Tvheadend

Describe more the crash. Ideally, provide the backtrace and watch the memory usage. Also, check the memory info tab for values (if something goes high).

Jaroslav Kysela wrote:

Describe more the crash. Ideally, provide the backtrace and watch the memory usage. Also, check the memory info tab for values (if something goes high).

This morning I looked at the cpu load and memory consumption and had 4 TV recordings in parallel.
Memory consumption stayed pretty much constant at about 390 Mbyte and CPU load for the TVheadend process was at about 15%.
Then CPU load went up to 25% and TVheadend was not reponding, nor did it further record the TV shows.

Then I observed a drop of memory consumption to about 320 MB and a CPU load of only 3-5 percent.

I had to stop the observation but I expect that the tvheadend process will be terminated a bit later.

If you need a backtrace, then I would need instructions on how to generate this information.

some additional information.

1. the 4 recordings did not crash simulatiously. One recording lastet 1:50 min, the other 3:00 - 4:00 minutes.
2. and as expected the server crashed finally later during the day.

I made another test, only recording 2 programs.

I have noticed a dramatically lower memory consumption which was 25 MB.
This is a factor of 15!

Recording of 2 programs persisted more than an hour and up to now did not crash the server

It looks like another problem. For DVR operation the buffers are unlimited, so if you have limited disk I/O bandwidth, you may hit this memory usage. It should be visible in the Configuration / Debugging / Memory information entries tab (Packets entries). In other words - it seems that your hw is not capable to handle this load.

ok. Interesting. thands for the feedback.
The version 3.x did not have the problem and it was running on the same hardware.
With respect of packet entries there are several item which refer to packets.

e.g. packet size 2650 peak 9504 count of objects 55 peak count of object 198.

How can an verify that your assumption is really true.

Also the harddisk are internal harddisk which should have orders of magnitude than a mp4 ts video stream.

OK, if 3.4 works my hypothesis is probably wrong. The peak count 200 is fine. Just try to look to these values, if tvh process grabs hundreds megs of memory.

I did some tests:

No parallel watching of TV recording.

HDTV Recording
one HDTV recording active: memory:190 MB, peak count: 28.
two HDTV recording active:
memory: 206 MB, growing to 250 MB after 1 Minute, 288 MB after 2 Min, 302 Mb after 2,5 min, 335 MB, 379 MB (here 93 % of memory is used).
This memory is kind of stable at about 370 -379 MB at a CPU Load of 18 %. But Webui does not respond anymore.
peak count:28

So far TVheadend did not crash but I cannot be control neither by Kodi nor Webui. Expect crash soon.

EPG Grabbing
CPU 26%, Memory 26 MB. All 4 tuners showed activities.

Restarted and did testing with SD TV streams.
one SDTV recording active: memory 32 MB, after 2 min 91 MB steadly growing, 3 min 117 MB, 4 min 150 MB,6 min 210 MB,8 291 MB,
After recording stops (the TV program was over and tvheadend idle) the memory was 268 MB constant.
It seems Tvheadend does not release the memory.
After 3 minutes idle the memory droped to 142 MB!

To me, the whole behaviour looks like a memory leak.

Unfortunately, I cannot reproduce this behaviour here on x86_64 - recording 4 streams, the RSS does not change in 5 minutes (stays around 110MB while 100MB was allocated on boot - I have many services here).

Do you use 32-bit binaries ?

The memory consumption should stay around value as you noted for 'EPG grabbing'. If you don't see the increasing packet peak counter, then the allocated memory does not belong to the stream data (packets).

In this case, it would be probably good to use some tools to detect the memory allocations like valgrind, but you need to have a debug version of the tvh binary (with the debug symbols).

I think the ARMADA 370 CPU is a 32 bit ARM CPU, therefore I assume I am using 32 bit binaries.

I am afraid using debug tools is beyond my abilities.
Also I am not able to "make" an ARMADA 370 debug build.

So I guess I am stuck!

Tested V 4.1.1953-1. Same behaviour.

Same Problem on a Synology DS213+ and version 4.1.1953-1 of tvheadend testing.
DSM version is 6.0-7321 Update 3.
The package doesnt even start, protocols are empty. No slowing down of the server recognized.

@Sylvan:
Your problem is not related.
If the server is crashing before having done anything, it is no surprise that you are see no slowing down.

Further tests with version 4.1.1996

TVheadend freshly started.

EPG Grabbing: 28 MB

Recording of 1 HDTV: 29 MB, konstant. Memory leak does not show up
Recording of 2 HDTV: Memory leak is appearing. Webui is still functional.
At about 210 MB I am stopping one recording. After this memory does not grow again but stays at 210 MB

Adding another SDTV Recording. Memory is staying at about 210 MB.
Adding antoher SDTV Recording. Memory is staying at about 210 MB. Gradually increasing again reaching 370 MB.
Trying to cancel the HDTV Recording is not possible because WebUi is not accepting the user action.

Killed TV Headend

TVheadend freshly startet
EPG Grabbing started at 17 MB ending at about 28 MB.

First SDTV Recording started. Memory is staying at about 28 MB. No increase of memory
added 2nd SDTV Recording started. Memory is not increasing.
added 3nd SDTV Recording. Noticed overall CPU of 100%: Memory is steady increasing.

Trying cancelled one of the SDTV Recordings. Webui is not executing request immediately but after some time only two SDTV
Recordings are active. Memory stays constand at 236 MB not increase, CPU total 93%
Cancelled another SDTV Recording. Memory usage drops to 68 MB, CPU total 67%
Cancelled remaining SDTV Recording. Memory stays at 68 MB.

• I seems that the CPU is overloaded and is not able to free memory.
• This load is dependent of the whether SD or HD TV is recorded.
• For HDTV basically only one recording can be active and potentially one additional SD Recording. (not quite sure on this)
• For SDTV two recording can be active.

@Jaroslav. Do you think increasing the priority of tvheadend is sensible?

I have seen you run Sat>IP. Are you using perexg's firmware on your Sat>IP box by any chance? If so, what minisatip version do you use?

I am using a Grundig /GSS DSI 400 GSS.box Sat IP Receiver with the software it came with and have no further information about any specific details of related module.

OK, I have had lots and lots of trouble until I started using perexg's firmware.

You will find it here:
https://github.com/perexg/satip-axe/tree/master/dist

You can try it from an USB stick without flashing. I recommend fw-11, but use it with standard minisatip configuration, NOT minisatip5.

And perexg = Jaroslav, actually... ;-)

Hi Martin,

I looked at the Github and saw some instruction how to flash my Grundig GSS receiver but I did not find a good explanation what this software actually improves and changes.

• Being able to stream / record multiple streams at the same time (didn't work with original firmware on my Digibit, which is a rebranded version)
• Lots and lots of continuity errors
• At least perceived stability. Had crashes from time to time and they disappeared. But I simulataneously updated to more and more recent 4.1.x backend versions, so cannot attribute the effect with certaintly to one of the other.
• I did have issues with the newer minisatip5 version though, and this is why I warned you about that.

Again, to be frank, I'm not sure that will solve your problem but I know what I have experienced and I would put perexg's firmware on an USB stick and try, at least to exclude the firmware to be causing (at least some of) your problems.

Hi Martin, thanks for the explanation.
I will try this new version and see what I will be getting. Booting from USB Stick seems to be a pretty safe approach.

I think I ran it from a USB stick for about three months before I dared flashing.

As a heads up. Finding a USB stick that works can be painful. Only 1 out of 5 USB sticks I have actually works with my SAT>IP box. Just if you should not see the "knight rider effect"...

HI,
I have installed the minsatip server and it work pretty nicely out of the box.
In TVheadend I noticed that the SNR Ratio that is shown much better.

With respect to my memory leak though, it was not the solution. Somehow the leak start a bit later.
I can not record two hd TV stream but additing another sd stream showed the same problem.

Thanks for the support anyway.

I am also testing the dvblink commerical server.
the minisatip is not recognized by dvbserver.

Could you try v4.1-2033-g99f199f ? I added some queue checks which drops the incoming data when CPU is not able to handle them. Also, new entries 'MPEG-TS input/table queue' were added to check if the data are not processed.

Also, it would be nice to know, where CPU spends the time - if you have 'top -H -p <pid_of_tvh>' command, could you send the output from it when CPU is almost on 100% ?

Also, you may try to use 'pass' profile for DVR (if you don't do this already). It should use less CPU than matroska.

OK, it seems that your version of top does not help us, too much. The right output should be:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                
 6585 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:02.74 tvheadend                                                                              
 6588 perex     20   0 1188540 124812  19776 S  0,0  0,8   1:14.89 tvh:log                                                                                
 6589 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.01 tvh:notify                                                                             
 6590 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.42 tvheadend                                                                              
 6591 perex     35  15 1188540 124812  19776 S  0,0  0,8   0:00.04 tvh:save                                                                               
 6592 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.83 tvh:mtick                                                                              
 6593 perex     39  19 1188540 124812  19776 S  0,0  0,8   0:00.28 tvh:tasklet                                                                            
 6594 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:fsmonitor                                                                          
 6595 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:imagecache                                                                         
 6596 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:httpc                                                                              
 6597 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:service                                                                            
 6598 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:iptv                                                                               
 6599 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.32 tvheadend                                                                              
 6600 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.48 tvh:hdhm-disc                                                                          
 6601 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:tshift-reap                                                                        
 6602 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.00 tvh:tcp-loop                                                                           
 6603 perex     20   0 1188540 124812  19776 S  0,0  0,8   0:00.94 tvh:upnp                                                                               
...

Version 2033 is available as a syno build.
Thanks to dierkse to create those builds so rapidly.

Unfortunately I cannot perform any tests in the next couple days because I am on biz trip.

I have installed and tested Version 4.1.2076.
Same behaviour, no improvement.

Same behaviour means what? Memory usage is increased?

1) show me 'top -H' when CPU is at 100%
2) watch memory info entries what increases (peaks)
3) look to logs if there are any suspicious lines

Memory Leak in Version 4.1.2140-1 on Armada 370 NOT resolved

Test on a 88f6281 Synology showed similar result. A
Also this Synology Plattform cannot support TVheadend sensibly.

I'm now unable to connect to the tv service. it is not responding..

Had to stop the service. after a restart it's reponding again.

The recording is fine, pictures is create no artifect.

I have installed Version 2190.
This version crashes as well. This seems to lead to nowhere.
Either my synology is too weak or the code is buggy.

I am giving up and will be moving to NextPVR running on a NUC.

The main problem that I can see is that volume usage is 100%. Which causes tvheadend not being able to write fast enough. Sabnzb has a disk speed test. That is able to write at about 80MB/s. Which should be more than enough for atleast 1 recording you would think.

You may try to change the 'Cache scheme' in 'DVR profiles' to 'System'. But the default 'Don't keep' should work, too...

Also, if the problem is slow packet handling, the packet counters in the 'memory info' debugging tab should be increasing....

Cache scheme is set to the defaul, Don't keep. Will try to set is to system

Setting the cache scheme to "System" seems to improve the situation.
I have successfully reordered three parallel HD TV streams without seeing the increase of memory.:)

I will do additional tests today.

Yes setting to system seems to solve the problem. I will do some further testing also, but 2 recordings have succeeded withuot the issue so far.

I have successfully recorded 5 parallel HD TV programs.
All went well and TVheadend is up and running. Memory did not grow and TVheadend stayed responsive.

Thanks for the support although it took 4 months to respond.

Yep recorded a few programs in parrallel without any problems.

Case closed as far as I concerned.

Cheers.

Jaroslav Kysela wrote:

Mark suggested to keep the settings to 'System' as default and I don't have arguments against this, so we "fixed" this in our code. v4.1-2221-ge02a5db

I personally prefer "don't keep", but if it prevents things like this from happening and it has no ill effects on other systems then it makes sense, there has to be a better way to detect issues like this though.

With the new setting TVheadend is stable since then.
About 30 recordings done. All flawless!

Great job!
The only weak point remaining it the pause function which seems to stop buffering after a very short time.