Bug #3770
Crashes of TVheadend V. 4.1.1943 on Synology
0%
Description
Version 4.1x seems to have a general problem that causes the server firstly to slow done and finally to crash.
Probably a memory leak.
I am running Version 4.1.1943 under DSM 6.0-7321 Update 3 but also previous version have been affected aswell.
Version 3.x did not show this behaviour.
I am using Sat>ip and runing on a Synology DS 213J using the Armada 370 build provided under http://syno.dierkse.nl/.
As frontend I am using Kodi 15.2.
The time to crash depends on the activities
When recording e.g. 5 TV programs simulateously, crash is occuring after about 30 minutes.
Just watching TV without other recordings, the server may work for about 12-24 h.
Files
History
Updated by Jaroslav Kysela over 8 years ago
Describe more the crash. Ideally, provide the backtrace and watch the memory usage. Also, check the memory info tab for values (if something goes high).
Updated by Martin Klayer over 8 years ago
Jaroslav Kysela wrote:
Describe more the crash. Ideally, provide the backtrace and watch the memory usage. Also, check the memory info tab for values (if something goes high).
This morning I looked at the cpu load and memory consumption and had 4 TV recordings in parallel.
Memory consumption stayed pretty much constant at about 390 Mbyte and CPU load for the TVheadend process was at about 15%.
Then CPU load went up to 25% and TVheadend was not reponding, nor did it further record the TV shows.
Then I observed a drop of memory consumption to about 320 MB and a CPU load of only 3-5 percent.
I had to stop the observation but I expect that the tvheadend process will be terminated a bit later.
Updated by Martin Klayer over 8 years ago
If you need a backtrace, then I would need instructions on how to generate this information.
Updated by Martin Klayer over 8 years ago
some additional information.
1. the 4 recordings did not crash simulatiously. One recording lastet 1:50 min, the other 3:00 - 4:00 minutes.
2. and as expected the server crashed finally later during the day.
Updated by Martin Klayer over 8 years ago
I made another test, only recording 2 programs.
I have noticed a dramatically lower memory consumption which was 25 MB.
This is a factor of 15!
Recording of 2 programs persisted more than an hour and up to now did not crash the server
Updated by Jaroslav Kysela over 8 years ago
It looks like another problem. For DVR operation the buffers are unlimited, so if you have limited disk I/O bandwidth, you may hit this memory usage. It should be visible in the Configuration / Debugging / Memory information entries tab (Packets entries). In other words - it seems that your hw is not capable to handle this load.
Updated by Martin Klayer over 8 years ago
ok. Interesting. thands for the feedback.
The version 3.x did not have the problem and it was running on the same hardware.
With respect of packet entries there are several item which refer to packets.
e.g. packet size 2650 peak 9504 count of objects 55 peak count of object 198.
How can an verify that your assumption is really true.
Also the harddisk are internal harddisk which should have orders of magnitude than a mp4 ts video stream.
Updated by Jaroslav Kysela over 8 years ago
OK, if 3.4 works my hypothesis is probably wrong. The peak count 200 is fine. Just try to look to these values, if tvh process grabs hundreds megs of memory.
Updated by Martin Klayer over 8 years ago
I did some tests:
No parallel watching of TV recording.
HDTV Recording
one HDTV recording active: memory:190 MB, peak count: 28.
two HDTV recording active:
memory: 206 MB, growing to 250 MB after 1 Minute, 288 MB after 2 Min, 302 Mb after 2,5 min, 335 MB, 379 MB (here 93 % of memory is used).
This memory is kind of stable at about 370 -379 MB at a CPU Load of 18 %. But Webui does not respond anymore.
peak count:28
So far TVheadend did not crash but I cannot be control neither by Kodi nor Webui. Expect crash soon.
EPG Grabbing
CPU 26%, Memory 26 MB. All 4 tuners showed activities.
Restarted and did testing with SD TV streams.
one SDTV recording active: memory 32 MB, after 2 min 91 MB steadly growing, 3 min 117 MB, 4 min 150 MB,6 min 210 MB,8 291 MB,
After recording stops (the TV program was over and tvheadend idle) the memory was 268 MB constant.
It seems Tvheadend does not release the memory.
After 3 minutes idle the memory droped to 142 MB!
To me, the whole behaviour looks like a memory leak.
Updated by Jaroslav Kysela over 8 years ago
Unfortunately, I cannot reproduce this behaviour here on x86_64 - recording 4 streams, the RSS does not change in 5 minutes (stays around 110MB while 100MB was allocated on boot - I have many services here).
Do you use 32-bit binaries ?
The memory consumption should stay around value as you noted for 'EPG grabbing'. If you don't see the increasing packet peak counter, then the allocated memory does not belong to the stream data (packets).
In this case, it would be probably good to use some tools to detect the memory allocations like valgrind, but you need to have a debug version of the tvh binary (with the debug symbols).
Updated by Martin Klayer over 8 years ago
I think the ARMADA 370 CPU is a 32 bit ARM CPU, therefore I assume I am using 32 bit binaries.
I am afraid using debug tools is beyond my abilities.
Also I am not able to "make" an ARMADA 370 debug build.
So I guess I am stuck!
Updated by Sylvan Deity over 8 years ago
Same Problem on a Synology DS213+ and version 4.1.1953-1 of tvheadend testing.
DSM version is 6.0-7321 Update 3.
The package doesnt even start, protocols are empty. No slowing down of the server recognized.
Updated by Martin Klayer over 8 years ago
@Sylvan:
Your problem is not related.
If the server is crashing before having done anything, it is no surprise that you are see no slowing down.
Updated by Martin Klayer over 8 years ago
Further tests with version 4.1.1996
TVheadend freshly started.
EPG Grabbing: 28 MB
Recording of 1 HDTV: 29 MB, konstant. Memory leak does not show up
Recording of 2 HDTV: Memory leak is appearing. Webui is still functional.
At about 210 MB I am stopping one recording. After this memory does not grow again but stays at 210 MB
Adding another SDTV Recording. Memory is staying at about 210 MB.
Adding antoher SDTV Recording. Memory is staying at about 210 MB. Gradually increasing again reaching 370 MB.
Trying to cancel the HDTV Recording is not possible because WebUi is not accepting the user action.
Killed TV Headend
TVheadend freshly startet
EPG Grabbing started at 17 MB ending at about 28 MB.
First SDTV Recording started. Memory is staying at about 28 MB. No increase of memory
added 2nd SDTV Recording started. Memory is not increasing.
added 3nd SDTV Recording. Noticed overall CPU of 100%: Memory is steady increasing.
Trying cancelled one of the SDTV Recordings. Webui is not executing request immediately but after some time only two SDTV
Recordings are active. Memory stays constand at 236 MB not increase, CPU total 93%
Cancelled another SDTV Recording. Memory usage drops to 68 MB, CPU total 67%
Cancelled remaining SDTV Recording. Memory stays at 68 MB.
- I seems that the CPU is overloaded and is not able to free memory.
- This load is dependent of the whether SD or HD TV is recorded.
- For HDTV basically only one recording can be active and potentially one additional SD Recording. (not quite sure on this)
- For SDTV two recording can be active.
@Jaroslav. Do you think increasing the priority of tvheadend is sensible?
Updated by Martin Walter over 8 years ago
I have seen you run Sat>IP. Are you using perexg's firmware on your Sat>IP box by any chance? If so, what minisatip version do you use?
Updated by Martin Klayer over 8 years ago
I am using a Grundig /GSS DSI 400 GSS.box Sat IP Receiver with the software it came with and have no further information about any specific details of related module.
Updated by Martin Walter over 8 years ago
OK, I have had lots and lots of trouble until I started using perexg's firmware.
You will find it here:
https://github.com/perexg/satip-axe/tree/master/dist
You can try it from an USB stick without flashing. I recommend fw-11, but use it with standard minisatip configuration, NOT minisatip5.
And perexg = Jaroslav, actually... ;-)
Updated by Martin Klayer over 8 years ago
Hi Martin,
I looked at the Github and saw some instruction how to flash my Grundig GSS receiver but I did not find a good explanation what this software actually improves and changes.
Updated by Martin Walter over 8 years ago
- Being able to stream / record multiple streams at the same time (didn't work with original firmware on my Digibit, which is a rebranded version)
- Lots and lots of continuity errors
- At least perceived stability. Had crashes from time to time and they disappeared. But I simulataneously updated to more and more recent 4.1.x backend versions, so cannot attribute the effect with certaintly to one of the other.
- I did have issues with the newer minisatip5 version though, and this is why I warned you about that.
Again, to be frank, I'm not sure that will solve your problem but I know what I have experienced and I would put perexg's firmware on an USB stick and try, at least to exclude the firmware to be causing (at least some of) your problems.
Updated by Martin Klayer over 8 years ago
Hi Martin, thanks for the explanation.
I will try this new version and see what I will be getting. Booting from USB Stick seems to be a pretty safe approach.
Updated by Martin Walter over 8 years ago
I think I ran it from a USB stick for about three months before I dared flashing.
As a heads up. Finding a USB stick that works can be painful. Only 1 out of 5 USB sticks I have actually works with my SAT>IP box. Just if you should not see the "knight rider effect"...
Updated by Martin Klayer over 8 years ago
HI,
I have installed the minsatip server and it work pretty nicely out of the box.
In TVheadend I noticed that the SNR Ratio that is shown much better.
With respect to my memory leak though, it was not the solution. Somehow the leak start a bit later.
I can not record two hd TV stream but additing another sd stream showed the same problem.
Thanks for the support anyway.
Updated by Martin Klayer over 8 years ago
I am also testing the dvblink commerical server.
the minisatip is not recognized by dvbserver.
Updated by Jaroslav Kysela over 8 years ago
Could you try v4.1-2033-g99f199f ? I added some queue checks which drops the incoming data when CPU is not able to handle them. Also, new entries 'MPEG-TS input/table queue' were added to check if the data are not processed.
Also, it would be nice to know, where CPU spends the time - if you have 'top -H -p <pid_of_tvh>' command, could you send the output from it when CPU is almost on 100% ?
Also, you may try to use 'pass' profile for DVR (if you don't do this already). It should use less CPU than matroska.
Updated by Martin Klayer over 8 years ago
Hi Jaroslav,
I will do so when this build is available under http://syno.dierkse.nl/
As I said before, I am not the Linux Guru who can create a synology build by himself.
I have tried to create the requested trace using the 1996 version.
Hope that provides some insight.
regards Martin
Updated by Jaroslav Kysela over 8 years ago
OK, it seems that your version of top does not help us, too much. The right output should be:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6585 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:02.74 tvheadend 6588 perex 20 0 1188540 124812 19776 S 0,0 0,8 1:14.89 tvh:log 6589 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.01 tvh:notify 6590 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.42 tvheadend 6591 perex 35 15 1188540 124812 19776 S 0,0 0,8 0:00.04 tvh:save 6592 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.83 tvh:mtick 6593 perex 39 19 1188540 124812 19776 S 0,0 0,8 0:00.28 tvh:tasklet 6594 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:fsmonitor 6595 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:imagecache 6596 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:httpc 6597 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:service 6598 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:iptv 6599 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.32 tvheadend 6600 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.48 tvh:hdhm-disc 6601 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:tshift-reap 6602 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.00 tvh:tcp-loop 6603 perex 20 0 1188540 124812 19776 S 0,0 0,8 0:00.94 tvh:upnp ...
Updated by Martin Klayer over 8 years ago
I used another option.
Should be more in the way you wanted it.
Updated by Martin Klayer over 8 years ago
Version 2033 is available as a syno build.
Thanks to dierkse to create those builds so rapidly.
Unfortunately I cannot perform any tests in the next couple days because I am on biz trip.
Updated by Martin Klayer over 8 years ago
I have installed and tested Version 4.1.2076.
Same behaviour, no improvement.
Updated by Jaroslav Kysela over 8 years ago
Same behaviour means what? Memory usage is increased?
1) show me 'top -H' when CPU is at 100%
2) watch memory info entries what increases (peaks)
3) look to logs if there are any suspicious lines
Updated by Martin Klayer over 8 years ago
Memory Leak in Version 4.1.2140-1 on Armada 370 NOT resolved
Updated by Martin Klayer over 8 years ago
Test on a 88f6281 Synology showed similar result. A
Also this Synology Plattform cannot support TVheadend sensibly.
Updated by Roland Sjouw about 8 years ago
- File 2016-08-21 09_40_07-Welcome.png 2016-08-21 09_40_07-Welcome.png added
- File 2016-08-21 09_41_34-Tvheadend.png 2016-08-21 09_41_34-Tvheadend.png added
- File 2016-08-21 09_42_56-192.168.178.200 - PuTTY.png 2016-08-21 09_42_56-192.168.178.200 - PuTTY.png added
Also have the same issue on synology.
DS1513+
During recording the memory usage keeps increasing. See memory usage screenshot.
In the debugging of TVHeadend memory information entries.
The packet only increases. See screenshot debugging.
Disk utilization is only 25%. See disk screenshot.
Updated by Roland Sjouw about 8 years ago
- File 2016-08-21 10_01_52-.png 2016-08-21 10_01_52-.png added
when recoring stopped, at 10:00, there where errors in the log, see screenshot.
Updated by Roland Sjouw about 8 years ago
I'm now unable to connect to the tv service. it is not responding..
Updated by Roland Sjouw about 8 years ago
Had to stop the service. after a restart it's reponding again.
The recording is fine, pictures is create no artifect.
Updated by Roland Sjouw about 8 years ago
- File IMG_0175.PNG IMG_0175.PNG added
Running version, screenshot
Updated by Martin Klayer about 8 years ago
I have installed Version 2190.
This version crashes as well. This seems to lead to nowhere.
Either my synology is too weak or the code is buggy.
I am giving up and will be moving to NextPVR running on a NUC.
Updated by Roland Sjouw about 8 years ago
The main problem that I can see is that volume usage is 100%. Which causes tvheadend not being able to write fast enough. Sabnzb has a disk speed test. That is able to write at about 80MB/s. Which should be more than enough for atleast 1 recording you would think.
Updated by Jaroslav Kysela about 8 years ago
You may try to change the 'Cache scheme' in 'DVR profiles' to 'System'. But the default 'Don't keep' should work, too...
Updated by Jaroslav Kysela about 8 years ago
Also, if the problem is slow packet handling, the packet counters in the 'memory info' debugging tab should be increasing....
Updated by Roland Sjouw about 8 years ago
Cache scheme is set to the defaul, Don't keep. Will try to set is to system
Updated by Martin Klayer about 8 years ago
Setting the cache scheme to "System" seems to improve the situation.
I have successfully reordered three parallel HD TV streams without seeing the increase of memory.
I will do additional tests today.
Updated by Roland Sjouw about 8 years ago
Yes setting to system seems to solve the problem. I will do some further testing also, but 2 recordings have succeeded withuot the issue so far.
Updated by Martin Klayer about 8 years ago
I have successfully recorded 5 parallel HD TV programs.
All went well and TVheadend is up and running. Memory did not grow and TVheadend stayed responsive.
Thanks for the support although it took 4 months to respond.
Updated by Roland Sjouw about 8 years ago
Yep recorded a few programs in parrallel without any problems.
Case closed as far as I concerned.
Cheers.
Updated by Jaroslav Kysela about 8 years ago
- Status changed from New to Invalid
Technical details: TVH uses "posix_fadvise(fd, pos, size, POSIX_FADV_DONTNEED);" call when 'Don't keep' settings is used in DVR profiles. I don't know why it causes delayed writes on Synology. The call should not flush data but inform kernel to not keep these data in the system cache.
If it's an issue with the used platform, I suggest to the package maintained to set 'System' as default for tvh packages:
diff --git a/src/dvr/dvr_config.c b/src/dvr/dvr_config.c index 4785d98..45e6f90 100644 --- a/src/dvr/dvr_config.c +++ b/src/dvr/dvr_config.c @@ -190,7 +190,7 @@ dvr_config_create(const char *name, const char *uuid, htsmsg_t *conf) cfg->dvr_cleanup_threshold_used = 0; // disabled /* Muxer config */ - cfg->dvr_muxcnf.m_cache = MC_CACHE_DONTKEEP; + cfg->dvr_muxcnf.m_cache = MC_CACHE_SYSTEM; /* Default recording file and directory permissions */
Closing. It's not a tvh bug.
Updated by Jaroslav Kysela about 8 years ago
- Status changed from Invalid to Fixed
Mark suggested to keep the settings to 'System' as default and I don't have arguments against this, so we "fixed" this in our code. v4.1-2221-ge02a5db
Updated by Mark Clarkstone about 8 years ago
Jaroslav Kysela wrote:
Mark suggested to keep the settings to 'System' as default and I don't have arguments against this, so we "fixed" this in our code. v4.1-2221-ge02a5db
I personally prefer "don't keep", but if it prevents things like this from happening and it has no ill effects on other systems then it makes sense, there has to be a better way to detect issues like this though.
Updated by Martin Klayer about 8 years ago
With the new setting TVheadend is stable since then.
About 30 recordings done. All flawless!
Great job!
The only weak point remaining it the pause function which seems to stop buffering after a very short time.