Bug #6052
packet buffer number increasing until out-of-memory with 1GB on RPi while DVR recording to disk/NFS
0%
Description
tvheadend is "reaped" by the Linux oom killer after exhausting all available memory when using DVR. I have turned swapping off to stay with a responsive system at least. Streaming is fine. The issue shows only when recording. I have tested
- 4.2.4-dmo1~bpo9+1~rpt1
- current stable (4.2.8?)
- and the current unstable 4.3-1964~g637844055
Suggestion: If a ring buffer is used for reusable packet buffers: maybe the write-to-disk code does not pass ownership to the allocator, again? Such a problem will not show on systems with todays amount of RAM (8GB or more + swap), as most recordings will be done within <5GB. I did not test with a short recording (<10 min.), whether the packet buffers will be deallocated after recording stops.
In such a case valgrind won't help much, because all memory is accounted for... Please find attached a valgrind log, regardless.
Files
History
Updated by Flole Systems over 3 years ago
It looks like you didn't have debug symbols for Tvheadend installed when you ran valgrind? Looks like it found something (not related to your issue, but still worth investigating).
Most likely your target is not fast enough for recording or the caching method is set to something inappropriate.
Updated by Alex Krupp over 3 years ago
Flole Systems wrote:
It looks like you didn't have debug symbols for Tvheadend installed when you ran valgrind? Looks like it found something (not related to your issue, but still worth investigating).
I did not notice the -dbg package because I got distracted by the -dbgsym, will give this a try.
Most likely your target is not fast enough for recording or the caching method is set to something inappropriate.
The target is a diskstation connected offering a transfer rate of >11Mbit when copying via scp from the source. Streaming over the same connection + WLAN works perfectly. Recording is done via an NFS mount. The recorded files show up with content, so it does not seem to be permission/squashing related. Any more ideas?
Updated by Alex Krupp over 3 years ago
Re. caching: Tvheadend has timeshift disabled. Cache scheme is set to "Don't keep" in the respective DVR profile.
Updated by Alex Krupp over 3 years ago
Seems I have to check transmission bandwith to the target, indeed. I assumed, that the stream had a bandwidth of approx. 4-5 Mbit/s, while it is closer to 10Mbit/s. Thank you for the assistance.
I would prefer, though, to be able to impose an upper limit to packet buffers inside tvheadend to be able to avoid the oom killer.
Updated by Alex Krupp over 3 years ago
If I switch from NFS to CIFS, recording works. And network bandwidth used (measured with iptraf-ng) stays at a similar rate of ~11Mbit.
I would prefer to use NFS, as it is said to require less resources (and be faster ?) than cifs, however it does not seem possible on this configuration, even though bonnie+ gives me similar throughput for NFS and CIFS.
I do not see how this would be a bug with tvheadend afterall, please feel free to close this.
Updated by Alex Krupp about 3 years ago
After checking throughput by means of bonnie++, nfs seems to be the faster choice, really. I found a blog entry where someone mentioned tvheadend on a Raspi Model B with 256 MB RAM, also recording via NFS. He described a similar problem and explained that he solved it by switching the "Recorder Profiles/Cache scheme" from "Do not keep" to "System". It is possible to record two channels simultaneously without packet buffers increasing on this system, now.
The Help text of tvheadend states: "Do not keep .... Useful e.g. in a RAM-limited system like a Pi (given that you’re unlikely to be watching while recording, so data can be discarded now and read back from disc later)." However, this setting seems to be causing problems on Raspbian 9, today.
I suggest to change the "Help" recommendation, and, maybe add information what differentiates "Do not keep" from "Sync".
Updated by dj bloc about 3 years ago
Set cache to "System" (should be default). I had major performance penalties when using "Do not keep" on RPI 3 in the past.