Project

General

Profile

Bug #6052

packet buffer number increasing until out-of-memory with 1GB on RPi while DVR recording to disk/NFS

Added by Alex Krupp over 3 years ago. Updated about 3 years ago.

Status:
Invalid
Priority:
Normal
Assignee:
-
Category:
PVR / DVR
Target version:
-
Start date:
2021-05-23
Due date:
% Done:

0%

Estimated time:
Found in version:
4.3-1964~g637844055
Affected Versions:

Description

tvheadend is "reaped" by the Linux oom killer after exhausting all available memory when using DVR. I have turned swapping off to stay with a responsive system at least. Streaming is fine. The issue shows only when recording. I have tested

  • 4.2.4-dmo1~bpo9+1~rpt1
  • current stable (4.2.8?)
  • and the current unstable 4.3-1964~g637844055

Suggestion: If a ring buffer is used for reusable packet buffers: maybe the write-to-disk code does not pass ownership to the allocator, again? Such a problem will not show on systems with todays amount of RAM (8GB or more + swap), as most recordings will be done within <5GB. I did not test with a short recording (<10 min.), whether the packet buffers will be deallocated after recording stops.

In such a case valgrind won't help much, because all memory is accounted for... Please find attached a valgrind log, regardless.


Files

valgrind2.log (55.7 KB) valgrind2.log valgrind --log-socket=127.0.0.1 --leak-check=yes --trace-children=yes /usr/bin/tvheadend -f -p /run/tvheadend.pid -u hts -g video Alex Krupp, 2021-05-23 10:34
Screenshot_2021-05-22 Tvheadend.png (78.2 KB) Screenshot_2021-05-22 Tvheadend.png huge amount of packet buffers, while increasing Alex Krupp, 2021-05-23 10:40
journal.txt (23.7 KB) journal.txt exemplary journal extract not related to valgrind2.log Alex Krupp, 2021-05-23 11:15

History

#1

Updated by Flole Systems over 3 years ago

It looks like you didn't have debug symbols for Tvheadend installed when you ran valgrind? Looks like it found something (not related to your issue, but still worth investigating).

Most likely your target is not fast enough for recording or the caching method is set to something inappropriate.

#2

Updated by Alex Krupp over 3 years ago

#3

Updated by Alex Krupp over 3 years ago

Flole Systems wrote:

It looks like you didn't have debug symbols for Tvheadend installed when you ran valgrind? Looks like it found something (not related to your issue, but still worth investigating).

I did not notice the -dbg package because I got distracted by the -dbgsym, will give this a try.

Most likely your target is not fast enough for recording or the caching method is set to something inappropriate.

The target is a diskstation connected offering a transfer rate of >11Mbit when copying via scp from the source. Streaming over the same connection + WLAN works perfectly. Recording is done via an NFS mount. The recorded files show up with content, so it does not seem to be permission/squashing related. Any more ideas?

#4

Updated by Alex Krupp over 3 years ago

Re. caching: Tvheadend has timeshift disabled. Cache scheme is set to "Don't keep" in the respective DVR profile.

#5

Updated by Alex Krupp over 3 years ago

Seems I have to check transmission bandwith to the target, indeed. I assumed, that the stream had a bandwidth of approx. 4-5 Mbit/s, while it is closer to 10Mbit/s. Thank you for the assistance.

I would prefer, though, to be able to impose an upper limit to packet buffers inside tvheadend to be able to avoid the oom killer.

#6

Updated by Alex Krupp over 3 years ago

If I switch from NFS to CIFS, recording works. And network bandwidth used (measured with iptraf-ng) stays at a similar rate of ~11Mbit.

I would prefer to use NFS, as it is said to require less resources (and be faster ?) than cifs, however it does not seem possible on this configuration, even though bonnie+ gives me similar throughput for NFS and CIFS.

I do not see how this would be a bug with tvheadend afterall, please feel free to close this.

#7

Updated by Flole Systems over 3 years ago

  • Status changed from New to Invalid
#8

Updated by Alex Krupp about 3 years ago

After checking throughput by means of bonnie++, nfs seems to be the faster choice, really. I found a blog entry where someone mentioned tvheadend on a Raspi Model B with 256 MB RAM, also recording via NFS. He described a similar problem and explained that he solved it by switching the "Recorder Profiles/Cache scheme" from "Do not keep" to "System". It is possible to record two channels simultaneously without packet buffers increasing on this system, now.

The Help text of tvheadend states: "Do not keep .... Useful e.g. in a RAM-limited system like a Pi (given that you’re unlikely to be watching while recording, so data can be discarded now and read back from disc later)." However, this setting seems to be causing problems on Raspbian 9, today.

I suggest to change the "Help" recommendation, and, maybe add information what differentiates "Do not keep" from "Sync".

#9

Updated by dj bloc about 3 years ago

Set cache to "System" (should be default). I had major performance penalties when using "Do not keep" on RPI 3 in the past.

Also available in: Atom PDF