Tvheadend

Edwin de Graaf wrote:

I see quite high memory usage of Tvheadend, so much that the Tvheadend process was killed on a 1GB memory VM because the memory was full.

As discussed in forum thread https://tvheadend.org/boards/5/topics/18091

This is the valgrind ouput included as a file.

Right now:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 6375 0.0 1.1 742112 23220 ? Ssl 09:35 0:01 tvheadend -f -u hts -g video

I'm no valgrind expert but that looks fine to me, btw you may want to install tvheadend-dbg package so it can fill in the gaps (those pesky ??? lines).

It could be a leak with hdhomerun which I think Tvheadend uses a built in lib.

1. free -m
   total used free shared buff/cache available
   Mem: 12078 286 68 65 11724 11638

Seems to not use much memory ;)

Rafal Kupiec wrote:

1. free -m
   total used free shared buff/cache available
   Mem: 12078 286 68 65 11724 11638

Seems to not use much memory ;)

This happens when the VM has 1GB of memory:

Oct 14 03:37:31 tvheadend kernel: [152017.938645] Out of memory: Kill process 3658 (tvheadend) score 614 or sacrifice child
Oct 14 03:37:31 tvheadend kernel: [152017.939386] Killed process 3658 (tvheadend) total-vm:2151828kB, anon-rss:237868kB, file-rss:0kB

But I don't mind if this issue is closed. It works for me when I give the VM 2GB.

in memory timeshift buffer???

B C wrote:

in memory timeshift buffer???

Timeshift is not enabled in the configuration.

I'm not sure if it's tvh which is causing the OOM. The key point is RSS. Could you also look when the RSS increased for the tvh process (an action started - see log) ?

egraaf@tvheadend:~/tvheadend/tvheadend$ zcat /var/log/syslog.?.gz | grep "invoked\|Tainted" 
Oct 14 03:37:31 tvheadend kernel: [152017.930233] tvhdhomerun_fro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 14 03:37:31 tvheadend kernel: [152017.930913] CPU: 0 PID: 5780 Comm: tvhdhomerun_fro Tainted: GF          O 3.13.0-24-generic #47-Ubuntu
Oct 13 02:05:04 tvheadend kernel: [60094.969473] tvhdhomerun_fro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 13 02:05:04 tvheadend kernel: [60094.970169] CPU: 0 PID: 3643 Comm: tvhdhomerun_fro Tainted: GF          O 3.13.0-24-generic #47-Ubuntu

It seems to happen when data is read from HD Homerun (not too surprising). And at 02:04 the EPG update starts (tv_grab_nl), so that also seems related.

RSS for Tvheadend when oom-killer activates is about 60MB, and we are out of SWAP:

Oct 14 03:37:31 tvheadend kernel: [152017.938594] Free swap  = 0kB
Oct 14 03:37:31 tvheadend kernel: [152017.938594] Total swap = 1044476kB
Oct 14 03:37:31 tvheadend kernel: [152017.938595] 262030 pages RAM
Oct 14 03:37:31 tvheadend kernel: [152017.938595] 0 pages HighMem/MovableOnly
Oct 14 03:37:31 tvheadend kernel: [152017.938596] 7531 pages reserved
Oct 14 03:37:31 tvheadend kernel: [152017.938596] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Oct 14 03:37:31 tvheadend kernel: [152017.938605] [  315]     0   315     4868        0      14       80             0 upstart-udev-br
Oct 14 03:37:31 tvheadend kernel: [152017.938607] [  319]     0   319    12805        1      28      143         -1000 systemd-udevd
Oct 14 03:37:31 tvheadend kernel: [152017.938608] [  378]   102   378     9806        0      23      107             0 dbus-daemon
Oct 14 03:37:31 tvheadend kernel: [152017.938610] [  420]     0   420    10862        1      27       90             0 systemd-logind
Oct 14 03:37:31 tvheadend kernel: [152017.938612] [  421]     0   421     5869        1      17       63             0 rpc.idmapd
Oct 14 03:37:31 tvheadend kernel: [152017.938613] [  422]     0   422     8228        1      21      102             0 rpc.gssd
Oct 14 03:37:31 tvheadend kernel: [152017.938614] [  433]   101   433    63960        0      28      256             0 rsyslogd
Oct 14 03:37:31 tvheadend kernel: [152017.938616] [  472]     0   472     3851        1      12      102             0 upstart-file-br
Oct 14 03:37:31 tvheadend kernel: [152017.938617] [  521]     0   521     2557       28       9      546             0 dhclient
Oct 14 03:37:31 tvheadend kernel: [152017.938619] [  589]     0   589    41641       23      18      546             0 userhdhomerun
Oct 14 03:37:31 tvheadend kernel: [152017.938620] [  737]     0   737     5855       14      17       62             0 rpcbind
Oct 14 03:37:31 tvheadend kernel: [152017.938622] [  803]   107   803     5385        1      16      115             0 rpc.statd
Oct 14 03:37:31 tvheadend kernel: [152017.938623] [  957]     0   957     3954        1      13       39             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938624] [  960]     0   960     3954        1      13       39             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938626] [  965]     0   965     3954        1      13       41             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938627] [  966]     0   966     3954        1      13       40             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938628] [  968]     0   968     3954        1      13       40             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938630] [ 1007]     0  1007    15341        0      34      169         -1000 sshd
Oct 14 03:37:31 tvheadend kernel: [152017.938631] [ 1013]     0  1013     1091        0       8       37             0 acpid
Oct 14 03:37:31 tvheadend kernel: [152017.938632] [ 1014]     0  1014     5913       17      18       38             0 cron
Oct 14 03:37:31 tvheadend kernel: [152017.938633] [ 1015]     0  1015     4784        0      13       49             0 atd
Oct 14 03:37:31 tvheadend kernel: [152017.938635] [ 1078]     0  1078     3989        0      12      213             0 upstart-socket-
Oct 14 03:37:31 tvheadend kernel: [152017.938636] [ 1116]     0  1116     3954        1      12       40             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938637] [ 1361]   106  1361     7861       30      19      121             0 ntpd
Oct 14 03:37:31 tvheadend kernel: [152017.938639] [ 3658]   105  3658   537957    59467     694   256683             0 tvheadend
Oct 14 03:37:31 tvheadend kernel: [152017.938641] [ 4323]     0  4323    26408        0      56      248             0 sshd
Oct 14 03:37:31 tvheadend kernel: [152017.938642] [ 4412]  1000  4412    26408        0      53      251             0 sshd
Oct 14 03:37:31 tvheadend kernel: [152017.938643] [ 4413]  1000  4413     5724        1      16      584             0 bash
Oct 14 03:37:31 tvheadend kernel: [152017.938645] Out of memory: Kill process 3658 (tvheadend) score 614 or sacrifice child

I found a critical error which may cause this. Could you upgrade to latest 4.0 code and retest ?

OK. If you do simple streaming (TV watch) and TVH does nothing else (epggrab), does the memory usage increase continuously ?

No, memory usage is very stable, both VSZ and RSS.