Project

General

Profile

Bug #3157

High memory usage

Added by Edwin de Graaf about 9 years ago. Updated about 9 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
General
Target version:
-
Start date:
2015-10-15
Due date:
% Done:

0%

Estimated time:
Found in version:
4.0.6-9~ga1170f9
Affected Versions:

Description

I see quite high memory usage of Tvheadend, so much that the Tvheadend process was killed on a 1GB memory VM because the memory was full.

As discussed in forum thread https://tvheadend.org/boards/5/topics/18091

This is the valgrind ouput included as a file.

Right now:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 6375 0.0 1.1 742112 23220 ? Ssl 09:35 0:01 tvheadend -f -u hts -g video


Files

valgrind-output.log (21.4 KB) valgrind-output.log Edwin de Graaf, 2015-10-15 10:23
pmap.txt (9.83 KB) pmap.txt Edwin de Graaf, 2015-10-19 15:06

History

#1

Updated by Mark Clarkstone about 9 years ago

Edwin de Graaf wrote:

I see quite high memory usage of Tvheadend, so much that the Tvheadend process was killed on a 1GB memory VM because the memory was full.

As discussed in forum thread https://tvheadend.org/boards/5/topics/18091

This is the valgrind ouput included as a file.

Right now:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 6375 0.0 1.1 742112 23220 ? Ssl 09:35 0:01 tvheadend -f -u hts -g video

I'm no valgrind expert but that looks fine to me, btw you may want to install tvheadend-dbg package so it can fill in the gaps (those pesky ??? lines).

It could be a leak with hdhomerun which I think Tvheadend uses a built in lib.

#2

Updated by Rafal Kupiec about 9 years ago

  1. free -m
    total used free shared buff/cache available
    Mem: 12078 286 68 65 11724 11638

Seems to not use much memory ;)

#3

Updated by Edwin de Graaf about 9 years ago

Rafal Kupiec wrote:

  1. free -m
    total used free shared buff/cache available
    Mem: 12078 286 68 65 11724 11638

Seems to not use much memory ;)

This happens when the VM has 1GB of memory:

Oct 14 03:37:31 tvheadend kernel: [152017.938645] Out of memory: Kill process 3658 (tvheadend) score 614 or sacrifice child
Oct 14 03:37:31 tvheadend kernel: [152017.939386] Killed process 3658 (tvheadend) total-vm:2151828kB, anon-rss:237868kB, file-rss:0kB

But I don't mind if this issue is closed. It works for me when I give the VM 2GB.

#4

Updated by B C about 9 years ago

in memory timeshift buffer???

#5

Updated by Edwin de Graaf about 9 years ago

B C wrote:

in memory timeshift buffer???

Timeshift is not enabled in the configuration.

#6

Updated by Jaroslav Kysela about 9 years ago

I'm not sure if it's tvh which is causing the OOM. The key point is RSS. Could you also look when the RSS increased for the tvh process (an action started - see log) ?

#7

Updated by Edwin de Graaf about 9 years ago

egraaf@tvheadend:~/tvheadend/tvheadend$ zcat /var/log/syslog.?.gz | grep "invoked\|Tainted" 
Oct 14 03:37:31 tvheadend kernel: [152017.930233] tvhdhomerun_fro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 14 03:37:31 tvheadend kernel: [152017.930913] CPU: 0 PID: 5780 Comm: tvhdhomerun_fro Tainted: GF          O 3.13.0-24-generic #47-Ubuntu
Oct 13 02:05:04 tvheadend kernel: [60094.969473] tvhdhomerun_fro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 13 02:05:04 tvheadend kernel: [60094.970169] CPU: 0 PID: 3643 Comm: tvhdhomerun_fro Tainted: GF          O 3.13.0-24-generic #47-Ubuntu

It seems to happen when data is read from HD Homerun (not too surprising). And at 02:04 the EPG update starts (tv_grab_nl), so that also seems related.

RSS for Tvheadend when oom-killer activates is about 60MB, and we are out of SWAP:

Oct 14 03:37:31 tvheadend kernel: [152017.938594] Free swap  = 0kB
Oct 14 03:37:31 tvheadend kernel: [152017.938594] Total swap = 1044476kB
Oct 14 03:37:31 tvheadend kernel: [152017.938595] 262030 pages RAM
Oct 14 03:37:31 tvheadend kernel: [152017.938595] 0 pages HighMem/MovableOnly
Oct 14 03:37:31 tvheadend kernel: [152017.938596] 7531 pages reserved
Oct 14 03:37:31 tvheadend kernel: [152017.938596] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Oct 14 03:37:31 tvheadend kernel: [152017.938605] [  315]     0   315     4868        0      14       80             0 upstart-udev-br
Oct 14 03:37:31 tvheadend kernel: [152017.938607] [  319]     0   319    12805        1      28      143         -1000 systemd-udevd
Oct 14 03:37:31 tvheadend kernel: [152017.938608] [  378]   102   378     9806        0      23      107             0 dbus-daemon
Oct 14 03:37:31 tvheadend kernel: [152017.938610] [  420]     0   420    10862        1      27       90             0 systemd-logind
Oct 14 03:37:31 tvheadend kernel: [152017.938612] [  421]     0   421     5869        1      17       63             0 rpc.idmapd
Oct 14 03:37:31 tvheadend kernel: [152017.938613] [  422]     0   422     8228        1      21      102             0 rpc.gssd
Oct 14 03:37:31 tvheadend kernel: [152017.938614] [  433]   101   433    63960        0      28      256             0 rsyslogd
Oct 14 03:37:31 tvheadend kernel: [152017.938616] [  472]     0   472     3851        1      12      102             0 upstart-file-br
Oct 14 03:37:31 tvheadend kernel: [152017.938617] [  521]     0   521     2557       28       9      546             0 dhclient
Oct 14 03:37:31 tvheadend kernel: [152017.938619] [  589]     0   589    41641       23      18      546             0 userhdhomerun
Oct 14 03:37:31 tvheadend kernel: [152017.938620] [  737]     0   737     5855       14      17       62             0 rpcbind
Oct 14 03:37:31 tvheadend kernel: [152017.938622] [  803]   107   803     5385        1      16      115             0 rpc.statd
Oct 14 03:37:31 tvheadend kernel: [152017.938623] [  957]     0   957     3954        1      13       39             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938624] [  960]     0   960     3954        1      13       39             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938626] [  965]     0   965     3954        1      13       41             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938627] [  966]     0   966     3954        1      13       40             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938628] [  968]     0   968     3954        1      13       40             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938630] [ 1007]     0  1007    15341        0      34      169         -1000 sshd
Oct 14 03:37:31 tvheadend kernel: [152017.938631] [ 1013]     0  1013     1091        0       8       37             0 acpid
Oct 14 03:37:31 tvheadend kernel: [152017.938632] [ 1014]     0  1014     5913       17      18       38             0 cron
Oct 14 03:37:31 tvheadend kernel: [152017.938633] [ 1015]     0  1015     4784        0      13       49             0 atd
Oct 14 03:37:31 tvheadend kernel: [152017.938635] [ 1078]     0  1078     3989        0      12      213             0 upstart-socket-
Oct 14 03:37:31 tvheadend kernel: [152017.938636] [ 1116]     0  1116     3954        1      12       40             0 getty
Oct 14 03:37:31 tvheadend kernel: [152017.938637] [ 1361]   106  1361     7861       30      19      121             0 ntpd
Oct 14 03:37:31 tvheadend kernel: [152017.938639] [ 3658]   105  3658   537957    59467     694   256683             0 tvheadend
Oct 14 03:37:31 tvheadend kernel: [152017.938641] [ 4323]     0  4323    26408        0      56      248             0 sshd
Oct 14 03:37:31 tvheadend kernel: [152017.938642] [ 4412]  1000  4412    26408        0      53      251             0 sshd
Oct 14 03:37:31 tvheadend kernel: [152017.938643] [ 4413]  1000  4413     5724        1      16      584             0 bash
Oct 14 03:37:31 tvheadend kernel: [152017.938645] Out of memory: Kill process 3658 (tvheadend) score 614 or sacrifice child
#8

Updated by Jaroslav Kysela about 9 years ago

I found a critical error which may cause this. Could you upgrade to latest 4.0 code and retest ?

#9

Updated by Edwin de Graaf about 9 years ago

I have updated to version 4.0.6-11~g1462871, but Tvheadend is still using a lot of memory:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 944 0.1 1.4 907500 28732 ? Ssl 14:41 0:01 tvheadend -f -u hts -g video

I have attached the output of pmap that shows a lot of anon allocations (malloc)

#10

Updated by Jaroslav Kysela about 9 years ago

OK. If you do simple streaming (TV watch) and TVH does nothing else (epggrab), does the memory usage increase continuously ?

#11

Updated by Edwin de Graaf about 9 years ago

No, memory usage is very stable, both VSZ and RSS.

Also available in: Atom PDF