Bug #3157
open
High memory usage
Added by Edwin de Graaf over 9 years ago. Updated over 9 years ago.
0%
Description
I see quite high memory usage of Tvheadend, so much that the Tvheadend process was killed on a 1GB memory VM because the memory was full.
As discussed in forum thread https://tvheadend.org/boards/5/topics/18091
This is the valgrind ouput included as a file.
Right now:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 6375 0.0 1.1 742112 23220 ? Ssl 09:35 0:01 tvheadend -f -u hts -g video
Files
| valgrind-output.log (21.4 KB) valgrind-output.log | Edwin de Graaf, 2015-10-15 10:23 | ||
| pmap.txt (9.83 KB) pmap.txt | Edwin de Graaf, 2015-10-19 15:06 |
Updated by Mark Clarkstone over 9 years ago
Edwin de Graaf wrote:
I see quite high memory usage of Tvheadend, so much that the Tvheadend process was killed on a 1GB memory VM because the memory was full.
As discussed in forum thread https://tvheadend.org/boards/5/topics/18091
This is the valgrind ouput included as a file.
Right now:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 6375 0.0 1.1 742112 23220 ? Ssl 09:35 0:01 tvheadend -f -u hts -g video
I'm no valgrind expert but that looks fine to me, btw you may want to install tvheadend-dbg package so it can fill in the gaps (those pesky ??? lines).
It could be a leak with hdhomerun which I think Tvheadend uses a built in lib.
Updated by Rafal Kupiec over 9 years ago
- free -m
total used free shared buff/cache available
Mem: 12078 286 68 65 11724 11638
Seems to not use much memory ;)
Updated by Edwin de Graaf over 9 years ago
Rafal Kupiec wrote:
- free -m
total used free shared buff/cache available
Mem: 12078 286 68 65 11724 11638Seems to not use much memory ;)
This happens when the VM has 1GB of memory:
Oct 14 03:37:31 tvheadend kernel: [152017.938645] Out of memory: Kill process 3658 (tvheadend) score 614 or sacrifice child
Oct 14 03:37:31 tvheadend kernel: [152017.939386] Killed process 3658 (tvheadend) total-vm:2151828kB, anon-rss:237868kB, file-rss:0kB
But I don't mind if this issue is closed. It works for me when I give the VM 2GB.
Updated by
Updated by Edwin de Graaf over 9 years ago
B C wrote:
in memory timeshift buffer???
Timeshift is not enabled in the configuration.
Updated by Jaroslav Kysela over 9 years ago
I'm not sure if it's tvh which is causing the OOM. The key point is RSS. Could you also look when the RSS increased for the tvh process (an action started - see log) ?
Updated by Edwin de Graaf over 9 years ago
egraaf@tvheadend:~/tvheadend/tvheadend$ zcat /var/log/syslog.?.gz | grep "invoked\|Tainted" Oct 14 03:37:31 tvheadend kernel: [152017.930233] tvhdhomerun_fro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Oct 14 03:37:31 tvheadend kernel: [152017.930913] CPU: 0 PID: 5780 Comm: tvhdhomerun_fro Tainted: GF O 3.13.0-24-generic #47-Ubuntu Oct 13 02:05:04 tvheadend kernel: [60094.969473] tvhdhomerun_fro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Oct 13 02:05:04 tvheadend kernel: [60094.970169] CPU: 0 PID: 3643 Comm: tvhdhomerun_fro Tainted: GF O 3.13.0-24-generic #47-Ubuntu
It seems to happen when data is read from HD Homerun (not too surprising). And at 02:04 the EPG update starts (tv_grab_nl), so that also seems related.
RSS for Tvheadend when oom-killer activates is about 60MB, and we are out of SWAP:
Oct 14 03:37:31 tvheadend kernel: [152017.938594] Free swap = 0kB Oct 14 03:37:31 tvheadend kernel: [152017.938594] Total swap = 1044476kB Oct 14 03:37:31 tvheadend kernel: [152017.938595] 262030 pages RAM Oct 14 03:37:31 tvheadend kernel: [152017.938595] 0 pages HighMem/MovableOnly Oct 14 03:37:31 tvheadend kernel: [152017.938596] 7531 pages reserved Oct 14 03:37:31 tvheadend kernel: [152017.938596] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Oct 14 03:37:31 tvheadend kernel: [152017.938605] [ 315] 0 315 4868 0 14 80 0 upstart-udev-br Oct 14 03:37:31 tvheadend kernel: [152017.938607] [ 319] 0 319 12805 1 28 143 -1000 systemd-udevd Oct 14 03:37:31 tvheadend kernel: [152017.938608] [ 378] 102 378 9806 0 23 107 0 dbus-daemon Oct 14 03:37:31 tvheadend kernel: [152017.938610] [ 420] 0 420 10862 1 27 90 0 systemd-logind Oct 14 03:37:31 tvheadend kernel: [152017.938612] [ 421] 0 421 5869 1 17 63 0 rpc.idmapd Oct 14 03:37:31 tvheadend kernel: [152017.938613] [ 422] 0 422 8228 1 21 102 0 rpc.gssd Oct 14 03:37:31 tvheadend kernel: [152017.938614] [ 433] 101 433 63960 0 28 256 0 rsyslogd Oct 14 03:37:31 tvheadend kernel: [152017.938616] [ 472] 0 472 3851 1 12 102 0 upstart-file-br Oct 14 03:37:31 tvheadend kernel: [152017.938617] [ 521] 0 521 2557 28 9 546 0 dhclient Oct 14 03:37:31 tvheadend kernel: [152017.938619] [ 589] 0 589 41641 23 18 546 0 userhdhomerun Oct 14 03:37:31 tvheadend kernel: [152017.938620] [ 737] 0 737 5855 14 17 62 0 rpcbind Oct 14 03:37:31 tvheadend kernel: [152017.938622] [ 803] 107 803 5385 1 16 115 0 rpc.statd Oct 14 03:37:31 tvheadend kernel: [152017.938623] [ 957] 0 957 3954 1 13 39 0 getty Oct 14 03:37:31 tvheadend kernel: [152017.938624] [ 960] 0 960 3954 1 13 39 0 getty Oct 14 03:37:31 tvheadend kernel: [152017.938626] [ 965] 0 965 3954 1 13 41 0 getty Oct 14 03:37:31 tvheadend kernel: [152017.938627] [ 966] 0 966 3954 1 13 40 0 getty Oct 14 03:37:31 tvheadend kernel: [152017.938628] [ 968] 0 968 3954 1 13 40 0 getty Oct 14 03:37:31 tvheadend kernel: [152017.938630] [ 1007] 0 1007 15341 0 34 169 -1000 sshd Oct 14 03:37:31 tvheadend kernel: [152017.938631] [ 1013] 0 1013 1091 0 8 37 0 acpid Oct 14 03:37:31 tvheadend kernel: [152017.938632] [ 1014] 0 1014 5913 17 18 38 0 cron Oct 14 03:37:31 tvheadend kernel: [152017.938633] [ 1015] 0 1015 4784 0 13 49 0 atd Oct 14 03:37:31 tvheadend kernel: [152017.938635] [ 1078] 0 1078 3989 0 12 213 0 upstart-socket- Oct 14 03:37:31 tvheadend kernel: [152017.938636] [ 1116] 0 1116 3954 1 12 40 0 getty Oct 14 03:37:31 tvheadend kernel: [152017.938637] [ 1361] 106 1361 7861 30 19 121 0 ntpd Oct 14 03:37:31 tvheadend kernel: [152017.938639] [ 3658] 105 3658 537957 59467 694 256683 0 tvheadend Oct 14 03:37:31 tvheadend kernel: [152017.938641] [ 4323] 0 4323 26408 0 56 248 0 sshd Oct 14 03:37:31 tvheadend kernel: [152017.938642] [ 4412] 1000 4412 26408 0 53 251 0 sshd Oct 14 03:37:31 tvheadend kernel: [152017.938643] [ 4413] 1000 4413 5724 1 16 584 0 bash Oct 14 03:37:31 tvheadend kernel: [152017.938645] Out of memory: Kill process 3658 (tvheadend) score 614 or sacrifice child
Updated by Jaroslav Kysela over 9 years ago
I found a critical error which may cause this. Could you upgrade to latest 4.0 code and retest ?
Updated by Edwin de Graaf over 9 years ago
I have updated to version 4.0.6-11~g1462871, but Tvheadend is still using a lot of memory:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
hts 944 0.1 1.4 907500 28732 ? Ssl 14:41 0:01 tvheadend -f -u hts -g video
I have attached the output of pmap that shows a lot of anon allocations (malloc)
Updated by Jaroslav Kysela over 9 years ago
OK. If you do simple streaming (TV watch) and TVH does nothing else (epggrab), does the memory usage increase continuously ?
Updated by Edwin de Graaf over 9 years ago
No, memory usage is very stable, both VSZ and RSS.