Bug #3305
Memory leak
0%
Description
With issue #3294 I've udpated to version 4.1-971~g5192186. Since then tvheadend is running out of meemory after a few recordings are started at the same time.
I've updated now to 4.1-988~g82776a4, but the memory issue is still there. I tried to get a coredump, but am unsuccessfull to get one trying with guide https://tvheadend.org/projects/tvheadend/wiki/Debugging.
Syslog:
Nov 15 20:14:03 mpw systemd1: Removed slice user-1001.slice.
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/386MHz/NPO 3 HD: H264 #3201 Continuity counter error (total 1) #3212 Continuity counter error (total 1)
Nov 15 20:14:13 mpw tvheadend[2994]: TS: Ziggo/386MHz/NPO 3 HD: AC3
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/386MHz/NPO 3 HD: TELETEXT #3301 Continuity counter error (total 1) #3211 Continuity counter error (total 1)
Nov 15 20:14:13 mpw tvheadend[2994]: TS: Ziggo/386MHz/NPO 3 HD: MPEG2AUDIO
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/386MHz/NPO 3 HD: MPEG2AUDIO #3213 Continuity counter error (total 1) #2001 Continuity counter error (total 1)
Nov 15 20:14:13 mpw tvheadend[2994]: TS: Ziggo/458MHz/RTL 4 HD: H264
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/458MHz/RTL 4 HD: TELETEXT #2101 Continuity counter error (total 1) #2012 Continuity counter error (total 1)
Nov 15 20:14:13 mpw tvheadend[2994]: TS: Ziggo/458MHz/RTL 4 HD: AC3
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/458MHz/RTL 5 HD: H264 #2301 Continuity counter error (total 1) #2311 Continuity counter error (total 1)
Nov 15 20:14:13 mpw tvheadend[2994]: TS: Ziggo/458MHz/RTL 5 HD: MPEG2AUDIO
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/458MHz/RTL 5 HD: TELETEXT #2401 Continuity counter error (total 1) #2011 Continuity counter error (total 1)
Nov 15 20:14:13 mpw tvheadend[2994]: TS: Ziggo/458MHz/RTL 4 HD: MPEG2AUDIO
Nov 15 20:14:13 mpw tvheadend2994: TS: Ziggo/458MHz/RTL 5 HD: AC3 @ #2312 Continuity counter error (total 1)
Nov 15 20:14:17 mpw kernel: [ 4483.564365] Purging GPU memory, 0 bytes freed, 8978432 bytes still pinned.
Nov 15 20:14:17 mpw kernel: [ 4483.564465] card_poll invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 15 20:14:17 mpw kernel: [ 4483.564467] card_poll cpuset=/ mems_allowed=0
Nov 15 20:14:17 mpw kernel: [ 4483.564473] CPU: 3 PID: 14309 Comm: card_poll Not tainted 3.16.0-50-generic #66~14.04.1-Ubuntu
Nov 15 20:14:17 mpw kernel: [ 4483.564475] Hardware name: /D54250WYK, BIOS WYLPT10H.86A.0040.2015.0612.1400 06/12/2015
Nov 15 20:14:17 mpw kernel: [ 4483.564477] 0000000000000000 ffff8800a9d7b8c8 ffffffff81766748 ffff8800aa6b0000
Nov 15 20:14:17 mpw kernel: [ 4483.564480] ffff8800a9d7b950 ffffffff817602f5 0000000000000000 0000000000000000
Nov 15 20:14:17 mpw kernel: [ 4483.564491] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Nov 15 20:14:17 mpw kernel: [ 4483.564494] Call Trace:
Nov 15 20:14:17 mpw kernel: [ 4483.564502] [<ffffffff81766748>] dump_stack+0x45/0x56
Nov 15 20:14:17 mpw kernel: [ 4483.564505] [<ffffffff817602f5>] dump_header+0x7f/0x1f1
Nov 15 20:14:17 mpw kernel: [ 4483.564512] [<ffffffff81165645>] oom_kill_process+0x205/0x360
Nov 15 20:14:17 mpw kernel: [ 4483.564516] [<ffffffff812ec305>] ? security_capable_noaudit+0x15/0x20
Nov 15 20:14:17 mpw kernel: [ 4483.564520] [<ffffffff81165df2>] out_of_memory+0x492/0x4d0
Nov 15 20:14:17 mpw kernel: [ 4483.564523] [<ffffffff81760e4d>] __alloc_pages_slowpath+0x63e/0x775
Nov 15 20:14:17 mpw kernel: [ 4483.564528] [<ffffffff8116bba3>] __alloc_pages_nodemask+0x273/0x2a0
Nov 15 20:14:17 mpw kernel: [ 4483.564533] [<ffffffff811ac6b4>] alloc_pages_current+0xa4/0x170
Nov 15 20:14:17 mpw kernel: [ 4483.564536] [<ffffffff81162057>] __page_cache_alloc+0x97/0xb0
Nov 15 20:14:17 mpw kernel: [ 4483.564540] [<ffffffff81164308>] filemap_fault+0x1a8/0x430
Nov 15 20:14:17 mpw kernel: [ 4483.564544] [<ffffffff8118aa99>] __do_fault+0x39/0x90
Nov 15 20:14:17 mpw kernel: [ 4483.564548] [<ffffffff8118d120>] do_read_fault.isra.56+0x50/0x2e0
Nov 15 20:14:17 mpw kernel: [ 4483.564551] [<ffffffff8118ee4d>] handle_mm_fault+0x62d/0x10b0
Nov 15 20:14:17 mpw kernel: [ 4483.564556] [<ffffffff8105af5e>] __do_page_fault+0x19e/0x560
Nov 15 20:14:17 mpw kernel: [ 4483.564561] [<ffffffff810a62e8>] ? __enqueue_entity+0x78/0x80
Nov 15 20:14:17 mpw kernel: [ 4483.564565] [<ffffffff810a8015>] ? set_next_entity+0x95/0xb0
Nov 15 20:14:17 mpw kernel: [ 4483.564569] [<ffffffff81011627>] ? __switch_to+0x167/0x580
Nov 15 20:14:17 mpw kernel: [ 4483.564573] [<ffffffff8105b351>] do_page_fault+0x31/0x70
Nov 15 20:14:17 mpw kernel: [ 4483.564577] [<ffffffff81770de8>] page_fault+0x28/0x30
Nov 15 20:14:17 mpw kernel: [ 4483.564578] Mem-Info:
Nov 15 20:14:17 mpw kernel: [ 4483.564580] Node 0 DMA per-cpu:
Nov 15 20:14:17 mpw kernel: [ 4483.564583] CPU 0: hi: 0, btch: 1 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564584] CPU 1: hi: 0, btch: 1 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564586] CPU 2: hi: 0, btch: 1 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564587] CPU 3: hi: 0, btch: 1 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564589] Node 0 DMA32 per-cpu:
Nov 15 20:14:17 mpw kernel: [ 4483.564591] CPU 0: hi: 186, btch: 31 usd: 30
Nov 15 20:14:17 mpw kernel: [ 4483.564593] CPU 1: hi: 186, btch: 31 usd: 7
Nov 15 20:14:17 mpw kernel: [ 4483.564595] CPU 2: hi: 186, btch: 31 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564597] CPU 3: hi: 186, btch: 31 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564598] Node 0 Normal per-cpu:
Nov 15 20:14:17 mpw kernel: [ 4483.564600] CPU 0: hi: 186, btch: 31 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564601] CPU 1: hi: 186, btch: 31 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564603] CPU 2: hi: 186, btch: 31 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564604] CPU 3: hi: 186, btch: 31 usd: 0
Nov 15 20:14:17 mpw kernel: [ 4483.564609] active_anon:622316 inactive_anon:198194 isolated_anon:0
Nov 15 20:14:17 mpw kernel: [ 4483.564609] active_file:78 inactive_file:18 isolated_file:6
Nov 15 20:14:17 mpw kernel: [ 4483.564609] unevictable:0 dirty:0 writeback:0 unstable:0
Nov 15 20:14:17 mpw kernel: [ 4483.564609] free:20733 slab_reclaimable:5016 slab_unreclaimable:5689
Nov 15 20:14:17 mpw kernel: [ 4483.564609] mapped:321 shmem:2632 pagetables:6281 bounce:0
Nov 15 20:14:17 mpw kernel: [ 4483.564609] free_cma:0
Nov 15 20:14:17 mpw kernel: [ 4483.564613] Node 0 DMA free:13844kB min:308kB low:384kB high:460kB active_anon:704kB inactive_anon:868kB active_file:0kB inactive_file:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:24kB slab_reclaimable:104kB slab_unreclaimable:132kB kernel_stack:16kB pagetables:88kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:72 all_unreclaimable? no
Nov 15 20:14:17 mpw kernel: [ 4483.564620] lowmem_reserve[]: 0 2935 3388 3388
Nov 15 20:14:17 mpw kernel: [ 4483.564624] Node 0 DMA32 free:60512kB min:58284kB low:72852kB high:87424kB active_anon:2282952kB inactive_anon:585344kB active_file:312kB inactive_file:552kB unevictable:0kB isolated(anon):0kB isolated(file):24kB present:3088116kB managed:3008820kB mlocked:0kB dirty:0kB writeback:0kB mapped:1076kB shmem:8716kB slab_reclaimable:17372kB slab_unreclaimable:16528kB kernel_stack:3232kB pagetables:21504kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1024 all_unreclaimable? no
Nov 15 20:14:17 mpw kernel: [ 4483.564630] lowmem_reserve[]: 0 0 452 452
Nov 15 20:14:17 mpw kernel: [ 4483.564633] Node 0 Normal free:8420kB min:8988kB low:11232kB high:13480kB active_anon:205608kB inactive_anon:206564kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:522240kB managed:463700kB mlocked:0kB dirty:0kB writeback:0kB mapped:192kB shmem:1788kB slab_reclaimable:2588kB slab_unreclaimable:6096kB kernel_stack:1200kB pagetables:3532kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 15 20:14:17 mpw kernel: [ 4483.564638] lowmem_reserve[]: 0 0 0 0
Nov 15 20:14:17 mpw kernel: [ 4483.564641] Node 0 DMA: 3*4kB (U) 5*8kB (UEM) 6*16kB (UEM) 2*32kB (E) 5*64kB (UEM) 2*128kB (EM) 1*256kB (M) 1*512kB (E) 2*1024kB (UE) 3*2048kB (EMR) 1*4096kB (M) = 13844kB
Nov 15 20:14:17 mpw kernel: [ 4483.564656] Node 0 DMA32: 308*4kB (UEM) 221*8kB (UEM) 264*16kB (UEM) 186*32kB (UEM) 96*64kB (UEM) 58*128kB (UE) 52*256kB (UEM) 30*512kB (UEM) 5*1024kB (UEM) 0*2048kB 0*4096kB = 60536kB
Nov 15 20:14:17 mpw kernel: [ 4483.564669] Node 0 Normal: 246*4kB (UEMR) 171*8kB (UEMR) 86*16kB (UEMR) 16*32kB (UER) 11*64kB (UE) 4*128kB (UE) 2*256kB (UR) 0*512kB 1*1024kB (R) 1*2048kB (R) 0*4096kB = 9040kB
Nov 15 20:14:17 mpw kernel: [ 4483.564684] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 15 20:14:17 mpw kernel: [ 4483.564686] 11979 total pagecache pages
Nov 15 20:14:17 mpw kernel: [ 4483.564687] 8996 pages in swap cache
Nov 15 20:14:17 mpw kernel: [ 4483.564690] Swap cache stats: add 1168276, delete 1159280, find 71008/101293
Nov 15 20:14:17 mpw kernel: [ 4483.564691] Free swap = 0kB
Nov 15 20:14:17 mpw kernel: [ 4483.564692] Total swap = 4084732kB
Nov 15 20:14:17 mpw kernel: [ 4483.564694] 906585 pages RAM
Nov 15 20:14:17 mpw kernel: [ 4483.564695] 0 pages HighMem/MovableOnly
Nov 15 20:14:17 mpw kernel: [ 4483.564696] 14635 pages reserved
Nov 15 20:14:17 mpw kernel: [ 4483.564698] 0 pages hwpoisoned
Nov 15 20:14:17 mpw kernel: [ 4483.564699] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Nov 15 20:14:17 mpw kernel: [ 4483.564719] [ 234] 0 234 7857 209 18 51 0 systemd-journal
Nov 15 20:14:17 mpw kernel: [ 4483.564725] [ 236] 0 236 11186 13 21 304 -1000 systemd-udevd
Nov 15 20:14:17 mpw kernel: [ 4483.564730] [ 447] 0 447 8419 0 19 107 0 lircd
Nov 15 20:14:17 mpw kernel: [ 4483.564735] [ 484] 0 484 9307 4 23 93 0 rpcbind
Nov 15 20:14:17 mpw kernel: [ 4483.564739] [ 633] 0 633 7234 20 19 50 0 cron
Nov 15 20:14:17 mpw kernel: [ 4483.564743] [ 634] 0 634 84322 60 68 393 0 ModemManager
Nov 15 20:14:17 mpw kernel: [ 4483.564748] [ 636] 0 636 70780 66 42 702 0 accounts-daemon
Nov 15 20:14:17 mpw kernel: [ 4483.564752] [ 637] 0 637 7157 49 19 52 0 systemd-logind
Nov 15 20:14:17 mpw kernel: [ 4483.564755] [ 638] 109 638 95503 29 81 388 0 whoopsie
Nov 15 20:14:17 mpw kernel: [ 4483.564758] [ 640] 102 640 10864 176 26 80 -900 dbus-daemon
Nov 15 20:14:17 mpw kernel: [ 4483.564761] [ 667] 101 667 64093 16 28 202 0 rsyslogd
Nov 15 20:14:17 mpw kernel: [ 4483.564764] [ 668] 0 668 112831 191 72 891 0 NetworkManager
Nov 15 20:14:17 mpw kernel: [ 4483.564766] [ 673] 111 673 10049 28 24 80 0 avahi-daemon
Nov 15 20:14:17 mpw kernel: [ 4483.564769] [ 674] 0 674 21541 2 46 275 0 cupsd
Nov 15 20:14:17 mpw kernel: [ 4483.564772] [ 683] 0 683 69637 71 39 471 0 polkitd
Nov 15 20:14:17 mpw kernel: [ 4483.564774] [ 686] 111 686 10017 1 23 78 0 avahi-daemon
Nov 15 20:14:17 mpw kernel: [ 4483.564776] [ 696] 0 696 68090 57 66 777 0 cups-browsed
Nov 15 20:14:17 mpw kernel: [ 4483.564779] [ 723] 0 723 17490 32 37 158 -1000 sshd
Nov 15 20:14:17 mpw kernel: [ 4483.564782] [ 735] 119 735 1119 2 7 38 0 mysqld_safe
Nov 15 20:14:17 mpw kernel: [ 4483.564785] [ 1108] 119 1108 362554 1054 182 63204 0 mysqld
Nov 15 20:14:17 mpw kernel: [ 4483.564788] [ 1119] 0 1119 9803 1 22 131 0 wpa_supplicant
Nov 15 20:14:17 mpw kernel: [ 4483.564792] [ 1149] 0 1149 5864 21 14 1703 0 dhclient
Nov 15 20:14:17 mpw kernel: [ 4483.564796] [ 1158] 65534 1158 12024 22 27 70 0 dnsmasq
Nov 15 20:14:17 mpw kernel: [ 4483.564800] [ 1382] 0 1382 5860 5 14 91 0 oscam-svn
Nov 15 20:14:17 mpw kernel: [ 4483.564804] [ 1383] 0 1383 123486 39 29 251 0 oscam-svn
Nov 15 20:14:17 mpw kernel: [ 4483.564806] [ 1478] 0 1478 4881 23 15 39 0 irqbalance
Nov 15 20:14:17 mpw kernel: [ 4483.564809] [ 1480] 106 1480 11319 23 25 74 0 kerneloops
Nov 15 20:14:17 mpw kernel: [ 4483.564811] [ 1498] 0 1498 88097 72 39 139 0 lightdm
Nov 15 20:14:17 mpw kernel: [ 4483.564814] [ 1511] 0 1511 40192 551 71 1189 0 Xorg
Nov 15 20:14:17 mpw kernel: [ 4483.564816] [ 1518] 0 1518 1099 3 8 42 0 acpid
Nov 15 20:14:17 mpw kernel: [ 4483.564819] [ 1519] 0 1519 90858 28 129 1710 0 apache2
Nov 15 20:14:17 mpw kernel: [ 4483.564821] [ 1522] 33 1522 90894 3 115 1737 0 apache2
Nov 15 20:14:17 mpw kernel: [ 4483.564824] [ 1523] 33 1523 90894 3 115 1737 0 apache2
Nov 15 20:14:17 mpw kernel: [ 4483.564826] [ 1524] 33 1524 90894 7 115 1733 0 apache2
Nov 15 20:14:17 mpw kernel: [ 4483.564828] [ 1525] 33 1525 90894 3 115 1737 0 apache2
Nov 15 20:14:17 mpw kernel: [ 4483.564831] [ 1526] 33 1526 90894 3 115 1737 0 apache2
Nov 15 20:14:17 mpw kernel: [ 4483.564833] [ 1531] 0 1531 3965 2 13 39 0 agetty
Nov 15 20:14:17 mpw kernel: [ 4483.564836] [ 1532] 0 1532 58074 2 48 211 0 lightdm
Nov 15 20:14:17 mpw kernel: [ 4483.564838] [ 1538] 1000 1538 11292 87 25 137 0 systemd
Nov 15 20:14:17 mpw kernel: [ 4483.564841] [ 1539] 1000 1539 15776 1 32 458 0 (sd-pam)
Nov 15 20:14:17 mpw kernel: [ 4483.564843] [ 1542] 1000 1542 3117 2 11 55 0 kodi.desktop
Nov 15 20:14:17 mpw kernel: [ 4483.564846] [ 1604] 1000 1604 2792 11 8 73 0 ssh-agent
Nov 15 20:14:17 mpw kernel: [ 4483.564849] [ 1607] 1000 1607 9721 0 20 97 0 dbus-launch
Nov 15 20:14:17 mpw kernel: [ 4483.564853] [ 1608] 1000 1608 10681 0 23 93 0 dbus-daemon
Nov 15 20:14:17 mpw kernel: [ 4483.564856] [ 1620] 1000 1620 34812 75 34 157 0 synergyc
Nov 15 20:14:17 mpw kernel: [ 4483.564861] [ 1623] 1000 1623 1119 0 7 26 0 kodi-standalone
Nov 15 20:14:17 mpw kernel: [ 4483.564865] [ 1628] 1000 1628 1119 0 7 34 0 kodi
Nov 15 20:14:17 mpw kernel: [ 4483.564868] [ 1630] 1000 1630 774953 2561 446 48183 0 kodi.bin
Nov 15 20:14:17 mpw kernel: [ 4483.564872] [ 1631] 0 1631 68985 83 51 271 0 upowerd
Nov 15 20:14:17 mpw kernel: [ 4483.564877] [ 1747] 120 1747 8500 29 21 127 0 ntpd
Nov 15 20:14:17 mpw kernel: [ 4483.564881] [ 2427] 0 2427 26516 0 53 257 0 sshd
Nov 15 20:14:17 mpw kernel: [ 4483.564885] [ 2472] 1000 2472 26516 0 52 254 0 sshd
Nov 15 20:14:17 mpw kernel: [ 4483.564890] [ 2473] 1000 2473 5781 0 16 621 0 bash
Nov 15 20:14:17 mpw kernel: [ 4483.564893] [ 2487] 1000 2487 14079 0 33 112 0 su
Nov 15 20:14:17 mpw kernel: [ 4483.564898] [ 2491] 0 2491 5796 55 16 585 0 bash
Nov 15 20:14:17 mpw kernel: [ 4483.564902] [ 2858] 1000 2858 6814 32 15 108 0 screen
Nov 15 20:14:17 mpw kernel: [ 4483.564906] [ 2859] 1000 2859 5803 3 17 646 0 bash
Nov 15 20:14:17 mpw kernel: [ 4483.564910] [ 2876] 1000 2876 14079 0 34 112 0 su
Nov 15 20:14:17 mpw kernel: [ 4483.564914] [ 2880] 0 2880 5804 3 16 646 0 bash
Nov 15 20:14:17 mpw kernel: [ 4483.564918] [ 2965] 0 2965 14079 0 33 111 0 su
Nov 15 20:14:17 mpw kernel: [ 4483.564922] [ 2966] 117 2966 5779 3 18 621 0 bash
Nov 15 20:14:17 mpw kernel: [ 4483.564927] [ 2994] 117 2994 2239953 802223 3383 880945 0 tvheadend
Nov 15 20:14:17 mpw kernel: [ 4483.564932] Out of memory: Kill process 2994 (tvheadend) score 890 or sacrifice child
Nov 15 20:14:17 mpw kernel: [ 4483.564940] Killed process 2994 (tvheadend) total-vm:8959812kB, anon-rss:3208892kB, file-rss:0kB
History
Updated by Jaroslav Kysela almost 9 years ago
Updated by I'm not sure if it's related to DVR. I tested several configuration and cannot reproduce the leak here. Could you try to isolate the behaviour to a restricted config?
Updated by Anonymous almost 9 years ago
I'm getting more and more issues last week. Let me recreate the config, then I will report back. This will be probably in a few days.
Updated by Anonymous almost 9 years ago
I've reinstalled tvheadend and recreated my config. The result is the same, tvheadend is running out of memory after a few recordings are started.
I've reverted now back to the prod version 4.0.7 and recreated the config again. The system is now running stable for almost 24 hours. Memory consumption of the tvheadend process is less then 1% of total memory.
Environment:
As OS I'm running Ubuntu 5.10. Hardware is Intel NUC D54250WYK with 4GB RAM and 60GB SSD. As a tuner I have HDHomerun HDHR3-4DC. My system is using latest oscam-svn for decoding my Ziggo Smart Card.
I'm recording directly on a samba share connected with 1GB to my Synology DS412+.
In the config I created 22 autorec entries. And as postprocessing command I'm running comskip (issue remains when disabling this).
Because it's my production system, and I need to keep my family happy, I needed to go back to a stable environment again. If you like to investigate this further, please let me know this and I can try to setup a similar test environment trying to reproduce the issue.
Updated by Rob Maas almost 9 years ago
Updated by Just a small addition, I have also memory problems since version 4.0.8, however it will not crash tvheadend, but the system gets very slow (due to the high memory) and when playing or recording a lot of continuity errors occur. After a reboot the system will work flawless for a couple of hours without any continuity error.
I already tried a fresh install, was running Ubuntu 14.10 and now on Debian 8, but both with the same result, the logging is clean.
Since I don't watch TV very often I can use this server for debugging purposes, but at this moment I have no idea where to start.