Project

General

Profile

Bug #3466

Out of Memory -> KILLED

Added by Alfred Zastrow almost 9 years ago. Updated almost 8 years ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
2016-01-01
Due date:
% Done:

0%

Estimated time:
Found in version:
HTS Tvheadend 4.1-1278
Affected Versions:

Description

Just as an adhoc info, it happend during a recording & live view:

[ 5780.489863] Out of memory: Kill process 913 (tvh:lnxdvb-fron) score 960 or sacrifice child
[ 5780.489930] Killed process 913 (tvh:lnxdvb-fron) total-vm:5666264kB, anon-rss:3593032kB, file-rss
:0kB

My tvh-server has 4 GB and this never happend.

Sorry for no-debug. I will install the debug version tomorrow, today I'm back to 1264 because I need to reccord for my wife (WAF-important :-))

Alfred


Files

Bildschirmfoto vom 2016-01-02 06-56-41.png (46.3 KB) Bildschirmfoto vom 2016-01-02 06-56-41.png Alfred Zastrow, 2016-01-02 07:06
vg.log (1.49 MB) vg.log C K, 2016-01-13 15:12

History

#1

Updated by Alfred Zastrow almost 9 years ago

This was logged yesterday, I will try latest today.

[ 5780.486718] tvh:tshift-rd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[ 5780.486723] tvh:tshift-rd cpuset=/ mems_allowed=0
[ 5780.486730] CPU: 2 PID: 919 Comm: tvh:tshift-rd Tainted: G O 4.3.0-1-amd64 #1 Debian 4.3.3-2
[ 5780.486732] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Q2900M, BIOS P1.00 07/24/2014
[ 5780.486734] 0000000000000000 000000008251b1cd ffffffff812ddc89 ffff880036a743c0
[ 5780.486739] ffffffff811c9ff6 ffff8800a8dd8918 0000000000000000 0000000000000000
[ 5780.486742] ffff8800a851a800 ffffffffa0544a5d ffff8800a8dd0000 000000008251b1cd
[ 5780.486746] Call Trace:
[ 5780.486755] [<ffffffff812ddc89>] ? dump_stack+0x40/0x57
[ 5780.486760] [<ffffffff811c9ff6>] ? dump_header+0x83/0x20a
[ 5780.486786] [<ffffffffa0544a5d>] ? i915_gem_shrinker_oom+0x19d/0x1f0 [i915]
[ 5780.486790] [<ffffffff8116282d>] ? oom_kill_process+0x1fd/0x400
[ 5780.486793] [<ffffffff81162201>] ? oom_unkillable_task+0xc1/0x100
[ 5780.486798] [<ffffffff8126103e>] ? security_capable_noaudit+0x3e/0x60
[ 5780.486801] [<ffffffff81162c7a>] ? out_of_memory+0x1fa/0x470
[ 5780.486804] [<ffffffff811684cd>] ? __alloc_pages_nodemask+0x8bd/0xa10
[ 5780.486808] [<ffffffff811abfba>] ? alloc_pages_current+0x8a/0xf0
[ 5780.486812] [<ffffffff81161301>] ? filemap_fault+0x151/0x430
[ 5780.486815] [<ffffffff81189cef>] ? __do_fault+0x4f/0xe0
[ 5780.486819] [<ffffffff8118eb4b>] ? handle_mm_fault+0xecb/0x16b0
[ 5780.486824] [<ffffffff81060338>] ? __do_page_fault+0x198/0x430
[ 5780.486828] [<ffffffff812131a3>] ? SyS_epoll_wait+0x93/0xd0
[ 5780.486832] [<ffffffff815834b8>] ? page_fault+0x28/0x30
[ 5780.486835] Mem-Info:
[ 5780.486842] active_anon:902427 inactive_anon:2166 isolated_anon:0
active_file:69 inactive_file:123 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:6015 slab_unreclaimable:4757
mapped:317 shmem:2323 pagetables:2434 bounce:0
free:6488 free_pcp:313 free_cma:0

#2

Updated by Alfred Zastrow almost 9 years ago

With HTS Tvheadend 4.1-1284~ga5085c8 I couldn't reproduce the crash so far but what I've seen is that in time shift mode old data is not deleted. I've configured 1 GB as Ram-Buffer and even with a time shift of some seconds this seems to be filled over the time (screenshoot) and then the next buffers are written on the disc.
I will cancel my ram buffer to not running into short of memory situations anymore.

Maybe you can close this ticket.

btw. the advantage of the memory buffer is smaler now, because the disc is needed anyway after some time.

#3

Updated by Alfred Zastrow almost 9 years ago

I'm back to 1264 - this is the last "stable" version for me.
1284 also as 1286 hangs without a usable message after few seconds :-(

#4

Updated by Alfred Zastrow almost 9 years ago

without using timeshift, only two clients.

#5

Updated by Njuskalo Njusko almost 9 years ago

4.1-1258~gb480c92 is the last stable version in my case, using IPTV only. In all subsequent releases after zapping a while the tvh process gets unresponsive. Not even using timeshift nor recording. Using Kodi Jarvis B4 as a client.

#6

Updated by Mark Clarkstone almost 9 years ago

Njuskalo Njusko wrote:

4.1-1258~gb480c92 is the last stable version in my case, using IPTV only. In all subsequent releases after zapping a while the tvh process gets unresponsive. Not even using timeshift nor recording. Using Kodi Jarvis B4 as a client.

This is pretty much the same for me but DVB-S/S2/T/T2, I will continue to test git master.

#7

Updated by Jaroslav Kysela almost 9 years ago

  • Status changed from New to Fixed

I believe, it's fixed in current master.

#8

Updated by C K almost 9 years ago

Jaroslav Kysela wrote:

I believe, it's fixed in current master.

Hmmm, just got a crash.

2016-01-13 15:08:43.192 [   INFO] htsp: Got connection from 192.168.178.22
2016-01-13 15:08:43.599 [   INFO] htsp: 192.168.178.22: Welcomed client software: Kodi Media Center (HTSPv18)
2016-01-13 15:08:43.878 [   INFO] transcode: 0001: 1:H264 1920x1080 ==> H264 1280x720 (libx264)
2016-01-13 15:08:43.884 [   INFO] transcode: 0001: 2:AC3 ==> Filtered
2016-01-13 15:08:43.888 [   INFO] transcode: 0001: 3:CA ==> Filtered
2016-01-13 15:08:43.888 [   INFO] transcode: 0001: 4:CA ==> Filtered
tvheadend: src/plumbing/globalheaders.c:284: gh_hold: Zusicherung »ssc != ((void *)0)« nicht erfüllt.
2016-01-13 15:08:44.208 [  ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (4.1-1354~g5966e61) [8f2762c52b2cb566663da57a27737fd119d48d86] CWD: /home/waldmeister/src/tvheadend  
2016-01-13 15:08:44.212 [  ALERT] CRASH: Fault address 0x3e800004322 (N/A)
2016-01-13 15:08:44.213 [  ALERT] CRASH: Loaded libraries: /usr/lib/valgrind/vgpreload_core-amd64-linux.so /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/liburiparser.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2  
2016-01-13 15:08:44.214 [  ALERT] CRASH: Register dump [23]: 000000000000006a0000000000000000000000000000000800000000000000010000000000b678340000000000b679b80000000037d81180000000001e910ae0000000000000432200000000000043590000000004061feb000000000402c00000000000000000060000000000000000000000000686ccc90000000037d80ec8000000000686ccc9000000000000000400000000000000000000000000000006000000000000000e0000000000000000000003e800004322
2016-01-13 15:08:44.215 [  ALERT] CRASH: STACKTRACE
2016-01-13 15:08:44.285 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/trap.c:148 0x46bed9
2016-01-13 15:08:44.335 [  ALERT] CRASH: ??:0 0x611a340
2016-01-13 15:08:44.336 [  ALERT] CRASH: gsignal+0x39  (/lib/x86_64-linux-gnu/libc.so.6)
2016-01-13 15:08:44.336 [  ALERT] CRASH: abort+0x148  (/lib/x86_64-linux-gnu/libc.so.6)
2016-01-13 15:08:44.360 [   INFO] htsp: 192.168.178.22 [ Kodi Media Center ]: Identified as user 'oe-buero'
2016-01-13 15:08:44.364 [   INFO] htsp: 192.168.178.22 [ oe-buero | Kodi Media Center ]: Privileges updated
2016-01-13 15:08:44.382 [  ALERT] CRASH: ??:0 0x6865b86
2016-01-13 15:08:44.423 [  ALERT] CRASH: ??:0 0x6865c32
2016-01-13 15:08:44.468 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/plumbing/globalheaders.c:284 0x4a1023
2016-01-13 15:08:44.521 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/plumbing/transcoding.c:858 0x519992
2016-01-13 15:08:44.590 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/plumbing/transcoding.c:1497 0x5189f8
2016-01-13 15:08:44.640 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/plumbing/tsfix.c:278 0x49fabf
2016-01-13 15:08:44.684 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/plumbing/tsfix.c:298 0x4a029a
2016-01-13 15:08:44.727 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/parsers/parsers.c:1826 0x491a98
2016-01-13 15:08:44.777 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/parsers/parsers.c:1402 0x4933e5
2016-01-13 15:08:44.825 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/parsers/parsers.c:437 0x492884
2016-01-13 15:08:44.879 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/input/mpegts/tsdemux.c:101 0x4d068e
2016-01-13 15:08:44.922 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/input/mpegts/tsdemux.c:268 0x4d0afe
2016-01-13 15:08:44.979 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/descrambler/tvhcsa.c:77 0x51a83d
2016-01-13 15:08:45.024 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/descrambler/tvhcsa.c:164 0x51a97d
2016-01-13 15:08:45.069 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/descrambler/descrambler.c:766 0x4c8af7
2016-01-13 15:08:45.120 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/input/mpegts/tsdemux.c:233 0x4d0a5d
2016-01-13 15:08:45.167 [  ALERT] CRASH: /home/waldmeister/src/tvheadend/src/input/mpegts/mpegts_input.c:1300 0x4ceac6

see attached valgrind log for more infos

#9

Updated by Jaroslav Kysela almost 9 years ago

It's not related.

#10

Updated by James Huxtable over 8 years ago

Jaroslav Kysela wrote:

I believe, it's fixed in current master.

Is this definitely fixed? I was testing a recording last night after I had enabled a new profile and timeshifting. Before finishing the recording the process was killed as all memory had been used (including swap).
It appears as if the buffer wasn't being written to the HDD and freed properly, meaning it kept growing until it was killed.

My build is:
Build: 4.1-1945~g5374573~vivid (2016-04-28T23:44:34+0000)

[204365.820566] tvheadend invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
[204365.820569] tvheadend cpuset=/ mems_allowed=0
[204365.820573] CPU: 1 PID: 7474 Comm: tvheadend Not tainted 4.4.0-21-generic #37-Ubuntu
[204365.820574] Hardware name: Xen HVM domU, BIOS 4.4.0 04/14/2014
[204365.820575] 0000000000000286 0000000099d2dd54 ffff8800764c3a58 ffffffff813e93c3
[204365.820577] ffff8800764c3c50 ffff8800776dee00 ffff8800764c3ac8 ffffffff81208b2e
[204365.820579] ffffffff81ccc43f 0000000000000000 ffffffff81e66660 0000000000000206
[204365.820581] Call Trace:
[204365.820587] [<ffffffff813e93c3>] dump_stack+0x63/0x90
[204365.820590] [<ffffffff81208b2e>] dump_header+0x5a/0x1c5
[204365.820592] [<ffffffff81190dd2>] oom_kill_process+0x202/0x3c0
[204365.820594] [<ffffffff811911f5>] out_of_memory+0x215/0x460
[204365.820597] [<ffffffff81197190>] __alloc_pages_nodemask+0x9d0/0xb60
[204365.820600] [<ffffffff811e084c>] alloc_pages_current+0x8c/0x110
[204365.820602] [<ffffffff8118d67b>] __page_cache_alloc+0xab/0xc0
[204365.820603] [<ffffffff8118f8aa>] filemap_fault+0x14a/0x3f0
[204365.820606] [<ffffffff811bc120>] __do_fault+0x50/0xe0
[204365.820609] [<ffffffff811bfb5b>] handle_mm_fault+0xf8b/0x1820
[204365.820611] [<ffffffff8125572b>] ? ep_poll+0x37b/0x3d0
[204365.820614] [<ffffffff8106b537>] __do_page_fault+0x197/0x400
[204365.820616] [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
[204365.820618] [<ffffffff81826678>] page_fault+0x28/0x30
[204365.820620] Mem-Info:
[204365.820622] active_anon:351582 inactive_anon:118804 isolated_anon:0
active_file:48 inactive_file:52 isolated_file:0
unevictable:1149 dirty:0 writeback:0 unstable:0
slab_reclaimable:5357 slab_unreclaimable:8373
mapped:678 shmem:91 pagetables:2659 bounce:0
free:3341 free_pcp:60 free_cma:0

#11

Updated by C K almost 8 years ago

Sorry to reopen it again, getting this error too while recording.

I have ~5-10 recordings at one time and measured my disc write speed with

dd if=/dev/zero of=/home/waldmeister/test123 bs=1G count=1 oflag=direct

110 - 140 MB/s

How can i provide more tests?

#12

Updated by C K almost 8 years ago

So i really don't think thats HDD speed related. There must be another problem. Maybe timeshift?

#13

Updated by Jaroslav Kysela almost 8 years ago

Create a new issue. Provide screenshot of 'Config / Debugging / Memory information entries' when tvh eats too much memory.

#14

Updated by Jaroslav Kysela almost 8 years ago

And sync with latest master code again...

Also available in: Atom PDF