Bug #3825
open
Timeshift to 'disk only' is consuming all system RAM
Added by Steve H almost 9 years ago. Updated over 5 years ago.
0%
Description
I've set timeshift to use disk only on the Wetek Play, but timeshifting eventually consumes available RAM (usually within about 15 minutes for an HD channel). I've even tried setting the "Maximum RAM size (MB)" to 1, just in case having it set as 0 was being interpreted as unlimited.
I'm currently running 4.1-2047 from the builds at http://build.mycvh.de/, but have also seen this in the older 4.0.9 from the repo
When available RAM eventually drops to around 80M, the dmesg log gets filled with:-
[ 745.936394@0] balance_pgdat end zone=1, pgdat_is_balanced=0
[ 745.936528@0] balance_pgdat, nr_reclaimed=8
[ 745.936559@0] balance_pgdat end zone=1, pgdat_is_balanced=0
[ 745.936706@0] balance_pgdat, nr_reclaimed=30
Once playback of the channel is stopped, all the comsumed RAM is released back to the system again.
The only way to keep timeshifting enabled without consuming RAM is to enable "On-demand (no first rewind)" so it would only start buffering if/when I pause TV.
Files
| Screen Shot 2016-05-22 at 22.27.26.png (36.2 KB) Screen Shot 2016-05-22 at 22.27.26.png | Steve H, 2016-05-25 20:41 |
Updated by Jaroslav Kysela almost 9 years ago
Provide '--trace timeshift' - https://tvheadend.org/projects/tvheadend/wiki/Traces .
Updated by Steve H almost 9 years ago
Jaroslav Kysela wrote:
Provide '--trace timeshift' - https://tvheadend.org/projects/tvheadend/wiki/Traces .
Since posting, I've updated to tvheadend 4.1-2088. The problem is still there
Log file uploaded to http://sprunge.us/aZfi
Updated by Jaroslav Kysela almost 9 years ago
OK, the timeshift code does not try to allocate RAM segments. It looks like another bug - hidden memory leak - which might be similar to #3770 . Unfortunately, I'm out of time to play with wetek now.
Updated by Steve H almost 9 years ago
OK, so what happens with this issue moving forward ?
Is it 'on the radar' to be investigated, or is the "I'm out of time to play with wetek now" the end of the story ?
Updated by Jaroslav Kysela almost 9 years ago
Unfortunately, these devices do not have tools to debug this issue in LE/OE, so I cannot instruct you what to do, unless you install debian or any other full distro on it (chroot might be sufficient).
I fixed some other memory leak issues, so you may give a try with newer versions of tvh.
Updated by Steve H almost 9 years ago
Thanks. I am staying up to date with the releases from http://build.mycvh.de/ but unfortunately, the memory issues that have been fixed have not cured this problem.
I appreciate it is difficult to debug these issues on ARM devices with stripped down OS running.
There is one further observation I have made, and maybe this might help shed more light on the possible cause.
If you record a TV programme without even watching a live broadcast, even with the "On-demand (no first rewind)" enabled, RAM is consumed until exhausted.
Updated by John Doe over 8 years ago
Jaroslav Kysela wrote:
Unfortunately, these devices do not have tools to debug this issue in LE/OE, so I cannot instruct you what to do, unless you install debian or any other full distro on it (chroot might be sufficient).
I fixed some other memory leak issues, so you may give a try with newer versions of tvh.
Unfortunately this issue seems to be there still. I'm running HTS Tvheadend 4.1-2277~gbf75b27 and tried to use timeshift with Kodi 16.1, but even without pausing it just eats up all my RAM.
Updated by Argus Nymus almost 8 years ago
I'm using Tvheadend server + pvr.hts client on LibreELEC (https://forum.libreelec.tv/thread/2156-8-0-1l-libreelec-8-0-for-s905-s905x), and had a similar issue.
Independent of any Tvheadend timeshift settings, the OS would go OOM after having paused live TV for a while. `top` showed that the kodi process' mem increased, while Tvheadend remained unchanged and also no timeshift files were created.
So I assumed that the 'pause' command did not go through to Tvheadend, and Kodi did the buffering...
But strangely, after having recently updated LibreELEC (from 8.0.1k to l) and thus Kodi, and w/o the Tvheadend server and client addons having been updated iirc, the issue is gone. Memory consumption is not increasing beyond the configured limit, and timeshift files are created after that.
Weird, but you might want to try again with recent tvheadend server+client + Kodi. :-)
Updated by th0ma7 ^ almost 6 years ago
I believe I'm having the exact same problem.
Using HTS Tvheadend 4.2.8 with Kodi 18.2 with tvheadend plugin and timeshift activated.
I first thought this was driver related:
https://github.com/b-rad-NDi/Embedded-MediaDrivers/issues/5
but now I strongly believe it's a tvheadend issue.
Updated by th0ma7 ^ almost 6 years ago
To add to this, I've tested multiple variances over the timeshift option.
In all cases it ended-up generating an OOM killer over tvheadend.
Duration varied depending of selected options.
- Enabled
- Max period: 60 minutes
- Storage Path: /volume1/TV
- Maximum size (MB): 10000
- Maximum RAM size (MB): 4000 (note that I have 16GB ram)
Nothing else selected (e.g. RAM only, On-demand (no first rewind), Unlimited time, Unlimited size, Fit to RAM (cut rewind), Include teletext)
My first assumptions are that setting Max period to 0 (e.g. unlimited) or selecting Unlimited time and/or Unlimited size did resulted in OOM killer due to memory leaking.
Also note that I am running on a Synology DS918+ with the following:
$ uname -a Linux th0ma7-nas 4.4.59+ #24922 SMP PREEMPT Thu Mar 28 11:07:03 CST 2019 x86_64 GNU/Linux synology_apollolake_918+
Updated by th0ma7 ^ almost 6 years ago
Still getting OOM killer, it just takes longer with above parameters (a few more hours to a few days) before it do hapen.
Let me know how I can help to gather data and catch where the bug is.
Cheers.
Updated by Flole Systems almost 6 years ago
Use valgrind to see if there is a memory leak.
Also you can use gdb to see what exactly is eating up the memory. You need to do this before the crash though.
Updated by th0ma7 ^ over 5 years ago
I can certainly do that! Having a few hints on how to proceed?
I can easily install gdb and/or valgrind as needed:
$ sudo /opt/bin/opkg list | grep -i -e valgrind -e gdb gdb - 8.2.1-1 - GDB, the GNU Project debugger valgrind - 3.15.0-1 - Valgrind...
Updated by Flole Systems over 5 years ago
Install valgrind and follow the steps in the wiki. Everything will be slow but it's normally sufficient to start a stream and pause it to cause timeshift to start. Only if that doesnt find anything you can use GDB, you want to look into the allocated memory there to see whats used for what.
Updated by th0ma7 ^ over 5 years ago
Humm, so I tried using either valgrind or gdb but without much luck (not using theses toold that often neither).
Synology boxes might not be that user friendly neither as I'm guessing debug libs may to be missing (guessing)?
A few hints into getting debug tools to work would be quite handy...
Valgrind output:
==23225== Memcheck, a memory error detector ==23225== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==23225== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info ==23225== Command: /volume1/@appstore/tvheadend/bin/tvheadend -f -u sc-tvheadend -g tvheadend --http_port 9981 --htsp_port 9982 -c /volume1/@appstore/tvheadend/var -p /volume1/@appstore/tvheadend/var/tvheadend.pid ==23225== valgrind: Fatal error at startup: a function redirection valgrind: which is mandatory for this platform-tool combination valgrind: cannot be set up. Details of the redirection are: valgrind: valgrind: A must-be-redirected function valgrind: whose name matches the pattern: strlen valgrind: in an object with soname matching: ld-linux-x86-64.so.2 valgrind: was not found whilst processing valgrind: symbols from the object with soname: ld-linux-x86-64.so.2 ...
And here is GDB attached to the PID:
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. 0x00007fcfd8b0ac48 in pthread_cond_timedwait () from /lib/libpthread.so.0 (gdb) set logging on Copying output to gdb.txt. (gdb) set pagination off (gdb) bt full #0 0x00007fcfd8b0ac48 in pthread_cond_timedwait () from /lib/libpthread.so.0 No symbol table info available. #1 0x00005617f8f3b5c7 in ?? () No symbol table info available. #2 0x00007fcfd826c030 in __libc_start_main () from /lib/libc.so.6 No symbol table info available. #3 0x00005617f8f3cb71 in ?? () No symbol table info available. (gdb) continue Continuing. [Detaching after fork from child process 25805] [New LWP 25813] [New LWP 25814] [LWP 25813 exited] [LWP 24797 exited] [New LWP 25945] [New LWP 25946] [New LWP 25947]