Bug #1713
Memory consumption keeps growing
0%
Description
It appears that tvheadend process continually consumes system memory (physical and swap) until such time that the OoM killer kills the process.
I have been having this issue with v3.4 and master branch of tvh.
Here is a process listing showing memory usage for tvh:
root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video
pmap says that there is roughly 14GB private memory mapped to this process.
mapped: 14836172K writeable/private: 13811404K shared: 4K
If there is any additional information I can provide please let me know.
if it helps, have built tvheadend as follows:
AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh -t debian
Files
History
Updated by Bruce McAlister over 11 years ago
Update:
Here is a sample message for when the OoM killer kicks in (circa 29GB virtual memory and 7.1GB physical memory used):
[61989.390964] Out of memory: Kill process 11963 (tvheadend) score 840 or sacrifice child
[61989.390979] Killed process 11963 (tvheadend) total-vm:28723844kB, anon-rss:7182860kB, file-rss:0kB
[61991.026644] init: tvheadend main process (11963) killed by KILL signal
I don't use tvheadend that often either which is what is strange. I may use it to watch a couple channels a night, but other than that, in terms of actual use, it is relatively idle.
Updated by Bjoern Franke over 11 years ago
Same issue here. tvh eats 500mb after running 3 days and some recordings.
Updated by Stig Larsson over 11 years ago
Yes, there has to be one (or more) memory leaks in recent versions of TVH.
I'm running build 3.5.101~gc80088a and after about two weeks of usage,
TVH's memory allocation is about 600 MB resident and 800 MB virtual.
I've now re-started TVH and I'll try to follow the increase.
Current usage is: 10 MB resident and about 200 MB virtual.
Updated by Andy Brown over 11 years ago
Confirmed also happening on 3.5.101~gc80088a (Latest git master)
Have had system force system to grind to a halt several times, doesn't seem related to viewing or not (memory climbs to critical overnight so difficult to tell)
I caught it several days ago with over 80% usage. Current usage is low (Started around 12 hours ago)
30723 2581 122220 351992 tvheadend ./build.linux/tvheadend -u tvheadend -g video -a 0,1,2 -s
However, now going to run through valgrind and see what I can find.
Updated by Pedro Hill over 11 years ago
I get these EOVERFLOW aswell:
ay 7 21:46:44 server tvheadend13636: dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 7 21:46:45 server tvheadend13636: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: CA #778: Continuity counter error, 1 duplicate log lines suppressed
#1130: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend[13636]: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: CA
May 7 21:46:45 server tvheadend13636: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: TELETEXT #7944: Continuity counter error, 1 duplicate log lines suppressed
#4101: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend[13636]: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: MPEG2VIDEO
May 7 21:46:45 server tvheadend13636: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: MPEG2AUDIO @ #4360: Continuity counter error, 1 duplicate log lines suppressed
free -m shows:
total used free shared buffers cached
Mem: 7589 7469 120 0 33 6380
-/+ buffers/cache: 1056 6533
Swap: 7785 0 7785
Updated by Adam Sutton over 11 years ago
- Status changed from New to Accepted
- Affected Versions 3.4 added
Hmmm, my message appears to have got lost from a few days back (think I sent from wrong email address):
--
Can you all give as detailed description of your setups as you can. Including details of tuners, config, use of descrambling, viewing habits.
We need to see if we can find some correlation.
I do think my own instance has a small leak, but its running at 180M after 2 weeks and its used for regular recording and live tv with mix of capmt and fta.
--
Update on that it's grown another 40M in 4 days, so clearly there is a leak, just that on my own setup its small enough to mostly go unnoticed (or at least it has little impact on my setup at the moment).
But I do want to get to the bottom of it.
Adam
Updated by Bruce McAlister over 11 years ago
I am also using a mixture of capmt and fta services.
I have a Hauppage WinTV Nova-T 500 DVB-T tuner and 1 quad port Digital Devices Cine-S2 DVB-S2 tuner.
Recently I have not been able to view much, but generally it is a mix between FTA and descrambled dvb-s/dvb-s2 channels.
I have not setup any recordings as of yet, but I inted to use it for recording down the line.
With regards the configuration of tvh itself, I dont think I have anything special. I add the generic config for dvb-t and delete all muxes except for 474Mhz which is where the FTA Irish channels are broadcast.
Then I have the 28.3 astra satalite config setup for the 4 ports on the dvb-s/s2 tuner, these pick up the freesat and several scrambled channels.
tvh eats memory pretty quick on my system, I can watch it grow by GB's per day. I now have a cron job scheduled every night to restart tvh or else it brings my box to its knees (literally eats all physical memory (12GB and all swap, 20GB). I only put the cron job in today, so hopefully my system will staty up for 24 hours. The last week it has crashed with 24 hours due to memory exhuastion.
I am currently running tvh 3.4 and this is happening, however, if needed, I can switch to master and run that if it helps with getting to the bottom of this (which was also presenting with the same memory issues)
Please let me know if you need additional info from me.
Updated by Pedro Hill over 11 years ago
I have a Asus E45M1-M Pro with 8gb of ram along with 2 Technotrend Budget C-1501 tv cards. I have 2 enabled cwc servers that i descramble the channels from. its pretty much running 18/24h through Xbmc, recording and just watching mixed (no i dont watch tv 18h a day). The issue seems to be most current when i record stuff witch results in alot of video drops, it records to a mdadm raid5 xfs.
Just let me know if you need any further information
Updated by Andy Brown over 11 years ago
Config data for mine (If everyone could use this basic template it will make bug-squashing this one easier):
General: Image Caching - ticked
re-fetch - 16800
re-try - 240
ignore invalid cert - unticked
Time Update - unticked
Enable ntp driver - unticked
Update tolerance - 0
Recording:
Media Container - same as source (pass-thru)
Skip commercials - checked
Post-processor command - none
Timeshift:
Enabled - ticked
On-Demand - unticked
Max.Period - n/a
Unlimited - ticked
Max.Size - n/a
Unlimited - ticked
CSA:
Code-Word-Client - enabled in use
Cap-mt connections - none
Debugging:
Debug to syslog - ticked
Debug trace - unticked
Updated by Pedro Hill over 11 years ago
A little update, this seems to happen to me when i have finished a recording. Here i recorded 2 programs at the same time, and when the first one was finished, the other one got EOVERFLOW errors:
May 10 21:02:00.014 subscription: "DVR: Sherlock Holmes" unsubscribing from "BBC HD"
May 10 21:02:00.015 dvr: Recording completed: "/home/hts/records/Sherlock.Holmes.2013-05-10.ts"
May 10 21:02:00.015 dvr: "Sherlock Holmes" on "BBC HD HD": End of program: Completed OK
May 10 21:03:21.110 dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 10 21:03:21.442 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: H264 #4130: Continuity counter error
#4642: Continuity counter error
May 10 21:03:21.452 dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 10 21:03:21.561 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: AC3
May 10 21:03:21.563 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: CA #1131: Continuity counter error
#772: Continuity counter error
May 10 21:03:21.676 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: CA
May 10 21:05:20.591 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: AC3 #4642: Continuity counter error
#4130: Continuity counter error
May 10 21:05:20.594 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: H264
Updated by Andy Brown over 11 years ago
I've setup memory logging for the TVH process, it logs every 5 minutes and shows vsz output:
Times shown beside show memory lost over this morning, so its gradually creeping upwards. 52% on my system now on tvh. I'm going to leave it and let it creep up, see if it causes the system failure.
Updated by Stig Larsson over 11 years ago
After 13 days of "normal usage", the memeory allocation is like this:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2978 hts 20 0 670m 472m 1572 S 16 23.6 2487:59 tvheadend
So the resident memory usage has increased from 10 MB to 472 MB.
Memory leak(s)? Very likely...
/Stig
Stig Larsson wrote:
Yes, there has to be one (or more) memory leaks in recent versions of TVH.
I'm running build 3.5.101~gc80088a and after about two weeks of usage,
TVH's memory allocation is about 600 MB resident and 800 MB virtual.
I've now re-started TVH and I'll try to follow the increase.
Current usage is: 10 MB resident and about 200 MB virtual.
Updated by Juanma Sanchez over 11 years ago
Hi,
I have been seeing this problem also. I have a somewhat busy tvheadend server with 28 dvb-s2 tuners, cwc, etc. If I disable idle scanning for all tuners the memory usage doesn't grow as fast as before, in fact it looks quite stable right now (the usage should be growing already after some hours since disabled idle scanning). I'll keep an eye on it and keep you informed.
If I don't see tvheadend consuming 8GB of RAM in the next 48-72 hours, for me it would be clear that the memory leak is in the idle scanning code.
Please, tell me if you need some tests from me.
Cheers.
Updated by Juanma Sanchez over 11 years ago
Hi,
This issue is definitely related to idle scanning. After 30hrs my tvheadend process should have grown to at least 3GB of resident mem and this is the situation right now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1115 hts 20 0 3786m 348m 2920 S 24 4.5 327:20.84 tvheadend
Could someone suffering this problem confirm if you have idle scanning enabled, and if disabling it makes the situation better?
Thanks.
Updated by Stig Larsson over 11 years ago
Hi Juanma (& all),
I have "idle scanning" enabled on my three DVB-C adapters and I have the problem as mentioned earlier in this thread/issue. I'll also see what happens when I turn it off.
"See you later"...
Juanma Sanchez wrote:
Hi,
This issue is definitely related to idle scanning. After 30hrs my tvheadend process should have grown to at least 3GB of resident mem and this is the situation right now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1115 hts 20 0 3786m 348m 2920 S 24 4.5 327:20.84 tvheadendCould someone suffering this problem confirm if you have idle scanning enabled, and if disabling it makes the situation better?
Thanks.
Updated by Bruce McAlister over 11 years ago
I have the Cine S2 with 4 ports, all 4 ports have idle scanning enabled. I will disable idle scanning shortly and restart TVH and see what it looks like. Right now my TVH process is using, according to top, 20.6GB virtual memory and 5.4GB physical memory. I'll disable idle scanning and see where we go afterwards:
3434 hts 20 0 20.6g 5.4g 1016 D 6 46.3 583:30.16 tvheadend
Updated by Juanma Sanchez over 11 years ago
Hi,
Wow, my tvheadend process grew a lot during the weekend:
1115 hts 20 0 4306m 1.2g 3028 S 34 16.0 2650:05 tvheadend
Anyway, when idle scanning was enabled the process used to last 3 or 4 days at most before the Oom killed it, so it looks like there are more than one memory leak.
Updated by Andy Brown over 11 years ago
Yes,
the same is happening to a few people, but not all.
PLEASE, could everyone having this problem log their settings so we can try and collate where the leak is taking place. See earlier posts for details.
Updated by Bruce McAlister over 11 years ago
- File DVB-S2_General_Settings.PNG DVB-S2_General_Settings.PNG added
- File general.PNG general.PNG added
- File recording_settings.PNG recording_settings.PNG added
- File time_shift_settings.PNG time_shift_settings.PNG added
Disabling the idle-scanning option certainly made a big difference, however, my tvh process still grows in memory usage. I have attached screenshots of most of my configuration settings.
I also have
[1] CWC enabled and working.
[2] No CAPMT configured
[3] Debugging tab, nothing configured
If I have missed something that would help please let me know.
Updated by Juanma Sanchez over 11 years ago
My settings:
Version: 3.5.121~g29f715f-dirty
./configure --enable-imagecache --enable-zlib --enable-inotify --enable-bundle --disable-libav --disable-timeshift
28 Adapters (7x Quad TBS 6985) with the following configuration:
{
"enabled": 1,
"fe_path": "/dev/dvb/adapter0/frontend0",
"dmx_path": "/dev/dvb/adapter0/demux0",
"dvr_path": "/dev/dvb/adapter0/dvr0",
"type": "DVB-S",
"displayname": "adapter0",
"autodiscovery": 1,
"idlescan": 0,
"idleclose": 0,
"skip_checksubscr": 0,
"sidtochan": 1,
"qmon": 1,
"poweroff": 0,
"nitoid": 0,
"diseqc_version": 0,
"diseqc_repeats": 0,
"extrapriority": 0,
"skip_initialscan": 1,
"disable_pmt_monitor": 0,
"full_mux_rx": -1,
"grace_period": 0
}
No IPTV Configuration
CWC is Enabled with 28 connections.
No recording is being done in this server. The Media container is set to pass through
All OTA EPG Grabbers are enabled. XMLTV and PyEPG are disabled
The DVR functionality is unused.
Updated by Andy Brown over 11 years ago
Potential memory leak fixes have just been commited. Could followers of the thread compile from master and see if these have plugged the hole?
Updated by Bruce McAlister over 11 years ago
- File tvh_compile_errors.txt tvh_compile_errors.txt added
I am getting compilation errors on master:
PS: I attached the full log as well
CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier from pointer target type [-Werror]
cc1: all warnings being treated as errors
make[2]: *** [/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make[2]: Leaving directory `/usr/local/src/tvheadend/master'
make[1]: *** [override_dh_auto_build] Error 2
make[1]: Leaving directory `/usr/local/src/tvheadend/master'
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
Updated by Adam Sutton over 11 years ago
Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.
I'll take a look.
Adam
On 5 July 2013 06:31, wrote:
Issue #1713 has been updated by Bruce McAlister.
- File tvh_compile_errors.txtadded
I am getting compilation errors on master:
PS: I attached the full log as well
CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make2:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make2: Leaving directory `/usr/local/src/tvheadend/master'
make1: * [override_dh_auto_build] Error 2
make1: Leaving directory `/usr/local/src/tvheadend/master'
make: ** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.I have been having this issue with v3.4 and master branch of tvh.
Here is a process listing showing memory usage for tvh:
root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g videopmap says that there is roughly 14GB private memory mapped to this process.
mapped: 14836172K writeable/private: 13811404K shared: 4K
If there is any additional information I can provide please let me know.
if it helps, have built tvheadend as follows:
AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh
t debian
-----------------------------You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account
Updated by Adam Sutton over 11 years ago
Actually, no, its more that I just didn't check with --enable-bundle, which
is required to hit that bit of code.
I'll put that back the way it was and that should fix things.
Adam
On 5 July 2013 09:28, Adam Sutton wrote:
Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.I'll take a look.
Adam
On 5 July 2013 06:31, wrote:
Issue #1713 has been updated by Bruce McAlister.
- File tvh_compile_errors.txtadded
I am getting compilation errors on master:
PS: I attached the full log as well
CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make2:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make2: Leaving directory `/usr/local/src/tvheadend/master'
make1: * [override_dh_auto_build] Error 2
make1: Leaving directory `/usr/local/src/tvheadend/master'
make: ** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.I have been having this issue with v3.4 and master branch of tvh.
Here is a process listing showing memory usage for tvh:
root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g videopmap says that there is roughly 14GB private memory mapped to this
process.mapped: 14836172K writeable/private: 13811404K shared: 4K
If there is any additional information I can provide please let me know.
if it helps, have built tvheadend as follows:
AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh
t debian
-----------------------------You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account
Updated by Adam Sutton over 11 years ago
Ok, code reverted, try it now.
Adam
On 5 July 2013 09:31, Adam Sutton wrote:
Actually, no, its more that I just didn't check with --enable-bundle,
which is required to hit that bit of code.I'll put that back the way it was and that should fix things.
Adam
On 5 July 2013 09:28, Adam Sutton wrote:
Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.I'll take a look.
Adam
On 5 July 2013 06:31, wrote:
Issue #1713 has been updated by Bruce McAlister.
- File tvh_compile_errors.txtadded
I am getting compilation errors on master:
PS: I attached the full log as well
CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make2:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make2: Leaving directory `/usr/local/src/tvheadend/master'
make1: * [override_dh_auto_build] Error 2
make1: Leaving directory `/usr/local/src/tvheadend/master'
make: ** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.I have been having this issue with v3.4 and master branch of tvh.
Here is a process listing showing memory usage for tvh:
root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g videopmap says that there is roughly 14GB private memory mapped to this
process.mapped: 14836172K writeable/private: 13811404K shared: 4K
If there is any additional information I can provide please let me know.
if it helps, have built tvheadend as follows:
AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh
t debian
-----------------------------You have received this notification because you have either subscribed
to it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account
Updated by Bruce McAlister over 11 years ago
OK, it compiled, and I have this version running. I have re-enabled idle scanning on the 4 ports so I'll watch it for a while, current memory usage is as follows:
19858 hts 20 0 2078m 109m 3284 D 0 0.9 0:04.32 tvheadend
2GB VIRT and 109MB RSS
Updated by Andy Brown over 11 years ago
- Status changed from Accepted to Need feedback
So far, looking very good. Code stable and no noticeable memory increases so far, same settings in use as before.
Updated by Bruce McAlister over 11 years ago
This change looks good, I've been running it for the weekend and the memory usage has stayed pretty constant over the weekend. Memory usage is still at 2GB VIRT and around 115MB RSS, pretty much the same as when I restarted TVH on Friday night.
Thanks for your time and effort everyone involved
Updated by Juanma Sanchez over 11 years ago
Looks good for me after 1 week.
I re-enabled the idle scanning for all 28 tuners and memory consumption is more or less the same as when I started the process.
Good job guys, thanks for fixing this
Updated by Adam Sutton almost 11 years ago
- Status changed from Need feedback to Fixed
I don't think any more work needs doing here.