Tvheadend

Update:

Here is a sample message for when the OoM killer kicks in (circa 29GB virtual memory and 7.1GB physical memory used):

[61989.390964] Out of memory: Kill process 11963 (tvheadend) score 840 or sacrifice child
[61989.390979] Killed process 11963 (tvheadend) total-vm:28723844kB, anon-rss:7182860kB, file-rss:0kB
[61991.026644] init: tvheadend main process (11963) killed by KILL signal

I don't use tvheadend that often either which is what is strange. I may use it to watch a couple channels a night, but other than that, in terms of actual use, it is relatively idle.

Same issue here. tvh eats 500mb after running 3 days and some recordings.

I can also confirm the growing memory consumption.

Yes, there has to be one (or more) memory leaks in recent versions of TVH.
I'm running build 3.5.101~gc80088a and after about two weeks of usage,
TVH's memory allocation is about 600 MB resident and 800 MB virtual.
I've now re-started TVH and I'll try to follow the increase.
Current usage is: 10 MB resident and about 200 MB virtual.

Confirmed also happening on 3.5.101~gc80088a (Latest git master)

Have had system force system to grind to a halt several times, doesn't seem related to viewing or not (memory climbs to critical overnight so difficult to tell)

I caught it several days ago with over 80% usage. Current usage is low (Started around 12 hours ago)

30723 2581 122220 351992 tvheadend ./build.linux/tvheadend -u tvheadend -g video -a 0,1,2 -s

However, now going to run through valgrind and see what I can find.

Output of valgrind session: http://pastebin.com/HSDPgvT2

I get these EOVERFLOW aswell:
ay 7 21:46:44 server tvheadend¹³⁶³⁶: dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 7 21:46:45 server tvheadend¹³⁶³⁶: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: CA #778: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend[13636]: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: CA #1130: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend¹³⁶³⁶: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: TELETEXT #7944: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend[13636]: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: MPEG2VIDEO #4101: Continuity counter error, 1 duplicate log lines suppressed
May 7 21:46:45 server tvheadend¹³⁶³⁶: TS: Philips TDA10023 DVB-C/314,000 kHz/TV3: MPEG2AUDIO @ #4360: Continuity counter error, 1 duplicate log lines suppressed

free -m shows:

total       used       free     shared    buffers     cached
Mem:          7589       7469        120          0         33       6380
-/+ buffers/cache:       1056       6533
Swap:         7785          0       7785

I am also using a mixture of capmt and fta services.

I have a Hauppage WinTV Nova-T 500 DVB-T tuner and 1 quad port Digital Devices Cine-S2 DVB-S2 tuner.

Recently I have not been able to view much, but generally it is a mix between FTA and descrambled dvb-s/dvb-s2 channels.

I have not setup any recordings as of yet, but I inted to use it for recording down the line.

With regards the configuration of tvh itself, I dont think I have anything special. I add the generic config for dvb-t and delete all muxes except for 474Mhz which is where the FTA Irish channels are broadcast.

Then I have the 28.3 astra satalite config setup for the 4 ports on the dvb-s/s2 tuner, these pick up the freesat and several scrambled channels.

tvh eats memory pretty quick on my system, I can watch it grow by GB's per day. I now have a cron job scheduled every night to restart tvh or else it brings my box to its knees (literally eats all physical memory (12GB and all swap, 20GB). I only put the cron job in today, so hopefully my system will staty up for 24 hours. The last week it has crashed with 24 hours due to memory exhuastion.

I am currently running tvh 3.4 and this is happening, however, if needed, I can switch to master and run that if it helps with getting to the bottom of this (which was also presenting with the same memory issues)

Please let me know if you need additional info from me.

I have a Asus E45M1-M Pro with 8gb of ram along with 2 Technotrend Budget C-1501 tv cards. I have 2 enabled cwc servers that i descramble the channels from. its pretty much running 18/24h through Xbmc, recording and just watching mixed (no i dont watch tv 18h a day). The issue seems to be most current when i record stuff witch results in alot of video drops, it records to a mdadm raid5 xfs.
Just let me know if you need any further information

Config data for mine (If everyone could use this basic template it will make bug-squashing this one easier):

General: Image Caching - ticked
re-fetch - 16800
re-try - 240
ignore invalid cert - unticked
Time Update - unticked
Enable ntp driver - unticked
Update tolerance - 0
Recording:
Media Container - same as source (pass-thru)
Skip commercials - checked
Post-processor command - none
Timeshift:
Enabled - ticked
On-Demand - unticked
Max.Period - n/a
Unlimited - ticked
Max.Size - n/a
Unlimited - ticked
CSA:
Code-Word-Client - enabled in use
Cap-mt connections - none
Debugging:
Debug to syslog - ticked
Debug trace - unticked

A little update, this seems to happen to me when i have finished a recording. Here i recorded 2 programs at the same time, and when the first one was finished, the other one got EOVERFLOW errors:

May 10 21:02:00.014 subscription: "DVR: Sherlock Holmes" unsubscribing from "BBC HD"
May 10 21:02:00.015 dvr: Recording completed: "/home/hts/records/Sherlock.Holmes.2013-05-10.ts"
May 10 21:02:00.015 dvr: "Sherlock Holmes" on "BBC HD HD": End of program: Completed OK
May 10 21:03:21.110 dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 10 21:03:21.442 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: H264 #4130: Continuity counter error
May 10 21:03:21.452 dvb: "_dev_dvb_adapter0_Philips_TDA10023_DVB_C" read() EOVERFLOW
May 10 21:03:21.561 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: AC3 #4642: Continuity counter error
May 10 21:03:21.563 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: CA #1131: Continuity counter error
May 10 21:03:21.676 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: CA #772: Continuity counter error
May 10 21:05:20.591 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: AC3 #4642: Continuity counter error
May 10 21:05:20.594 TS: Philips TDA10023 DVB-C/314,000 kHz/RAI HD: H264 #4130: Continuity counter error

I've setup memory logging for the TVH process, it logs every 5 minutes and shows vsz output:

http://pastebin.com/e79kvDkN

Times shown beside show memory lost over this morning, so its gradually creeping upwards. 52% on my system now on tvh. I'm going to leave it and let it creep up, see if it causes the system failure.

After 13 days of "normal usage", the memeory allocation is like this:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2978 hts       20   0  670m 472m 1572 S   16 23.6   2487:59 tvheadend

So the resident memory usage has increased from 10 MB to 472 MB.
Memory leak(s)? Very likely...

/Stig

Stig Larsson wrote:

Yes, there has to be one (or more) memory leaks in recent versions of TVH.
I'm running build 3.5.101~gc80088a and after about two weeks of usage,
TVH's memory allocation is about 600 MB resident and 800 MB virtual.
I've now re-started TVH and I'll try to follow the increase.
Current usage is: 10 MB resident and about 200 MB virtual.

Hi,

I have been seeing this problem also. I have a somewhat busy tvheadend server with 28 dvb-s2 tuners, cwc, etc. If I disable idle scanning for all tuners the memory usage doesn't grow as fast as before, in fact it looks quite stable right now (the usage should be growing already after some hours since disabled idle scanning). I'll keep an eye on it and keep you informed.
If I don't see tvheadend consuming 8GB of RAM in the next 48-72 hours, for me it would be clear that the memory leak is in the idle scanning code.

Please, tell me if you need some tests from me.

Cheers.

Hi,

This issue is definitely related to idle scanning. After 30hrs my tvheadend process should have grown to at least 3GB of resident mem and this is the situation right now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1115 hts 20 0 3786m 348m 2920 S 24 4.5 327:20.84 tvheadend

Could someone suffering this problem confirm if you have idle scanning enabled, and if disabling it makes the situation better?

Thanks.

Hi Juanma (& all),

I have "idle scanning" enabled on my three DVB-C adapters and I have the problem as mentioned earlier in this thread/issue. I'll also see what happens when I turn it off.

"See you later"...

Juanma Sanchez wrote:

Hi,

This issue is definitely related to idle scanning. After 30hrs my tvheadend process should have grown to at least 3GB of resident mem and this is the situation right now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1115 hts 20 0 3786m 348m 2920 S 24 4.5 327:20.84 tvheadend

Could someone suffering this problem confirm if you have idle scanning enabled, and if disabling it makes the situation better?

Thanks.

I have the Cine S2 with 4 ports, all 4 ports have idle scanning enabled. I will disable idle scanning shortly and restart TVH and see what it looks like. Right now my TVH process is using, according to top, 20.6GB virtual memory and 5.4GB physical memory. I'll disable idle scanning and see where we go afterwards:

3434 hts       20   0 20.6g 5.4g 1016 D    6 46.3 583:30.16 tvheadend

Hi,

Wow, my tvheadend process grew a lot during the weekend:
1115 hts 20 0 4306m 1.2g 3028 S 34 16.0 2650:05 tvheadend

Anyway, when idle scanning was enabled the process used to last 3 or 4 days at most before the Oom killed it, so it looks like there are more than one memory leak.

Yes,
the same is happening to a few people, but not all.

PLEASE, could everyone having this problem log their settings so we can try and collate where the leak is taking place. See earlier posts for details.

My settings:

Version: 3.5.121~g29f715f-dirty
./configure --enable-imagecache --enable-zlib --enable-inotify --enable-bundle --disable-libav --disable-timeshift

28 Adapters (7x Quad TBS 6985) with the following configuration: {
"enabled": 1,
"fe_path": "/dev/dvb/adapter0/frontend0",
"dmx_path": "/dev/dvb/adapter0/demux0",
"dvr_path": "/dev/dvb/adapter0/dvr0",
"type": "DVB-S",
"displayname": "adapter0",
"autodiscovery": 1,
"idlescan": 0,
"idleclose": 0,
"skip_checksubscr": 0,
"sidtochan": 1,
"qmon": 1,
"poweroff": 0,
"nitoid": 0,
"diseqc_version": 0,
"diseqc_repeats": 0,
"extrapriority": 0,
"skip_initialscan": 1,
"disable_pmt_monitor": 0,
"full_mux_rx": -1,
"grace_period": 0
}

No IPTV Configuration

CWC is Enabled with 28 connections.
No recording is being done in this server. The Media container is set to pass through

All OTA EPG Grabbers are enabled. XMLTV and PyEPG are disabled

The DVR functionality is unused.

Potential memory leak fixes have just been commited. Could followers of the thread compile from master and see if these have plugged the hole?

Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.

I'll take a look.

Adam

On 5 July 2013 06:31, wrote:

Issue #1713 has been updated by Bruce McAlister.

- File tvh_compile_errors.txtadded

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make²:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make²: Leaving directory `/usr/local/src/tvheadend/master'
make<sup>1</sup>: * [override_dh_auto_build] Error 2
make<sup>1</sup>: Leaving directory `/usr/local/src/tvheadend/master'
make: ** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing

- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6

It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh t debian
-----------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

Actually, no, its more that I just didn't check with --enable-bundle, which
is required to hit that bit of code.

I'll put that back the way it was and that should fix things.

Adam

On 5 July 2013 09:28, Adam Sutton wrote:

Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.

I'll take a look.

Adam

On 5 July 2013 06:31, wrote:

Issue #1713 has been updated by Bruce McAlister.

- File tvh_compile_errors.txtadded

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make²:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make²: Leaving directory `/usr/local/src/tvheadend/master'
make<sup>1</sup>: * [override_dh_auto_build] Error 2
make<sup>1</sup>: Leaving directory `/usr/local/src/tvheadend/master'
make: ** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing

- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6

It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this
process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh t debian
-----------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

Ok, code reverted, try it now.

Adam

On 5 July 2013 09:31, Adam Sutton wrote:

Actually, no, its more that I just didn't check with --enable-bundle,
which is required to hit that bit of code.

I'll put that back the way it was and that should fix things.

Adam

On 5 July 2013 09:28, Adam Sutton wrote:

Yes, I wasn't sure about this mod, I had a feeling there must have been a
reason why I created a seemingly redundant copy of the data. My own zlib
doesn't appear to have this limitation (i.e that the input data is not
const, but presumably different implementations may do things slightly
differently.

I'll take a look.

Adam

On 5 July 2013 06:31, wrote:

Issue #1713 has been updated by Bruce McAlister.

- File tvh_compile_errors.txtadded

I am getting compilation errors on master:

PS: I attached the full log as well

CC src/filebundle.o
src/filebundle.c: In function ‘_fb_inflate’:
src/filebundle.c:90:18: error: assignment discards ‘const’ qualifier
from pointer target type [-Werror]
cc1: all warnings being treated as errors
make²:
[/usr/local/src/tvheadend/master/build.linux/src/filebundle.o] Error 1
make²: Leaving directory `/usr/local/src/tvheadend/master'
make<sup>1</sup>: * [override_dh_auto_build] Error 2
make<sup>1</sup>: Leaving directory `/usr/local/src/tvheadend/master'
make: ** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
------------------------------
Bug #1713: Memory consumption keeps growing

- Author: Bruce McAlister
- Status: Accepted
- Priority: Normal
- Assignee:
- Category: General
- Target version:
- Found in version: 3.4 and 3.5.116~ga2ccbb2
- Affected Versions: 3.4, 3.6

It appears that tvheadend process continually consumes system memory
(physical and swap) until such time that the OoM killer kills the process.

I have been having this issue with v3.4 and master branch of tvh.

Here is a process listing showing memory usage for tvh:

root@vms:~# ps -e -opid,ppid,rss,vsz,comm,args | sort -rnk4 | head -10
3535 1 10023280 14836172 tvheadend tvheadend -f -u hts -g video

pmap says that there is roughly 14GB private memory mapped to this
process.

mapped: 14836172K writeable/private: 13811404K shared: 4K

If there is any additional information I can provide please let me know.

if it helps, have built tvheadend as follows:

AUTOBUILD_CONFIGURE_EXTRA="--enable-bundle" ./Autobuild.sh t debian
-----------------------------

You have received this notification because you have either subscribed
to it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

OK, it compiled, and I have this version running. I have re-enabled idle scanning on the 4 ports so I'll watch it for a while, current memory usage is as follows:

19858 hts 20 0 2078m 109m 3284 D 0 0.9 0:04.32 tvheadend

2GB VIRT and 109MB RSS

This change looks good, I've been running it for the weekend and the memory usage has stayed pretty constant over the weekend. Memory usage is still at 2GB VIRT and around 115MB RSS, pretty much the same as when I restarted TVH on Friday night.

Thanks for your time and effort everyone involved :)

Looks good for me after 1 week.
I re-enabled the idle scanning for all 28 tuners and memory consumption is more or less the same as when I started the process.

Good job guys, thanks for fixing this :)