Bug #1383
Memory leak in git as from yesterday, 4th november 2012
0%
Description
Hi,
I do hope info is as required.
I'm stressing tvheadend probably to much because when I start 6 recordings from 1 DVB-S mux, memory usage starts to increase after a few minutes with about 4MB per second.
After stopping tvheadend, and restarting it's also impossible to restart the recordings, subsequently adding a recording works but if there are more then 1 there are all kinds of error messages like
Nov 06 08:23:43 [WARNING]:TS: Sundtek DVB-S/S2/CANALDIGITAAL: 12,187,000 kHz Horizontal (Default (Port 0, Universal LNB))/NET5 HD: Transport error indicator
Nov 06 08:23:43 [WARNING]:TS: Sundtek DVB-S/S2/CANALDIGITAAL: 12,187,000 kHz Horizontal (Default (Port 0, Universal LNB))/Veronica/DisneyXD HD: Transport error indicator
Nov 06 08:23:43 [WARNING]:TS: Sundtek DVB-S/S2/CANALDIGITAAL: 12,187,000 kHz Horizontal (Default (Port 0, Universal LNB))/RTL7 HD: Transport error indicator
Nov 06 08:23:43 [WARNING]:TS: Sundtek DVB-S/S2/CANALDIGITAAL: 12,187,000 kHz Horizontal (Default (Port 0, Universal LNB))/RTL5 HD: Transport error indicator
Nov 06 08:23:43 [WARNING]:TS: Sundtek DVB-S/S2/CANALDIGITAAL: 12,187,000 kHz Horizontal (Default (Port 0, Universal LNB))/Canvas HD: Transport error indicator
If multiple recordings start at the same time, which often occurs, the problem is the same.
History
Updated by Adam Sutton about 12 years ago
- Status changed from New to Need feedback
- Affected Versions 3.3 added
- Affected Versions deleted (
3.4)
See if the performance is the same if you disable full mux RX (in the adapter config tab).
Adam
Updated by Henk Schoneveld about 12 years ago
First of all thank you very much for your prompt reaction. Very much appreciated. Did a fresh install, new git clone, I don't see the mentioned option in adapter configuration. Do I have another version ?
Now it worked Ok for 10 minutes then the continuity errors apear and the memory consumption starts. With 7 recordings with 10MB per second.
I also get a segmentation fault when memory consumption is about 1.2GB of 4GB available,
Fault address 0x1374e8c (address not mapped)
Henk Schoneveld
Updated by Adam Sutton about 12 years ago
Hmm, Andreas has been plugging a few memory leaks last few days but I don't think he has anything that directly relates to this issue.
On the adapter config tab there should be a whole load of options (in the middle box), there should be something in there about full mux RX ('Disable full MUX reception'). If you can't see that option something is very wrong or you are not building from master.
Maybe include a full debug log just for clarity.
Adam
Updated by Henk Schoneveld about 12 years ago
Hi Adam,
I think I found the reason for the memory-eating. Played with 2.99 and did see the same behaviour. After stopping the last recording and restarting everything went fine with 5 concurrend recordings.
Tried again with current git, know what, also runs fine. It looks like somewehere 5 is defined as max concurrend streams from a given mux, I'm not a programmer, but (address not mapped) I could associate with that. Don't laugh please ;-)
Updated by Adam Sutton about 12 years ago
There is nothing that limits the number of streams from a given mux, I know several users that stream more than that (IPTV services etc..) using TVH.
However that doesn't mean that there isn't an issue in the DVR code that causes problems above some certain threshold, though not sure why it would.
I'll try and have a look to see if I can repeat when I get a chance.
Adam
Updated by Henk Schoneveld about 12 years ago
Thanks for your answers. Also added another adapter, but if total recordings exceeds 5 I get in the same boat again. Parser errors, resulting in "blockines" video.
Nov 07 19:29:00 [ERROR]:parser: transport stream H264, DTS discontinuity. DTS = 1672200, last = 793800
Nov 07 19:29:00 [ERROR]:parser: transport stream MPEG2AUDIO, DTS discontinuity. DTS = 1646692, last = 724372
Nov 07 19:29:10 [ERROR]:parser: transport stream H264, DTS discontinuity. DTS = 1773000, last = 869400
Nov 07 19:29:11 [ERROR]:parser: transport stream MPEG2AUDIO, DTS discontinuity. DTS = 1696378, last = 787018
Those IPTV services, how many channels do they deliver per tvheadend instance ?
I can imagine 1 machine 5 adapters with 5 channels to x users would work without any problem. If done otherwise I would like to know how they do it.
Updated by Adam Sutton about 12 years ago
The particular system I'm thinking of has several setups. One has a single quad tuner card running approx 40 services (constantly streamed to another machine - transcoder). But also has a setup with 4 quad tuner cards, not sure how many services (I don't think its many more services, just greater number of muxes covered).
Now that system does experience some issues, probably because that really is pushing the boundaries but we are trying to fix them.
I can certainly imagine that the full mux RX might add a slight increase in load, so its worth trying to disable it and seeing what happens.
And maybe attach a more complete debug log, just so we can see what's going on.
Adam
Updated by Henk Schoneveld about 12 years ago
As mentioned my git version doesn't have that option. System load is 5%, so that isn't the problem either. I tried TS (pass through) but that results in .bin files where mplayer and mediainfo cannot find any video or audio in.
Looks like I'm having a something different version from git then wat you are expecting I should get. Tomorow I'll try different instances of tvheadend each with it's own adapter and see what happens.
Henk
Updated by Adam Sutton about 12 years ago
If you're not seeing the disable full mux option and getting .bin files for passthru then you need to update your git tree.
Adam
Updated by Henk Schoneveld about 12 years ago
I've done that in the weekend and yesterday as well. The used command:
git clone https://github.com/tvheadend/tvheadend.git
Do i have to specify anything else ?
Henk
Updated by Adam Sutton about 12 years ago
Can you double check the git revision being reported by TVH on startup?
It's just the .bin issue should have been fixed, and there should definitely be a disable full mux option on the adapter tab.
Adam
Updated by Henk Schoneveld about 12 years ago
Did a new git clone, version is HTS Tvheadend 3.3.133~gddad1d2
There isn't mux RX ('Disable full MUX reception') in configuration of the card in the midle. Further testing follows
Updated by Henk Schoneveld about 12 years ago
Things arent' getting different. Which logs would you like to receive ? I see I can set log to console, but how do I capture that to a file ?
Henk
Updated by Henk Schoneveld about 12 years ago
What I thin to see that I have 5 active subscriptions, with unrealistic low bandwith and only 1 file is written. I almost looks like the NOT written files endup in memory.
Henk
Updated by Adam Sutton about 12 years ago
There may be an element of truth to that last statement. We did recently add some buffer limiting in some aspects of the code that had previously been missing. However nothing was added to the DVR chain as it was assumed that you should be able to write to file at greater than the incoming data rate else stuff would just start to fall apart.
So indeed if your disks simply cannot keep up with the number of things you are trying to record it is indeed possible that it could be buffering it all (unlimited) to memory.
I could possibly create a patch that would limit the buffering to at least stop the memory leak, but in reality that won't solve much (long term) since you will simply get corrupted recordings which will probably not be off much use.
If you start TVH manually ./build.linux/tvheadend, then add -d (debug to console) and pipe stderr to file, i.e:
./build.linux/tvheadend -d 2> tvh.log
Adam
Updated by Henk Schoneveld about 12 years ago
I think i know when things go wrong. If ts pass through is chosen there is no memory eating. Only the resulting file doesn't have audio or video according to mplayer and mediainfo. Disk I/O isn't the problem as you were assuming. When setting recording to mkv there almost isn't any disk io. When choosing ts pass through and there then is normal bandwith use according to "active subscriptions" then according to iostat there isn't a I/O problem.
Linux 3.3.6-desktop-2.mga2 (i5) 11/08/2012 i686 (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
2.73 0.00 0.88 2.22 0.00 94.17
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 3.46 76.79 29.35 286344 109446
sdb 21.32 1.61 2446.91 5993 9124160
As you described your "particular system" you probably are streaming in ts-format to the encoding machines and that will work Ok
When I start to see follwing in the log
08 13:53:44 [ERROR]:parser: transport stream H264, DTS discontinuity. DTS = 5776200, last = 4807800
Nov 08 13:53:44 [ERROR]:parser: transport stream MPEG2AUDIO, DTS discontinuity. DTS = 5741338, last = 4728298
I do observe the dropping in bandwith in active subscriptions to values of 250 - 350k where it should be between 5000 and 20.000k.
In my opinion there are 2 things that have to be investigated, probably the mentioned buffering and ts pass through should result in playable ts files.
Updated by Adam Sutton about 12 years ago
Hmmm interesting,
I will definitely try this out. I can say that originally the setup I described was using MKV, however since there were general concerns about loading some of it was switched to TS (however that pre-dated John's fixes and so performance was actually worse at the time). No idea whether the guy is using MKV or TS at the moment, probably MKV (which reminds me I need to check).
I think the two issues you mention are somehow linked, though I don't know how yet. Clearly others are able to stream far higher loads than you are currently having problems with which does suggest there is something specific about your setup which is causing problems.
That's not to say those problems aren't bugs in TVH, they may well be, just might make it a little tougher to track down.
I still haven't had a go and recording lots of simultaneous streams myself, but since I've just upgraded my server maybe I'll see if I can find a mux with plenty of services to test.
Adam
Updated by Henk Schoneveld about 12 years ago
Installed 3.2.18. 7 simultaneous ts pass through recordings looks very stable and the ts files are playable. mkv has troubles as in git. So it looks I've a workable situaution, at least with 1 adapter. Will try adding another later today.
Updated by Adam Sutton about 12 years ago
- Category set to Muxers
- Assignee set to John Törnblom
OK, couldn't re-produce this one on my 3.2 installation, so I'm going to pass to John to take a look as if there are problems in the MKV code he will most probably already know about them.
Adam
Updated by Henk Schoneveld about 12 years ago
Hi Adam,
Been playing around last few days, I don't see any problems as long as source material is SD, at least not with the 3.2.18 version I'm running atm.
Updated by Adam Sutton about 12 years ago
Hmmm, ok, I don't have any way to test with HD. I have a total of 5 HD channels and the most I have on 1 mux is 2, which means the most I can receive at the same time is 3
I regularly record from the 2 (on the same mux) in MKV format, but not seen any issues. But as you mentioned might be that it gets worse with time? Or there might be a q'ing problem.
I'll leave it to John to take a look.
Adam
Updated by Henk Schoneveld about 12 years ago
Hi Adam,
Something maybe related to the case. I just happened to have discontinuity errors and dmesg tells me:
saa7146 (0) vpeirq: used 1 times >80% of buffer (168824 bytes now)
saa7146 (0) vpeirq: used 5 times >80% of buffer (65424 bytes now)
But no growing memory usage :-)
Henk
Updated by Adam Sutton about 12 years ago
That could be important, not sure.
John, when you get a chance can you have a look?
Adam
Updated by John Törnblom about 12 years ago
I'll see what I can do. I had memory leaks myself sometime last week but I haven't seen them since my last sync with master. Perhaps they are related to the memory fixes commited by andoma the past few week?
Updated by Henk Schoneveld about 12 years ago
the mentioned vpeirq problem can be solved by creating /etc/modprobe.d/budget_core with
options budget_core bufsize=1410
Hope it helps someone.
Updated by Henk Schoneveld about 12 years ago
New insight of possible cause. The 'irq' problems were Asus SandyBridge board related, so I bought a GigaByte board to solve this. It didn't. I programmed 64 programmes from 1 mux, 2/3 failed, from start or somewhere halfways. But what I observed then that 1 station, didn't have any fault all were OK. That was the only FTA channel on that mux. Finally I grabbed an old PC, 6 years old, stuffed it with 2 PCI cards and started recording a mux with only FTA channels. In a few hours I recorded 63GB, all without any failing.
So that makes me think it has to be something related to the way the Code Word Client part of tvheadend works. Any idea how I could diagnose this further. BTW the memory leak isn't observible when recording only FTA channels.
Updated by Adam Sutton about 12 years ago
Henk,
When did you last update? I know manio has fixed some stuff, but I don't think that affected cwc stuff only capmt. But the descrambling code is a mystery to me.
I don't use anything but FTA which is probably why I've not seen this myself.
Adam
Updated by Henk Schoneveld about 12 years ago
Haven't updated yet, will do so in an hour. Started recording of Pay TV/scrambled channels, after a recording starts one sees cwc Obtained key ... after 1h30m these messages aren't in the logs anymore. After another hour new recordings can't be opened, NO VIDEO NO AUDIO stream detected. Earlier started recordings stop playing when that timepoint arives and mplayer segfaults.
There are no errors/warnings in tvheadend or the system, except for the segfaults of mplayer.
Oscam.log tells it's still delivering keys for decoding after the problems arrive.
Updated by Henk Schoneveld about 12 years ago
Compiled latest git, behaviour is the same. Before starting a descrambled recording I had to tune to a FTA channel, watch it to see the recording is what it should be, after that the recording of pay-tv channels is OK.
By starting the 1st record I get "cwc: Obtained key", in total I do get 6 of them, the last one 2 hours later. After another about 45 minutes all recordings fail. Oscam happily keeps providing keys which tvheadend seems to ask for but doesn't integrate them in the stream to make them watchable.
Updated by Henk Schoneveld about 12 years ago
FWIW tvheadend version 3913532, a modified version by EricV, the cwc functions and keeps functioning as it should, at least for 12hrs on SD channels.
Updated by Henk Schoneveld about 12 years ago
Discovered another major problem, not the cause of tvheadend ;-), but a kernel problem. I often saw major IO wait situations where kswapd0 used one kernel 99,99%, some of my mentioned problems may be related to that. Google kswapd0 and cpu usage and you'll find a lot of problems specially when moving relatively a lot or greater files. Parallel recording of 6 streams, with sometimes overlapping recordings, which result in even more parallel streams, the kswapd0 problem was easily triggered. Compiled a lot of older kernels and found that stock kernels until 2.6.33 don't seem to be 'infected' while a distribution kernel 2.6.33.7-2 does have those, even tried until 3.7-rc6, all of them have it.
Situation now.
GIT of today delivers unreadable pay-tv channels. FTA is ok
3.2.18 delivers unreadable pay-tv channels every now and then. FTA is ok
EricV versions 3913532 and ca11b8b record the pay-tv channels and FTA ok but suffers from memory leaks when too many parallel streams are recorded.
So I would very much welcome a GIT version which knows how to handle pay-tv channels;-)
Henk
Updated by Henk Schoneveld almost 12 years ago
Finally got rid of all the 'system errors'. No more IOwait etc. Recording from FTA is perfect. Recording 5 channels and simultaneous transcoding goes perfect.
However choosing PayTV channels and using cwc with oscam server is VERY unreliable, unreadable files and/or Discontinuity errors with memory leaks ultimately resulting in a crash .
Updated by Adam Sutton almost 12 years ago
- Category changed from Muxers to Descrambling
- Assignee deleted (
John Törnblom)
As I've said before I'm not very well versed on the descrambling code, but I know there are 2 ways in which you can interact with oscam, i.e. cwc or capmt. Manio has been working on the capmt code recently, where as cwc is mostly untouched for a long time.
Is the use of capmt something you can try? Again I have no idea what's involved as I've never had call to use such stuff myself.
Adam
Updated by Adam Sutton almost 12 years ago
Can we get an update on whether or not this is still an issue? Else I'll close it.
Ta
Adam
Updated by Henk Schoneveld almost 12 years ago
No the problem is still there, nothing changed. Still have to investigate capmt, but AFAI can see capmt is on top of cwc, so I didn't/don't see as a viable solution, but I'll try coming days.
Updated by Adam Sutton almost 12 years ago
Henk,
I've been trying to read back over the notes and looks like there are several issues that we discussed some of which may have been resolved/worked around etc...
So could you just give a summary of what teh current problem(s) is.
Ta
Updated by Henk Schoneveld almost 12 years ago
Adam,
Sorry for being so late with responding. The current problem is that reliable concurrent recording pay-tv channels isn't possible. After a while the decoding stops.
Henk
Updated by Henk Schoneveld almost 12 years ago
Recording pay tv channels from the same mux, debug enabled gives:
Feb 08 12:38:36 cwc: Received ECM reply for service "RTL7 HD" even: aa.15.69.28.5e.4b.25.ce odd: 0d.9f.20.cc.de.cb.e3.8c (seqno: 55 Req delay: 339 ms)
Feb 08 12:38:36 cwc: Received ECM reply for service "NET5 HD" even: a2.90.d1.03.e7.7e.6d.d2 odd: d4.1d.02.f3.17.ac.75.38 (seqno: 56 Req delay: 684 ms)
Feb 08 12:38:37 cwc: Received ECM reply for service "een HD" even: e2.af.45.d6.28.a7.70.3f odd: 57.13.06.70.20.06.66.8c (seqno: 57 Req delay: 1021 ms)
Every additional channel on the same mux Req delay: increases with about 340ms.
Additional recording on another mux, on another adapter, the 1st one starts with the same start delay as the 1st recording ~ 340ms
As long as delay is smaller then a certain value everything is Ok. If the threshold is reached all recordings from all channels from all muxes go bad.
Hope it helps.
Henk
Updated by Henk Schoneveld almost 12 years ago
Hi Adam,
After long searching and experimenting it appears that the used paytv service card from CanalDigitaal has a limit of serving max 4 channels. Nothing wrong with tvheadend or oscam software. The above mentioned threshold of delay below a certain threshold is correct but not the cause of failing. Added another pc, if the 1st one records 4 channels and 1 start recording the next on the 2nd pc, after a short while all recordings fail.
Sorry for bothering you and wasting your time.
I'm very grateful for all the help I received.
Henk Schoneveld
Updated by Adam Sutton over 11 years ago
- Status changed from Need feedback to Invalid
Closing this as I think the last message indicated this was outside of TVH.
Adam