Tvheadend

First of all thank you very much for your prompt reaction. Very much appreciated. Did a fresh install, new git clone, I don't see the mentioned option in adapter configuration. Do I have another version ?
Now it worked Ok for 10 minutes then the continuity errors apear and the memory consumption starts. With 7 recordings with 10MB per second.
I also get a segmentation fault when memory consumption is about 1.2GB of 4GB available,
Fault address 0x1374e8c (address not mapped)

Henk Schoneveld

Hmm, Andreas has been plugging a few memory leaks last few days but I don't think he has anything that directly relates to this issue.

On the adapter config tab there should be a whole load of options (in the middle box), there should be something in there about full mux RX ('Disable full MUX reception'). If you can't see that option something is very wrong or you are not building from master.

Maybe include a full debug log just for clarity.

Adam

Hi Adam,

I think I found the reason for the memory-eating. Played with 2.99 and did see the same behaviour. After stopping the last recording and restarting everything went fine with 5 concurrend recordings.
Tried again with current git, know what, also runs fine. It looks like somewehere 5 is defined as max concurrend streams from a given mux, I'm not a programmer, but (address not mapped) I could associate with that. Don't laugh please ;-)

There is nothing that limits the number of streams from a given mux, I know several users that stream more than that (IPTV services etc..) using TVH.

However that doesn't mean that there isn't an issue in the DVR code that causes problems above some certain threshold, though not sure why it would.

I'll try and have a look to see if I can repeat when I get a chance.

Adam

Thanks for your answers. Also added another adapter, but if total recordings exceeds 5 I get in the same boat again. Parser errors, resulting in "blockines" video.
Nov 07 19:29:00 [ERROR]:parser: transport stream H264, DTS discontinuity. DTS = 1672200, last = 793800
Nov 07 19:29:00 [ERROR]:parser: transport stream MPEG2AUDIO, DTS discontinuity. DTS = 1646692, last = 724372
Nov 07 19:29:10 [ERROR]:parser: transport stream H264, DTS discontinuity. DTS = 1773000, last = 869400
Nov 07 19:29:11 [ERROR]:parser: transport stream MPEG2AUDIO, DTS discontinuity. DTS = 1696378, last = 787018

Those IPTV services, how many channels do they deliver per tvheadend instance ?
I can imagine 1 machine 5 adapters with 5 channels to x users would work without any problem. If done otherwise I would like to know how they do it.

The particular system I'm thinking of has several setups. One has a single quad tuner card running approx 40 services (constantly streamed to another machine - transcoder). But also has a setup with 4 quad tuner cards, not sure how many services (I don't think its many more services, just greater number of muxes covered).

Now that system does experience some issues, probably because that really is pushing the boundaries but we are trying to fix them.

I can certainly imagine that the full mux RX might add a slight increase in load, so its worth trying to disable it and seeing what happens.

And maybe attach a more complete debug log, just so we can see what's going on.

Adam

As mentioned my git version doesn't have that option. System load is 5%, so that isn't the problem either. I tried TS (pass through) but that results in .bin files where mplayer and mediainfo cannot find any video or audio in.
Looks like I'm having a something different version from git then wat you are expecting I should get. Tomorow I'll try different instances of tvheadend each with it's own adapter and see what happens.

Henk

If you're not seeing the disable full mux option and getting .bin files for passthru then you need to update your git tree.

Adam

I've done that in the weekend and yesterday as well. The used command:
git clone https://github.com/tvheadend/tvheadend.git

Do i have to specify anything else ?

Henk

Can you double check the git revision being reported by TVH on startup?

It's just the .bin issue should have been fixed, and there should definitely be a disable full mux option on the adapter tab.

Adam

Did a new git clone, version is HTS Tvheadend 3.3.133~gddad1d2
There isn't mux RX ('Disable full MUX reception') in configuration of the card in the midle. Further testing follows

Things arent' getting different. Which logs would you like to receive ? I see I can set log to console, but how do I capture that to a file ?

Henk

What I thin to see that I have 5 active subscriptions, with unrealistic low bandwith and only 1 file is written. I almost looks like the NOT written files endup in memory.

Henk

There may be an element of truth to that last statement. We did recently add some buffer limiting in some aspects of the code that had previously been missing. However nothing was added to the DVR chain as it was assumed that you should be able to write to file at greater than the incoming data rate else stuff would just start to fall apart.

So indeed if your disks simply cannot keep up with the number of things you are trying to record it is indeed possible that it could be buffering it all (unlimited) to memory.

I could possibly create a patch that would limit the buffering to at least stop the memory leak, but in reality that won't solve much (long term) since you will simply get corrupted recordings which will probably not be off much use.

If you start TVH manually ./build.linux/tvheadend, then add -d (debug to console) and pipe stderr to file, i.e:

./build.linux/tvheadend -d 2> tvh.log

Adam

I think i know when things go wrong. If ts pass through is chosen there is no memory eating. Only the resulting file doesn't have audio or video according to mplayer and mediainfo. Disk I/O isn't the problem as you were assuming. When setting recording to mkv there almost isn't any disk io. When choosing ts pass through and there then is normal bandwith use according to "active subscriptions" then according to iostat there isn't a I/O problem.
Linux 3.3.6-desktop-2.mga2 (i5) 11/08/2012 i686 (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
2.73 0.00 0.88 2.22 0.00 94.17

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 3.46 76.79 29.35 286344 109446
sdb 21.32 1.61 2446.91 5993 9124160

As you described your "particular system" you probably are streaming in ts-format to the encoding machines and that will work Ok
When I start to see follwing in the log
08 13:53:44 [ERROR]:parser: transport stream H264, DTS discontinuity. DTS = 5776200, last = 4807800
Nov 08 13:53:44 [ERROR]:parser: transport stream MPEG2AUDIO, DTS discontinuity. DTS = 5741338, last = 4728298
I do observe the dropping in bandwith in active subscriptions to values of 250 - 350k where it should be between 5000 and 20.000k.
In my opinion there are 2 things that have to be investigated, probably the mentioned buffering and ts pass through should result in playable ts files.

Hmmm interesting,

I will definitely try this out. I can say that originally the setup I described was using MKV, however since there were general concerns about loading some of it was switched to TS (however that pre-dated John's fixes and so performance was actually worse at the time). No idea whether the guy is using MKV or TS at the moment, probably MKV (which reminds me I need to check).

I think the two issues you mention are somehow linked, though I don't know how yet. Clearly others are able to stream far higher loads than you are currently having problems with which does suggest there is something specific about your setup which is causing problems.

That's not to say those problems aren't bugs in TVH, they may well be, just might make it a little tougher to track down.

I still haven't had a go and recording lots of simultaneous streams myself, but since I've just upgraded my server maybe I'll see if I can find a mux with plenty of services to test.

Adam

Installed 3.2.18. 7 simultaneous ts pass through recordings looks very stable and the ts files are playable. mkv has troubles as in git. So it looks I've a workable situaution, at least with 1 adapter. Will try adding another later today.

Hi Adam,

Been playing around last few days, I don't see any problems as long as source material is SD, at least not with the 3.2.18 version I'm running atm.

Hmmm, ok, I don't have any way to test with HD. I have a total of 5 HD channels and the most I have on 1 mux is 2, which means the most I can receive at the same time is 3 :(

I regularly record from the 2 (on the same mux) in MKV format, but not seen any issues. But as you mentioned might be that it gets worse with time? Or there might be a q'ing problem.

I'll leave it to John to take a look.

Adam

Hi Adam,

Something maybe related to the case. I just happened to have discontinuity errors and dmesg tells me:
saa7146 (0) vpeirq: used 1 times >80% of buffer (168824 bytes now)
saa7146 (0) vpeirq: used 5 times >80% of buffer (65424 bytes now)
But no growing memory usage :-)
Henk

That could be important, not sure.

John, when you get a chance can you have a look?

Adam

I'll see what I can do. I had memory leaks myself sometime last week but I haven't seen them since my last sync with master. Perhaps they are related to the memory fixes commited by andoma the past few week?

the mentioned vpeirq problem can be solved by creating /etc/modprobe.d/budget_core with
options budget_core bufsize=1410
Hope it helps someone.

New insight of possible cause. The 'irq' problems were Asus SandyBridge board related, so I bought a GigaByte board to solve this. It didn't. I programmed 64 programmes from 1 mux, 2/3 failed, from start or somewhere halfways. But what I observed then that 1 station, didn't have any fault all were OK. That was the only FTA channel on that mux. Finally I grabbed an old PC, 6 years old, stuffed it with 2 PCI cards and started recording a mux with only FTA channels. In a few hours I recorded 63GB, all without any failing.
So that makes me think it has to be something related to the way the Code Word Client part of tvheadend works. Any idea how I could diagnose this further. BTW the memory leak isn't observible when recording only FTA channels.

Henk,

When did you last update? I know manio has fixed some stuff, but I don't think that affected cwc stuff only capmt. But the descrambling code is a mystery to me.

I don't use anything but FTA which is probably why I've not seen this myself.

Adam

Haven't updated yet, will do so in an hour. Started recording of Pay TV/scrambled channels, after a recording starts one sees cwc Obtained key ... after 1h30m these messages aren't in the logs anymore. After another hour new recordings can't be opened, NO VIDEO NO AUDIO stream detected. Earlier started recordings stop playing when that timepoint arives and mplayer segfaults.
There are no errors/warnings in tvheadend or the system, except for the segfaults of mplayer.
Oscam.log tells it's still delivering keys for decoding after the problems arrive.

Compiled latest git, behaviour is the same. Before starting a descrambled recording I had to tune to a FTA channel, watch it to see the recording is what it should be, after that the recording of pay-tv channels is OK.
By starting the 1st record I get "cwc: Obtained key", in total I do get 6 of them, the last one 2 hours later. After another about 45 minutes all recordings fail. Oscam happily keeps providing keys which tvheadend seems to ask for but doesn't integrate them in the stream to make them watchable.

FWIW tvheadend version 3913532, a modified version by EricV, the cwc functions and keeps functioning as it should, at least for 12hrs on SD channels.

Discovered another major problem, not the cause of tvheadend ;-), but a kernel problem. I often saw major IO wait situations where kswapd0 used one kernel 99,99%, some of my mentioned problems may be related to that. Google kswapd0 and cpu usage and you'll find a lot of problems specially when moving relatively a lot or greater files. Parallel recording of 6 streams, with sometimes overlapping recordings, which result in even more parallel streams, the kswapd0 problem was easily triggered. Compiled a lot of older kernels and found that stock kernels until 2.6.33 don't seem to be 'infected' while a distribution kernel 2.6.33.7-2 does have those, even tried until 3.7-rc6, all of them have it.
Situation now.
GIT of today delivers unreadable pay-tv channels. FTA is ok
3.2.18 delivers unreadable pay-tv channels every now and then. FTA is ok
EricV versions 3913532 and ca11b8b record the pay-tv channels and FTA ok but suffers from memory leaks when too many parallel streams are recorded.

So I would very much welcome a GIT version which knows how to handle pay-tv channels;-)

Henk

Finally got rid of all the 'system errors'. No more IOwait etc. Recording from FTA is perfect. Recording 5 channels and simultaneous transcoding goes perfect.
However choosing PayTV channels and using cwc with oscam server is VERY unreliable, unreadable files and/or Discontinuity errors with memory leaks ultimately resulting in a crash .

Can we get an update on whether or not this is still an issue? Else I'll close it.

Ta
Adam

No the problem is still there, nothing changed. Still have to investigate capmt, but AFAI can see capmt is on top of cwc, so I didn't/don't see as a viable solution, but I'll try coming days.

Henk,

I've been trying to read back over the notes and looks like there are several issues that we discussed some of which may have been resolved/worked around etc...

So could you just give a summary of what teh current problem(s) is.

Ta

Adam,

Sorry for being so late with responding. The current problem is that reliable concurrent recording pay-tv channels isn't possible. After a while the decoding stops.

Henk

Recording pay tv channels from the same mux, debug enabled gives:
Feb 08 12:38:36 cwc: Received ECM reply for service "RTL7 HD" even: aa.15.69.28.5e.4b.25.ce odd: 0d.9f.20.cc.de.cb.e3.8c (seqno: 55 Req delay: 339 ms)

Feb 08 12:38:36 cwc: Received ECM reply for service "NET5 HD" even: a2.90.d1.03.e7.7e.6d.d2 odd: d4.1d.02.f3.17.ac.75.38 (seqno: 56 Req delay: 684 ms)

Feb 08 12:38:37 cwc: Received ECM reply for service "een HD" even: e2.af.45.d6.28.a7.70.3f odd: 57.13.06.70.20.06.66.8c (seqno: 57 Req delay: 1021 ms)

Every additional channel on the same mux Req delay: increases with about 340ms.
Additional recording on another mux, on another adapter, the 1st one starts with the same start delay as the 1st recording ~ 340ms

As long as delay is smaller then a certain value everything is Ok. If the threshold is reached all recordings from all channels from all muxes go bad.

Hope it helps.

Henk

Hi Adam,

After long searching and experimenting it appears that the used paytv service card from CanalDigitaal has a limit of serving max 4 channels. Nothing wrong with tvheadend or oscam software. The above mentioned threshold of delay below a certain threshold is correct but not the cause of failing. Added another pc, if the 1st one records 4 channels and 1 start recording the next on the 2nd pc, after a short while all recordings fail.
Sorry for bothering you and wasting your time.
I'm very grateful for all the help I received.

Henk Schoneveld