Tvheadend

I'll have a test here and see if I get the same issue - will report back, if I forget please bump this thread.

Great - thanks, much appreciated.

Just to say a little more about my configuration in this respect:

• The channel "Use EPG Running State" flag is enabled
• DVR Profile "Use EPG Running State" flag is enabled
• DVR Profile pre and post padding set to 10 minutes

ITV is particularly bad unless you have had a tuner tuned to this channel for some considerable time prior to the recording. In my case, everything is generally idle (no tuners active) prior to a recording starting.

Ok I've setup a recording with debugging for dvr.

Could you do the same?

Yes, sure - will need to do it later though as I don't have access at the moment.

I've done a fair bit of tracing already. I may very well not be right (and I'm happy to be corrected), but I believe I understand what I have seen. At the very least I have made up a good theory that seems to fit my understanding

My understanding of how it should work (and how it seems to work on e.g. the BBC channels):

• Due to the pre-recording padding on my profile tvh will tune to the right channel for the recording 10 minutes before the scheduled start time
• tvh will then monitor for EIT packets referring to the programme (event) I want to record
• tvh will start to record the programme when an EIT packet for the correct event is seen with a running status of 2 (starts in a few seconds) or 4 (running)
• tvh will stop the recording when an EIT packet for the correct event with a running status of 1 (not running) is seen ...... or it might be when a status of 2 or 4 is seen for a different event - haven't looked that deeply into the source

What I see in the traces for channels like BBC is that prior to the programme starting, EIT packets with a running status of 1 (not running) are transmitted for the programme due to start at fairly regular intervals prior to the programme actually starting to air. This, I think, is part of the problem with ITV.

What I have seen in the ITV traces (and some other channels) is that the EIT packets are not transmitted at such regular intervals prior to the recording starting. This is compounded by the fact that the Event ID transmitted in the EIT data by ITV is not the same as the event ID transmitted in the Freesat EPG (as far as I can see anyway).

In normal cirumstances, this wouldn't be an issue as TVH already has code that matches the EIT event to an EPG event using some fuzzy matching and updates the EPG event ID if necessary. It will also update any DVR timers associated with the old (EPG) event ID with the new (EIT) event ID. So, in an ideal world, things happen like this:

1. tvh monitoring for EPG Event ID 1234
2. tvh sees EIT with running status 1 (not running) for event ID 5678
3. tvh uses fuzzy matching to realise that EIT event ID 5678 is the same programme as EPG Event ID 1234
4. tvh updates the EPG event to have ID 5678
5. tvh updates any DVR entries for event ID 1234 to have event ID 5678
6. tvh sees EIT for Event ID 5678 with a running status of 2 or 4
7. tvh matches event ID 5678 to the DVR entry updated earlier and recording is started

The problem is that tvh must see an EIT event with the "wrong" event ID early enough before the programme actually starts to compelte steps 3, 4 and 5 below. If it doesn't, when it sees the EIT event in step 6 it isn't able to match it to a DVR entry so the recording never starts.

ITV create an ideal scenario for this to happen by transmitting the EIT data for the required event less frequently than other channels and also becuase the event ID in the EIT is different to the event ID in the Freesat EPG. This is also why, if you have been watching ITV for a while before the programme starts, things tend to be a lot more reliable (because there has been more of a change to see and process EIT data and to update the event IDs).

I'm hoping to find some time soon to test a change to TVH that forces the DVR entry update (step 5 above) as soon as tvh has realised that the event IDs are the same (and before the running status is evaluated for the purposes of starting the recording). I'm fairly sure that will fix the issue for me, but before doing that, I was hoping that someone else would confirm that there really IS a problem and that I'm not just seeing an issue that doesn't exist because of a problem with my configuration. It could easily be that all the above is an absolute load of twoddle and I'm just making things up to fit what I think I have seen in the traces.

You should be seeing "marking start/stop" messages in the DVR like this.. This is the eitp/f

2017-10-26 15:58:13.071 [  TRACE]:dvr: dvr event running check for New: Tipping Point on ITV HD-SAT running 2
2017-10-26 15:58:13.071 [  DEBUG]:dvr: dvr entry ecddd3f07c9a8e02f0f0fa8a72086149 event New: Tipping Point on ITV HD-SAT - EPG marking start
2017-10-26 15:58:13.080 [  TRACE]:dvr: entry ecddd3f07c9a8e02f0f0fa8a72086149 - set state - state RECORDING rec_state RUNNING error 'OK'
2017-10-26 15:58:13.080 [  TRACE]:dvr: entry ecddd3f07c9a8e02f0f0fa8a72086149 - set state - state RECORDING rec_state WAIT error 'OK'
2017-10-26 15:58:13.081 [  TRACE]:dvr: Creating directory "/md0/tvheadend/dvr/ITV HD-SAT" with octal permissions "750" gid -1 uid -1
2017-10-26 15:58:13.081 [  DEBUG]:dvr: File "/md0/tvheadend/dvr/ITV HD-SAT/New_ Tipping Point2017-10-26.ts" -- No such file or directory -- Using for recording
2017-10-26 15:58:13.081 [   INFO]:dvr: /md0/tvheadend/dvr/ITV HD-SAT/New_ Tipping Point2017-10-26.ts from adapter: "Montage Technology M88DS3103 #2 : DVB-S #0", network: "DVB-S - Astra 28.2°E", mux: "11097V", p$
2017-10-26 15:58:13.081 [   INFO]:dvr:  #  type              lang  resolution  aspect ratio  sample rate  channels
2017-10-26 15:58:13.081 [   INFO]:dvr:  1  H264                    ?           ?
2017-10-26 15:58:13.081 [   INFO]:dvr:  2  MPEG2AUDIO        nar                             96000        ?
2017-10-26 15:58:13.081 [   INFO]:dvr:  3  DVBSUB            eng
2017-10-26 15:58:13.081 [   INFO]:dvr:  4  TELETEXT
2017-10-26 15:58:13.081 [   INFO]:dvr:  5  AC3               eng                             96000        ?
2017-10-26 15:58:13.081 [   INFO]:dvr:  6  TEXTSUB           eng
2017-10-26 15:58:13.150 [  TRACE]:dvr: entry ecddd3f07c9a8e02f0f0fa8a72086149 - set state - state RECORDING rec_state RUNNING error 'OK'
2017-10-26 15:58:13.305 [  TRACE]:dvr: dvr event running check for New: Alphabetical on ITV HD running 0
2017-10-26 15:58:13.305 [  TRACE]:dvr: dvr event running check for New: Tipping Point on ITV HD running 2

This is just with the plain EIT now/next not the freesat data. I'm still waiting for it to populate..

Ahh - okay. I can see why that would work if you're only using the now/next data.

But I think part of the problem is the mis-matched event IDs between the EIT and EPG data.

One last thing I should have said is that most of my scheduled recordings are "series linked" recordings (using the Series CRID). But they are all, without exception, set up from the Freesat EPG.

Adrian Smith wrote:

Ahh - okay. I can see why that would work if you're only using the now/next data.

But I think part of the problem is the mis-matched event IDs between the EIT and EPG data.

Yes, there is some issues. previous tvh versions merged the data but this caused problems so was removed.

I'm hoping to find some time soon to test a change to TVH that forces the DVR entry update (step 5 above) as soon as tvh has realised that the event IDs are the same (and before the running status is evaluated for the purposes of starting the recording).

Tvh really does need more people contributing, so more the merrier. If you want help testing your fix, just open a PR & I'll (and I'm sure others will) give it try.

No problem .... I'll do some reading on how to open a pull request - it'll be a first for me!

I have implemented the fix I had theorised would solve the problem in my own compiled version of tvh and initial indications are positive. I'll do some much more thorough testing over the weekend before patting myself on the back though!

Would still be interested in knowing if you (or anyone else) is experiencing the same thing on ITV when using the Freesat EPG data. I'm still not 100% convinced that I'm solving a problem that is not of my own making somehow. I appreciate that it requires time and effort but having confirmation that it's broken is surely going to be necessary at some point when the fix is considered for inclusion in the code.

Adrian Smith wrote:

No problem .... I'll do some reading on how to open a pull request - it'll be a first for me!

I have implemented the fix I had theorised would solve the problem in my own compiled version of tvh and initial indications are positive. I'll do some much more thorough testing over the weekend before patting myself on the back though!

Would still be interested in knowing if you (or anyone else) is experiencing the same thing on ITV when using the Freesat EPG data.

I'll look into setting up a Pi with my old Technisat I have going spare within the next few days & let you know. Please do bump this thread if you haven't heard anything from me!

I'm still not 100% convinced that I'm solving a problem that is not of my own making somehow. I appreciate that it requires time and effort but having confirmation that it's broken is surely going to be necessary at some point when the fix is considered for inclusion in the code.

Hi Mark,

I tested my fix over the weekend and everything worked fine. 100% of ITV (and other channel) programmes which were scheduled to record from the Freesat EPG were successfully recorded using accurate recording (EIT data). For me the problem experienced previously has gone away.

I have opened a pull request (I think). If I haven't done it correctly for some reason please let me know.

Since you mentioned you have a genuine Freesat recorder, could you try recording the start of a series episode on Horror (like Star Trek)?

I've always found that Tvheadend would miss the first few seconds of episodes I record on Horror. I think the programme signal is normally broadcast every ten seconds and I always wondered whether genuine recorders pre-buffer the recording so when they get the signal a programme has started, they write the buffer from the previous ten seconds, whereas Tvheadend only starts from that exact point in time so misses the start of the programme. Of course, I'd be surprised if Horror were so keen they exactly set "now" on the millisecond the programme changed, but many other channels seem to have a pre-programme ad nowadays that perhaps mitigates the problem on them.

I don't use accurate recording any more ever since I had several occasions when Tvheadend didn't detect the end of programmes and so kept recording for another 15 hours or more until I noticed, and a couple of other times it started recording half-way through.

Nowadays I just bind separate keys on my remote to -30s, +10s, +30s, +4m, +5m which makes it surprisingly easy and fast enough to skip to start and skip over ads.

By way of update before answering the question, the initial issue is now resolved. There was a problem with tvheadend not starting recordings when using EITp/f (accurate recording) against the Freesat EPG.

That said, I set up some recordings on both my "proper" Freesat box and tvh in the last day or so. The "good news" (as far as the initial problem with tvh is concerned) is that the recording start and stop time was identical on both devices. The bad news I'm afraid is that the Horror Channel do seem genuinely awful at broadcasting the EITp/f information at the correct time. My experience on both boxes was that the beginning of programmes were missed because of a late "start of programme" EITp/f being broadcast. So I would say this is down to the broadcaster rather than anything else.

Thanks all for the investigation and fix.

Thanks for testing Horror. That's a pity they don't broadcast it right. It makes me wonder why people with official boxes haven't noticed they don't get complete recordings since I'm guessing configuring EIT p/f isn't something most people will (or even can) mess about with on those boxes.

Em Smith wrote:

Thanks all for the investigation and fix.

Thanks for testing Horror. That's a pity they don't broadcast it right. It makes me wonder why people with official boxes haven't noticed they don't get complete recordings since I'm guessing configuring EIT p/f isn't something most people will (or even can) mess about with on those boxes.

I think most will notice it once & set pre/post-padding & forget about it. I'll be very surprised if boxes ignore custom padding!

Mark Clarkstone wrote:

I think most will notice it once & set pre/post-padding & forget about it. I'll be very surprised if boxes ignore custom padding!

Correct Mark. All Freesat boxes that I know of give the option to use accurate recording or EPG time based recording. If the latter is chosen then there are options for pre and post padding for the recording. I've always used the former because my viewing habits are mostly restricted to the "main five" channels who seem to get the EITp/f correct. I've never had any issue with accurate recording on these channels on my Freesat box.

Thanks for the info. The last accurate recording hardware I had was VHS.

Two more things: could one of you post your grabbers enabled and priorities? I remember there was discussion and fixes a while ago which I believe means the priorities are no longer necessary for EIT p/f to work, but it might be useful to be able to point people to "here's a config that works."

Second, when I watched normal tv I noticed that the 9pm film started at 9.02pm. I deliberately had the info box up and that was when the info box of "now playing" updated. I assume this is basically the tv parsing the EIT p/f. So I can potentially use it to decide on which channels I should enable accurate recordings?

Em Smith wrote:

Thanks for the info. The last accurate recording hardware I had was VHS.

Two more things: could one of you post your grabbers enabled and priorities? I remember there was discussion and fixes a while ago which I believe means the priorities are no longer necessary for EIT p/f to work, but it might be useful to be able to point people to "here's a config that works."

No problem. I am using:

• Over the air: EIT: DVB Grabber - Priority 10
• Over the air: UK: Freesat - Priority 5
• Internal XMLTV: United Kingdom (bleb.org)- Priority 3

Em Smith wrote:

Second, when I watched normal tv I noticed that the 9pm film started at 9.02pm. I deliberately had the info box up and that was when the info box of "now playing" updated. I assume this is basically the tv parsing the EIT p/f. So I can potentially use it to decide on which channels I should enable accurate recordings?

From your description it sounds very much like this is what your particular make/model of TV is showing you in the info box.

Forgot about this!

Mine is:-

EIT: 1
Freesat: 2
OpenTV: 3
Freeview: 4

- Edit - removed : after each number (doing too much Python!)