Tvheadend

Nobody knows anything?

OK, I've done some digging and I think I see what is going wrong. I believe the code is wrong and the documentation is wrong/incomplete.

I think my problem is caused by the dreaded & invisible serieslink.

Specifically this code in dvr_autorec.c:

/* Do not check title if the event is from the serieslink group */
  if(dae->dae_serieslink == NULL &&
     dae->dae_title != NULL && dae->dae_title[0] != '\0') {

IIUC this is saying that the contents of the title regexp are completely ignored if the first event happened to have a serieslink.

This seems ass-backwards to me. If I want to ignore the title regexp I can make it blank. Or there could be a toggle-button in the interface to ignore it or not, or something (anything!).

What makes this IMHO coding error insulting as well as injurious is that the interface doesn't make the presence of a serieslink visible in any way, and that the documentation is totally silent about this unwanted side-effect of their presence.

Now I'm aware that I'm running 4.2.6 and that is not the latest version of the program, so let me ask - is the latest version any better and if so how?

And I've been looking at the online docs, so they're just plain wrong. I'll happily submit a patch if we can agree what needs to be done.

I have never tried this buy you could try a preprocessing script?

If title = the strain and season = 1, abort recording.\

Also try this regex ^New: The Strain$

Vik Toras wrote:

I have never tried this buy you could try a preprocessing script?

If title = the strain and season = 1, abort recording.\

Where would it get the season from? I use UK EIT data.

Also try this regex ^New: The Strain$

Yes, I tried that regex with no change.

If the season is not there you cant get it... If its there hit the help button for variables available for your script.

On the regex... try escaping the ':' like so '^New\: The Strain$'

Also, you can try deleting the autorec. Instead of recreating it through your EPG guide, create it manually from scratch by hitting the add button in autorec tab. Try the regex then.

On the regex... try escaping the ':' like so '^New\: The Strain$'

Thanks, I just tried that. It still recognizes just 'The Strain'

Also, you can try deleting the autorec. Instead of recreating it through your EPG guide, create it manually from scratch by hitting the add button in autorec tab. Try the regex then.

Yes, I've deleted the original autorecs a while ago. These are ones I've typed in by hand.

I can't explain it. I understood originally that it generated a hidden series link from the first match and used that in preference and to the exclusion of the regex for all subsequent matches. But I deleted those autorecs and typed in some new ones that include the word New and the only programs being broadcast do not include that word, but they're still matching somehow.

The only way I've found to stop the false matches is to disable the autorecs, which kind of defeats the purpose of having them!

I just looked, there is no "New: The Strain", only repeats.

(^new\:\s)?the strain

should find new and old

(^new\:\s)the strain

should just find new

I just looked, there is no "New: The Strain", only repeats.

Yes, that's the whole point. Why are they being matched by autorecs with regex such as ^New: The Strain$ or ^New\: The Strain$.

What does (^new\:\s)the strain add? Will that make any difference?

OK I dug into this a bit more and tested a bit more (with tipping point).

These should do what you want, ugly but it works for tipping point.

^(?!The\sStrain)^New:\sThe\sStrain

^(?!Tipping\sPoint)^New:\sTipping\sPoint

HTH

Let's take a step backwards.

Try your regex on the EPG grid first (not autorec rule) since it's easier to see changes.

If your text is "^New: " then it can not match something without that text (unless a bug, which I'd expect would affect lots of people).

Is the autorec rule definitely the one that is creating the recording? Or could it be that some other rule is matching? Can you include a picture of your rule?

Did your autorec rule have "full-text" box ticked?

The series link usually wouldn't have linked "new series 1" to "old repeat series 1", though it's broadcaster-dependent. The "ignore title" is probably correct since I recall "The Last Leg" once renamed a final episode to something else, but then had to rename back because it broke many series-link implementation which did check title.

Do you only have DVB-T, or do you have DVB-S too? Merged channels across different aerials/sources? In the past, I have seen occasions where the OTA guide differs so DVB-T has "new:" but DVB-S does not have "new:" (or vice-versa), though that seems unlikely in your case since you say these are repeats rather than new episodes.

In 4.3 (alpha/dev branch), we added "record show if unique", "extract season/episode from OTA description", and "only record season > X". It sounds like Paramount doesn't provide season/episode information, so these features probably won't benefit you for that channel. But if you decided to use an xmltv provider like SchedulesDirect (which does support most UK channels) then this provides good consistent guide data with season/episode details and fanart and so would definitely identify and record the right episodes.

Let's take a step backwards.

Sounds a good idea to me.

Try your regex on the EPG grid first (not autorec rule) since it's easier to see changes.

If your text is "^New: " then it can not match something without that text (unless a bug, which I'd expect would affect lots of people).

That appears to work as I would expect

Is the autorec rule definitely the one that is creating the recording? Or could it be that some other rule is matching? Can you include a picture of your rule?

I believe it was. Originally the rule was 'New: The Strain' (without the quotes) and it started recording the repeats, which do not have New in the title. Since then I have tried many variations, including deleting the original autorec which was created from the button in the EPG entry and creating new ones by typing using Add or Edit in the autorec tab.

Did your autorec rule have "full-text" box ticked?

No

The series link usually wouldn't have linked "new series 1" to "old repeat series 1", though it's broadcaster-dependent. The "ignore title" is probably correct since I recall "The Last Leg" once renamed a final episode to something else, but then had to rename back because it broke many series-link implementation which did check title.

That seems like a good argument for a user interface feature to disable/enable series link checking and title checking. Are things any different in the current software? Can I see/edit series link matches?

Do you only have DVB-T, or do you have DVB-S too? Merged channels across different aerials/sources? In the past, I have seen occasions where the OTA guide differs so DVB-T has "new:" but DVB-S does not have "new:" (or vice-versa), though that seems unlikely in your case since you say these are repeats rather than new episodes.

I just have Freeview, no satellite or other sources. Some channels are merged SD and HD but not Paramount where these problems have occurred.

In 4.3 (alpha/dev branch), we added "record show if unique", "extract season/episode from OTA description", and "only record season > X". It sounds like Paramount doesn't provide season/episode information, so these features probably won't benefit you for that channel. But if you decided to use an xmltv provider like SchedulesDirect (which does support most UK channels) then this provides good consistent guide data with season/episode details and fanart and so would definitely identify and record the right episodes.

The broadcast description contains things like '(S1 Ep 6)' at the end for most series including the problematic ones but the EPG never displays any episode information, and a regex can't search both title and description AIUI. I have philosophical objections to subscription services.

Currently my hand-typed regexes are not selecting the repeats. I'll wait and see if that remains true when the next programmes appear in the EPG later in the week. I have no idea what has changed.

Did you try to remake the autorec manually (not from epg tab) like i suggested?

Did you try to remake the autorec manually (not from epg tab) like i suggested?

You mean as in

Since then I have tried many variations, including deleting the original autorec which was created from the button in the EPG entry and creating new ones by typing using Add or Edit in the autorec tab.

Well thats odd. I've had a similar issue where regex would not be enforced if the rule was created from the EPG. There is some stuff you dont see from the Webui such as program ID (Depending on the guide).

You could compare both autorecs. They reside in the drv logs. Maybe that will lead you somewhere.

Alternatively you could try the preprocesssing script.

Not sure if you are open to 4.3, but it does have a 'new' filter, along with a minimum season filter. Need to have better EPG source for that, not something that stuffs additional information in the description.

Since your EPG has '(S1 Ep 6)' then 4.3 should be able to extract that to populate season/episode details (in the epg grabber tab). In 4.3 we added a variety of automatic "EPG regexes" to scrape those details (and "new" flag) from the description/title as long as the tickbox in that tab is ticked.

I don't think you can see episode link info in 4.3, which I agree is a little annoying, but ISTR it's not guaranteed to be readable, though I'd have to re-check the spec.

If you're technically minded, then you could possibly dump your epgdb and see the details. I don't know if the installed package has the tool, or if you need to "git clone", but then you could possibly do:

support/epgdump $HOME/.hts/tvheadend/epgdb.v2 | less

And then find your programmes and look for the "crid" and see if it's the same for "New" and non-New episodes.

You could compare both autorecs. They reside in the drv logs.

Sorry, I don't know what the drv logs are or how to access them.

If you're technically minded, then you could possibly dump your epgdb and see the details.

I'm fairly technical but ...

I don't know if the installed package has the tool, or if you need to "git clone", but then you could possibly do:

support/epgdump $HOME/.hts/tvheadend/epgdb.v2 | less

My installed package doesn't have it. I don't know git, though I could probably figure it out. I don't have a .hts directory. What/where else might whatever it is be?

And then find your programmes and look for the "crid" and see if it's the same for "New" and non-New episodes.

There are no new episodes at present so there won't be any such episodes in EPG data (unless it keeps historical records?)

https://tvheadend.org/projects/tvheadend/wiki/Building

You can probably get away with just doing:

git clone https://github.com/tvheadend/tvheadend.git
cd tvheadend
git checkout origin/release/4.2
support/epgdump $HOME/.hts/tvheadend/epgdb.v2 | less

The .hts directory will be under the user that runs tvheadend. Perhaps it is /home/hts/.hts or /home/tvheadend/.hts?

You might not have read permissions on the file but you can sudo/chmod the file.

There will be a lot in your EPG dump, but you'd eventually expect to see entries like this:

{ u'channel': u'b3cc43fc2aadacdd3273cbaa2fa0e4dd',
  u'dvb_eid': 17910,
  u'episode': u'crid://www.sky.com/G_S36195226',
  u'grabber': u'eit',
  u'id': 722089,
  u'is_subtitled': 1,
  u'serieslink': u'crid://www.sky.com/G_SKYB1574286',
  u'start': 1511996400,
  u'stop': 1511998200,
  u'summary': { u'eng': u"Jim Bowen hosts the dartboard quiz game that tests contestants' darts skills and their general knowledge to win prizes. [S]"},
  u'type': 4,
  u'updated': 1511359708}

I'd expect you'd be able to find a matching entry for "New" and non-New by searching for their description. Then you can see the serieslink details and also see if the details are duplicated in the file but some with "new" and some without "new".

I don't know if it would show anything useful (since you don't have merged DVB-T/DVB-S so unlikely to have conflicting entries), but it might show something unusual when you next have the problem.

Sorry, I forgot to answer your other bits.

The epgdb.v2 is the EPG database that is saved by Tvheadend (assuming you have the tickbox to save EPG ticked in the grabber). It doesn't keep historical data, though it might only be saved occasionally so it could have a few hours of old data.

The epgdump program reads the database and outputs a more readable version to screen. However the database contains many sections so the serieslink section is usually at the end of the file.

The .hts directory will be under the user that runs tvheadend. Perhaps it is /home/hts/.hts or /home/tvheadend/.hts?

tvheadend is run by user 'hts'. According to /etc/passwd its home directory is /var/lib/tvheadend/ But there's no .hts subdirectory in that directory? There is /var/lib/tvheadend/config/epgdb.v2 Could that be it? I don't have permission to access it so using sudo:

$ support/epgdump /var/lib/tvheadend/config/epgdb.v2 
ERROR: epg "/var/lib/tvheadend/config/epgdb.v2" was not found!
$ sudo support/epgdump /var/lib/tvheadend/config/epgdb.v2
__iter__()
{ }
Traceback (most recent call last):
  File "support/epgdump", line 37, in <module>
    for msg in htsmsg.deserialize(fp, True):
  File "support/../lib/py/tvh/htsmsg.py", line 250, in next
    raise Exception('failed to read from fp')
Exception: failed to read from fp
$ sudo ls -l /var/lib/tvheadend/config/epgdb.v2
-rw------- 1 hts video 1996171 Mar 26 15:39 /var/lib/tvheadend/config/epgdb.v2

Given the timestamp on the file, it does seem like it is the correct file.

It could be that the database is compressed since I think the epgdump can't handle that.

Could you untick Config->General->Base EPG-Settings:Compress EPG Database?
Then you need to wait for the file to be re-written which is in Config->EPG->Grabber Periodically save.

Thanks, Em. Uncompressing the database has let me read it and I've found the relevant entries for the repeated episodes. I then did something I'm surprised I haven't done before which is search for CRID on the intertubes. According to wikipedia a CRID is a descriptor for particular content, so I'd expect the CRID for the original broadcasts to be the same as for the reruns. Wikipedia says that something called a locator distinguishes between different broadcasts. So I'm not clear how having a CRID as the value of a serieslink makes sense?

I expect I'm misunderstanding something

Edit: I've been doing a bit more reading of the scurrilous intertubes, after being slightly surprised that the crids of these programmes on Paramount were surprisingly five.tv like

u'episode': u'crid://www.five.tv/V44GNEX'
u'serieslink': u'crid://www.five.tv/R0J8O'

It seems* that channel 5 is somewhat unique in the way it uses CRIDs. A bit more like I would expect from reading the wikipedia article. That is, I would expect all broadcasts of this episode on all channels at all times (or even streamed) to have the same CRIDs.

oops - edit2: It seems as stated on https://hummy.tv/forum/threads/bush-btu500dtr-after-tutv.4580/

What you need is "ETSI TS 102 323" document. I believe Sec 5.2.2 says that CRIDs can be permanent or re-used.

So, BBC appears to have permanent CRIDs (for episodes/movies) that match the iplayer URLs. Any time the programme is on, it has the exact same CRID. I think this is probably the case for most (all?) channels, though I've never investigated. So you should see the same CRID for the same programme on 4, 4HD, 4+1, 47, and 47HD.

The authorities are frequently shared amongst related channels, so Paramount is probably owned/run by the same parent company as Five. Unfortunately, if you have a mixed DVB-T/DVB-S system then you find that they have inconsistent authorities such as five.com and five.tv for the two different sources, which makes duplicate handling a bit more annoying.

I'd expect "series link" CRIDs to be re-used over time. Think of it as a simple hash such as "Wed/09:00", and it links the programmes with the same value.

What you need is "ETSI TS 102 323" document.

Thanks, that looks like excellent sleep-inducing reading

Paramount is probably owned/run by the same parent company as Five

They're both owned by Viacom I believe.

Both my brand new, typed-in autorecs ^New\: Arrow$ and (^new\:\s)the strain have now decided to record future episodes of Arrow and The Strain respectively, without the word New in the title in either case. Or at least, they are the only autorecs still listed that have 'arrow' or 'strain' in them so I'm presuming it is them that are matching.

I really don't understand what is going on. It's very frustrating. I've typed these regex in at a time when there's been nothing broadcast that matches them, so I don't see how they can have matched anything to be able to subsequently match a crid or whatever. Is TVH holding data somewhere that doesn't get deleted when I delete an autorec?

Can you try entering the "^New: Arrow$" in to the EPG window?
(My test system doesn't seem to want to get EPG for Paramount so I can't try it).

For autorecs, the comment field on an autorec can be used to say what created the autorec as this is displayed in the upcoming tab.