Tvheadend

@Em Smith, can you please provide a finished xml (epg) file, I would like to compare it with webgrabplus and maybe i can implement it there so that regexp recognizes it, thanks

Sure.

There's nothing special about my epg, it just puts "title - season/episode - subtitle" on one line in the description.

So an example in the client looks like this:

NCIS - S07E19 - Guilty Pleasure.

S7/15 E19/24. (2010-04-06). (TV-14). When the team members link the death of a Navy officer to a call girl, they are forced to ask Holly Snow for help to find the culprit.

Cast: Mark Harmon, Michael Weatherly, David McCallum, Pauley Perrette, Sean Murray, Cote de Pablo, Rocky Carroll, Dina Meyer, Adam Kaufman, Jason London, Taylor Cole, Jillian Bach, Bruce Nozick, Hayley Chase, Cliff Weissman, Kallee Brookes, Chris Sheffield, Jeff Wolfe.

Categories: Action, Adventure, Crime drama, Episode, Mystery, Series, Show.

The description is from this subset of the file at the end of the post (my whole file is too big to include).

So, each "programme" has a "desc" tag which contains the description. I include some extra information in the description since most skins don't show the information I want to see (such as age rating).

The season/episode number is taken from the "episode-num" tag, but be careful since there are different formats for episode-num so some (such as xmltv_ns) has "6 / 15" to mean "S7/15" and "18 / 24" to mean "E19/24".

 <programme channel="I52335.json.schedulesdirect.org" start="20180809190000 +0000" stop="20180809200000 +0000">
  <title>NCIS</title>
  <sub-title>Guilty Pleasure</sub-title>
  <desc lang="en-GB">NCIS - S07E19 - Guilty Pleasure.

S7/15 E19/24. (2010-04-06). (TV-14). When the team members link the death of a Navy officer to a call girl, they are forced to ask Holly Snow for help to find the culprit.

Cast: Mark Harmon, Michael Weatherly, David McCallum, Pauley Perrette, Sean Murray, Cote de Pablo, Rocky Carroll, Dina Meyer, Adam Kaufman, Jason London, Taylor Cole, Jillian Bach, Bruce Nozick, Hayley Chase, Cliff Weissman, Kallee Brookes, Chris Sheffield, Jeff Wolfe.

Categories: Action, Adventure, Crime drama, Episode, Mystery, Series, Show.
&lt;img src="https://s3.amazonaws.com/schedulesdirect/assets/p11857845_b_h13_ab.jpg" width="120"/&gt;
&lt;img src="https://json.schedulesdirect.org/20141201/image/assets/p8059916_e_h9_ab.jpg" width="120"/&gt;</desc><credits><director>James Whitmore Jr.</director><actor role="NCIS Special Agent Leroy Jethro Gibbs">Mark Harmon</actor><actor role="NCIS Special Agent Anthony &quot;Tony&quot; Dinozzo">Michael Weatherly</actor><actor role="Dr. Donald &quot;Ducky&quot; Mallard">David McCallum</actor><actor role="Abby Sciuto">Pauley Perrette</actor><actor role="NCIS Special Agent Timothy McGee">Sean Murray</actor><actor role="Ziva David">Cote de Pablo</actor><actor role="NCIS Director Leon Vance">Rocky Carroll</actor><writer>Reed Steiner</writer><writer>Christopher J. Waild</writer><producer>Donald P. Bellisario</producer><guest>Dina Meyer</guest><guest>Adam Kaufman</guest><guest>Jason London</guest><guest>Taylor Cole</guest><guest>Jillian Bach</guest><guest>Bruce Nozick</guest><guest>Hayley Chase</guest><guest>Cliff Weissman</guest><guest>Kallee Brookes</guest><guest>Chris Sheffield</guest><guest>Jeff Wolfe</guest></credits>

  <date>2010-04-06</date>
  <category>series</category>
  <category>Crime drama</category>
  <category>Action</category>
  <category>Adventure</category>
  <category>Mystery</category>
  <category>Series</category>
  <category>Episode</category>
  <category>Show</category>
  <category>2000s</category>
  <category>Colleague</category>
  <category>Crime</category>
  <category>Government agents</category>
  <category>Headquarters</category>
  <category>Homicide</category>
  <category>Investigation</category>
  <category>Naval base</category>
  <category>Naval officer</category>
  <category>Navy</category>
  <category>Special agent</category>
  <category>Washington, D.C.</category>
  <length units="seconds">2640</length>
  <icon src="https://s3.amazonaws.com/schedulesdirect/assets/p11857845_b_h13_ab.jpg" width="480" height="270" />
  <episode-num system="dd_progid">EP01260840.0154</episode-num>
  <episode-num system="xmltv_ns"> 6 / 15 . 18 / 24 .  </episode-num>
  <episode-num system="thetvdb.com">series/72108</episode-num>
  <video></video>
  <audio>
   <stereo>stereo</stereo>
  </audio>
  <previously-shown start="20100406 +0000" />
  <rating system="VCHIP">
   <value>TV-14</value>
  </rating>
  <rating system="CHVRS">
   <value>14+</value>
  </rating>
  <rating system="ClassInd">
   <value>16</value>
  </rating>
  <rating system="FPB">
   <value>16</value>
  </rating>
  <rating system="MEKU">
   <value>K16</value>
  </rating>
  <rating system="Régie du cinéma">
   <value>13+</value>
  </rating>
 </programme>

HTH.

Wow - thank you, Em Smith.

This development looks absolutely amazing.

Other than the fact that I was unable to specify which episodes to record, two things about my system remain problematic:

1. A news break, in the middle of a film, would often mean that I would only record the first half of it.
2. I like to watch recordings from "TV Series" in Kodi, rather than from the Recordings folder in the plugin. However, it is not uncommon for the Schedules Direct episode information to differ from that of the TVDB, so I either get informed that the wrong episode is there or (if the episode is not in the TVDB), I won't know it has been recorded at all.

Having just read your posts, your grabber seems to be absolutely perfect for me!

I am still pretty much a beginner, when it comes to Linux, so I still have to be spoon fed instructions, when it comes to these sorts of things.

I should have no problem in adding the new grabber and making it executable.

Em Smith wrote:

1. Backup your current .xmltv configuration from ~/.xmltv (since my version patches the DB to download artwork details);
2. Change the grabber to tv_grab_az_sdjson_sqlite (look for it in "path") in tvheadend's configuration->channel/epg->epg grabber modules, and add extra arguments of "--update-description-with-title" or "--update-description-with-all" (without the quotes) which has more description fixups such as adding year, age rating and stars. I'd also add "--artwork-max-width=720" to avoid slow artwork downloads so it becomes "--update-description-with-all --artwork-max-width=720 --merge-split=5";
3. ???
4. Profit.

Could you possibly clarify why I need to backup my current .xmltv configuration, please? Is this strictly necessary?

What are the next steps, ??? and Profit?

Sorry if I am being really thick with this, I still find programming with Linux a bit of a struggle .

Sorry, "???, Profit" is an old "joke" where if you give a list of things, the last two steps are always that. It comes from bad business plans where you only know the first two steps but you know you'll make a profit.

No need to backup your .xmltv config (actually the SchedulesDirect.DB inside the .xmltv directory), but it just allows you to revert to the "zz" grabber if you don't like the "az" grabber since the "az" makes some changes to the DB which won't be compatible if you later revert.

So (for others) you'd do something like:

cd ~/.xmltv &&                                      (or wherever the .xmltv is on your system)
cp -p SchedulesDirect.DB SchedulesDirect.DB.backup  (miss out the "-p" if your system doesn't have it).

dirname `which tv_grab_zz_sdjson_sqlite`     (Note these are backquotes, maybe it's left of your "1" key)
cd /usr/local/bin     (or wherever the "zz" grabber is)

sudo wget https://raw.githubusercontent.com/azlm8t/tv_grab_zz_sdjson_sqlite/master/tv_grab_az_sdjson_sqlite &&
sudo chmod 755 tv_grab_az_sdjson_sqlite

Then to revert from "az" to "zz":

cd ~/.xmltv &&
mv SchedulesDirect.DB.backup SchedulesDirect.DB

Sorry - I did think it may well have been a joke, but then you never know with Linux commands

This is perfect - I am going to have a go at this, when I get in tonight.

Rather than blindly type stuff into the command line, I try and understand everything I do so I can try and remember/learn if for the future.

May I just check with you, what I think will be going on with the above, please?

1. [code] cd ~/.xmltv && [/code] - change directory to wherever your XMLTV folder is kept.

2. [code] cp -p SchedulesDirect.DB SchedulesDirect.DB.backup [/code] - make a copy of the current database in the same folder, but don't change datestamp.

3. [code] dirname `which tv_grab_zz_sdjson_sqlite` - no idea.

4. [code] cd /usr/local/bin - change directory to bin.

Is that broadly correct?

I have tried looking up dirname with which but cannot seem to find what this command does.

Yes, broadly correct.

The "-p" also preserves permissions (that you can read/write to the file) just in case you have a funny setup where your files are owned by one person but you have someone else run the grabber. Surprisingly common.

The "which" says "where is the program called tv_grab_zz_sdjson_sqlite". So it might print /usr/bin/tv_grab_zz_sdjson_sqlite, or /usr/local, or /storage/.kodi/somewhere. The dirname is just to print out the directory name just to make it easier to read so you know to where you need to change directory to store your executable.

Effectively you would normally just combine the two commands in to one if you knew the file existed:

cd $(dirname `which tv_grab_zz_sdjson_sqlite`)

The command to lookup commands is "man" such as "man dirname" or "man 1 dirname" (since commands are in section 1 of the manual just in case of a bad setup where you get "dirname" from the developer manual in section 3).

And if the "cd ~/.xmltv" fails then it could be that it's on some LibreElec or some place odd. So you could possibly use find to find the directory:

find /storage/.kodi -type d -name .xmltv -print

Oh, and you probably have to restart tvheadend since it only finds grabbers at startup.

Oh, and the "profit" is basically from South Park S2E17.

So a local example could be:

1. Purchase failing DIY chain on the other side of the planet for a high price;
2. Spend a fortune to refit stores to remove all high profit items and sell only low profit margin items for prices far higher than your competitors;
3. ???
4. Profit.
   Allegedly.

So the "???" is where you imagine the board presenting their plan and you wonder what that stage could possibly be and why they couldn't see the plan would fail.

Thanks very much for that - you really know your stuff. I am getting a Linux lesson at the same time!

I followed the prior instructions and I am not sure that it has worked for me - I must have forgotten to do something (or messed something up).

The screenshot shows my current EPG date for Casualty and you can see the grab log below - what do you think I am doing wrong?

Perhaps you forgot to update "extra arguments" on the epg grabber tab to contain "--update-description-with-all --artwork-max-width=720 --merge-split=5" (without the quotes)?

this working on webgrabplus @em smith, and i have the old tvh server 4.2.6 has no PCRE on this..

the trick is, the title, season, episode stand in the description on first line, see Screenshot 1.jpg..

"^Mythbusters - Die Wissensjäger - S13" for only recording S13 but u must make fulltext on, see Screenshot 2.jpg
----------------

to Webgrabplus..

It must be a provider, that provides you with episode information..

on Webgrabplus.config, you should turn on "rex" <postprocess run="y" grab="y">rex</postprocess>
make a fresh rex.config file and add this lines to..

<settings>

<filename>\path\to\yours\rex.guide.xml</filename>

<title>'title'</title>
<desc>'title'{ - 'episode'}{ - 'subtitle'.\n}'description'</desc>
<star-rating></star-rating>

</settings>

special thanks to the webgrabplus support, Blackbear199

Fantastisch, thanks for sharing the webgrab+ config. I'm glad you got it working, and the webgrab config looks easy to expand.

My guess is that "desc" should be:

<desc>'title'{ - 'episode'}{ - 'subtitle'}{.\n}'description'</desc>

Em Smith wrote:

Perhaps you forgot to update "extra arguments" on the epg grabber tab to contain "--update-description-with-all --artwork-max-width=720 --merge-split=5" (without the quotes)?

No - I updated that too, as per the attached.

How odd.

Further quick update.

As per the attached, it seems that the new grabber is in fact working but I am only getting the updated EPG data for new programs added to the database (from 30/8/18 onwards).

Is there any way to remove all the programs from the database and grab from scratch?

That is very strange. I've never had that problem since all Tvheadend data gets overwritten if it is modified (at least in newer versions of Tvheadend). I can't think what would cause that.

Tvheadend stores a copy of its epg inside the tvheadend directory. I'm not sure where it is on your system, but you may be able to find it with this:

find /storage/.kodi ~/.hts -name epgdb.v2 -print

For the grabber there is the "--force-download" option to force it to remove its data. It should only be used once otherwise it puts a strain on the SD servers, but you can add that to the "extra arguments".

One other thing, I noticed you don't have channel icons in the information panel. If you go in to Tvheadend epg grabber (where the schedule is) and tick the "update icon" button and save and re-run grabber. Then, if that doesn't work, then you may also have to go in to the configuration->channel/epg->epg grabber channels (may be called something different on your version since things were renamed), highlight the ones with XMLTV, hit edit, then update "channel update options" to include icons, press the tick box next to it then save and re-run grabber.

Em Smith wrote:

Fantastisch, thanks for sharing the webgrab+ config. I'm glad you got it working, and the webgrab config looks easy to expand.

My guess is that "desc" should be:
[...]
(With .\n outside subtitle). So if your episode does not have a subtitle then you still have a newline before description. Just my guess though since I've never run it and perhaps all your programmes have a subtitle and the original version will always work.

hi guys,just joined to add a comment.

yes you would be correct that is no subtitle was present the \n wouldnt be added.

a more correct way would be..
[code]<desc>'title'{ - 'episode'}{ - 'subtitle'}.\n'description'</desc>[/code]

the way you did it would work also,the purpose of using { and } qouted from the documentation..

text and element-names can be linked together by enclosing them by {}. This will ensure that, when the element in it is empty, everything between the {} is ignored.

Em Smith wrote:

That is very strange. I've never had that problem since all Tvheadend data gets overwritten if it is modified (at least in newer versions of Tvheadend). I can't think what would cause that.

Tvheadend stores a copy of its epg inside the tvheadend directory. I'm not sure where it is on your system, but you may be able to find it with this:
[...]
You'd have to shutdown tvheadend completely (not sure how you'd do that, maybe "service tvheadend stop" or "systemctl stop tvheadend" and then use "start" later to restart it), then remove the file ("rm epgdb.v2") or preferably move it out the way with "mv epgdb.v2 epgdb.v2.old". Then restart tvheadend and re-run the grabber.

For the grabber there is the "--force-download" option to force it to remove its data. It should only be used once otherwise it puts a strain on the SD servers, but you can add that to the "extra arguments".

One other thing, I noticed you don't have channel icons in the information panel. If you go in to Tvheadend epg grabber (where the schedule is) and tick the "update icon" button and save and re-run grabber. Then, if that doesn't work, then you may also have to go in to the configuration->channel/epg->epg grabber channels (may be called something different on your version since things were renamed), highlight the ones with XMLTV, hit edit, then update "channel update options" to include icons, press the tick box next to it then save and re-run grabber.

It is odd - I also thought that the existing EPG would have been updated.

I shall try the extra argument, to see if that resets everything.

BTW - thank you for the channel icon advice. I never thought about doing that but it makes everything much more complete.

My hero - I could kiss you (if the "Bigger Brothers" didn't make me sign a form promising I wouldn't).

I removed the original epgdb.v2 and restarted TVH.

I now have a fully updated EPG and can now specify which seasons I want to record.

Thanks again - ever so grateful.

I have another slight issue that I hope you guys might be able to help with.

When I type go to manage my lineups (see the first attachment), I note that I have four sources set.

However, when I have had a look at the configuration file itself (second attachment), it states that only on of the sources is being used.

How can this be? I assumed that when you added an EGP source from Scheduled Direct (via tv_grab_az_sdjson_sqlite --manage-lineups), it was the configuration file that saved the changes.

What happens is that it's the SchedulesDirect server that knows about lineups you've created, even if you no longer have any config files referencing those lineups.

IIRC there's some limit to say you can't keep changing lineups to avoid people having 30 lineups (by constantly add/delete lineup) but only paying for four, so that's why SD server maintains the lineups. The configuration file then just says which lineup and options to use when generating the output.

You need a separate configuration file per lineup, and each configuration file would have a single lineup line.

Easiest way to create the configuration file is to just "cp" the existing file to a new filename and then it looks like you use "nano" to edit the new file and just replace the "lineup=" line with one from your manage-lineups output.

Thanks again Em - another perfect explanation.

Given that I would need three copies of the existing tv_grab_az_sdjson_sqlite.conf file, what should I rename each them, so they each get used for grabbing?

Unfortunately there's no easy way to get the grabber to pick up multiple configuration files/lineup.

Partly I guess it's because many PVRs require a separate xmltv lineup/file per "source", so it's assumed that your xmltv config and channels for "Freeview (DVB-T)" would be completely different than channels for "Freesat (DVB-S)" or "Astra 19.2 (DVB-S)".

For example with one PVR I know you have to feed freeview.xmltv in to freeview channels, freesat.xmltv in to freesat channels, and they are all kept separate. Whereas with Tvheadend, all the channels are lumped together so you can't say "this xmltv file is only for this subset of channels on this source network."

Assuming you want to feed multiple lineup configuration files in to tvheadend, then the approach I take is here:

[[https://tvheadend.org/boards/5/topics/33827?r=33859#message-33859]]

You'd need to replace "*.xml" with "*.conf" in the script. But, basically, you could name the files anything you want as long as they ended in ".conf". So you could name them "freesat.conf", "freeview.conf", "skyhd.conf".

The xmltv grabbers have a "--config-file" option to tell them which file to use.

Just remember that the last (alphabetical) file wins. By that I mean that if Freeview and Freesat has channel Bob, but it's only available 6pm--10pm on Freeview but 24h on Freesat, then you'd probably want a Freesat filename to be last so you get 24 hour listings, and then give higher priority to DVB-S tuners. I don't know if Freeview still has those partial channels, but I know it used to be annoying when half-way through a Freeview programme would cut out as the channel ceased broadcasting.

Thanks for clarifying that - makes perfect sense when you spell out.

Whilst I could combine multiple lineups (as you have cleverly done), I think I am going to stick with just one source (at least for the time-being).

I am slightly worried that I going to start getting out of my depth with my setup.

The reason I had multiple lineups was that Schedules Direct does not appear to provide EPG data for all of its lineup channels.

At the moment, I am using the Sky London HD bouquet with the matching SD lineup but noticed that there were lots of channels for which there was no guide information.

I was rather hoping that adding additional lineup channels would help to get round this but a bit of testing has found this not to be the case.

For the time-being, I have had to use the Sky OpenTV grabber to fill in the gaps, which is a bit of a pain - I really wanted to have all my channel data to be from the same source so it would all match up.

As a general rule, it does not matter that much as most of the missing data is for channels that I am unlikely to watch.

However, a particular sticking point is Quest HD. Perhaps, as it is new, it is going to take a little while for SD to add it to its various lineups.

Yes, SD is a pain. When I contacted them, they hinted that they don't get much feedback from GB. In theory, the 'ZZZZ 28.2E' lineup should have all the channels, but it doesn't. You can always raise lineup issue tickets with them via their website.

You should get Quest HD from your 100017 Freesat lineup or Freeview lineup. And use Freesat EIT to get the extended listings on remaining channels.

I have 20 or more channels without listings, but they are mostly fringe channels.

If you are running on a beefy box with lots of memory, you can run tv_grab_combiner which is fairly easy to configure and runs multiple grabbers (such as tv_grab_az_sdjson_sqlite with different config files) and combines the results in to one file. That avoids having to run shell scripts/crontab, but, IME, it would hammer my machine.

I had a bit of spare time so updated my tv_grab_az_sdjson_sqlite on github to support multiple lineups in one config file. Seems to work for me. Guess I'll know for sure in a month or two.

This means you can have one config file and the output xmltv will be a merged Freeview/Freesat/Sky.

I've not added UI support for it (--manage-lineups, etc.) so you have to create a new config file manually (and use the old one if you need to manage lineups on SD server).

So, based on your jpgs, you'd create a new .conf file similar to:

database=/home/hts/.xmltv/SchedulesDirect.DB
lineup=GBR-1000197-DEFAULT
lineup=GBR-1000015-DEFAULT
lineup=GBR-1000017-DEFAULT
lineup=GBR-1000051-DEFAULT
mythtv-categories=disabled

I made a few other intrusive changes to try and speed it up. These may cause some problems in edge cases that I've not experienced yet. But it also now can cache xmltv output segments to improve runtime since programmes rarely change between runs so getting from a Redis or File cache is faster that regenerating, but that's probably a bit niche setup, and is a bit experimental at this moment. But it means the output xmltv file (with cache configuration from the README) only contains changed programmes rather than every programme on every channel for X days. Needs a few more weeks of testing though to ensure all works well as programmes expire.

FWIW I seem to have issues only with the 28.2E lineup (many channels SD says are in the lineup then produce an SD error saying the channel isn't in any of my lineups, so I added code to log the error so it's not invisible any more). Not sure of the cause yet, so caveat emptor and always backup first!

please, can you kindly explain me why
"^Autosalon" works (in title)
"test" works (in description)
but
"^Autosalon.*test.*" does not work?

What am I doing wrong?

thx