Tvheadend

Sorry for the delay. The alert hopped into my junk email. I am assuming you are using the tv_grab_file from the code section of the guide. It looks like that may not be the case. A direct link to the file is
https://github.com/rocky4546/script.xmltv.tvheadend/tree/master/usr/bin
It is what I am using and has been working for over a year with 4.0.x and now 4.1.x tvheadend. Once the grabber is present in the "EPG Grabber Modules", it should all work.

Now IF your xmltv.xml file stops generating, then the EPG will slowly disappear because the episodes are from the past. Check the ~/.kodi/temp/zap2it.log to see what the errors are.

As for step 4, you are indicating that the .zap2xmlrc was not updated with your zap2it username and password (or maybe mistyped) If the username and password are correctly listed in the file, it should work.

Thanks for trying the guide.

George Kazaroth wrote:

Sorry for the delay. The alert hopped into my junk email. I am assuming you are using the tv_grab_file from the code section of the guide. It looks like that may not be the case. A direct link to the file is
https://github.com/rocky4546/script.xmltv.tvheadend/tree/master/usr/bin
It is what I am using and has been working for over a year with 4.0.x and now 4.1.x tvheadend. Once the grabber is present in the "EPG Grabber Modules", it should all work.

Now IF your xmltv.xml file stops generating, then the EPG will slowly disappear because the episodes are from the past. Check the ~/.kodi/temp/zap2it.log to see what the errors are.

As for step 4, you are indicating that the .zap2xmlrc was not updated with your zap2it username and password (or maybe mistyped) If the username and password are correctly listed in the file, it should work.

Thanks for trying the guide.

Thanks for the response.
Yeah, I downloaded the tv_grab_file from that link.
I do not have a zap2it.log file. I searched my whole system for any log file that would be related to zap2it and could not find anything on the system.
I'm working on a bash script that completes all the steps from your guide automatically. I'll post it later once I have had more time to test it on a virtual machine running Ubuntu Server 16.04.

• category-filter.pl
• xmltv.xml
• xmltv_default.xml
• zap2xml.pl
• zap2xml.sh
• cache/
• icons/ (optional)

You should be running zap2xml.sh in your cronjob that will put the log file in ~/.kodi/temp/
Normally, if everything work, it will say something like below at the end of the log file

  Downloaded 701188 bytes in 557 http requests.
  Writing XML file: /home/[user]/xmltv/xmltv_default.xml
  Completed in 1829s (Parse: 1827s) 55 stations, 8779 programs, 24446 scheduled.
  Reading XML file
  Parsing Sections 
  Pass 1 complete
  Updating Genre
  SUCCESS:: New xmltv.xml file generated

George Kazaroth wrote:

Check the date/time of your xmltv.xml file. It should be new unless the script failed. Let's check your setup. In ~/xmltv/ you should have

• category-filter.pl
• xmltv.xml
• xmltv_default.xml
• zap2xml.pl
• zap2xml.sh
• cache/
• icons/ (optional)

You should be running zap2xml.sh in your cronjob that will put the log file in ~/.kodi/temp/
Normally, if everything work, it will say something like below at the end of the log file
[...]
I run the script with the same user that I installed kodi. Kodi puts all its log files in the ~/.kodi/temp/ folder, so the script puts its log file in the same location.

I don't have the following:

category-filter.pl
zap2xml.sh

I don't recall ever seeing that mentioned that I should have those files. Where do I get those?
Please note in regards to Kodi: My current Ubuntu 16.04 machine has the latest version of Kodi installed but I have no plans of installing it on my new Ubuntu Server 16.04 build once I get all this perfected. I'll just have TVheadend installed.

Check out the Wiki section "Setup to update xmltv.xml" It contains the info on the last changes. The files are in the code area in the xmltv folder. (same area you found the grabber file)

George Kazaroth wrote:

Check out the Wiki section "Setup to update xmltv.xml" It contains the info on the last changes. The files are in the code area in the xmltv folder. (same area you found the grabber file)

Okay. Got it. Trying to wrap my head around all this.
So, I don't see it in your guide: Is there no need to change the following in the /usr/bin/tv_grab_file?

if (( $# < 1 ))
then
#  cat ~/.xmltv/tv_grab_file.xmltv
  cat /home/ed/xmltv/xmltv.xml
  exit 0
fi

I really don't know where you got that file. Mine does not have "cat ~/.xmltv/tv_grab_file.xmltv" in it, so I am concerned you have the wrong file.

I try not to change files in /usr/bin/ so instead of making any changes to the file, I do a soft link as described in the install procedure (step 5) to /home/hts/.xmltv
The grabber file I have is the original file without any changes.

Hmmmm... I believe I downloaded it from this:

https://github.com/Rigolo/tv-grab-file

Also, going forward, for the zap2xml.sh file to work, I would need to have Kodi installed on my future Ubuntu Server build?

No, you don't have to have kodi installed. Just manually create the folder structure where the log file is suppose to appear. You can also edit the zap2xml.sh file and change line 3 to point to a different location.

It looks like the grabber file from Rigolo should work, so it is up to you.

I can see that a bundled zip file with an install script would be much better. The current Wiki details what we have found and determined to install the capability up to today. I will take that info and put together an install script that can be used to setup the environment.

I'm actually in the process of writing something. I can post what I have so far and we both can collaborate, if you want. I'm still learning shell scripting so my script would possibly be very basic.
See attached.
The install script I already know works flawlessly. I have not tested the EPG script fully yet.

Looking over my EPG script now and your zap2xml.sh script I am a little confused at your step 4 from your guide. Is that really supposed to be a rename of the xmltv_default.xml file or a copy?
The reason I ask is that in your post above you also state the xmltv_default.xml should be in the xmltv directory

I will review the scripts tonight. Step 4 validates that you have zap2xml.pl and associated file installed correctly and integrated into TVHeadEnd. As you are aware, it takes a little effort to get the grabber installed and working through TVHeadend. This step validates zap2xml.pl can correctly generate the xmltv file and that TVHeadend can read and collect the data from it. The rename is to simulate the execution of the category*pl file that will generate the xmltv.xml file from the xmltv_default.xml file. Once the test works, the rest of the install is simple. If we use an install script, it would probably complete the entire install without the intermediate check and then we need to determine what tests the user makes to validate the pieces are integrated correctly.

I uploaded an install.sh script to the code/xmltv folder on github.
https://github.com/rocky4546/script.xmltv.tvheadend
I used some of your lines. It is my first cut, so give it a try. You can run it multiple times. Also, it will check to see if you have the required files in the installation folder, if not, it will let you know what it needs. I assumed the installation folder is not the ~/xmltv folder.

In many cases it will avoid overwriting the files on the system and will inform the installer when files are skipped. You can remove them and re-run the script to have the files update.

Awesome! I'll try this out sometime this weekend.

I haven't really been following what you are doing here, having set up zap2xml some time back (using the page at https://freetoairamerica.wordpress.com/2014/12/03/some-hints-for-getting-free-to-air-satellite-channels-into-the-electronic-program-guide-in-kodi-or-xbmc-or-another-frontend/ as a reference, among others) but I did briefly glance at your script and noticed you are having it do updates at 8 AM every morning. This is bad, because this means that every user of your script will be hitting the server at the exact same time, bringing unwanted attention to what we are doing. People should change the time of the cron job to some random time in the early AM hours, preferably a time that is not at the top or bottom of the hour, so that it doesn't crash the servers if everyone tries to pull listings at once. If we aren't considerate about server usage, there soon may not be any server for us to be considerate of. All it would take would be for them to make a few tweaks to their web page code, and zap2xml wouldn't work anymore.

Thanks for looking at the script. The install script actually asks for the hour and minute that you want to generate the xmltv.xml from zap2it.com. The 8:00am is to import the xmltv file that was created sometime during the night. So, the time each person grabs the data from zap2it.com will be random. I think it would be good to auto-generate the tvheadend grab time as 3-4 hours after the zap2it script executes. That way it should have completed and be able to be imported. This would work except on the first run where it needs to cache all 2 weeks of data. That can take a long time to run.

Sorry, I misunderstood then. I just get a little concerned about overloading the servers, especially in such a way that will make it obvious that people are running scripts.

In my experience, it doesn't take that long for zap2xml to run, even on the first run, if you don't use the options that make 1 extra http request per program (-D, -I, -W). Without those you get enough data for Kodi's PVR grid to work and you are doing far fewer page loads from the server. I tried using the -W option recently and it took what had been about a one minute process and turned it into something that took between a half hour and an hour, and that was only the listings tagged as movies! So I will live without that bit of extra information in Kodi; if I really want it I can log into my Zap2it account and look it up there. I do wish there was a way to get a bit more information without having to do all those additional page loads.

So, after the script is run, how long before I see a schedule grid in the EPG tab? Do I have to wait until that crontab executes?

I have not determined where to put the initial run that should be accomplished after installing the scripts. You can wait overnight, but most, if not all of us, what to get it done. Be aware the first run will take a while (a few hours). This is because of the settings we are using to reduce the load on the zap2it server and the amount of data you are requesting. You can run by executing the zap2xml.sh file. While it is running, you can tail the zap2xml.log and see how it is progressing. It should just keep scrolling and look boring. Only the end of the log will contain the issues, if they happen. Once that is completed, you will need to ask TVHeadend to manually import the xmltv.xml file that you generated. See step 9 on the Wiki for details on how to run it manually.

Thanks again for the help.

Thank you. Again, you've been phenomenal in assisting me to get this setup. Your quick responses and help has been greatly appreciated. I thank you, and my wife thanks you even if she doesn't know what I (you) am doing to get this PVR up for her.
I hope to be "cutting the cord" soon so having this server setup to record all her favorite shows will be extremely convenient. After all is said and done, I believe the actual biggest effort will be to train her on how to access the recorded media from our TV.

You are where I was a little over 1 year ago. We were looking to disconnect Dish Network and go to an internet only service. It was extremely hard for her to do that, but with Netflix, TVheadend and a few Kodi addons like Pankodi and ABCFamily, she was happy. She has been watching the ABCFamily kodi addon that allows you to view shows that FreeForm allows for free. The next level integration for me looked like this.
TVHeadend records TV shows and places them on file storage.
Kodi is installed and hooked to a TV via HDMI. Kodi has the TVHeadend pvr client which allows watching live TV with timeshifting, recorded TV Shows and control of the TVHeadend server by recording TV shows, as needed. I have had issues with the TVHeadend not maintaining the recorded shows list correctly after a reboot, so my wife uses the mount point through Kodi to access and manage the recorded files directly. So, Kodi must be able to see the mount point where the shows are being recorded. Then, I also have a media server that catalogs movies, tvshows and music called Emby. It is beyond nice. It also has a TVHeadend plugin and I can access the shows from Emby. The nice part is ... my Emby is available on the internet as well, so I can watch my recorded shows (an even live shows) from anywhere. Finally, we have tried Roku Streaming box. It can see any DLNA service and has a Emby client, so that means it can see Windows 10 media on my local PC, and Emby. So we can carry that $29 device to hotels and still access our recordings.

She still wishes she can see HGTV and TLC, but that is it. My biggest issue is that sports are moving more toward cable channels that I have very restricted access (like go to a sports bar). The entire college football bowl games were on ESPN.

Finally, I put all of this on one NUC5i3 little box running around 30 watts and it normally runs around 25% CPU when two shows are running. I also have Nextcloud also available from the internet and the NUC acts as a NAS as well, so it is pretty much amazing that it handles all of that.

Assuming you are using tv-grab-file as your grabber, what you can do is go and temporarily pick "no grabber" from the dropdown list, click save, then go back and reselect tv-grab-file and save again. It seems the act of disabling and re-enabling the grabber forces an immediate re-read of the xml file.

If there's an easier way, I'd like to know about it.

I am using TVHeadend 4.1.x and if you go to the
Configuration -> Channel / EPG -> EPG Grabber Modules
There is a button that says "Re-run Internal EPG Grabbers". Clicking that will reimport the xmltv.xml file. If you run the re-import with the console log open at the bottom, you will see the info that TVHeadend re-imported the xmltv.xml file.

Not sure how to do it with 4.0. That was a long time ago.

George Kazaroth wrote:

I am using TVHeadend 4.1.x and if you go to the
Configuration -> Channel / EPG -> EPG Grabber Modules
There is a button that says "Re-run Internal EPG Grabbers". Clicking that will reimport the xmltv.xml file. If you run the re-import with the console log open at the bottom, you will see the info that TVHeadend re-imported the xmltv.xml file.

Not sure how to do it with 4.0. That was a long time ago.

I see this when I "Rerun..."

2017-03-25 19:43:36.009 xmltv: /usr/bin/tv_grab_file: grab /usr/bin/tv_grab_file
2017-03-25 19:43:36.013 spawn: Executing "/usr/bin/tv_grab_file"
2017-03-25 19:43:36.017 spawn: cat: /home/hts/.xmltv/xmltv.xml: No such file or directory
2017-03-25 19:43:36.018 xmltv: /usr/bin/tv_grab_file: no output detected
2017-03-25 19:43:36.018 xmltv: /usr/bin/tv_grab_file: grab returned no data

Here is the softlink info:

hts@ubupvr:~$ ls -l .xmltv/
total 0
-rw-rw-r-- 1 hts hts  0 Mar 11 14:57 tv_grab_na_tvmedia.conf.TMP
lrwxrwxrwx 1 hts hts 14 Mar 11 14:51 xmltv -> /home/ed/xmltv

ed@ubupvr:~$ ll xmltv/
total 1.4M
drwxrwxr-x 2 ed ed  60K Mar 25 17:00 cache
-rwxr--r-- 1 ed ed  11K Mar 25 15:41 category-filter.pl
-rw-rw-r-- 1 ed ed 1.3M Mar 25 17:00 xmltv_default.xml
-rwxr--r-- 1 ed ed  50K Mar 25 15:41 zap2xml.pl
-rwxr--r-- 1 ed ed  602 Mar 25 15:41 zap2xml.sh