Tvheadend

well, it looks like the xmltv_default.xml file might have completed, but the next part did not. If you ran the zap2xml.sh file, there should be a zap2xml.log in /home/ed/.kodi/temp/
I updated the zap2xml.sh script so it will place the log file in the .../xmltv/ folder if the kodi folder does not exist. So, if the kodi folders do not exists, you can
1) grab the latest zap2xml.sh that will put the *.log in your current .../xmltv/ folder or
2) create the folders that the log is looking for (/home/ed/.kodi/temp/)

edit the ~/.zap2xmlrc file and change the line
ncsdays=2
to
ncsdays=0
which will cause the script to not contact zap2it.com and will complete fast.
re-execute zap2xml.sh and review the end of the zap2xml.log and see what is going on.

Ran the script and tailed the log file that was located in the .kodi/temp (I have Kodi currently installed). This is the last few lines:

Downloaded 19626 bytes in 21 http requests.
Writing XML file: /home/ed/xmltv/xmltv_default.xml
Completed in 71s (Parse: 71s) 5 stations, 1278 programs, 2159 scheduled.
Reading XML file
Parsing Sections
Pass 1 complete
Updating Genre
SUCCESS:: New xmltv.xml file generated

Logged into the web console and saw a blank EPG tab. Went into Configuration>Channel/EPG>EPG Grabber and selected the "Re-run..." and still saw the same output:

2017-03-25 20:56:35.000 xmltv: /usr/bin/tv_grab_file: grab /usr/bin/tv_grab_file
2017-03-25 20:56:35.004 spawn: Executing "/usr/bin/tv_grab_file" 
2017-03-25 20:56:35.027 spawn: cat: /home/hts/.xmltv/xmltv.xml: No such file or directory
2017-03-25 20:56:35.028 xmltv: /usr/bin/tv_grab_file: no output detected
2017-03-25 20:56:35.028 xmltv: /usr/bin/tv_grab_file: grab returned no data

Even attempted a reboot.
Is it possible that softlink isn't setup correctly?
I have a VM running Ubuntu Server I plan on testing your install script to see what happens also. I have a second zap2it account I can use to test.

The log indicates the xmltv.xml file was successfully created. For some reason in your previous post, the xmltv.xml was not listed in the xmltv folder. Please confirm that the file exists in the folder. If it is there, then your softlink was not generated when you ran the install.sh script. Not sure how that can happen. To confirm the link is missing,
cd /home/hts
ls -al
you should a line that looks like
lrwxrwxrwx 1 hts hts 20 Dec 29 2015 .xmltv -> /home/[user]/xmltv
but I suspect it is somehow missing. To create the softlink manually,
cd /home/hts
sudo ln -s /home/[user]/xmltv .xmltv
sudo chown hts:hts .xmltv
ls -al
Check to see if the link is now present. You can also verify you can see the xmltv.xml file by
ls .xmltv
If so, try to re-run the grabber in tvheadend.

The softlink is there:

ed@ubupvr:~$ sudo su - hts
hts@ubupvr:~$ ls -l .xmltv/
total 0
-rw-rw-r-- 1 hts hts  0 Mar 11 14:57 tv_grab_na_tvmedia.conf.TMP
lrwxrwxrwx 1 hts hts 14 Mar 11 14:51 xmltv -> /home/ed/xmltv

So, I ran the script on my test VM and two things happened:

1. The chown of the softlink did not work, it is owned by root:

ed@ubuntusrvtest:~$ sudo su - hts
[sudo] password for ed: 
hts@ubuntusrvtest:~$ ls -l .xmltv/
total 0
lrwxrwxrwx 1 root root 14 Mar 25 21:10 xmltv -> /home/ed/xmltv

2. As your script instructed at the end, since the config file from tvheadend did not recognize the new tv_grab_file I rebooted and ran the script again and received the output ""Softlink not found, Adding softlink" even though the softlink exist.

I'll post again in a few to give you the output of the EPG Grabber "Re-run..." to see if I get the same as my post above from my "production" machine.

That looks really wrong to me. Looks like you have a folder blocking the softlink which is probably why the install could not correctly install it. I would recommend removing the folder and recreating it as hts.
rm -rf /home/hts/.xmltv
ln -s /home/hts/.xmltv /home/ed/xmltv
ls -al
Check to see if it correct this time as show in previous post
lrwxrwxrwx 1 hts hts 20 Dec 29 2015 .xmltv -> /home/[user]/xmltv

If it still look like what you posted, you really need to remove/delete/get_rid_of the .xmltv folder. It is way wrong.

Everything was setup by the tvheadend install and your script. I ran it from a fresh base install of Ubuntu Server 16.04.

hts@ubuntusrvtest:~$ ls -la
total 20
drwxr-xr-x 4 hts  hts  4096 Mar 25 21:13 .
drwxr-xr-x 4 root root 4096 Mar 25 21:08 ..
-rw------- 1 hts  hts    52 Mar 25 21:27 .bash_history
drwxr-xr-x 3 hts  hts  4096 Mar 25 21:08 .hts
drwxrwxrwx 2 hts  hts  4096 Mar 25 21:10 .xmltv

hts@ubuntusrvtest:~$ ls -l .xmltv/
total 0
lrwxrwxrwx 1 root root 14 Mar 25 21:10 xmltv -> /home/ed/xmltv

ed@ubuntusrvtest:~$ ls -l
total 100
drwxrwxr-x 2 ed ed  4096 Mar 25 21:28 -
-rwxr--r-- 1 ed ed 10449 Mar 25 21:06 category-filter.pl
-rw-rw-r-- 1 ed ed  1744 Mar 27  2013 repo.gpg.key
-rw-rw-r-- 1 ed ed  1027 Mar 25 21:06 tv_grab_file
-rwxr--r-- 1 ed ed  4853 Mar 25 21:06 tvheadend_EPG_setup.sh
-rwxr--r-- 1 ed ed   317 Mar 25 21:06 tvheadend_install.sh
drwxrwxr-x 3 ed ed  4096 Mar 25 22:42 xmltv
-rw-rw-r-- 1 ed ed   539 Mar 25 21:06 x.zap2xmlrc
-rwxr--r-- 1 ed ed 50190 Mar 25 21:06 zap2xml.pl
-rwxr--r-- 1 ed ed   602 Mar 25 21:06 zap2xml.sh

ed@ubuntusrvtest:~$ ls -l xmltv/
total 2760
drwxrwxr-x 2 ed ed   53248 Mar 25 22:42 cache
-rwxr--r-- 1 ed ed   10449 Mar 25 21:14 category-filter.pl
-rw-rw-r-- 1 ed ed 1300690 Mar 25 22:42 xmltv_default.xml
-rw-rw-r-- 1 ed ed 1399442 Mar 25 22:42 xmltv.xml
-rwxr--r-- 1 ed ed   50190 Mar 25 21:14 zap2xml.pl
-rwxr--r-- 1 ed ed     602 Mar 25 21:14 zap2xml.sh

hts@ubuntusrvtest:~$ rm -rf .xmltv
hts@ubuntusrvtest:~$ ls -la
total 16
drwxr-xr-x 3 hts  hts  4096 Mar 25 22:48 .
drwxr-xr-x 4 root root 4096 Mar 25 21:08 ..
-rw------- 1 hts  hts    84 Mar 25 22:47 .bash_history
drwxr-xr-x 3 hts  hts  4096 Mar 25 21:08 .hts

hts@ubuntusrvtest:~$ ln -s /home/hts/.xmltv /home/ed/xmltv
ln: failed to create symbolic link '/home/ed/xmltv/.xmltv': Permission denied

hts@ubuntusrvtest:~$ ln -s /home/ed/xmltv .xmltv                                                              
hts@ubuntusrvtest:~$ ls -la
total 16
drwxr-xr-x 3 hts  hts  4096 Mar 25 22:58 .
drwxr-xr-x 4 root root 4096 Mar 25 21:08 ..
-rw------- 1 hts  hts   248 Mar 25 22:55 .bash_history
drwxr-xr-x 3 hts  hts  4096 Mar 25 21:08 .hts
lrwxrwxrwx 1 hts  hts    14 Mar 25 22:58 .xmltv -> /home/ed/xmltv

hts@ubuntusrvtest:~$ ls -l .xmltv
lrwxrwxrwx 1 hts hts 14 Mar 25 22:58 .xmltv -> /home/ed/xmltv

I was able to reproduce the error. It was caused by running the install script with the .xmltv already present. The install script did error, but did not exit immediately. I have corrected the script so it does exit if the link fails to generate so the user can correct the issue. Based on what I see above, it should now work as expected by the grabber in TVHeadend. It is always good to get someone else to run the software. thanks!

I'll try to test the updates on my Ubuntu Server VM today. I may even try a screen record.

So attached is my first failure. I'll fix and try again with a screen cast later. Gotta go grocery shopping now...

unable to view the mp4, but let me know what happened when you can.

I posted on You Tube, for the time being. I haven't had the chance to reconfigure and try again. It may not be until later this evening before I can get to it.

Edited:
Forgot to post the You Tube link
https://youtu.be/17d398i2iO8

YouTube worked. Thanks for the video. I have been building releases in the source area and would probably recommend using that instead of making your own tvheadend_EPG fileset. Download the release from
https://github.com/rocky4546/script.xmltv.tvheadend/releases
unzip the fileset somewhere
place the zap2xml.pl where the install.sh file exists
run ./install.sh

I have updated the Wiki to include the Automated Installation procedure at the top.
https://github.com/rocky4546/script.xmltv.tvheadend/wiki/Guide:-How-to-Setup-XMLTV-for-TVHeadEnd

I still think the way to get the grabber recognized by TVHeadend is unknown, but the rest should be much cleaner.

If one were to download either compressed file from the automated installation, extract and launch the install.sh, shouldn't the install.sh move the files from the uncompressed folder structure to the proper location (root of the user's home directory?) before the file existence check in the install.sh starting at line 17? And, shouldn't it also automatically create the ~/xmltv folder before the file check? Just wondering.

I don't think the install.sh script accounts for the extracted folder. This is what happens when I unzip the file and just launch the script:

ed@ubuntusrvtest:~$ unzip script.xmltv.tvheadend-0.1-RC.4.zip 
Archive:  script.xmltv.tvheadend-0.1-RC.4.zip
3e26c4ccf406c7f17f4f7ba48d31a11f06fded39
   creating: script.xmltv.tvheadend-0.1-RC.4/
  inflating: script.xmltv.tvheadend-0.1-RC.4/LICENSE.md  
  inflating: script.xmltv.tvheadend-0.1-RC.4/README.md  
  inflating: script.xmltv.tvheadend-0.1-RC.4/install.sh  
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/bin/
  inflating: script.xmltv.tvheadend-0.1-RC.4/usr/bin/tv_grab_file  
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/
   creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/
  inflating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/MyPVRChannels.xml  
  inflating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/MyPVRGuide.xml  
  inflating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/README.md  
  inflating: script.xmltv.tvheadend-0.1-RC.4/x.zap2xmlrc  
   creating: script.xmltv.tvheadend-0.1-RC.4/xmltv/
  inflating: script.xmltv.tvheadend-0.1-RC.4/xmltv/category-filter.pl  
  inflating: script.xmltv.tvheadend-0.1-RC.4/xmltv/zap2xml.sh  
  inflating: script.xmltv.tvheadend-0.1-RC.4/zap2xml.sh

ed@ubuntusrvtest:~$ script.xmltv.tvheadend-0.1-RC.4/install.sh 

### Checking for files to install
usr/bin/tv_grab_file not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

x.zap2xmlrc not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

xmltv/category-filter.pl not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

xmltv/zap2xml.sh not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

zap2xml.pl not found
Please download file from http://zap2xml.awardspace.info
and place into the folder containing install.sh

got it. I assumed it ran inside the folder. it is an easy change to cd into the folder at the top of the script. Then it should run fine. I'll make the update

Sorry for the late response. I've been under the weather lately.
So, I tried using the recent updated install script and this is what happened:

https://youtu.be/CkpSO6df7jc

Looked in the web console and still got this output from the EPG Grabber tab (also seen at the end of the video):

2017-03-28 21:26:58.000 xmltv: /usr/bin/tv_grab_file: grab /usr/bin/tv_grab_file
2017-03-28 21:26:58.002 spawn: Executing "/usr/bin/tv_grab_file" 
2017-03-28 21:26:58.003 spawn: cat: /home/hts/.xmltv/xmltv.xml: No such file or directory
2017-03-28 21:26:58.004 xmltv: /usr/bin/tv_grab_file: no output detected
2017-03-28 21:26:58.004 xmltv: /usr/bin/tv_grab_file: grab returned no data

Questions regarding the install script:
1. Would it be beneficial to include a 'mkdir xmltv' at the beginning, say around line 11?
2. Also, in case one would not move the zap2xml.pl to the uncompressed directory where the install.sh file is before launching the script, would it also be beneficial to insert a find and move of the zap2xml.pl file in the install.sh script? Something like 'find /home -name "zap2xml.pl" -exec mv {} `dirname $0` \;
3. As far as the softlink error goes, that output will generate the error until the zap2xml.sh file executes, correct? If so, would it make since at the end of the install.sh file to give the user the option or not to execute that zap2xml.sh file?

Saw you video. Looks real good. I am thinking the reboot is not needed if the TVHeadend shows the tv_grab_file in the list. So, I will probably take that out. Also, I did find a bug in the crontab update. It has 2 .../xmltv/xmltv/... in the path. I corrected that and reposted a release. The reason it failed when you saved is because you have not executed the zap2xml.sh script to create it. I will update the end of the script to be clearer. I recommend running the zap2xml.sh with the option fast. So the command would be
zap2xml.sh fast
This will remove the delays we normally add between the queries to zap2it.com and will make it run very fast. It is that or wait a lot of hours for the first run. This info has been posted in the manual update section on the Wiki.

question #1: The mkdir is on line 169. The purpose of moving to a earlier point would be to make sure it can happen. It there a concern that it would not work? I figured since it was in your home area, it would never have an issue. Let me know your thoughts on why to move it earlier and sure...
Question #2: I have a big concern with #2. It is very likely a user could have multiple copies of the file somewhere in /home/* and picking up the wrong version is a big issue. We could see if it is present and if not ask where is it located and give the URL where it can be downloaded. If they provide a good folder where it exists, then we could move it and finish the execution. Question: Should we ask for the file location or the folder where the file exists?
Question #3: I will add the question and execution at the end of the script. and will note this can take a while.

A note on the category*pl file. If you notice the translations are in English. TVHeadend C-code has hard-coded the genre to be English only. Of course, a person could go to zap2it and say give me French and then the genre would be in that language and the translations to English for tvheadend would misfire since all the translations in the category*pl file are in English. I suspect in the long run, it should probably have multiple language files available for different language translations.

I will let you know when I have the updates completed

updates posted. I came up with a fix for #1. Give it a try and see what you think.

I didn't get a chance to create a video but this is what I found in testing the new install.sh:

1. zap2xml.pl directory is asked and answer given. However, it doesn't look as if after the answer is given the file is copied (or moved) to a new, correct location/directory. After selecting the tv_grab_file in the web console, I executed the install.sh again and was asked for the location again and received this output even though I gave the correct location:

### Deploying scripts to /home/ed/xmltv
chmod: cannot access 'zap2xml.pl': No such file or directory
cp: cannot stat 'zap2xml.pl': No such file or directory

I looked and the zap2xml.pl still existed at the root of my home directory. Perhaps, as you mentioned, asking for the zap2xml.pl is not a good idea.

2. On line 202 of the install.sh, I see this: Do you want to quickly create a 2 day EPG? [Y|y]
Shouldn't that be Do you want to quickly create a 2 day EPG? [Y|N]

3. Found this on line 209:

/home/mccaslin/xmltv/zap2xml.sh fast -d 2

I got this error output on second execution of the install.sh before I found that line:

script.xmltv.tvheadend-0.1-RC.7/install.sh: line 209: /home/mccaslin/xmltv/zap2xml.sh: No such file or directory

Also, I just noticed the uncompressed file/folder structure of the new zip file:

ed@ubuntusrvtest:~$ ll script.xmltv.tvheadend-0.1-RC.7
total 116
-rwxr--r-- 1 ed ed  7083 Mar 28 23:30 install.sh
-rw-rw-r-- 1 ed ed 35141 Mar 28 23:30 LICENSE.md
-rw-rw-r-- 1 ed ed  1141 Mar 28 23:30 README.md
-rwxr--r-- 1 ed ed 50190 Mar 29 05:26 script.xmltv.tvheadend-0.1-RC.7
drwxrwxr-x 4 ed ed  4096 Mar 28 23:30 usr
drwxrwxr-x 2 ed ed  4096 Mar 28 23:30 xmltv
-rw-rw-r-- 1 ed ed   540 Mar 28 23:30 x.zap2xmlrc
-rwxr--r-- 1 ed ed  1115 Mar 28 23:30 zap2xml.sh

Maybe that script.xmltv.tvheadend-0.1-RC.7 file may have happened as a result of the new zap2xml.pl question?

Sorry about that.
#1 fixed issues. It should now work as expected. Enter either a relative path or an absolute path. It will continue to work with you until either it finds the file or you hit return with no path. So, if it is in the folder where you execute, entering a dot "." should find the file.
#2 Fixed. I looked up the standard. Uppercase represents the default and lowercase represent alternative answers. So [N|y] means N is default.
#3 Oops. Fixed path
Also made a few additional changes. Published a new pre-release.

Here is the test. It still doesn't seem to correctly accept the path for the zap2xml.pl for some reason. The questioned is asked 31 seconds into the video:

https://youtu.be/BM8bDK6HDz0

Everything else seemed to work just fine. I'll be trying the automated process on my production machine with Ubuntu 16.04 either early tomorrow morning or tomorrow night. Then, maybe this weekend, I plan to rebuild my production machine with Ubuntu Server 16.04 and start from scratch.
Question: Once the the 2 day EPG is pulled, should the EPG tab be populated with the TV schedule? Or will that be populated later after the cronjob?

Added support for absolute path when finding zap2xml.pl file.
If you go ahead and say yes to generating the initial high speed 2 day xmltv.xml file, then you can import that file into TVHeadend manually and the EPG data will appear in TVHeadend. You can then tie the EPG data to your channels and services. I would recommend rerunning ./zap2xml.sh file, but it will take a long while to generate the entire 14 days in your cache. You may want to do that just to get the cache filled up with episode data. I have over 11,000 files in my cache. That would take 9 hours to recreate!

Just completed it on my prod machine and it works well.
Thank you so much for all your hard work. If you had a Paypal button on your site I'd make a donation. In the meantime, I'll buy you a virtual beer. Cheers!

Hi all,

It looks like Zap2It has redone their website, and the script can no longer log in. Anyone have an idea if this procedure can still work with some tweaks? Thanks

You can check out my zap2epg version that is built for the new site:

https://github.com/edit4ever/script.module.zap2epg

It is designed to be used in Kodi with Tvheadend - but I'm sure you can modify to use on other platforms if you want.

edit4ever ! wrote:

You can check out my zap2epg version that is built for the new site:

https://github.com/edit4ever/script.module.zap2epg

It is designed to be used in Kodi with Tvheadend - but I'm sure you can modify to use on other platforms if you want.

Well here is the problem with that, you made it a Kodi addon rather than a standalone script like zap2xml. I don't run Kodi on my backend system, and also don't want to have to depend on a Kodi addon to configure Tvheadend. Tvheadend and Kodi are two completely separate software projects that just happen to work together in certain ways, but Tvheadend shouldn't be dependent of the presence of Kodi for configuration. Not saying that your software won't be useful for some people; I'm sure it will be, but it's not a substitute for zap2xml.

A larger problem at the moment, and I am HOPING this is just a temporary glitch, is that existing logins don't seem to work on the Zap2it site. So, you cannot access your favorite channels list. The introductory popup seems to indicate that existing logins should work, but for the moment they just don't; you get an error that you have entered an invalid username and password. I did not try creating a new account, because I'm hoping Zap2it will fix it so that existing accounts work, but if anyone else wants to try creating a new account and then logging out and back in, feel free. So in order to get full functionality back, first Zap2it needs to fix the authentication issue (or maybe we will all need to create new accounts, but I hope not), and then the Zap2xml author will need to modify his software to make it work with the new listings, assuming he chooses to do so (i suspect he will if he can, but have no way of knowing that). EDIT: Existing logins DO work, but only at tvschedule.zap2it.com