Tvheadend

Thanks Gary. Nice guide. I had to modify the code as the gunzip line was making the file disappear into thin air.

What were the changes that you made to get the script to work?

Hey Gary,

I know this thread is quite old, but i'm pulling my hair out on this one (and i don't have much hair)

I'm not sure if there's a syntax issue or something but i'm getting errors from gunzip regarding the extension being incorrect

Can you elaborate on the escape character as i don't really understand it as much as i should

gzip: /home/hts/.xmltv/tv_grab_file.gz: not in gzip format
gzip: /home/hts/.xmltv/tv_grab_file.xmltv: unknown suffix -- ignored
cat: /var/lib/hts/.xmltv/tv_grab_file.xmltv: No such file or directory
hts@tvr:/usr/bin$

not sure if the format has changed that freeepg uses? is it still in a gzip formazt?

Simply place the 'escape' character' exactly where it says to. What is the exact string you are entering in order to receive the errors? I am using the OTA epg at this time but am willing to try and replicate and or attempt to correct your issue.

Hi Gary

I did what you said, put an escape character "\" for ever "?" and "="
In your example you escaped the & as well?
I ended up pasting my region, uid and key into your command to be sure but the result is the same error


19-07-04 22:39:57.777 xmltv: /usr/bin/tv_grab_file: grab /usr/bin/tv_grab_file
2019-07-04 22:39:57.783 spawn: Executing "/usr/bin/tv_grab_file"
2019-07-04 22:39:57.809 spawn: --2019-07-04 22:39:57-- http://au.freepg.tv/xmltv/oztivo/VIC.Melbourne.gz?UID=XXX9BF76&K=XXX3393AACCBE4A973D83E6F999E5318
2019-07-04 22:39:58.031 spawn: Resolving au.freepg.tv (au.freepg.tv)... 65.182.107.36
2019-07-04 22:39:58.225 spawn: Connecting to au.freepg.tv (au.freepg.tv)|65.182.107.36|:80... connected.
2019-07-04 22:39:58.421 spawn: HTTP request sent, awaiting response... 200 OK
2019-07-04 22:39:58.421 spawn: Length: 511407 (499K) [text/xml]
2019-07-04 22:39:58.421 spawn: Saving to: ‘/home/hts/.xmltv/tv_grab_file.gz’
2019-07-04 22:39:59.004 spawn: 0K .......... .......... .......... .......... .......... 10% 85.7K 5s
2019-07-04 22:39:59.199 spawn: 50K .......... .......... .......... .......... .......... 20% 257K 3s
2019-07-04 22:39:59.202 spawn: 100K .......... .......... .......... .......... .......... 30% 18.8M 2s
2019-07-04 22:39:59.393 spawn: 150K .......... .......... .......... .......... .......... 40% 261K 1s
2019-07-04 22:39:59.396 spawn: 200K .......... .......... .......... .......... .......... 50% 15.0M 1s
2019-07-04 22:39:59.399 spawn: 250K .......... .......... .......... .......... .......... 60% 16.6M 1s
2019-07-04 22:39:59.402 spawn: 300K .......... .......... .......... .......... .......... 70% 14.1M 0s
2019-07-04 22:39:59.587 spawn: 350K .......... .......... .......... .......... .......... 80% 270K 0s
2019-07-04 22:39:59.591 spawn: 400K .......... .......... .......... .......... .......... 90% 13.0M 0s
2019-07-04 22:39:59.594 spawn: 450K .......... .......... .......... .......... ......... 100% 14.4M=1.2s
2019-07-04 22:39:59.595 spawn: 2019-07-04 22:39:59 (426 KB/s) - ‘/home/hts/.xmltv/tv_grab_file.gz’ saved [511407/511407]
2019-07-04 22:39:59.708 spawn: gzip: /home/hts/.xmltv/tv_grab_file.xmltv: unknown suffix -- ignored
2019-07-04 22:39:59.712 spawn: cat: /var/lib/hts/.xmltv/tv_grab_file.xmltv: No such file or directory
2019-07-04 22:39:59.714 xmltv: /usr/bin/tv_grab_file: no output detected
2019-07-04 22:39:59.714 xmltv: /usr/bin/tv_grab_file: grab returned no data

If I run an ls in /home/hts/.xmltv while the script runs i can see the .gz file appear and then disappear and left with a tv_grab_file

root@tvr:/home/hts/.xmltv# ls
tv_grab_file.gz
root@tvr:/home/hts/.xmltv# ls
tv_grab_file

I can see in the first reply to this post another user had to modify the code as the file was disappearing

Breaking down the commands

Running the wget command seems to grab the gz correctly

@root@tvr:/home/hts/.xmltv# wget O /home/hts/.xmltv/tv_grab_file.gz http://au.freepg.tv/xmltv/oztivo/VIC.Melbourne.gz\?UID\=XXX9BF76\&K\=XXX3393AACCBE4A973D83E6F999E5318
--2019-07-04 23:03:12- http://au.freepg.tv/xmltv/oztivo/VIC.Melbourne.gz?UID=XXX9BF76&K=XXX3393AACCBE4A973D83E6F999E5318
Resolving au.freepg.tv (au.freepg.tv)... 65.182.107.36
Connecting to au.freepg.tv (au.freepg.tv)|65.182.107.36|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 511407 (499K) [text/xml]
Saving to: '/home/hts/.xmltv/tv_grab_file.gz'

/home/hts/.xmltv/tv_grab_file.gz 67%[============================================================================> ] 337.06K 360KB/s s
/home/hts/.xmltv/tv_grab_file.gz 100%[==================================================================================================================>] 499.42K 444KB/s in 1.1s

2019-07-04 23:03:13 (444 KB/s) - '/home/hts/.xmltv/tv_grab_file.gz' saved [511407/511407]

root@tvr:/home/hts/.xmltv# ls
tv_grab_file.gz@

The format appears correct

root@tvr:/home/hts/.xmltv# file tv_grab_file.gz
tv_grab_file.gz: gzip compressed data, max speed, from FAT filesystem (MS-DOS, OS/2, NT)


Running the gzip command results in failure

root@tvr:/home/hts/.xmltv# gunzip -f /home/hts/.xmltv/tv_grab_file.gz /home/hts/.xmltv/tv_grab_file.xmltv
gzip: /home/hts/.xmltv/tv_grab_file.xmltv.gz: No such file or directory
root@tvr:/home/hts/.xmltv#

It looks like gunzip has unpacked the file but has not written it as the correct target filename/extension

I can't remember exactly what I did to correct the same issue and I won't be able to muck around at a terminal until tonight.

In the mean time, try gunzip-ing the xmltv file from a different folder to its correct location. From memory I wrote a bash script to grab the file, gunzip, mv the xmltv file to its correct location then delete the the gz file.

I'm using TV Headend with a Synology NAS and IPTV.

None of the linux paths indicated above seem to line up with anything like what I'm seeing on this thing. As an example, the path to which the TVHeadend software is looking for the XMLTV grabber settings is this - /volume1/@appstore/tvheadend-testing/var/epggrab/xmltv.sock .

That said, I am able to get a basic EPG (program names and times only) and am able to watch live and record, all perfectly, so I can't really complain too much!

I am, however, really struggling to take the information posted here and apply it to my installation because of the differences in where the files are all supposed to be located.

Any help would be very much appreciated!

Thanks,

MJJ

In case this is useful to anyone:

gunzip takes a list of files on its command line and replaces each file
       whose  name ends with .gz, -gz, .z, -z, or _z (ignoring case) and which
       begins with the correct magic number with an uncompressed file  without
       the  original extension.

root@tvr:/home/hts/.xmltv# gunzip -f /home/hts/.xmltv/tv_grab_file.gz /home/hts/.xmltv/tv_grab_file.xmltv
gzip: /home/hts/.xmltv/tv_grab_file.xmltv.gz: No such file or directory

The above error is happening because you're telling gunzip to decompress tv_grab_file.xmltv which doesn't exist.
What you want is:

gunzip -f /home/hts/.xmltv/tv_grab_file.gz
 mv /home/hts/.xmltv/tv_grab_file /home/hts/.xmltv/tv_grab_file.xmltv

Alternately, this would work too:

gunzip -fc /home/hts/.xmltv/tv_grab_file.gz > /home/hts/.xmltv/tv_grab_file.xmltv

And finally, why not just name the file properly in the 1st place? tv_grab_file.xmltv.gz will unzip to tv_grab_file.xmltv

Also a note regarding escaping URLs for wget, much simpler just to put the URL in single quotes, so what you really want is:

XMLTV=/home/hts/.xmltv
 wget -O $XMLTV/tv_grab_file.xmltv.gz 'COPY-AND-PAST-LINK-FROM-EMAIL-HERE'
 gunzip -f $XMLTV/tv_grab_file.xmltv.gz

Set XMLTV to the appropriate path & you're good to go.

John Jones:

The path you've given is the path to the TVHeadend epggrab socket, which is a different beast entirely. You can use that if
tv_grab_file doesn't work. You'll need 'socat' for that.

e.g. In your case that would be:

TVHSOCK=/volume1/@appstore/tvheadend-testing/var/epggrab/xmltv.sock
 XMLTV=/home/hts/.xmltv # Set this path according to your setup
 XMLFILE=$XMLTV/tv_grab_file.xmltv
 wget -O $XMLFILE.gz 'COPY-AND-PAST-LINK-FROM-EMAIL-HERE'
 gunzip -f XMLFILE.gz
 cat $XMLFILE | socat - UNIX-CONNECT:$TVHSOCK

Thanks so much Hiro. Really appreciate you helping us all out like this!

I'm still slightly at a loss, however, to understand how to replace the /home/hts/.xmltv path with the path for my system, which has none of those directories in it. You tell me to "Set this path according to your setup" however, I'm simply not experienced enough to understand how that path is represented in my setup.

I don't have a "home" folder. I don't have a "hts" folder, and I don't know where to find the ".xmltv" folder either if it exists on my system. The busybox linux OS used in the Synology NAS seems to require the TVHeadend software installation to be set up somewhat differently, and I'm certain that if someone could help me understand how that /home/hts/.xmltv path is expressed in my system, I might have half a chance. If you could perhaps explain to me what those paths represent to the OS, and the TVHeadend installation, I might be able to somehow find their equivalents in the Synology ecosystem.

Thanks in advance!

MJJ

Ok, so I think that your /home/hts/.xmltv path is the same as my /usr/bin path. That's where TVHeadend puts the tv_grab_file files and reads them from there too. However, turns out that the Synology OS doesn't have socat, so that's out.

In any case, I'm finding that I can still execute that tv_grab_file to a point. It seems to be downloading and then saving the tv_grab_file.xmltv.gz file successfully, but then gzip is complaining "gzip: XMLFILE.gz: No such file or directory" when it gets to that line of the script. Permissions?

This is how that section of the script now reads with my paths in it -

#!/bin/bash
dflag=
vflag=
cflag=
if (( $# < 1 ))
then
TVHSOCK=/volume1/@appstore/tvheadend-testing/var/epggrab/xmltv.sock
XMLTV=/usr/bin
XMLFILE=$XMLTV/tv_grab_file.xmltv
wget -O $XMLFILE.gz 'http://au.freepg.tv/xmltv/oztivo/NSW.Sydney.gz?UID=<myid>&K=<mykey>'
gunzip -f XMLFILE.gz
cat $XMLFILE | socat - UNIX-CONNECT:$TVHSOCK
exit 0
fi

Thanks again for looking at this for us. Really appreciate it!

MJJ

OK, firstly, that's my mistake, the gunzip -f XMLFILE.gz line should be:

gunzip -f $XMLFILE.gz

/home/hts/.xmltv path is the same as my /usr/bin path

No, what directory are you in when you log in? [the 'pwd' command will tell you if you don't know].
That's your home directory. You should put data files in a directory under there. Not in /usr/bin,
that's for programs.

For the purposes of stuff like this, the path you store in XMLTV isn't critical, other than
it must of course exist, so you could make it $HOME/junk [$HOME is where you end up when you log in],
or even /tmp [that's where I put mine]. The main thing is that you know where you're putting it so you can
tell the other stuff in your script where to find it.

Of course the script above won't work if you don't have socat. you can use nc, but you need the 'right' nc, only some versions
can write to a unix socket - what fun.

You're probably better off sticking with tv_grab_file - in which case you must store your file
where tv_grab_file is going to look for it. You can easily modify tv_grab_file to look in the right place.

So something like this should work.

XMLTV=/tmp
 XMLFILE=$XMLTV/tv_grab_file.xmltv
 wget -O $XMLFILE.gz 'http://au.freepg.tv/xmltv/oztivo/NSW.Sydney.gz?UID=<myid>&K=<mykey>'
 gunzip -f $XMLFILE.gz
 cat $XMLFILE

Hi Hiro,

That's got it! Thanks so much!

The script tv_grab_file seems to now do it's thing when I invoke it via SSH, and the data all looks ok when it flashes by.

The EPG data only, however, appears in some of the shows, and not in others. When I download the EPG manually with a web browser, I can see details on every channel and every program, but it's seems to be sparsely scattered on the EPG in TVHeadend. I've restarted TVHeadend, I've gone through and manually chosen the EPG Source in the Configuration / Channel-EPG / Channels tab but I'm a little confused as all the XMLTV grabbers seem to be in there twice for some reason, once at /bin/ and the other one is at /usr/bin/, even the one that you have been helping me set up, so that's confusing. If I change the description in one with nano, it changes in both. I've Re-run Internal EPG Grabbers in both the EPG Grabber Modules tab and the EPG Grabber tab.

Any suggestions on what I could look at as being the reason the import of the data is intermittent? Some data is definitely getting through, just not all of it and I can't work out any pattern in the failures.

Thanks again!

MJJ

Hi again,

Just saw this in the log - 2019-09-23 00:04:04.591 spawn: 2019-09-23 00:04:04 (276 KB/s) - '/tmp/tv_grab_file.xmltv.gz' saved [454166/454166]
2019-09-23 00:04:04.597 spawn: gzip: /tmp/tv_grab_file.xmltv: Operation not permitted
2019-09-23 00:04:04.680 xmltv: /bin/tv_grab_file: grab took 2 seconds
2019-09-23 00:04:04.736 xmltv: /bin/tv_grab_file: parse took 0 seconds
2019-09-23 00:04:04.737 xmltv: /bin/tv_grab_file: channels tot= 26 new= 0 mod= 0
2019-09-23 00:04:04.737 xmltv: /bin/tv_grab_file: brands tot= 0 new= 0 mod= 0
2019-09-23 00:04:04.737 xmltv: /bin/tv_grab_file: seasons tot= 0 new= 0 mod= 0
2019-09-23 00:04:04.737 xmltv: /bin/tv_grab_file: episodes tot= 0 new= 0 mod= 0
2019-09-23 00:04:04.737 xmltv: /bin/tv_grab_file: broadcasts tot= 0 new= 0 mod= 0

Operation not permitted? Have I still got a bad path or permissions problem?

MJJ

gzip: /tmp/tv_grab_file.xmltv: Operation not permitted

This is probably because you created tv_grab_file.xmltv and now the tvheadend user doesn't have permission to write to it.

Simplest way to fix is to modify your script thus:

XMLTV=/tmp
 XMLFILE=$XMLTV/tv_grab_file.xmltv
 wget -O $XMLFILE.gz 'http://au.freepg.tv/xmltv/oztivo/NSW.Sydney.gz?UID=<myid>&K=<mykey>'
 gunzip -f $XMLFILE.gz
 chmod 666 $XMLFILE
 cat $XMLFILE

This will allow any user to overwrite that file.

I'm a little confused as all the XMLTV grabbers seem to be in there twice for some reason, once at /bin/ and the other one is at /usr/bin/

Ignore this, it's a non-issue.

The EPG data only, however, appears in some of the shows, and not in others.

If you check under Channel/EPG, look in the EPG Grabber channels tab. Make sure each entry in the 'Name' column has the correct channel set under the 'Channels' column.

If that's all OK, run the grabber manually twice - the 1st time nothing much will happen, the 2nd time you should get EPG data appearing in all the channels. You only need to do this twice the 1st time round, subsequently it should update properly each time it's run [assuming there is new EPG data].

If the change to the script works properly, tvheadend should be able to run the update without problems.

Hi Hiro,

Thanks again. I think I'd worked out what has been happening with the permission issue. When I ran that script manually, I did it as the user "admin" not as the user the TVHeadend application is, so this is why it couldn't write or even chmod that file. I've deleted the ones created when I manually ran the script and now that's all working fine.

I run two EPG services. One is the Matt Huisman one and the other is the freepg one. Matt's one only contains program information for about two channels, where the freepg one has most of the rest. I'm suspecting that the script that parses the XML is not aligned with the XML schema used in the freepg file and this might be why hardly anything turns up in the EPG. Matt's schema and parsing script works perfectly. The other one looks very similar to Matt's one, but the code around the parse is compeltely different. None of it makes any sense to me, but just looking at them side by side and they seem competely different.

I'm a little unsure about the etiquite around posting this data in an open forum like this, but I feel that if you could have a look at this data you might be able to let me know if I'm using the correct parsing code for that schema. Any ideas?

Again, thank you SO very much for all your help with this. You've been incredible. I really appreciate it! With your help, I'm certain we're going to get to the bottom of this very soon!

MJJ

Hi again,

Just a note to let you know that I have gone through the EGP Grabber Channels tab and made sure to assign only one EPG to each channel service and make sure all the names are aligned.

I also went into Configuration/Channel-EPG/Channels tab and selected the desired EPG source and made there was only one for each channel.

Not sure where to go from here.

MJJ

John Jones wrote:

Matt's one only contains program information for about two channels, where the freepg one has most of the rest.

I'm a bit puzzled by this - I use Matt's EPG, & it contains over 20 AU channels [plus most of the NZ ones] - why are you only getting 2?
The limitation with Matt's EPG data that bothers me is there's no program metadata, & quite often just the program name doesn't
tell you much.

I'm a little unsure about the etiquite around posting this data in an open forum like this, but I feel that if you could have a look at this data you might be able to let me know if I'm using the correct parsing code for that schema. Any ideas?

I think it's fine to put your file[s] on a site like pastebin & post a link to it here. A query like this would be better posted in the XMLTV forum.

Hi,

I too get all of Matt's 20 channels for Sydney, but only two of them have the additional program information, and they actually work, so that's why I set the system to use those two from Matt's list and the others from the other one, as it does contain EVERYTHING! Program description, episode numbers, cast names, etc. Not much use to me if I can't get TVHeadend to pull them in, though. I'll put a question into that forum and see if those guys and gals have any suggestions for things for me to try. If I have no luck there, if it's ok with you, I'll come back here and put some example files together in a package on postbin as you suggest.

Thanks again and again!

MJJ

Not sure where you were planning on putting your question, but I was referring to the XMTV forum on this site.

Me neither, to be honest. I didn't understand that you meant the one on this site until now.