Tvheadend

I'm intrigued enough by this issue that I thought I'd try freepg out myself.
I'll let you know how it looks in due time.

Meanwhile, there's something you could try. Looking at the TVH web interface, open the log viewer [that's a button that looks like two ^ characters at the bottom right of the page] and then run your EPG grabber. You should see something similar to this in the log:

2019-09-27 11:22:48.020 xmltv: xmltv: grab took 1 seconds
 2019-09-27 11:22:48.202 xmltv: xmltv: parse took 0 seconds
 2019-09-27 11:22:48.202 xmltv: xmltv:  channels   tot=   26 new=    0 mod=    0
 2019-09-27 11:22:48.202 xmltv: xmltv:  brands     tot=    0 new=    0 mod=    0
 2019-09-27 11:22:48.202 xmltv: xmltv:  seasons    tot=    0 new=    0 mod=    0
 2019-09-27 11:22:48.202 xmltv: xmltv:  episodes   tot=   90 new=    0 mod=    0
 2019-09-27 11:22:48.202 xmltv: xmltv:  broadcasts tot=   90 new=    0 mod=    0

It would be helpful if you could post the output you get.

Hi Hiro,

I've been interstate working and have just finally had a chance to look at this now, and I'm stuffed if I know why, but it's all working perfectly now. I've got no idea what's happened, but every channel that freepg has data on, has data in my Kodi EPG. It's wonderful!

Here's that log anyway if it helps -

2019-09-30 15:00:18.235 htsp: 192.168.2.148: Welcomed client software: Kodi Media Center (HTSPv34)
2019-09-30 15:00:18.236 htsp: 192.168.2.148 [ Kodi Media Center ]: Identified as user 'admin'
2019-09-30 15:00:18.236 htsp: 192.168.2.148 [ admin | Kodi Media Center ]: Privileges updated
2019-09-30 15:05:53.759 xmltv: /bin/tv_grab_file: grab /bin/tv_grab_file
2019-09-30 15:05:53.762 spawn: Executing "/bin/tv_grab_file"
2019-09-30 15:05:53.771 spawn: --2019-09-30 15:05:53-- http://au.freepg.tv/xmltv/oztivo/NSW.Sydney.gz?UID=myid&K=mypass
2019-09-30 15:05:53.954 spawn: Resolving au.freepg.tv... 65.182.107.36
2019-09-30 15:05:54.168 spawn: Connecting to au.freepg.tv|65.182.107.36|:80... connected.
2019-09-30 15:05:54.392 spawn: HTTP request sent, awaiting response... 200 OK
2019-09-30 15:05:54.392 spawn: Length: 449601 (439K) [text/xml]
2019-09-30 15:05:54.392 spawn: Saving to: '/tmp/tv_grab_file.xmltv.gz'
2019-09-30 15:05:55.034 spawn: 0K .......... .......... .......... .......... .......... 11% 77.9K 5s
2019-09-30 15:05:55.250 spawn: 50K .......... .......... .......... .......... .......... 22% 231K 3s
2019-09-30 15:05:55.463 spawn: 100K .......... .......... .......... .......... .......... 34% 235K 2s
2019-09-30 15:05:55.468 spawn: 150K .......... .......... .......... .......... .......... 45% 9.33M 1s
2019-09-30 15:05:55.681 spawn: 200K .......... .......... .......... .......... .......... 56% 235K 1s
2019-09-30 15:05:55.686 spawn: 250K .......... .......... .......... .......... .......... 68% 10.5M 1s
2019-09-30 15:05:55.891 spawn: 300K .......... .......... .......... .......... .......... 79% 244K 0s
2019-09-30 15:05:55.898 spawn: 350K .......... .......... .......... .......... .......... 91% 6.82M 0s
2019-09-30 15:05:55.902 spawn: 400K .......... .......... .......... ......... 100% 10.3M=1.5s
2019-09-30 15:05:55.902 spawn: 2019-09-30 15:05:55 (291 KB/s) - '/tmp/tv_grab_file.xmltv.gz' saved [449601/449601]
2019-09-30 15:05:56.017 xmltv: /bin/tv_grab_file: grab took 2 seconds
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: parse took 0 seconds
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: channels tot= 26 new= 0 mod= 0
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: brands tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: seasons tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.388 xmltv: /bin/tv_grab_file: episodes tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.388 xmltv: /bin/tv_grab_file: broadcasts tot= 2310 new= 14 mod= 0
2019-09-30 15:05:56.388 xmltv: /bin/tv_grab_au: grab /bin/tv_grab_au
2019-09-30 15:05:56.390 spawn: Executing "/bin/tv_grab_au"
2019-09-30 15:05:56.396 spawn: Downloading https://i.mjh.nz/au/Sydney/epg.xml...
2019-09-30 15:05:56.562 xmltv: /bin/tv_grab_au: grab took 0 seconds
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: parse took 0 seconds
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: channels tot= 29 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: brands tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: seasons tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: episodes tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: broadcasts tot= 360 new= 9 mod= 0

MJJ

Hi again,

Looking closer, I'm seeing only about two days of EPG on all channels, some have more, but not more than 4 days, so not a 100% satisfactory situation just yet!

MJJ

Yeah, I've been having a look at the XML and it looks OK, but the big difference is that the data for todays
EPG starts at 28/09 [useful!] and only goes out to 02/10, I presume once Matt's stuff has aged out I'll start seeing the new data, however it's
not a great choice - a 7 day EPG with no metadata, or a 2 Day EPG with metadata.

I think I might hassle Matt again & see if he can include the metadata.

Hi Hiro,

Thanks so much again for all this. Especially for offering to reach out to Matt to see if he can help as well. That's a really exciting prospect because I get the impression that his epg would have to have the best prospects for longevity and consistency.

I do have a working Windows Media Centre that uses DVB-T tuners, but I only have two, and that system is so notoriously unreliable. It does, however, have a working two week EPG that comes in via the DVB-T tuners with some metadata. I wonder if there would be a way to generate my own epg.xml file from this information somehow... A scheduled script running under Windows 7 to extract that data on a daily basis from the tuners, and have TVHeadend suck it in and use it for the IPTV services... ?

MJJ

This is Matt's response: "Its in the future plans, but just not yet unfortunately :)"

Pulling the EPG from DVB-T is definitely possible, I do that for the NZ EPG, but the Oz setup is different.

There's a guide here that might be useful: https://www.reddit.com/r/PleX/comments/c580f4/epg_collector_for_tv_guide_australia/
Or you could just download from here: http://www.xmltv.net/

The xmltv.net data seems to also be a couple of days only, and the reddit guide pulls down EIT data.
With EIT you only get a limited EPG, and I did read that AU stopped their MHEG5 EPG [in NZ you get a 7 day EPG via MHEG].
If as you say you have a 2 week EPG via DVB, it's just a matter of figuring out how to extract & decode it.

Hiro Protagonist wrote:

I presume once Matt's stuff has aged out I'll start seeing the new data

Now that the old stuff has aged out, I'm seeing the new stuff OK, overall I'm pretty pleased with it.

When you say you're seeing new stuff, do you mean you're now seeing program detail and episode numbers? Are you now seeing metadata besides the program names, times, dates?

John Jones wrote:

When you say you're seeing new stuff, do you mean you're now seeing program detail and episode numbers? Are you now seeing metadata besides the program names, times, dates?

Indeed. Episode numbers are sporadic, but that's nothing unusual. I'm seeing a weeks worth of EPG for the ABC channels, and a couple of days for most of the rest.

Did anyone end up getting a good resolution for this? My EPG has recently stopped working so looking for a new solution. I am using Matt's IPTV setup in tvheadend

Not yet. Would be wonderful if someone did have a solution they could share with the forum!

John Jones wrote:

Not yet. Would be wonderful if someone did have a solution they could share with the forum!

Hi John,

What I ended up doing was adding xmltv [[https://www.xmltv.net/]] EPG to tvheadend. There are a few steps to this, including a script I have in crontab.

1) Download the EPG from xmltv. Pick which city you want, this is what I have in the script that is on crontab daily:

#!/bin/sh
cd /home/hts/.xmltv/
wget http://xmltv.net/xml_files/Sydney.xml
mv Sydney.xml /home/hts/.xmltv/tv_grab_file.xmltv

It doesn't really where it goes or what the file name is, as long as it matches up with the next script...

2) Create the following script for tvheadend to use:

#!/bin/bash
dflag=
vflag=
cflag=
if (( $# < 1 ))
then
  cat /home/hts/.xmltv/tv_grab_file.xmltv
  exit 0
fi

for arg
do
    delim="" 
    case "$arg" in
    #translate --gnu-long-options to -g (short options)
       --description) args="${args}-d ";;
       --version) args="${args}-v ";;
       --capabilities) args="${args}-c ";;
       #pass through anything else
       *) [[ "${arg:0:1}" == "-" ]] || delim="\"" 
           args="${args}${delim}${arg}${delim} ";;
    esac
done

#Reset the positional parameters to the short options
eval set -- $args

while getopts "dvc" option 
do
    case $option in
        d)  dflag=1;;
        v)  vflag=1;;
        c)  cflag=1;;
        \?) printf "unknown option: -%s\n" $OPTARG
            printf "Usage: %s: [--description] [--version] [--capabilities] \n" $(basename $0)
            exit 2
            ;;
    esac >&2
done

if [ "$dflag" ]
then
   printf "tv_grab_file is a simple grabber that just read the ~/.xmltv/tv_grab_file.xmltv file\n" 
fi
if [ "$vflag" ]
then
   printf "0.1\n" 
fi
if [ "$cflag" ]
then
   printf "baseline\n" 
fi

exit 0

Set the last script up as a EPG module within tvheadend. It will get the crontab xmltv guide into tvheadend.

3) After that I had to go into each channel within tvheadend and select the new EPG for each channel individually. (Config -> Channel / EPG -> Channels -> Edit - EPG Source - Select xx.yourtv.com.au)

You may need to run the internal grabber every now & again throughout the process to populate.

Hopefully this makes sense and/or is useful for someone...

Hi Dr Spalding,

This all looks fantastic, but it's been a number of years since I've had any idea on how to create a script or a cron job in unix.

I've put the scripts into .sh files and put the GetEPG.sh in the /etc/ folder with crontab. I've edited crontab so that it runs that script once a day.

I've put the ProcessEPG.sh file into the /volume1/@appstore/tvheadend-testing/var/epggrab/ folder. Under DSM, this is where the TVH files go and TVH had read/write access to this folder. There's a file in that folder called XMLTV, I might have put it there in some sort of flailing attempt to get something like this working in the past, but clearly unsuccessfully.

Now what? What do I do to tell TVH to run that script? Your instruction to "Set the last script up as a EPG module within tvheadend." means nothing to me. I don't understand how to do this...

Any help appreciated!

MJIII

I fear what is needed here for people like me is a step-by-step instruction manual for setting this up and making it work, which would clearly require a lot of work, but would help lots of people like me. I'd be happy to help someone make something of a guide for people like me, but I really need someone with some unix chops who understands intimately how to get this to work in the Synology OS with considerations for the various different builds of TVHeadend that are installed in the different types of Synology NASes.

Hi John, yes it definitely can be a bit of extra work with the nix based systems. Especially with Synology OS. It adds just that little bit of extra difficulty and eccentricity. If you save the larger (second) script I have above in the /usr/bin folder, then it should pop up after you reboot tvheadend in the Configuration -> Channel/EPG -> EPG Grabber Modules area.

I would however probably change both scripts a little in hindsight for Synology.

1) Create a folder/share where you will store all your scripts and won't be deleted. The reason will come later. For me let's say I have /volume2/downloads/

2) xmltv download script:

#!/bin/sh
cd /volume2/downloads/
wget http://xmltv.net/xml_files/Sydney.xml
mv Sydney.xml /volume2/downloads/tv_grab_file.xmltv

3) tvheadend EPG grabber module script:

#!/bin/bash
dflag=
vflag=
cflag=
if (( $# < 1 ))
then
  cat /volume2/downloads/tv_grab_file.xmltv
  exit 0
fi

for arg
do
    delim="" 
    case "$arg" in
    #translate --gnu-long-options to -g (short options)
       --description) args="${args}-d ";;
       --version) args="${args}-v ";;
       --capabilities) args="${args}-c ";;
       #pass through anything else
       *) [[ "${arg:0:1}" == "-" ]] || delim="\"" 
           args="${args}${delim}${arg}${delim} ";;
    esac
done

#Reset the positional parameters to the short options
eval set -- $args

while getopts "dvc" option 
do
    case $option in
        d)  dflag=1;;
        v)  vflag=1;;
        c)  cflag=1;;
        \?) printf "unknown option: -%s\n" $OPTARG
            printf "Usage: %s: [--description] [--version] [--capabilities] \n" $(basename $0)
            exit 2
            ;;
    esac >&2
done

if [ "$dflag" ]
then
   printf "tv_grab_file is a simple grabber that just read the ~/.xmltv/tv_grab_file.xmltv file\n" 
fi
if [ "$vflag" ]
then
   printf "0.1\n" 
fi
if [ "$cflag" ]
then
   printf "baseline\n" 
fi

exit 0

Instead of crontab, for the first script, I also saved it in the /volume2/downloads/ folder and then just added a daily task in Synology to run that script. (Control Panel -> Task Scheduler -> Create scheduled task -> User defined script).

Regarding the second script, the EPG grabber module one, that needs to be saved in /usr/bin/ for it to be recognised. Once it is saved in that folder, then you reboot tvheadend and it should pop up. However, this is the annoying part. As soon as Synology updates the DSM, it wipes all files in /usr/bin/. So, I save that second script also in /volume2/downloads/ and you can either just copy it across after the DSM update, or also set up a second script to copy it from /volume2/downloads/ to /usr/bin/ on some type of regular time frame.

Hopefully this makes sense. If not, I can start from scratch and type out the whole lot of instructions in a longer version. I may not be until the weekend though... Good luck.

Hi again,

Thanks again.

What do you save that script as? Is there a file name or extension it needs to be for TVH to pick it up when it's placed into /usr/bin/ ? I've saved it as ProcessEPG.sh but TVH doesn't see it.

Thanks,

em

Try tv_grab_au with no extension. If that doesn’t work try chmod 777 tv_grab_au as a last resort. Make sure you restart tvheadend instance between each step

Thank-you Dr Spalding. It works wonderfully. I think the permissions thing was what got it going. chmod 777 and a restart, and it's all there, and working wonderfully! Cheers to you and everyone else who has been helping me with this thing. Very much appreciated!

No worries, glad it worked. Often I change & read so many things I forget what fixed what!

Now THAT I can believe. I swear @Hiro Protagonist sorted all this out for me ages ago, but no way in hell was I going to remember everything we did. He was going to help me again when he had a sec, as I wasn't holding much hope in my ability to do what you explained to do, but I've let him know that he's off the hook now and that thanks to you, I've got it going again. Huge relief. :-)