Feature #3190
closed
Implementing over-the-air EPG grabber for stateside ATSC broadcast (PSIP)
Added by G E over 9 years ago. Updated over 9 years ago.
0%
Description
Hello,
I just finished with a fellow from the #htc IRC channel, username lsd he was remote into my system here and put together a file from the local channel Muxes for parsing info to hopefully implement EIT for hts.
EIT would be a great addition to TVHeadend here in the states and would be much easier to setup then using XML grabbers, setting up scripts and cron jobs to keep it updated. I have used Myth and it sees the local channel EIT information; but Myth is, in my opinion, too much to have to go through to set up to use with Kodi or even it's own frontend.
I hope the attached file helps. If not contact me and I will put together another file.
Thank you
gee
Updated by luuthe . over 9 years ago
Another full mux capture:
https://drive.google.com/open?id=0BzvRZDja5fbydXE3Rkh5YWozeEU
Updated by G E over 9 years ago
Well so far most sites are broken. :(
This one better work.
https://www.dropbox.com/s/rt2m4eivli8pjdb/test.028MHz%2520%252F%2520Tampa%2520Bay?dl=0
Updated by luuthe . over 9 years ago
This might be a good place to start, but I've yet to experiment with it.
Updated by B C over 9 years ago
luuthe . wrote:
This might be a good place to start, but I've yet to experiment with it.
oh great, so no need to actually reinvent the wheel. @Jaroslav Kysela: take a look at the psip module
Updated by G E over 9 years ago
One more for good measure. This is from a different local channel.
https://www.dropbox.com/s/4ct61li5jg05463/wfla-test.ts?dl=0
Updated by Jaroslav Kysela over 9 years ago
I sent an e-mail to Lauri to sign CLA and create the pull request so we can adapt his code.
Updated by Lauri Mylläri over 9 years ago
Jaroslav Kysela wrote:
I sent an e-mail to Lauri to sign CLA and create the pull request so we can adapt his code.
After replying to you I found this issue, so I'm commenting here directly.
I created a pull request at https://github.com/tvheadend/tvheadend/pull/734
Can you see from this if I already signed the CLA - I think I did when I started working on this.
Updated by G E over 9 years ago
This question will show my lack of knowledge in this.
"_dvb_support: change GPS UTC offset to one less …_"
Is this syncing the time up and if so could you use a NTP server for time sync or the broadcast station's timecode which should be in the data stream.
Just curious how the time syncing is done.
Updated by G E over 9 years ago
Also, maybe look at myth's code. The EIT information parsed and then displayed within Kodi seems to be on time and accurate.
Sorry if I am being a pain. Just throwing ideas up in the air here.
Updated by B C over 9 years ago
you can let tvh sync your time from the networks if you have a computer without internet, but you must not.
Updated by B C over 9 years ago
G E wrote:
Also, maybe look at myth's code. The EIT information parsed and then displayed within Kodi seems to be on time and accurate.
Sorry if I am being a pain. Just throwing ideas up in the air here.
G E, it seems Lauri has all you need, just be a little bit patient for getting his PR merged and then we see if anything needs to be fixed. Don't worry about accuracy, that's the smallest problem.
Updated by G E over 9 years ago
A Broadcasters’ Guide to PSIP
http://www.atscforum.org/documents/PSIP_guide.pdf
Digital Video Broadcasting (DVB);
Specification for Service Information (SI) in DVB systems
http://www.etsi.org/deliver/etsi_en/300400_300499/300468/01.03.01_60/en_300468v010301p.pdf
Broadcast Engineer's Reference Book
https://books.google.com/books?id=8YDOAwAAQBAJ&pg=PA339&lpg=PA339&dq=How+EIT+is+used+in+broadcast&source=bl&ots=CXII4JJwCA&sig=Y7hm8ePVI2pjdbxaqJHXR-x2sz8&hl=en&sa=X&ved=0CD8Q6AEwBmoVChMIxOuvzLfUyAIVAakeCh3efwJD#v=onepage&q=How%20EIT%20is%20used%20in%20broadcast&f=false
Updated by G E over 9 years ago
Oh I'm in no rush here just inquisitive on some of this and also just pulled some references that I used when I used to work in broadcast.
Updated by Jaroslav Kysela over 9 years ago
- Subject changed from Mux Information for implementing EIT for stateside ATSC broadcast EPG to Implementing over-the-air EPG for stateside ATSC broadcast (PSIP)
- Target version set to 4.2
Updated by Jaroslav Kysela over 9 years ago
- Subject changed from Implementing over-the-air EPG for stateside ATSC broadcast (PSIP) to Implementing over-the-air EPG grabber for stateside ATSC broadcast (PSIP)
Updated by Jaroslav Kysela over 9 years ago
Discussion on forum: https://tvheadend.org/boards/12/topics/6573
Updated by Jaroslav Kysela over 9 years ago
I merged with some fixes the code from Lauri, so we need testers now :-) v4.1-724-gc08bf56
Updated by luuthe . over 9 years ago
Tested latest master (rev-list#7104).
The EPG is populated successfully, times are correct along with show names and descriptions.
Great work Lauri and perexg!
Updated by G E over 9 years ago
Thank you very much Gentleman on a job well done.
This is the fastest that I have ever seen a group of Devs go to work on a request for a upgrade or fix. Everything is looking great so far.
I have just one small suggestion; and this is probably due to my lack understanding of the program.
In the EPG panel view, second column over, "Progress", the title can be a bit confusing, well at least to me. I thought at first it was the progress of the download of EIT information and I discovered that is the percentage of the program time past.
My suggestion, the title of the second column be changed to Time, Time Past or change the count to show the time that is left and call it Time Left. If this can be done also replace the "%" with "m" with the math to show actual minutes.
Just a thought.
Again thanks for getting the OTA/EIT working.
Updated by Chris Lee over 9 years ago
Here is 7 ATSC recordings from all 7 channels in Edmonton Alberta Canada.
https://drive.google.com/open?id=0B7PjDRC3SaJxZEh1SVRZZkVqZDg
I'll install tvheaded and test what I can as well.
UDL
Updated by Mark Clarkstone over 9 years ago
Chris Lee wrote:
Here is 7 ATSC recordings from all 7 channels in Edmonton Alberta Canada.
https://drive.google.com/open?id=0B7PjDRC3SaJxZEh1SVRZZkVqZDg
I'll install tvheaded and test what I can as well.
UDL
Tried two of those TS files and they both failed to play in VLC. It reports..
core debug: processing request item: atsc_2.ts, node: Playlist, skip: 0 core debug: resyncing on atsc_2.ts core debug: atsc_2.ts is at 0 core debug: starting playback of the new playlist item core debug: resyncing on atsc_2.ts core debug: atsc_2.ts is at 0 core debug: creating new input thread core debug: Creating an input for 'atsc_2.ts' core debug: requesting art for atsc_2.ts core debug: using timeshift granularity of 50 MiB, in path 'C:\Users\Mark\AppData\Local\Temp' core debug: `file:///D:/Downloads/atsc_2.ts' gives access `file' demux `' path `/D:/Downloads/atsc_2.ts' core debug: specified demux `any' core debug: creating demux: access='file' demux='any' location='/D:/Downloads/atsc_2.ts' file='D:\Downloads\atsc_2.ts' core debug: looking for access_demux module matching "file": 12 candidates core debug: looking for meta fetcher module matching "any": 1 candidates core debug: no access_demux modules matched core debug: creating access 'file' location='/D:/Downloads/atsc_2.ts', path='D:\Downloads\atsc_2.ts' core debug: looking for access module matching "file": 21 candidates filesystem debug: opening file `D:\Downloads\atsc_2.ts' lua debug: Trying Lua scripts in C:\Users\Mark\AppData\Roaming\vlc\lua\meta\fetcher core debug: using access module "filesystem" core debug: Using stream method for AStream* core debug: starting pre-buffering lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\fetcher core debug: received first data after 0 ms core debug: pre-buffering done 1024 bytes in 0s - 1000000 KiB/s core debug: looking for stream_filter module matching "any": 6 candidates core debug: no stream_filter modules matched core debug: looking for stream_filter module matching "record": 6 candidates core debug: using stream_filter module "record" core debug: creating demux: access='file' demux='any' location='/D:/Downloads/atsc_2.ts' file='D:\Downloads\atsc_2.ts' core debug: looking for demux module matching "any": 66 candidates lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\fetcher\tvrage.luac lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\fetcher\tvrage.luac core debug: no meta fetcher modules matched core debug: searching art for atsc_2.ts core debug: looking for art finder module matching "any": 2 candidates lua debug: Trying Lua scripts in C:\Users\Mark\AppData\Roaming\vlc\lua\meta\art lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\art lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\00_musicbrainz.luac lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\00_musicbrainz.luac lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\01_googleimage.luac lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\01_googleimage.luac lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\02_frenchtv.luac lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\02_frenchtv.luac lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\03_lastfm.luac lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\03_lastfm.luac core debug: no art finder modules matched core debug: art not found for atsc_2.ts qt4 debug: IM: Setting an input ts debug: eof ? ts debug: Force Seek Per Percent: Seeking failed at 10%. ts debug: Force Seek Per Percent: PCR's not found, ts debug: pid[49] unknown ts debug: pid[65] unknown ts debug: pid[48] unknown ts debug: pid[68] unknown ts debug: pid[7424] unknown ts debug: pid[7680] unknown ts debug: pid[8187] unknown ts debug: PATCallBack called ts debug: new PAT ts_id=16429 version=9 current_next=1 ts debug: * number=1 pid=48 ts debug: PMTCallBack called ts debug: new PMT program number=1 version=9 pid_pcr=49 ts debug: * es pid=49 type=2 fcc=mpgv core debug: selecting program id=1 ts debug: * es pid=52 type=129 dr->i_tag=0xa ts debug: * es pid=52 type=129 dr->i_tag=0x81 ts debug: found language: eng ts debug: * es pid=52 type=129 fcc=a52 ts debug: * es pid=53 type=129 dr->i_tag=0x81 ts debug: * es pid=53 type=129 dr->i_tag=0xa ts debug: found language: enm ts debug: * es pid=53 type=129 fcc=a52 core debug: using demux module "ts" core debug: looking for a subtitle file in D:\Downloads\ core debug: looking for decoder module matching "any": 43 candidates avcodec debug: CPU flags: 0x0107d3db avcodec debug: trying to use direct rendering avcodec debug: allowing 4 thread(s) for decoding avcodec debug: avcodec codec (MPEG-1/2 Video) started core debug: using decoder module "avcodec" core debug: looking for packetizer module matching "any": 23 candidates core debug: using packetizer module "packetizer_mpegvideo" core debug: looking for decoder module matching "any": 43 candidates core debug: using decoder module "a52" ts debug: DEMUX_SET_GROUP 0 0000000000000000 core debug: looking for meta reader module matching "any": 2 candidates lua debug: Trying Lua scripts in C:\Users\Mark\AppData\Roaming\vlc\lua\meta\reader lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\reader lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\reader\filename.luac core debug: no meta reader modules matched core debug: `file:///D:/Downloads/atsc_2.ts' successfully opened ts debug: pid[7425] unknown ts debug: pid[7427] unknown ts debug: pid[7682] unknown ts debug: pid[7683] unknown ts debug: pid[7681] unknown ts debug: pid[7426] unknown ts debug: eof ? core debug: EOF reached core debug: finished input core debug: removing module "avcodec" avcodec debug: ffmpeg codec (MPEG-1/2 Video) stopped core debug: killing decoder fourcc `mpgv', 0 PES in FIFO core debug: removing module "packetizer_mpegvideo" core debug: removing module "a52" core debug: killing decoder fourcc `a52 ', 0 PES in FIFO core debug: removing module "ts" ts debug: pid list: ts debug: - pid[0] seen ts debug: - pid[48] seen ts debug: - pid[49] seen core debug: Program doesn't contain anymore ES ts debug: - pid[65] seen ts debug: - pid[68] seen ts debug: - pid[7424] seen ts debug: - pid[7425] seen ts debug: - pid[7426] seen ts debug: - pid[7427] seen ts debug: - pid[7680] seen ts debug: - pid[7681] seen ts debug: - pid[7682] seen ts debug: - pid[7683] seen ts debug: - pid[8187] seen ts debug: - pid[8191] seen core debug: removing module "record" core debug: removing module "filesystem" core debug: dead input core debug: changing item without a request (current 0/1) core debug: nothing to play qt4 debug: IM: Deleting the input
Updated by Ron L over 9 years ago
I only looked at atsc_1.ts but I think perhaps only the psip PID was recorded, not the raw mux with video and audio.
Updated by G E over 9 years ago
I'm not sure if this is an issue with EPG/EIT or due to our time change and the local stations.
We just switched our time here in the US one hr back DST (Daylight Savings Time). The system clock here is on the right time.
The EPG is apparently broken again according to the browser interface. It is only getting some of the stations information as it did earlier on as the EIT was being worked on. It was working up to a few days ago Wednesday,but this evening is the first time since the time change that I had a chance to check on it. I also just took an update for TVheadend earlier today.
v. 4.1-812~ge85a3c7
I did restart hts also tried clearing the information so hts can repopulate the information.
Updated by Jaroslav Kysela over 9 years ago
This issue was fixed in v4.1-822-g18d993f . Update.
Updated by G E over 9 years ago
I just took a upgrade from htc v4.1-822-g18d993f to v4.1-830~gaeb20ca
Still no go.
Damn DST.
Updated by G E over 9 years ago
Scratch last... It took a few minutes over an hour to populate all the channels.
Kodi EPG is now populated.
Thanks again
Updated by Jaroslav Kysela over 9 years ago
Just a little note: I fixed many things in v4.1-894-ga27b0af regarding the psip parser - events should be fetched much faster and the description text should be fetched, too (it didn't work for many events). v4.1-894-ga27b0af
Report issues to bug #3272 .
Updated by Jaroslav Kysela over 9 years ago
Just a little implemntation note: Only 5 PIDs are used to scan ATSC EIT/ETT tables now (the subscribed EIT/ETT tables are changed in a loop). It might fix some issues with tuners which are not capable to do hardware filtering of many PIDs.