Project

General

Profile

XMLTV - Schedules Direct JSON service backend migration 2022-05-20

Added by Gary Buhrmaster over 2 years ago

The following was posted on the Schedules Direct forum at:
https://forums.schedulesdirect.org/viewtopic.php?f=18&t=3201


Notice: JSON service migration from PHP to Python

Schedules Direct will be migrating our backend service from PHP to
Python starting on 2022-05-20.

What should you expect?

The Python provided API has been created to be protocol compatible, so
it's likely that you won't need to do anything if your grabber has
implemented the published API.
The new service is provided by AWS API Gateway, so the issues you may
have experienced if you were updating your schedule at your local
midnight time should now be addressed.
The methods used to generate data on our backend servers have been
updated, and there are a number of fields in the new JSON which will
cause the MD5 values to change, so your application will refresh its
local data. This is known as a cache invalidation - expect that there
will be a large amount of data downloaded.

What changes will you see?

The schedules on the backend will be refreshed more often; we have
migrated to a new data delivery mechanism from our upstream provider.
You should see 14 to 18 days of schedule data.
Images will be updated more often.
Lineup updates will be updated more often.
Schedules Direct will send lineup deletion notifications to your
registered email address 30 days before a lineup is deleted, in order
to give you time to migrate. Lineups are deleted by upstream providers
if they're consolidating headends, or ending an offering - this
typically occurs if a provider is migrating from analog (usually, the
-DEFAULT lineup) to digital-only (-X)

What should you do if you have issues after the migration?

Check to see if you're running the latest version of your application
/ grabber. Developers were provided early access to the new API, and a
number of issues were resolved.
Check the support forum for your application and engage the developer;
they'll be the ones that can help you troubleshoot the application and
the migration.
If you're still stuck, your developer will tell you to open a ticket
with Schedules Direct. Please open a lineup support ticket on the
SchedulesDirect webpage https://www.schedulesdirect.org/lineupsupport
and indicate that you're having an issue, and that the developer of
your application has directed you to open a ticket.


Replies (11)

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Rachel Greenham over 2 years ago

... and sure enough suddenly I don't seem to be getting any new programmes through it.

This is logged when I try to force-run the internal grabbers:

2022-05-21 16:53:19.025 xmltv: /usr/bin/tv_grab_zz_sdjson: grab /usr/bin/tv_grab_zz_sdjson
2022-05-21 16:53:19.027 spawn: Executing "/usr/bin/tv_grab_zz_sdjson" 
2022-05-21 16:53:19.344 spawn: Initializing...
2022-05-21 16:53:24.970 spawn: ** POST https://json.schedulesdirect.org/20141201/token ==> 100%200 OK (5s)
2022-05-21 16:53:30.039 spawn: ** GET https://json.schedulesdirect.org/20141201/status ==>  92%100%200 OK (6s)
2022-05-21 16:53:30.039 spawn: Updating lineups...
2022-05-21 16:53:36.874 spawn: ** GET https://json.schedulesdirect.org/20141201/lineups/GBR-1000071-DEFAULT ==>   0%  1%  3%  5%  7%  8%  9% 11% 12% 13% 15% 16% 18% 19% 21% 22% 24% 25% 26% 27% 29% 30% 32% 33% 34% 36% 37% 39% 40% 42% 43% 45% 46% 48% 49% 51% 52% 54% 55% 57% 58% 60% 61% 62% 64% 65% 67% 68% 70% 71% 73% 74% 76% 77% 79% 80% 82% 83% 85% 86% 88% 89% 90% 92% 93% 95% 96% 98% 99%100%200 OK (6s)
2022-05-21 16:53:43.930 spawn: ** GET https://json.schedulesdirect.org/20141201/lineups/GBR-1000208-DEFAULT ==>   0%  1%  2%  3%  4%  5%  6%  7%  8%  9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100%200 OK (7s)
2022-05-21 16:53:50.803 spawn: ** GET https://json.schedulesdirect.org/20141201/lineups/GBR-1002100-DEFAULT ==>   0%  1%  2%  3%  4%  5%  6%  7%  8%  9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100%200 OK (7s)
2022-05-21 16:53:50.809 spawn: Indexing channels...
2022-05-21 16:53:50.810 spawn: Channel ID 68053 in the current configuration is not found in any enabled lineup.
2022-05-21 16:53:50.810 spawn: Updating schedules...
2022-05-21 16:53:55.223 spawn: ** POST https://json.schedulesdirect.org/20141201/schedules/md5 ==>   1%  3%  6%  9% 11% 14% 15% 18% 21% 23% 26% 27% 28% 31% 32% 35% 38% 39% 42% 45% 48% 51% 54% 55% 58% 61% 64% 67% 70% 73% 76% 79% 82% 85% 88% 91% 94% 97%100%200 OK (5s)
2022-05-21 16:54:00.102 spawn: ** POST https://json.schedulesdirect.org/20141201/schedules ==> 303 See Other (5s)
2022-05-21 16:54:00.103 spawn: Use of uninitialized value in pattern match (m//) at /usr/bin/tv_grab_zz_sdjson line 283.
2022-05-21 16:54:00.129 xmltv: /usr/bin/tv_grab_zz_sdjson: no output detected
2022-05-21 16:54:00.129 xmltv: /usr/bin/tv_grab_zz_sdjson: grab returned no data

TVHeadend version is 4.3-2010~g3b1d7a928 on Ubuntu 22.04
xmltv version is xmltv-util 0.6.1-1 from the Ubuntu repos.

There doesn't seem to be a straightforward way to upgrade to the latest xmltv; but also looking at the xmltv changelogs it doesn't look like they have an update for this anyway. There are some sdjson_sqlite updates mentioned but tvheadend doesn't appear to be using that...

I note that "303 See Other" response above. I also note you have a PR open that looks like it might be relevant... https://github.com/XMLTV/xmltv/pull/180

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Gary Buhrmaster over 2 years ago

Yes, that PR might fix the issue with that grabber not following the redirect (the 303). You might want to apply it to your local copy of the grabber and if it works for you provide feedback to the XMLTV project so that the fix will be merged sooner rather than later.

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Rachel Greenham over 2 years ago

OK, that appears to have worked. TVHeadend's programme list looks to be updated. Kodi, however, is crashing when I try to view the EPG, but I have a feeling this is something that broke before I tried this upgrade, when I first tried just clearing the EPG data.

... ah no wait, it's full of junk data. Schedules are a complete mess.

Also btw attached the logs as output into the tvheadend log. I didn't see this when running from the commandline. There are errors being reported relating to uninitialized values.

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Gary Buhrmaster over 2 years ago

That grabber appears to have insufficient data validation/checking (and the cache sometimes contains "challenged" data; removing the grabber cache may make some of the errors disappear for at least a while).

You might want to open issues against that grabber upstream to get the attention of the original developer, or move to the alternative sdjson grabber which tends to have more of the data validation checks already in place (but moving to the alternative grabber has a different set of setup/maintenance considerations).

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Rachel Greenham over 2 years ago

To fill out a little:

The fetching process looked like it worked, it appeared to be fetching data OK, following redirects etc. and it loaded into tvheadend. But the programmes themselves were hopelessly jumbled.

Capturing the XML output from the grabber and looking at it, the data is wrong there. The programmes I'm expecting to see are present, but they're assigned to all the wrong channels, apparently at random (ie: programmes for one channel are not being reliably reassigned to a single other channel, but all over the place. I'm not sure I'm reading the timestamps right either. The first part of the programme start value is clearly a date, but the timestamp part after that is a bit more opaque. Either that or just wrong. (it doesn't seem to be seconds into the day for instance, the values are still wrong.) eg: A programme I knew to be starting at 20:00 today (May 21), with a 5-minutes-before start time configured, has a programme start time of "20220526185500 +0000" The first part's a date, what's the second part? Could do with some ISO8601 love here I reckon! And anyway it's set to completely the wrong channel. (Supposed to be BBC Four HD, is Children's ITV (UK btw)) The other programmes I wanted were supposed to be on the same channel but the data I got back put them on Al Jazeera and CBBC HD respectively...

I suspect your code is fine and Schedules Direct's back-end upgrade has gone horribly wrong...

I then went through a bunch of adventures trying to fix things, I believe hampered by the actual DVB-S2 receiver needing a reset, so all my recordings today were missed... I'm pretty much on the edge of giving up on the whole business, as there's not much I even use it for these days (those missed programmes? got 'em already, they would just be fresher versions) Currently it's trying again to do an epg ota fetch and a uk-tvguide fetch, not tbh with a lot of success so far (there being reasons why I'd ended up with Schedules Direct already).

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Rachel Greenham over 2 years ago

your post arrived while I was typing. I did try removing the cache, to little avail. I'm not sure what you mean by the alternative grabber. The sqlite one, or what they describe as the legacy SD-DD one?

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Gary Buhrmaster over 2 years ago

I'm not sure what you mean by the alternative grabber. The sqlite one, or what they describe as the legacy SD-DD one?

I was talking about the sdjson_sqlite variant. TTBOMK the legacy SD-DD one does not support countries other than the US.

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Gary Buhrmaster over 2 years ago

... and Schedules Direct's back-end upgrade has gone horribly wrong...

I do not know if anyone tested the non-US guide data. You may need to open a lineup ticket with SD.

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Rachel Greenham over 2 years ago

ok, there isn't an internal xmltv grabber for the sqlite variant. you can't change the paths on the ones that do exist. (and i seem to have two of each one btw - this is why i tried to start from scratch earlier, but the wizard failed me). and i can't find any information on what to do about setting up the external xmltv thing, with the socket...

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Rachel Greenham over 2 years ago

(I found it. The sqlite variant, staring me in the face, but not adjacent to the non-sqlite version which threw me. It seems to be working. I have a plausible-looking EPG now anyway.)

RE: XMLTV - Schedules Direct JSON service backend migration 2022-05-20 - Added by Gary Buhrmaster over 2 years ago

Thee sqlite variant .... It seems to be working. I have a plausible-looking EPG now anyway.

That is at least a mitigation until/unless someone spends some time understanding the issues with the zz_sdjson grabber.

    (1-11/11)