Tvheadend

Why invalid?

The EPG data for arte coming from the internal OTA grabber is obvious wrong, broken.

Other devices are working like expected and not showing this record which is not a television program.

Because this is not a valid issue report:

Issue reports without debug logs/a stacktrace and instructions on how to reproduce the issue will not be accepted.

Flole Systems wrote:

Because this is not a valid issue report:

Issue reports without debug logs/a stacktrace and instructions on how to reproduce the issue will not be accepted.

To reproduce: Open tvheadend with a DVB-S source (Astra 19.2°) and have a look in the EPG for arte HD

First now the requested 30 seconds ts sample

The logfile for epg, epggrab giving nothing. It seems this comment event is sent not regular like normal EPG.
Keeping it runing and watching if it is showing up somewhen.

The 30 seconds sample ts with 144MiB was curt off: " Request Entity Too Large"

You're probably not looking at the correct debug log, the EIT table contains all the EPG stuff, so that one is relevant here.

The TS probably contains unnecessary data aswell and not just the EPG data, if you remove audio and video data it should be possible to upload it.

Read this old problem with EPG
https://tvheadend.org/issues/5062

and use https://www.digitalekabeltelevisie.nl/dvb_inspector/ to see what is wrong in your case.

I also have more advanced tools to inspect the EIT available, they even find and show errors/specification violations automatically. I can definitely load a mpeg-ts file to verify it's contents.

Thanks for the information.
This will help to get the needed information out of logs an the ts stream.

Unfortunate my time is at the moment very limited and a broken EPG for Arte has not the highest priority.
Might need some time.

I see the same problem in kodi's epg, bit as soon as I actually tune to ARTE the EPG
updates to the correct one.

So probably what happens is that the "ARTE" title is downloaded from some other OTA source
during automated EPG scanning.

Flole Systems wrote:

I also have more advanced tools to inspect the EIT available, they even find and show errors/specification violations automatically. I can definitely load a mpeg-ts file to verify it's contents.

I have now recorded 24 hours Arte.
How can I remove audio and video to make this 118GB file smaller.
My tries with ffmpeg seem being not sucessfull.
The "Stream #0:0[0x12]: Data: epg" changed in the output to "Stream #0:0[0x100]: Data: bin_data ([6][0][0][0] / 0x0006)"
I tried with

ffmpeg -i /recordings/ARTE2023-02-12.ts -map 0 -map -0:v -map -0:a -c copy -copy_unknown /storage/ARTE2023-02-12.ts

Maybe you should only record the EPG stuff instead of trying to filter it later on. I don't think ffmpeg knows anything about the concept of EPG, otherwise the map option should work.

Maybe a simple -map 0:0?

Flole Systems wrote:

Maybe you should only record the EPG stuff instead of trying to filter it later on.

How?

If you want to download the 118G file. I can upload it to my esternal server.

Using SAT>IP/RTP. Simply only subscribe to PIDs 1-20, that should include all the tables but no audio/video/teletext/whatever.

There's probably no point in recording that long, 5 minutes should be more than enough.

I just checked the file and in this file there is no EPG entry named "ARTE". I assume it is broadcasted on another transponder, which is clearly a fault of the channel and not of Tvheadend. When changing to the correct transponder the correct EPG data is parsed, replacing the bad data again. It should even correct the data if you select a different channel on the same transponder. If you switch to a channel on the "bad" transponder it will most likely replace the good data with the bad data again. Someone mentioned on another forum that 10788:V is the "bad" transponder, if you grab EPG data from that one I can verify that.

Flole Systems wrote:

I just checked the file and in this file there is no EPG entry named "ARTE". I assume it is broadcasted on another transponder, which is clearly a fault of the channel and not of Tvheadend. When changing to the correct transponder the correct EPG data is parsed, replacing the bad data again. It should even correct the data if you select a different channel on the same transponder. If you switch to a channel on the "bad" transponder it will most likely replace the good data with the bad data again. Someone mentioned on another forum that 10788:V is the "bad" transponder, if you grab EPG data from that one I can verify that.

This is the correct transponer.<
The EPG data is for
svcId | EIT avail | provider | service name
10301 | p/f sched | ARD - Das Erste HD
10302 | p/f sched | ARD - arte HD
10303 | p/f sched | ARD - SWR BW HD
10304 | p/f sched | ARD - SWR RP HD

At time of recording the "El canal de televisiðn cultural." did not show up.
But I continued the recording.
Now the file is 2.4GB.

The assumption is that there is a bogus EIT entry on that other transponder which overwrites the EIT on this transponder.

You can also periodically restart the recording if you don't see the bad entry, that will reduce the file size significantly.

You can also periodically restart the recording if you don't see the bad entry, that will reduce the file size significantly.

I can not find this record.
I subscribed to the PIDs 0-20 with vlc and running wireshark for the case that vlc is dropping or correcting this record.

Could not find this "El canal de televisiðn cultural." in the wireshark records. No idea how it is encoded.
Wireshark was running from Sunday evening until now.

In the tvheadend EPG it showed up again for 21. March starting at 6:00 with duration 24h and the same for 22. March.
Today it disappeared again in tvheadend.

This problem was already existing in 2017.

This record is from a different transponder.
Seems tvheadend is trusting channel and transponder IDs like vdr before it was patched 2018 and is merging EPG data from different transponders.

https://www.vdr-portal.de/forum/index.php?thread/131650-arte-epg-verhindert-suchtimeraufnahmen/&postID=1303361#post1303361

A solution here:

https://www.vdr-portal.de/forum/index.php?thread/131650-arte-epg-verhindert-suchtimeraufnahmen/&postID=1303292#post1303292

If you don't watch anything on that transponder you can just disable it.

Otherwise feel free to contact Arte again and ask them why they are transmitting that stuff on a different transponder, there's really no point in doing that and I suggest that they stop doing that ASAP.

Flole Systems wrote:

If you don't watch anything on that transponder you can just disable it.

If this would work....

I went to config - Muxes and set 10788V to Disabled. The "Enabled" field and the "EPG scan" field.
I stopped tvheadend, deleted /home/hts/.hts/tvheadend/epgdb.v3 and started tvheadend again.
After a few seconds: "El canal....."

Otherwise feel free to contact Arte again and ask them why they are transmitting that stuff on a different transponder, there's really no point in doing that and I suggest that they stop doing that ASAP.

This is not a problem with Arte.
This transponder is not Arte. On this transponder are spanish free and pay-tv stations.

Andreas Tauscher wrote:

Flole Systems wrote:

If you don't watch anything on that transponder you can just disable it.

If this would work....

Sorry. Also 11097:V (also Spanish) is transmitting this crap.

Andreas Tauscher wrote:

This is not a problem with Arte.
This transponder is not Arte. On this transponder are spanish free and pay-tv stations.

Well you can also try to contact the operator of the transponder. Usually the channel operator has a better way of reaching people who understand the issue and can fix it though.

And 11038:V, 10979:V, 11453.5:V, 10817.5:V, 11435.5:V, 10729:V, 11258.5:V, 11685.5:V, 10906:V, 10935.5:V, 11156:V, 11126.5:V, 10758.5:V, 10876.5:V, 11317.5:V

Not sure which it is exactly but at least one of this is sending with the Transport stream ID of Arte in the EIT
Seems is messed up by the spanish pay-tv M+

Disabling all this muxes can be only a temporary workaround and not a solution.

IMHO it would be a good idea to handle this data like vdr doing it: Do not trust that the TSI is set correct. Verify that the actual transponder and the data in the SIT/EIT is matching.

Flole Systems wrote:

Andreas Tauscher wrote:

This is not a problem with Arte.
This transponder is not Arte. On this transponder are spanish free and pay-tv stations.

Well you can also try to contact the operator of the transponder. Usually the channel operator has a better way of reaching people who understand the issue and can fix it though.

Are you kidding?

Asking such organisations about techical issues?
This is like asking microsoft support when they fucked up again their mailservers.

Play with TSDuck, this problem not exist in my Neutrino box.
This is some glitch in TVH code.

saen acro wrote:

This is some glitch in TVH code.

No, Tvheadend follows exactly the specifications that clearly says

Each service_id shall be unique within each original_network_id. Therefore the combination of service_id and original_network_id uniquely identifies a service.

Each unique instance of a service can be uniquely referenced through the path original_network_id/transport_stream_id/service_id (see EN 300 468 [i.1])

If some operator is violating this that's not a bug in Tvheadend. If you want to implement such a feature it could be tracked as a feature request.

You can achieve what you want by commenting out line 1011 in src/epggrab/modules/eit.c. However this will break EPG on 29.2E, where the data is carried on a separate dedicated transponder.

Dave Pickles wrote:

You can achieve what you want by commenting out line 1011 in src/epggrab/modules/eit.c. However this will break EPG on 29.2E, where the data is carried on a separate dedicated transponder.

Also no solution.
Only a woraround and not every user is able to build TVH from source.

If some operator is violating this that's not a bug in Tvheadend. If you want to implement such a feature it could be tracked as a feature request.

But only TVH having this problem.
From about 10 different devices I tested (even a cheap crappy receiver I bought ~10 years ago for less than 30€) not a single one having this problem.
Only TVH is providing broken EPG data.

There is also one product from company M. from Redmond breaking the SMTP standard.
This is why mail servers reply on a EHLO with one double line:

250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN

It is not a problem of postfix, exim and others. M$ is breaking the SMTP and not able/willing to fix this, so others fixing it.

Flole Systems wrote:

No, Tvheadend follows exactly the specifications that clearly says

https://github.com/tvheadend/tvheadend/blob/ef13a600afb35905ddfa84447073c016d320c185/src/input/mpegts/satip/satip.c#L586-L637

Exact exact or little bit not exact

saen acro wrote:

Flole Systems wrote:

No, Tvheadend follows exactly the specifications that clearly says

https://github.com/tvheadend/tvheadend/blob/ef13a600afb35905ddfa84447073c016d320c185/src/input/mpegts/satip/satip.c#L586-L637

Exact exact or little bit not exact

All that isn't executed unless workarounds are enabled, literally the first line in that function says

if(sd->sd_disable_workarounds)
    return;

Anyways, it's a feature request and not a bug, so the direction is clear: Either someone here wants to work on it (not entirely sure how a solution should look like if other satellites are carrying the EPG data on separate transponders, but I'll happily take a look at the PR once it's ready) or the status is correct as it's not a bug. Or you could still ask the operator or the channel to fix their stuff rather than working around it, that might be easier for you to do than changing Tvheadend itself.

Violations of the dvb standards are frequent on satellite: muxes lie about frequencies, satellite positions,
confuse east with west, and even contain contradictory data. That means workarounds are typically needed.
Standards are only useful when respected.

I have worked on code like this in the past, and perhaps the following info could be useful:

Some possible heuristics to get rid of crappy epg records are:

-if no epg data for the timeslot/service exists yet, accept any data from any source. In this case,
the spanish crap would be stored only if there is no valid epg yet.

Reasoning: better some epg than no epg at all, even if the result in this specific bug would not be
be helpful, it can help for other services and does no harm.

-if epg data exists, only overwrite it if the new epg data has higher priority than the old one
With a proper defintion of priority, the spanish crap will be overwritten when tuning to the arte transponder
and will not be overwritten later, when tuning to the spanish provider.

priority can be defined in multiple ways, which can be combined. None is perfect.

examples:
1. priority is higher if the mux providing the epg data also contains the actual service. In this case,
if spanish crap is present, epg grabbing of the mux containing arte will fix it and it will remain fixed.

2. epg data contains language encodings and I have seen muxes that broadcast data in multiple
languages simulatenously. So it would be useful as a feature anyway to ask the user to provide a list of language priories as configuration. E.g., 1st choice german, 2nd french ... spanish.

With such a preference, if data is received for a higher priority language it will overwrite existing data, otherwise not. This example would also fix the specific problem
of this bug by prefering french over spanish. Not a solution for everyone of course + language codes
can also be wrong.

3. priority can also be defined based on the length of "story". Records with longer length have priority.
Reasoning: in many cases this leads to more useful information for the user.
This too would probably fix the specific bug in question.

4. priority can be based on a user setting on the mux.

5. or on the duration of the program. Typically placeholders are much longer (many hours) than real epg.
So if new data overlaps with old data, prefer the shorter one. The details may require some work.
This might work in the case of the arte bug

6. if priority of old and new data is equal, and the records differ, prefer newer over older information.

All of the above solutions will make no difference at all if there is no conflicting data on satellite.
So they should be safe. Some are easier to implement than others, and they can be combined to some degree.

They are all heuristics and given the amount of crap broadcasters are making up,
they will not be full solutions and can still fail, but they can only fail on data for which
dvb standards are violated (=do no harm).

As all of this is a pain to debug, it may also be useful (as debugging option) to include the network_id and
ts_id of the mux which provided the data, along with the date when it was captured. This can be easier to check than in debug logs (but debug logs are of course still valuable. One does not replace the other)

PS. Why do have to annotate so many buses, just to log in on this forum? It is quite annoying.

Good luck finding someone who implements all that, but that seems like a good way of doing that.