Tvheadend

Another idea:

I have also tried in the channel tab the column "Use EPG of" to set. For example: For the channel "arte" use EPG of "arte" (of itself). No luck: the EPG of "arte" is then empty.

Obviously, the stray entry is in tvheadend WebUI and xmltv export.

It seems like neither Kodi18 nor VLC do show them - they seem to interpret the data properly and filter the overlapping entry, whereas Yatse does not.

Can anybody advise what tvh modules to debug or trace to find out where this entry is from?

There also was some tool to grab EPG right from a running channel, right now don't remember the name - must go searching my bookmarks ._)

/HXZ

da h4xX0rz1sT wrote:

There also was some tool to grab EPG right from a running channel, right now don't remember the name - must go searching my bookmarks ._)

Found it: https://github.com/hiroshiyui/epgrab

HTH,
/HXZ

Unfortunately, I cannot use the above "epgrab" tool - it only works with local DVB devices, and I have a SAT>IP server for DVB-S.

Anyone knows a tool that can grab raw EPG data, also from SAT>IP? Or a way to get the source of an EPG entry (i.e. the channel that broadcasted it) in tvheadend?

@peter peter -----: can you check with the tool I posted above if that stray entry really comes from Movistar? In VLC you won't be able to see it.

/HXZ

I'm sorry, I'm also using only SelfSat SAT>IP IP36 and IP21.

@da h4xX0rz1sT:

The informations are surely from Moviestar+: I've taken the spanish EPG-informations from tvheadend from "BBC Eorld Europe HD" and had googled it: "Canal de información 24 horas, producido por la BBC que emite en inglés en más de 200 países alrededor del mundo."

-> BINGO! Moviestar+: [[http://www.movistarplus.es/canal/bbc-world?id=BBC]]

Peter ----- wrote:

The informations are surely from Moviestar+: I've taken the spanish EPG-informations from tvheadend from "BBC Eorld Europe HD" and had googled it: "Canal de información 24 horas, producido por la BBC que emite en inglés en más de 200 países alrededor del mundo."

I tried the same with the arte entry, but without success. But I think we can be reasonably sure that messy entry really comes from Movistar. @peter peter -----: Can you see EPG on your SAT>IP boxes?

Then the second question comes up: what logic could TVh use to filter these entries?

• hard to detemine which EPG data is "authoritative" for which channel
• hardcoded filters are probably not the way to go
• we could have some language preference order, e.g. ger,eng,spa to only spit out one preferred language
• TVh could also detect it is an overlapping entry

Any other ideas?

HTH,
/HXZ

Is in Settings > General selected preferred languages?

You're right, that language preference setting is already there! Forgot about that!

It says "German, English" but nothing of "Spanish". I also tried to change it to "German, English, Spanish" but the Spanish entry for "Sky News", "BBC World HD" still "wins" and replaces other languages.

Seems this setting is not properly considered when serving EPG.

/HXZ

Export epg and see what languages is available for services.

The output from the export is exactly the same I also see in TVh WebUI.

For "BBC World HD" I only have the Spanish entry:

<programme start="20190319060000 +0100" stop="20190320060000 +0100" channel="4c78c12726f12e1176e953c256963463">
    <title lang="spa">BBC WORLD HD</title>
    <desc lang="spa">Canal de informaciðn 24 horas, producido por la BBC que emite en inglØs en mÆs de 200 paTses alrededor del mundo. </desc>
</programme>

For "Sky News Intl", I get both Spanish and English:

<programme start="20190319060000 +0100" stop="20190320060000 +0100" channel="5e44629c727d4808f13922be9cf52065">
    <title lang="spa">SKY NEWS</title><desc lang="spa">Canal de noticias británico que se caracteriza por la inmediatez de su cobertura y su espíritu fresco e innovador.   </desc>
</programme>
<programme start="20190323063000 +0100" stop="20190323070000 +0100" channel="5e44629c727d4808f13922be9cf52065">
    <title lang="eng">Press Preview with Anna Jones</title>
    <sub-title lang="eng">Anna Jones and guests with a review of tomorrow's newspapers and passionate debate about the stories they cover</sub-title>
    <category lang="en">News / Current affairs : News magazine</category>
</programme>
<programme start="20190324063000 +0100" stop="20190324070000 +0100" channel="5e44629c727d4808f13922be9cf52065">
    <title lang="eng">Press Preview with Anna Jones</title>
    <sub-title lang="eng">Anna Jones and guests with a review of tomorrow's newspapers and passionate debate about the stories they cover</sub-title>
    <category lang="en">News / Current affairs : News magazine</category>
</programme>

For "arte" it's the same problem, I got both German and Spanish entries which Kodi and VLC seem to filter, but 2 languages make Yatse weird.

The EPG "preferred language" setting obviously does not filter languages.

/HXZ

For the sake of completeness, here is what I see for arte:

<programme start="20190319060000 +0100" stop="20190320060000 +0100" channel="f902dfdfe9808e1fe8e90bb2b3ddada9">
    <title lang="spa">ARTE</title>
    <desc lang="spa">El canal de televisiðn cultural. </desc>
</programme>
<programme start="20190320060000 +0100" stop="20190321060000 +0100" channel="f902dfdfe9808e1fe8e90bb2b3ddada9">
    <title lang="spa">ARTE</title>
    <desc lang="spa">El canal de televisiðn cultural. </desc>
</programme>
    <programme start="20190321061000 +0100" stop="20190321071000 +0100" channel="f902dfdfe9808e1fe8e90bb2b3ddada9">
    <title lang="ger">Das Rätsel unseres Bewusstseins</title>
    <sub-title lang="ger">Dokumentation Frankreich 2015</sub-title>
    <desc lang="ger">Das menschliche Bewusstsein ist einzigartig in der Natur [...] # continues</desc>
</programme>
# other EPG entries following, all normal

So it seems EPG data from arte and some Movistar channel are mixed together.

Can anyone please check how the situation is with "BBC World HD" and "Sky News Intl" on Astra1x? Do they really have that empty or almost empty EPG like shown above?

/HXZ

Try with this software
https://github.com/hiroshiyui/epgrab
just tune to channel and wait to finish

Sorry, does not work. Posted the link myself 2 days ago, but this one won't work with my SAT>IP server, unfortunately.

/HXZ

it work with /dev/dvb adapters on box
at end of week can precise point one of my dish /after wind/ to 19.2 to be shure

make curl -m 60 "http://service url

saen acro wrote:

make curl -m 60 "http://service url

Here is my first problem, I don't know the URL that is requested from SAT>IP. Any way to find those out easily?

Second: will it work to get EPG from raw stream? Those Movistar channels are encrypted, my SAT>IP does not decrypt in hardware, I use TVheadend to decrypt.

@Jaroslav: can you please explain how the EPG "preferred language" is meant to work? Manual says

"Select the list of languages (in order of priority) to be used for supplying EPG information to clients that don't provide their own configuration" 

HTH,
/HXZ

In TVH you have ;)

U can use german xml epg :) (ch + at also included)

https://takealug.de/wordpress/2019/02/10/fertiges-de-at-ch-xml-epg-zum-download/

Bastian,

thank you very much for joining and your efforts.

I strongly agree with you that we all should switch to XMLTV and people are just too lazy to do that step (that includes me:) ). Your reasoning IMHO is also very close to the point, I didn't go to XMLTV because all of the sources have different lacks of information, I'm too lazy to build a config, and it it often means a loss of information.

Do you have some github repo for your filter confguration?

/HXZ

saen acro wrote:

In TVH you have ;)

There is output but not input URL, and input is what I'm looking for. Or can you see SAT>IP source URLs somewhere in TVh and I just don't know?

/HXZ

You can record mux /without pid filter/ on mux tab /play button url/

da h4xX0rz1sT wrote:

I tried the same with the arte entry, but without success. But I think we can be reasonably sure that messy entry really comes from Movistar. @peter peter -----: Can you see EPG on your SAT>IP boxes?

I have only Libreelec with tvheadend, no other SAT>IP boxes. On the SelfSAT IP36 or IP21 there you don't see any EPG-informations.
I see on "arte", "BBC World News Europe HD" and "Sky News Intl" the spanish EPG in tvheadend and Kodi at the same time, because Kodi will be served from tvheadend.

When I will look into the VLC-EPG-Data then I start the stream with the "Play"-Button on the selected channel in tvheadend with the "pass"-stream-profile and VLC decodes the EPG data by itself. There I see the correct german EPG ("arte"), correct english EPG ("Sky News Intl") and correct no EPG ("BBC World News Europe HD"). The pass-profile from tvheadend serves the correct EPG-Data.

@Bastian Kleinschmidt: Thank you very much, but when it's possible I don't want to use external EPG-data. For me the EIT-OTA-EPG is good enough.

I have found out something else, where you can see the problem too:
On transponder 12603.75H you have 8 "WDR" channels (german), "Sky News Intl" and 3 radio channels. "Sky News Intl" will be overwritten with spanish EPG from somewhere and "destroys" the german EPG of all 8 "WDR" channels. In VLC-EPG everything is normal. Maybe this helps too...

saen acro wrote:

You can record mux /without pid filter/ on mux tab /play button url/

Got it. Yeah, thank you, that was a lot more helpful than your last answer ;)

/HXZ :)

The english EPG information for "Sky News Intl" (Transponder 12603.75H) will be definitely overwritten from transponder 10729V!

Peter -----:

thank you very much for your observations. We might get closer to the culprit :)

I've to think about your observations across the equinox, will come back to you :)

Please post meanwhile which version of Kodi/what platform/what hardware you're using.

BBL,
/HXZ

Libreelec-Generic.x86_64-9.0.1 (Kernel Linux 4.19.23) [[https://libreelec.tv/downloads_new/generic-pc/]]
Kodi 18.1-Leia-2019-02-21
tvheadend 4.3-1774 from [[http://build.mycvh.de/Tvh-LE9.0/4.3-1774-01.03.2019/]] but the problem was an earlier versions too.

Running on an Intel J4105 with 8GB RAM.

Jaroslav Kysela wrote:

I did a quick look on 19.2E and it seems that EIT data are inconsistent among transpoders for some noted channels (ARTE & etc). The only way is to disable the EIT grabber for the problematic mux/transponder (EPG Scan = disable), but it will affect all services. If some services in this mux should be updated, then we're in the trouble.

There's another problem: I've tested this "solution" - disable EPG-Grabber on all spanish transponders - but when I start a spanish TV-channel ("MULTIDEPORTE"), then EPG will be grabbed and the problem is back aigan. Maybe I have to disable EPG on every service, not only transponder - I will test it.

Peter ----- wrote:

Jaroslav Kysela wrote:

I did a quick look on 19.2E and it seems that EIT data are inconsistent among transpoders for some noted channels (ARTE & etc). The only way is to disable the EIT grabber for the problematic mux/transponder (EPG Scan = disable), but it will affect all services. If some services in this mux should be updated, then we're in the trouble.

There's another problem: I've tested this "solution" - disable EPG-Grabber on all spanish transponders - but when I start a spanish TV-channel ("MULTIDEPORTE"), then EPG will be grabbed and the problem is back aigan. Maybe I have to disable EPG on every service, not only transponder - I will test it.

No, does also not work!

Peter ----- wrote:

When I will look into the VLC-EPG-Data then I start the stream with the "Play"-Button on the selected channel in tvheadend with the "pass"-stream-profile and VLC decodes the EPG data by itself. There I see the correct german EPG ("arte"), correct english EPG ("Sky News Intl") and correct no EPG ("BBC World News Europe HD"). The pass-profile from tvheadend serves the correct EPG-Data.

I can also see "Sky News Intl" EPG builds up slowly when playing the channel, but that data should already be known by grabbing.

I have found out something else, where you can see the problem too:
On transponder 12603.75H you have 8 "WDR" channels (german), "Sky News Intl" and 3 radio channels. "Sky News Intl" will be overwritten with spanish EPG from somewhere and "destroys" the german EPG of all 8 "WDR" channels. In VLC-EPG everything is normal. Maybe this helps too...

Can't confirm the WDR problem, everything seems normal here.

@peter peter -----: do you have those issues in Kodi, too? Here, everything is normal and the Spanish entries seem to get filtered, but Kodi gets them, since I can see them in Yatse, which loads the data from Kodi.

HTH,
/HXZ

Well, I experienced this issue nearly two years ago, but never managed to solve it.
As a workaround I am using external xmltv-data, provided by epgdata.com and webgrabplus.

Main problem and disadvantage of external epg data: They are not as up to date as the ota-data.

Another nice problem you will encounter: EPG data of SD and HD channels are not the same, showing minor differences (3sat SD <--> 3 sat HD etc.)

And I never received epg-data of spanish speaking channels like Canal 24 horas or TVE Internacional.
International channels like CNN International or CNBC Europa are also difficult.
IMHO provision of epg-data works well in the UK, Germany and Scandinavian countries, maybe also in France and Poland, but in the rest of the world they are a neglected genre. And even in the mentioned countries broadcast stations seem to implement it insufficiently.

Some comments:

1) the preferred EPG language is just preferred language - it does not do and mean any filtering - tvh maintains all languages in the database; usually, the internal language list for clients is 'user language + preferred languages', first available text is returned

2) disable 'EPG Scan' in the mux settings won't help (my fault), it just configures the EPG grabbing for the mux, but if you tune to the disabled mux (scan, streaming service etc.), the EIT data are grabbed

3) if EIT data for one service are available on multiple transponders, last data wins (no merging in the latest code)

In my opinion, the best methods to avoid those collisions are:

1) force EIT mux per service and do not update EPG database from others

2) per service and per language import - update EPG database only when one language (languages) are seen in the EPG event

Start to ting that this is same as this
https://tvheadend.org/issues/5062

Jaroslav Kysela wrote:

1) the preferred EPG language is just preferred language - it does not do and mean any filtering - tvh maintains all languages in the database; usually, the internal language list for clients is 'user language + preferred languages', first available text is returned

Thank you for that clarification.

3) if EIT data for one service are available on multiple transponders, last data wins (no merging in the latest code)

OK, this makes the problem clear.

In my opinion, the best methods to avoid those collisions are:

1) force EIT mux per service and do not update EPG database from others

IMHO, this is a bad idea, since updates for other services are generally a good thing, besides those Movistar people that fiddle around with services they don't own.

2) per service and per language import - update EPG database only when one language (languages) are seen in the EPG event

I think the global configuration could be enough here. But then, EPG would need to be filtered to drop languages with lower preference level.

Another idea would be to allow or forbid those cross-channel updates per provider (Movistar=no)

HTH,
/HXZ

The characterset I've set in networks is ISO-8859-15. But also no luck with ISO-8859-1 and UTF-8.

Peter ----- wrote:

da h4xX0rz1sT wrote:

I can also see "Sky News Intl" EPG builds up slowly when playing the channel, but that data should already be known by grabbing.

No it does not! Here you can see for "Sky News Intl" in the pictures the big difference, which EPG-Data tvheadend stores and which EPG-Data VLC grabs while playing with the same Libreelec/Kodi-Box. In tvheadend look also at the starttime column.

You seem to be right, it builds up slowly, but stays incomplete, having only the same two shows each morning but not more.

So what shall I do? I don't know. Is my configuration wrong, but where?

As Jaroslav explained, there is nothing you could do in the moment, except not to panic ;)

Peter ----- wrote:

The characterset I've set in networks is ISO-8859-15. But also no luck with ISO-8859-1 and UTF-8.

You probably don't have a charset problem here, but FYI those Sky channels use ISO-8859-9.

HTH,
/HXZ

da h4xX0rz1sT wrote:

Jaroslav Kysela wrote:

2) per service and per language import - update EPG database only when one language (languages) are seen in the EPG event

I think the global configuration could be enough here. But then, EPG would need to be filtered to drop languages with lower preference level.

I meant to ignore imported events which do not contain a language from the specified list (in the service settings).

Another idea would be to allow or forbid those cross-channel updates per provider (Movistar=no)

Unfortunately, it might not be easy to detect who created the EIT table. For example, the 10729V contains services from three broadcasters:

sdt: mux 10729V in S19.2E
sdt:  type 19 (25) name [ALQUILER HD] provider [Movistar+] def_auth []
sdt:  type 19 (25) name [BEINLALIGA2] provider [Movistar+] def_auth []
sdt:  type 19 (25) name [AXN HD] provider [DIGITAL+] def_auth []
sdt:  type 19 (25) name [M.COMEDIA HD] provider [Movistar+] def_auth []
sdt:  type 19 (25) name [LALIGATV BAR] provider [Movistar+] def_auth []
sdt:  type 19 (25) name [ALQUILER 2 HD] provider [Movistar+] def_auth []
sdt:  type 19 (25) name [M.PartidazoHD] provider [CANAL+] def_auth []
sdt:  type 19 (25) name [M.LCAMPEON1] provider [Movistar+] def_auth []
sdt:  type 19 (25) name [LA 1 HD] provider [Movisrta+] def_auth []
sdt:  type 19 (25) name [M.LCAMPEON2] provider [Movistar+] def_auth []

Jaroslav Kysela wrote:

da h4xX0rz1sT wrote:

Jaroslav Kysela wrote:

2) per service and per language import - update EPG database only when one language (languages) are seen in the EPG event

I think the global configuration could be enough here. But then, EPG would need to be filtered to drop languages with lower preference level.

I meant to ignore imported events which do not contain a language from the specified list (in the service settings).

OK, but in Peter's and my case, we would have to specify "ger" "eng" and "spa", which leads us to where we already are - some clients simply don't handle multi-language EPG properly, that's why we have to filter if we want them to work. On, the other hand, does anybody want to have more than one EPG language? (also regarding most clients do not handle it properly; this use case might exist but be rare - filtering could be optional for that)

Another idea would be to allow or forbid those cross-channel updates per provider (Movistar=no)

Unfortunately, it might not be easy to detect who created the EIT table.

Yes I understand that. But when filtering by language, you don't have cope with that issue ;)

HTH,
/HXZ

Jaroslav Kysela wrote:

Unfortunately, it might not be easy to detect who created the EIT table. For example, the 10729V contains services from three broadcasters:

What about SID filter compared to current transponder SID,
everything with is != "in current transponder" services list to be ignored.

Jaroslav Kysela wrote:

Unfortunately, it might not be easy to detect who created the EIT table.

There should be "original_network_id" in each entry - have not checked yet whether they always set it properly.

Another possibility might be a kill switch for "alien" language EPG entries that are in a language the channel does not provide, e.g. "arte" that is only German or French => put Spanish to /dev/null. Only problem is channel languages might change from time to time or within the stream itself...

HTH,
/HXZ

saen acro wrote:

What about SID filter compared to current transponder SID,
everything with is != "in current transponder" services list to be ignored.

Some broadcasters, e.g. public TV here in Germany update across transponders, which is perfectly legal - not a bug, really a feature :)

Or should we file a bug against Astra? Their backup uplink is just some kilometers away from here, I could visit them. *justjoking

HTH,
/HXZ

da h4xX0rz1sT wrote:

Peter ----- wrote:

The characterset I've set in networks is ISO-8859-15. But also no luck with ISO-8859-1 and UTF-8.

You probably don't have a charset problem here, but FYI those Sky channels use ISO-8859-9.

Aha, thank you for the information that Sky uses ISO-8859-9. Where do I get the information which channel/provider uses which character-set?

ISO 8859 character naming explained
https://tvheadend.org/issues/5425

Thank you, I know what a character-set is, but I don't know which channel/provider uses which character-set...

Peter ----- wrote:

da h4xX0rz1sT wrote:

You probably don't have a charset problem here, but FYI those Sky channels use ISO-8859-9.

Aha, thank you for the information that Sky uses ISO-8859-9. Where do I get the information which channel/provider uses which character-set?

@peter peter -----:
Generally, you don't have to set a charset, the "AUTO" setting will be fine in most circumstances. The broadcast usually contains the character set used. The only channels I have explicitly set a charset for are those from Sky, because they seem not to broadcast their used encoding correctly.

If you find weird characters in some EPG, and you can't find what character set the broadcaster uses on any search engine, you still have guessing/trial & error as a last resort.

HTH,
/HXZ