Feature #227
Add multiple language in EIT
100%
Description
Hello, I would like to see multiple language support for EIT in tvheadend. I attached muxdump from Total TV provider which have at least 4 different languages in EIT.
Now tvheadend mixes all language into EPG and it looks really strange for me. One Program has slovenian language, other serbian and some of them english.
I would also like to mention that in slovenian language some of the character are partly supported (čšž and probablj ćđ also). I wrote down partly because some Programs have them in EPG and some don't.
This should be also included in attached muxdump.
Files
History
Updated by Alan - over 14 years ago
Yes i also have the same problem with UPC Direct broadcast with Hungarian, Slovakian, English, Romanian language EPG. Character Encoding is OK though. Maybe a setting for preferred EPG language and a setting with character conversion options would be great. The preference could be for not only one but for two (three) languages. So that One can tell that in case the first preferred language is not there the second preferred language should step in place ( just like in VDR i think)
Thanks for the great work Andoma.
Updated by joakim - over 14 years ago
This would be great. My EPG is in swedish instead of Norwegian.
alanwww1: Thats how mythtv does it aswell.
Updated by Gujs - over 14 years ago
Hi,
I don't think it is related to #163 because this is EPG language grabbed from DVB provider and not web interface language.
Updated by Hein Rigolo over 14 years ago
Gujs, you are right ... but still part of the whole "we need to support other languages" part of Tvheadend (both EPG, WebGUI, HTSP etc etc)
Updated by Gujs - about 14 years ago
I just changed the milestone. That way this ticket is visible to all of us and it will not be forgotten.
Updated by Riku H almost 14 years ago
preferred epg language should be configurable
DVB-DescriptorTag: 77 (0x4d) [= short_event_descriptor]
descriptor_length: 139 (0x8b)
ISO639_2_language_code: fin
event_name_length: 33 (0x21)
event_name: "House 7. kausi, 2. jakso: Selfish" -- Charset: Latin alphabet
text_length: 101 (0x65)
text_char: "Maailman suosituimman sarjan 7. tuotantokaudella House ja Cuddy yrittävät saada suhteensa toimimaan." -- Charset: Latin alphabet no. 5
DVB-DescriptorTag: 77 (0x4d) [= short_event_descriptor]
descriptor_length: 83 (0x53)
ISO639_2_language_code: swe
event_name_length: 20 (0x14)
event_name: "House Del 2, Selfish" -- Charset: Latin alphabet
text_length: 58 (0x3a)
text_char: "Amerikansk dramaserie med Hugh Laurie och Lisa Edelstein." -- Charset: Latin alphabet no. 5
Updated by Jarmo Keränen over 13 years ago
This really is annoying feature and very important feature to be corrected. At least for me this is the question to use or not to use TVheadend at the moment, and it seems that this is not suitable for me at the moment... Is it really so that this cannot be patched somehow to work when we're waiting for ver. 3.0? Other option could be (at least for me) to have patch to eliminate EPG data receiving from DVB and use only XMLTV (there is another issue for that, but I mean some temporary solution with patch before we have ver 3.0).
If someone has made patch for this or disbling EPG from DVB signal I would like to have it also...
Anyway, if we do not look at these two problems I mentioned, great work and easy to configure to use. applause ;o)
Updated by Riku H over 13 years ago
Updated by Adam Sutton over 12 years ago
- Category changed from DVB to EPG
- Status changed from New to Accepted
- Assignee set to Adam Sutton
I've been discussing this really with some people and it does indeed need to be included, I just need to look at how etc..
Updated by Adam Sutton over 12 years ago
Been giving this some thought, it would be quite straightforward to hack this into the EIT handler. However I'm not sure that's the right way to go. I'm beginning to think that it might require a more fundamental update to the EPG code so that this is properly handled throughout the EPG code.
This doesn't mean I'm thinking about storing multiple languages in the EPG database (though that is one way to go, I think its a bit much at this stage). More that I'm thinking it needs to consider the fact we also have inputs from other sources, XMLTV etc.. which might conflict and/or have multi language support.
I need to think this through properly.
Updated by Riku H over 12 years ago
the above patch doesn't work anymore for the new version (I'm using 3.1.539.g51595.dirty)
can I someway force the language to "fin" for short event 0x4d?
epggrab/module/eit.c has:
/* Language (skip) */
len -= 3;
ptr += 3;
Updated by Adam Sutton over 12 years ago
Well, someone persuaded me it might be a good idea to actually do things properly and provide full multi-lingual support to the EPG. I.e. I track all available languages in the DB and then filter on request.
Seems a great idea and much of the implementation is no more complicated than basic filtering on input. However since I use title/desc hashing for episode IDs, you get a horrible chicken/egg situation plus a general processing nightmare.
However I'm going to persevere and see what comes of it. But might not be ready today.
Updated by Adam Sutton over 12 years ago
- % Done changed from 0 to 80
OK,
I have a "working" solution. I'm using the word "working" very loosely here basically I've done all the basic work to include full multi-lingual support to the EPG code. But its VERY experimental, as in I've done almost no testing as I don't have access to a multi-lingual source (at the moment).
I've done basic testing with my own setup and XMLTV input (I'll try and do a bit more tonight). But if anyone wants to give it a got I'd really appreciate that. But do expect some failures/crashes etc...
The EPG DB (as well as related code) is now fully multi-lingual (or at least that's the theory) so info for ALL available languages will be stored in the database. And its then filtered on request.
To make use of it, goto WebUI->Configuration->General and set default languages. This is a comma separated list of ISO-639 (1 and 2T and 2B) codes (infact it'll parse a string as expected by HTTP Accept-Language). This will define the default options, useful for HTSP (to showtime/XBMC). Note: if you don't do this the values you get back could be a random collection of langs (though generally it'll be the first, alphabetically, language available).
From the WebUI itself it will try and pick up the Accept-Language header (as sent by browser) so beware if your browser is configured differently to what you expect.
Anyway let me know how you get on.
Code is on github @ https://github.com/adamsutton/tvheadend and make sure you use the feature/227-multilang-eit branch.
Adam
Updated by Riku H over 12 years ago
seems to be working, thanks!
I stopped tvheadend, deleted epgdb.v2 and started again.
EPG Grabber OTA Interfaces "EIT: DVB Grabber" should be selected by default.
Updated by Adam Sutton over 12 years ago
Hi Riku,
All OTA grabbers are meant to be enabled by default for new installations and on migration from old configuration. However there was a bug in the migration that meant it did not do this, so if you have used several git master builds recently you might have fallen into that trap (i.e. it migrated the config incorrectly, it won't do it a second time). If this was your use of latest code then let me know and I'll check there isn't another mistake in the migration.
I'm glad the code appears to be working, especially since it was a bit of a blind shot as I'd done very little testing.
I'd be quite interested to know how the language selection works, if you're able to change the language your browser requests does the EPG come up in a different language?
Alternatively if you could mail me your EPG database and possibly the contents of the .hts/tvheadend/channels, I might be able to check it myself on my machine.
Regards
Adam
Updated by Adam Sutton over 12 years ago
- Status changed from Accepted to Resolved
- % Done changed from 80 to 90
Marking as resolved, although will not close until I have merged code to master. But that might not be for a little while. Need to do a proper review of what I've changed.
Updated by Adam Sutton over 12 years ago
- Status changed from Resolved to Fixed
- % Done changed from 90 to 100
Applied in changeset commit:00902f46135ddbfa86047192652d4f14611f09c2.
Updated by doron rabia about 12 years ago
For some reason I can't get it to work - I have waited for it for so long (thanks a lot for the development work !!)
I am trying to filter out hebrew out of 4 languages in EIT EPG , but still get all 4 of them randomly.
I am using version 3.1.608.g08434 and have tried to set he language as "heb" "he" and even "hebrew" as per Iso639, but nothing worked.
Any idea ?
Thanks in Advance,
Doron.
Updated by Adam Sutton about 12 years ago
Can you attach your epgdb.v2 file so I can take a quick look at what's in it.
Adam
Updated by doron rabia about 12 years ago
sure- but I think that currently it is in RAM after I have deleted and restarted the daemon. I am waiting for tvheadend to write it to disk.
Updated by Adam Sutton about 12 years ago
It will write to disk as soon as you shutdown tvheadend, should take a second or two. Then just restart straight away, so as not to disrupt service
Updated by doron rabia about 12 years ago
Sorry about the tar - very old habit of mine ,even if a single file inside
Updated by Adam Sutton about 12 years ago
Are you sure you're actually running the right version/binary? That database does not correspond to the format of the latest code.
Can you check that you're definitely running the right version, check what version it says in the UI (About tab). And make sure you refresh incase it's cached something from a previous instance you had running.
Adam
Updated by doron rabia about 12 years ago
I am pretty sure, as I previous versions were 2.99* and only tonight I have changed to 3.1 tree.
I must say that my config is not fresh ,and I am using channel configs from previous versions.
I did notice that when started with new version (3.1 based) it did create epgdb.v2 which was a new file.
I will refresh my local git and rebuild and see if this helps.
many thanks,
Doron.
Updated by Adam Sutton about 12 years ago
Hold on, my bad. I was looking at someone else's database. I've got WAY too many users config's on this machine I need to put a rule about how to provide stuff in nicely named directories
OK, so having got that out of the way, what exactly is the problem? I can see that it definitely has the hebrew data in (lang code should be "heb") as well as English (eng), Russian (rus) and Arabic (ara).
Although not every entry appears to have every language, probably a limitation of the EIT data. And in many cases although its listed as one of the non-English entries its in English. Again most likely a problem with the upstream data.
Updated by doron rabia about 12 years ago
My problem is that I see the EPG in all 4 languages , although I explicitly set it to heb.
Updated by Adam Sutton about 12 years ago
Right. It's not actually a filter, its a prioritisation.
If there is no Hebrew entry it will return the next entry in your list. If none of your selections match it will pick whatever it can from what is in the database. This will typically be the first available lang code (alphabetically).
And as mentioned several of the Hebrew entries appear to actually be in English (though they report Hebrew).
The alternative would be to simply return blank fields, where the explicit selections are not available. And I'm not sure that would be right. Certainly most people would not want that.
Updated by doron rabia about 12 years ago
Fair enough, and good approach.
I will try to filter both eng and heb and see what happends.
Many thanks for your effort - keep up the good work !
Doron.