Tvheadend

Add missing string to '_epg_genre_names' array in src/epg.c . The xmltv strings are translated to DVB categories.

Jaroslav Kysela wrote:

Add missing string to '_epg_genre_names' array in src/epg.c . The xmltv strings are translated to DVB categories.

thank you - looks like it was a grabber issue! Now that I know where this is - if I find any additional genres I will submit.

Fixed the grabber and now I'm passing all the genre (category) names into the xmltv file.

After looking at epg.c it seems like this is a relatively short list compared to what is used here in the US for genre information. Is there a reason to not just pass through the genre names as they are in the xmltv file?

TVH is being coded in Europe, so we're using the 8-bit value according the ETSI DVB standard - _epg_genre_names table follows this standard. Note that the table is splitted to major / minor part (and only major genre group is filtered in the web EPG window - we just realized that too many categories are not much helpful for the standard usage).

I see two ways to handle additional categories:

- extend the DVB table (handle the clashes which many come with DVB standard updates)
- use own categories mapping (create completely own table and map also DVB genre there)

Could you gather the names of missing categories ?

Jaroslav Kysela wrote:

Could you gather the names of missing categories ?

I have contacted Gracenote - provider of the data - to see if they can provide a complete list of categories used. I've checked the list in the ETSI TS 102 822-3-1 document and it is quite extensive. I'm happy to parse it and work on extending the table, but will a list that long have any impact on how the program performs? Also - I'm not sure just the major category is enough information. For example - the major category of "Movie" wouldn't tell you "Documentary" or "Western" or "Thriller" which is much more useful information to see in a guide. Seems like at least two levels of content should be passed on.

Another option would be for there to be a setting to select "enable all genre/category/content without filter" - this would just pass all the "category" data through to the frontend and then that system can decide how to filter the data. This takes the burden off Tvh and allows customization based on the frontend system a user chooses to run.

The current genre scheme follows EN 300 468 - see 6.2.9 content descriptor table.

Jaroslav Kysela wrote:

The current genre scheme follows EN 300 468 - see 6.2.9 content descriptor table.

Correct - but without including the level 2 information, these genres are too vague to be relevant to the end user. I would be happy to build out the full table in order to enable the level 2 descriptors.

It is often these level 2 descriptors that are the only information input buy the content providers here in the US - therefore most of the epg content fields are empty in Tvh.

Should I rework the epg.c file, test and then submit for pull to enable level 2?

Ok, let's look to "A.8 ContentCS", you're right that there are up to three level information in the newer standard:

3.1 - NON-FICTION/INFORMATION
  3.1.1 - Time-sensitive information
  3.1.1.1 - Daily news
  3.1.1.2 - Special news/edition

While we have:

1[.1] - Movie/Drama
  1.2 - Detective / Thriller
  1.3 - Adventure / Western / War

I think that we should improve this and follow the new standard. We can probably use 16-bit unsigned int instead 8-bit value like:

5 bits - level 0 (major)
  5 bits - level 1
  6 bits - level 2

... and provide mapping from old genre value into this new one.

In the meantime - I've been experimenting with adding missing genres to the _epg_genre_names array in epg.c

This works well and actually displays the genres in the tvheadend content type field of the web epg. If we could take this string and pass it to kodi as the sGenre field - the string would be available in the epg in kodi. Right now tvheadend is not passing anything to the sGenre field.

Then all that would need to be changed is using a skin that has the option to use the sGenre string instead of the iGenreType and iGenreSubType fields.

This is what I would like to test next, but I need to figure out how to pass the content type string that is used in the tvheadend web interface to the sGenre field in kodi. Any thoughts?

I think part of what edit4ever ! wants is implemented in #4441 and a recent change in pvr.hts. Tvheadend passes through all categories to pvr.hts. In the case where Tvheadend has mapped a category to a genre then pvr.hts will use the 8-bit genre code.

However, in the case where no genre code mapping has occurred, pvr.hts will pass through to Kodi strGenreDescription and passes through all the categories as a CSV which is displayed in Kodi UI.
[[https://github.com/kodi-pvr/pvr.hts/issues/340]].

Recording via categories ("Movie" + "Western") is also available in the Tvheadend UI (#4665) but it is not supported in the Kodi API.

As with all newer standards, the problem with ETSI would be that everyone has to move. So we'd need to get Kodi to change their API too.

My one problem with genres is that they always seem arbitrary. Sure it has romcom ("Romantic Comedy") but where's the category for zomcom ("Zombie Comedy")? :-)

Also, SD doesn't seem to follow the wording of ETSI. For example a grep of my xml shows I have categories of "Yacht Racing" (ETSI has Yachting), and also categories of "Cheerleading" and "Standup" that I couldn't easily find in A.8. So we'd probably need a mapping layer.

The nice thing about genres though is that you get colours in the UI, but I've never really known which colour means which category, and programmes frequently fit in multiple genres.

In my mind, ETSI and EN300 are both quite limited sets, and having free-form text keywords can have some benefits. I can see the benefit of having "3.5.7.3" from a supplier POV but from a user-POV I prefer "Sitcom". So perhaps ETSI should just map to free-form string categories rather than categories mapping to a more restricted ETSI.

I think you may be right that chasing the proper genre format may not work. However, I would suggest including an option to always pass through the category list - even if a match is made. Otherwise what happens, at least here in the US, is the word comedy gets matched to the EN300 Movie genre and only comedy is displayed, despite something being a sitcom type of show and not a movie. So the coloring in that case doesn't make sense as it is colored like a movie.

This would of course turn off the coloring - but that may be preferable to some people would still provide information in the kodi genre display.

Maybe we can get the kodi team to switch to using the 8 bit code just for coloring, but always put in an option to read the genre string from the sGenre field. This would allow tvh to send an 8 bit code for coloring (based on whatever options we want to include) but the display in kodi would always use that sGenre string field.

Just a thought. :-) Otherwise - I'm really considering having to customize the Estuary skin to get rid of the genre display on the EPG!

Yes, I've seen the same problem here that I can't really tell what type of programme something is from the colour, but colours do make it look nicer. Kodi could just assign random colours and most people (including me) probably wouldn't tell the difference.

The one problem with only seeing categories in Kodi (instead of genres) is that some skins currently expect a short word ("Drama") so you can get a long tedious scroll, but I get that in other dialogs such as movie keyword scrolling of 50 keywords displayed in a scrolling label of a dozen characters.

I'm not sure the best UI approach in Kodi. As you say, I would imagine Kodi could use the 8-bit code and the string together since they can check for a non-empty string which does seem the best of both worlds.

To clarify: Tvheadend passes through both genre and categories, but the Kodi API currently only allows one or the other so pvr.hts just sends the one through (currently preferring genre).

I guess it depends how people use the genre/category. My autorec scheduling is convoluted with rules such as "Action, Movie, Stars>80%", but at an EPG-level, I ignore genre (and most other fields) completely.

Perhaps what would be useful for multiple categories/genres is be to display multiple small icons for the programmes. So, if it displayed a film reel and a smiley face, I would know it's a comedy film; a smiley face and series icon and I'd know what it is; a film reel and magnifying glass for a detective film; an elephant and I'd know it was a nature programme; musical note, etc.

"To clarify: Tvheadend passes through both genre and categories, but the Kodi API currently only allows one or the other so pvr.hts just sends the one through (currently preferring genre)."

Since I have not been able to test an updated pvr.hts to see how tvh is handling the categories...can you clarify the following:

Currently tvh passes the 8 bit category and sub category codes through pvr.hts to kodi - these fill in the iGenreType and iGenreSubType fields in kodi's epg database. Does tvh/pvr.hts send anything to the sGenre field in that database?

I have not seen that field filled in - but maybe that is due to not having the update pvr.hts - this field could contain the catgories/genres string all the time. Then all we need to do is have the kodi team add an infolabel for sGenre since the current VideoPlayer.Genre infolabel pulls from the iGenreType and iGenreSubType fields.

This would simplify things and allow the skin or user to decide what info to display.

See logic at line 1370:
[[https://github.com/kodi-pvr/pvr.hts/commit/9e58cbb0650a23a14709329050ee439cae9f219f#diff-3d7c5d1c125334f96674ced9f9ac2cdfR1370]]

So, yes, it will now populate the sGenreDescription field from the categories, but only if the 8-bit genre isn't set. The reason for this is that Kodi only looks at the sGenreDescription if we pass through a special "use string" value as the 8-bit value. I don't know why Kodi does it like that, rather than allowing both to be passed and just checking non-empty string.

Anyway: I implemented (but not checked in) my idea of adding icons to represent categories, but only in tvh UI and only for the top few categories. So, I append them to the title in the EPG and it's actually almost useful. I can easily see this afternoon's programme is a Western and there's some horror on later and some other shows are musicals. I'll play with it for a few days and see if it's worth submitting.

OK, the system makes sense. Maybe we could just add an option, either in tvh or in pvr.hts, to set "use string" for all and pass through the categories. This would allow the end user to see the full list of categories (genres) in the kodi epg by overriding any original matching to the 8 bit info.

Thanks for the info!

I think in order of preference it would be kodi, pvr.hts, tvh. The reason being that kodi could handle the integer for colour and the text for displaying and other clients could also handle it their way.

However, thinking about it we'd lose internationalization of the category names which you get with the integer. Sample xmltv for Denmark in issue 1774 uses English words for categories, but perhaps it just does that to maximize compatibility.

If not done in Kodi, then the next best place would be pvr.hts since changing tvh to strip fields would mean other clients on Android/Apple would also get the stripped fields and it might become a mess trying to strip data for some clients and not others.

Otherwise, perhaps it can go on the user settings in tvh so different frontends can login with different details. I've never really investigated that area so don't know if that's a good idea.

One other place we could probably fix it would be to have a checkbox on the xmltv importer to say "map xmltv category to genres", default to true (as currently happens). Users who don't want genres can untick it and will only get categories throughout the system including at the Kodi UI.

I like that idea. If you want to make a patch for that I'm happy to test it. :-)

I'll take a look at the weekend. Should be easy enough.

Em Smith wrote:

I had to reverse the logic of the checkbox since it seems default values only apply to completely new records not existing ones (or my def.i=1 logic was incorrect).

You should set the defaults in C when the structure is created (before idnode_load()). The def.X contants are only for webui at the moment. For xmltv it means:

  epggrab_module_ext_t *mod = epggrab_module_ext_create();
  mod->xmltv_var = 1;

Similar code should be added for epggrab_module_int_create().

I see it, thanks.

I think my earlier comment was confusing, so to clarify: The default (if the user does nothing) is to map categories to genres as currently happens, which I think is what most users will want even if the mapping doesn't handle all categories, so the default is false. If they tick the box then we will no longer do the mapping of categories to genres at all.

Em Smith wrote:

See logic at line 1370:
[[https://github.com/kodi-pvr/pvr.hts/commit/9e58cbb0650a23a14709329050ee439cae9f219f#diff-3d7c5d1c125334f96674ced9f9ac2cdfR1370]]

So, yes, it will now populate the sGenreDescription field from the categories, but only if the 8-bit genre isn't set. The reason for this is that Kodi only looks at the sGenreDescription if we pass through a special "use string" value as the 8-bit value. I don't know why Kodi does it like that, rather than allowing both to be passed and just checking non-empty string.

For the sGenreDescription to be pulled into kodi - do we need a specific updated version of pvr.hts?? Right now the TVHweb client is showing the categories - but they are not being passed into kodi.

You need a relatively new version of pvr.hts/Kodi. Looking at the pvr.hts commit, a nightly build after 8th November should be fine.

Although Kodi itself supported the API for a few years, I don't think the pvr.hts change was back-ported since some cleanups were made to the Kodi API at the same time as the pvr.hts commit to clarify that fields are separated by commas and then those Kodi API changes were merged back in to the pvr.hts commit to make it all clean.

OK tested and working as expected.

Also working is the patch to disable the categories matching to genres function. Using latest pvr.hts and tvheadend builds on latest LibreELEC 9 (Kodi 18) Alpha - I have:

• Categories listed in the Tvh web UI and no genre matches
• Kodi is showing the genre (using the categories list) but not coloring the epg (as no genre codes are sent)

Perfect! Would love to see this patch added as a commit. Thanks!

One note - it seems that when the categories are parsed, they are sorted alphabetically. This normally makes sense - except in the case of a movie. It would seem that the category "Movie" should always be listed first as the rest of the categories will be subcategories of the movie.

For example, Battle of Britain is listed as Action, Drama, Movie, War in the categories field in the web UI details. In the column with the emojis, the movie emoji is first. Since this info will now be passed to Kodi (or other clients) it would be better to be displayed as Movie, Action, Drama, War. This way you can always quickly see that you are looking at a movie - and in this case, a movie that is action, drama and war based.

I have the code in my xmltv grabber built to sort this way - but I realized that the Tvh function was overriding the order of the categories.

Thanks for considering!

Em Smith wrote:

Kodi could just assign random colours and most people (including me) probably wouldn't tell the difference.

FYI: This seems to be a skin issue within Kodi. For example, the Titan skin supports customizing the EPG colors.

tl;dr: I'll take a look at giving priority to some categories (such as Movies, News) in the list we send to pvr.hts so they are appear at the front.

You're right, the tvheadend UI puts movie at the front since at the server they are held in a sorted string tree for easy searching.

It's a little more complicated since the UI separates the sorted categories in to "major" and "minor" categories and moves the major ones to the front. So, it's not just movie, but also news, radio, series, and sports that are all moved to the front since, as you say, these seemed to me to reflect something you don't want lost in the middle of all the icons or hidden in the overflow.

So the daily financial news series for closing bell would be "news, series, finance" in the icons (since "finance" is a minor category), but "Bus./financial, News, Series, Show" in the categories.
[[https://github.com/tvheadend/tvheadend/blob/master/src/webui/static/app/tvheadend.js#L46]]

If categories aren't enabled then it does something similar for the major genres nybble.
[[https://github.com/tvheadend/tvheadend/blob/master/src/webui/static/app/tvheadend.js#L146]]

Finally, the "is new" icon is moved to the very front of the list; I know it's not a category but that's added in that same UI code.

Unfortunately, the tv_grab_zz_sdjson_sqlite grabber I use doesn't have the categories sorted, which is why I sorted them in the server.

For example I have "movie, Action, Drama, War, Feature Film, Movie" and "tvshow, War, News, Special, Show". The lack of sorting made it difficult for to see what categories a programme had.

(The initial lowercase movie/tvshow is only because I have the grabber's mythtv-categories enabled which fakes a first lowercase entry, and I think is the one that fakes a "radio" category if station isRadioStation. I think your grabber fakes "Movie / Drama" for similar reasons.)

But this means people using that grabber would have "Action" or "War" as the first category (assuming mythtv-categories is disabled, which is the default), rather than "Movie" or "Show".

It seems SD entityType is Show, Episode, Sports, or Movie, so I guess that's the one you shuffle to be the first category output and the one that (mostly by coincidence) matches my "major category" (except for radio).

For (my) future reference, looking at SD, for MV000056140000 (Battle of Britain) we get:

'genres' => [ 
                          'War',
                          'Action',
                          'Drama'
                        ],
'showType' => 'Feature Film',
'entityType' => 'Movie',

Partly I think it's a presentation issue that the UI (or Kodi UI) should take the data and present it in the appropriate way for the user, such as removing movie from the list and putting an icon, or backdrop, or whatever. But, at the same time, I agree that it's annoying that you sort your list and want it used that way.

Swapping the sorted tree for categories with an unsorted htsmsg is a little effort (probably 100 lines or so since it shares code with keywords). I think we'd need (yet another) tick box to say "use unsorted category order from xmltv file" since otherwise the unsorted list would unduly penalize people using the tv_grab_zz_sdjson_sqlite grabber and potentially others that do not order categories.

The simpler method may be to prioritize certain categories (Movie, News, etc) before we send them to the pvr.hts. That is less impact. I've checked pvr.hts and that doesn't sort the categories we send.

The third approach would be for pvr.hts or Kodi to give priority to certain categories, which is the ideal place since each user can sort depending on their preference.

I probably won't have time until late this year/early next year, but I am edging towards the second approach (us modify the data we send to pvr.hts) since it offers some consistency between different grabbers.

Re: Titan.
I used to love that skin for its configurability. I might take another look, especially since there's a tickbox somewhere for having snow and a huge present as the busy cursor which is great at this time of year. For the moment, I use Estuary Mod which is nice since it has an actors tab when you press "info" during playback.

I think your right about option 2. It is simple and accomplishes the goal for multiple platforms.

Gave the latest unstable version a try and it seems to work. The only annoying thing is that the string category has an captibale H at the end.

Example:
Categories:serie/dramaH

sebastian s. wrote:

Gave the latest unstable version a try and it seems to work. The only annoying thing is that the string category has an captibale H at the end.

Example:
Categories:serie/dramaH

Yeah having a lot of problems with categories here.
Lot of strange letters and wrong named categories.

Categories:newses, series
Categories:animation�G, comedyion�G, series
Categories:newsow, tvshow
Categories:golfts, sports
Categories:animation�G, dramation�G, kidsation�G, series
Categories:

A Few examples it seems it adds the 2 last letters in last category to the first name golfes, series adds 'es' from series to golf for example.

And forgot to mention: only small letters are used even if the xml has capitalized letters.

sebastian s. wrote:

Gave the latest unstable version a try and it seems to work. The only annoying thing is that the string category has an captibale H at the end.

Example:
Categories:serie/dramaH

Yeah having a lot of problems with categories here.
Lot of strange letters and wrong named categories.

Categories:newses, series
Categories:animation�G, comedyion�G, series
Categories:newsow, tvshow
Categories:golfts, sports
Categories:animation�G, dramation�G, kidsation�G, series
Categories:golfes, series

A Few examples it seems it adds the 2 last letters in last category to the first name golfes, series adds 'es' from series to golf for example.

sebastian s. wrote:

And forgot to mention: only small letters are used even if the xml has capitalized letters.

Yeah that is annoying too.

After a while of thinking i it seems it fills with data from the longest categoryname.

Example:
Categories:animation�G, dramation�G, kidsation�G, series

In this case "animation" is the longest name and fills from that to all categorynames "drama(tion)" "kids(ation)" exept last one.

I don't have multiple category lines in my xml, but yeah. Strange :)

"special interest/sonstige� H"
"unterhaltung/realityG"

Some others are okay

@Em Smith
Please let me know if you need further information

No it's only 1 category line.
I just pasted a few examples from a few programs.

Andreas Fornberg wrote:

No it's only 1 category line.
I just pasted a few examples from a few programs.

Ok i checked in the xml file and it's more than 1 category line but it's on 1 line in tvheadend info.