Tvheadend

Norm,

I'm intrigued, what are you trying to do?

Currently I don't think there is anything that does this. I started to put together a small HTSP python lib mainly for testing purposes. That could probably be updated to subscribe to channels and just drop any incoming data packets.

Adam

Well I wrote a perl script that listens to the transponder for EIT data, and builds an xml file out of it - it is then absorbed into TVH via the xml interface. Right now I have to manually tune to a channel on the specific transponder (NA sats use only one to send a 9-day guide), run my script, and then I can change channels or do whatever while the script processes.

I know we talked about integrating DVB EIT into TVH but I just haven't had time to pursue this and the script was written quite a while ago as a stop gap meassure LOL - which seems to have become more of a regularly used script now.

So the script to get EIT works but must be manually executed after tuning to a specific channel - I want to automate this process. I am just typing outloud here, but perhaps I can use the autoscan when idle to launch the script when it tunes the transponder I want... hmmm... not sure if that's what I really want to do though.

Norm

This sounds like something that really should be integrated into TVH directly. This is really not the way to do, it may work but its just sub-optimal and you'll end up duplicating a lot of code.

Why not simply create another pseudo EIT module (like uk_freesat/uk_freeview), that will install the relevant table handlers (for the correct mux and possibly non-standard PID?). The only other issue I can think of is does NA use any kind of compression (huffman etc..) on the strings?

Adam

Of course you are correct - and I did start to look at it but wasn't able to work out how the modules worked. NA uses (I will use Bell as an example) use 1 transponder for 9 day (12224000 on 91) for both of their sats, which I have set up as 2 different sources in TVH. The second sat (82) only has now/next. I believe, actually all transponders send now/next but only one gives 9-day for both sats - this is the same on Dish.

It also uses Huffman compression.

How would I go about creating a new module to begin working on it? It's been a few months since I looked at it (work interfering with fun again...)

Norm

Norm,

take a look at src/epggrab/module/eit.c. This is the EIT handler, but it has 2 pseudo modules called uk_freesat and uk_freeview. The purpose of those pseudo modules is 3 fold:

1. It makes it clear to users that those specific services are supported (gives them a warm fuzzy feeling ).
2. It enables me to decide whether or not to use the huffman decoding (I'm concerned that since they use a private field value to define this is being used some other network might existing in the future that uses the same value to mean something else, this is my way out).
3. They pre-define which multiplexes and PIDs carry the relevant info. Note: they don't create multiplex instances, you have to already have done that.

1. You'll need an entry in eit_init() to create the new module. Copy, paste and change names sensibly.

2. If you know which muxes carry the extended info put something in _eit_enable() to pre-register the muxes with the EPG scanning system. Otherwise it will just have to wait until something gets detected and will then be logged in the config (which persists a restart, so they will be added again anyway next time).

3. Then you if NA uses non-standard PIDs you need to register them in _eit_start(). There is some trickery goes on in here because the uk_free* modules are not real modules. Ignore that I can help there if anything needs tweaking.

4. If NA is using huffman encoding as you suggest you need to modify _eit_get_string_with_len() to add the relevant rules. I can help with that. For free* this involves registering a custom string conversion routine called freesat_huffman_decode() if the string format byte is 0x1f. I'd need some more input on how the strings are presented to offer some advice on what is necessary here.

5. And finally you need to implement the huffman decoding for NA. Depending on what format your tables are in, there are already 2 huffman decoders in TVH, the freesat one which is one I nicked from elsewhere and is intimately tied to the optimised freesat tables (it can be re-used you just need to know how to optimise the tables, and I don't ). The other is a very simplistic huffman decoder I wrote that uses the basic dictionary with no optimisation. If you have the tables and convert them to the appropriate format that would probably be easy to use.

That might sound like a lot, but in reality 1-3 are one liners, as most likely will be 4. The only real work is in 5. The system will take care of the rest for you.

Adam

P.S.
Might be worth posting your perl script? might be able to help offer better advice based on that (assuming I can decipher your perl code ).

Norm,

Can you just attach the code to this issue, I've no idea what that site was you've put it on but I can't download stuff from it.

Adam

Sure.. I removed the post to avoid confusion.

I will take a look at this and see what I can come up with.

Norm

Actually here is the tar/gz file since you will need the SI::Parse module as well.

Norm

You sadist!

I can't believe you wrote all that in perl :p

So looks like you need to enable a few custom PIDs (which are obviously hidden in the SI module) specifically on various networks. On issue (at present) is that TVH does not record NID, so best you can do is register network name.

Also looks like DISH are using some custom descriptors, they'll be easy enough to add.

And you have some huffman compressed strings that are being decoded by library code, so will need to understand what's in there.

But this should definitely be doable and include in TVH would be a much better solution in my opinion :0

If you want to have a go I'm happy to help with the basic structure etc.. if you can do the guts.

Adam

I certainly didn't write this all my self - I started out with code that had most of the work done already but for Mythtv

I will give it a go and see what I come up with and go from there. When I get stuck I will call on you.

Norm

Had a quick look at the huffman code, it has similar optimisations to Freesat, one day I will try and unify these. However it uses a detection (i.e. how it knows its huffman encoded) spec similar to opentv, i.e its only used in custom descriptors and varies depending on table.

I would just translate the perl to C copies the tables in and create src/epggrab/support/dish_huffman.c. And we'll find a way to hook in, I didn't think about this at the time and know its easy enough to do (I just forget how ).

Adam

Wow, ok I got two new eit modules setup - na_bell and na_dish with all the various links in place. The last piece as you said is the hardest and that's the huffman stuff. I created a new file called naeit_huffman since its the same for dish and bell.

Will update you as I get along.

Norm

Adam, I'm looking at the huffman tables in hts versus the tables I have in my script's SI::Parse package and also in vdr.

Hts breaks it down as:
struct naeittab {
unsigned int value;
short bits;
char next;
};

Where as I have:
struct HuffmanTable {
unsigned int startingAddress;
unsigned char character;
unsigned char numberOfBits;
};
static HuffmanTable Table128[SIZE_TABLE_128];
static HuffmanTable Table255[SIZE_TABLE_255];

Then 2 tables, Table128 and Table255 with a series of elements in each.

How would I translate into what HTS needs?

Norm

Norm,

At this stage, for simplicity I would just translate the algorithms.

Adam

Easy for you to say -- I'm not even certain I know what its doing to decompress it. Also the programs I have for reference incorporate the categories/genres, etc., into them so I would have to pull that apart as well. This is where I got stuck before too LOL

Norm

Adam,

Ok, I'm trying to convert the c++ code from a vdr plugin that works with NA eit to HTS. I'm having trouble with the code below - specifically the getBits routine where it does a union. I'm not that familiar with how c++ does a union nor how I would redress it here... can you offer any suggestions on it or any of the other bits here?

  static unsigned int getBits(int bitIndex, int bitCount, const unsigned char *byteptr,
      int length) {
    union
    {
      unsigned char b[4];
      unsigned long val;
    } chunk;
  }

size_t naeit_huffman_decode(char *dst, size_t* dstlen, const uint8_t *data, size_t datalen)
    {
        const char *str = data;
        const unsigned char *cmp = NULL;
        int length = 0; // Length of compressed data
        unsigned int dLength = 0; // Length of decompressed data
        if((str[3] & 0xFC) == 0x80){
            length = str[1] - 2;
            dLength = (str[2] & 0x40) ? ((str[3] << 6) & 0xFF) | (str[2] & 0x3F) : str[2] & 0x3F;
            cmp = str + 4;
        }else{
            length = str[1] - 1;
            dLength = str[2] & 0x7F;
            cmp = str + 3;
        }
        if(length <= 0 || !dLength)
            return NULL;

        //char* dst = char[dLength + 1];
        struct naeittab *table;
        unsigned int tableSize, numBits;
    if (data[1] == 1) {
        table = Table128;
        tableSize = sizeof table / sizeof *table;
        numBits = 11;
    } else {
        table = Table255;
        tableSize = sizeof table / sizeof *table;
        numBits = 13;
     }
        int bLength = length << 3; // number of bits
        int currentBit = 0, count = 0;
        while(currentBit < bLength - 1 && count < dLength){
            // Find the interval containing the sequence of length numBits starting
            // at currentBit. The corresponding character will  be the one encoded
            // at the begin of the sequence.
            unsigned int code = getBits(currentBit, numBits, cmp, length);
            // We could use a binary search, but in practice this linear search is faster.
            int index = 0;
            while(table[index].startingAddress <= code && index < tableSize){
                index++;
            }
            index--;
            dst[count++] = table[index].character;
            currentBit += table[index].numberOfBits;
        }

        dst[count] = 0;
    *dstlen = count;
        return 0;
    }

Adam, you can ignore the above post as I have gotten it to compile (I was missing part of the code relating to the union!).

The issue I'm having right now is getting it to be called. I have made the changes to eit.c as you suggested but the naeit_huffman_decode never seems to get called?

/* ************************************************************************
 * EIT Event descriptors
 * ***********************************************************************/

static dvb_string_conv_t _eit_freesat_conv[2] = {
  { 0x1f, freesat_huffman_decode },
  { 0x00, NULL }
};

static dvb_string_conv_t _eit_naeit_conv[2] = {
  { 0x1f, naeit_huffman_decode },
  { 0x00, NULL }
};

/*
 * Get string
 */
static int _eit_get_string_with_len
  ( epggrab_module_t *m,
    char *dst, size_t dstlen, 
        const uint8_t *src, size_t srclen, char *charset )
{
  dvb_string_conv_t *cptr = NULL;

  /* Enable huffman decode (for freeview and/or freesat) */
  m = epggrab_module_find_by_id("uk_freesat");
  if (m && m->enabled) cptr = _eit_freesat_conv;
  m = epggrab_module_find_by_id("uk_freeview");
  if (m && m->enabled) cptr = _eit_freesat_conv;
  m = epggrab_module_find_by_id("na_bell");
  if (m && m->enabled) cptr = _eit_naeit_conv;
  m = epggrab_module_find_by_id("na_dish");
  if (m && m->enabled) cptr = _eit_naeit_conv;

I added some tvhlog commands in the naeit_huffman_decode sub but they never get called so not sure what is happening. I also defined the EIT_EPG_TRACE flag as I saw there was debug built in but even then I don't see any output. On the frontend I now have NA Dish and NA Bell as OTA modules, I enable them and I see in the debug log:

Oct  6 09:39:21 tv tvheadend[31085]: na_dish: install table handlers
Oct  6 09:39:21 tv tvheadend[31085]: na_bell: install table handlers
... nothing of interest between here other then epg expiring items
Oct  6 09:49:21 tv tvheadend[31085]: na_dish: processing complete
Oct  6 09:49:21 tv tvheadend[31085]: na_bell: processing complete

EPG still has items in the database from my xml load but same results if I remove the database and start from scratch. Any suggestions where to look?

Norm

Indeed,

the problem you have is that piece of code is designed to work with standard EIT fields, this requires that they use a field that allows them to indicate the string is encoded using a custom encoding (that's what the 0x1f is for).

However NA (much like opentv) do not use the the standard descriptors to carry the strings (they use their own custom ones) therefore they don't need to indicate such a thing (its implicit from the descriptor type).

Therefore you will need to alter the NA descriptor handlers to go straight to doing a huffman decode on the appropriate fields (taking the length etc.. into account).

Adam

Woohoo ... got it working for NA Bell - still testing for NA Dish but based on the dvbsnoop info, the table id's are the same so it 'should' process as well (famous last words? LOL). The table id's for the extended eit is in the range of 0x81 to 0xa4 so it was a quick change to get it to process the events. My guide 'mostly' filled within a few minutes - there are a few empty spots so I will let it loop for a while and see if it fills in the blanks.

Once I have it all tested and working I will issue a pull request Thanks for your input Adam.

Norm

Norm,

That's great news, I look forward to seeing the PR

Adam

So it turns out that NA Bell doesn't use compression while NA Dish does. My decompress routine is not working properly in that I'm getting results that are jumbled and only partial words - Not entirely sure I know why yet but I will continue to look at it. I spoke to soon in thinking I had it figured out

Norm

So I finally got the huffman decoding working for Dish and Bell. The issue was that the data being sent to the converter was 'incomplete' for it to do its job. I had to do the following in eit.c:
case DISH_DVB_DESC_NAME:
tvhlog(LOG_DEBUG, mod->id, "Processing prog short description tableid:0x%02X",tableid);
dllen += 6;
ptr -= 6;
r = _eit_desc_short_event(mod, ptr, dlen, &ev);
dllen = dlen;
// _eit_dtag_dump(mod, dtag, dlen, ptr);
break;

DISH_DVB_DESC_NAME is a new dtag that only dish uses - there are a few others as well that I need to look at.

Interestingly, when I add in the extended description the data for the results for the short description come out corrupted. Something interesting is happening with the pointer (ptr) that isn't clear to me right now. As you can see I had to roll the ptr back 6 bytes just to get it to decode it with accurate results. I have even tried to roll forward the ptr the number I rolled back after I run the _eit_desc_short_event but it made no difference. I don't understand how turning on the extended description can corrupt the short description for an event - what is the difference between the two? I assumed the extended is where you would put the extra or long description for the event?

Thanks
Norm

Norm,

Where are you managing this code? is it on github so I can see what you're doing etc...?

Adam

Unfortunately I don't have it on git since its just me hacking around with it. I did download the latest git and I guess there have been many changes to eit.c? I will branch the latest git and apply my changes and let you know - I tried this the other day and the changes made seem to affect the changes I made so need to work through it.

I have a better idea of what the issue is and I need to find someone with more knowledge of the dish stream to determine my next steps.

One other thing I've noticed is that there are a number of channels that don't get eit. This is something known with dish in that there is expected matching regardless of the network id - they do this so that they can have one eit stream for all sats.

Bet you wish I just did the XML stuff now LOL

Norm

Adam, I've got most of this working but have run into a few smaller things.

What format does the 'crid' take for program ID and episode/series from here:
if (crid) {
if (strstr(buf, "crid://") == buf) {
strncpy(crid, buf, clen);
} else if ( *buf != '/' ) {
snprintf(crid, clen, "crid://%s", buf);
} else {
char *defauth = svc->s_default_authority;
if (!defauth)
defauth = svc->s_dvb_mux_instance->tdmi_default_authority;
if (defauth)
snprintf(crid, clen, "crid://%s%s", defauth, buf);
}

Specifically what are you expecting buf to contain here? NA EIT does this quite differently and I have the data - just need to match the format that TVH is expecting.

I will start with that and see where it goes. By the way, I have effectively integrated the changes with the latest code which includes the onid. That piece of code you added actually helps me out considerably in that it allows me to differentiate between multiple sources more easily. I did have to manually edit the mux files to added in the onid value to get it to work though - the autoscan did not do it.

Norm