Tvheadend

Again, set the ignore flag for the overlapping muxes. Unfortunately, the data from broadcasters are heavy inconsistent. I need to find a better way to handle this.

Ignoring rather than deleting would be fine with me, but the hard part is that channel mappings may be spread across duplicate muxes. If all channel mappings could be gathered into a single mux of a duplicate set, then all would be fine.

It's very hard to see which mux is "better".
I checked a set of 10758.5V muxes on Astra 1, and the only difference was Stream ID. 0 on one, -1 on another.

With Stream ID, I mean "ISI".
Transport Stream ID is 1052 for both.

I don't see an ISI parameter for muxes on Lyngsat or Kingofsat or in /usr/share/dvbv5/dvb-s/Astra-19.2E.
Then how could such a minor difference cause a new mux to be created?

The old tvh had the ISI (Stream ID) value as zero by default. The recent code sets the default to -1 (which is more appropriate - no filter - zero value has unknown behaviour). If the stream id number is different, tvh thinks that it's different mux, so new mux is created. Just set this value to -1 for you old muxes and everything will be fine. If Lyngsat does not specify the ISI number, it means 'no multistream' -> 'no filter' -> '-1 value in tvh'.

My point is that every change to a mux, either from the network or from changing tvheadend defaults, cause a new mux to be created. Along with duplicate services. Why not just update the existing mux? Did that break something in the past? If so, could it become per-network optional behaviour?

I would opt for it.
The current behaviour is less then ideal.
The most currently created mux is obviously the "best", but all services mapped to channels remain in the old mux.
You end up deleting/disabling/ignoring the "better" new mux, and keeping the "bad" old one because remapping channels is such a pain.
You lose your link to recordings, timers, autorecs, icons, EPG sources and maybe more.

If there is no way around duplicate muxes getting created then I could think of two ways to ease the pain:
- a tool that could transfer services mapped to channels to another mux
- a comparison tool that would allow one to copy properties from one mux to another, sort of like Beyond Compare. Or a smart one that just works automatically.

The first tool would allow one to move services from the old mux so the old mux can be deleted.
The second tool would allow one to improve the old mux so the new mux can be deleted, without it coming back.

I was thinking of the possibility of writing a python script using the API to perform either.
I doubt transferring services is possible, but improving old muxes might be possible.

Here's my solution for the moment for the issue:

https://github.com/pimzand/tvh-muxdedup

It's a script, using the API to remove duplicate muxes.

Before, I had some 400+ duplicate muxes, now 0, and none are coming back after a scan.
I did have to remap some channels, as they were spread over duplicate muxes.
I also had to disable both external and internal fast scan files, as they appear to be inaccurate, causing bad muxes to come back.

It would help if Tvheadend would add a field to each new mux, describing what caused its creation.

• created from manual entry
• created from external scan file
• created from internal scan file
• created from nit in this other mux (isn't that how it works?)

This could help finding the cause for duplicate muxes and bad new muxes

I have recently seen a mux being duplicated on Astra 2, but on closer inspection the new mux had a different FEC value because the broadcaster actually changed FEC. The old mux definition is now defunct but it is still being referenced in the Channels (Mapped count on the new mux was 0). A dedup algorithm should also consider and resolve such minor differences.

This is exactly what the script was designed to handle. Duplicate muxes are rarely completely identical. They get created because of minor differences, like the FEC value.

The script considers two muxes duplicate if orbital and polarisation matches, and the frequencies are identical of differ less then 1 MHz. There can be only one such mux, and if the values differ, then one of them is wrong. Usually the old one.

The script would copy the FEC value from the new mux to the old one and then delete the new one. The next time tvheadend will scan the network, your old mux should be good again, and no new duplicate should be created because the old mux value now matches the scanned values. And you will not lose your channel mappings.

Even with a brand new installation I get duplicates, presumably because the preset muxes (for Astra 28.2E) uses to bootstrap are outdated and/or have some frequencies which differ from those in the SI due to rounding. Possibly there are also rounding differences between the BSkyB and Freesat SI. These duplicates can easily be detected because they have the same transport_stream_id etc. tvheadend should be filtering them out, and dealing properly with updates itself, not relying on the community to write scripts.