Tvheadend

You can help finding the error by going through the versions. i would pick a point between the bad and good version and start from there, if it's an issue pick a point between this and the good version and so on. Basically you're halving the search space every time until you find the commit that introduced it.

Hi guys

Please share deb file download link for 4.3-1857 jessie and bionic.

Thanks

Build instructions are in the Wiki, Readme.md links to them. Just check out an older revision.

Maybe not everyone knows how to check out a specific version. It's not so easy
because the TVHeadend version numbering is different.

First of all: We need to find the unique identifier for every commit.

A) Short version only with git

1) clone the latest master : 

git clone https://github.com/tvheadend/tvheadend.git

Then simply change to the tvheadend dir

cd tvheadend 

2) look into the commit history log:

git log

Now you will see all commits - every commit has a unique identifier

Lets choose e.g. a version which was valid on 1st Dec 2019. The last commit before was

"commit 221c29b40b1e53ae09a69d9458442dd4fea665f5
Author: Flole998 <[email protected]>
Date: Sat Nov 23 17:11:58 2019 +0100

Fix #5782
" 

3) So we checkout this version unsing the hash.

git checkout 221c29b40b1e53ae09a69d9458442dd4fea665f5

4) now build your specific version

configure && make 

5) stop, install and restart tvh. 

systemctl stop tvheadend.service && make install && systemctl start tvheadend.service

Step 5 may be different if your are not using systemctl to control init procdures.

You can install without stop service it will autorestart

saen acro wrote:

You can install without stop service it will autorestart

Maybe - but systemd is bullshitish in many,many ways - so it is
better to explicitly start and stop it.

I have the very same problem. However I'm using the linuxserver/tvheadend (latest) docker container which appears to be the same version mentioned, 4.3-1896~gce0907705.
Reading that version 4.3-1857~g221c29b40 did not have this problem I went searching previous versions of linuxserver/tvheadend in the docker hub and found: linuxserver/tvheadend:221c29b4-ls61 to be that exact version.
This version is running rather smooth with sometimes a very short glitch. With version 4.3-1896 the distortion took on average 15 seconds.

My setup is: Synology DS415+ with tvheadend on docker and OSCam, SAT>IP (Telestar digibit twin) and KODI on Android TV (Sony).

The only possible cause that comes to my mind in that timeframe could be https://github.com/tvheadend/tvheadend/commit/25e9c0600b6090335cebee2854bea1f9b2fecaa4, please try reverting that one and test again.

I cloned the current master and reverted the commit you mentioned

git revert 25e9c0600b6090335cebee2854bea1f9b2fecaa4

Never compiled tvheadend myself, but finally I managed it. Hard work to remove a few lines of code.

Unfortunately the errors still raise up in the same amount. Tomorrow I will try to reset to commit 749f519 and compile again...

Wait a second, you are using a FritzBox? Is that a 6591?

6590, which ran stable for months (even with multiple streams hd+uhd), before I updated Ubuntu.

Also I did not update the firmware. The only thing that should have changed was that I updated my tvheadend server with apt. Rolling back tvheadend did not solve the issue. I could only try to jump back to a backup I made with timeshift, to make 100% sure it has to do with the software of my server.

First you should check with Wireshark if the RTP Stream is complete or if (for some reason) the FritzBox is sending out an incomplete/broken stream. If on Wireshark it is fine (that means the traffic reached your server correctly) then you can start looking where it got dropped.

Thank you for the guidance.

I installed Wireshark, and observed the RTP traffic going from my fritzbox to my tvheadend server, but I didn't see anything obvious. How can I detect if "the traffic reached my server correctly"?

In the meantime I made the observation that this phenomenon exclusively happens if the descambler is active (I use oscam for this). I have updated oscam now to the most recent version, but it hasn't changed anything. The oscam log also looks fine to me.

You start the capture, AFTER that you start the channel, wait for the errors, stop the channel, stop the capture. Click on Telephony -> RTP -> RTP Streams, wait for it to load, select your Stream, click on "Analyze" and a Window will pop up with the packets. You can then sort by status and it will show you that packets are missing (Status will be "wrong sequence number" for the following packet) and they will also be red. Also on the left it will show Lost count (on the left under the "Forward" Section).

You can go one packet up and see if the RTP Sequence Number Counter increased by one or if a RTP packet got lost. Then you know if it's a reception problem or network problem.

If you run a capture on the FritzBox itself you can see if the packets are already lost there or if they get lost in your network somewhere. Also with UHD Channels the FritzBox can cause issues.

Generally the FritzBox SAT>IP Server is not good in my opinion, getting a 6591 won't be better (at the moment).

Thanks, the RTP sequence number of the previous packet is 43261, so it means the tvheadend server never receives the missing packages.

Yes the Fritzbox is not that good for multiple streams or uhd streams. But it had never that much problems with a single hd stream. There are firmware mods, which overcome this problems. For whatever reason this is not working anymore, I think its time for a new tuner.

Do you have any recommendations for an external quad DVB-C tuner that works good with tvheadend?

That would indicate a network problem rather than a FritzBox problem in my opinion. I'm not sure why the FritzBox would increase the sequence counter without ever sending out the packet. If I remember correctly the classic issue was not causing sequence errors in RTP, so the stream itself was perfectly fine and the FritzBox was skipping MPEG packets.

It could also be a driver issue on your server, if the Ethernet card or driver drops the packet it would show the exact same thing. Maybe a driver update was done in the meantime aswell.

I am using an ethernet controller with RTL8125b chipset and ubuntu 18.04. I have read that this chipset is supported natively from kernel 5.4 upwards, which is available at Ubuntu 20.04. I will give a dist upgrade a try, because then this controller shouldn't need a separate driver.

I think I found the reason in my case. The implementation of the RTL cards into the Linux Kernels was associated with a lot of changes in the drivers of these chipsets (Kernel 5.3+). In my case it was not solved by upgrading the kernel. Instead I found this ubuntu issue

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1880076

and investigated the output of "ethtool -S enp3s0" and discovered a lot of "rx_missed"-Errors. The solution was to disable the "aspm"-settings of the pcie-controllers inside the Bios of my server. Since the RTL cards are common, it might be helpfull to someone else having issues like that.

What remains is that the counting for continuity errors in "Status->Streams" differs drastically from the ones logged in the tvheadend log. This appears only on some channels for me, such that I have a counter of continuity error in the order of 10000 in "Status->Streams", whereas no continuity error has been logged inside the tvheadend log (see also the screenshot in #5937-11). If I record such a channel in "Digital Video Recorder->Upcoming/Current Recording" I see 0 Errors and 0 Data Errors. What could be the reason for this deviation?

I assume you're seeing this for example on 370MHz?

The deviation can happen when there is a Continuity Error in a Stream other than Audio/Video/HbbTV/Teletext/Subtitle. For example on the CAID-Stream used for descrambling. If my guess which provider you're using is correct using is correct it is affected by that issue.

Yes, you are perfectly right again, the affected frequencies belong all to the same provider for which an oscam descrambler is active (via dvbapi).

Is it something to worry about, because I see no negative consequences in my reordings? Or is there any way to fix this?

I have enabled debugging and all errors show the same structure

tbl-csa: emm: 370MHz in Fritzbox: PID 1509 CC error 12 != 7

I have no clue what that means, but some comparison seems to fail, which leads to all these counted CC errors, but not in a interruption of the stream.

And it's failing because there are packets missing. As the keys are still received properly it's still working.

The only thing you can do to fix this is contact the provider, nobody else will be able to repair the broken datastream. You can't fix this in Tvheadend, it's not a Tvheadend issue. And they will probably not understand the issue, so good luck with that.

OK, thank you for your helpfull support so far. Since I don't see any negative consequences of these errors, I will live with it. Maybe some day they are gone, when the provider changes its stream again.

Since these CC errors have no negative consequences it would be very nice to have a workaround. Perhaps a checkbox in the network or mux to disable the counting.

You should contact your provider and ask them to fix their stream. These are continuity errors and they should be counted. If you don't want to see them in the Webinterface just disable that column.

Flole Systems wrote:

Closing this for now until someone else who reported problems is able to provide further debug information that shows it's a Tvheadend issue.

Actually there should be no doubt that this is an tvh issue.

-> old tvh version: some CC glitches , some counted CC errors

Nothing changed !

-> new tvh version: some CC glitches, numerous counted CC errors.

The counter goes up and up and up....

I don't have a clue what this counter now counts: for sure not CC errors.

It counted CC errors in a CAID stream in those cases.... As already mentioned above (and there is even a screenshot of Wireshark showing the CC error). You still sure it's not counting CC errors?

Flole Systems wrote:

It counted CC errors in a CAID stream in those cases.... As already mentioned above (and there is even a screenshot of Wireshark showing the CC error). You still sure it's not counting CC errors?

Besides the simple fact that "CC Error" is a well-defined terminus technicus is was already stated that there are major differences between logged and now counted CC errors. The WS Screenshot timestamp doesn't match the timestamps from the log - so it's difficult to decide what has been counted.

Unfortunatly it seems that the meaning of the counter has changed whithout adaptation of the release numbers. That's really bad, because real CC Erors are often correlated with attenuation issues. Environment debugging is getting more and more difficult.

The Wireshark screenshot shows one of numerous examples. I have checked it on my Installation and confirmed the issue. It is also logged if the appropriate logging is enabled, so that claim is also not true (there is also a line from the logs that shows it above).

The meaning of the counter did not change (it indicates that the mpeg continuity counter was not continuos), if a provider provides a faulty signal this will happen. You should then report an issue to your provider because they are violating the specification.

There is no way to distinguish missing packets that were dropped at the provider from missing packets caused by bad signal.

I apologise if this adds 'noise' without anything especially technical backing it up - however, I built a stripped down version of TVheadend and it seems to have gone a long way to improving things for me.

My setup is a Sat>IP client, I then export the M3u from TVheadend to something else. So I have very basic needs, TVheadend is essentially a UI/frontend for my Sat>IP server, it does no recording, transcoding, etc.. When I built with the below options, I no longer see the spikes of uncorrected errors.

AUTOBUILD_CONFIGURE_EXTRA="--python=/usr/bin/python3 --disable-pcloud_cache --disable-ddci --disable-tvhcsa --disable-epoll --disable-ffmpeg_static --disable-libav --disable-libx264_static --enable-libx264 --disable-libx265_static --enable-libx265 --disable-libvpx_static --disable-libtheora_static --disable-libvorbis_static --disable-libfdkaac_static --disable-libopus_static --disable-nvenc --disable-hdhomerun_static --disable-hdhomerun_client --disable-cardclient --disable-cwc --disable-cccam --disable-capmt --disable-constcw --disable-linuxdvb --disable-satip_server --disable-iptv --disable-dvbscan --disable-timeshift --disable-trace --disable-avahi --disable-inotify --disable-ccache  --disable-pie" ./Autobuild.sh -t focal-amd64

I don't truly understand all the ramifications of the above, but what it does seem to do, is stop the large spikes of uncorrected errors. I now perhaps see 1 instance of minor pixelation per evening, compared to multiple and/or the odd complete stall. I did also experiment with the cpu frequency scaling I saw in another bug report, whilst it improved things, did not help as much as the above. Would be interested to know if anyone else sees the same.

I've been running this for almost a week now and the improvement is definitely significant in my use case. I now perhaps get 20-30 uncorrected per 1 hour TV programme and maybe 2-10 continuity errors - rather than 10's of thousands of the former.

I guess the next step for me is to start adding options back in again, to try to determine the case.