Bug #3810
Digital Devices SAT>IP server reports a lot of errors
0%
Description
Hi,
my SAT>IP server Digital Devices octopusNET S2/4 reports a lot of Uncorrected/Transport/Continuity errors during operations. Errors do not affect live view (kodi client) but I found a lot of planned recording failed.
SAT>IP tuners (4) are connected to the same multiswitch used by my others PCie tuners, so LNBs, multiswitch and cables are not the sources of these errors. Digital Devices support told me to ask here, because it seems to be a tvheadend problem.
I do not know with subsystem log I need to activate to help debugging, but my system is not yet in production, so tell me what you need and i'll produce it.
Thanks
Gigius
Files
History
Updated by Jaroslav Kysela over 8 years ago
It looks that some UDP packets are lost on the network. You may verify this using the '--trace satip' trace subsystem - https://tvheadend.org/projects/tvheadend/wiki/Traces . If you see 'RTP discontinuity' lines then TVH didn't received all UDP packets.
You should tune UDP buffers in linux and also tune your ethernet card to receive all UDP packets.
Also, you may try if digital devices supports the RTSP/TCP data mode - RTSP/TCP (embedded data) field in the tvh config.
Updated by Gigius 2016 over 8 years ago
Jaroslav Kysela wrote:
It looks that some UDP packets are lost on the network. You may verify this using the '--trace satip' trace subsystem - https://tvheadend.org/projects/tvheadend/wiki/Traces . If you see 'RTP discontinuity' lines then TVH didn't received all UDP packets.
You should tune UDP buffers in linux and also tune your ethernet card to receive all UDP packets.
Also, you may try if digital devices supports the RTSP/TCP data mode - RTSP/TCP (embedded data) field in the tvh config.
Hi Jaroslav,
Yes, I have a lot of 'RTP discontinuity' errors in the log. Tryed to tune IP stack (up to 8MB receive buffer) but nothing changes.
netstat -su report no errors:
===
root@tvheadend:~# netstat -su
IcmpMsg:
OutType3: 172
Udp:
1249746 packets received
256 packets to unknown port received.
0 packet receive errors
3714 packets sent
UdpLite:
IpExt:
InMcastPkts: 2587
OutMcastPkts: 121
InBcastPkts: 4457
OutBcastPkts: 2788
InOctets: 1687295848
OutOctets: 20959117
InMcastOctets: 588821
OutMcastOctets: 7936
InBcastOctets: 237304
OutBcastOctets: 133824
InNoECTPkts: 1367319
===
but when I do a 'force network scan' (grep discontinuity) I got:
===
2016-05-18 11:02:52.891 [ TRACE]:satip: RTP discontinuity (27209 != 27210)
2016-05-18 11:02:53.287 [ TRACE]:satip: RTP discontinuity (27212 != 27214)
2016-05-18 11:02:53.683 [ TRACE]:satip: RTP discontinuity (27217 != 27218)
2016-05-18 11:02:53.980 [ TRACE]:satip: RTP discontinuity (27220 != 27221)
2016-05-18 11:02:54.277 [ TRACE]:satip: RTP discontinuity (27222 != 27224)
2016-05-18 11:02:54.475 [ TRACE]:satip: RTP discontinuity (27225 != 27226)
2016-05-18 11:02:54.673 [ TRACE]:satip: RTP discontinuity (27227 != 27228)
2016-05-18 11:02:55.267 [ TRACE]:satip: RTP discontinuity (27231 != 27234)
2016-05-18 11:02:55.663 [ TRACE]:satip: RTP discontinuity (27236 != 27238)
2016-05-18 11:02:55.861 [ TRACE]:satip: RTP discontinuity (27239 != 27240)
2016-05-18 11:02:56.158 [ TRACE]:satip: RTP discontinuity (27241 != 27243)
2016-05-18 11:02:56.356 [ TRACE]:satip: RTP discontinuity (27244 != 27245)
2016-05-18 11:02:56.653 [ TRACE]:satip: RTP discontinuity (27246 != 27248)
2016-05-18 11:02:56.851 [ TRACE]:satip: RTP discontinuity (27249 != 27250)
2016-05-18 11:02:57.148 [ TRACE]:satip: RTP discontinuity (27251 != 27253)
2016-05-18 11:02:57.643 [ TRACE]:satip: RTP discontinuity (27256 != 27258)
2016-05-18 11:02:58.039 [ TRACE]:satip: RTP discontinuity (27260 != 27262)
2016-05-18 11:02:58.237 [ TRACE]:satip: RTP discontinuity (27263 != 27264)
2016-05-18 11:02:58.534 [ TRACE]:satip: RTP discontinuity (27265 != 27267)
2016-05-18 11:02:58.991 [ TRACE]:satip: RTP discontinuity (27270 != 27272)
2016-05-18 11:03:10.977 [ TRACE]:satip: RTP discontinuity (27431 != 27433)
2016-05-18 11:03:21.039 [ TRACE]:satip: RTP discontinuity (28099 != 28101)
===
I'll try to bypass my two switches to see if the problem is there!
Thanks
Gigius
Updated by Gigius 2016 over 8 years ago
22.05.2016
First switch bypassed without result. Waiting on an ordered low-profile network card to bypass the second :-)
Gigius
Updated by Ricardo Rocha over 8 years ago
Gigius 2016 wrote:
22.05.2016
First switch bypassed without result. Waiting on an ordered low-profile network card to bypass the second :-)
Gigius
any news on your problem?
Updated by Gigius 2016 over 8 years ago
Ricardo Rocha wrote:
Gigius 2016 wrote:
22.05.2016
First switch bypassed without result. Waiting on an ordered low-profile network card to bypass the second :-)
Gigiusany news on your problem?
Hi, sorry for be too late. I've tested with and without network switches, and I also bought another SAT-IP tuner (DVB-T this time, always from digital devices. The sat-ip part is the same, only the tuners change). Problem is always there. Tuning the linux IP stack does not make any changes. A lot of errors but they seems to be 'cosmetic errors. I mean TV watching and recording does not suffers from these 'RTP discontinuity' errors.
Also tested tvheadend on physical (actually Production machine) and on a VMware virtual machine.
Tests done:
- changed the sat-ip tuner
- changed the cabling (end-to end)
- with or without network switches on the path
- tuning the ip stack (Buffers)
- Firmware upgrade on the two tuners (latest beta)
RTP Discontinuity always there but it seems without impact on TV Watching.
Thanks
Gigius
Updated by seb min over 8 years ago
Same problem here.
Latest stable FW on the Octopus Net V2 box:
FW Date 30 Jul 2016 18:44
MAC 54:84:7b:00:32:52
Linux 3.17.7 armv5tejl
SAT>IP Server 1.0.71
FPGA 300-0.46
Boot ID 1
Device ID 5
Tuner 1 DVB-C/C2
Tuner 2 DVB-C/C2
Tuner 3 DVB-C/C2
Tuner 4 DVB-C/C2
CableTV <-> Octopus net <-> Gigabit Switch <-> debian8.5(with tvheadend stable)
I can reproduce the problem with 2 recording running at the same time.
Recording the same two channels with vdr+satip plugin works without any error.
If you need additional info, let me know.
Thanks!
Updated by Gigius 2016 over 8 years ago
seb min wrote:
Same problem here.
Latest stable FW on the Octopus Net V2 box:
FW Date 30 Jul 2016 18:44
MAC 54:84:7b:00:32:52
Linux 3.17.7 armv5tejl
SAT>IP Server 1.0.71
FPGA 300-0.46
Boot ID 1
Device ID 5
Tuner 1 DVB-C/C2
Tuner 2 DVB-C/C2
Tuner 3 DVB-C/C2
Tuner 4 DVB-C/C2CableTV <-> Octopus net <-> Gigabit Switch <-> debian8.5(with tvheadend stable)
I can reproduce the problem with 2 recording running at the same time.
Recording the same two channels with vdr+satip plugin works without any error.If you need additional info, let me know.
Thanks!
Hi seb, thanks for your report.
If with vdr all works good, something should relly be wrong with tvheadend.
Not less important, with firmware 1.0.71 octopusNET devices are finally SES SAT>IP certified.
Thanks
Gigius
Updated by seb min over 8 years ago
Gigius 2016 wrote:
Hi seb, thanks for your report.
If with vdr all works good, something should relly be wrong with tvheadend.
Not less important, with firmware 1.0.71 octopusNET devices are finally SES SAT>IP certified.
Thanks
Gigius
After some more testing I found that vdr also produces those stream errors, it just did not log them.
But the recording were full of errors.
After I got a little frustrated over the situation I pulled some old hardware out and tried connecting the DD Octopus directly to it.
Steps:
1. Fresh install of debian8.5 + HTS Tvheadend 4.0.9-7~gbf276e0~wheezy on an old asus board with embedded intel atom cpu and Realtek LAN Chip
2. Connecting the DD Octopus NET V2 (Cable) directly to the debian box (To one of the Gigabit ports of the DD device).
3. Connecting the DD device with another Gigabit Port to my Router/Rest of the network.
And with that setup all problems went away.
No more stream errors at all within tvh!
So my guess it that something in the LAN is the problem, as other ppl already reported problems with Intel NICĀ“s, that might be also the problem for me. The Intel NIC setup was full of stream errors. The new setup with Realtek NIC is just fine.
Models of the NICs:
Produces problems for me:
Intel(R) 82579V Gigabit Network Connection
Fine for me with tvh:
Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)
I will try to put my gigabit switch in between the realtek NIC and the DD Box to see if that will cause trouble again.
Will report back then.
Updated by seb min about 8 years ago
The gigabit in between is also NOT a problem, streams are still error free.
@ Gigius 2016: I would recommend you to test a different NIC.
Maybe it also helps to disable some of the Intel NIC feature such as UDP Offloading (Just a guess).
Updated by Gigius 2016 about 8 years ago
seb min wrote:
The gigabit in between is also NOT a problem, streams are still error free.
@ Gigius 2016: I would recommend you to test a different NIC.Maybe it also helps to disable some of the Intel NIC feature such as UDP Offloading (Just a guess).
Thanks Seb, I'll try as soon I'll find an 'non-intel' network card in my stock :-(
Regards
Gigius
Updated by seb min about 8 years ago
seb min wrote:
The gigabit switch in between is also NOT a problem, streams are still error free.
@ Gigius 2016: I would recommend you to test a different NIC.Maybe it also helps to disable some of the Intel NIC feature such as UDP Offloading (Just a guess).
Updated by Anonymous over 7 years ago
seb min wrote:
seb min wrote:
The gigabit switch in between is also NOT a problem, streams are still error free.
@ Gigius 2016: I would recommend you to test a different NIC.Maybe it also helps to disable some of the Intel NIC feature such as UDP Offloading (Just a guess).
I use a Intel NIC without a issue with default settings (No feature like UDP offloading disabled).
Intel NIC:
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM (rev 31)
Kernel:
Linux Server 4.9.19 #1 SMP PREEMPT Thu Mar 30 08:32:28 PDT 2017 x86_64 Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz GenuineIntel GNU/Linux
Had the same issues in the beginning as my Tvheadend erver runs in docker container ("bridge" network mode). In "bridge" mode (NAT & portforwarding is used) you may have these "satip: RTP discontinuity" problems with artifacts.
After setting the Tvheadend docker container to "host" network mode or better to "macvlan" (since docker v1.12), the problem went away. No Continuity errors anymore.
Running a recording since ~52 minutes with zero errors: