Tvheadend

the problem most likely is a poor design of the power circuit on your dvb card or your lnb. Try to disdable all power saving option in tvheadend so that there are no spikes generated when enabling the second tuner path. tvh is just forwarding what it receives from the kernel driver, if that's garbage it can't do better. Most E2 boxes supply power to the LNB all the time. I'm running tvh with 4 dvb tuners and do not experience this behaviour, but I did have it some time ago with a different dvb card. Also try with latest master as there are constant improvements in the code.

B C wrote:

the problem most likely is a poor design of the power circuit on your dvb card or your lnb. Try to disdable all power saving option in tvheadend so that there are no spikes generated when enabling the second tuner path. tvh is just forwarding what it receives from the kernel driver, if that's garbage it can't do better. Most E2 boxes supply power to the LNB all the time. I'm running tvh with 4 dvb tuners and do not experience this behaviour, but I did have it some time ago with a different dvb card. Also try with latest master as there are constant improvements in the code.

I myself have experienced this in the past too (with a DVBSky card) especially when cables have come loose however Tvheadend shouldn't crash, I have seen crashes/hangs because the driver screws up due to constant interference though.

The OP hasn't mentioned what card they're using.

Actually the same problem was present with two completely differnet DVB tuner cards as well when they were located in two separate HTPCs. Furthermore the same problem is present with three completey different type of LNBs.
Anyway, as I wrote my intention with this ticket is to improve Tvheadend stability. Whatever is the trigger of incoming signal disturbances the Tvheadend should not crash or malfunction.
I have just upgraded to 4.0.7 and this is the latest available addon for Openelec. I am ready to test with newer or debug releases however I cannot build such addons by myself.

I've been struggling with Tvheadend for years now. I would prefer investigation of the actual fault instead of trying newer and newer releases.

I forgot to mention that the attached service.log backup file ends at Tvheadend crash.

Mark Clarkstone wrote:

The OP hasn't mentioned what card they're using.

This is Openelec (Wetek) box with built in tuners.

Akos Sz wrote:

Actually the same problem was present with two completely differnet DVB tuner cards as well when they were located in two separate HTPCs. Furthermore the same problem is present with three completey different type of LNBs.

You got interference with two different DVB cards and has Tvheadend crashed on these systems as well ? low quality cable maybe? Could also be caused by cheap motherboards with bad insulation (I've seen this with some cards).

Anyway, as I wrote my intention with this ticket is to improve Tvheadend stability. Whatever is the trigger of incoming signal disturbances the Tvheadend should not crash or malfunction.

In order to help, perexg (he's the main dev atm) would need debug logs even if Tvheadend is crashing it shouldn't completely kill the wetek so you should be able to ssh in and at least get the crash lines from dmesg or wherever OE dumps them.

I have just upgraded to 4.0.7 and this is the latest available addon for Openelec. I am ready to test with newer or debug releases however I cannot build such addons by myself.

I don't personally use OE as I find it too limiting and I don't think anyone here would be able to give you newer builds either, you may find some on the openelec forum though.

I've been struggling with Tvheadend for years now. I would prefer investigation of the actual fault instead of trying newer and newer releases.

I've been using Tvheadend since version 2.99 and it's worked fine for me and no doubt many others, granted there have been ups and downs lately with major changes to the code which perexg is extremely fast at fixing (I don't know how he does it, his fingers must be on fire most of the time) but in general it works pretty well.

I forgot to mention that the attached service.log backup file ends at Tvheadend crash.

Yes. The OS on wetek is not affected by Tvheadend crash. I can SSH and collect dmesg output.
Is there any further debug options in Tvheadend which provides more info?

actually if you get a crash we need some log, you should see something in dmesg
have you disabled power save?

Unfortunately dmesg has rolled over since yesterday crash. I have to reproduce the fault first then play with settings.

Akos Sz wrote:

I managed to reproduce a crash again. I am not sure if it is completely the same but it is definitely a crash and very similar to yesterdays case. In this case the two tuners were tuned to different satellites (LNBs) hence the interference can be excluded, however the crash happend after a short "Transport error indicator", it involved a descrambler fault and ended up in a TVheadend restart.

I attached a screenshot, the saved backup of service.log and the dmesg output and oscam log. For your reference the crash happened at 18:48 which is at 6195 in dmesg.

Looking at that image something sticks out at me when Tvheadend restarts, "Device or resource busy"! No wonder you're having so many problems you have something on that system that is stealing the cards, VDR installed perhaps?

Mark Clarkstone wrote:

Looking at that image something sticks out at me when Tvheadend restarts, "Device or resource busy"! No wonder you're having so many problems you have something on that system that is stealing the cards, VDR installed perhaps?

Thanks for your response. Actually, VDR backend and client addons are installed but disabled. There is also a VDR configuration addon which were not disabled. Now it is. Let me test the system behavior and come back with the results if this solved the issues.

Akos Sz wrote:

Still "ECM late", crash, crash, crash and now also freeze.

I disabled VDR then completely removed all of its components however I still can find sporadic "DVB-S #0 - failed to config dmx for pid 133 [e=Device or resource busy]"

I captured a new service.log with "*** Error in `tvheadend': free(): invalid next size (normal): 0x02ec9cd0 ***" as last line before crash.
See backup_crash3.log

Today I upgraded the box to Openelec 6.0 (latest stable release). And now I have a new behavior. Tvheadend got stuck. The last log items are
`tvheadend': malloc(): smallbin double linked list corrupted: 0x03339340 ***
2015-11-20 19:50:40.892 [ INFO] htsp: 192.168.10.25 [ htpc2 | Kodi Media Center ]: Write error -- Broken pipe
dmesg is flooded with
[19153.819781@1] balance_pgdat end zone=1, pgdat_is_balanced=0
[19153.819863@1] balance_pgdat, nr_reclaimed=13
[19153.819875@1] balance_pgdat end zone=1, pgdat_is_balanced=0
[19153.819978@1] balance_pgdat, nr_reclaimed=23
[19153.819995@1] balance_pgdat end zone=1, pgdat_is_balanced=0
[19153.820380@1] balance_pgdat, nr_reclaimed=46

See service_freeze.log, output of ps -ef and top.

Now I am going to install previous version of Tvheadend addon and compare the log content and behavior.

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Mark Clarkstone wrote:

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Ok.This freeze happened only once. What abot the crash?
I plan but I have never tested VDR. I put so huge effort into this that I do not want to start everything again.
(e.g. Due to very poor GUI performance at the beginnig, I had to create my own ugly channel list editor for Tvheaend which is still used by many others)

Akos Sz wrote:

Mark Clarkstone wrote:

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Ok.This freeze happened only once. What abot the crash?

It's serious issue probably visible on the Wetek platform only. I have Wetek here, but I've not had a time to do a serious debugging for this platform. I'll try to find a time.

Also, there are many changes in our development tree which may fix this issue (and maybe add others :-)). I'll try to compile the latest for Wetek. But no ETA. Sorry.

Jaroslav Kysela wrote:

Akos Sz wrote:

Mark Clarkstone wrote:

The only thing I can find is this http://openelec.tv/forum/146-wetek-play/78060-balance-pgdat-nr-reclaimed-xx-errors-in-dmesg but that has no responses, it looks like a wetek issue, have you tried anything else? Does VDR work as expected?

Ok.This freeze happened only once. What abot the crash?

It's serious issue probably visible on the Wetek platform only. I have Wetek here, but I've not had a time to do a serious debugging for this platform. I'll try to find a time.

Also, there are many changes in our development tree which may fix this issue (and maybe add others :-)). I'll try to compile the latest for Wetek. But no ETA. Sorry.

OK. Thanks. Waiting for your result. Anyway if you need a debug on my system I am open to run any build provided by you.
FIY this morning I rolled back to TVH 4.0.3 and it seems much more stable.

Also, provide output from 'journalctl' when tvheadend crashes.

There's nothing in logs as in previous. I need to find a way, how to debug tvh on openelec. Could you install gdb ? I saw that there should be gdb packages. Run tvh from cmd line as 'gdb --args <standard_tvh_cmd_line>' . The type 'r' (run) and wait for crash. If it crashes, type 'bt' and provide me these lines.

I am afraid that gdb is not included in OE, neither cen be installed due to RO filesystem.

Do you have coredumps ? https://github.com/OpenELEC/OpenELEC.tv/blob/master/packages/sysutils/busybox/sysctl.d/99-coredump.conf

I am not familiar with this.

OpenELEC:~/.cache/cores #
OpenELEC:~/.cache/cores # cat /proc/sys/kernel/core_pattern
/storage/.cache/cores/core.%E.%t.%p
OpenELEC:~/.cache/cores # ls /storage/.cache/cores/ -a
. ..
OpenELEC:~/.cache/cores #


For me it seems that I have the mentioned settings, however the destination folder is empty. Should I reproduce the fault again and check if something appears in that folder?

From https://tvheadend.org/projects/tvheadend/wiki/Debugging: you should try a preliminar command 'ulimit -c unlimited' and then the extra option '-D' for tvheadend.

I applied the command 'ulimit -c unlimited' and then the extra option '-D' for tvheadend however I could not find coredump after crash.
Tried with '-A' option as well. No coredump found either.

Any further idea?

Last event.

Journal_curl.log:

Dec 05 19:04:11 OpenELEC tvheadend[22413]: epggrab: EIT: DVB Grabber - data completion timeout for 11765.84V in Thor 0.8W
Dec 05 19:04:11 OpenELEC tvheadend[22413]: subscription: 00AA: "epggrab" unsubscribing
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH FIFO_0
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH ok
Dec 05 19:04:11 OpenELEC tvheadend[22413]: TS: Thor 0.8W/12034V/RTL Klub HD Transport error indicator (total 21)
Dec 05 19:04:11 OpenELEC tvheadend[22413]: TS: Thor 0.8W/12034V/RTL Klub HD: H264 #251 Continuity counter error (total 18)
Dec 05 19:04:11 OpenELEC systemd¹: service.multimedia.tvheadend.service: main process exited, code=killed, status=11/SEGV
Dec 05 19:04:11 OpenELEC systemd¹: Unit service.multimedia.tvheadend.service entered failed state.
Dec 05 19:04:11 OpenELEC systemd¹: service.multimedia.tvheadend.service failed.
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH FIFO_1
Dec 05 19:04:11 OpenELEC kernel: wetek-dvb dvb.8: FLUSH ok
Dec 05 19:04:13 OpenELEC systemd¹: service.multimedia.tvheadend.service holdoff time over, scheduling restart.
Dec 05 19:04:13 OpenELEC systemd¹: Started TVHeadend Service.
Dec 05 19:04:13 OpenELEC systemd¹: Starting TVHeadend Service... @

If it helps...

Hi
I have similar experience:(
The latest tvheadend compiled from sources is runing on Intel atom with latest Debian.
I have two DVB-S2 cards:
TechnoTrend TT3200 -PCI
TechnoTrend TT3600 -USB
Connected to one Quad LNB
I can successfully use each card separately.
When both are enabled, second card always produces garbage and I have drops of signal on first card.

Regards
Michał Konopiński

did you disable all power save options on the tuners/lnbs?

In my case "power save" option has no effect on the behavior. The disturbance is present when power safe is disabled on both tuners.

Michal, I opened another ticket. https://tvheadend.org/issues/3356 With that solution the tuners will not disturb each other. Unfortunately it is not fully functional yet.

Just a note: I'm trying to reproduce the crash on x86 (discarding and modifying the incoming MPEG-TS packets to emulate the broken stream) and after three hours of testing - streaming/recording three channels, I've not had any crash based on these wrong data. So things are not so bad. It's probably platform related (or the issue is just more visible on Wetek). It would be much better to run full Linux on Wetek (debian/ubuntu) to have all debugging tools available to resolve this.

I have already disabled all powersave options.

I think, in my case, there is something wrong with hardware(LNB?).
I configured tvheadend to use only one card.
Every time I stop or start dvbv5-scan(powering and depowering LNB), on second card I see damaged stream on first card for several seconds

I had similar problem with an old twin LNBs: damages on the stream when switching on the brother LNB.
It is clear that tvheadend should not crash in this situation but:
- Jaroslav can't make miracles without the information comining from proper tools on your platform;
- even if he fixes the problem, you would get a (stable) system with problems on the streams...

Think about changing the LNBs.

my 2 cents

I would like to clearly separate the triggering condition and the consequences

- TRIGGER –
Actually I have the third type of LNBs from different vendors producing the same issue. The current ones are german premium quality quad LNBs http://www.amazon.de/Edition-Premium-Anschl%C3%BCsse-Garantie-schwarz/dp/B004TTNJQA Also the cabling is replaced.
I had the same issue before with Inverto RED twin LNBs and no-name low-end LNBs. The ultimate solution would be a satellite multiswitch but it is a bit overkill.

Reports from other users confirm that this phenomenon exists and not specific to my setup. Anyway, I do not expect that anyone makes miracles, however the problem can be mitigated by disabling scanning activity when any streaming is going on. Now this can be achieved by appropriate priority settings. Thanks for Jaroslav’s improvement in https://tvheadend.org/issues/3356

- CONSEQUNECES -
My goal with this ticket was to highlight the faults in the SW when the above triggering conditions occur.
Now, I am in a trouble with further steps. I have been really trying to support the team by providing the needed information however the installation of full Linux seems problematic.

Even though I managed to reproduce this problem many times the behavior is not fully consistent. Normally it takes less than 30 minutes to have the crash but once this SW run for 3 days without any issue. Then did a restart and it crashed within 10 minutes. On the other hand the problem was seen on 4.0.7 only. I could never see this crash either on 4.0.3 or 4.1.x.

I will make effort to get core dump on Openelec but the wetek upgrade to full Linux is not feasible because the device is used for everyday TV watching in my family. And there is no guarantee if the fault is reproducible on the other Linux installation.
Since the issue is specific for 4.0.7 probably it is not worth wasting too much effort. I am much more interested in the other problem seen sporadically on recent releases also. I mean the
- "descrambler: ECM late" and picture freeze for 8-10 seconds despite ECMs are received - issue.

I think my task is now to collect useful logs for further investigations about either fault. I propose to suspend your activity until I come back with some new input.

Akos Sz wrote:

I need help how to collect debug info on "[ ERROR] descrambler: ECM - key late" issue in Openelec. Tried to enable debug and trace on descrambler and cwc subsystems. See attched picture. I cannot see any further info in the service.log file.

Try setting the lpg path.

Set like this: /storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/debug.txt
Nothing happens. No file created.

Hi,

I still need help with enabling debug and trace in Tvheadend addon in Openelec. There are sporadic "[ ERROR] descrambler: ECM - key late" errors causing 5-10 seconds of dropout despite there are no ECM loss or delay according to Oscam logs.

Hi,

I observed a big stability improvement somewhere between 4.1.1354 and 4.1.1497. After upgrading the system got very robust. It has not been crashed since the upgrade. It survived whatever I tried and caused crash before. Either the ECM late problem was seen only once.

I propose to close this ticket as fixed.
Thanks for your efforts.