Tvheadend

I've just this minute gotten around to pushing a new 3.4 build, this includes a whole host of memleak (and some other stuff) fixes submitted by a user. We all knew it was leaking, just hadn't had time to investigate.

I'm not sure this will explain a lockup, but its definitely worth upgrading, I'm doing my system as we speak!

Adam

Are you using a TBS6280 Dual Tuner card under Linux by any chance? If so, try disabling 'Idle Scanning' for both tuners. This generally increases the length of time you can go without rebooting to a couple of days as it decreases the number of 'mux locks'. It isn't a fix though, there is still an issue with this card and TVHeadend which causes the tuner to crash.

OK,

next person to suggest that Tvheadend is somehow responsible for "crashing" or "breaking" a DVB driver (particularly a TBS one) is likely to suffer my wrath! I completely accept there are some error scenarios (after the tuner has failed) that Tvheadend doesn't gracefully deal with.

However if a driver fails, its the drivers fault. If Tvheadend dies because a client sends it crap, I wouldn't blame the crash on the client (even if it dig trigger it) Tvheadend should be robust to such things.

Sorry if that sounds harsh, just been having a fun week trying to point out to a particular DVB hardware vendor (I'll let you guess which) that Tvheadend is not to blame for the tuner performing badly, but the shoddy driver is the problem.

Adam

It doesn't sound harsh at all, I have looked through the TBS linux driver and agree that it is shoddy at best and poorly maintained. The only reason I point the finger at both TVHeadend and the driver is that it worked reasonably well for a long time prior to version 3.4 and I am basing my assumptions on theory rather than fact as I have not been able to look into the issue more thoroughly. I appreciate efforts that have been made to diagnose the issue and interact with the vendor themselves to try and resolve the issue at a driver level. I hope they listen and gratefully receive any patches handed their way and don't put Linux users at the bottom of the pile like normal!

I am using a TBS6284 quad tuner

Thanks Scott, I'll try turning off idle scanning and report back on Monday.

Adam, based on your comment, what is the driver doing wrong? I want to email TBS, and just want to point them in the right direction?

In the short term, can anything be done with tvheadend to stop this happening?

I Will post the debug output with the streaming locked-up on Monday.

Thanks for all your input

I'm not sure this issue is definitely related, though its possible. We've already been in contact with TBS, and have an open dialogue with them, unfortunately they seem to be in denial. We've given them clear test cases that demonstrate the problems in the drivers, and even though they admit they added a fix (to one driver) based on a similar test case (submitted by another user) they seem to have completely disregarded our tests showing that the "fix" is nothing but a nasty workaround that at best creates terrible zap times.

I should point out that all of the above is focused primarily on the DVB-S2 TBS6981 (and 6980), though the 6984 has similar issues. Although you're using a T2 tuner, its entirely possible that similar issues exist. However we cannot say that for certain, for example I have a 6985 (newer version of the 6984) and because of the different HW architecture it doesn't suffer the same problems.

I'm keeping a blog of the email thread between us and TBS, which can be found here. I've been a bit cagey about the exact nature of the underlying problems, though I've dropped enough hints that a decent software engineer should get the gist. This is simply because at this stage I'm not in the habit of doing TBS' job for them, Luis has already done that (by writing an OSS driver for the 6981 that overcomes the issues and performs flawlessly) which they're free to review. And yet they're still in denial.

However if you want to know, the basic problem with the 6981 (and 6984) is that they share an I2C interface (and address space) for the pair of demods (6984 has 2x6981 demod chips), there is a complete lack of locking to ensure no concurrent access occurs. As a result if you tune both devices within a very short period of each other, the commands can trash each other. Typically this results in one tuner working and the other not, though in some cases neither work. Adding a short delay between tuning appears to resolve the issue, but I'm loathed to add such an ugly kludge to TVH to fix their broken drivers (no other devices have this problem). Though I'm tempted to add a "Fix broken TBS drivers" option.

But like I said at teh start, there is no clear evidence this problem is caused by the above. This could well be a bug in Tvheadend, its certainly got plenty of them lurking around.

Interestingly this does seem similar to a problem that has been reported in the XBMC PVR IRC channel (#xbmc-pvr), and I'm beginning to wonder if there is some sort of issue being caused by long running HTSP sessions.

Personally my TVH server runs 24/7 and so far my last few uptimes have been measured in months (only downed so I could upgrade etc..), without any significant issues (except the memleak'ing which was a known issue and is hopefully now fixed, fingers crossed). However my XBMC clients are just that and so they are off most of the day, so the HTSP sessions are pretty short lived (a few hours).

What we really need to see is the debug log when things have failed, and ideally if you can get it into the failed state, maybe we can poke it a bit to see what's going on. Maybe try streaming from it using HTTP, rather than HTSP (XBMC), just to rule out a client issue. Also look at the status page and see if data is actually flowing from the tuners etc...

Adam

The locking issue in the driver between tuners makes a lot of sense with regards to the TBS6280 also. The issue tends to appear when there is sequential activity on both tuners. For instance the other day I was watching a channel through XBMC when another recording started on another station which crashed both tuners. In the status panel when the error occurs, the throughput of the card drops from say 2000kbps (normal) to around 300kbps and you get random noise and pixels on screen. Both tuners stay locked to the frequency they crashed on and do not respond to further commands unless the kernel module is reloaded. The issue also occurs when no XBMC clients are open and connected (such as overnight). I can go to bed with everything working fine and no clients connected, go to watch something on my tablet using tvhguide in the morning and find the card has crashed overnight doing epg updates etc... I am going to try and take an hour out later on to look into the issue and maybe try and merge some of the 6981 (fixed) driver with the 6280 driver. I will let you know how it goes and once again, thank you for all the efforts on this issue so far.

Scott,

If you're thinking about trying to write an open source version of the driver, its worth contacting Luis (lja on freenode or ljalves on github). As he has spent a lot of time analysing the 6981 problem and writing the open source driver for that. He's hoping to submit it upstream in the coming weeks, so at some point we might even have in-tree kernel support for some TBS devices.

Adam

Can I just add that I have the exact same issue with the same hardware. I too can rule out network issues and client issues as local DVR recordings also drop to a bandwidth of less than 200 forcing a reboot.

Jamie,

This issue-thread seems to have gone a little cold.

I see that someone has written their own TBS driver which fixes this problem. However it does not support TBS6284 yet.

Their page is at https://github.com/ljalves/linux_media

I have emailed the developer asking if he has any plans to add support for TBS6284 in the future.

I will post his reply here.

Hi all,

About XBMC being the problem, I don't think so... at least not for me.
I'm running tvheadend (from git/master) with xbmc 12.2 + tvheadend pvr plugin and it runs for weeks now without a single issue.

About the driver: unfortunately building a driver without having the real hardware to debug and test is a P.I.T.A.
I have no plans to release other TBS drivers unless I have access to the cards (although I'm making some effort to have an OSS driver for the 6984).

And I would say that it is definitively a driver issue.

Regards,
Luis

Thanks for the reply Luis.

Excuse my ignorance, but what is an OSS driver? Is that one that is part of the operating system?

This issue is a real pain, so if you ever need a guinea pig to test a future 6984 driver on, please let me know.

Adam, in the mean time, is there anything that can be done to tvheadend to stop this from happening? Before you mentioned adding a setting to the web interface?

Cheers,

West

There is a new Linux driver out on the tbsdtv website today. I will compile it later and see if any fixes have been implemented or whether they continue to ignore the issue. Looking at the changelog there doesn't seem to be anything mentioned specifically...

I have the new driver from TBS and am 99% sure the issue remains.
W Trainer - Thanks you for the heads up, would be great to fix this issue.

Based on Konstantin's complete lack of understanding/acceptance of what the underlying issue is, I'd very much doubt its resolved. If you're lucky you'll end up, as we did during testing, with a driver that "works" at the expense of horrendous performance.

I'm beginning to get the stage of, if anyone reports any DVB/streaming related issues the first question I'll ask is "is your card TBS" and if yes I'll ignore people (that's somewhat a joke obviously, but I'm pretty pissed about this driver debacle).

Adam

What is the best alternative? I need DVB-T2.

I have a TBS6280 and had similar symptoms - problem was solved by changing the IRQ handling for the card. See http://www.tbsdtv.com/forum/viewtopic.php?f=52&t=7631

Interesting idea. I am getting there with Linux but IRQ stuff is all new, here is my /proc/interrupts - does this suggest that currently there is no sharing? I woke up this morning and it was working - which is unusual.

CPU0 CPU1
0: 44 0 IO-APIC-edge timer
1: 3 0 IO-APIC-edge i8042
8: 1 0 IO-APIC-edge rtc0
9: 0 0 IO-APIC-fasteoi acpi
12: 4 0 IO-APIC-edge i8042
17: 4268652 565101 IO-APIC-fasteoi SAA716x Core
23: 115 0 IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2
41: 0 0 PCI-MSI-edge xhci_hcd
42: 1592480 0 PCI-MSI-edge eth0
43: 648687 0 PCI-MSI-edge ahci
44: 25 0 PCI-MSI-edge mei
45: 306 0 PCI-MSI-edge snd_hda_intel
46: 1 0 PCI-MSI-edge i915
NMI: 2032 1166 Non-maskable interrupts
LOC: 7008938 3292135 Local timer interrupts
SPU: 0 0 Spurious interrupts
PMI: 2032 1166 Performance monitoring interrupts
IWI: 0 0 IRQ work interrupts
RTR: 1 0 APIC ICR read retries
RES: 264258 277407 Rescheduling interrupts
CAL: 272 502 Function call interrupts
TLB: 31846 21464 TLB shootdowns
TRM: 0 0 Thermal event interrupts
THR: 0 0 Threshold APIC interrupts
MCE: 0 0 Machine check exceptions
MCP: 202 202 Machine check polls
ERR: 0
MIS: 0

If you still have a syslog from the last time the problem happened, search for the irq error msg shown in the tbs forum post to confirm this is indeed the problem.

17: 4268652 565101 IO-APIC-fasteoi SAA716x Core

shows your tbs card is using the old style of interrupts - you want to change this to use PCI-MSI-edge type.

Create /etc/modprobe.d/tbsdriver.conf with this line in it:
options saa716x_tbs_dvb int_type=1

then reboot and check /proc/interrupts again to make sure it's using the new interrupt type.

This is working for me so far. Been running for 2 days with 'Idle Scanning' enabled and still going strong.

When i grep syslog for IRQ i get the following:-

$ grep IRQ syslog
Aug  2 14:53:21 box kernel: [24120.626668]  <IRQ>  [<ffffffff810e4c6d>] __report_bad_irq+0x3d/0xe0
Aug  2 14:53:21 box kernel: [24120.626720]  [<ffffffff816a6eca>] do_IRQ+0x5a/0xe0
Aug  2 14:53:21 box kernel: [24120.626946] Disabling IRQ #17

$ grep irq syslog
Aug  2 14:53:21 box kernel: [24120.626596] irq 17: nobody cared (try booting with the "irqpoll" option)
Aug  2 14:53:21 box kernel: [24120.626668]  <IRQ>  [<ffffffff810e4c6d>] __report_bad_irq+0x3d/0xe0
Aug  2 14:53:21 box kernel: [24120.626693]  [<ffffffff810e28f9>] handle_irq_event_percpu+0xa9/0x210
Aug  2 14:53:21 box kernel: [24120.626699]  [<ffffffff810e2aae>] handle_irq_event+0x4e/0x80
Aug  2 14:53:21 box kernel: [24120.626705]  [<ffffffff810e5c24>] handle_fasteoi_irq+0x64/0x120
Aug  2 14:53:21 box kernel: [24120.626713]  [<ffffffff810161c2>] handle_irq+0x22/0x40
Aug  2 14:53:21 box kernel: [24120.626809] [<ffffffff814bcab0>] usb_hcd_irq
Aug  2 14:53:21 box kernel: [24120.626845] [<ffffffff814bcab0>] usb_hcd_irq
Aug  2 14:53:21 box kernel: [24120.626893] [<ffffffffa0434a30>] saa716x_tbs6284_pci_irq [saa716x_tbs_dvb]

Is this an IRQ error?

When i checked /proc/interrupts i found this line:-

17:    4293298       6111        274        320   IO-APIC-fasteoi   ehci_hcd:usb1, ehci_hcd:usb2, SAA716x Core

Assuming this is the old style i created the tbsdriver.conf file as specified by Alex.

After a reboot my /proc/interrupts shows this:-

17:          0          2          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, ehci_hcd:usb2

Does this look like the change has taken effect?

I will leave my server on and report back. Hopefully i will have the same result as Scott.

Thanks all for your input!!

W Trainer, that isn't correct. You should get a line like this when you run 'cat /proc/interrupts':

44:         60   11986691   PCI-MSI-edge      SAA716x Core

The file you need to create is NOT tbsdriver.conf, it needs to be /etc/modprobe.d/tbs.conf with the following line in it:

options saa716x_tbs-dvb int_type=1

Save that and reboot and then check /proc/interrupts for the line posted above.

Hope it helps...

W Trainer, I think you're good as you are without following Scott's advice. When you change IRQ type for SAA716x it seems to change its IRQ number to one in the 40s - just check /proc/interrupts for the line containing the SAA716x module, you just pasted the line for IRQ17 and SAA716x isn't there so looks like your change has taken effect.

Scott's advice is likely correct for whatever version of the tbs drivers he's running. Dashes and underscores are often used interchangeably between developers to whatever suits their coding style at the the time and may well have changed between versions - just run 'lsmod | grep -i SAA716' to get the style for your machine - this is what the put in the /etc/modprobe/*.conf file. The name of the file doesn't really matter, it's the name of the module that does.

Alex is right with regards to filename, it only occurred to me once I had sent the message that it makes no difference in the.modprobe.d directory, it just looks for .conf files. Pretty sure the module name has been saa716x-_tbs-dvb for quite a few driver revisions now though.

That is meant to say saa716x_tbs-dvb. Serves me right for typing on my phone!

Ah! turns out it doesn't matter if it's dashes or underscores that are used in module names - from the modprobe man page:

modprobe intelligently adds or removes a module from the Linux kernel: note that for convenience, there is no difference between _ and - in module names (automatic underscore conversion is performed)

that simplifies things a bit :)

Yes, once i applied the change i now have the following line in /proc/interrupts:-

58:    6480548        595        592        665   PCI-MSI-edge      SAA716x Core

So far so good. Server up for 21 hours and still going strong.

I will post back in a day or two to update on TBS status. But feeling optimistic.

Thanks all!!

So far so good with me....:)

Right, it's been about a week now and everything is running smoothly.

Thanks for everyone's input, especially Alex for identifying the IRQ bug. That fix saved me having to get a new card!

Cheers Adam, loving tvheadend. I can finally cancel my Virgin TV subscription!

Thanks. I've just upgraded. I'll see how it goes for a day or two, then
report back on the thread.

All the best,

West

West

On 11 July 2013 20:44, wrote:

Issue #1767 has been updated by Adam Sutton.

I've just this minute gotten around to pushing a new 3.4 build, this
includes a whole host of memleak (and some other stuff) fixes submitted by
a user. We all knew it was leaking, just hadn't had time to investigate.

I'm not sure this will explain a lockup, but its definitely worth
upgrading, I'm doing my system as we speak!

Adam
------------------------------
Bug #1767: TV stops streaming and needs a reboot

- Author: W Trainer
- Status: New
- Priority: Normal
- Assignee: John Törnblom
- Category: Streaming
- Target version:
- Found in version: 3.4
- Affected Versions:

Hello there,

First off, great project, and i'm on the verge of Tvheadend replacing my
virgin subscription. Just one annoying thing keeps happening...

I have an Ubuntu server with Tvheadend installed. I view Tvheadend on a
seperate PC running XBMC. When the server has been turned on, all works
perfectly.

However, if i leave the server on for an extended period, eg overnight,
Tvheadend stops streaming. That is, in the morning, i switch on my PC and
access XBMC. It connects to the server and loads the channels, but when i
select a channel the buffering stays at zero, and no picture appears. If i
leave it for a few minutes a frozen lined image might appear, but nothing
viewable. And that is it, no picture.

The server, in all other aspects is working fine and is accessible as a
file-server. But Tvheadend seems to have stalled. Any recording run as
scheduled, and are for the correct length, but they have this frozen image
in them for the entire duration.

The only way to fix this is to reboot the server, and everything is fine
again, including recordings. But after several hours, the picture is
stalled again.

Is this a known issue, or is there something i can do to debug this?

Here is my set-up

Server - Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-28-generic x86_64), Tvheadend
3.4~precise, Tuner card - TBS6284 DVB-T2/T (latest drivers)
PC - XBMCbuntu, XBMC 12.2 Frodo

Here is my output from running "tvheadend -d" on the server

Jul 11 18:39:58.799 [ DEBUG]:config: no configuration, loading defaults
Jul 11 18:39:59.001 [ INFO]:charset: 71 entries loaded
Jul 11 18:39:59.005 [ ALERT]:dvb: /dev/dvb/adapter0/frontend0: unable to
find dvr
Jul 11 18:39:59.022 [ ALERT]:dvb: /dev/dvb/adapter1/frontend0: unable to
find dvr
Jul 11 18:39:59.022 [ ALERT]:dvb: /dev/dvb/adapter2/frontend0: unable to
find dvr
Jul 11 18:39:59.022 [ ALERT]:dvb: /dev/dvb/adapter3/frontend0: unable to
find dvr
Jul 11 18:39:59.024 [ INFO]:webui: Running web interface in debug mode
Jul 11 18:39:59.024 [ INFO]:CSA: Using SSE2 128bit parallel descrambling
Jul 11 18:39:59.024 [ INFO]:epggrab: module eit created
Jul 11 18:39:59.024 [ INFO]:epggrab: module uk_freesat created
Jul 11 18:39:59.024 [ INFO]:epggrab: module uk_freeview created
Jul 11 18:39:59.024 [ INFO]:epggrab: module viasat_baltic created
Jul 11 18:39:59.028 [ DEBUG]:opentv: dictionary skyit loaded
Jul 11 18:39:59.028 [ DEBUG]:opentv: dictionary skyeng loaded
Jul 11 18:39:59.029 [ DEBUG]:opentv: dictonaries loaded
Jul 11 18:39:59.030 [ DEBUG]:opentv: genre map skyuk loaded
Jul 11 18:39:59.030 [ DEBUG]:opentv: genre maps loaded
Jul 11 18:39:59.030 [ INFO]:epggrab: module opentv-ausat created
Jul 11 18:39:59.030 [ DEBUG]:opentv: provider ausat loaded
Jul 11 18:39:59.031 [ INFO]:epggrab: module opentv-skyit created
Jul 11 18:39:59.031 [ DEBUG]:opentv: provider skyit loaded
Jul 11 18:39:59.031 [ INFO]:epggrab: module opentv-skyuk created
Jul 11 18:39:59.031 [ DEBUG]:opentv: provider skyuk loaded
Jul 11 18:39:59.031 [ DEBUG]:opentv: providers loaded
Jul 11 18:39:59.033 [ INFO]:epggrab: module pyepg created
Jul 11 18:39:59.033 [ INFO]:epggrab: module xmltv created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_il created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_huro created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_uk_bleb
created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_is created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_fi_sv created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_combiner
created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_se_swedb
created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_uk_rt created
Jul 11 18:40:00.006 [ INFO]:epggrab: module /usr/bin/tv_grab_pt created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_nl created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_es_laguiatv
created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_no_gfeed
created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_na_dd created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_hr created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_fr created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_pt_meo created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_eu_epgdata
created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_dtv_la created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_fi created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_za created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_it created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_fr_kazer
created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_ar created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_ch_search
created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_in created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_ee created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_es_miguiatv
created
Jul 11 18:40:00.007 [ INFO]:epggrab: module /usr/bin/tv_grab_dk_dr created
Jul 11 18:40:00.008 [ INFO]:epgdb: loaded v2
Jul 11 18:40:00.008 [ INFO]:epgdb: channels 0
Jul 11 18:40:00.008 [ INFO]:epgdb: brands 0
Jul 11 18:40:00.008 [ INFO]:epgdb: seasons 0
Jul 11 18:40:00.008 [ INFO]:epgdb: episodes 0
Jul 11 18:40:00.008 [ INFO]:epgdb: broadcasts 0
Jul 11 18:40:00.008 [ INFO]:dvr: Creating new configuration ''
Jul 11 18:40:00.008 [WARNING]:dvr: Output directory for video recording is
not yet configured for DVR configuration "". Defaulting to to "/home/west".
This can be changed from the web user interface.
Jul 11 18:40:00.008 [ NOTICE]:START: HTS Tvheadend version 3.4~precise
started, running as PID:3020 UID:1000 GID:1000, settings located in
'/home/west/.hts/tvheadend'
Jul 11 18:40:00.012 [ DEBUG]:AVAHI: Adding service 'Tvheadend'
Jul 11 18:40:00.855 [ INFO]:AVAHI: Service 'Tvheadend' successfully
established.

Any help you can give would be much appreciated!

Cheers,

West
------------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

Hi,
Im having the same issue but im using a Pinnacle PCTV DVB-S2 461e ([[http://www.linuxtv.org/wiki/index.php/Pinnacle_PCTV_DVB-S2_Stick_(461e)]])
TvHeadend works perfectly for hours but then stops streaming the signal strength is good but
when looking at the status it goes from testing to bad which keeps looping, however there are no errors.

All this interrupt stuff is a bit beyond me, but im not sure if it will help?
Any help would be appreciated.

Thanks