Tvheadend

Alright. I'm dying to test this but in all my farting around I've hosed tvheadend... or something. I was unable to play any channel through tvheadend, so I cleared out the muxes and added my channels.conf back in for it to rescan. It found 6 channels instead of the hundreds it should have found. I have no idea what I did so you'll have to bear with me until I am able to get services/channels set back up again.

Hi Villy,

Sorry for the delay, but I finally got to test your experimental fix. It worked great. I was able to record and playback the hacked channel successfully.

Any idea how much effort it would be to turn this into a real fix?

Thanks again!

Thanks Jim! I've all but given up on tvheadend since making the terrible mistake of updating the firmware on my prime to the newest beta. At least, that's what I'm convinced broke the set up as tvheadend scans and scans and finds a hand full of services, where before it would find every last channel.

I need to stop pushing buttons.

Hi Jim/John,

Sorry for the delay, but I finally got to test your experimental fix. It worked great. I was able to record and playback the hacked channel successfully.

Good, proof of concept for that at least.

So for next hackish step, we need to get a mapping of PID<->Program, and then teach userhdhomerun to use that for tuning and see what happens. Fortunately there is already a tool that will scan everything and print out the mapping for us - so if one you could do the below and paste the output here - I'll see if I can teach userhdhomerun to use that.

Remember to stop tvheadend before doing this!
 sudo apt-get install w-scan
 sudo w_scan -f a -A 3 -c US > wscan_output.log

That will simply go scan through all frequencies and find what stuff there is, and output a line for each program. Long operation, takes half an hour for me here in DVB-C land. Info on w_scan here: http://linuxtv.org/wiki/index.php/W_scan you may want to tweek the -A parameter, read on that first.

Output in the file wscan_output.log is what we are interested in. For me I get something like this:

NatGeo HD;Stofa:610000:M256:C:6900:1540=27:1541=eng;1542:0:0:1154:1:31:0
 BBC World News;Stofa:626000:M256:C:6900:1330=2:1331=eng:1339:0:1133:1:33:0
 ZDF HD;Stofa:626000:M256:C:6900:2560=27:2561=deu;2562:2569:0:1256:1:33:0
 Eurosport HD;Stofa:626000:M256:C:6900:2680=27:2681=dan:2689:0:1268:1:33:0
 Eurosport 2 HD;Stofa:626000:M256:C:6900:2690=27:2691=dan,2692=eng:0:0:1269:1:33:0

Which contain both program number and PID's for video/audio/teletext in one convienient line per program.

Any idea how much effort it would be to turn this into a real fix?

Very good question. To me it seems like no matter how it is done, it is going to be somewhat complex.

There is still some technical details/analysis I/we need to figure out. One thing that pops to mind is that right now in tvheadend (and any other linux dvb program), you can easily record more than one program on the same mux using only one tuner. You simply feed it with a filter contained the programs you want to record and that works magically. But PID filtering don't work for cablecard, here your explicitle need to ask a tuner to tune to a specific program - I haven't found in the API the possibility to record more than one program on one tuner/mux. So tvheadend expects to be able to record more than one program on a mux, and if that is not possible we need to fail that gracefully somehow.

Also there is the whole story of providing the PID<->program mapping before everything starts to work. That could/would work with the w_scan application - but is going to be difficult to explain to users. "Hey run this for a hour, then you can use tvheadend. Oh, and don't run tvheadend while you run, and remember to start it afterwards"...

Anyway, lets just try with output from w_scan, teach userhdhomerun to use that for mapping and see what happens. ;-)

Oh, and John, the above is a good test for you also - if w_scan finds a lot more the tvheadend does, then tvheadend is of course the problem... ;-)

I had ran one of these earlier last week using "scan" on my first adapter and using my custom list of frequencies as reported from hdhomerun_config. You can check out the dump from that at http://pastebin.com/FMZJFEdY

It only grabbed names for the local channels, the remainder are just the hex values for the programs, which I was mapping manually to the found services in tvheadend (before it started crapping out on me). Just for reference AMC HD, which I was using in the dvbsnoops above is line:

[0014]:319750000:QAM_256:3746:3747:20

I've tried to run w_scan a few different ways, but it just runs through 8vsb, finds nothing, then runs through qam256 on HRC's in like a second and reports back:

initial_tune:2167: Setting frontend failed QAM256   f=837000 kHz
843000: QAM256(time: 04:31) set_frontend:1761: ERROR: Setting frontend parameters failed (API v5.x)
: 22 Invalid argument

initial_tune:2167: Setting frontend failed QAM256   f=843000 kHz
849000: QAM256(time: 04:31) set_frontend:1761: ERROR: Setting frontend parameters failed (API v5.x)
: 22 Invalid argument

initial_tune:2167: Setting frontend failed QAM256   f=849000 kHz

ERROR: Sorry - i couldn't get any working frequency/transponder
 Nothing to scan!!

As for my personal tvheadend/hdhomerun-beta problem, I am still able to run "scan" and it dumps fine, although on any program that it comes across where there isn't a name found (where it only reports the virtual hex) it says "WARNING: filter timeout pid 0x1fff"... although it writes the correct video pid and audio pid to the dump, so it may not mean much.

Let me know if I can push any more buttons to help make this happen.

Hi John/Jim,

Hmmm, weird w_scan doesn't work for you.

Jim: can you try the same?

Scan and w_scan are quite different applications - w_scan provides a bit more information which is why it would be nice to be able to use that one.

well.. w_scan didn't work on my main server or my laptop... so I tried it on my web dev server in the basement that is running ubuntu 11.04 and it worked. http://paste.ubuntu.com/1334960/

I ran it using:

 w_scan -f a -c US -A 2 -I us-IL-Waukegan -o 7 > scan_output.txt

where us-IL-Waukegan is my dvbscan formatted list of frequencies (which comprises every frequency for which I have programs).

I'll run it again using your parameters in case there was something important I was missing.

[UPDATE]
So.. no go on the parameters suggested. 8VSB returned no hits and the list of frequencies it runs automagically for the US return nothing. I believe using my us-IL-Waukegan list is the best way to go.

Sorry, I didn't get to trying w_scan yesterday, but I'll give it a try this evening.

Jim

On a seperate note, I believe I've figured out what they hosed in their latest firmware beta. Lucky me... My device only reports back 0x0000 as a tsid regardless of the frequency. I believe that tvheadend relies on the tsid (which I assume it calls a MUX ID) to do it's scanning, so only channels that are in the first scanned frequency are found.

Just Peachy. So yea... don't drop the beta on your device since you can't downgrade, the source is kept secret and it borks anything that relies on the tsid of a mux.

Villy,

I've been working on an alternate approach to doing the PID <-> Virtual Channel mapping in dvbhdhomerun. Basically, in HdhomerunTuner::StartStreaming, I call hdhomerun_device_get_tuner_streaminfo to get virtual channel information for the currently tuned frequency. For each program found, I select it using hdhomerun_device_set_tuner_program, then call hdhomerun_device_get_tuner_filter to get the PID filter that has been applied as a result of the set_tuner_program call. If it matches the PID passed to the StartStreaming call, then I have the correct program. If I can't find a program that matched the PID, I fall back to the original logic of setting the PID filter directly.

I've got this implemented, but I still have to test it. I'd be interested in any comments or suggestions you might have. In particular, do you see any reason why this can't work or is a really bad idea? If not that, then I'd appreciate anything you can tell me about how to test this.

Thanks!

Hi Jim,

I've been working on an alternate approach to doing the PID <-> Virtual Channel mapping in dvbhdhomerun. Basically, in HdhomerunTuner::StartStreaming, I call hdhomerun_device_get_tuner_streaminfo to get virtual channel information for the currently tuned frequency. For each program found, I select it using hdhomerun_device_set_tuner_program, then call hdhomerun_device_get_tuner_filter to get the PID filter that has been applied as a result of the set_tuner_program call. If it matches the PID passed to the StartStreaming call, then I have the correct program. If I can't find a program that matched the PID, I fall back to the original logic of setting the PID filter directly.

I did not know about the hdhomerun_device_get_tuner_streaminfo(), sounds like that'll make everything much easier.

I've got this implemented, but I still have to test it. I'd be interested in any comments or suggestions you might have. In particular, do you see any reason why this can't work or is a really bad idea? If not that, then I'd appreciate anything you can tell me about how to test this.

Hmmm, to keep it simple just do the hdhomerun_device_get_tuner_streaminfo() + hdhomerun_device_set_tuner_program() for starters. get_tuner_filter() and checking wait with that.

One thing to keep in mind, is that StartStreaming() will be called a bunch of times when a program is tuned. One for each video/audio/teletext PID. That is just the way it is done in the linux kernel. But I don't see any harm in calling streaminfo() + set_tuner_program() each time it is hit.

Also keep in mind that set_tuner_program() only seems to work for one program per mux, so don't schedule two program on the same mux for starters - that'll screw things up. Otherwise, go go go! :-)

Villy Thomsen wrote:

Hi Jim,

I've been working on an alternate approach to doing the PID <-> Virtual Channel mapping in dvbhdhomerun. Basically, in HdhomerunTuner::StartStreaming, I call hdhomerun_device_get_tuner_streaminfo to get virtual channel information for the currently tuned frequency. For each program found, I select it using hdhomerun_device_set_tuner_program, then call hdhomerun_device_get_tuner_filter to get the PID filter that has been applied as a result of the set_tuner_program call. If it matches the PID passed to the StartStreaming call, then I have the correct program. If I can't find a program that matched the PID, I fall back to the original logic of setting the PID filter directly.

I did not know about the hdhomerun_device_get_tuner_streaminfo(), sounds like that'll make everything much easier.

Hmmm, no on closer inspection that doesn't look like it will work. get_tuner_streaminfo() returns all programs on the mux no matter what you have set for PID filtering (I think), but do try it anyway if I am wrong.

In any case I think you are on to something, there may be something else in the API that can gives us what we need. I don't have much time tonight (I'm in denmark btw, so there is some time difference) so do keep looking.

Villy Thomsen wrote:

Villy Thomsen wrote:

Hi Jim,

I've been working on an alternate approach to doing the PID <-> Virtual Channel mapping in dvbhdhomerun. Basically, in HdhomerunTuner::StartStreaming, I call hdhomerun_device_get_tuner_streaminfo to get virtual channel information for the currently tuned frequency. For each program found, I select it using hdhomerun_device_set_tuner_program, then call hdhomerun_device_get_tuner_filter to get the PID filter that has been applied as a result of the set_tuner_program call. If it matches the PID passed to the StartStreaming call, then I have the correct program. If I can't find a program that matched the PID, I fall back to the original logic of setting the PID filter directly.

I did not know about the hdhomerun_device_get_tuner_streaminfo(), sounds like that'll make everything much easier.

Hmmm, no on closer inspection that doesn't look like it will work. get_tuner_streaminfo() returns all programs on the mux no matter what you have set for PID filtering (I think), but do try it anyway if I am wrong.

In any case I think you are on to something, there may be something else in the API that can gives us what we need. I don't have much time tonight (I'm in denmark btw, so there is some time difference) so do keep looking.

Seems like simply:

hdhomerun_device_get_tuner_program()

Is exactly what we need. So instead of get_tuner_streaminfo() just use that one. That gives exactly what program is tuned.

Villy Thomsen wrote:

Hmmm, no on closer inspection that doesn't look like it will work. get_tuner_streaminfo() returns all programs on the mux no matter what you have set for PID filtering (I think), but do try it anyway if I am wrong.

Right, that's why I took sort of a backwards/indirect approach. I iterate through each program returned by get_tuner_streaminfo(), tune to it using set_tuner_program(), then call get_tuner_filter() to see if the PID matches, which means I've found the right one. That's kind of strange and possibly inefficient, but it's the only way I could figure out so far.

Hi Jim,

Hmmm, no on closer inspection that doesn't look like it will work. get_tuner_streaminfo() returns all programs on the mux no matter what you have set for PID filtering (I think), but do try it anyway if I am wrong.

Right, that's why I took sort of a backwards/indirect approach. I iterate through each program returned by get_tuner_streaminfo(), tune to it using set_tuner_program(), then call get_tuner_filter() to see if the PID matches, which means I've found the right one. That's kind of strange and possibly inefficient, but it's the only way I could figure out so far.

You may have missed a posting there - but simply use this instead of _get_tuner_streaminfo():

hdhomerun_device_get_tuner_program()

It returns the program that is tuned now, so you can use that directly in set_tuner_program()

Villy Thomsen wrote:

Seems like simply:

hdhomerun_device_get_tuner_program()

Is exactly what we need. So instead of get_tuner_streaminfo() just use that one. That gives exactly what program is tuned.

Perhaps I'm misunderstanding something (wouldn't be the first time). The way I understand the problem with the HDHR Prime is that just setting a PID filter doesn't cause the box to do the additional CableCARD magic to decrypt the stream. In order to do that you have to call set_tuner_program() instead of just . But that requires a program number, when all we have is a PID. Are you suggesting setting the PID filter, then calling get_tuner_program() to get the program number? I can try that, but I'd be surprised if it worked. Am I missing your point?

Ok here is the w_scan output (both stderr and stdout) and the output of hdhomerun_config 131605CE scan 0 scan0.txt so you can compare that.

Jim Ancona wrote:

Villy Thomsen wrote:

Seems like simply:

hdhomerun_device_get_tuner_program()

Is exactly what we need. So instead of get_tuner_streaminfo() just use that one. That gives exactly what program is tuned.

Perhaps I'm misunderstanding something (wouldn't be the first time). The way I understand the problem with the HDHR Prime is that just setting a PID filter doesn't cause the box to do the additional CableCARD magic to decrypt the stream. In order to do that you have to call set_tuner_program() instead of just . But that requires a program number, when all we have is a PID. Are you suggesting setting the PID filter, then calling get_tuner_program() to get the program number? I can try that, but I'd be surprised if it worked. Am I missing your point?

Hi Jim,

That was my thought exactly. It's worth a try at least. Long shot of course, since if it actually knew which program it was tuned to via the PID's it would probably have decoded it....

Jim Ancona wrote:

Ok here is the w_scan output (both stderr and stdout) and the output of hdhomerun_config 131605CE scan 0 scan0.txt so you can compare that.

Thanx. Well, it does look like it contains what we need for a mapping - I'll see if I can teach userhdhomerun to eat the output from w_scan.

I should probably make it flexible so that you can use it as a "cache" if your method works. E.g. when you tune to a frequency the first time, store the PID's + program you get from it to the cache, so that next time you tune to that frequency you can just look it up in the cache instead of having to go through all the programs again.

Maybe your method could be used as a startup thing? Hmmm. Need more thinking.

I tried adding the get_tuner_program() call immediately after the set_tuner_filer() in StartStreaming() (without calling set_tuner_program()). Unfortunately it always returns "0". :-(

As I'm doing more testing there's something I don't understand about the way TVheadend uses the DVB interfaces. I'm working in the StartStreaming() method and when I make a request to record or stream a channel I see multiple calls to StartStreaming for multiple PIDs. The logs look like this:

FE_SET_FRONTEND, freq: 495000000
sym qual: 64
START FEED: Pid = 0
In StartStreaming: _pid: 0
PidFilter: 0x0 
hdhomerun_device_set_tuner_filter: 0x0 
hdhomerun_device_stream_start: 1
FE_READ_STATUS
Open data device: /dev/hdhomerun_data2
Open data device: /dev/hdhomerun_data2
sym qual: 64
START FEED: Pid = 1
In StartStreaming: _pid: 1
PidFilter: 0x0 0x1 
hdhomerun_device_set_tuner_filter: 0x0 0x1 
START FEED: Pid = 1ffb
In StartStreaming: _pid: 1ffb
PidFilter: 0x0 0x1 0x1FFB 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 
START FEED: Pid = 1a01
In StartStreaming: _pid: 1a01
PidFilter: 0x0 0x1 0x1FFB 0x1A01 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 0x1A01 
START FEED: Pid = 1a03
In StartStreaming: _pid: 1a03
PidFilter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 
START FEED: Pid = 1a0d
In StartStreaming: _pid: 1a0d
PidFilter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 
START FEED: Pid = 1a00
In StartStreaming: _pid: 1a00
PidFilter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 0x1A00 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 0x1A00 
START FEED: Pid = 17a0
In StartStreaming: _pid: 17a0
PidFilter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 0x1A00 0x17A0 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 0x1A00 0x17A0 
START FEED: Pid = 1a10
In StartStreaming: _pid: 1a10
PidFilter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 0x1A00 0x17A0 0x1A10 
hdhomerun_device_set_tuner_filter: 0x0 0x1 0x1FFB 0x1A01 0x1A03 0x1A0D 0x1A00 0x17A0 0x1A10 
FE_READ_STATUS

Note that the above is all to stream/record a single program. Do you know what TVHeadend is trying to do?

So... given my Comcast/tsid issue I had the great thought that maybe I could spoof the tsid for each mux so tvheadend doesn't balk at all the 0x00's. I've seen a few other folks out there dump a hdhomerun_config scan from comcast and it looks like I'm not the only one that suffers from this issue.

Villy, any suggestions you might have for hacking on userhdhomerun to pull off a tsid spoof. From what I've seen the TSID needs to be static for each mux, and it looks like a 16 bit positive integer, so given that the range of potential frequencies is fixed, I figured I could just divide each frequency by a sufficiently large number and pass that back up so tvheadend is none the wiser.

Jim, if there is anything I can help with or test on the virtual channel front, let me know. I haven't written in C++ before, but I do code in a handful of other languages so I can stumble around without breaking things too badly.

You guys rock.

I have a patch that works sometimes. I'm trying to understand why it doesn't work all the time. One possible explanation is that dvbhdhomerun plus patch is working fine, but there are other problems with my TVHeadend setup.I'm going to try to do some more troubleshooting on that front this evening.

I still haven't been able to figure out what distinguishes channels that work from those that don't with my patch. If anyone wants to test it, I've uploaded it to the dvbhdhomerun issue tracker here: https://sourceforge.net/apps/trac/dvbhdhomerun/ticket/12

I'd appreciate any feedback. Thanks!

My patch now works reliably and tvheadend now can record and stream reliably from my HDHR Prime. I've attached the patch to the issue on the dvbhdhomerun tracker. If you can build dvbhdhomerun, please give it a try and let me know how it works.