Tvheadend

Does this conversation ring true for you, Ben?

https://tvheadend.org/boards/5/topics/6808?r=7082

Try .ts and see if that fixes your problem - if it does, you're probably victim of the same issue. I don't know if the fix made it to whatever version you're running, but you could try switching to master (or use an unstable nightly) and see if your mkv audio problems are fixed there. I certainly haven't noticed any issues for some time now.

Yeah - that thread was what prompted trying the TS, and it's certainly better that way.

However, even with TS, nothing appears able to convert it and retain the 5.1, and I can't playback 5.1 AAC. I think latest beta TVH + TS looks to reliably downmix to ac3 2.0, which is something at least!

Hmmm, well, you've just nudged me to go and check an mkv I recorded the other night, to see what soundtrack that had and whether I could replicate your issue. Unfortunately, that's back to where I was at Christmas, in that there only appear to be a couple of random channels instead of the 5.1 I'd expect - so the recording is useless. And that's on 3.5.74~g8bceacd as well, so a very recent build.

Ah, well, my wife didn't really want to watch that film... I hope...

Bum. Maybe the Matroska gods just hate me and I'm better off sticking with .ts instead.

Lol.

My experience from a lot of messing about with demuxing the streams is that the data is generally there…. it's just not packaged properly/got the right meta data (no idea if those are the right terms…. !!).

eg: the issue on that thread where you seem to get the surround channels coming through as if they are 2.0 channels - I've managed to demux the audio there, and then convert it with a bit of forcing to 6 channel (with faad/aften) - it is all there. But when converted to 6 channel, it is all out of sync, because of the 2 channel bits - ads/before start.

On Tuesday, 9 April 2013 at 17:49, [email protected] wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/boards/5/topics/8194?r=8213#message-8213)
Prof Yaffle
Hmmm, well, you've just nudged me to go and check an mkv I recorded the other night, to see what soundtrack that had and whether I could replicate your issue. Unfortunately, that's back to where I was at Christmas, in that there only appear to be a couple of random channels instead of the 5.1 I'd expect - so the recording is useless. And that's on 3.5.74~g8bceacd as well, so a very recent build.
Ah, well, my wife didn't really want to watch that film... I hope...
Bum. Maybe the Matroska gods just hate me and I'm better off sticking with .ts instead.
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: https://tvheadend.org/my/account

Maybe you're right - the log suggests that tvheadend never even saw the extra channels, though:

Apr  7 22:58:30 revo tvheadend[30771]: dvr: "Dead Poets Society" on "BBC Two HD" recorder starting
Apr  7 22:58:30 revo tvheadend[30771]: subscription: "DVR: Dead Poets Society" subscribing on "BBC Two HD", weight: 300, adapter: "Sony CXD2820R (PCTV 290e - DVB-T2)", network: "Gloucestershire", mux: "Gloucestershire: 482,200 kHz", provider: "", service: "BBC TWO HD", quality: 100
Apr  7 22:59:01 revo tvheadend[30771]: dvr: /home/xbmc/Recordings/Dead Poets Society.2013-04-07.23-00.mkv from adapter: "Sony CXD2820R (PCTV 290e - DVB-T2)", network: "Gloucestershire", mux: "Gloucestershire: 482,200 kHz", provider: "", service: "BBC TWO HD" 
Apr  7 22:59:01 revo tvheadend[30771]: dvr:  #  type              lang  resolution  aspect ratio  sample rate  channels
Apr  7 22:59:01 revo tvheadend[30771]: dvr:  1  H264                    1920x1088   30:17
Apr  7 22:59:01 revo tvheadend[30771]: dvr:  2  AAC               eng                             48000        2
Apr  7 22:59:01 revo tvheadend[30771]: dvr:  3  AAC               eng                             24000        1
Apr  7 22:59:01 revo tvheadend[30771]: dvr:  4  DVBSUB            eng
Apr  8 01:03:00 revo tvheadend[30771]: subscription: "DVR: Dead Poets Society" unsubscribing from "BBC Two HD" 

... so tvh saw two channels but VLC reports that there should be five (and my ears confirm that dialogue is missing).

Sorry, hijacking your thread a bit!

Ah... VLC tells me that there were two streams initially, but then it changes - so maybe I can viciously cut the file and save the transition. The syslog output above is simply saying what was there as the recording started, not what went into it later...

I'm sure this had been fixed. It was a limitation on mkv containers IIRC.

I've been trying to read up on it... it's a confusing world though. I think UK DVB-T2 (aka Freeview HD) uses AAC LATM for audio, I'm not quite sure what the LATM means as such, to do with how the data is packed I think. But that's the bit that's not so widely supported.

Based on reading + experimenting, I /think/ it's to do with the metadata on the stream as well as the stream itself. I also think some tools use/pay more attention to one/the other, hence the inconsistency. I also think there's differences between 3.2 and 3.3 TVH (latest in both cases).

I've not yet found a stream, mkv or ts, where the audio data is missing (there are some permutations I've not looked into as much though) - it's just labelled/packed/something'd incorrectly - so you may be able to force the correct audio out. TS seems better/more consistent than MKV though, which is inline with what other threads say, and also what adamsutton said in a brief IRC conversation last night.

Couple of points:

1. https://en.wikipedia.org/wiki/Advanced_Audio_Coding (LATM = Low Overhead
Audio Transport Multiplex - don't ask me the details)

2. If its TS then its nothing to do with the contents of the file, since
(assuming its TS passthru) it's exactly how the broadcaster provides the
data (hence the argument for using TS to record stuff). So its the player.

3. For MKV there are definitely issues with stream changes, though in most
cases the data is there (if mislabelled) and XBMC may be able to handle
that. However I've not tried any of this as I pretty much only use freesat
(which doesn't use AAC) and TS passthru.

Adam

On 9 April 2013 21:05, wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3?
Ben Lumley

I've been trying to read up on it... it's a confusing world though. I
think UK DVB-T2 (aka Freeview HD) uses AAC LATM for audio, I'm not quite
sure what the LATM means as such, to do with how the data is packed I
think. But that's the bit that's not so widely supported.

Based on reading + experimenting, I /think/ it's to do with the metadata
on the stream as well as the stream itself. I also think some tools use/pay
more attention to one/the other, hence the inconsistency. I also think
there's differences between 3.2 and 3.3 TVH (latest in both cases).

I've not yet found a stream, mkv or ts, where the audio data is missing
(there are some permutations I've not looked into as much though) - it's
just labelled/packed/something'd incorrectly - so you may be able to force
the correct audio out. TS seems better/more consistent than MKV though,
which is inline with what other threads say, and also what adamsutton said
in a brief IRC conversation last night.
------------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

Thanks for that - definitely seeing more success with TS. The problem transcoding seems to be tools (specifically, linux/osx cli ones) that can see the channel change in the stream - presumably the same issue as lots of players seem to have with the switch.

Latest ffmpeg compiled from source according to the docs here https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide seems to be ok though. It flashes up that audio changes are detected, and provided you force 6 channel output, it will then correctly transcode. This does have the downside that if the content is 2 channel, you get 4 blank channels… so you can't use PLII on it.

Working with TS, currently a command like this is working:

ffmpeg -ac:a 6 -i "$file.ts" -map 0:0 -map 0:1 -map 0:1 -c:a ac3 -r:a 48000 -b:a:0 192k -b:a:1 448k -ac:a:0 2 -ac:a:1 6 -c:v copy -strict 2 -sn "$file.ac3.ts"

End up with a 2 channel and a 6 channel soundtrack after that, both ac3. If the 6 channel has the blank channels, then you can use the 2 channel for PLII.

Ideally it'd be good if the ac3 switched from 2-6 channels with the source, but can't find a way to do that. My post process script wraps the above command, and ffmpeg does spit out the channel info as it goes, including changes in channel config with config info, so it may be that I can grep that to decide whether it is 5.1 or 2 channel and drop the streams as appropriate.

Above doesn't work with MKV at all though - get something like:

[mpegts @ 0xaacfce0] pts (2764800) < dts (2772000) in stream 0
av_interleaved_write_frame(): Invalid argument

Ben

On Wednesday, 10 April 2013 at 15:08, [email protected] wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/boards/5/topics/8194?r=8222#message-8222)
Adam Sutton
Couple of points:
1. https://en.wikipedia.org/wiki/Advanced_Audio_Coding (LATM = Low Overhead
Audio Transport Multiplex - don't ask me the details)
2. If its TS then its nothing to do with the contents of the file, since
(assuming its TS passthru) it's exactly how the broadcaster provides the
data (hence the argument for using TS to record stuff). So its the player.
3. For MKV there are definitely issues with stream changes, though in most
cases the data is there (if mislabelled) and XBMC may be able to handle
that. However I've not tried any of this as I pretty much only use freesat
(which doesn't use AAC) and TS passthru.
Adam
On 9 April 2013 21:05, wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3?
Ben Lumley
I've been trying to read up on it... it's a confusing world though. I
think UK DVB-T2 (aka Freeview HD) uses AAC LATM for audio, I'm not quite
sure what the LATM means as such, to do with how the data is packed I
think. But that's the bit that's not so widely supported.
Based on reading + experimenting, I /think/ it's to do with the metadata
on the stream as well as the stream itself. I also think some tools use/pay
more attention to one/the other, hence the inconsistency. I also think
there's differences between 3.2 and 3.3 TVH (latest in both cases).
I've not yet found a stream, mkv or ts, where the audio data is missing
(there are some permutations I've not looked into as much though) - it's
just labelled/packed/something'd incorrectly - so you may be able to force
the correct audio out. TS seems better/more consistent than MKV though,
which is inline with what other threads say, and also what adamsutton said
in a brief IRC conversation last night.
------------------------------
You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: https://tvheadend.org/my/account

I think I'm finally going to take the hint on this one and move to .ts, despite my better judgement. I've lost too many recordings to .mkv now.

As for this particular file - I've mmg-ed it, ffmpeg-ed it, subjected it to the full force of avidemux and Handbrake, and hurled faad, lame and others at it with extreme prejudice; I've chopped it, transcoded it, demuxed and remuxed it; I've downmixed it, upmixed it, passed it through, and stuffed it into every number of channels from mono to 7.1 surround; I've tried it in XBMC, Mplayer, VLC, BSPlayer. Nothing.

So, unless the family really likes subtitles, I surrender. I don't suppose anyone's got the soundtrack to Dead Poet's Society to hand, have they? :-)

Adam - serious point - if this is a genuine limitation of mkv that can't be easily worked around, is it actually worth keeping mkv available? Or, at the very least, should the limitation be more publicly advertised?

Well done, I made that move about 6 months ago and have been very happy

I still convert the stuff I keep to MKV, but I do the original recording to TS to ensure I have the right data and let me control the remuxing off-line where I know if things go wrong I've still got the original source data.

With regard to changes in the streams, yes this is a limitation of MKV. It can handle some changes in the stream, if the audio codec changes but the stream ID doesn't we just keep bundling the new codec into the old stream. XBMC (other players) are smart enough to detect that change.

There are things you can do about this, such as segmented MKV, but this has very little support. It's simply a case that MKV was never designed to deal with broadcast content where this sort of thing is common. In recorded material its less frequent, since generally a given programme/show will only ever have one set of codecs for its entirety.

Adam

Yeah, the fact that you keep saying that you use TS has swayed me a little... I should've taken the hint months ago...

I stayed away from TS because of the stream errors: it makes it much harder to cut the file down without transcoding or running it through ProjectX first - and if you cut the transcoded version, you frequently have a damaged index, which causes seeking issues on VLC (at least), although XBMC copes.

However, after several years' use, I've only just noticed that Handbrake allows you to select a portion of a file to recode. So I can chop out pre/post-amble segments and re-encode what's left in one step, so I can sleep easily now!

Do you not find that handbrake loses the 5.1 audio if/when there is any… unless the clip starts with 5.1, because it analyses the start to find out what the streams are? So you probably still need to pre-cut into the 5.1 section, rather than cut with handbrake, so that it will pick up 5.1 on it's analysis. (I've not tried this - I wanted something to work unattended, so didn't want to be manually cropping)

Out of interest, what handbrake settings are you using for HD content? Specifically the video? I'm still trial+erroring what I'm using.

On Wednesday, 10 April 2013 at 22:31, [email protected] wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/boards/5/topics/8194?r=8230#message-8230)
Prof Yaffle
Yeah, the fact that you keep saying that you use TS has swayed me a little... I should've taken the hint months ago...
I stayed away from TS because of the stream errors: it makes it much harder to cut the file down without transcoding or running it through ProjectX first - and if you cut the transcoded version, you frequently have a damaged index, which causes seeking issues on VLC (at least), although XBMC copes.
However, after several years' use, I've only just noticed that Handbrake allows you to select a portion of a file to recode. So I can chop out pre/post-amble segments and re-encode what's left in one step, so I can sleep easily now!
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: https://tvheadend.org/my/account

I cant say if Handbrake eats the audio on TS yet - I hope it doesn't! - but I'm busy seeking things out on BBC HD that may have a 5.1 track so I can do some testing. If it does then I'll have little choice but to ProjectX it first because mmg won't chop a TS file without choking otherwise. I could try avidemux, I suppose, but that pretty universally borks out on H.264 streams on any platform/version I've tried.

HB settings... constant quality, RF19, HP, CBR (comskip on Windows chokes on VBR). Audio, I then just pass through. I really transcode MPEG2/SD programmes to tidy them up, sort out some interlacing issues, a squash them; HD programmes don't really compress much (well, films don't - cartoons do), so I'm only really doing that to either crop out C4HD's logo, prepare them for comskip or fix the indexing artefacts left by mmg. Which I can't use on TS anyway, which is why I went down the mkv route in the fist place...

I can check the full suite of settings later if you wish, but I think I've some experimenting of my own to do yet. If HB eats the audio on these as well, I think I'll have to resign myself to (a) ProjectX, which I've no idea if it runs in batch/unattended/automated/CLI mode, or (b) just leaving the blighters alone!

Ta for that. Your comment on comskip explains a few things!

If you want to get a sample from BBC - record the 'this is bbc hd' bits that get put on BBC One HD in place of the local news. They start as stereo and transition to 5.1 fairly early in the programme - assume they are all the same. I've used the lunch time one for testing a few times.

Re the audio - be interested to know what you find, see if it matches what I've seen/think I've seen. Re the audio issue - if you aren't on freeview, it'll be ok - satellite use aac ac3. If it's freeview, with the AAC LATM audio, handbrake can't pass it through for some reason like it can normal AAC. So it re-encodes and drops it to stereo with varying success.

On Thursday, 11 April 2013 at 10:37, [email protected] wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/boards/5/topics/8194?r=8233#message-8233)
Prof Yaffle
I cant say if Handbrake eats the audio on TS yet - I hope it doesn't! - but I'm busy seeking things out on BBC HD that may have a 5.1 track so I can do some testing. If it does then I'll have little choice but to ProjectX it first because mmg won't chop a TS file without choking otherwise. I could try avidemux, I suppose, but that pretty universally borks out on H.264 streams on any platform/version I've tried.
HB settings... constant quality, RF19, HP, CBR (comskip on Windows chokes on VBR). Audio, I then just pass through. I really transcode MPEG2/SD programmes to tidy them up, sort out some interlacing issues, a squash them; HD programmes don't really compress much (well, films don't - cartoons do), so I'm only really doing that to either crop out C4HD's logo, prepare them for comskip or fix the indexing artefacts left by mmg. Which I can't use on TS anyway, which is why I went down the mkv route in the fist place...
I can check the full suite of settings later if you wish, but I think I've some experimenting of my own to do yet. If HB eats the audio on these as well, I think I'll have to resign myself to (a) ProjectX, which I've no idea if it runs in batch/unattended/automated/CLI mode, or (b) just leaving the blighters alone!
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: https://tvheadend.org/my/account

I have both Freeview (T2) and Freesat (S2) on the same system, so that's part of the fun - most of the time, I have no idea which tuner is feeding me the HD signal. I seem to recall that BBC 1 is at least "BBC One HD" on Freeview and "BBC 1 England/Wales/NI/... HD" on Freesat, so that should give me a clue.

Thanks for the tip on the sample, I can maybe use that to do some investigation, rather than waiting for random films to come on that may or may not have the right soundtrack.

Where's my shovel, time for some digging...

EDIT

BBC ONE HD is the same on both S2 and T2 - but BBC TWO HD only exists on T2, with regional BBC 2 variants on S2. So, with some gratuitous tuner blocking before the event, I should be able to get a usable sample from each tuner.

Okay, I've got a couple of samples that I've been playing with. I'm not sure I've yet solved my problem of how to chop down .ts files reliably - the Freesat one plays nicely, but the Freeview one chokes. However, you might like to try this with a recentish ffmpeg build ... I'm playing with 1.1.1:

./ffmpeg -i input.ts -sn -vcodec copy -acodec ac3 -ac 6 output.mkv

My cloth ears and this stereo PC can't determine whether or not it really is still six channels - but the initial stereo mapping seems to move to (some components of) 5.1 and the main six-channel stream seems to be there. I'd need to play it on something with real depth to test it, though. It does report as being a 6-channel, ac3-encoded track, though...

On the other hand, you may have already tried this, given your initial post!

EDIT

Again, I haven't thumped it through a 6-channel amp, but Handbrake now correctly sees the audio as 5.1 AC3 and allows AC3 passthru while cutting the ends off the file as required. So I took a 1Gb sample and turned it into a quarter of that size on RF20 (all that red background in the sample compresses well, I suspect) and it appears to be fine. I'll play it on the amp later to see what it sounds like versus the original or the Freesat sample.

Fingers crossed. No fat ladies breaking into song just yet, however.

Yeah - that's pretty much my workflow too...

Your ffmpeg command does the same as the one i posted above, except i'm adding a stereo channel too, so that for non 5.1 content, I can pass that to the receiver - it won't PLII process a 5.1 stream, even if the surround/center/lfe channels are empty.

Do you use handbrake on the command line? My command is this at the moment... loosely based on the 'high' preset in the handbrake gui, plus dropping the res down to 720 from 1080 (can't really spot a difference - suspect my tv is too small)

HandBrakeCLI -i "$path" -o "$dir/$name.processed.mkv" -e x264 -q 19 -Y 720 -a 1,2 -E copy -f mkv --decomb --loose-anamorphic --crop 0:0:0:0 -m -x b-adapt=2:rc-lookahead=50:level=-1

Not quite happy with that yet though...

I'm a GUI user at the moment - just shifting HB from my Windows laptop to my 'buntu server, now I've got something with enough horsepower to be worth trying. I know that the Windows version will tell me what the CLI options are, so I can pull those for you if you like?

I don't know how much I'm prepared to automate... I could use ffmpeg to identify AAC with a simple grep, and then re-code the audio track and/or re-package the streams as part of that. But that won't help me tune the crop parameters (e.g. adding a few pixels to C4HD autocrop takes out the very top of the image but obliterates the channel DOG).

Happy to dig out the coding parameters I'm using, though, so I'll pull those when I get to that PC.

Cropping - I've settled on no crop rather than doing it manually - handbrake's auto crop seems to lose the top/bottom bit of the frame, same as you are seeing.

FWIW, this seems to be the command line my current GUI profile is generating...

HandBrakeCLI.exe -i inputfile -t 1 -c 1 -o outputfile  -f mkv --decomb -w 1920 --loose-anamorphic  -e x264 -q 20 --cfr  -a 2 -E copy -B 0 -6 auto -R Auto -D 0 --gain=0 --audio-copy-mask none --audio-fallback ffac3 -x b-adapt=2:rc-lookahead=50 --verbose=1

That works really well on SD/MPEG-2 files, and avoids any nasty combing artefacts due to interlacing. Perhaps 40% reduction in size plus I'd say the final image is better than the source because of some of the averaging-out over the macroblocks.

It doesn't do so much on OTA HD/H.264 sources since they're already pretty well compressed (it's more useful on BR rips where you can easily get a 60%+ filesize reduction with no discernible loss, even at lower RF values). It typically shrinks files by maybe 10%-20%, I think.

Ta for that - very similar to what I'm using. Will try the cfr option, see what difference it makes though.

On Friday, 12 April 2013 at 14:06, [email protected] wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/boards/5/topics/8194?r=8249#message-8249)
Prof Yaffle
FWIW, this seems to be the command line my current GUI profile is generating...
HandBrakeCLI.exe -i inputfile -t 1 -c 1 -o outputfile -f mkv --decomb -w 1920 --loose-anamorphic -e x264 -q 20 --cfr -a 2 -E copy -B 0 -6 auto -R Auto -D 0 --gain=0 --audio-copy-mask none --audio-fallback ffac3 -x b-adapt=2:rc-lookahead=50 --verbose=1
That works really well on SD/MPEG-2 files, and avoids any nasty combing artefacts due to interlacing. Perhaps 40% reduction in size plus I'd say the final image is better than the source because of some of the averaging-out over the macroblocks.
It doesn't do so much on OTA HD/H.264 sources since they're already pretty well compressed (it's more useful on BR rips where you can easily get a 60%+ filesize reduction with no discernible loss, even at lower RF values). It typically shrinks files by maybe 10%-20%, I think.
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: https://tvheadend.org/my/account

Another thought - I use StaxRip for BR encodes where I want to keep the HD audio stream, since Handbrake always transcodes these. I wonder if it would do a better job for you for these Freeview recordings?

It uses several common underlying tools - eac3to, x264 et al - but packages them in a pretty tidy front end.

Windows only I think? Come across it before, and eac3to, which otherwise
looked good.

I use a Mac and a headless ubuntu server, so not too handy for me.
On 14 Apr 2013 11:11, wrote:

Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3?
Prof Yaffle

Another thought - I use StaxRip for BR encodes where I want to keep the HD
audio stream, since Handbrake always transcodes these. I wonder if it would
do a better job for you for these Freeview recordings?

It uses several common underlying tools - eac3to, x264 et al - but
packages them in a pretty tidy front end.
------------------------------

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://tvheadend.org/my/account

Damn these operating system wars... whatever happened to POSIX? Can't be that hard to update it for AV encoding ...

Yes, Windows-only, which is an increasing problem for me as the number of Linux derivatives proliferate in the house. So unless you want to bootcamp your machine or can do it as a VM (which is what I'm considering), I guess that's a dead end. Ah, well, it was worth a punt!