Bug #1592
MKV generated is still incorrect for hardware decoders (e.g panasonic TV)
0%
Description
I have a brand new panasonic TV, with ethernet connexion and DLNA capabilities. I tested several material on the TV and they mostly worked. But none of the tvheadend recording were working although the TV supports mkv, H264, EAC3, ... Saving as TS and using mkvmerge to produce MKV make it work on TV. Same channel, same codec identifications by mediainfo. Firts MKV generated by mkmerge works, second not. The things I see is the
Height : 1 088 pixels
Original height : 1 080 pixels
Frame rate mode : Variable
Whereas working one has only
Height : 1 080 pixels
Frame rate mode : Constant
Can send sample of files.
Here are the complete mediainfo output:
mediainfo TF1-Le-transporteur.2013-02-03.20-50.mkv
General
Unique ID : 20373502504667733180652587223598795499 (0xF53CB3A0DBDB2F8C0BE7DB62D008AEB)
Complete name : TF1-Le-transporteur.2013-02-03.20-50.mkv
Format : Matroska
Format version : Version 4 / Version 2
File size : 1.64 GiB
Duration : 30mn 57s
Overall bit rate : 7 588 Kbps
Encoded date : UTC 2013-02-03 21:57:55
Writing application : mkvmerge v6.0.0 ('Coming Up For Air') built on Jan 21 2013 10:34:30
Writing library : libebml v1.3.0 + libmatroska v1.4.0
Video
ID : 2
Format : AVC
Format/Info : Advanced Video Codec
Format profile : [email protected]
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 30mn 57s
Bit rate : 7 053 Kbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : MBAFF
Bits/(Pixel*Frame) : 0.136
Stream size : 1.53 GiB (93%)
Default : Yes
Forced : No
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Audio #1
ID : 1
Format : E-AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_EAC3
Duration : 30mn 57s
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -119ms
Stream size : 28.3 MiB (2%)
Language : French
Default : Yes
Forced : No
Audio #2
ID : 3
Format : E-AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_EAC3
Duration : 30mn 57s
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -101ms
Stream size : 28.3 MiB (2%)
Default : No
Forced : No
Audio #3
ID : 4
Format : E-AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_EAC3
Duration : 30mn 57s
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -101ms
Stream size : 28.3 MiB (2%)
Default : No
Forced : No
root@funtwist:/videoRecords# mediainfo TF1-Doc-Martin.2013-01-28.20-50.mkv
General
Unique ID : 206141404959851820191439781740831413314 (0x9B1563F51F5FF7478327902710119442)
Complete name : TF1-Doc-Martin.2013-01-28.20-50.mkv
Format : Matroska
Format version : Version 2
File size : 3.35 GiB
Duration : 1h 6mn
Overall bit rate : 7 162 Kbps
Movie name : Doc Martin
Writing application : Tvheadend 3.3.338~gbb51160
Writing library : Tvheadend Matroska muxer
Original source form : TV
DATE_BROADCASTED : 2013-01-28 20:50:00
TVCHANNEL : TF1
PART_NUMBER : 3
Video
ID : 2
Format : AVC
Format/Info : Advanced Video Codec
Format profile : [email protected]
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Format settings, GOP : M=8, N=32
Codec ID : V_MPEG4/ISO/AVC
Bit rate : 6 635 Kbps
Width : 1 920 pixels
Height : 1 088 pixels
Original height : 1 080 pixels
Display aspect ratio : 16:9
Original display aspect ratio : 16:9
Frame rate mode : Variable
Original frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : MBAFF
Language : French
Default : Yes
Forced : No
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Audio #1
ID : 1
Format : E-AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_EAC3
Duration : 1h 6mn
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : 1mn 5s
Stream size : 61.3 MiB (2%)
Language : French
Default : Yes
Forced : No
Audio #2
ID : 3
Format : E-AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_EAC3
Duration : 1h 6mn
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : 1mn 5s
Stream size : 61.3 MiB (2%)
Language : qaa
Default : Yes
Forced : No
Audio #3
ID : 4
Format : E-AC-3
Format/Info : Audio Coding 3
Format settings, Endianness : Big
Codec ID : A_EAC3
Duration : 1h 6mn
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : 1mn 5s
Stream size : 61.3 MiB (2%)
Language : qaa
Default : Yes
Forced : No
Text #1
ID : 5
Format : S_DVBSUB
Codec ID : S_DVBSUB
Language : French
Default : Yes
Forced : No
Text #2
ID : 6
Format : S_DVBSUB
Codec ID : S_DVBSUB
Language : French
Default : Yes
Forced : No
Files
History
Updated by Eric Valette over 11 years ago
Updated by So I've tried various things that may help others hunting the bug. I first edited video track header to describe the video height and width in the same way mkvmerge does => no change. I then used mkvmerge to extract H264 track as elementary stream from a tvheadend non working mkv and remuxed it using mkvmerge => works (no audio of course but will see that later). Seems to prove that its not the H264 recorded byte stream per se that is problematic but its description or the way the panasonic TV interprets it.
I really would like to get exactly the same ts stream remuxed both by tvheadend and mkvmerge to analyze headers. Looking at ffmpeg code that correspond to what is in the avc.c file, I saw some additional checks have been introduced to the routine that rewrites the sps/pps and that is used in the video header to provide codec private data. Will try to update the code to ffmpeg current and see if it changes anything. BTW now that it link with libav, ffmpeg routines could be reused as is...
Updated by John Törnblom over 11 years ago
Updated by I'm guessing the subtitles might have something to do with it.
Updated by Eric Valette over 11 years ago
John Törnblom wrote:
I'm guessing the subtitles might have something to do with it.
Can try to remove subtitle from faulty mkv to see if this changes something. Will do and report. Note that the tv works with blueray-rip that have subtitles.
Updated by Eric Valette over 11 years ago
Eric Valette - wrote:
John Törnblom wrote:
I'm guessing the subtitles might have something to do with it.
Can try to remove subtitle from faulty mkv to see if this changes something. Will do and report. Note that the tv works with blueray-rip that have subtitles.
Also the PS as subtitle too and works...
Updated by Eric Valette over 11 years ago
Next batch of tests:
1) Removing all video subtitle tracks doesn't change anything,
2) Putting the video track as first track in the file without subtitles idem
3) Extracting the video track as elementary stream, deleting the original video track and re-adding the extracted ones produce a working mkv,
So I really believes this has to do with the video track description. Furthermore I believe this is in the video segment description as remuxing the complete files via mkvmerge changes the clusters layout but not the track description and also produce a non working mkv.
Updated by Eric Valette over 11 years ago
After analysis by one of our H264 expert it turns out that, as expected, the CodecPrivateData field is indeed incorrect and that it is probably due to the avc_find_startcode routine. The original code in ffmpeg has now evolved. This field is used by hardware to preprogram the decoder before it finds the sps/pps info directly in the stream.
So I'm going to retake the current ffmpeg code (and add the credits in this file).
Updated by Eric Valette over 11 years ago
Even the improved current ffmpeg avc_find_startcode code is incorrect and lead to generation of wrong CodecPrivateData field (and wrong sps/pps info). We had to dump the original stream and simulate parsing done during mkv remuxing to find what is wrong.
If someone has a 100% proven avc_find_startcode routine in C, I'm all ears. In the meantime I will test a fix to original ffmpeg code (mkvmerge code is correct by highly depends on C++).
Updated by Eric Valette over 11 years ago
I have a working version tested with two hardware decoders (including the panasonic TV) that also works fine on PC with hardware decoding like vdpau, xvba, software decoding. Anyone having problem with a hardware decoder may test
Updated by Adam Sutton over 11 years ago
Updated by - Status changed from New to Accepted
Eric,
Can you submit this via github PR and I'll see about getting it merged.
Ta
Adam
Updated by Eric Valette over 11 years ago
Adam Sutton wrote:
Eric,
Can you submit this via github PR and I'll see about getting it merged.
Yes I was just expecting some wider testing before pushing it. But I can do a PR for this single file for sure (any way 90% of it is ffmpeg code like previous version). I just fixed the location where ffmpeg code does not conform to standard!
Updated by Adam Sutton over 11 years ago
Sure, but if you think it fixes it and John gives the nod I'll merge it into master (post 3.4 branch) and we can use trial by crowd (it works quite well 
and you'll at least get feedback if it breaks something else).
Adam
Updated by Eric Valette over 11 years ago
Adam Sutton wrote:
Sure, but if you think it fixes it and John gives the nod I'll merge it into master (post 3.4 branch) and we can use trial by crowd (it works quite well
Adam
Works for me on all my hardware platform. Run it at home. Have quite a few but...
Updated by Eric Valette over 11 years ago
Adam Sutton wrote:
Sure, but if you think it fixes it and John gives the nod I'll merge it into master (post 3.4 branch) and we can use trial by crowd (it works quite well
Adam
PR done.
Updated by Adam Sutton over 11 years ago
I can't seem to find the PR? Eric do you still want this included?
Adam
Updated by Eric Valette over 11 years ago
Adam Sutton wrote:
I can't seem to find the PR? Eric do you still want this included?
Well I have it in my tree so personnaly I don't care. The bug is definitively there and you do have the file. I did a PR but now I shall create a dedicated branch to reuse the PR and I do not knpw how to do it... I could return the question : do you want to have correct code?
Updated by Rob vh over 11 years ago
Updated by I have seen 15-20% of the MKVs fail on a Samsung TV, so I am interested in inherent improvements of MKV build too.
Updated by xraynorm - over 11 years ago
Updated by Just tested TVH recordings (.mkv) on a Panasonic TXL42E5B over DLNA and the all work fine (SD HD DVB-T)
HTS Tvheadend 3.3.491~g6990005~precise
Updated by Eric Valette over 11 years ago
xraynorm - wrote:
Just tested TVH recordings (.mkv) on a Panasonic TXL42E5B over DLNA and the all work fine (SD HD DVB-T)
HTS Tvheadend 3.3.491~g6990005~precise
Do not understand what we shall understand. First Did you test with or without the patch applied? The bug is there but depends on the way the NALU padding is done and thus depend on the DVB source H264 stream itself. When you are in a situation wrere the compuation for avc_find_startcode is incorrect the PrivateCodecDate generation and PPS/SPS generation is bugged. The stream I have with upstream generation ARE bugged. I do not see it on many devices because some chipsets use them for decoding while others do not. The TV is just what enabled to discover the bug.
Updated by xraynorm - over 11 years ago
No patch just standard install of 3.3.491.
UK Freeview.
I was not implying this was not an issue. Just that it works on my TV which is probably very similar to yours.
Updated by Eric Valette over 11 years ago
xraynorm - wrote:
No patch just standard install of 3.3.491.
UK Freeview.
I was not implying this was not an issue. Just that it works on my TV which is probably very similar to yours.
Your model is a LCD model while I have a plasma 3D one and the price range is quite diffrent so I doubt it uses the same hardware components. Can send you a bugged mkv if for testing purpose if you want to discriminate chipsets vs dvb source.
The fact that ffmpeg did change the code for detection compared to the old one still used in tvheadend is just an hint that the bug exists. I just needed to change a for loop, to a while one to skip one more 0 padding compared to current ffmpeg routine.
Updated by Eric Valette over 11 years ago
BTW : someone gave me the way to do a PR from an alaredy forked branch. And its not at all that simple when you main tree contains already many patches. Will try to send a new PR but I'm in the process of swapping my main PC so not before a few days... If someone as simple git commands to create a new branch to and existing tree with the patch already included I'm glad to try it.