Adding h264_v4l2m2m encoder for RPi4 to TVH...

I'm working on adding h264_v4l2m2m encoding support to TVH on Raspberry Pi. OMX is deprecated, and no longer works on the latest RPi4 OSes (Raspbian nor Ubuntu) in TVH. It seems v4l2m2m will be the method of choice in future for hardware encoding via ffmpeg on Raspberry Pi.
And luckily the ffmpeg that tvheadend builds already supports it without needing any special libs.

The h264_v4l2m2m encoder is added to the webui using the changes described below.

But right now I'm not getting any video at the output. The TVH debug log looks like this:

2022-03-07 22:19:50.962 subscription: 0087: "192.168.0.214 [ user | Kodi Media Center ]" subscribing on channel "Virgin Media 1", weight: 100, adapter: "DiBcom 7000PC #0 : DVB-T #0", network: "DVB-T Network", mux: "562MHz", provider: "RTÉNL", service: "Virgin Media 1", profile="webtv", hostname="192.168.0.214", username="user", client="Kodi Media Center" 
2022-03-07 22:19:51.439 transcode: 0007: 01:H264: ==> Using profile webtv-h264-v4l2m2m
2022-03-07 22:19:51.439 transcode: 0007: 02:MPEG2AUDIO: ==> Copy
2022-03-07 22:19:51.439 transcode: 0007: 03:MPEG2AUDIO: ==> Copy
2022-03-07 22:19:51.439 transcode: 0007: 06:PCR: ==> Filtered out
2022-03-07 22:19:51.439 transcode: 0007: 04:TEXTSUB: ==> Filtered out
2022-03-07 22:19:52.420 libav: AVCodecContext: Using device /dev/video11
2022-03-07 22:19:52.420 libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-07 22:19:52.420 libav: AVCodecContext: requesting formats: output=YU12 capture=H264
2022-03-07 22:19:52.428 libav: AVCodecContext: Encoder does not support b-frames yet
2022-03-07 22:19:52.429 libav: AVCodecContext: DTS/PTS calculation for V4L2 encoding
2022-03-07 22:19:52.429 libav: AVCodecContext:  is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

The first three lines in the log above from AVCodecContext are good. It shows its using the correct HW encoder device and bcm2835-codec driver, and these messages match what I get when using ffmpeg on a file using the h264_v4l2m2m encoder. So far, so good.

The next lines in the log are about B-frames and DTS/PTS calculation not being supported/implemented. I don't know if these are warnings or errors. From looking at libavcodec source code, they seem to be warnings. I'm guessing no B-frames is OK, OMX doesn't have this supported either.

I'm also guessing DTS/PTS calculation not implemented could be where the problem is.

But then I had a similar problem getting OMX working, no errors in the log, and the DTS/PTS do get set there, but I still get no video output: https://tvheadend.org/boards/4/topics/46727

To add h264_v4l2m2m in the GUI, these were the changes. I can submit a patch once I get it working and tested.

1. In /src/transcoding/codec/codec.c:
- add extern TVHCodec tvh_codec_v4l2m2m_h264; to the encoders section, and
- add tvh_codec_register(&tvh_codec_v4l2m2m_h264); to the tvh_codecs_register() in the TVHCodec section.

2. create new file /src/transcoding/codec/codecs/libs/v4l2m2m.c which is similar to omx.c, but replacing omx with v4l2m2m and with libname, libprefix and zerocopy removed from the struct.

3. add LIBS-CODECS += v4l2m2m in Makefile.

4. add ENCODERS += h264_v4l2m2m in Makefile.ffmpeg.

I'd also add a config option for this before making any patch.

With the above edits, TVH builds OK (just using ./Autobuild.sh -j4 with no other options for now), and h264_v4l2m2m is then available in the codecs list (screenshot attached).

Using ffprobe, I see the output has the copied audio streams, but no video:

ffprobe http://user:[email protected]:9981/stream/channel/6725f4a2a58c319c5645cb8a40434c06?profile=webtv
ffprobe version 5.0 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/opt/ffmpeg-builds/build/linux64/usr --extra-ldexeflags=-static --pkg-config-flags=--static --extra-cflags=-I/opt/ffmpeg-builds/lib/linux64/include --extra-ldflags=-L/opt/ffmpeg-builds/lib/linux64/lib --extra-libs='-lpthread -lm -lz' --extra-ldexeflags=-static --enable-gpl --enable-nonfree --enable-version3 --disable-debug --disable-indev=sndio --disable-outdev=sndio --enable-fontconfig --enable-frei0r --enable-openssl --enable-libaom --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --disable-libvidstab --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libdav1d --enable-libxvid --enable-libfdk-aac --enable-ffplay --enable-pic
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
[http @ 0x4c045c0] Stream ends prematurely at 71628, should be 18446744073709551615
Input #0, mpegts, from 'http://user:[email protected]:9981/stream/channel/6725f4a2a58c319c5645cb8a40434c06?profile=webtv':
  Duration: N/A, start: 1.446278, bitrate: 320 kb/s
  Program 1 
    Metadata:
      service_name    : RTÉ One
      service_provider: Tvheadend 0.0.0-unknown
  Stream #0:0[0x100](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s
  Stream #0:1[0x101](und): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, mono, fltp, 64 kb/s
  Stream #0:2[0x102](nar): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, mono, fltp, 64 kb/s

Would be great if anyone has any ideas on how to further debug why no video is getting to the output stream, either for this, or the OMX thread linked above.

v4l2m2m_tvh.png (20.1 KB) v4l2m2m_tvh.png

Replies (8)

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by Jay O over 2 years ago

This is from the debug log.

The H264 stream gets disabled by globalheaders after a timeout waiting for it:

2022-03-12 14:58:18.528 [  DEBUG]:libav: AVCodecContext: ct_type:0 pic_struct:2
2022-03-12 14:58:18.531 [  DEBUG]:transcode: 0001: 01:H264: [h264 => h264_v4l2m2m]: opts: flags=+global_header,b=1000000,zerocopy=0,tvh_filter_deint=0,width=362,height=384,pix_fmt=0,bf=3,tvh_require_meta=1,profile=0
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: probing device /dev/video12
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: v4l2 capture format not supported
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: probing device /dev/video16
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in unknown mode
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: probing device /dev/video15
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: v4l2 output format not supported
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: probing device /dev/video14
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: v4l2 output format not supported
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: probing device /dev/video13
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: v4l2 capture format not supported
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: probing device /dev/video11
2022-03-12 14:58:18.532 [  DEBUG]:libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-12 14:58:18.532 [   INFO]:libav: AVCodecContext: Using device /dev/video11
2022-03-12 14:58:18.532 [   INFO]:libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-12 14:58:18.532 [   INFO]:libav: AVCodecContext: requesting formats: output=YU12 capture=H264
2022-03-12 14:58:18.539 [  DEBUG]:libav: AVCodecContext: output: YU12 16 buffers initialized: 0362x0384, sizeimage 00221184, bytesperline 00000384
2022-03-12 14:58:18.541 [  DEBUG]:libav: AVCodecContext: capture: H264 04 buffers initialized: 0362x0384, sizeimage 00524288, bytesperline 00000000
2022-03-12 14:58:18.542 [WARNING]:libav: AVCodecContext: Encoder does not support b-frames yet
2022-03-12 14:58:18.542 [  DEBUG]:libav: AVCodecContext: Failed to set number of B-frames: Invalid argument
2022-03-12 14:58:18.542 [  DEBUG]:libav: AVCodecContext: Failed to get number of B-frames
2022-03-12 14:58:18.542 [WARNING]:libav: AVCodecContext: DTS/PTS calculation for V4L2 encoding
2022-03-12 14:58:18.542 [WARNING]:libav: AVCodecContext:  is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2022-03-12 14:58:19.041 [  DEBUG]:http: 192.168.0.100: HTTP/1.1 POST (3) /api/service/list -- 401
2022-03-12 14:58:19.102 [  DEBUG]:http: 192.168.0.100: HTTP/1.1 POST (3) /api/service/list -- 401
2022-03-12 14:58:23.614 [  DEBUG]:globalheaders: gh disable stream 1 H264 (PID 1106) threshold 1 qd 1 qd_max 451440
2022-03-12 14:58:23.614 [  DEBUG]:htsp: 192.168.0.100 [ user | Kodi Media Center ] - subscription start
2022-03-12 14:58:23.614 [  DEBUG]:htsp: 192.168.0.100 [ user | Kodi Media Center ] - first packet

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by Stefan Dietzel over 2 years ago

In the begining there existed a similar problem with VAAPI.

During this time the following line of code was added to vaapi.c (which I removed in my last pull request for VAAPI) maybe it helps for v4l2 now too:

// max_b_frames
    // XXX: remove when b-frames handling in vaapi_encode is fixed
    AV_DICT_SET_INT(opts, "bf", 0, 0);

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by Jay O over 2 years ago

Stefan Dietzel wrote:

In the begining there existed a similar problem with VAAPI.

During this time the following line of code was added to vaapi.c (which I removed in my last pull request for VAAPI) maybe it helps for v4l2 now too:

// max_b_frames
// XXX: remove when b-frames handling in vaapi_encode is fixed
AV_DICT_SET_INT(opts, "bf", 0, 0);

Thanks Stefan, this solved it and I'm now getting H264 output using v4l2m2m. I thought libav was just warning about B-frames and assumed it was then setting it 0 itself, but no, it was terminating.

There are some colour problems on the output, and aspect ratio is not being set, but I recognize these as issues other people have seen (and solved) in ffmpeg.

I think I know how to fix these in tvheadend, hopefully I can submit a pull request during the next week or so for enabling h264_v4l2m2m in tvheadend.

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by aim aim about 2 years ago

After reading up on how to get hw transcoding to work on the rpi4 I came to the realization that omx is deprecated and doesn't work under arm64. By googling for v4l2 as an alternative I stumbled upon this thread.
There are lots of threads by people struggling to get hw-accelerated transcoding to work on their rpis. I think if your solution worked properly this would be a game changer for many users. Because at today's energy prices an rpi is the perfect little home server and it would be so cool to make use of its hw-accelartion capabilities.

Sorry for the long preamble. I just wanted to ask, if you made any further progress. Can I look forward to a pull request (and eventually this being integrated)?

Cheers aimaim

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by Jay O about 2 years ago

I was never able to get it working. Close I think, but in the end I got an Up Board SBC, it's smaller than an RPi, for €40 with an Intel Atom processor which supports h264 hw encoding in tvheadend with vaapi.

I agree it would be great to get it working on RPi 4 with v4l2m2m.

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by aim aim about 2 years ago

What a pity :(
Thanks for the quick reply though :)
Have a good one.

EDIT:
It seems, that this guy figured it out: [[https://www.kodinerds.net/index.php/Thread/75265-Transcoding-auf-Handy-Tablet-mit-Tvheadend-Raspberry-4-l%C3%A4uft-endlich/]]
It works for me. The post is in German but I think the relevant parts are mostly language-independent. I can translate, in case there are any questions.

EDIT2:
I didnt't realize, that using MPEG-TS Spawn has the drawback, that it's not supported by the hts protocol. So the solution I linked to only works, if the client doesn't rely on htsp (TVHClient on Android e.g.).

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by aim aim about 2 years ago

I've just tried to reproduce your work but I'm unsure about:
"2. create new file /src/transcoding/codec/codecs/libs/v4l2m2m.c which is similar to omx.c, but replacing omx with v4l2m2m and with libname, libprefix and zerocopy removed from the struct."

For me the new codec profile does appear, yet it's not enabled and the enabled-flag is grayed out.

Could you please post your /src/transcoding/codec/codecs/libs/v4l2m2m.c ?

RE: Adding h264_v4l2m2m encoder for RPi4 to TVH... - Added by Jay O about 2 years ago

This is my v4l2m2m.c: https://pastebin.com/raw/5u30j9Ue

Actually looking at my notes from back in March, I think I did get this working, but the aspect ratio was not correct (but this may have been using the spawn settings method). I moved my focus away from Rpi shortly after that, so I haven't looked at this in over six months, but I'd be interested to hear if this compiles for you with the other edits I have posted above, and whether you get any video out from it.

(1-8/8)

Project

General

Profile

Tvheadend