Tvheadend

Hello everyone,

I run TVH on a headless server equipped with an NVIDIA GPU for transcoding and I have identified an issue possibly related to VAAPI transcoding.

System Settings¶

OS: Ubuntu 20.04
CPU: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GH
GPU: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti]
TVH Version: 4.3.0~pre+202010141717-0~built202010151647~git7e1dac822~ubuntu20.04.1 (from the mamarley/tvheadend-git ppa)
Nvidia Driver Version: 450.80.02 (from Ubuntu repo)
Nvidia Cuda Compiler: release 10.1, V10.1.243 (from Ubuntu repo)
Ffmpeg Version: 4.2.4-1ubuntu0.1

The NVIDIA driver is patched using https://github.com/keylase/nvidia-patch

The Error¶

I get the following errors on the logs every time a stream is requested to the server:

Oct 20 02:59:22 Ubuntu_Server tvheadend[32285]: mpegts: playlist.m3u - Channel in IPTV Network - tuning on IPTV #1
Oct 20 02:59:22 Ubuntu_Server tvheadend[32285]: subscription: 0001: "HTTP" subscribing on channel "ChannelName", weight: 100, adapter: "IPTV #1", network: "IPTV Network", mux: "playlist.m3u - Channel", provider: "FFmpeg", service: "Service01", profile="nvidia-hevc-aac-webm", hostname="XXX.XXX.XXX.XXX", username="username", client="VLC/3.0.11.1 LibVLC/3.0.11.1" 
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: transcode: 0001: 01:H264: ==> Using profile nvidia-hevc
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: transcode: 0001: 02:AAC: ==> Copy
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: transcode: 0001: 01:H264: ==> Using profile nvidia-hevc
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: transcode: 0001: 02:AC3: ==> Using profile webtv-aac
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: libav: AVHWDeviceContext: libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: libav: AVHWDeviceContext: Failed to initialise VAAPI connection: -1 (unknown libva error).
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: vaapi: failed to create a context for device: /dev/dri/renderD128
Oct 20 02:59:24 Ubuntu_Server tvheadend[32285]: libav: AVCodecContext: Increasing reorder buffer to 2
Oct 20 02:59:27 Ubuntu_Server tvheadend[32285]: htsp: Got connection from XXX.XXX.XXX.XXX
Oct 20 02:59:27 Ubuntu_Server tvheadend[32285]: htsp: XXX.XXX.XXX.XXX: Disconnected
Oct 20 02:59:29 Ubuntu_Server tvheadend[32285]: subscription: 0001: "HTTP" unsubscribing from "ChannelName", hostname="XXX.XXX.XXX.XXX", username="username", client="VLC/3.0.11.1 LibVLC/3.0.11.1" 
Oct 20 02:59:29 Ubuntu_Server tvheadend[32285]: libav: AVCodecContext: Qavg: 118.000

The libav error is independent to the network source (IPTV vs DVB-T) and it happens every time a stream is requested.

Despite of the error, the stream starts and is transcoded by the NVIDIA GPU -after a start delay of 1-2 seconds:

$ nvidia-smi

Tue Oct 20 03:13:00 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   39C    P0    N/A /  72W |     80MiB /  4040MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     32285      C   /usr/bin/tvheadend                 78MiB |
+-----------------------------------------------------------------------------+

The TVH Transcoding Settings¶

These are the transcoding settings used in TVH to transcode the video:

Stream Profile:¶

Profile name: nvidia-hevc-aac-webm
Enabled: yes
Default: yes
Comment: NVIDIA profile HEVC/AAC/WEBM
Data timeout (sec) (0=infinite): 0
Default priority: Normal
Force priority: 0
Restart on error: yes
Continue if descrambling fails: yes
Descrambling timeout (ms): 2000
Switch to another service: yes
Preferred service video type: FHD: full high definition
Container: WEBM/av-lib
Video codec profile: nvidia-hevc (NVIDIA codec HEVC)
Source video codec: MPEG2VIDEO,H264,VP8,VP9,THEORA
Audio codec profile: webtv-aac (WEBTV codec AAC)
Source audio codec: MPEG2AUDIO,AC3,MP4A,EAC3,VORBIS,OPUS
Subtitle codec profile: Disabled
Source subtitle codec: None

Codec Profile:¶

Name: nvidia-hevc
Description: NVIDIA codec HEVC
Codec: hevc_nvenc: NVIDIA NVENC hevc encoder
Codec name: hevc_nvenc
Type: video
Enabled: yes
Deinterlace: yes
Height (pixels) (0=no scaling): 0
Hardware acceleration: yes
GPU number: 0
Preset: Default
Bitrate (kb/s) (0=auto): 0
Rate control: Auto
Pixel format: auto
Profile: auto
Quality (0=auto): 0

Ffmpeg¶

I can confirm that my Ffmpeg binary has NVENC support.

If I run:

for i in encoders decoders filters; do
    echo $i:; ffmpeg -hide_banner -${i} | egrep -i "npp|cuvid|nvenc|cuda|nvdec" 
done

I get:

encoders:
 V..... h264_nvenc           NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc                NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc_h264           NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc_hevc           NVIDIA NVENC hevc encoder (codec hevc)
 V..... hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)
decoders:
 V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)
 V..... hevc_cuvid           Nvidia CUVID HEVC decoder (codec hevc)
 V..... mjpeg_cuvid          Nvidia CUVID MJPEG decoder (codec mjpeg)
 V..... mpeg1_cuvid          Nvidia CUVID MPEG1VIDEO decoder (codec mpeg1video)
 V..... mpeg2_cuvid          Nvidia CUVID MPEG2VIDEO decoder (codec mpeg2video)
 V..... mpeg4_cuvid          Nvidia CUVID MPEG4 decoder (codec mpeg4)
 V..... vc1_cuvid            Nvidia CUVID VC1 decoder (codec vc1)
 V..... vp8_cuvid            Nvidia CUVID VP8 decoder (codec vp8)
 V..... vp9_cuvid            Nvidia CUVID VP9 decoder (codec vp9)
filters:
 ... hwupload_cuda     V->V       Upload a system memory frame to a CUDA device.

I am not sure how ffmpeg is called by TVH, but I have tried the following direct command (based on https://trac.ffmpeg.org/ticket/8376), which outputs the same error reported by the TVH logs:

$ /usr/bin/ffmpeg -vaapi_device /dev/dri/renderD128 -i yks.ts -vf 'format=nv12,hwupload' -c:v hevc_vaapi -qp 30 -c:a copy out.ts

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[AVHWDeviceContext @ 0x55b47547b8c0] libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[AVHWDeviceContext @ 0x55b47547b8c0] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
Failed to set value '/dev/dri/renderD128' for option 'vaapi_device': Input/output error
Error parsing global options: Input/output error

Thinking that it could have been a problem with the ffmpeg binary, I have compiled the latests ffmpeg sources with NVENC support following the guide at https://atlex00.com/linux/ffmpeg/. Sadly, I get the same error when running the command above.

VAAPI¶

When running "vainfo" on my box, I get the following:

$ vainfo

error: can't connect to X server!
libva info: VA-API version 1.7.0
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
vaInitialize failed with error code -1 (unknown libva error),exit

From what I have read, VAAPI does not need to have an X server running for transcoding. Yet, the error related to the vainfo command seems to be related to what's displayed on the TVH logs and Fffmpeg.

Final Thoughts¶

I am no expert on the topic. Based on what I have been able to gather, the problem might be related with libva, which might require to open a bug directly with Ubuntu. However, I wanted to open a ticket here first in case the issue has to do with your code or if someone can point me in the right direction explaining how ffmpeg and VAAPI are used by TVH.

I usually would not care about this bug because transcoding seems to be working fine. However, I am trying to set up a second TVH server that gets some of its content from the original server via IPTV and I am unable to create new services due to a "scan no data, failed" error while scanning the muxes.

What can be causing the libva error? Should I be using different versions of the Nvidia drivers? TVH? FFMPEG? Has anyone experienced similar issues in the past?

Thank you.