Tvheadend

Hi, I have the same issue. I am using Ubuntu 18.04 (HTS Tvheadend 4.2.8-4~g9f7ab681a~bionic) on a new Dell mini i3 with 4g memory. I get freezing and log indicates (transport stream AAC-LATM, DTS discontinuity. DTS = 169337376, last = 168884256 and stream H264, DTS discontinuity. DTS = 169333164, last = 168881213). I never had this issue when I used libreelec tvheadend. I thought it was the Ethernet data being too fast but the problem still exist with wifi connection. It maybe a bug within bionic tvheadend. Did you file a ticket? Respond back if you come up with a solution. It takes a long time for a specialist to respond here.

Lisa

Only other thing you haven't mentioned is amount of swap space/swap used, though I'd expect some out of memory (OOM) messages to be logged if you had a memory leak using all your memory.

It's possible that the vaapi is buggy and causes a lock up since it's very low-level. If you don't use vaapi does your machine lock up? That would use a lot more CPU so would stress your machine/fan far more so would generally rule out most hardware issues.

Have you tried running "ffmpeg" with "vaapi" outside tvheadend (at a console) to convert a large recording and determine if vaapi is causing the problem. You could run it in a loop overnight and see if your machine survives.

Is your machine completely locked up so not even the keyboard works?

I'd run a "top" in another virtual console and leave that console as the active one, so when the machine dies you can see what was running at the time and amount of cpu/memory used. On a second run, I'd probably run "powertop" and see if that gave an indication of your machine doing things (and not responding) or not doing anything (and not responding).

I'm not sure this will help resolve the issue, but might help rule out something causing issues.

Thanks @Em Smith for your review of this issue.

Only other thing you haven't mentioned is amount of swap space/swap used, though I'd expect some out of memory (OOM) messages to be logged if you had a memory leak using all your memory.

The machine has 4 GB RAM and also 3.7 GB swap, but I think swap is never used. In fact, I checked free mem and there is always available

If you don't use vaapi does your machine lock up?

I created a stream profile to transcode using libx264 without hw acceleration, it´s continuously buffering (top reports 140% CPU usage) but the machine didn't freezes. It seems to freeze only using VAAPI.

Have you tried running "ffmpeg" with "vaapi" outside tvheadend (at a console) to convert a large recording and determine if vaapi is causing the problem. You could run it in a loop overnight and see if your machine survives.

Could you please confirm what would be the command to run ffmpeg with vaapi using a H264 stream as input, or where I can find a guide for this?

Is your machine completely locked up so not even the keyboard works?

Yes, correct. Nothing works, the console is totally black and keyboard is not working. After the freeze I need to power off the machine, then when it comes up I tried to find any crash info or log, but I am not able to find anything.. perhaps I am not searching correctly..

I'd run a "top" in another virtual console and leave that console as the active one, so when the machine dies you can see what was running at the time and amount of cpu/memory used.

This is the output of the top at the moment of a freeze:

top - 20:12:55 up 22:13,  1 user,  load average: 0.42, 0.50, 0.60
Tasks: 140 total,   1 running,  83 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.6 us,  4.7 sy,  0.0 ni, 83.3 id,  6.1 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  3738940 total,   783892 free,   594664 used,  2360384 buff/cache
KiB Swap:  3738620 total,  3738620 free,        0 used.  1989736 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  691 hts       20   0 3509524 226992  34728 S  33.1  6.1  61:10.91 tvheadend
19489 root      20   0       0      0      0 I   6.3  0.0   0:06.50 kworker/1:2
    3 root      20   0       0      0      0 I   1.7  0.0   0:46.66 kworker/0:0
 1292 asterisk  20   0 2666736  46392  24744 S   0.7  1.2  11:11.80 asterisk
17766 root      20   0   42800   4104   3456 R   0.7  0.1   0:19.23 top
    8 root      20   0       0      0      0 I   0.3  0.0   0:33.11 rcu_sched
    1 root      20   0  225172   9092   6892 S   0.0  0.2   0:08.65 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.08 kthreadd

As you can see CPU usage is low, in this case I was running a single stream with transcoding (H264 1080i -> H264 720p with 2000 bitrate). It also reports 783K memory free. I will run powertop this night.

Thanks!

Glad it's not swap problem. I didn't think it would be, but best to check.

Since it doesn't hang when not using vaapi, that suggests to me your hardware is physically ok (not overheating since CPU encoding is more demanding), so it's either a driver problem or how the code uses the converter. I'm not familiar with that code.

I believe you need something like this:

ffmpeg -init_hw_device vaapi=foo:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device foo -i input.mp4 -filter_hw_device foo -vf 'format=nv12|vaapi,hwupload' -c:v h264_vaapi output.mp4

from here: https://trac.ffmpeg.org/wiki/Hardware/VAAPI

(I don't have an Intel Linux machine I can test it on at the moment).

So "-i input.mp4" is saying filename to read. "output.mp4" is the output file.

You'd put it in a loop like this:

while :   <---note the colon
do
date
ffmpeg -y -init_hw_device vaapi=foo:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device foo -i input.mp4 -filter_hw_device foo -vf 'format=nv12|vaapi,hwupload' -c:v h264_vaapi -b:v 2M output.mp4
done

Assuming it's working, it will start output "fps" (frames per second) messages and creating the output.mp4. At the end, it will do it again until you stop it with Ctrl-C. (Note arguments are slightly different so I've added the "-y" to mean "yes overwrite the output file if it already exists" and "-b:v 2M" to mean 2mb/s). The guide suggests you change hwupload to be "hwupload,scale_vaapi=w=1280:h=720:format=nv12" (to do your scaling).

So, find a big recording such as the one that crashed, and copy it to "input.mp4" and try it. If you get an "output.mp4" then switch consoles and run top and see if anything locks up after a few hours.

So, I'd expect your frames per second to be quite high since you're using vaapi (70fps+?). I'd also hope it would lock up.

I don't know if the "sensors" command will help? It will log temperatures on your machine. You can probably run it with "watch sensors" or just "sensors".

One other thing, on your top it says "20:12:55" (which is the current time). I assume that your actual current time (when you checked the PC) was later than that so it crashed at 20:12:55 but you checked at, say, 9pm.

...because I was thinking it might be an issue like this:
https://bugs.launchpad.net/ubuntu/+source/intel-vaapi-driver/+bug/1751492
("VAAPI encoding is broken in Skylake").
I know yours isn't Skylake (looks like Bay Trail), but might be a similar issue that this ppa has fixed that isn't available in your distribution.

Using the link you provided I was able to run the ffmpeg command line. The following is the command I used, to simulate the transcoding parameters in TVHeadend:

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i test.ts -vf 'scale_vaapi=w=1280:h=720' -c:v h264_vaapi -b:v 2M -c:a copy output.ts

The output from ffmpeg is the following:

Input #0, mpegts, from 'test.ts':
  Duration: 01:00:58.88, start: 16352.112411, bitrate: 9614 kb/s
  Program 1
    Metadata:
      service_name    : CARACOL TV HD
      service_provider: CARACOL TELEVISION SA
    Stream #0:0[0x400](spa): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:1[0x3e9]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 59.94 tbr, 90k tbn, 59.94 tbc
    Stream #0:2[0x3ea](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 448 kb/s
    Stream #0:3[0x3eb](spa): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 256 kb/s
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (native) -> h264 (h264_vaapi))
  Stream #0:2 -> #0:1 (copy)
Press [q] to stop, [?] for help
WARNING: Invalid RefPicListX[] entry!!! It is not included in DPB
Output #0, mpegts, to 'output.ts':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (h264_vaapi) (High), vaapi_vld, 1280x720 [SAR 1:1 DAR 16:9], q=0-31, 2000 kb/s, 29.97 fps, 90k tbn, 29.97 tbc
    Metadata:
      encoder         : Lavc57.107.100 h264_vaapi
    Stream #0:1(eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 448 kb/s
[mpegts @ 0x5581e415fb60] PES packet size mismatch1:00:56.88 bitrate=2677.6kbits/s speed=3.49x
frame=109622 fps=105 q=-0.0 Lsize= 1195950kB time=01:00:58.48 bitrate=2677.9kbits/s speed=3.49x
video:891375kB audio:200044kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.577609%

I created the loop and it did run for about 6 times without a single issue! So the ffmpeg with VAAPI is working fine. It seems to be something related to the driver in TVHeadend.

BTW, can I use TVHeadend with ffmpeg as transcoder for these H264 streams? Thanks!

If you mean transcode after recording has completed then this might help to run a post-processing script:
https://tvheadend.org/boards/4/topics/23915?r=23940#message-23940

with a bit more general info here:
https://tvheadend.org/projects/tvheadend/wiki/Tvheadend_post_recording_scripts

So, you'd replace the ffmpeg command in the first link, using "$1" where we have "input.mp4" and "converted-$1" is where we have "output.mp4". End the whole command with " &&" so it looks something like this:

#!/bin/bash
...
/usr/bin/ionice...ffmpeg…-i "$1"...."converted-$1" > /dev/null 2>&1 &&
mv -f "converted-$1" "$1" 

Make MPEG-TS SPAWN profile.
It will be more easy.

Thanks for the answer, but really I was thinking in using ffmpeg to transcode live streams (not recordings). I was searching in the forum and found that using spawn profiles I can include a ffmpeg command line for the transcoding. However I was not able to make it work.. I did the following steps:

1) Created the streaming profile using MPEG-TS/Spawn built-in
2) Added the ffmpeg command in "Command line". Replaced the input with INPUTLINK and output with pipe:1. Tested with and without "pipe://" at the start.
3) Used "video/mp2t" as mime type. Tried also with "video/h264"
3) Created a user "test" and assigned this profile to the user
4) In Kodi client, configured to log in with this user

I attached a screenshot of the profile configuration.

However the profile is not used! When a stream is started in the Kodi client is uses the default htsp profile.. I know this works as I have other user with the VAAPI transcoding profile and when that user is logged in the VAAPI profile is being used. Also the spawn profile is not found in TVHClient in Android.. Is there something that I am missing?

If profile isn't found then perhaps try Config->Users->Access Entries, find the user, edit, advanced settings, "streaming profiles". Perhaps you need to tick the ones to use? Though, since your image says your profile is default, it should just use it.

I don't use those settings, but hopefully someone else will know.

Any errors logged on the tvheadend log?

This page suggests you might need some extra flags:
https://tvheadend.org/projects/tvheadend/wiki/Custom_MPEG-TS_Input

-mbd rd -copyinkf -flags +ilme+ildct -fflags +genpts

Command line in profile need to have input and output pipe

Confirm that in documentation, is not clear and informative about spawn profile,
but there is some examples in forum.

ffmpeg -i pipe:0 /processing commands/ pipe:1

Thanks to Saen and Em I was able to have the spawn profile working, but only for Dream Player in Android. Whenever I connect with Kodi clients it uses the htsp profile.. although in both cases are connected with the same user "test", which is configured to use only the ffmpeg profile.

The command line in the spawn profile is the following:

/usr/bin/ffmpeg -loglevel fatal -threads 4 -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i pipe:0 -vf 'scale_vaapi=w=1280:h=720' -c:v h264_vaapi -b:v 2M -c:a copy -f mpegts pipe:1

When using Dream Player the stream is working fine and I get the following in the log:

2019-02-05 11:47:15.269 subscription: 004E: "HTTP" subscribing on channel "CARACOL HD 2", weight: 100, adapter: "Silicon Labs Si2168 #0 : DVB-T #0", network: "Col", mux: "473MHz", provider: "CARACOL TELEVISION SA", service: "CARACOL HD 2", profile="ffmpeg", hostname="192.168.15.252", username="test", client="VLC/3.0.4 LibVLC/3.0.4" 
2019-02-05 11:47:15.270 spawn: Executing "/usr/bin/ffmpeg" 
2019-02-05 11:47:20.570 spawn: WARNING: Invalid RefPicListX[] entry!!! It is not included in DPB
2019-02-05 11:47:58.404 subscription: 004E: "HTTP" unsubscribing from "CARACOL HD 2", hostname="192.168.15.252", username="test", client="VLC/3.0.4 LibVLC/3.0.4" 

When using Kodi I get the following:

2019-02-05 11:46:46.469 subscription: 004D: "192.168.15.136 [ test | Kodi Media Center ]" subscribing on channel "CARACOL HD 2", weight: 100, adapter: "Silicon Labs Si2168 #0 : DVB-T #0", network: "Col", mux: "473MHz", provider: "CARACOL TELEVISION SA", service: "CARACOL HD 2", profile="htsp", hostname="192.168.15.136", username="test", client="Kodi Media Center" 
2019-02-05 11:46:53.434 subscription: 004D: "192.168.15.136 [ test | Kodi Media Center ]" unsubscribing from "CARACOL HD 2", hostname="192.168.15.136", username="test", client="Kodi Media Center" 

Could you please confirm if there is something I must configure in Kodi or in the tvheadend server to allow Kodi to connect using that profile? Thanks!

I disabled the htsp profile in Tvheadend (just to test) and I got the following in the log after trying to open a stream from Kodi:

2019-02-05 13:57:35.186 profile: unable to select a working profile (asked '(null)' alt 'htsp')
2019-02-05 13:57:35.186 htsp: unable to create profile chain 'ffmpeg'

Any idea why Kodi cannot open the stream? Thanks!

try setting the profile by hand in Kodi. that is done in the tvheadend pvr settings in Streaming settings under Profile to use.

You can force use of profile in user profiles in TVH

Thanks for your answers, Dorin and Saen

try setting the profile by hand in Kodi. that is done in the tvheadend pvr settings in Streaming settings under Profile to use.

I already tried this, but when Kodi TVHeadend addon restarts, it generates an error message: "Streaming profile ffmpeg is not available"

You can force use of profile in user profiles in TVH

What I understand from your suggestion is to configure a specific streaming profile for the user. This is basically what I did, I assigned the profile ffmpeg to the user test, I am attaching an screenshot of the config for this user. However, when connected with this user in Kodi the profile htsp is always used, Kodi ignores the profile defined for this user..

Is there anything else I can check? Additional logging, debug, etc. that can be enabled to provide more info on the machine freeze? Thanks!

Exactly in this case no need to set it in client side, it will set by default.

I have the user configured with the profile, but Kodi is not using that profile, it uses "htsp". In the client side (kodi) I am not configuring a profile, I did it just for testing but it didn't work also..

Why Kodi ignores the profile configured for this user? Thanks.

Streaming
and streaming profile is different

In the "Streaming" field I initially had all the options enabled. I disabled HTSP, and now the config for this user is like in the attached screenshot. However Kodi client continues using HTSP as the streaming profile for this.. Is that the option you are refering to? Could you please provide more clarity in what setting should I modify? Thanks!

All applications with use API to communicate with TVH use HTSP protocol rest use streaming.

Ok, so my understanding is that Kodi addon will not work with spawn profiles as it uses API and therefore HTSP, correct? Is there any other addon or any similar app for a HTPC connected to a TV that allows spawn profiles?

Nope it will use set streaming profile on htsp protocol