Tvheadend

Hi Antonio,

great to see you moving forward and I really appreciate it. I could imagine to spend some spare time working on it.

Well, talking about sw and hw en-/decoding is a bit "non-expert" as everything is running on hw :). The point is - as you already pointed out - the balancing, sequential vs. parallel. The RPi accommodate a NDA SoC with a very powerful GPU. So I believe the GPU has four cores (abbreviated from the name VideoCore IV). If you want less load on your CPU (which is doing more general tasks) you can offload most tasks to the highly parallel GPU. This is achieved by programming the CPU to offload tasks to the GPU. That means, there will be load on CPU - more or less for control, devices, dma etc. - and you will need or create something that will offload the tasks for you. For Multimedia there is OpenMaX (OMX) from the Krhonos Group that is also responsible for most Open* (OpenGL, OpenCL, OpenAL, OpenGL ES ....) standards.
To get the turn, there are OMX parts missing for ffmpeg and also the openmaxil library on raspbian.
MMAL is something ARM specific (MultiMediaAbstractionLayer I think) and therefore is CPU and not GPU related. Broadcom produces SoCs with IP from ARM and a custom GPU in the RPi case. There are also Broadcom SoCs for STBs that do "hw" transcoding running MIPS CPUs and custom GPUs.
It's a bit tricky as SoC can mean CPU + GPU but also only CPU and there are even more IP like Modems (3G, 4G ...), GPS and so on. The classic PC can do have all this too but it is commonly a separate device. Even PCs do have some special interfaces like vdpau, vaapi, nvenc.

The answer to Frank would be yes, but only with mpeg license and omxplayer. Sorry, I'am not really sure about omxplayer.

Antonio, what would you do or plan next?

MMAL can include hardware decoding - it's one of the two hardware acceleration options in Raspberry Pi Kodi. OMXPlayer and MMALPlayer are two different acceleration options. AIUI they access the VPU and GPU hardware via different software routes - but both use hardware acceleration. (De-interlacing was shifted between VPU and GPU I believe - but still doesn't hit the CPU)

Hi Alexander, on Monday I will write some details on the architecture I have in mind.

So, here I am :)
Lukely the current transcoding architecture, should make the modification for parallel transconding almost transparent to the rest of the TVHeadend system:

static void
transcoder_stream_video(transcoder_t *t, transcoder_stream_t *ts, th_pkt_t *pkt)

This function perform a complete transcode over one input video packed. The good thing is that the transcoded packet is sent via an async function called later in this function body:

if (got_output)
    send_video_packet(t, ts, pkt, &packet2, octx);

So, in my mind I have this idea: in the function transcoder_init_video together with the codec pickup we create also the threads we need to run in parallel. The transcoder_stream_video, instead of doing the transcoding, will sync these threads to perform their own transcoding step (or steps) iteration. The last stage (must be the encoder), if a packet is ready, calls send_video_packet. Every AVFrame output from one stage will be an AVFrame input to the next one. But there must be a pipeline of frames, since we run in parallel. So we need a number of AVFrame equal to the number of stages we run in parallel, and the stage process the older frame out from the previous, something like:

                
         | Stage0| Stage1| Stage2|
         | frame | frame | frame |
         | In|Out| In|Out| In|Out|     
In pkt 0 | 0 | 0 | x | x | x | x | None
In pkt 1 | 1 | 1 | 0 | 0 | x | x | None
In pkt 2 | 2 | 2 | 1 | 1 | 0 | 0 | Out pkt 0|
In pkt 3 | 0 | 0 | 2 | 2 | 1 | 1 | Out pkt 1|
In pkt 4 | 1 | 1 | 0 | 0 | 2 | 2 | Out pkt 2|
In pkt 5 | 2 | 2 | 1 | 1 | 0 | 0 | Out pkt 3|
In pkt 6 | 0 | 0 | 2 | 2 | 1 | 1 | Out pkt 4|
In pkt 7 | 1 | 1 | 0 | 0 | 2 | 2 | Out pkt 5|
In pkt 8 | 2 | 2 | 1 | 1 | 0 | 0 | Out pkt 6|
In pkt 9 | 0 | 0 | 2 | 2 | 1 | 1 | Out pkt 7|
In pkt a | 1 | 1 | 0 | 0 | 2 | 2 | Out pkt 8|
In pkt b | 2 | 2 | 1 | 1 | 0 | 0 | Out pkt 9|
In pkt c | 0 | 0 | 2 | 2 | 1 | 1 | Out pkt a|
In pkt d | 1 | 1 | 0 | 0 | 2 | 2 | Out pkt b|

So we have a frame pipeline of N stages and also a packet output latency of N, but it should not be a problem.
What do you think about?

Sounds like a really good plan! So this will actually be a general improvement that will benefit not only the Raspberry Pi builds?
Unfortunately I don't have the skills to help with the coding - I wish I could. Can you do it?

Thumbs up!

Antonio S wrote:

So, here I am :)
Lukely the current transcoding architecture, should make the modification for parallel transconding almost transparent to the rest of the TVHeadend system:

[...]

This function perform a complete transcode over one input video packed. The good thing is that the transcoded packet is sent via an async function called later in this function body:

[...]

So, in my mind I have this idea: in the function transcoder_init_video together with the codec pickup we create also the threads we need to run in parallel. The transcoder_stream_video, instead of doing the transcoding, will sync these threads to perform their own transcoding step (or steps) iteration. The last stage (must be the encoder), if a packet is ready, calls send_video_packet. Every AVFrame output from one stage will be an AVFrame input to the next one. But there must be a pipeline of frames, since we run in parallel. So we need a number of AVFrame equal to the number of stages we run in parallel, and the stage process the older frame out from the previous, something like:

[...]

So we have a frame pipeline of N stages and also a packet output latency of N, but it should not be a problem.
What do you think about?

This would be a welcomed feature for many that use Tvheadend on their Pis. If you're using the master branch please do consider making a PR

Hey Antonio,

nice to hear from you. I had to think about it to get your idea and resulting benefits. The conclusion is, that I am not your man for this. Nevertheless, I have some questions.

Is it right that your going to create the threads in init, sync them via stream and send them via send after all frames has been transocded?
Do you think it will be possible to split transcoding between the threads with ffmpeg?
Would the architecture make it possible to pass the number of threads to ffmpeg?

So it looks like I can't imagine what is going to be done in parallel. There is the pipeline where every stage does it's own thing and outputs to its successor stage. Last stage will output the the full transcoded frame within a packet.

Sorry for delay,

Alex.

Alexander Meinke wrote:

Hey Antonio,

nice to hear from you. I had to think about it to get your idea and resulting benefits. The conclusion is, that I am not your man for this. Nevertheless, I have some questions.

Is it right that your going to create the threads in init, sync them via stream and send them via send after all frames has been transocded?

Yes, this is the plan, however the _send will be called when the last stage (the encoder) has one packet to send

Do you think it will be possible to split transcoding between the threads with ffmpeg?
Would the architecture make it possible to pass the number of threads to ffmpeg?

It's possible and tvheadend apparently ask ffmpeg to run encoding in parallel with a number of threads equal to the CPU core detected. I have not checked if the information passed are consistent, but this is another topic. Parallelizing heavy computation stage over thread is of course good for multi core architecture, but still with the current transcoding code, every stage (decoding, deinterlacing, encoding) waits each other. With this architecture you will never get benefits from GPU accelerated capabilities

So it looks like I can't imagine what is going to be done in parallel. There is the pipeline where every stage does it's own thing and outputs to its successor stage. Last stage will output the the full transcoded frame within a packet.

Let me try to be more simplier here. With the current code, as said every stage waits each other. Suppose we have a GPU that is capable to accelerate each stages, le'ts say that this acceleration won't consume any CPU power. But let's say that every stage gets 1/50 seconds for complete the job. So we would have

decoding (takes 1/50sec, no CPU) -> deinterlacing (takes 1/50sec, no CPU) -> encoding (takes 1/50sec, no CPU)

For completing a single frame trascoding we need 3/50 sec. If the source frame rate is 1/25 (which is 2/50 sec) we don't do transcoding in real time, even if we are using 0% of the CPU!!
Can you see now the difference between CPU usage and real time? ;)
With a pipelined transcoding we will have, after the third frame punt in the pipeline

1) decoding of frame 2 (takes 1/50sec, no CPU)
2) deinterlacing frame 1 (takes 1/50sec, no CPU)
3) encoding of frame 0 (takes 1/50sec, no CPU) ---> output

So we perform the entire trnascoding in 1/50 sec with no CPU usage and we run realtime

Sorry for delay,

Alex.

No problem. I'm done with the coding of the pipeline, debugging now. Next step is to implement specific handler for the decoding/filtering/enconding stage (trivial) and then put these handlers in task (not complicated). Stay tuned

Hi
pipeline is working now, I'm developing on branch: https://github.com/Menion2k/tvheadend/tree/rpi/encoding
Still it run serial, so there is no improvement, but now I can move forward to the last two steps
Bye

Really great job. Antonio!
Can't wait to test it...

This is fantastic! Will this benefit hardware that has no hardware acceleration?

Hi all
I have completed (and committed in my branch) the implementation of the pipeline
Unfortunately I'm in a dead path now
This are the figures from a Rpi3 with standard clocking

1) Decoding SW stage takes betwen 18ms and 35ms
2) Deinterlacing with no resize takes between 35ms and 50ms
3) Encoding HW takes between 8ms and 15ms

So the problem is with the deinterlacing. It's purely done in SW and Rpi3 is too slow for the task (we must stay 100% of the times above 40ms)
There are two problems here: I cannot make the hw accelerated decoder working, I don't know why, probably there is something needed in the codec open. This would lower a little bit the CPU load and give more power to the deinterlacing, BUT, I'm not sure that ffmpeg supports multhread deinterlacing...
If, and only IF, we can fix the HW mpeg decoder and when ffmpeg would implements some MMAL accelerated deinterlacing filter, we may have some success, but for the moment there is nothing we can do for it. I will write a question to libav mailing list for the mmal deinterlacing, chance to get some answer are really low...

Hi Antonio,

I know I speak for many reading this thread when I say that your efforts are very much appreciated.

It would indeed appear to be a dead end. Having said that, thank you for taking the time and effort to investigate this to the degree that you have.

Regards
AAA

Agreed. Fantastic work on this, and thanks to everyone involved.

I wonder if all is not lost though.

1) Is it possible to not deinterlace and simply leave it for the player to do it? In Australia we broadcast interlaced h264, so it must be a thing...

2) I personally am interested not to much in a live stream but in the final file format. Perhaps this development could be leveraged to allow TVHeadend to record and compress to h264, if not in realtime? It would be nice to have it all built in without having to fuss about with post processing options etc (currently I use post processing to call a bash script which then calls ffmpeg)

Thanks again
Matt

Matt T wrote:

Agreed. Fantastic work on this, and thanks to everyone involved.

I wonder if all is not lost though.

1) Is it possible to not deinterlace and simply leave it for the player to do it? In Australia we broadcast interlaced h264, so it must be a thing...

It should be possible to do this IF you don't scale vertically - i.e. keep 1080i as 1080i or 576i as 576i and keep an identical field-rate.

It IS possible to scale in the interlaced domain too - by simply scaling fields, not frames, but you have to do it in the field domain, which may be tricky with some codecs. You don't get the same quality - but it was how early 'DVE's (Digital Video Effects devices that squeezed or zoomed and repositioned video digitally) worked.

If you try scaling in the frame domain without deinterlacing, you end up with all sorts of nasty artefacts (commonly seen as line-grouping, banding or 'wobbly edged motion) as field 1 motion leaks into field 2 and vice versa.

You've done an amazing job on improving the code!

I was wondering if de-interlacing is neccesarry when the source codec is H.264? So maybe we can still benefit from your improvements when the goal is to transrate (H.264->H.264 with lower bitrate)?

A big thanks also from me. I would also like what Morten is stating. For me all HD programs are H.264 and those are the ones I would like to transcode. The SD channels are MPEG-2 and those are at an acceptable bandwidth rate.

Dorin

Matt T wrote:

Agreed. Fantastic work on this, and thanks to everyone involved.

I wonder if all is not lost though.

1) Is it possible to not deinterlace and simply leave it for the player to do it? In Australia we broadcast interlaced h264, so it must be a thing...

2) I personally am interested not to much in a live stream but in the final file format. Perhaps this development could be leveraged to allow TVHeadend to record and compress to h264, if not in realtime? It would be nice to have it all built in without having to fuss about with post processing options etc (currently I use post processing to call a bash script which then calls ffmpeg)

Thanks again
Matt

No, it's not possible, at least it is not possible with the OMX encoder of Rpi3, that woeks only for progressive inputs

dorin ilut wrote:

A big thanks also from me. I would also like what Morten is stating. For me all HD programs are H.264 and those are the ones I would like to transcode. The SD channels are MPEG-2 and those are at an acceptable bandwidth rate.

Dorin

Hi
I will give a try with the HD channels

Antonio S wrote:

Matt T wrote:

Agreed. Fantastic work on this, and thanks to everyone involved.

I wonder if all is not lost though.

1) Is it possible to not deinterlace and simply leave it for the player to do it? In Australia we broadcast interlaced h264, so it must be a thing...

2) I personally am interested not to much in a live stream but in the final file format. Perhaps this development could be leveraged to allow TVHeadend to record and compress to h264, if not in realtime? It would be nice to have it all built in without having to fuss about with post processing options etc (currently I use post processing to call a bash script which then calls ffmpeg)

Thanks again
Matt

No, it's not possible, at least it is not possible with the OMX encoder of Rpi3, that woeks only for progressive inputs

Ah - so that means 576/50i and 1080/50i (i.e. all UK TV) would have to be re-encoded either to 25p or 50p (both requiring a deinterlace)... What a pain. (I guess the encoder was designed primarily to compress camera output in a mobile phone-type scenario - which would be natively progressive)

Hi Antonio,

Do not know if it is relevant but all my HD channels look like this. Check the attachment.

As anyone tried the omxtx transcoding option [[https://github.com/dickontoo/omxtx]]

Following the author it should be able to transcode on the pi
[[https://www.raspberrypi.org/forums/viewtopic.php?f=70&t=25022]]

What about using the hardware MMAL decoder to deinterlace? It is definitely capable of doing it, because my copy of Rasplex (based on Libreelec) does it. So the MMAL could decode, deinterlace, and then send it to the OMX for encoding.

And come to think of it, it can probably hardware scale too. I can play a 576i mpg2 video upscaling to 1080p with CPU at like 3%. So the deinterlacing and upscaling must be happening in hardware.

Hi.
MMAL supports deinterlacing, it works and it is used in Kodi for Rpi for example.
The problem is that ffmpeg doesn't support it. It would mean that someone should port the mmal deinterlacing from a SW that use it, such Kodi, VLC, gstreamer, to tvheadend.
And this task requires an amount of time that I simply don't have right now.
Moreover with my pipeline works, I have noticed some problem with the video rate. I suspect that having decoupled the sync between packet input and output requires some change in the internal engines, and this can be only done by some core developers... I'm here if the tvheadend team would talk about it, just don't ask me to use IRC, men we are in the 2016"
, there are better tools for collaboration