Project

General

Profile

Bug #5417

Each time the channel is changed, the memory is more occupied by the GPU -> NVENC -> Transcode

Added by Ronny M. almost 6 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Transcoding
Target version:
999
Start date:
2018-12-11
Due date:
% Done:

0%

Estimated time:
Found in version:
4.3
Affected Versions:

Description

Hallo @all,
with each channel change (with HW Transcode at NVENC) the memory is occupied more by the GPU -> NVENC since when this is so, unfortunately not known, since I tested this release last year ago with other hardware.

In the version 4.2.7 one can observe clearly that the GPU Ram is released after a switching process again.
At about 2500 MB occupancy comes in the TVH log the following message
"libav: AVHWDeviceContext: Failed to initialize VAAPI connection: -1 (unknown libva error)."
Until the restart of TVH no transcode is possible.
In my opinion, only the "every old" transcode process is not closed properly.

Thanks and greetings

Enclosed an output from nvidia-smi

Tue Dec 11 18:01:36 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P5000        Off  | 00000000:65:00.0 Off |                  Off |
| 34%   56C    P0    46W / 180W |   1499MiB / 16278MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     66068      C   /usr/bin/tvheadend                          1487MiB |
+-----------------------------------------------------------------------------+

Tue Dec 11 18:01:43 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P5000        Off  | 00000000:65:00.0 Off |                  Off |
| 34%   56C    P0    44W / 180W |   1507MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     66068      C   /usr/bin/tvheadend                          1495MiB |
+-----------------------------------------------------------------------------+

Tue Dec 11 18:01:44 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P5000        Off  | 00000000:65:00.0 Off |                  Off |
| 34%   56C    P0    44W / 180W |   1640MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     66068      C   /usr/bin/tvheadend                          1628MiB |
+-----------------------------------------------------------------------------+

Files

fix (1.76 KB) fix Ronny M., 2021-04-24 17:32

History

#1

Updated by Ronny M. almost 5 years ago

Hello John,

its been a year - and I have still the same issue.
On using htsp client memory is not freed up when stop streaming the channels but is stacks until the server goes down.
On M3U playlist + simple-XML it works like a charm.

Any Ideas? If you need more info - do not hesitate to come back to me.

BR,
Ronny

#2

Updated by Flole Systems almost 5 years ago

John was last active 2015, I'd assume that he's not working on this.

Also I can't reproduce this issue here.

#3

Updated by saen acro almost 5 years ago

Maby devs need to add https://github.com/NVIDIA/NvPipe to TVH

#4

Updated by Ronny M. almost 5 years ago

I think that's just an overlay to the "normal" nvdia sources,

"Please note that NvPipe acts as a lightweight synchronous convenience layer around the NVIDIA Video Codec SDK and doesn't offer all high-performance capabilities. If you're looking for ultimate encode/decode performance, you may want to consider using NvCodec directly."

Therefore, normal IPTV: destroy -> a -> no problem
HTPS IPTV: destroy -> b -> not complete -> problem

in version 4.2, a new subprocess has been built for each stream.
I found this better because Sub destroy and done.

BR
Ronny

#5

Updated by Rene Wagler over 4 years ago

PLS Someone must solve the Problem

#6

Updated by Flole Systems over 4 years ago

Rene Wagler wrote:

PLS Someone must solve the Problem

Please go ahead and do it then.

#7

Updated by Rene Wagler over 4 years ago

Nope, im not a programmer. But someone is working on 4.3

#8

Updated by Flole Systems over 4 years ago

Sorry to disappoint you but nobody is actively working on tvheadend at the moment, there are a few changes now and then but other than that there is not much work being done.

#9

Updated by Rene Wagler over 4 years ago

But TVH is the one and only DVB Server, isnt it? Why nobody cares about it? Cant understand this

#10

Updated by Flole Systems over 4 years ago

Let's start with yourself: Why don't you care? Why are you not spending time to investigate this issue? Seems like this issue is not that important if you don't put effort in it, totally fine with me but if your issue isn't important to yourself it's not important to me either (especially since I can't reproduce it and I am not affected by it at all).

#11

Updated by Rene Wagler over 4 years ago

Are u using nvenc with TVH? i think not, so u cant reproduce it. The Problem is i dont know witch source file is responsible for transcoding and channel switching

#12

Updated by Flole Systems over 4 years ago

Yes I am using nvenc on the latest tvheadend version.

#13

Updated by Rene Wagler over 4 years ago

and what happens it you stop live tv in the nvidia-smi ? Is the transcoding tread gone as it should ?

#14

Updated by Flole Systems over 4 years ago

Yes, Tvheadend completely disappears from that list when i stop the stream.

#15

Updated by Rene Wagler over 4 years ago

ok.. are u using cuda or only nvenc and cuvid?

#16

Updated by Flole Systems over 4 years ago

I am using nvenc and enabled Hardware acceleration, cuda is not really used in Tvheadend as far as I know.

#17

Updated by saen acro over 4 years ago

NVENC is part of CUDA instructions
/can be seen with "netstat" command/

@ Rene Wagler update drivers

#18

Updated by Rene Wagler over 4 years ago

i have 440.xx its actual enough. you can compile TVH with and without CUDA. NVENC and CUVID are seperate compile-options

#19

Updated by Flole Systems over 4 years ago

  • Assignee deleted (John Törnblom)
  • Target version set to 999

If you don't even want to spend the time to update to the latest version you really shouldn't complain that nobody is working on it. You don't even try to fix the issue, so I won't either.

Also no accurate Tvheadend version or debug log was presented either....

#20

Updated by Rene Wagler over 4 years ago

I cant switch to newest driver. Cause i use TVH in docker. And the driver is not the Problem!!

#21

Updated by Flole Systems over 4 years ago

  • Status changed from New to Invalid

Old driver is used, Tvheadend is used in Docker. Once someone can reproduce this on the latest Tvheadend version with the latest Nvidia Driver on a native System I can consider reopening this.

#22

Updated by Rene Wagler over 4 years ago

Flole Systems can u post your transcoding settings ?

#23

Updated by Rene Wagler over 4 years ago

what codec are u using? 264 or 265 ??

#24

Updated by Ronny M. about 4 years ago

Rene Wagler wrote:

what codec are u using? 264 or 265 ??

both codec´s have the same Problem, i dosn´t think this is a Problem with the Codec, i think this is a Problem with the Opening and Close from the Sub-Process.

Kind Regards
Ronny

#25

Updated by saen acro about 4 years ago

Ronny M. wrote:

both codec´s have the same Problem, i dosn´t think this is a Problem with the Codec, i think this is a Problem with the Opening and Close from the Sub-Process.

Kind Regards
Ronny

+1
Flushing unneeded memory need to be debugged.

#26

Updated by Rene Wagler about 4 years ago

glad im not alone.. but why Flole Systems does not have this bug ?

#27

Updated by saen acro about 4 years ago

Rene Wagler wrote:

glad im not alone.. but why Flole Systems does not have this bug ?

Because no one report bugs corectly
What OS/Kernel is used
What hardware platform
What devices/driver used
etc.

Valgrind debugging for example

#28

Updated by Rene Wagler about 4 years ago

what hardware are u using? me: runnign tvh in an unraid nvidia docker conatainer build with newest ffmpeg

#29

Updated by Ronny M. about 4 years ago

saen acro wrote:

Rene Wagler wrote:

glad im not alone.. but why Flole Systems does not have this bug ?

Because no one report bugs corectly
What OS/Kernel is used
What hardware platform
What devices/driver used
etc.

Valgrind debugging for example

OS: Debian (10.5)
Kernel: 4.19.132-1
Hardware: Motherboard Supermicro, CPU Intel(R) Xeon(R) Silver 4110
GPU: Nvidia Quadro P5000
Driver Version: 450.57
NVIDIA-SMI: 450.57
Cuda Version: 11.0
Docker: No
ffmpeg: Static from TVH Build
TVH Version: 4.3-1857

Debug: show the first Post.

#30

Updated by Leon Andersen almost 4 years ago

Ronny M. wrote:

Hallo @all,
with each channel change (with HW Transcode at NVENC) the memory is occupied more by the GPU -> NVENC since when this is so, unfortunately not known, since I tested this release last year ago with other hardware.

In the version 4.2.7 one can observe clearly that the GPU Ram is released after a switching process again.
At about 2500 MB occupancy comes in the TVH log the following message
"libav: AVHWDeviceContext: Failed to initialize VAAPI connection: -1 (unknown libva error)."
Until the restart of TVH no transcode is possible.
In my opinion, only the "every old" transcode process is not closed properly.

Thanks and greetings

Enclosed an output from nvidia-smi
[...]

Ronny M. wrote:

saen acro wrote:

Rene Wagler wrote:

glad im not alone.. but why Flole Systems does not have this bug ?

Because no one report bugs corectly
What OS/Kernel is used
What hardware platform
What devices/driver used
etc.

Valgrind debugging for example

OS: Debian (10.5)
Kernel: 4.19.132-1
Hardware: Motherboard Supermicro, CPU Intel(R) Xeon(R) Silver 4110
GPU: Nvidia Quadro P5000
Driver Version: 450.57
NVIDIA-SMI: 450.57
Cuda Version: 11.0
Docker: No
ffmpeg: Static from TVH Build
TVH Version: 4.3-1857

Debug: show the first Post.

Hello Ronny

did you find any solutions for this ?

Im have the same problem

OS: Ubuntu 20.04
Kernel: 5.4.0-58-generic
Hardware Motorbord MSI: CPU Core i7
GPU: Nvidia Quadro p2000
Driver 460.27.04
Nvidia-smi: 460.27.04
Cuda: 11.2
Docker: no
FFMPEG: Static self build
TVH Version: 4.3-1919

#31

Updated by Ronny M. almost 4 years ago

Hallo @Leon,
no, i have only a not so Wonderful Workaround ....

a Cronjob restart at 5:00 AM o'clock the TVHeadend Service ;-(

Sorry!

#32

Updated by Ronny M. over 3 years ago

Problem is Solve with this work / Change..
https://tvheadend.org/boards/5/topics/43530?r=44328#message-44328

File to change /src/profile.c

@ Admin
Ticket can close.

Regards Ronny

#33

Updated by Flole Systems over 3 years ago

It's not a proper fix (the bottom half of it needs to stay the old version) and the bug report will stay open until it's merged ;)

#34

Updated by Ronny M. over 3 years ago

okay we renamed is to
Worked but not correctly
or
a quick and dirty workaround ;-)

#35

Updated by Flole Systems over 3 years ago

  • Status changed from Invalid to Resolved
#36

Updated by stef stef over 3 years ago

bonjour
même problème pour moi.
OS: Ubuntu 21.04 server
Hardware CPU bi-xeon
GPU: Nvidia Quadro p2000
Driver 460.80
Nvidia-smi: 460.80
Cuda: 11.2
TVH Version: HTS Tvheadend 4.3.0~pre+202105051438-0

#37

Updated by Flole Systems over 3 years ago

Use english language and as you're not using an official build please contact the maintainer of your build for support.

Also available in: Atom PDF