Tvheadend

I have seen this problem with 1 user too when IPTV sources are unstable ot unavailable.

This might be related to the issue I found: #5295

Can you try with latest github master?
More specifically with this fix: https://github.com/tvheadend/tvheadend/commit/77fc1c05f1b6e9db34ea498063eff0f6bd0a2d37

I've noticed same when some IPTV source hangs.

Tvh keeps running previously executed tasks but stops accepting new connections, webui stops responding.

On top of that I've noticed following error messages in the log:

[ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?)

kodiaq kodiaq wrote:

I've noticed same when some IPTV source hangs.

Tvh keeps running previously executed tasks but stops accepting new connections, webui stops responding.

On top of that I've noticed following error messages in the log:

[ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?)

That's the behavior I had on #5295

The commit above fixed my issues, where you able to try it?

I have rolled it to all my machines and will take me more than a day to confirm, thanks for making me aware :)

@luis Parada - I am still getting same issue. Tvh webui becomes unavaliable, no new connections accepted with message [ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?). Tvh restart is the remedy. Weird thing about this is that there were no issues with IPTV streams at all since server started, so I will have to probably start new bug thread on this.

It looks like that some resources are exhausted. Could you look to number of the open file descriptors (ls -la /proc/PID/fd).

@Jaroslav - Not sure what to search for, there is just long list of pipes and sockets. But I've checked old logs and found a pattern. It seems that after cca 2000 started pipes Tvh gets exhausted.

How many files are there? Note that the kernel have limit for the file descriptor per process - https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

If I use (ls -la /proc/PID/fd) than there are 160 results with pipes a sockets.

Other commands from the link you shared for "hts" user:

cat /proc/sys/fs/file-max -> 808571
ulimit -Hn -> 1048576
ulimit -Sn -> 1024

So for test I have changed it to:

cat /proc/sys/fs/file-max -> 1000000
ulimit -Hn -> 1048576
ulimit -Sn -> 16384

@Jaroslav - I can now confirm that there were alltogether 1024 pipes as result of (ls -la /proc/PID/fd) when Tvh became unavailable. I can relate this to "ulimit -Sn" which I have previously changed from 1024 to 16384, but after server restart it changed back to 1024 (* I was not aware of it until now). Not sure how to enforce "ulimit -Sn" to higher number. Is there anything else I can try?

I've made a mistake by changing "ulimit -Sn" for user "hts". All pipes are executed under "root" so I have changed it to max limit 1048576 for root. Will test it once again.

@Jaroslav - I can now confirm that exhaustion is not related to "ulimit -Sn" limit. Tvh got exhausted again at the 1024 pipes even though I have set it to 1048576 yesterday. Weird thing about it is that there were only 3 channels running. Why there are so many pipes executed? It seems that the executed pipes are just getting acumulated during channel zapping and never properly closed. Or something like that.

The pipes should be closed correctly. Do you have zombie tasks in your system? Do you use the right signal to kill the spawned task?

You're right, pipes are not correctly closed. I've put "kill -- -$$" to the end of the bash scripts to see if it makes any difference.

@Jaroslav - Pipes are still getting acumulated. Command "kill -- -$$" is not being executed when changing stations. It seems that bash script containing pipe:///usr/bin/ffmpeg ... pipe:1 ignores all commands placed after pipe:1. What way shall we control correct closure of spawned tasks properly?

I meant KILL signal in the tvh mux settings (but it's the default, so it should be set already).

All IPTV muxes have kill signal set as SIGKILL. This is result of (ls -la /proc/PID/fd) after Tvh running for cca 24 hours: https://pastebin.com/BJXBXLmQ . There are no zombie tasks in the background. I was able to raise ulimit -n from 1024 to 65535 and this prevented Tvh from becoming stuck. Note that every hour there is acumulated "block with 10 pipes" regardless amount of real executed pipe tasks. Hard to understand correlation here for me. One last observation, if there is no channel watched for an hour, there is no "block with 10 pipes" generated for that particular hour (I have checked it on system which is rarelly used).

v4.3-1519-g1222de114 - enable '--trace iptv' and show logs when first pipe descriptors are not released (I need the /proc directory listing, too)

• v4.3-1519-g1222de114 - '--trace iptv' enabled -> https://pastebin.com/AnzxN3ZQ
• first pipe descriptors not released -> https://pastebin.com/va7AtwtQ
• /proc directory listing -> part 1 https://pastebin.com/qWrUiRds -> part 2 https://pastebin.com/V9XRV5Ee -> part 3 https://pastebin.com/Ytzgvvr9

You are creating the pipes with user "root".
Then you are trying to close the pipes with your tvheadend user (hts?) - I think that's why tvh can't close them.

Try to chmod the pipes to something like 777 so that any user can remove them.

Tvheadend should also send an error/warning message when it can't remove the pipe due to permissions.

I added the kill/permissions log line to v4.3-1547-ge9aefbf2c .