Bug #4774
Tvheadend hanging when IPTV source unavailable
0%
Description
It seems that tvheadend is hanging when multiple users connect to IPTV sources that are not available. The tvheadend process shows it is running but it does not respond at all (CPU usage is less than 1% with top command).
Of course A/V library is being used here to change containers and I think the a/v library pipe is not timing out properly with SIGKILL or something.
History
Updated by Andreas Fornberg almost 7 years ago
I have seen this problem with 1 user too when IPTV sources are unstable ot unavailable.
Updated by Luis Alves about 6 years ago
This might be related to the issue I found: #5295
Can you try with latest github master?
More specifically with this fix: https://github.com/tvheadend/tvheadend/commit/77fc1c05f1b6e9db34ea498063eff0f6bd0a2d37
Updated by kodiaq kodiaq about 6 years ago
I've noticed same when some IPTV source hangs.
Tvh keeps running previously executed tasks but stops accepting new connections, webui stops responding.
On top of that I've noticed following error messages in the log:
[ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?)
Updated by Luis Alves about 6 years ago
kodiaq kodiaq wrote:
I've noticed same when some IPTV source hangs.
Tvh keeps running previously executed tasks but stops accepting new connections, webui stops responding.
On top of that I've noticed following error messages in the log:
[ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?)
That's the behavior I had on #5295
The commit above fixed my issues, where you able to try it?
Updated by kodiaq kodiaq about 6 years ago
I have rolled it to all my machines and will take me more than a day to confirm, thanks for making me aware
Updated by kodiaq kodiaq about 6 years ago
luis Parada - I am still getting same issue. Tvh webui becomes unavaliable, no new connections accepted with message [ ERROR]:iptv: Unable to start pipe 'pipe:///.../script.sh parameter' (wrong executable?). Tvh restart is the remedy. Weird thing about this is that there were no issues with IPTV streams at all since server started, so I will have to probably start new bug thread on this.
Updated by Jaroslav Kysela about 6 years ago
It looks like that some resources are exhausted. Could you look to number of the open file descriptors (ls -la /proc/PID/fd).
Updated by kodiaq kodiaq about 6 years ago
@Jaroslav - Not sure what to search for, there is just long list of pipes and sockets. But I've checked old logs and found a pattern. It seems that after cca 2000 started pipes Tvh gets exhausted.
Updated by Jaroslav Kysela about 6 years ago
How many files are there? Note that the kernel have limit for the file descriptor per process - https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Updated by kodiaq kodiaq about 6 years ago
If I use (ls -la /proc/PID/fd) than there are 160 results with pipes a sockets.
Other commands from the link you shared for "hts" user:
cat /proc/sys/fs/file-max -> 808571
ulimit -Hn -> 1048576
ulimit -Sn -> 1024
So for test I have changed it to:
cat /proc/sys/fs/file-max -> 1000000
ulimit -Hn -> 1048576
ulimit -Sn -> 16384
Updated by kodiaq kodiaq about 6 years ago
@Jaroslav - I can now confirm that there were alltogether 1024 pipes as result of (ls -la /proc/PID/fd) when Tvh became unavailable. I can relate this to "ulimit -Sn" which I have previously changed from 1024 to 16384, but after server restart it changed back to 1024 (* I was not aware of it until now). Not sure how to enforce "ulimit -Sn" to higher number. Is there anything else I can try?
Updated by kodiaq kodiaq about 6 years ago
I've made a mistake by changing "ulimit -Sn" for user "hts". All pipes are executed under "root" so I have changed it to max limit 1048576 for root. Will test it once again.
Updated by kodiaq kodiaq about 6 years ago
@Jaroslav - I can now confirm that exhaustion is not related to "ulimit -Sn" limit. Tvh got exhausted again at the 1024 pipes even though I have set it to 1048576 yesterday. Weird thing about it is that there were only 3 channels running. Why there are so many pipes executed? It seems that the executed pipes are just getting acumulated during channel zapping and never properly closed. Or something like that.
Updated by Jaroslav Kysela about 6 years ago
The pipes should be closed correctly. Do you have zombie tasks in your system? Do you use the right signal to kill the spawned task?
Updated by kodiaq kodiaq about 6 years ago
You're right, pipes are not correctly closed. I've put "kill -- -$$" to the end of the bash scripts to see if it makes any difference.
Updated by kodiaq kodiaq about 6 years ago
@Jaroslav - Pipes are still getting acumulated. Command "kill -- -$$" is not being executed when changing stations. It seems that bash script containing pipe:///usr/bin/ffmpeg ... pipe:1 ignores all commands placed after pipe:1. What way shall we control correct closure of spawned tasks properly?
Updated by Jaroslav Kysela about 6 years ago
I meant KILL signal in the tvh mux settings (but it's the default, so it should be set already).
Updated by kodiaq kodiaq about 6 years ago
All IPTV muxes have kill signal set as SIGKILL. This is result of (ls -la /proc/PID/fd) after Tvh running for cca 24 hours: https://pastebin.com/BJXBXLmQ . There are no zombie tasks in the background. I was able to raise ulimit -n from 1024 to 65535 and this prevented Tvh from becoming stuck. Note that every hour there is acumulated "block with 10 pipes" regardless amount of real executed pipe tasks. Hard to understand correlation here for me. One last observation, if there is no channel watched for an hour, there is no "block with 10 pipes" generated for that particular hour (I have checked it on system which is rarelly used).
Updated by Jaroslav Kysela about 6 years ago
v4.3-1519-g1222de114 - enable '--trace iptv' and show logs when first pipe descriptors are not released (I need the /proc directory listing, too)
Updated by kodiaq kodiaq about 6 years ago
- v4.3-1519-g1222de114 - '--trace iptv' enabled -> https://pastebin.com/AnzxN3ZQ
- first pipe descriptors not released -> https://pastebin.com/va7AtwtQ
- /proc directory listing -> part 1 https://pastebin.com/qWrUiRds -> part 2 https://pastebin.com/V9XRV5Ee -> part 3 https://pastebin.com/Ytzgvvr9
Updated by Luis Alves almost 6 years ago
You are creating the pipes with user "root".
Then you are trying to close the pipes with your tvheadend user (hts?) - I think that's why tvh can't close them.
Try to chmod the pipes to something like 777 so that any user can remove them.
Tvheadend should also send an error/warning message when it can't remove the pipe due to permissions.
Updated by Jaroslav Kysela almost 6 years ago
I added the kill/permissions log line to v4.3-1547-ge9aefbf2c .