Feature #3885
respawn pipe:// process if no data received for x time
0%
Description
I would like to suggest a small enhancement, an option to kill/respawn pipe:// process if no data received from it for x timeout...
Reason: I use pipe:// a lot with ffmpeg to grab input from live stream sources, problem is that frequently (due to network glitches, etc) ffmpeg with some streams just "stalls" reading input live source and outputs no more data at all (it outputs error messages if not set to run quiet), but also doesn't terminate process... in this case tvheadend just continues waiting for data forever... on Status - Stream tab in Tvheadend it shows stream with "Bandwidth (kb/s): 0" forever as seen in the screenshot. This is making the pipe:// feature a pita for me, as I need to frequently manually kill ffmpeg processes to get them respawn
Files
History
Updated by Nuno Silva over 8 years ago
Just to add a quick note: I did even noticed one other situation (other than when ffmpeg stalls) I did also reproduced a different issue, when ffmpeg actually terminates and respawns, but if it's source stream is temporarily unavailable then it will terminate again immediately (as it should) with no data outpt and this is shown on tvheadend log:
2016-07-09 08:13:05.427 spawn: Executing "/usr/bin/ffmpeg"
2016-07-09 08:13:05.729 iptv: stdin pipe unexpectedly closed: No data
however then tvheadend will not respawn again anymore (I expected it to respawn forever until it data is available) and subscription will stall like that forever.
Updated by Nuno Silva almost 8 years ago
Btw, I did found a good way to workaround the problem (in fact both the one I describe on the ticket and the one on my comment above) using a script like this:
#!/bin/bash pid=$$ ( while [ 1 ]; do bb=$aa; aa=$(stat -L /proc/$pid/fd/1 | grep Modify); if [ "$aa" = "$bb" ]; then pkill -P $pid ffmpeg; echo "kill ffmpeg" >/proc/$pid/fd/2; fi; sleep 20; done )& while [ 1 ]; do /usr/bin/ffmpeg -loglevel fatal -fflags +igndts -re -i "$1" -acodec copy -vcodec copy -metadata service_provider="$2" -metadata service_name="$2" -f mpegts pipe:1 sleep 3 echo "respawn ffmpeg" >&2 done
To use the script, on Mux settings I set URL like this:
pipe:///storage/scriptname.sh <live_stream_url> ${service_name}
fill "Mux Name" and "Service Name" fields... and no need to enable "Respawn (pipe)" option anymore...
The script will handle respawn and will kill it's ffmpeg "child" if there is no data flowing on the pipe for more than 20 seconds... I'm testing it for weeks of continuous streams and so far seems perfect, no stall streams on tvheadend anymore
I know this is not a forum... but hope my workaround can be useful to other users, that's why I'm posting it.
Updated by J Smith about 7 years ago
Nuno Silva wrote:
Btw, I did found a good way to workaround the problem (in fact both the one I describe on the ticket and the one on my comment above) using a script like this:
[...]
To use the script, on Mux settings I set URL like this:
[...]
fill "Mux Name" and "Service Name" fields... and no need to enable "Respawn (pipe)" option anymore...The script will handle respawn and will kill it's ffmpeg "child" if there is no data flowing on the pipe for more than 20 seconds... I'm testing it for weeks of continuous streams and so far seems perfect, no stall streams on tvheadend anymore
I know this is not a forum... but hope my workaround can be useful to other users, that's why I'm posting it.
Thanks for posting this, much appreciated. Using perl to rewrite the downloaded m3u to include this as a pipe, rather than just http directly. Stalling problems gone.
Updated by Dean Steele about 6 years ago
Nuno Silva wrote:
Btw, I did found a good way to workaround the problem (in fact both the one I describe on the ticket and the one on my comment above) using a script like this:
[...]
To use the script, on Mux settings I set URL like this:
[...]
fill "Mux Name" and "Service Name" fields... and no need to enable "Respawn (pipe)" option anymore...The script will handle respawn and will kill it's ffmpeg "child" if there is no data flowing on the pipe for more than 20 seconds... I'm testing it for weeks of continuous streams and so far seems perfect, no stall streams on tvheadend anymore
I know this is not a forum... but hope my workaround can be useful to other users, that's why I'm posting it.
Hi - does the script need ${service_name} to work? Could I just strip that out of the script? My M3U file doesn't have that at present, so I'm hoping it's just a nice to have...
(I'm at work right now, so I'm unable to check, else I would have just coded it and seen if it would work! )
Thanks.
Updated by Hiro Protagonist about 6 years ago
Dean Steele wrote:
Hi - does the script need ${service_name} to work? Could I just strip that out of the script? My M3U file doesn't have that at present, so I'm hoping it's just a nice to have...
(I'm at work right now, so I'm unable to check, else I would have just coded it and seen if it would work! )
What you put for the service name doesn't matter too much, so just use the channel name if you don't have a service name [remember to quote if the name has spaces].
Updated by Dean Steele about 6 years ago
Hiro Protagonist wrote:
Dean Steele wrote:
Hi - does the script need ${service_name} to work? Could I just strip that out of the script? My M3U file doesn't have that at present, so I'm hoping it's just a nice to have...
(I'm at work right now, so I'm unable to check, else I would have just coded it and seen if it would work! )What you put for the service name doesn't matter too much, so just use the channel name if you don't have a service name [remember to quote if the name has spaces].
Thanks - I've managed to embed the script inside a pipe, inside my m3u (as per this post - https://tvheadend.org/boards/4/topics/29249) but I'm now getting a fair few more respawns turning up in my log window. Loads, in fact.
Any ideas? My script is exactly what the one here is, but my system seems to be respawning ffmpeg all over the place at times...
Updated by Dean Steele about 6 years ago
UPDATE: I think I've worked out the respawn problem. Either due to the "clone entry on error" in the recording profile, or the "restart on error" option in stream ("pass") profile - switching both off seems to have stopped the respawns.
Updated by Ricardo Rocha about 6 years ago
J Smith wrote:
Nuno Silva wrote:
Btw, I did found a good way to workaround the problem (in fact both the one I describe on the ticket and the one on my comment above) using a script like this:
[...]
To use the script, on Mux settings I set URL like this:
[...]
fill "Mux Name" and "Service Name" fields... and no need to enable "Respawn (pipe)" option anymore...The script will handle respawn and will kill it's ffmpeg "child" if there is no data flowing on the pipe for more than 20 seconds... I'm testing it for weeks of continuous streams and so far seems perfect, no stall streams on tvheadend anymore
I know this is not a forum... but hope my workaround can be useful to other users, that's why I'm posting it.
Thanks for posting this, much appreciated. Using perl to rewrite the downloaded m3u to include this as a pipe, rather than just http directly. Stalling problems gone.
Could you be more specific about that implementation?
Updated by Dean Steele about 6 years ago
Ricardo Rocha wrote:
Could you be more specific about that implementation?
Check out this post - https://tvheadend.org/boards/4/topics/29249 - and the sixth reply (I think). It shows using a respawn script (continuousffmpeg) embedded in a pipe, that "envelopes" the web addresses in your referenced M3U. I've been running with this for about a week now...
Updated by Jaroslav Kysela about 6 years ago
- Status changed from New to Rejected
I think that a lot of confusion is here. The input restarts are controlled through the streaming profiles. It's a higher level in the tvh code. If the pipe stalls or aborts, then the problem is serious (it's not a normal situation).
So, if you like to abort on error, just turn off 'Restart on error' in the streaming profiles. If you like to restart the input completely, then turn this option on. I changed the timeout (data timeout) explanation in the streaming profiles, because the timeout is not only for the stream start but also for the stalled data checks. Basically, if there are no new data in the specified interval, the timeout error is sent to the upper levels and the abort/restart depends on the 'Restart on error' settings in the streaming profiles.