Bug #3732
IPTV network max input streams broken after #3273
100%
Description
Commit 2f71335 intended to fix #3273 seems to have broken the max input streams functionality of IPTV networks. I have two IPTV networks each configured with a max streams = 3. As soon as I build with this commit, the startup scan when the server is started tries to scan all muxes from both networks all at once. This brings the server to a halt where the web UI is not responding.
When I build with up to 1c95a19, everything is fine. Also, if I disable all muxes prior to switching to 2f71335, and then enable only one mux, I'm able to successfully scan and use that one mux.
As requested, logs with '--trace mpegts,service' are here:
1c95a19: http://sprunge.us/NeQC
2f71335: http://sprunge.us/GgUi
For 2f71335 the log grew to over 100meg in a very short time before I had to terminate the process (wouldn't respond to a normal interrupt). It appears to be in an uncontrolled endless loop. Due to size limits, I pasted just the first 10meg of the log.
Files
History
Updated by
Updated by Phil Uithoven over 8 years ago
Updated by Jaroslav Kysela wrote:
Could you retest with v4.1-1922-g3d4ac3e ?
Doesn't fix it - it looks the same I'm afraid:
g3d4ac3e: http://sprunge.us/RKie
Updated by David jrm over 8 years ago
Updated by - File tvheadend2.log tvheadend2.log added
Jaroslav Kysela wrote:
Could you retest with v4.1-1922-g3d4ac3e ?
Guys, I have 2 IPTV networks as well + HDHR device connected and outcome is a bit different. I don't see the above behavior, so when TVH start, the only stuff working is the HDHR as usual, but nothing with IPTV channels.
I have tested this once again and definitely the number of max streams match what the availability to play from clients, so if it is set as 1, just 1 mux can be run at same time, if set as 2, then 2, etc... However something strange happen as for each connected client, appears 2 connection to the server in the Web UI and in the subscriptions tab only the muxes running as expected. So not sure if this is related to the same root cause for Phil's issue or not, but something is not working fine.
See attached my logs as well (just the portion where I was playing with 2 muxes from same IPTV network with max#input set as 1 )
Thanks
Updated by Phil Uithoven over 8 years ago
Jaroslav Kysela wrote:
Could you retest with v4.1-1924-gb17d157 ?
Well, it's better, in that it doesn't go runaway and hang. It appears now though to only be using one stream at a time from each of my two networks instead of the 3 that I have them set for.
gb17d157: http://paste2.org/3EJsbNg1
Updated by Phil Uithoven over 8 years ago
Jaroslav Kysela wrote:
v4.1-1927-gac2d90e ?
Ah! Looking much better. I saw on the startup scan it was using exactly 3 streams at a time from both of my networks (correct behavior). I'll run this build and bang away at it for a while and let you know if I see any further issues.
gac2d90e: https://ptpb.pw/-YXs
sorry about all the different pastebins - seem to be having different issues with different ones at different times 
Updated by Phil Uithoven over 8 years ago
Phil Uithoven wrote:
Jaroslav Kysela wrote:
v4.1-1927-gac2d90e ?
Ah! Looking much better. I saw on the startup scan it was using exactly 3 streams at a time from both of my networks (correct behavior). I'll run this build and bang away at it for a while and let you know if I see any further issues.
Ok, issue right away. In this log you'll see a fresh restart, all the muxes get scanned (with the behavior looking good like I mentioned). Then at 10:49:31 I start a live playback from Kodi (this is at weight 150). I then start some recordings to fill up the other streams (recordings are weight 300). All looks good up to 3 streams. (BTW - I have priorities set on my 2 networks such that it fills up the higher priority network first, then should move on to the second). When I add a recording to make the 4th stream, I can see in the status that it first attempts to add a 4th stream from the first network - which fails (because the HDHR box only had 3 tuners - so it gets no output from the mux) - and then it fails over to the second network. This happens again for each additional stream instead of first trying an available stream from the second network. At the end, I then try to add another recording which would make it a total of 7 streams.
New log: https://ptpb.pw/JUqw
Updated by Phil Uithoven over 8 years ago
I just went back to 1c95a19 to verify - and it exhibits the expected behavior. When the first three streams are filled (1 Kodi weight 150, and 2 recordings weight 300), the next stream is started from the 2nd network (without trying the first). Then when I start the 6th recording (7th stream), it drops the Kodi stream because of the higher weight (with the above, the recording failed because it didn't kick Kodi).
Updated by David jrm over 8 years ago
Phil Uithoven wrote:
I just went back to 1c95a19 to verify - and it exhibits the expected behavior. When the first three streams are filled (1 Kodi
weight 150, and 2 recordingsweight 300), the next stream is started from the 2nd network (without trying the first). Then when I start the 6th recording (7th stream), it drops the Kodi stream because of the higher weight (with the above, the recording failed because it didn't kick Kodi).
Max#input is ok for me as well. On the other hand I still seeing these 2 connections per client show on Status-Connections tab in the Web UI (1 connection just when the client is connected and another one when the channel is being played) Is this normal?
Updated by Phil Uithoven over 8 years ago
David jrm wrote:
Phil Uithoven wrote:
I just went back to 1c95a19 to verify - and it exhibits the expected behavior. When the first three streams are filled (1 Kodi
weight 150, and 2 recordingsweight 300), the next stream is started from the 2nd network (without trying the first). Then when I start the 6th recording (7th stream), it drops the Kodi stream because of the higher weight (with the above, the recording failed because it didn't kick Kodi).Max#input is ok for me as well. On the other hand I still seeing these 2 connections per client show on Status-Connections tab in the Web UI (1 connection just when the client is connected and another one when the channel is being played) Is this normal?
I want to clarify that there is still an issue with the weighting and max streams as stated above in comment #8. I was just comparing the behavior change from 1c95a19.
Updated by Phil Uithoven over 8 years ago
Phil Uithoven wrote:
Phil Uithoven wrote:
Jaroslav Kysela wrote:
v4.1-1927-gac2d90e ?
Ah! Looking much better. I saw on the startup scan it was using exactly 3 streams at a time from both of my networks (correct behavior). I'll run this build and bang away at it for a while and let you know if I see any further issues.
Ok, issue right away. In this log you'll see a fresh restart, all the muxes get scanned (with the behavior looking good like I mentioned). Then at 10:49:31 I start a live playback from Kodi (this is at weight 150). I then start some recordings to fill up the other streams (recordings are weight 300). All looks good up to 3 streams. (BTW - I have priorities set on my 2 networks such that it fills up the higher priority network first, then should move on to the second). When I add a recording to make the 4th stream, I can see in the status that it first attempts to add a 4th stream from the first network - which fails (because the HDHR box only had 3 tuners - so it gets no output from the mux) - and then it fails over to the second network. This happens again for each additional stream instead of first trying an available stream from the second network. At the end, I then try to add another recording which would make it a total of 7 streams.
New log: https://ptpb.pw/JUqw
This is still an issue with latest build up to f027e3c
Updated by Phil Uithoven over 8 years ago
Phil Uithoven wrote:
Phil Uithoven wrote:
Phil Uithoven wrote:
Jaroslav Kysela wrote:
v4.1-1927-gac2d90e ?
Ah! Looking much better. I saw on the startup scan it was using exactly 3 streams at a time from both of my networks (correct behavior). I'll run this build and bang away at it for a while and let you know if I see any further issues.
Ok, issue right away. In this log you'll see a fresh restart, all the muxes get scanned (with the behavior looking good like I mentioned). Then at 10:49:31 I start a live playback from Kodi (this is at weight 150). I then start some recordings to fill up the other streams (recordings are weight 300). All looks good up to 3 streams. (BTW - I have priorities set on my 2 networks such that it fills up the higher priority network first, then should move on to the second). When I add a recording to make the 4th stream, I can see in the status that it first attempts to add a 4th stream from the first network - which fails (because the HDHR box only had 3 tuners - so it gets no output from the mux) - and then it fails over to the second network. This happens again for each additional stream instead of first trying an available stream from the second network. At the end, I then try to add another recording which would make it a total of 7 streams.
New log: https://ptpb.pw/JUqw
This is still an issue with latest build up to f027e3c
Still an issue with f34fac1aab4635c83f209ae31564ddf62c870f21 - 4.1-2140~gf34fac1
Updated by BoB The Cherub over 7 years ago
Updated by I am seeing this issue as well. It only appears to affect DVR connections. If I try to stream from multiple players (VLC, Kodi, etc...) then I am told an adapter is available. However, the DVR will create a new stream even though none are available, without first terminating an existing player.
Updated by Jaroslav Kysela over 7 years ago
Could you provide another log for this situation with '--trace service,mpegts,iptv-sub' ? https://tvheadend.org/projects/tvheadend/wiki/Traces
Please, use the latest v4.1-2442-g82aea16 .
Updated by BoB The Cherub over 7 years ago
- File tvheadend-subsystems.log tvheadend-subsystems.log added
- File tvheadend-iptv-sub-live-then-dvr.log tvheadend-iptv-sub-live-then-dvr.log added
- File tvheadend-iptv-sub-live-then-live.log tvheadend-iptv-sub-live-then-live.log added
Apologies for the delay, it took me a while to compile the latest build and deploy it to my systems.
I tried to set the debug settings to service,mpegts,iptv-sub as requested. However, the service subsystem was always removed whenever selected 'Apply Configuration'. I checked with 'tvheadend --subsystems' and can confirm 'service' is listed as a subsystem (see attached). Let me know what I am doing wrong here, and I will create another log.
In the meantime I have attached the log with just mpegts and iptv-sub trace enabled. During the period logged I started a channel (BBC1) on Kodi, and then selected a program to record on another channel (ITV3) via the tvheadend web interface. Both channels were fulfilled using the LDVB network, which has 'max-streams' set to 1. Its probably not easy to see from the log, but the kodi stream freezes shortly after the recording starts, as the IPTV provider detects multiple streams and closes the older stream. With both channels there is another IPTV network with available streams mapped which it could have used. This is tvheadend-iptv-sub-live-then-dvr.log
As mentioned previously, this only occurs when the dvr is pulling streams. I have also attached another log using live streaming from kodi only (tvheadend-iptv-sub-live-then-live.log). In this snippet I first start BBC1, and the LDVB stream is selected. When I start ITV3 via a live stream as well, it correctly falls back to the lower priority FPIPTV network, which has an available mux. I then stop ITV3 and attempt to start the Lifetime channel. This channel only has the a LDVB mux mapped, and the single LDVB stream is being used by BBC1 still. Tvheadend correctly fails to find an adapter for the channel, and this is reported back to kodi.
Let me know if you need any more information.
Updated by Jaroslav Kysela over 7 years ago
- Status changed from New to Fixed
- % Done changed from 0 to 100
Applied in changeset commit:tvheadend|52d1c4d4e53747cfad8e9cc5b31534dcbc2543b4.
Updated by Jaroslav Kysela over 7 years ago
Please, retest with v4.1-2448-g52d1c4d. Provide the same log, if things does not work as expected.
Updated by BoB The Cherub over 7 years ago
- File tvheadend-iptv-sub-live-then-dvr-2.log tvheadend-iptv-sub-live-then-dvr-2.log added
- File tvheadend-iptv-sub-new-mux-scan.log tvheadend-iptv-sub-new-mux-scan.log added
Apologies for the delay again. It takes a while for me to build and deploy.
This is certainly better when it comes to handling the max streams, but I think there might still be a couple of issues.
When I am watching a live stream, and a DVR recording begins, I was expecting the recording to either fall down to a lower priority network, or if one doesn't exist reschedule the recording. However, it actually removes the mux from the live stream for the DVR entry, and then tries to find a lower priority mux for the live stream. This is much more advanced behaviour than I was expecting. Unfortunately it doesn't quite work. The DVR entry takes the mux from the high priority network, and a lower priority mux is started for the live stream. When I look in the 'Status/Subscriptions' tab, I can see both muxes running with 'input' bandwidth. However, the live stream has no 'output' bandwidth. So I am guessing that even though both muxes are running, the live stream hasn't been hooked up to the output. See tvheadend-iptv-sub-live-then-dvr-2.log for the log output.
Unfortunately, it seems the issue with run-away infinite mux scanning reported earlier in this bug has come back as well. When scanning a new automatic IPTV network, it gets into a infinite loop, and effectively hangs tvheadend. The only option is to shutdown. I have attached the log (tvheadend-iptv-sub-new-mux-scan.log), but beware, its big due to the infinite loop. Some things I noticed when testing:
- The infinite loop only occurs if max streams is 0 or greater than 1.
- I couldn't cause the same error using a standard IPTV network, and manually adding muxes through the web interface.
- There is no infinite loop when adding an automatic IPTV network with only 1 mux.
So I think the loop happens only when more than 1 mux needs to be scanned, and there is more than 1 stream available.
Updated by BoB The Cherub over 7 years ago
I have noticed something else a little odd.
If I set 'max streams' to 2 on one of my IPTV networks, and then attempt to watch 2 programs simultaneously, it seems get into loop as well. It seems that both live streams are overriding each other in turn. Which is odd, as two streams should be available. See tvheadend-iptv-sub-live-then-live-2.log.
I think it must be kodi that is constantly re-initializing the live streams, causing them to take the stream from each other, as the loop doesn't occur if I initiate via the web interface and VLC, rather than kodi.
Updated by Mark Clarkstone over 7 years ago
Updated by BoB The Cherub wrote:
I have noticed something else a little odd.
If I set 'max streams' to 2 on one of my IPTV networks, and then attempt to watch 2 programs simultaneously, it seems get into loop as well. It seems that both live streams are overriding each other in turn. Which is odd, as two streams should be available. See tvheadend-iptv-sub-live-then-live-2.log.
I think it must be kodi that is constantly re-initializing the live streams, causing them to take the stream from each other, as the loop doesn't occur if I initiate via the web interface and VLC, rather than kodi.
Does turning off predictive tuning (in the pvr-hts client) help?
Updated by BoB The Cherub over 7 years ago
I checked, and can see I already have predictive tuning off.
Updated by Phil Uithoven over 7 years ago
- File tvheadend-bad.log tvheadend-bad.log added
- File tvheadend-good.log tvheadend-good.log added
I'm still running 1c95a19 because nothing since has worked properly - see comment #8. Now with the latest build, I'm right back to the same symptoms that I reported initially: start the process, and the web interface initially responds/loads, but as soon as it starts scanning muxes, the web interface becomes unresponsive. The log shows the endless looping of scanning that is just SPEEDING by (instead of the normal slow/controlled entries seen during proper scanning). Have to kill -9 in order to terminate it, as it will respond to no other signal.
I've attached logs with '--trace service,mpegts,iptv-sub' from both 1c95a19 (tvheadend-good.log) and 583493d (tvheadend-bad.log) to show the differences.
Updated by BoB The Cherub over 7 years ago
This is much better, with just 2 small issues.
Firstly the problem with infinite loops when scanning more than 1 mux is resolved.
It is also now correctly taking streams from the right networks, in the right order according to the priority. However, this only works if you add 1 to the max-streams value. To be precise:- If you set max-streams to 1, it will only use 1 stream from the network, but not necessarily correctly when it comes to priorities.
- If you set max-streams to 2, it will only use 1 stream from the network, and correctly follow the priority rules.
- If you set max-streams above 2, it will use n-1 streams from the network, and correctly follow the priority rules.
If a I higher priority profile (e.g. DVR) requests a stream, it still attempts to take an active stream from the lower priority profile and replace it with a stream from another network. As before this stream appears to start, but doesn't seem to be hooked up to the output.
Let me know for which scenarios you would like logs.
Updated by G Kazaroth over 7 years ago
Updated by Jaroslav Kysela wrote:
Try v4.1-2454-gd08a8b9 .
I would like to see if this also fixes issue #4272
I am on unstable and do not see 2454. How would I get the update in order to test it? or should I wait until it comes out on unstable? Ty
Updated by Mark Clarkstone over 7 years ago
George Kazaroth wrote:
Jaroslav Kysela wrote:
Try v4.1-2454-gd08a8b9 .
I would like to see if this also fixes issue #4272
I am on unstable and do not see 2454. How would I get the update in order to test it? or should I wait until it comes out on unstable? Ty
The repo is broken (still). Your only option for now is either build it yourself or you can manually install an unstable deb package from my doozer build branch (which is just master with changes to the build system), see here.
Updated by Phil Uithoven over 7 years ago
Jaroslav Kysela wrote:
Try v4.1-2454-gd08a8b9 .
Sorry for being slow, have had very little time for testing. Just tested with 4.1-2482~g0dbc5c986 and still seeing some of the N-1 described above. I have two IPTV networks defined with max streams set to 3. On startup, the scan used only two streams per network - only scanned two muxes at a time per network. In the middle of the scan progress, I started a couple Kodi streams (which are higher priority than scanning of course) and all scanning on that network stopped, even though there was a 3rd stream capability still possible.
Once the scans finished, I was able to do a mix of Kodi clients and recordings which filled all three streams per network (total of 6). It correctly chose the second network when starting the 4th stream (first network was filled with 3 streams) and then when starting a 7th higher-priority stream, it kicked the lower priority as expected.
Updated by Mark Clarkstone over 7 years ago
Phil Uithoven wrote:
Is this still being looked at?
Jaroslav is a very busy guy at the moment, but I'm sure he'll get around to looking at it eventually. Tvh really does need more developers!
Updated by Phil Uithoven over 7 years ago
Jaroslav Kysela wrote:
Upload commented logs, please.
See attached log with --trace service,mpegts,iptv-sub.
This shows a clean startup. It starts scanning muxes 2-at-a-time on both of my two networks. I start a kodi playback and a recording. Both subscriptions are serviced by muxes from network HDHR-6C1. At this point scanning on that network is stopped (even though the network is set for max of 3 streams).
at 6:47, scanning on the other network HDHR-E25 completes, so there are only two active streams. Yet there are still 49 muxes in the scan queue for network HDHR-6C1.
At 6:48, I stop Kodi playback. Then at 6:49 I stop the recording. shortly thereafter, scanning muxes in network HDHR-6C1 resumes.
I stop the server and pull the log.
Updated by Jaroslav Kysela over 7 years ago
I tried to make fix in v4.1-2501-gdb52b06 (just do git pull from the main tvh repo). Upload commented logs when something does not work.
Updated by Phil Uithoven over 7 years ago
- File tvheadend-crash.log tvheadend-crash.log added
Jaroslav Kysela wrote:
I tried to make fix in v4.1-2501-gdb52b06 (just do git pull from the main tvh repo). Upload commented logs when something does not work.
Unfortunately it now crashes shortly after startup. It runs long enough for Kodi to connect, but then it looks like it crashes immediately when the mux scan starts. Log attached.
Updated by Jaroslav Kysela over 7 years ago
Could you provide a backtrace from gdb ? https://tvheadend.org/projects/tvheadend/wiki/Debugging , I cannot do anything without the resolved symbolic names.
Updated by Jaroslav Kysela over 7 years ago
I found it - fixed in v4.1-2511-g77e6de9 (latest master).
Updated by Phil Uithoven over 7 years ago
- File 4.1-2498~gda7321a5d.log 4.1-2498~gda7321a5d.log added
- File 4.1-2520~geb3e25fb1.log 4.1-2520~geb3e25fb1.log added
Jaroslav Kysela wrote:
I found it - fixed in v4.1-2511-g77e6de9 (latest master).
Yes, that fixes the crash.
Yes, it now uses 3 streams per network when doing the scanning.
However, it now is having a different priority related issue. Now when adding a 4th stream, if that 4th stream is higher priority than any of the first 3, it'll kick one of those streams instead of just starting the 4th stream from the second network. I'm including logs from both the previous version I tested above (where I was saying it showed the N-1 in the scanning - but doesn't have this issue), and latest master.
--------------------------
4.1-2498~gda7321a5d
Start a stream in Kodi
Stream: E25 - 940 TWC in HDHR-E25
Subscription/service: IPTV/HDHR-E25/E25 - 940 TWC/940 TWC
6:51 start two recordings
6:52 start third recording
The third recording correctly starts on the second network - 6C1 - and everything is working, showing bandwidth, Kodi keeps playing
--------------------------
4.1-2520~geb3e25fb1
7:12 start a stream in Kodi
Stream: E25 - 940 TWC in HDHR-E25
Subscription/service: IPTV/HDHR-E25/E25 - 940 TWC/940 TWC
7:13 start two recordings
7:14 start third recording
Third recording is on channel 943 - which starts stream "E25 - 943 CNBC in HDHR-E25"
The stream for Kodi (channel 940 TWC) was booted off network E25, and starts a stream on "6C1 - 940 TWC in HDHR-6C1". Stream shows bandwidth. Subscription now shows service "IPTV/HDHR-6C1/6C1 - 940 TWC/940 TWC". Input shows bandwidth, output shows 0, Kodi is getting no incoming traffic and so picture hangs.
Updated by Jaroslav Kysela over 7 years ago
- Status changed from Accepted to Fixed
Applied in changeset commit:tvheadend|06a07bb5eb9cdd28fc93b431dda7de98fa8a3736.
Updated by G Kazaroth over 7 years ago
Thanks for the fix. I installed 4.1-2527~g3b2a616 and it does work switching to higher priority channels when I am not watching live TV. My setup is 2 tuners IPTV on a single network (very simple). The issue is if I am watching the tv channel that is recording, after the recording completes for a few minutes I stop playing live TV, the recording stream keeps going. It is no longer in the up and coming recording tab and has moved to the finished recording, but the stream is still going and the file being recorded is still growing. The stream is the matroska stream, which is the DVR recording vs the htsp stream going to the live TV on kodi. To stop the recording, I had to restart the tvheadend service.
Updated by G Kazaroth over 7 years ago
Additional notes: tvheadend use to record two shows on the same channel at the same time using a single stream. As an example TVshow 1 ends at 8:02pm on channel 5 while TVshow 2 begins at 7:58pm on channel 5. So from 7:58 to 8:02, both files are recording the same channel. It now has the second TVshow in idle and the log states:
subscription: 0006: No input source available for subscription
during that time.
Updated by G Kazaroth over 7 years ago
sorry, I am new to tvheadend dev/test and do not know how to get that exact version. Above, Mark Clarkstone said to go to https://doozer.io/mpmc/tvheadend/builds
But that version is not listed. I would be glad to test it, if I could get the amd64 deb file for it.
Updated by Mark Clarkstone over 7 years ago
George Kazaroth wrote:
sorry, I am new to tvheadend dev/test and do not know how to get that exact version. Above, Mark Clarkstone said to go to https://doozer.io/mpmc/tvheadend/builds
But that version is not listed. I would be glad to test it, if I could get the amd64 deb file for it.
You're looking to install 2530 if using the builds from my branch on doozer.io. The slight increase in version numbering is because the branch has commits related doozer/packaging that the main tvheadend repo doesn't (see the PR).
Updated by G Kazaroth over 7 years ago
Thanks for the info. I installed 4.1-2530~gd57fb7e from Doozer. Setup test as before. The critical problem with the channel not stopping when the timer finishes has been fixed. The secondary issue with the overlap recording between the previous recording and the next recording on the same channel is still present. I setup to record two shows on the same channel. One ending at 6:31am and one starting at 6:29am. Between 6:29am and 6:31am, the first recording continued, the second recording was added to subscriptions in the idle state. The log stated:
subscription: 0009: No input source available for subscription
At 6:31am, the first recording completed and the second recording started.
Updated by Phil Uithoven over 7 years ago
Jaroslav Kysela wrote:
Applied in changeset commit:tvheadend|06a07bb5eb9cdd28fc93b431dda7de98fa8a3736.
Just retried my last scenario with 4.1-2526~g5cbaac172, and tuners filled as expected. Lower-priority Kodi stream was untouched until all 6 tuners were in use, and I started a 6th recording. It then got bumped as it should have.
Things seem to be working correctly now. I'm happy to be able to call this one fixed from what I can see. Thanks!