Tvheadend

NOTE: While working with the new XUMO, I found that python's multiprocessing Process is NOT thread safe. I could not believe it, so I ran a number of tests and it cannot launch multiple process at the same time. Once launched, everything is OK. This happens when I do a Force Scan with tuners set to 4 or 5 in TVH. The result is a failed channel and hanging processes. I have fixed this is the new version, but be aware of launching multiple channels at the exact same time.

This reminds me of a time when our company's server was crashing every 3-6 hours with a Java dump. Found out the Oracle classes12.zip file for connecting to their database was not thread safe. At the time, I thought it was crazy for a company like Oracle who is big on multi-threaded process to develop software that was not thread safe.

Next significant upgrade is available. 0.9.3.2
Refer to the release info for changes
https://github.com/cabernetwork/cabernet/releases/tag/0.9.3.2

Major updates include¶

• XUMO plugin now available: Note, XUMO instance default is "disabled" in the config so you can setup the settings before requesting Channels and EPG data.
• Be aware that the first time the XUMO EPG runs, it will take a while to pull down all the program data and store it in the database
• Removed Locast Plugin and moved it to its own repo
• Fixed all the Python limitations with multiprocessing and HTTPS, so PlutoTV is much more stable
  There is a small patch to the file log handler settings that are applied when you upgrade automatically.
  I recommend deleting the plugins/provider_video_locast folder since it should be empty anyways. I need to update the patcher to remove the folder; just have not done that yet.

XUMO is being goofy. It seems to reset every so often and start from the beginning. Working on testing it out.

I have a patch in place that seems to be working. 0.9.3.3
To let you know what I am facing... XUMO has more connection issues than PlutoTV. It will hang on a URL request, for video, forever and not respond. The answer is to use a HTTP timeout; however, Python's great implementation works only on HTTP and not HTTPS. In today's world, much of the URLs are HTTPS, so the requests that stick, stick forever. Python implemented a secondary level timeout at the socket layer, but is a global and not associated with each request. Using globals with multi-threaded processes is normally not a good idea, but that is what we have.

So what happened. I implemented a socket timeout to handle the video sections requests from XUMO that get stuck by setting the timeout to 1 second. This worked to clear the timeouts that stuck from XUMO. BUT... This timeout was also applied (do to it being a global) to the video stream to the client (or TVH). That is why the video stream keeps resetting!

What is the solution? The socket timeout is scoped to a process and fortunately, we fork a new process for each tuner. This allows us to set the timeouts for each tuner separately from the timeouts to the clients. Not sure if it works totally, but initial tests are working. The big issue is when running multiple tuners, XUMO get hung on HTTPS requests. This shows up as a ffmpeg process that does not terminate when the tuner shuts off. My test that seems to reproduce the stuck condition is to run TVH Rescan on the Networks.

Oh, one more thing. The XUMO EPG update reminds me of zap2it perl script. If gets the 10000-20000 program datasets from the website and stores them into a cache folder. We do the same, but use a database. Currently, I have the aging set to 30 days, so the database will trim old programs off after a while.

I have reworked all the socket timeout, process and thread issues; moving the URL requests with the same socket timeouts together. It looks like that worked and I am no longer getting any timeouts or hanging url calls. Current version is 0.9.3.6.

Hi

Thanks for all your work on this app! Have made some good recordings of movies from PlutoTV.

I was wondering how easy it would be to add another IPTV provider? I am interested in SamsungTVPlus. There is some overlap with PlutoTV but it has lots of other channels.

However it seems to differ between regions. A list of streams per region can be found here, http://www.apsattv.com/streams.html. This site seems to provide an epg in xmltv format which is updated daily, https://i.mjh.nz/SamsungTVPlus/.

Thanks.

I would also like to say thanks for your work on this. I just installed the latest upgrade. I hope it will fix a problem I have been having, where the web interface becomes inaccessible and the only way to get it to come up again is to do a "sudo systemctl restart cabernet.service" (you may recall I mentioned I had renamed it from locast to cabernet). I did record a movie last night off of Xumo in the middle of the night but haven't had a chance to watch it yet.

Those m3u8 lists on the page that Robert Coupee linked to are interesting but the granddaddy of them all appears to be that one at https://iptv-org.github.io/iptv/index.m3u - try loading that into VLC and there are just tons of channels in there. But here is the thing I wish you could explain to me, if it doesn't mystify you as much as it does me. I'm going to be a little bit circumspect about how I say this because I don't want things to go away, if you catch my drift. But anyway, load that link into VLC (under "Open Network"), and just to pull out one at more or less random, go to the one that begins with the letters "KD" and ends with "[Not 24/7]". Try to play it and you will see that it takes several seconds to start playing (depending on the time of day) but if it is up, it appears to be playing in real time. Once it is playing, in VLC bring up the Media Information window and note the Location (the URL of the stream). Now take that and in Tvheadend create an IPTV Network (not an Automatic IPTV network, just a normal one) and under that network, a mux, using that link as the URL Go through the steps to create a channel, and then go and view the channel. What you will find, for no explainable reason, is that you have gone back in time and are viewing something that was aired much earlier. Play the stream through Tvheadend simultaneously with the same stream in VLC, and you will note that they are not playing the same thing! I don't get that at all.

And to further confuse the issue, change the link in the Tvheadend mux to use an ffmpeg pipe, for example

pipe:///usr/local/bin/ffmpeg -loglevel fatal -i link.m3u8 -c copy -metadata service_provider=name_of_stream -metadata service_name=name_of_stream -f mpegts pipe:1

Now suddenly the stream in Tvheadend is current and pretty much mirrors the stream in VLC, with one exception - the video and the audio are out of sync in Tvheadend. If you are really lucky that may not happen at the time you look, but try again a few times over a day or two and you'll see it. But in VLC, the audio and video are always in perfect sync.

This behavior has me totally baffled and I really wish I had an explanation, but I don't. But anyway, what I was going to suggest is that you could have a plugin that would allow entering links to one or more .m3u/.m3u8 lists and select channels from those. Or if not that, individual stream links, that would then become part of an automatic network in Tvheadend. Of course you would not have guide data in most cases, unless you could obtain it from another source. But it would allow you to pluck individual streams from lists such as these, that might be pertinent to your area (for example a stream of local newscasts), and create an customized automatic network of such streams, if that makes any sense. And I would sort of hope that by funneling the stream through Cabernet it just might solve the "watching the past" and/or audio/video sync issues.

Also if you are looking for another streaming service to offer a plugin for, I'll again mention that Sling TV has a bunch of free streaming channels, go to https://www.sling.com/deals/sling-free and click the "Watch Free Now" button and then at the top click "Guide" and you'll see that for several channels they have a full week of guide data. The downside is there is another group of channels for which there is no guide data at all, those appear to actually be links to "on demand" videos but for some reason they are listed as channels in the guide.

Anyway, thank you once again for all the work you have done on this!

Thanks for the inputs... The new version seems to be stable, so far.
As for the timing issue from Sean, It is associated with that list of 10,000 channels. I played the channel in VLC and it looked fine. Did not try it in TVH. Some interesting channels, but not really what they say they are.
As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

Nice links from both; many have requests like these. Since we are still early in the plugins, I would like not to have to spend too much time trying to get the basics. So, to have one picked, you must show a python solution is available. This can come from GitHub or Kodi as a general standard. (yes, I am aware of the GitHub streamlink site and no I'm not going to implement all of them.) Then, there are the defaults, like import a m3u file or import a URL (like Sean provided) with over 10,000 channels (I saw this one on the TVH forum with someone trying to import a filtered list of it.) I was curious how much it would blow up in Cabernet trying to put that into a channel editor. Cabernet would probably have to limit the number of channels listed per page.

I also found some links by a Kodi developer. This one contains almost 100 video cameras from around the world showing nature (like one in alaska)
https://omega.explore.org/api/get_cam_group_info.json?id=79
which references
https://www.youtube.com/embed/VhFpQsemOx0?rel=0&showinfo=0&autoplay=1&playsinline=1

I suspect adding a general m3u plugin should be the next thing. Point to a file or URL and it will load/reload it based on the scheduler. Default EPG will generate hourly programs for each channel, so it will appear correctly. Some m3u providers also provide guide data. Not sure what format those are in, but we could use a default xmltv.xml format to start. Probably need to also add the limited channels per page for the channel editor since these m3u files can get harry. Also note that we currently only support unauthenticated m3u8 protocols. In the future, this could be expanded.

G Kazaroth wrote:

As for the timing issue from Sean, It is associated with that list of 10,000 channels. I played the channel in VLC and it looked fine. Did not try it in TVH. Some interesting channels, but not really what they say they are.

That's part of what is strange. It always play perfectly fine in VLC, no looking back in time and no audio/video sync issues. It's only when you try it in Tvheadend that one of those issues pops up.

As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

Well what was odd about it was that two days in a row that happened. I was in the web interface one day and it was fine, came back the next day and it was not responsive. No idea what happened in the meantime.

So, to have one picked, you must show a python solution is available. This can come from GitHub or Kodi as a general standard.

I'm not a programmer, unfortunately, so would not know a Python solution if I were staring right at one.

(yes, I am aware of the GitHub streamlink site and no I'm not going to implement all of them.)

I would hope not, that would be way too many to deal with. That's why I was thinking there could be a way to select the ones you want.

Then, there are the defaults, like import a m3u file or import a URL (like Sean provided) with over 10,000 channels (I saw this one on the TVH forum with someone trying to import a filtered list of it.) I was curious how much it would blow up in Cabernet trying to put that into a channel editor. Cabernet would probably have to limit the number of channels listed per page.

I definitely agree with that. A browser might well run out of memory if all those were on the same page. In fact when I loaded that list into VLC I was holding my breath waiting for it to crash, but it didn't.

I suspect adding a general m3u plugin should be the next thing. Point to a file or URL and it will load/reload it based on the scheduler. Default EPG will generate hourly programs for each channel, so it will appear correctly. Some m3u providers also provide guide data. Not sure what format those are in, but we could use a default xmltv.xml format to start. Probably need to also add the limited channels per page for the channel editor since these m3u files can get harry. Also note that we currently only support unauthenticated m3u8 protocols. In the future, this could be expanded.

Well I will just say that this could get REALLY interesting if there were some way to log into a commercial streaming provider and then bring all their channels (at least those in a package you subscribe to) into Tvheadend, but of course the big problem with that is you'd have to have an account with any service you wanted to implement. The main reason for doing that would be so you could PVR shows and watch them at your convenience. And also I suspect they encrypt those things ridiculously.

But another option would be to be able to use a streaming provider login with various cable network sites, at least those that offer live streams of their networks (if any still do). Typically they ask you to select your provider from a list, and while most of those providers are cable or satellite services, some do also accept logins from streaming providers (for example, DirecTV has a streaming bundle that includes a number of cable channels). So let's say, just as an example, that you wanted to watch or PVR a specific cable channel, and that on that channel's site there is a live stream available but you have to log in with your DirecTV Stream (or other streaming service) login to do it. Note that you would not be getting the stream from DirecTV Stream (or whoever) in this case, but instead you'd be getting it from the cable network and just using your streaming service login to facilitate that. However I haven't checked in a long time to see if any of those cable networks still offer live streams.

I do realize that would probably be waaaaaaay more difficult than doing PlutoTV or XUMO, and I'm not even sure such a thing would be possible, but it would be REALLY interesting if it could be done. I am not holding my breath though!

Reviewed the samsung tv and it has m3u and xmltv.xml files, so this may be a great test when creating the m3u/xmltv plugin. There is also a json; not sure if it provides important additional metadata for each channel.

0.9.3.8
Found bug with XUMO when the provider terminates the program, cabernet tuner process will continue to run at 100% CPU (forever). Update fixes issue. It will not run at 100% CPU and will terminate the tuner process thread after 20 seconds of no data from provider.

On another note... It looks like the samsung tv xmltv file provides 8 hours of programming at a time.

Hello

I am running 0.9.3.9 and cabernet/tvheadend messed up multiple recordings today. When playing the program back later it plays and then periodically it jumps ahead (i.e. a chunk of the show is missing). Everything recorded ok yesterday with previous version. I am recording on channel '614 DABL' and the show is called 'Escape to the Country'. I record 3 episodes each day. The shows start back to back at 1pm, 2pm and 3pm pacific time. I have start time and end time adjustment set to 0. I have enabled URL Filtering and PTS/DTS Resync. I have 'Extra warming up time (seconds)' set to 30 in TVHeadend.

Thanks for your great work on this!

Thanks for the info. I have a number of DABL shows recording and will see if I can have it happen. DABL is a unique channel has its own quirks. About the only way to get a skip is to have an HTTP error of some kind. If you look at the log, you will see INFO and WARNING lines. By default, the system log should at least have the WARNINGs. Below are examples of what you might see.
2021-09-22 17:30:52,838-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:53,840-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:54,842-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:55,844-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 434226 <urlopen error [Errno xxx] xxxxx

If you find the WARNING saying a timeout has occurred, then you can make a change to possibly correct it. Cabernet currently retries 3 times to get a connection, before it fails and drops the attempt. What changed? Well, I found that requests from providers were hanging forever causing tuner process to also hang. the result was to put a timeout on the connection. The question is... What should the timeout be? By default, python sets it to forever. Cabernet has it set to 1.0 seconds. If the PlutoTV connection is consistently taking a while, that could cause the issue. You can make the change and see if that helps.

Go to ./lib/streams/m3u8_queue.py
There are 2 lines that cause the timeout (set to 1.0 seconds) in the file. Change the value to something higher like 5.0. The lines are

   with urllib.request.urlopen(req, timeout=1.0) as resp:
   socket.setdefaulttimeout(1.0)

hmmm. I looked in the logs. I am set to log INFO. Here are some of my log entries during the recording....

2021-09-22 15:22:25,363-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:25,521-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00001.ts from play queue 2914
2021-09-22 15:22:25,522-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00000.ts to play queue 2914
2021-09-22 15:22:25,654-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:25,840-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:25,862-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:25,980-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/1.ts from play queue 1937
2021-09-22 15:22:25,980-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/2.ts from play queue 1937
2021-09-22 15:22:25,981-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222215_5_00065.ts to play queue 1937
2021-09-22 15:22:29,037-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:29,226-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00002.ts from play queue 2914
2021-09-22 15:22:29,227-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00001.ts to play queue 2914
2021-09-22 15:22:29,446-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:30,197-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:30,664-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:32,742-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:34,517-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:34,645-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/3.ts from play queue 1937
2021-09-22 15:22:34,646-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222221_5_00066.ts to play queue 1937
2021-09-22 15:22:35,705-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:36,435-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:36,599-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00003.ts from play queue 2914
2021-09-22 15:22:36,600-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00002.ts to play queue 2914
2021-09-22 15:22:36,707-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:37,059-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:38,862-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:38,978-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/4.ts from play queue 1937
2021-09-22 15:22:38,979-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222227_5_00067.ts to play queue 1937
2021-09-22 15:22:40,116-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:40,448-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00004.ts from play queue 2914
2021-09-22 15:22:40,448-DEBUG:m3u8_queue Added https://siloh.pluto.tv/5b080d_25_ad/creative/611bf4edc453cce4242cb4de_ad/720p/20210817_174205/hls/hls_2400-00000.ts to play queue 2914
2021-09-22 15:22:40,664-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
/error

I will set the timeout to 5.0

Agree, you have the "Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 1937 <urlopen error timed out>" indicating either you have a issue with the internet at the time, or the timeout is set too low.

I just thought of something... The reason I had set the timeout to 1 second is because TVH does not use the stream "Data timeout" during the initial transmission. TVH has it hard-coded to terminate the connection, if no data is received in 10 seconds. After it receives data, then it will use the stream "Data timeout" (which I have set to 480 seconds). What I could try is to send one of those ATSC messages that contain basically no info and see if TVH will think the video has started. If so, then the timeout can be much bigger. I'll give it a try tomorrow, when I am more awake...

Hmmm... Don't know if this helps but, this tends to happen in the middle of a recording. In the case of my log I sent, the recording started at 3pm (15:00) and the log shows the issue at 15:22

It's random, so it can happen anywhere. It is like having a random number from 1-300 with the number 1 being a really bad number. The problem is it can happen at the beginning. When I ran the rescan on the plutotv and xumo (over 600), it would hang forever on at least one of the 600. It could be caused by having 4 channels run at the same time, but it does occur.

I have a Pull Request with all the changes for the timeout. Doing some final checks and it will be ready in a short.

With timeout of 5.0, I ran a test with the initial ATSC msg sent, then it had a delay of 15 seconds between each section, so tons of delays. TVH did not terminate the connection as it would normally do and cabernet did not complain about any timeouts. Looks like it worked and TVH believed the stream was active.

BTW, All the DABL tests I ran had only 2 timeouts, but none had more than one hit, so did not get the warning or deleted sections. As I figure, we all have different internet connections and need the timeout to be a variable. That will be in the next major release with the M3UGeneric plugin.

I had an interesting test. Was watching DABL live on Kodi through TVH 4.3 and when the first ad stopped, it was fine with a still screen. The next time it stopped, Kodi went to a delay (round circle with a number in the middle). This time, the log stated that Kodi disconnected and resubscribed to the channel. When it did this, the m3u8 data backed up about 60 seconds and played the last 60 seconds again. It was like deja vue, but cabernet actually did was it was told and performed correctly. Not sure if Kodi has any settings to stop the termination and resubscription. According to the log, the last transmit was at 09:24:48 and kodi disconnected at 09:25:14, which is exactly 26 seconds. Now, just to let you know, cabernet does not just sit there during ads, it does send the ATSC blank message as a keep alive, but Kodi must think that is not enough. After the deja vue, it stopped again and again Kodi disconnected (this time after 24 seconds). It looks like it is in some kinda of loop, which would eventually end when the commercial phase was over.

Cabernet is setup to have config parameters at the plugin and at the instance level. I kinda guess at where to put parameters (and their view level) and it is not hard to move them. As part of the M3UGeneric plugin, I am moving most of the plugin level parameters to the instance level. It does mean you may have a little more management if you have multiple instances, but it does have benefits. You could have two instances of PlutoTV; one with ads removed (for records) and one with ads in place (for live). Then have 2 group tags, one for each instance. You could then use the tags to switch between live and recording lists in Kodi. The flexibility of Cabernet is its power.

I think I might know what happens to cause the deja vue. I noticed that whenever a stream starts the first m3u8 playlist contains quite a few items from the past. Thus if you try to start a stream at noon it will give you a playlist with about 10 or 20 items prior to noon and in effect you will actually start a minute or two prior to noon due to the first playlist giving you those items from the past. Thus in your interesting test when Kodi disconnects and reconnects a new stream will be opened and a minute or two of old data will be played.

About 2 weeks ago I created with a modification that marked these extra playlist items as played in an attempt to stop the stream from playing stuff from the past (this is when we were having problems with PTS). I abandoned the mod because you came up with PTS/DTS resync which worked well. However the mod may have some value to avoid the deja vue situation you encountered. The mod I made was was to the function called stream_direct in internal_proxy.py. To use it in your latest code I think it would need to go into m3u8_queue.py in the 'run' function of the M3U8Process thread.

This is what I put into your old version (in internal_proxy.py):

        # cookieisland
        new_stream = True
        # cookieisland

        while True:
            try:
                added = 0
                removed = 0
                i = 3
                while i > 0:
                    try:
                        playlist = m3u8.load(stream_uri)
                        break
                    except urllib.error.HTTPError as e:
                        self.logger.info('M3U8 Exception caught, retrying: {}'.format(e))
                        time.sleep(1.5)
                        i -= 1
                if i < 1:
                    break
                removed += self.remove_from_stream_queue(playlist, play_queue)
                added += self.add_to_stream_queue(playlist, play_queue)
                # cookieisland
                if new_stream == True:
                    '''
                    mark playlist as played except newest uri so that we do not play old content.
                    e.g. if stream was started at noon, content prior to noon will be skipped
                    '''
                    found_newest = False
                    for uri, data in reversed(play_queue.items()):
                        if not data["played"]:
                            if found_newest:
                                data['played'] = True
                                self.logger.debug(f"Cookieisland marked {uri} as played")
                            else:
                                found_newest = True
                    new_stream = False
                # cookieisland

P.S. I wonder if this is also what causes Sean's anomaly where some stream were playing past content. However he was getting content from an hour or more in the past. Maybe the first m3u8 playlist in his case had a HUGE amount of items from the past???

I'm not anywhere close to being a programmer but I thought I'd mention something that may be related to the Deja Vue. I use Lunatixz Pluto TV Kodi addon and IPTV Merge. The nice thing about this is I can select a show from the guide (even from several hours earlier), select the context menu, and I have the option to "Play programme". The show will start from the beginning and I can skip forward and back. This is really handy but doesn't work on all shows. I think this is different from the On Demand option (which I use most) since the not all shows in the guide are available on demand and visa versa.

I'm just wondering if somehow Pluto TV thinks you are skipping back in the show.

C Island wrote:

P.S. I wonder if this is also what causes Sean's anomaly where some stream were playing past content. However he was getting content from an hour or more in the past. Maybe the first m3u8 playlist in his case had a HUGE amount of items from the past???

That is an interesting thought. I downloaded that long .m3u file using wget and searched it to find the channel that I saw the problem on. It gave a m3u8 link for that channel, so I downloaded that using wget. It contains this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=2000,RESOLUTION=768x432
https://(redacted)/event/(redacted)/768x432.m3u8?start_time=2021-05-16T23%3A45%3A00%2B00%3A00&end_time=2023-05-17T05%3A00%3A00%2B00%3A00&env=prd&uid=(redacted)
#EXT-X-STREAM-INF:BANDWIDTH=4500,RESOLUTION=1280x720
https://(redacted)/event/(redacted)/1280x720.m3u8?start_time=2021-05-16T23%3A45%3A00%2B00%3A00&end_time=2023-05-17T05%3A00%3A00%2B00%3A00&env=prd&uid=(redacted)

Note I am redacting the actual links and other data that might identify the link because I don't want them to be searchable in this thread, since I don't know anything about their origin. But anyway, it appears there are streams in two different resolutions, but so far both VLC and Tvheadend (with or without the ffmpeg pipe) have used the larger one (I think VLC will sometimes play a couple seconds of the smaller one before switching to the larger). So again, I used wget on that link and got back a file that contains this:

(Lines 1-30):

#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:18.929+0000
#EXTINF:6.000,
1280x720/(redacted)0458.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:24.929+0000
#EXTINF:6.000,
1280x720/(redacted)0459.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:30.929+0000
#EXTINF:6.000,
1280x720/(redacted)0460.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:36.929+0000
#EXTINF:6.000,
1280x720/(redacted)0461.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:42.929+0000
#EXTINF:6.000,
1280x720/(redacted)0462.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:48.929+0000
#EXTINF:6.000,
1280x720/(redacted)0463.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:54.929+0000
#EXTINF:6.000,
1280x720/(redacted)0464.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:32:00.929+0000
#EXTINF:6.000,
1280x720/(redacted)0465.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:32:06.929+0000
#EXTINF:6.000,
1280x720/(redacted)0466.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:32:12.929+0000
#EXTINF:6.000,
1280x720/(redacted)0467.ts

(Lines 2032-2052):

#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:37:48.929+0000
#EXTINF:6.000,
1280x720/(redacted)1123.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:37:54.929+0000
#EXTINF:6.000,
1280x720/(redacted)1124.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:00.929+0000
#EXTINF:6.000,
1280x720/(redacted)1125.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:06.929+0000
#EXTINF:6.000,
1280x720/(redacted)1126.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:12.929+0000
#EXTINF:6.000,
1280x720/(redacted)1127.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:18.929+0000
#EXTINF:6.000,
1280x720/(redacted)1128.ts
#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:24.929+0000
#EXTINF:6.000,
1280x720/(redacted)1129.ts

So it looks like the first link is from 15:31:18.929 UTC while the last one was from 16:38:24.929 UTC, which would have been very close to the current time at the time I downloaded it. But apparently VLC is smart enough to skip to the end of the list and start playing there, and ffmpeg also goes to the end of the list but for some reason screws up the audio and video sync. But for whatever reason, if you put the original m3u8 link in a Tvheadend IPTV mux, it has no problem keeping the audio and video in sync but it's not smart enough to skip to the end of the list and start playing there. When I say "end of the list" I'm assuming that is the current time, but I may be wrong about that. Anyway, your theory appears to be correct! And I suspect that file can be MUCH longer because I have at times seen video from several hours in the past.

Code from C Island does not really work since each disconnect/reconnect from Kodi/TVH causes a separate tuner instance and the loss of what was played. I thought about using a service/mux method to provide a single tuner for all streams of the same channel might work, but that too would fail since it stops the tuner and restarts it on a different thread. One way to handle past playback would be to use the #EXT-X-PROGRAM-DATE-TIME: field (as shown by Sean) if present and if accurate, but that depends on the quality of data coming from the provider, which I am not fond of doing. Since we now handle the add/remove/played queue in a separate Process and not just a separate Thread, we could keep a short term list of all sections played on any tuner (how long is short term?). That could then be used to remove sections already played, even on new tuners. But, a possible solution.

Having some video buffer created on playback start is nice, but also playing near the current time would also be nice.

As for starting way back... Cabernet sends the data stream at a rate of 2-3 times faster than the play rate, if the provider makes the sections available. So, you could skip forward fairly quickly.

For the where to start playback, we have two choices. Either current or start at beginning. Currently, Cabernet just starts at beginning, always. There could be a setting to change that, but it would not be as nice as on Kodi. The URL from Cabernet could include an argument like
http://ipaddress:5004/PlutoTV/watch/xxxxx?start=current
http://ipaddress:5004/PlutoTV/watch/xxxxx?start=beginning
With a default value being set in the instance level of the plugin. I know that TVH would not understand using different M3U URLs, so it won't be much help with TVH.

Well all I will say about that is that I rarely watch anything live, and even if I do I usually start a recording and then watch the recording, so I can skip back or pause if I need to (only real exception is live news feeds, and you really wouldn't want those delayed). So personally I would much prefer that recordings work as expected. And that means that if I set a recording to start at (for example) 8 PM and end at 9 PM, I don't want to get a one hour recording of whatever aired from say 2:39 PM to 3:39 PM or some other random earlier times. I want what is actually airing from 8 PM to 9 PM.

It's not a problem if it goes back say 30 seconds in the stream because I can compensate for that in Tvheadend (in fact my default recording profile always records some extra padding at the end, just in case a program goes past its scheduled end a little) but if it goes back to the start of a very long list, that's absolutely useless, because I can't even predict how much to delay the start of the recording to compensate. Of course this is only a problem with streams like the one I was talking about where it can go way back in time. If we are only talking about less than a minute of previous stream available then that's really not a problem.

So, I'd vote for always starting at the current time by default; you could always have the option to start at the beginning but I know I'd personally never use it, at least not for any reason I can think of. And rather than a "beginning" option it would be a lot more useful to have options that go back exactly one, two, or three hours, but of course those would only work if that much data was available, and you could never guarantee that. It's the whole "jumping back to some random point in the past" that makes no sense at all to me. Possibly others will disagree, but if so I'd love to know why anyone would want to go back a random amount of time.

I am not sure what you mean when you say "each disconnect/reconnect from Kodi/TVH causes a separate tuner instance and the loss of what was played". That is the point of the code I wrote. When a new tuner starts I delete all old sections. I basically throw away everything in the first playlist pulled by a tuner by marking it as played. In doing so content from the past is not played. In the case of a disconnect and reconnect from Kodi/TVH the current tuner would stop, a new tuner is started and the old content/sections will be marked as played and then subsequent content would play on. This should eliminate your deja vue situation and also fix Sean's problem where he gets an hour or more of old content he doesn't want. If you want I can create a PR and you can test it out.....

If you do what you suggest when you said "we could keep a short term list of all sections played on any tuner" and remove sections already played even on new tuners I think you might run into a new problem. For instance when I record 3 back to back shows on Dabl and I record an extra 30 seconds at the beginning of each show. And Sean always records and extra minute or too at the end. In back to back recoding of shows you will have times when 2 tuners are running on the same stream. In that case you do not want to remove duplicate sections on either tuner because due to the overlapping recording times you want both tuners to play that content.

BTW: my recordings seem to be ok now and the DABL shows I record are no longer missing chunks of content.

G Kazaroth wrote:

As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

I am still seeing this issue of the web interface becoming inaccessible, and the bad part about that is apparently it can't respond to requests for guide data when it's down. The channels still play, so it's not that Cabernet is completely gone, just the web interface. What I have no idea how to do is tell if it is throwing exceptions, or if it is, why it's doing so. I only have the default logging enabled, and as far as I can tell it's not showing anything - cabernet.log is 0 bytes. Should I make any changes in the logging to help diagnose this issue?

OK, I reread the code and you are correct. On the first m3u8 list it throws away everything except the last entry. Based on some preliminary findings, it does look like the m3u8 files list based on having a buffer of around 2 minutes where the last one is around current time. This does seem to be useful and is an easy fix.

Alternatively, a slightly more complex solution would be to use the date_time field (which looks like it basically determines the same results), but would also handle the case where the entire movie is provided in the m3u8 file and Cabernet may need to pick how far down the list to start.

I would be interested in having Sean provide his insight on this as to whether always throwing away the first m3u8 set, or use the data_time stamps inside the file to determine where to start. I am good with either way.