Tvheadend

The last time it happened was 5 minutes after midnight. I don't se anything in the Cabernet log at that time:

2021-11-23 23:33:07,162-DEBUG:epg USTVGO:default Processing Program 2030281611 from USTVGO
2021-11-23 23:33:07,226-INFO:plugin_epg USTVGO:default EPG update completed
2021-11-24 01:49:28,326-DEBUG:schedule Running job Job(interval=200, unit=minutes, do=exec_trigger, args=({'uuid': '196C18A7-3DB8-11EC-BF37-D05F64D13909', 'area': 'EPG', 'title': 'Refresh PlutoTV EPG', 'timetype': 'interval', 'timelimit': -1, 'timeofday': None, 'dayofweek': None, 'interval': 200, 'randdur': 80, 'taskid': '196A9F60-3DB8-11EC-9176-D05F64D13909', 'namespace': 'PlutoTV', 'instance': None, 'funccall': 'refresh_epg', 'lastran': datetime.datetime(2021, 11, 21, 13, 17, 39, 684984), 'duration': 11, 'priority': 10, 'threadtype': 'thread', 'active': 0, 'description': 'Pulls channel program data from PlutoTV'},), kwargs={})
2021-11-24 01:49:28,338-NOTICE:scheduler Running threaded task EPG:Refresh PlutoTV EPG
2021-11-24 01:49:41,579-INFO:epg Missing PlutoTV genre translation for: Dance
2021-11-24 01:49:44,396-DEBUG:epg Refreshed EPG data for PlutoTV:default day 2021-11-24
2021-11-24 01:49:44,583-INFO:plugin_epg PlutoTV:default EPG update completed
2021-11-24 02:36:31,370-DEBUG:schedule Running job Job(interval=160, unit=minutes, do=exec_trigger, args=({'uuid': '6310EB33-4954-11EC-9690-D05F64D13909', 'area': 'EPG', 'title': 'Refresh USTVGO EPG', 'timetype': 'interval', 'timelimit': -1, 'timeofday': None, 'dayofweek': None, 'interval': 160, 'randdur': 40, 'taskid': '630EBFF7-4954-11EC-929E-D05F64D13909', 'namespace': 'USTVGO', 'instance': None, 'funccall': 'refresh_epg', 'lastran': datetime.datetime(2021, 11, 21, 15, 48, 17, 843749), 'duration': 219, 'priority': 10, 'threadtype': 'thread', 'active': 0, 'description': 'Pulls channel program data from USTVGO'},), kwargs={})
2021-11-24 02:36:31,376-NOTICE:scheduler Running threaded task EPG:Refresh USTVGO EPG
2021-11-24 02:36:36,306-DEBUG:epg USTVGO: Adding minimal EPG data for channel 9200017734

I rebooted my Tvheadend box this morning so it is working now.

Edit:

I did find this warning but it wasn't around the time when the grabber runs:

2021-11-23 23:30:32,124-DEBUG:epg USTVGO:default Processing Program 6032412973 from USTVGO
2021-11-23 23:30:42,831-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/6031461719/web
2021-11-23 23:30:54,339-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/6031461719/web
2021-11-23 23:30:55,341-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping PluginEPG.get_uri_data() 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/6031461719/web
2021-11-23 23:31:05,911-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/2030120261/web
2021-11-23 23:31:17,256-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/2030120261/web
2021-11-23 23:31:18,258-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping PluginEPG.get_uri_data() 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/2030120261/web
2021-11-23 23:31:28,702-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/6031014239/web
2021-11-23 23:31:41,106-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/6031014239/web
2021-11-23 23:31:42,108-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping PluginEPG.get_uri_data() 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/6031014239/web
2021-11-23 23:31:53,140-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/2030120261/web
2021-11-23 23:32:05,531-INFO:decorators Socket Timeout Error in function PluginEPG.get_uri_data(), retrying 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/2030120261/web
2021-11-23 23:32:06,532-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping PluginEPG.get_uri_data() 5251 The read operation timed out https://cmg-prod.apigee.net/v1/xapi/tvschedules/tvguide/programdetails/2030120261/web
2021-11-23 23:32:07,462-DEBUG:epg USTVGO:default Processing Program 6003260677 from USTVGO

Those should be considered normal issues. I was concerned about the Connection Refused which could be caused by exceptions, but since you don't have any, there is no issue.

Hi there,
I use the tool to bring PlutoTV into TVH.
Is there a way to get the MUXes / services without a channel number like "Avatar" instead of "115 Avatar"?
In TVH / CAB i used default settings.

yes, I can add a setting in the client to not include the channel number. Will make an update in a short...

Hey, thank you very much, I have your GIT Master at work anyway.

Please think of XMLTV that the number will also be dropped there if the option is set.

Have you seen the following bug, is probably the cause of the dropouts when switching to advertising, etc.
Unfortunately I can't build my own ffmepg with this patch.

https://trac.ffmpeg.org/ticket/5419

btw do you know why some services don't get real names but only Service01?

Updated 0.9.6.5 with patch. Flag is in settings > client > epg. Will update the xmltv.xml and channels.m3u to not include channel number with name. Let me know if you find any issues.

Service01 is normally from ffmpeg. This could be from the provider or from Cabernet. Sometimes the ATSC stream from the provider does not include the basic packets causing this to occur. If you have "Enable PTS/DTS Resync" enabled, then it should update the stream to include the basic packets and Cabernet will update the stream with the channel name based on having Clients > Stream > ATSC SDT Update enabled (which adds the channel name to the service name)

As for the EXT-X-DISCONTINUITY tag. Providers abuse the tag, so using it probably does not make sense. We use an algorithm to determine what to do. For example, one provider resends the exact same commercial/stream over and over. Around each repeat is a EXT-X-DISCONTINUITY tag. Normal processing would think this commercial as a dupe, but Cabernet knows better.

I'm currently testing it, but it looks very good

• EPG works correct
• MUX Names correct
• Service names have still the number | but after mapping service to channel the number is gone ... strange
• ATSC SDT is enabled, still much Service01, but will look again after a while

Another really cool feature would be "start your own command after epg scheduled task" in my case "systemctl start get_cabernetepg.service"

It would also be nice to influence the channel numbers in Cabernet they are in my case between 10, 15, 30 ... 2000.
I have configured numbers from 200, shouldn't it be 200,201,202 ...

I attempted the upgrade but it came back and still says I am on Version: 0.9.6.4 and that 0.9.6.5 is available, turns out Github is returning 500 error pages this afternoon.

Looks like they are aware of the problem: https://twitter.com/githubstatus

Hi

Been using the PlutoTV option in Cabernet for a while now and it is working well. Just wondering if it is possible to update the URL Filter to include “Sponsors” which is a screen that appears during ad breaks on the PlutoTV uk version?

Thanks.

The URL Filter is a setting that users can change/add. The plutotv option is an example for users to see and use/adjust as they need. If you look at the log at the INFO level, it should tell you the difference in the URLs that will allow you to add filters.

G Kazaroth wrote:

Those should be considered normal issues. I was concerned about the Connection Refused which could be caused by exceptions, but since you don't have any, there is no issue.

I'm still having the connection refused problem.

I posted this problem on the Tvheadend forum and was told to “Allow host to “without credentials” user “*” with same password.” However, I have no clue where to do that nor why the credentials expire after a couple of days.

Could the problem have something to do with something it doesn't like in the extra argument "-u http://192.168.1.83:6077/xmltv.xml" for the tv_grab_url grabber?

Based on the service log, it looks like the Cabernet service is shutting down at some point from something outside of Cabernet. Again, if you are not seeing any exceptions in the log, then the issue is with your system. It looks like your tv_grab_url executable is running just fine. The log indicates that both the tuner and web server interfaces are terminated at the same time. This is extremely unlikely from inside Cabernet, and I would look externally for a solution.

G Kazaroth wrote:

Based on the service log, it looks like the Cabernet service is shutting down at some point from something outside of Cabernet. Again, if you are not seeing any exceptions in the log, then the issue is with your system. It looks like your tv_grab_url executable is running just fine. The log indicates that both the tuner and web server interfaces are terminated at the same time. This is extremely unlikely from inside Cabernet, and I would look externally for a solution.

I understand. I'm just trying to figure out whether I should be focusing on the Tvheadend server, CoreELEC, my box, or router. My router seems unlikely since Tvheadend is on the box that is refusing the connection and zap2epg gets connected.

G Kazaroth wrote:

Based on the service log, it looks like the Cabernet service is shutting down at some point from something outside of Cabernet. Again, if you are not seeing any exceptions in the log, then the issue is with your system. It looks like your tv_grab_url executable is running just fine. The log indicates that both the tuner and web server interfaces are terminated at the same time. This is extremely unlikely from inside Cabernet, and I would look externally for a solution.

I'm starting to think this is a problem involving Cabernet. I was going to try to restart Cabernet after the problem occurred again and see if it cleared the problem. The last successful tv_grab_url run was this morning at 8:05 am. I got the connection refused message in the Tvheadend log at 4:05 pm today. I tried to log into Cabernet's web page after that but I get a connection refused message in my browser. I can log into Tvheadend on the same device, and I can browse files, etc. on the device. And Tvheadend is still running fine. There is a lot of activity going on in the Cabernet logs - almost like it is in a loop.

I've attached the last two Cabernet logs to cover the 8:05 am and 4:05 pm timeframes.

Those logs had exceptions in them. You indicated that no exceptions were found. Since there are exceptions, I will take a look and see what is the cause.

G Kazaroth wrote:

Those logs had exceptions in them. You indicated that no exceptions were found. Since there are exceptions, I will take a look and see what is the cause.

Thanks. I guess didn't understand what was meant by exceptions. I just looked for errors around the time I was having problems.

For some reason I am having a problem with the EPG not sending program data for any of the m3u channels. If I disable the EPG for everything but one m3u provider and then look at the xmltv.xml file, I see all the channels listed between channel tags, but there are no programme tags at all in the file. But, if I enable PlutoTV then I do get program listings for that. I have checked that the XMLTV File URL's are correct, and I figure Cabernet must be getting them or else it would not be able to get the channels to put between the channel tags.

I know at one time it used to bring in program data for those m3u instances and I am a little confused as to why it's not working now.

EDIT: Figured it out. For some reason the scheduler had lost its EPG timers for the m3u instances (but not for PlutoTV) and it hadn't updated the guide data in several days (probably not since the last reboot).

I guess I need to add a conditional popup on deleting schedule tasks. Most likely was caused by hitting the trash instead of the play button.

To Sterling:
An exception is not an error. An error is an issue that has been processed, appropriately. An exception can be an issue that is either processed or not processed. Exceptions are found in the log by searching for the word exception. In your log, you have processed and unprocessed exceptions. None are considered errors.

As an example, you could have a FileNotFoundException, but it may be expected at times, so you log it as INFO. Once it is logged as INFO, it is a processed exception. Unprocessed (or unexpected) exceptions happen all the time. This is caused by unknown operator conditions that the author was not expecting. So, when an unprocessed exception occurs, authors are very interested and normally will patch the code to handle the condition.

As I mentioned, I will be looking into the unprocessed exception that was found. As for the connection refused, I did find some interesting info. It looks like you have the network refresh set to 60 minutes (called Re-fetch period). What you are saying is you believe the channel listing is changing every hour to have different channels. In reality, if channels change weekly, I would be surprised. Although this setting probably does not cause the connection refused, it is probably something you should set to a MUCH higher value. 1440 min is what I use. You are basically banging on all channels every hour looking for new channels (that rarely change). Run a search on "web_handler" in the log to see what is being requested, if you want. The xmltv.xml is successfully being executed twice each time. (TVH sends 2 requests as expected) The xmltv.xml requests took about 60 seconds according to the log to complete. Check the bottom of the tv_grab_url file and see if the -T option is greater that 60 seconds, but I believe it should be.

G Kazaroth wrote:

To Sterling:
An exception is not an error. An error is an issue that has been processed, appropriately. An exception can be an issue that is either processed or not processed. Exceptions are found in the log by searching for the word exception. In your log, you have processed and unprocessed exceptions. None are considered errors.

As an example, you could have a FileNotFoundException, but it may be expected at times, so you log it as INFO. Once it is logged as INFO, it is a processed exception. Unprocessed (or unexpected) exceptions happen all the time. This is caused by unknown operator conditions that the author was not expecting. So, when an unprocessed exception occurs, authors are very interested and normally will patch the code to handle the condition.

As I mentioned, I will be looking into the unprocessed exception that was found. As for the connection refused, I did find some interesting info. It looks like you have the network refresh set to 60 minutes (called Re-fetch period). What you are saying is you believe the channel listing is changing every hour to have different channels. In reality, if channels change weekly, I would be surprised. Although this setting probably does not cause the connection refused, it is probably something you should set to a MUCH higher value. 1440 min is what I use. You are basically banging on all channels every hour looking for new channels (that rarely change). Run a search on "web_handler" in the log to see what is being requested, if you want. The xmltv.xml is successfully being executed twice each time. (TVH sends 2 requests as expected) The xmltv.xml requests took about 60 seconds according to the log to complete. Check the bottom of the tv_grab_url file and see if the -T option is greater that 60 seconds, but I believe it should be.

OK, I understand what you mean by "exception" now.

I changed Re-fetch to 1440 (I had left it at the default) for Pluto TV and USTVGO.

The -T option appears to be set to 60 seconds:
wget -q -T 60 -O - ${url}

Should I increase it? I've noticed that the grab can take longer than 60 seconds but it still completes.

Thanks

Cabernet does chunk the xmltv.xml output in instance/day chunks, so it should send something to the client fairly often, but if it somehow hangs due to I/O buffering or whatever, wget will terminate the connection after 60 seconds of dead space. Per the man page on wget:

"if, at any point in the download, no data is received for more than the specified number of seconds, reading fails and the download is restarted."

You can make a decision on what timeout you want to use. It was just another possible reason you could be having issues.

Got a question for you. Up until recently, if you went to https://i.mjh.nz/PBS/ and used the all.m3u8 file as a source in VLC, it would load up a whole list of dozens of PBS stations from across the country, and you could play any of them. And of course you could select any of the streams from that m3u8 file that you might have had an interest in, and add it to your own custom list of m3u streams for use with Cabernet.

But sometime recently things changed, for one thing the list of stations shrank considerably, but even the ones that remain mostly are unplayable. The only exception I could find that would still play was the one from Nebraska Public Media, and who knows how long that one will continue working.

However there is a new m3u8 link on that page at https://i.mjh.nz/PBS/kodi.m3u8 and at first I could not understand what it was doing (it doesn't work in VLC except for that same Nebraska feed) so I went to the Kodi forum and found this thread: https://forum.kodi.tv/showthread.php?tid=357548 - If you look at message #4 in that thread (https://forum.kodi.tv/showthread.php?tid=357548&pid=3071833#pid3071833) it says this:

But yes, there were changes at PBS - 95% of the streams are now using Widevine DRM.
Add-on 0.1.5 added support for these.

https://i.mjh.nz/PBS/all.m3u8 now only has the non DRM streams

There is a new https://i.mjh.nz/PBS/kodi.m3u8 which includes the DRM streams.
But will only work on players that support Widevine and the #KODIPROP tags.

So using that addon you could watch any of those streams live in Kodi, but not using Tvheadend.

What I am curious about is if you understand what he's doing in his Kodi addon to support these streams, and if there would be any easy way to add that same ability to Cabernet. Personally I am not really interested in having every one of those PBS stations appear in the Tvheadend guide, but there were a few select ones that I had some interest in.

There are 3 types of streams: Open, Encrypted and DRM. Open is by definition a clear stream that needs no additional processing. Encrypted is one where the provider gives the http URL to the decryption key in the m3u8 file so we can decrypt it. DRM provides a unique url string that informs DRM knowledgeable software how to obtain the key needed to decrypt the stream. The ones I looked into required a login to obtain the key, so Cabernet will log that it cannot DRM the stream if it finds one.

Since Kodi is providing one type of DRM, I will take a look. My understanding is Kodi uses OS DRM libraries to obtain the key.

Thanks. I had a feeling that if Kodi could do it, Cabernet just might be able to do it as well.

All of a sudden I was getting no guide data. The http://localhost:6077/xmltv.xml file only contained:

HTTP/1.0 501 Not Implemented
Server: BaseHTTP/0.6 Python/3.7.3
Date: Thu, 30 Dec 2021 10:33:48 GMT
Content-type: text/html

<html>
        <head></head>
        <body>
            <h2>501 - Server Error</h2>
        </body>
    </html>

If I tried to get an individual instance, for example http://localhost:6077/plutotv/xmltv.xml that worked. So I unchecked "XMLTV Enabled" and saved the change for ALL the providers except PlutoTV, same problem - http://localhost:6077/xmltv.xml returned the error, while http://localhost:6077/plutotv/xmltv.xml worked fine.

Turned out I had deleted a m3u instance from the config file but forgot to go into Data Mgmt and clear the data for that instance there. So, when Cabernet was trying to collect EPG data, it was throwing errors like this:

Dec 30 05:59:34 systemname locast[2935]: 2021-12-30 05:59:34,487-ERROR:web_admin UNEXPECTED EXCEPTION on GET='m3u_custom2'
Dec 30 05:59:34 systemname locast[2935]: Traceback (most recent call last):
Dec 30 05:59:34 systemname locast[2935]:   File "/home/user/cabernet/lib/clients/web_admin.py", line 104, in do_GET
Dec 30 05:59:34 systemname locast[2935]:     elif getrequest.call_url(self, self.content_path):
Dec 30 05:59:34 systemname locast[2935]:   File "/home/user/cabernet/lib/common/decorators.py", line 228, in call_url
Dec 30 05:59:34 systemname locast[2935]:     self.url2func[_name](_webserver, *args, **kwargs)
Dec 30 05:59:34 systemname locast[2935]:   File "/home/user/cabernet/lib/clients/epg2xml.py", line 40, in xmltv_xml
Dec 30 05:59:34 systemname locast[2935]:     epg.get_epg_xml(_webserver)
Dec 30 05:59:34 systemname locast[2935]:   File "/home/user/cabernet/lib/clients/epg2xml.py", line 87, in get_epg_xml
Dec 30 05:59:34 systemname locast[2935]:     self.gen_channel_xml(xml_out, channel_list)
Dec 30 05:59:34 systemname locast[2935]:   File "/home/user/cabernet/lib/clients/epg2xml.py", line 172, in gen_channel_xml
Dec 30 05:59:34 systemname locast[2935]:     if not self.config[config_section]['enabled']:
Dec 30 05:59:34 systemname locast[2935]: KeyError: 'm3u_custom2'

I note that the program is still identifying as locast in the log so you might want to think about changing that to cabernet.

On another note, I appreciate that you can disable "Channel # in Name" but kind of wish that option were available on a per-instance basis. The reason is I'd like to turn it off, but don't want to have to deal with having several hundred channels all getting renamed at once. I'd prefer to deal with one instance at a time. Also, I wish there was a way on a per-instance basis to append a string to each channel NAME (not the number), for example to append " (PlutoTV)" after all the PlutoTV channels. I realize that can be done manually in the channel editor, but that's kind of a pain if you are doing several hundred channels, and it's a greater pain if you ever decide you want to change that suffix for some reason. The point is that I'd like to get rid of channel numbers as part of the channel name entirely (but preferably not all at once), but still have a way to know which streaming service any given channel is coming from.

And speaking of the channel editor, one thing that would be useful is a button that downloads the icons of all currently displayed and enabled channels in the largest available size, giving them the name of the channel (but keeping the original extension such as .png). These files could then be loaded into a program such as GIMP for additional processing. For example, one thing I hate is when they leave a big transparent border around an icon, which has the effect of making it look smaller than all the other icons in Kodi. In such a case I can bring it into GIMP and do a "Zealous Crop" to remove the border but it is a pain to have to download each such icon manually, which is why it would be nice to be able to download all of them en masse.

None of this is really urgent, it's just things that I have been thinking for a while that it would sure be easier if Cabernet could do this or that. So those are just suggestions in case you feel like tweaking Cabernet a bit more.