Project

General

Profile

Forums » General »

Has tvhmeta ever worked when the tvh api wants authentication?

Added by James Bevan almost 2 years ago

I'm running tvheadend in a docker container, using the docker-compose template from https://github.com/linuxserver/docker-tvheadend. I'd been ignoring failing tvhmeta runs for a while & eventually decided to take a look.

What I found was that when a username and password are supplied, urlopen() can't grok the URL that tvhmeta constructs for the tvh api - eg http://user:pass@hostname:portnum/url, and instead treats 'user:pass@hostname' as the whole hostname. I'm certain this is the case as I saw exactly this kind of failing lookup requests in my DNS server logs. Here's what the error looks like in the tvheadend logs:

2023-01-06 15:14:17.989 spawn: Traceback (most recent call last):
2023-01-06 15:14:17.989 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open
2023-01-06 15:14:17.990 spawn:     h.request(req.get_method(), req.selector, req.data, headers,
2023-01-06 15:14:17.990 spawn:   File "/usr/lib/python3.9/http/client.py", line 1285, in request
2023-01-06 15:14:17.991 spawn:     self._send_request(method, url, body, headers, encode_chunked)
2023-01-06 15:14:17.991 spawn:   File "/usr/lib/python3.9/http/client.py", line 1331, in _send_request
2023-01-06 15:14:17.991 spawn:     self.endheaders(body, encode_chunked=encode_chunked)
2023-01-06 15:14:17.991 spawn:   File "/usr/lib/python3.9/http/client.py", line 1280, in endheaders
2023-01-06 15:14:17.992 spawn:     self._send_output(message_body, encode_chunked=encode_chunked)
2023-01-06 15:14:17.992 spawn:   File "/usr/lib/python3.9/http/client.py", line 1040, in _send_output
2023-01-06 15:14:17.992 spawn:     self.send(msg)
2023-01-06 15:14:17.992 spawn:   File "/usr/lib/python3.9/http/client.py", line 980, in send
2023-01-06 15:14:17.993 spawn:     self.connect()
2023-01-06 15:14:17.993 spawn:   File "/usr/lib/python3.9/http/client.py", line 946, in connect
2023-01-06 15:14:17.993 spawn:     self.sock = self._create_connection(
2023-01-06 15:14:17.993 spawn:   File "/usr/lib/python3.9/socket.py", line 823, in create_connection
2023-01-06 15:14:17.994 spawn:     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
2023-01-06 15:14:17.994 spawn:   File "/usr/lib/python3.9/socket.py", line 954, in getaddrinfo
2023-01-06 15:14:17.994 spawn:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2023-01-06 15:14:17.994 spawn: socket.gaierror: [Errno -3] Try again
2023-01-06 15:14:17.994 spawn: During handling of the above exception, another exception occurred:
2023-01-06 15:14:17.994 spawn: Traceback (most recent call last):
2023-01-06 15:14:17.994 spawn:   File "/usr/bin/tvhmeta", line 458, in 
2023-01-06 15:14:17.994 spawn:     process(sys.argv[1:])
2023-01-06 15:14:17.994 spawn:   File "/usr/bin/tvhmeta", line 453, in process
2023-01-06 15:14:17.994 spawn:     tvhmeta.fetch_and_persist_artwork(opts.uuid, opts.force_refresh, opts.modules_movie, opts.modules_tv)
2023-01-06 15:14:17.994 spawn:   File "/usr/bin/tvhmeta", line 225, in fetch_and_persist_artwork
2023-01-06 15:14:17.995 spawn:     recjson = self.request("api/idnode/load", data)
2023-01-06 15:14:17.995 spawn:   File "/usr/bin/tvhmeta", line 150, in request
2023-01-06 15:14:17.995 spawn:     req = urlopen(full_url, data=bytearray(data, 'utf-8'))
2023-01-06 15:14:17.995 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
2023-01-06 15:14:17.995 spawn:     return opener.open(url, data, timeout)
2023-01-06 15:14:17.995 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 517, in open
2023-01-06 15:14:17.995 spawn:     response = self._open(req, data)
2023-01-06 15:14:17.995 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 534, in _open
2023-01-06 15:14:17.996 spawn:     result = self._call_chain(self.handle_open, protocol, protocol +
2023-01-06 15:14:17.996 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
2023-01-06 15:14:17.996 spawn:     result = func(*args)
2023-01-06 15:14:17.996 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 1375, in http_open
2023-01-06 15:14:17.996 spawn:     return self.do_open(http.client.HTTPConnection, req)
2023-01-06 15:14:17.996 spawn:   File "/usr/lib/python3.9/urllib/request.py", line 1349, in do_open
2023-01-06 15:14:17.997 spawn:     raise URLError(err)
2023-01-06 15:14:17.997 spawn: urllib.error.URLError: 
2023-01-06 15:15:00.161 spawn: Done

For reference, this is one such tvhmeta invocation:

# tvhmeta --force-refresh --host localhost --user my-username --password my-password --tmdb-key my-tmdb-key --tvdb-key my-tvdb-key --uuid f5be8a96ea70376a0cf298c0ca873b21

I've written-up a replacement request() function along the lines of the urlopen documentation. I'm no regular python programmer but it appears to work, at least in the authenticated situation:

  def request(self, url, data):                                                                  
    """Send a request for data to Tvheadend."""                                                  
    full_url = "http://{}:{}/{}".format(self.host,self.port,url)                                 
    req = urllib.request.Request(full_url, method="POST", data=bytearray(data, 'utf-8'))         
    if self.user is not None and self.password is not None:                                      
        base64string = base64.b64encode(bytes('{}:{}'.format(self.user, self.password), 'ascii'))
        req.add_header('Authorization', 'Basic {}'.format(base64string.decode('utf-8')))

    logging.info("Sending %s to %s" % (data, full_url))                        

    ret = urllib.request.urlopen(req).read().decode('UTF-8', errors='replace');       
    logging.debug("Received: %s", ret)                                                 
    return ret

Note there's also an 

import base64
declaration among the other imports at the top of the tvhmeta script. CAVEAT, in the current docker image python is at version 3.9, so I don't know if the current function might work/have worked OK in earlier versions.

So back to the question in $TITLE, has this ever worked? Bonus question, if it indeed currently doesn't work, does the above look like a viable fix, or is there a simpler and more obvious fix?


Replies (17)

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

I am unable to get posters from TVDB. Are you able to get posters working?

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

Yes, at least I assume that's where the artwork is coming from. Here's what I have for "Additional command line options...":

--host localhost --user <local-fanart-account-id> --password <local-fanart-account-password> --tmdb-key <my-tmdb-key> --tvdb-key <my-v4-tmdb-key-enclosed-in-quotes> --uuid %U

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

What is local-fanart-account-id? How exactly did you get v4 tvdb working? When I try I get this

INFO:tv_meta_tvdb:Got exception: 'token'

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

I have user authentication enabled on my setup, so that's the username (and password) that I told tvhmeta to use to authenticate to tvheadend, to store any metadata it got back from tmdb/thetvdb. Most likely you don't need this if your tvheadend is unauthenticated.

I don't think I did anything extraordinary to get v4 tvdb going once I figured out I needed a V4 key, though I did have to enclose it in quotes on the commandline as my key does include a couple of 'special' characters. Other than that it has just worked OK.

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

Weird that it doesn't work for me. This is all I get. Are you sure v4 api is working? From what I read, you need a pin and I don't see a way to do this with tvheadend. The legacy api works but there are no posters.

tv_meta_tvdb.py --tvdb-key "6xxxx643-e0d7-4afd-b23e-xxxxxxxxxxxx" --title "Seinfield" :debug
tv_meta_tvdb:Got exception: 'token'

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

Well, one thing that jumps straight out for me is that you & I are invoking different scripts - I'm running tvhmeta, and you're running tv_meta_tvdb.py. I've not ever tried tv_meta_tvdb.py so I wouldn't know what to suggest. You get the same result when you run tvhmeta?

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

I think your artwork comes from tmdb and not tvdb. I enter the API key in the tvheadend artwork settings and removed tmdb. These are the results with the v4 API.I get the same token error. Check it out.


2023-04-05 20:21:22.610 subscription: 0087: "DVR: The Andy Griffith Show" subscribing on channel "MeTV NY", weight: 300, adapter: "IPTV #2", network: "IPTV", mux: "5Nxxxx3WGDh - MeTV NY", service: "MeTV NY", profile="pass", username="" 
2023-04-05 20:21:22.718 spawn: 2023-04-05 20:21:22,718:INFO:tvhmeta:455:Got movie modules: [tv_meta_tmdb] tv modules [tv_meta_tmdb,tv_meta_tvdb]
2023-04-05 20:21:22.718 spawn: 2023-04-05 20:21:22,718:INFO:tvhmeta:153:Sending uuid=e2d53031a491b00cb39a23ac539b982e&list=uuid%2Cimage%2Cfanart_image%2Ctitle%2Ccopyright_year%2Cepisode_disp%2Curi&grid=1 to http://localhost:9981/api/idnode/load
2023-04-05 20:21:22.720 spawn: 2023-04-05 20:21:22,719:INFO:tvhmeta:292:Trying title The Andy Griffith Show year None uri ddprogid:///bin/tv_grab_zz_sdjson_sqlite/EP00000324.0247 in language eng with client tv_meta_tmdb
2023-04-05 20:21:22.720 spawn: 2023-04-05 20:21:22,720:CRITICAL:tv_meta_tmdb:41:Need a tmdb-key. No lookup available with this module.
2023-04-05 20:21:22.720 spawn: 2023-04-05 20:21:22,720:INFO:tvhmeta:314:Failed to import and create module tv_meta_tmdb: Need a tmdb key
2023-04-05 20:21:22.720 spawn: 2023-04-05 20:21:22,720:INFO:tvhmeta:343:Lookup failed with module tv_meta_tmdb for uuid e2d53031a491b00cb39a23ac539b982e title The Andy Griffith Show year None in language eng
2023-04-05 20:21:22.720 spawn: 2023-04-05 20:21:22,720:INFO:tvhmeta:292:Trying title The Andy Griffith Show year None uri ddprogid:///bin/tv_grab_zz_sdjson_sqlite/EP00000324.0247 in language eng with client tv_meta_tvdb
2023-04-05 20:21:22.720 spawn: 2023-04-05 20:21:22,720:INFO:tv_meta_tvdb:68:{'key': 'xxxxxx-e0d7-4afd-xxzz-xxxxxx', 'languages': 'en'}
2023-04-05 20:21:22.907 spawn: 2023-04-05 20:21:22,907:INFO:tvhmeta:314:Failed to import and create module tv_meta_tvdb: 'token'
2023-04-05 20:21:22.907 spawn: 2023-04-05 20:21:22,907:INFO:tvhmeta:343:Lookup failed with module tv_meta_tvdb for uuid e2d53031a491b00cb39a23ac539b982e title The Andy Griffith Show year None in language eng
2023-04-05 20:21:22.908 spawn: 2023-04-05 20:21:22,908:ERROR:tvhmeta:347:Lookup completely failed for uuid e2d53031a491b00cb39a23ac539b982e title The Andy Griffith Show year None
2023-04-05 20:21:22.908 spawn: 2023-04-05 20:21:22,908:ERROR:tvhmeta:464:Failed to process with error: 'Lookup completely failed for uuid e2d53031a491b00cb39a23ac539b982e title The Andy Griffith Show year None'
2023-04-05 20:21:22.928 dvr: /mnt/Medialibrary/Recordings/The Andy Griffith Show/The Andy Griffith Show - S04E06 - Gomer the House Guest.ts from adapter: "IPTV #2", network: "IPTV", mux: "5NFqxxxGDh - MeTV NY", provider: "", service: "MeTV NY

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

Looks like you're probably right:

2023-04-06 06:23:21.675 spawn: 2023-04-06 06:23:21,675:INFO:tvhmeta:314:Failed to import and create module tv_meta_tvdb: 'token'
2023-04-06 06:23:21.675 spawn: 2023-04-06 06:23:21,675:INFO:tvhmeta:343:Lookup failed with module tv_meta_tvdb for uuid b4922d63f65a5a608f2f6b5ba6508e42 title Location, Location, Location year None in language eng

A further couple of things worth noting as of this morning's reading-up, firstly your key looks very different to mine - although with that said clearly neither of us are getting results back from tvdb; and secondly it appears thetvdb is moving to a $ub$cription model but the website isn't very clear about when this will happen, or indeed if it's happened already. I haven't $ub$cribed so perhaps it shouldn't be surprising that my key isn't working either? Whatever the case the metadata I am getting from tmdb seems OK for my purposes.

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

The update from this morning's experiment is that I can get tvdb lookup to work with my Legacy API key:

root@64f6da68c0bd:/# /usr/bin/tv_meta_tvdb.py --tvdb-key <my-legacy-api-key> --title "Bottom" 
2023-04-07 07:51:40,763:INFO:tv_meta_tvdb:{'key': '<my-legacy-api-key>', 'languages': None}
{"poster": null, "fanart": "https://thetvdb.com/banners/fanart/original/78050-1.jpg"}
root@64f6da68c0bd:/# /usr/bin/tv_meta_tvdb.py --tvdb-key <my-legacy-api-key> --title "Race Across The World" 
2023-04-07 07:52:44,699:INFO:tv_meta_tvdb:{'key': '<my-legacy-api-key>', 'languages': None}
{"poster": null, "fanart": "https://thetvdb.com/banners/fanart/original/5c790600db31d.jpg"}
root@64f6da68c0bd:/#

So clearly I'd completely failed to understand which representation of the key to send. Nice to know, and glad to have been prompted to actually do the legwork, less good to know that this is the API they're going to be switching off imminently in favour of their V4 subscription based API. Whatever the case, I'm still happy with what tmdb has on offer.

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

The legacy key works for me but only gets me fanart and no posters. It looks to be the same for you too. TMDB mismatches a lot of the metadata for tv shows for me.

Is there no way to correct the metadata when the match is wrong?

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

Turns out tv_meta_tvdb.py is failing to return posters due to a typo referencing variable "language":

def get_poster(self, title = None, tvdbid = None, languague = "en"):

If you fix this on line 138, you start getting posters returned, where they exist anyway:

root@64f6da68c0bd:/# /usr/bin/tv_meta_tvdb.py --tvdb-key <my-legacy-api-key> --title "Race Across The World" 
2023-04-08 08:27:39,957:INFO:tv_meta_tvdb:{'key': '<my-legacy-api-key>', 'languages': None}
{"poster": "https://thetvdb.com/banners/posters/5c7cef5b3b416.jpg", "fanart": "https://thetvdb.com/banners/fanart/original/5c790600db31d.jpg"}
root@64f6da68c0bd:/#

Note, to steer tvhmeta to use thetvdb artwork instead of tmdb, you'll want to add '--modules-tv tv_meta_tvdb' to your tvhmeta command line.

Nice to find and fix this one, same caveat applies though about the legacy API closing down at some point in the future. Also note tv_meta_tvdb.py is only coded for the V3 API and would need a rewrite to support V4. On reading more into the V4 documentation, it looks like a very different setup - one where the project (tvheadend in this case obvs) registers, and us individual users $ub$cribe to get a PIN that allows us to make queries to the API that associate with the project's ID. Coding-up a new version of tv_meta_tvdb.py to work with this V4 arrangement is beyond my skills, plus I'm unlikely to sign up to a paid service when tmdb artwork is good enough for me, so for now I'll open a PR for this fix and enjoy the artwork for whatever remains of the V3 API's lifespan.

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

James Bevan wrote:

The above fix is now merged upstream.

Thanks for the fix! It now works.

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by D M over 1 year ago

James Bevan wrote:

The above fix is now merged upstream.

This commit managed to break it tvhmeta completely. https://github.com/tvheadend/tvheadend/commit/18effa8ad93e901f3cdaa534123d910f14453d1f

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan over 1 year ago

It does look like a sizeable change all right, best open a detailed bug report. A bit relieved I hadn't updated to it yet...!

RE: Has tvhmeta ever worked when the tvh api wants authentication? - Added by James Bevan about 1 year ago

OK well it looks like "okay" should be set to True, not False, at line 245. As it stands currently, every lookup is going to fail. I'll open a bug and put in a PR.

    (1-17/17)