Tvheadend

Yes, I did it for a while but it becomes tedious to do it perfectly. Unfortunately I can't easily share the scripts since they have a lot of hard-coded scraper credentials and are a bit ugly as I found more and more annoyances.

I found the problem with the scraper is that they often have completely made-up episode numbers that don't correspond to the actual episodes, especially for specials. So, my OTA EPG for the channel that made the programme tells me it is S6E2, but the first person to insert details in the scraper insists it is S0E14, and then locks it to prevent any discussion/updates since they've already named their file S0E14.

Similarly, movies can be a year-or-two out since they used to be released at different times in different countries until, was it Hulk that got the really bad reviews and made studios decide to change so they release at roughly the same time everywhere so people in other countries don't bypass bad movies?

With the %U (uuid) option in commands in tvh, it should be easier to write nowadays. For basic nfo, you could basically get the record from tvh based on uuid (similar to python fanart code in tvh), extract title, description, etc. (perhaps using jq or xslt), and then write to the nfo file, with exact format depending on whether it was EP or MV. The other tedious bit is that episodes also need a tvshow.nfo, but I think that can be minimal.

Thanks Em - sounds rather complicated (I really am still not very good with Linux).

I wonder if I should change tact and try something else.

I totally agree about TVDB - is a wonderful idea if you are cataloguing all your favourite shows and films and have the time to go over everything.

However, as you say, random stuff that you record just doesn't seem to nicely fit in with the scraper. I have been using the Missing Movie Scanner add-on and it has found masses of recordings that have never been added at all, for one reason or another.

Furthermore I have increasingly found that lots of new series that I record have to have the year in brackets in the title. I guess that this is inevitable - as more and more programs get made, the likelihood of a duplication increases.

This has meant that lots of new series get scraped as older, completely unrelated, series.

To tell you the truth, at the moment, I am watching everything via file mode in Kodi (i.e. not library mode and not via the recording folder in the TVHeadend add-on).

My biggest bugbear with this is the watched status tick on the folders - folders seemed to get ticked, when none of the videos inside have been ticked and then, conversely, when all the files inside are ticked, the folder itself sometimes does not produce a tick.

Not the worst issue in the world but still very irritating - I either think that there is a new episode of a series (and get disappointed to find that there isn't) or I don't watch new episodes because there is no way of me knowing if from the folder.

May I ask what strategy you are using?

Have you tried adding a Kodi file source of "pvr://" (one of the options in advanced IIRC). That then gives you your file hierarchy but is using htsp (TVHeadend protocol) to actually play the files and you get proper watched status since it is from tvh. You can then long-click (or press C for custom menu) to change the scraper for "good" directories and they'll get scraped in to your library; others you can just browse.

I've tried a number of strategies but now just use LiveTV.

There was a time last year when htsp would constantly crash for me, so I added code to tvh to generate "scraper friendly names" so I could browse the file hierarchy directly and later started scraping to the library. But I then realized that scraping often doesn't work.

So I wrote a bunch of nfo generators, and spent quite a while improving them with extra SD information and tvdb/tmdb lookups to get full information since I could do better fuzzy matching (this movie says 1999 but tmdb says 1998, this episode says S5E4 but the title matches S5E3). But, there were still some annoyances in how Kodi worked for me, especially with deleting watching programmes.

So then I added code for artwork/fanart to my SD grabber, tvh and Kodi plugin. And so now I'm back full circle at using the standard "LiveTV" in Kodi, but at least having decent artwork, apart from the fact some skins decide to ignore my artwork and scrape their own incorrect one (even in LiveTV) which is very annoying.

One reason I prefer the LiveTV to scraping is because my grabber adds things like age ratings of "U" or "Adult language", and that is useful to me when looking for films to watch when there is a mixed age group in the room since 99% of my programmes are recorded via generic rules ("movies after 1990 rated 4* or higher", "films starring Tom Cruise"), hence I have no idea what most programmes are until I read the description! Most skins I've tried don't show age ratings, but do tell me the show is stereo.

I had added to tvh a "sub-directory by age rating" but I had things like some episodes of old Top Gear rated 18 with others as PG, or some episodes of a cookery show as 15! So, it proved too annoying.

Thanks for that.

I added "pvr://" as a source and was pretty chuffed when three folders showed up - one for channels, one for frequencies and one for recordings.

I was hoping that every file in my recordings folder would be in there (even those that were old and not logged by TVHeadend), but clicking on the recording folder did nothing - i.e. it wouldn't open. Why do you think that is?

I then noticed that recordings was an option in my sources (recordings/tv/active) - I tried adding this as a source but this just gave me the same files as the recording folder.

Unless I am doing something wrong, I am not sure what more I can do.

I can try importing every single recording back into TVHeadend, which I am told is very lengthy or perhaps find a way of hacking Kodi itself, to try and get the folder tick back in file mode (as mentioned above). This used to happen in previous versions of Kodi (I am sure it was there in Helix), so there must be a way of getting it back.

That I don't know. I've used pvr on a number of different machines and versions and it works, but I don't have the very latest versions, mine are a few months old.

I go "Videos->Files->Add. Browse button. Recordings." This then shows "pvr://recordings/tv/active", which sounds like what you have. Inside that I see "tvmovies" and "tvshows", but I click "ok" to get the top level directory added as a source. You can then press "c" (or long press if your remote supports it) to change content type on the sub-directories to allow them to be scanned. It seems a bit temperamental sometimes and so sometimes setting shows removes movies, but eventually it will work.

The only thing I can think of is maybe it is a tvh permission problem? But, I don't know why your LiveTV would work if such a problem existed.

The main reason to do it this way (rather than directly browse the filesystem) is because the file tick should be synched back with tvheadend (since it is using htsp instead of direct file reads), so it may workaround your problems with ticks.

You mention you have lots of files not imported in tvh?

Sorry - yes,I record to single hard drive that has only two folders, "TV" and "Movie".

I have two presets set up in TVHeadend, so that the default is TV but if I know that a recording is a Movie, I choose the other one.

I have to reinstall TVH a number of times over the last few years and each time I do, I have to start from scratch - i.e. TVH only logs the recordings made since that install, so there are masses of recordings in the recording folder that TVH is completely unaware of.

Since you're using Schedules Direct, if you also use tvh dev branch (4.3) then you can use the "$q" format specifier to automatically split tvshows and tvmovies, so no need to do it manually. The autorec rules can also "match if unique" which will use the Schedules Direct unique identifier for duplicates.

(If you use my unsupported version of the grabber then you also get artwork and it will prefer tvdb season/episode numbering if SD has specific knowledge of a difference; but take a backup of your .xmltv directory first since mine modifies the database which means you won't be able to re-run the official version without restoring the db).

Unfortunately it's a bit difficult to import the old files if you have lost the original dvr/log entries. I think the filenames have some specific format so you can't just generate a new unique id filename and say 'filename: "abc"' in it.

Right - I see.

I was hoping that all files that happened to be in the recording path (regardless of whether they were in the TVH database or not) would appear, but sadly not - as you say, trying to add these files back into TVH is just not going to be worth it.

Maybe I should just accept that I have to have two sets of file, those in my TVH Recording folder and those in my Library.

Out of interest - how stable is 4.3?

What you have mentioned above sounds amazing, but I am slightly worried about upgrading from 4.2 if it is too buggy.

I've not upgraded my 4.3 for a while due to lack of time, but it's usually stable, but sometimes might have a week of less reliability where some fixes might aggravate something else that was broken but had always seemed to work.

Once you get a version that works then stick with it! Don't go chasing the latest and greatest version unless you see some feature you want, and always take a quick peek at the bug reports before upgrading just in case.

Features which might be of use to you are the "$q" [[https://github.com/tvheadend/tvheadend/blob/master/docs/property/pathname.md]]; "record if unique" option in autorec (so just works regardless of if it is a movie or episode); "mark as previously recorded" (so if your autorec matches a movie you've seen then you can flag it so it isn't recorded again); save epg after xmltv import; and artwork poster/fanart passed to Kodi (v18+/Leia+). But, there are thousands of behind the scenes improvements.

But, ensure you backup your dvr/log before trying to run it since the format of entries has changed so recordings won't be backward compatible (can't be imported if you later downgrade).

Thanks Em - that is really sensible advice.

I am always excited about new features but, as you say, the most important thing is a stable PVR system that just works (plus I really want to avoid irritating my wife and son with missed recordings).

I think I will wait for the next stable version - the things you have mentioned above are going to be really useful to me.

Thanks so much for you help with this (and another thread you have been helping me with).

You clearly have spent a lot of time with your answers and I just wanted to say that it is very much appreciated.

Thanks, much appreciated.

Don't let me put you off trying the 4.3 though since it is usually stable.

Since you're moving machine, you could test it for a few days by setting up your stable machine as a "sat>ip server" and use your other machine as a "sat>ip client". So your tuners from stable would then be shared with your other machine, and you can record the same shows on both machines (since the tuner mux would be shared and could record on both machines).

So on stable machine you'd go config->general->satip server, change port number from 0 to 9983 to enable the server; update "exported dvb-s tuners" to 2; then go to config->dvb->network and set "satip source" to 1 for your dvb-s network. Then on the other machine you'd start with a copy of your original machine's config and should be able to just config->general->satip server->discover and they should show up as adapters which you map to your "dvb-s" network. Sounds complicated, but it's not too difficult when you have the UI in front of you.

Might be more time/effort than it's worth, but it means you can run both systems for a few days to see if it is stable/worth the risk.

A rather intriguing development.

I have come accross this page:

https://kodi.wiki/view/Video_file_tagging

which suggests that you can now put files into Kodi's libraries, scraping from metadata contained in the file.

I know that .ts files cannot include metadata, but .mkv files can.

If I start saving my files as MKVs and check the box for "Tag files with metadata", this might just work.

It could be the perfect solution that we have been looking for - I use your version of the Schedules Direct grabber. If I could have this data in my library, it would be the perfect solution.

As Sir Humphrey might say, that's a very courageous decision

I've not used mkv tagging for a while, but it used to be recommended to record ts and then post-process to mkv if required. I occasionally would have missing audio/subtitles with mkv, possibly because the channel stream can "reconfigure", so you'd have a 4:3 mp2 programme followed by a 16:9 aac/audio described programme (that you wanted to record), but tvh had already decided the streams to use so would ignore the aac stream.

Looks like it might be fixed though [[https://tvheadend.org/issues/4882]].

I'm not sure if the files are tagged with artwork and don't know if it uses the exact same tags that Kodi now reads since their wiki suggests they don't use the iTunes tags that most software uses. But it's certainly worth a try.

I looked a little in to the Kodi scraping and my initial thought is that it wouldn't solve your problem of having episodes not scraped or scraped incorrectly.

I looked more closely at the wiki since I couldn't determine how it would work with episodes (which need tvshow info when using separate nfo files) vs. the easier case of movies which only really have title+year+plot, and perhaps "movie set" info.

The wiki said:

There are three more tags that should be filled in for TV Shows: Show, Season and Episode Number. At the moment these tags do not seem to be of any practical use in Kodi, but should nevertheless be filled out as this may change.

It still wasn't clear, so I looked at the original Kodi PR [[https://github.com/xbmc/xbmc/pull/13106/commits/47bb0b69bb07c9a6499e00dcbcabbee66a98ef29]] and the latest version of the code in xbmc/video/tags/VideoTagLoaderFFmpeg.cpp.

A casual inspection (of avtag->key) suggests it parses "title", "director", "date_released", "title", "composer", "genre", "date", "description", "synopsis", "track", "album", "artist"; and a couple of other places where is parses artwork, "kodi-metadata", "kodi-override-metadata", and a couple of places where it uses a different context for "IMDBURL" or "TMDBURL".

Perhaps it parses season and episode and associated show details elsewhere (so you know an episode is part of a specific series).

I could easily be wrong though since I haven't done much research, just a quick look.

Perhaps you could just add tags to one of your own existing files and give it a try?

One of my concerns is that the metadata option seems to be an "all or nothing" button (rather than per-directory so you can enable it on file hierarchy you know is good).

I'm also not sure what happens if you have, say, a title in the mp4 but no plot or artwork. This probably means you don't get any art. And since they don't parse cast, it means the one feature I like of displaying actor thumbnails would presumably be lost.

It seems the best tag to add would be the TMDBURL or IMDBURL and then, perhaps, it would scrape the extra info it needed for both episodes and movies.

For your missing recordings, you could try the attached script to re-create a basic dummy dvr/log so you are not missing so many items from your reinstall.

Download the script, then "chmod 755 tvhimport" to make it executable. Backup.

Run it with the name of a media file and it will print to the screen what it would generate in the logfile (so you can check it). Only fields you're probably interested in are title and episode. We assume files are roughly named "title - SxxEyy.ts" or "title (yyyy).ts".

Run it with "./tvhimport --dvr-log-dir /path/to/hts/dvr/log file_to_import.ts" and it will generate a file for you in that directory. You can specify a wildcard such as *.ts. It's not a good idea to include files that already have an existing dvr/log.

I'd probably output to a temporary directory first, check a few files to see if they look correct, then chown/chmod/"mv -i" a couple of the files to the correct location, reboot tvheadend and see if it works in the UI/Kodi.

Seems to work on tvh4.3, but some fields were changed in 4.3 so I don't know if it works on tvh4.2. Obviously many of the fields are dummy since the information has been lost. I can't guarantee it won't mess something up since you're not really meant to mess around with internals like this, but you pays your money, you takes your chances.

Maybe it would be better to use the api to "import" the new dvr log entries?
From this thread ([[https://tvheadend.org/boards/5/topics/34366?r=34637#message-34637]]
here is a partial script which adds the "conf=" prefix to a dvr/log file and then uses curl to create the dvr entry and test if it worked.

# Use logfile to create json data for import

dir="/path/to/recordings/json/"  # note important trailing /  
filename=${file##*/}  # get just filename

printf "conf=" | cat - "$logfile" > "$dir$filename".import.json

#wait to make sure the network file is written
sleep 5

# Create recording entry on play2
curl -q  --data @"$dir$filename".import.json 'http://user:pass@wetek_play2:9981/api/dvr/entry/create' > "$dir$filename".reply

if grep -q uuid "$dir$filename".reply ;then
        echo "worked" 
        mv "$dir$filename".import.json "$dir"$filename"import.json.done
else
        echo "failed" 
fi

Definitely use a temp directory to create the entries for import. (do not touch the dvr/log directory)

This way you know which imported ok and you do not have to stop and restart tvheadend...

Sorry, I do not have time to write a full script, but the important parts are there (add "conf=", curl command and test for result).

That's very good advice.

I think that in the off-chance we create the same uuid as an existing entry then we'd get an error back from the api and could just re-run the generation to create a new uuid.

Using the api, you do not need to generate any uuid (it is not in the conf ) - tvheadend will create the uuid when it creates the dvr log entry (and file).
A temporary file you create is only used to upload the data into tvheadend, which itself will create the real file in the dvr/log dir.

Newer version attached that uses the api directly, with "--dry-run" to test it out without doing an actual import.

I am wondering whether one of the Kodi Library Managers [[https://kodi.wiki/view/Library_managers]] will help with your file naming/nfo generation? Perhaps they have better algorithms for matching shows to metadata since they have to deal with badly named items?

Thank you, both, for your further input on this.

A few further interesting developments:

According to the Kodi forum:

https://forum.kodi.tv/showthread.php?tid=339620

It does seem that the issue that I was dealing with has been raised as a bug. With a bit of luck, if they introduce folder watched statuses back into file mode, this would pretty much solve my problem.

I also spent the week messing around with Emby.

Initially, things went particularly well - it did seem to solve my issues. Files were added to the TV and Movie library without needing to be scraped (which was exactly what I was after), but then I started to get a few issues with the Emby server suddenly not being available.

I ended up uninstalling it again. It is a great piece of software but it did seem massively over-complicated/bloated for my setup and what I want to achieve.

I also did some testing with my Metadata idea above and, as you predicted Em, it did not quite work. None of the important additional information was included in the .mkv file, so I was no better off than having recorded it as a .ts file.

I wonder whether all I can do is just wait for the fix in Kodi - I seem to have exhausted all the reasonable options.

I'll add the link to the Kodi issue you mentioned:
https://github.com/xbmc/xbmc/issues/15324

Though, that issue says "when content type is none." Do you get the same problem if you change the content type to "tv" and scraper to "local nfo" (of which you will have zero). You can still browse the file hierarchy but do you get the correct watch status?

There was another "just the filename" scraper that relied on a php script on localhost rather than Google, so perhaps that might also work better. I don't recall the name of the scraper though.

But, yes, looks like a frustrating week.

Unfortunately, picking "TV" and "local nfo" makes no difference - still no watched status on folders.

I did actually have a go with "justthefilename" too - unfortunately, that only works with movies, rather than TV Shows. Furthermore, I did try it with some of my movies (just to see if it worked) and it kept stopping - something about problems with the remote server.

The search goes on - just typical! Trust me to find the only missing link in an otherwise perfect system