Tvheadend

At startup, tvheadend goes through all the recordings (tvheadend/dvr/log/) and checks if the file exists. If so, an inotify is started for the directory in which the file resides and it keeps a list of the files. Inotify works on idnodes and not by filename, so if a file is "converted", in reality a new file is created (new idnode) and then renamed to match the file which was removed. Inotidy sees the old idnode being removed, but has no idea that the new idnode is associated with the old one. So, tvheadend only sees that the file (idnode) was removed and thus marks the file as being removed. Normally just restarted tvheadend is enough since it will scan for all the files at startup. There should be no need for a reboot. Possibly tvheadend is not really stopping???

As for a fix which does not require restarting, you should be able to use the "filemoved" api (with same source and destination) and see if that triggers tvheadend to re-look for the file..
[[https://github.com/dave-p/TVH-API-docs/wiki/Dvr]]

dvr/entry/filemoved
Informs TVH that a recording has been relocated (by external means) within the filesystem.

src The original full path to the file
dst The new full path to the file

If that does not work, you may be able to do some trick to add the new file. In this thread: [[https://tvheadend.org/boards/5/topics/34366]] I gave some information about exporting/importing recordings from one instance of tvheadend to another. Possibly you could write a script that would "export" all recordings and then periodically check if the file has been modified (replaced) and then try to "import" the recording again.

So, I've played with this a bit by trying curl and doing GETs with a web browser - and I've successfully locked up tvheadend requiring reboots and no other success.

I'm going to try Postman and see what I can do with that. I don't use it very often so I'm going to have to play with it. (As a side note, I've got a project at work that is opensource and uses XMLRPC so the exercise is not wasted. Again, an API that is written in c++, with minimal documentation, and only partially implemented ) However, I'm a bit confused on the documentation.

It looks like the requests should be json objects? However, if you look @ the examples page, they are specified as standard url encoded requests?

Also, how do you know if you need to do a GET or a POST? I've been working on the assumption that to execute a change aka filemoved I would have to perform a POST.

try:

wget -qO- 'http://admin:pass@tvheadend:9981/api/dvr/entry/filemoved?src=/path/to/old/file&dst=/path/to/new/file'

Make sure there are not quotes around the /path/filname. Tvheadend probably should accept quoted files, but it doesn't.
Paths can include spaces.
Also make sure the old (src) is exactly how it is in the dvr/log file (ie. full path or relative) because tvheadend just does a strcmp (string compare) of the "filename" value (without quotes) to the src given.
Tvheadend also makes sure the dst exists, or it will not change.

So I'm going to use Postman instead of wget - there's more info from Postman.

Here's the output of http://admin:[email protected]:9981/api/dvr/entry/grid_removed for a file I know exists:

{
            "uuid": "5ba769a864db21aa31dfbdfb0d96ace7",
            "enabled": true,
            "start": 1548039600,
            "start_extra": 0,
            "start_real": 1548039595,
            "stop": 1548043200,
            "stop_extra": 3,
            "stop_real": 1548043380,
            "duration": 3780,
            "channel": "5d803aacd2bd6db4aaa7c55ad3104a9c",
            "channel_icon": "https://zap2it.tmsimg.com/h3/NowShowing/106839/s90401_h3_aa.png",
            "channelname": "Heroes",
            "title": {
                "eng": "Star Trek: Deep Space Nine" 
            },
            "disp_title": "Star Trek: Deep Space Nine",
            "subtitle": {
                "eng": "S04E05 Indiscretion" 
            },
            "disp_subtitle": "S04E05 Indiscretion",
            "description": {
                "eng": "(1995/10/23) Adventure / Fantasy / Science fiction /  S04E05\nKira reluctantly allows Dukat to accompany her on a mission to find lost Bajoran prisoners -- a mission with emotional consequences for them both." 
            },
            "disp_description": "(1995/10/23) Adventure / Fantasy / Science fiction /  S04E05\nKira reluctantly allows Dukat to accompany her on a mission to find lost Bajoran prisoners -- a mission with emotional consequences for them both.",
            "pri": 2,
            "retention": 2147483646,
            "removal": 5,
            "playposition": 0,
            "playcount": 0,
            "config_name": "fb5df3fef206dbe4870f0b79a4910b53",
            "owner": "kodi",
            "creator": "kodi",
            "filename": "",
            "directory": "Star Trek Deep Space Nine",
            "errorcode": 0,
            "errors": 0,
            "data_errors": 0,
            "dvb_eid": 0,
            "noresched": true,
            "norerecord": false,
            "fileremoved": 0,
            "autorec": "589a6e97f58af6f54d4e62d33372ede2",
            "autorec_caption": "Star Trek: Deep Space Nine (Created from EPG query)",
            "timerec": "",
            "timerec_caption": "",
            "parent": "",
            "child": "",
            "content_type": 1,
            "broadcast": 0,
            "episode": "Season 4.Episode 5",
            "url": "dvrfile/5ba769a864db21aa31dfbdfb0d96ace7",
            "filesize": 0,
            "status": "File missing",
            "sched_status": "completedError",
            "duplicate": 0,
            "comment": "Auto recording: Created from EPG query" 
        },

Note that filename is blank. But the filename exists after MCEBuddy processed it. See below:

root@tvheadend:/# find ./ -name 'St*.ts'
./home/hts/Star Trek Deep Space Nine/Star Trek_ Deep Space Nine-S04E05 Indiscretion.ts
./home/hts/Star Trek Deep Space Nine/Star Trek_ Deep Space Nine-4.ts
./home/hts/Star Trek Deep Space Nine/Star Trek_ Deep Space Nine-S03E20 Improbable Cause.ts
./home/hts/Star Trek Deep Space Nine/Star Trek: Deep Space Nine-S03E12 Past Tense.ts
./home/hts/Star Trek Deep Space Nine/Star Trek_ Deep Space Nine-3.ts
./home/hts/Star Trek Deep Space Nine/Star Trek: Deep Space Nine-S04E06 Rejoined.ts
./home/hts/Star Trek Deep Space Nine/Star Trek: Deep Space Nine-S03E13 Life Support.ts
./home/hts/Star Trek Deep Space Nine/Star Trek: Deep Space Nine.ts

So from Postman:
Headers are set for application/x-www-form-urlencoded

If I manually add the "=" for src to make it a more complete call - same result.

And if I url encode the request:

If I send the request(s) with no headers I get the same responses.

I'm putting waayy too much time into this. I'm seriously considering NextPVR to keep everything on Windows. I went with MCEBuddy because it was more of an Install and Forget solution. ComSkip for Linux didn't seem to run correctly as a post processing script and didn't add all the metadata info.

This after I realized that the Hauppage USB tuners were deaf, on top of not working well under ESXi.
FWIW - the SiliconDust HDHomerunDuo has a better RX than the TV. And I can actually measure the RX with a calibrated device (IFR-1200 Service Monitor US Extra Class Ham License)

The problem is not wget (or curl or postman).
The problem is you cannot change something that doesn't exist (filename).

When the original file is deleted (replaced) inotify tells tvheadend and tvheadend considers the recording as being removed (hence in "removed" list) and removes the "filename" from the record since it no longer exists.

You need to keep the original file until after you execute the wget for filemoved.

One could argue that tvheadend should not remove the filename from the record and only somehow mark it as missing.

Here's the final fix - since I cat back over the original, the old file never disappears. Keeps the inotify monitoring that TVH does happy.

#!/bin/bash

# Script to move MCEBuddy processed files from the receipt folder and overwrite the old existing
# by cat'ing the file back over the old one.
# This is a workaround for how TVHeadend handles file monitoring with inotify.
# by Wurmborn Ranger 1/23/19

# This script makes the following assumptions:
#
# 1. That it is located in the folder that MCEBuddy will be placing the files after running any
#    conversion/comskip processing.
# 2. That the original file is located somewhere in the default /home/hts directory that
#    TVHeadend installs to.
# 3. That this script will be run by root in an appropriate cron job. I'm thinking every 15 min.

# Just to limit the log file size.
function cleanlog {
newlog=`tail -n 50 /home/hts/convert.log`
echo "$newlog" > /home/hts/convert.log
}

cd /home/kodi/processed
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

# Keep a count.
i=0

# If nothing there, then exit
files=`ls /home/kodi/processed/*.ts 2>/dev/null`
if [ ${#files} -eq 0 ]; then
#       printf "$(date) No files found\n" >> /home/hts/convert.log
#       cleanlog
        exit 0
fi

# Main loop
for filename in $(ls *.ts 2>/dev/null); do
printf "$(date) Filename processed = $filename\n" >> /home/hts/convert.log
        # Make sure the file is not being written to before we process it.
        if [ `lsof $filename 2>/dev/null|grep -v 'RW'|awk '{ print $4 }'|grep -c 'w'` -lt 1 ]; then

                # Find the original file
                originalfile=`find /home/hts/ -name "*.ts"|grep "$filename"`
                if [ "$originalfile" = "" ]; then
                        printf "$(date) Original copy of $filename not found\n" >> /home/hts/convert.log
                        mv "$filename" lost/. &>/dev/null
                        continue
                fi

                # Sanity check to make sure we're not overwriting a newer copy.
                olddate=`date +%s -r "$originalfile"`
                newdate=`date +%s -r "$filename"`
                if [ $olddate -gt $newdate ]; then
                        printf "$(date) $filename is older than $originalfile. Moving to lost folder\n" >> /home/hts/convert.log
                        mv "$filename" lost/. &>/dev/null
                        continue
                fi
                cat "$filename" > "$originalfile" 

                # Fix for some weirdness in permissions on my CIFS shares
                        setfacl -b "$originalfile" 
                        chmod 775 "$originalfile" 

                # Back to original script
                printf "$(date) $originalfile has been updated\n" >> /home/hts/convert.log

                # Send a notification to Kodi
                curldata="'{"id":1,"jsonrpc":"2.0","method":"GUI.ShowNotification","params":{"title":"Converstion Script","message":"File $filename has been converted"}}'" 
                curl -v -H "Accept: application/json" -H "Content-type: application/json" -d $curldata http://kodi:XXXXX@homepvr:8080/jsonrpc
                rm "$filename" 
                ((i++))
        fi
done

# If there were files processed, then log the number.
if [ $i -gt 0 ]; then
        printf "$(date) $i files processed - exiting\n" >> /home/hts/convert.log
fi

cleanlog

exit 0

Now I just need to figure out how to get MCEBuddy to use different comskip.ini files for different channels.

If the script is working for you, that is great, but I would make a few suggestions to simplify.

You can remove the section:

# If nothing there, then exit
files=`ls /home/kodi/processed/*.ts 2>/dev/null`
if [ ${#files} -eq 0 ]; then
#       printf "$(date) No files found\n" >> /home/hts/convert.log
#       cleanlog
        exit 0
fi

And then just move the cleanlog call into the if statement

# If there were files processed, then log the number.
if [ $i -gt 0 ]; then
        printf "$(date) $i files processed - exiting\n" >> /home/hts/convert.log
        cleanlog
fi

since you know the filename, you do not need to grep for it:
instead of

originalfile=`find /home/hts/ -name "*.ts"|grep "$filename"`

just use:

originalfile=`find /home/hts/ -name "$filename"`

no need to make videos executable:

chmod 664 "$originalfile" 

That said, I would do it differently:
1. record as usual with tvheadend
2. use mcebuddy the same, (keeping the same filename but putting processed file in different directory)
3. in script just mv the processed file to the original directory with an added string to indicate it was processed by mcebuddy. Something like basename-mce.ts. (Bash has good tools to tools to get filenames without the extension or path, ie basename="${filename%*.ts}") That way it is always clear which files have been processed and which ones haven't.
4. Use the api (like I wrote above) to tell tvheadend of the change.
5. I would NEVER delete an original file automatically without verifying the "processing" worked as intended.

I do not have time to rewrite your whole script, but main part would be something like:

cd /path/to/files
for filename in *.ts; do
     originaldir=`find /home/hts/ -name "$filename" -printf '%h\n'`
     if [ -z "$originaldir" ]; then
          mv "$filename" lost/
          continue;
     fi
     file="${filename%.ts}" 
     mv "$filename" "$originaldir/$file-mce.ts" 
     wget -qO- 'http://admin:pass@tvheadend:9981/api/dvr/entry/filemoved?src=$originaldir/$filename&dst=$originaldir/$file-mce.ts'
.
.
.

(may need to check the quotes needed for the wget command- I do not have time to test...)

This way you would only write each file twice and read it once instead of writing each file three times and reading it twice.