Tvheadend

Instead of just moving the file, the safest way is probably to use the "filemoved" API call to inform Tvheadend about the new file:

#!/bin/sh
# $1 = File
# $2 = Error

_ERROR=${2}

#_TVH_USER="" 
#_TVH_PASS="" 
_TVH_IP="127.0.0.1" 
_TVH_PORT="9981" 
_API_ENDPOINT="/api/dvr/entry/filemoved" 

_BIN_FFMPEG="/usr/bin/ffmpeg" 
_BIN_CURL="/usr/bin/curl" 

_FILE_FULL=$(readlink -f "${1}")
_FILE_PATH=$(dirname "${_FILE_FULL}")
_FILE_NAME=$(basename "${_FILE_FULL}" .ts)

if [ ${_ERROR} != "OK" ] ; then
  echo "$(date): Error with conversion of ${_FILE_NAME}" 
  exit 1
fi

echo "$(date): Converting ${_FILE_NAME}" 
nice -20 ${_BIN_FFMPEG}        \
  -loglevel fatal -hide_banner \
  -i "${_FILE_FULL}"           \
  -map 0 -c copy               \
  -y "${_FILE_PATH}/${_FILE_NAME}.mkv" 

# Uncomment if you wish to ensure group-level access 
#echo "$(date): Changing permissions on ${_FILE_NAME}.mkv" 
#chmod 664 "${_FILE_PATH}/${_FILE_NAME}.mkv" 

# If you need user/pass for Tvheadend access, change the URL to:
# "http://${_TVH_USER}:${_TVH_PASS}@${_TVH_IP}:${_TVH_PORT}${_API_ENDPOINT}".
# Also, check that the user/pass variables are set
echo "$(date): Notifying Tvheadend of the change to ${_FILE_NAME}.mkv" 
${_BIN_CURL} -s "http://${_TVH_IP}:${_TVH_PORT}${_API_ENDPOINT}" \
  --data-urlencode "src=${_FILE_FULL}"                           \
  --data-urlencode "dst=${_FILE_PATH}/${_FILE_NAME}.mkv" > /dev/null

echo "$(date): Conversion of ${_FILE_NAME} finished" 

echo "$(date): Removing ${_FILE_NAME}.ts" 
rm "${_FILE_FULL}" 

I know it's not quite perfect, but here's a quick and dirty based on a script I used to do nearly the same thing you were trying to do. I added your error-checking and call to nice, although the nice call probably isn't really necessary since ffmpeg is only repackaging the streams and not really converting anything.

Awesome Robert, thanks a ton! :)

This is exactly what I was looking for (did not know about the API calls) and it's working perfectly. Also it gives me the advantage of just a very minimal "outage" when still watching a recording from "behind" as the .mkv is already present before the .ts file is deleted - so I don't need to wait the time when it copies (overwrites) the original file. In theory I should be able to immediately restart the playback once the output froze due to the missing .ts file which was playing.

Working perfectly by calling the script via the shell, I'll schedule some real life test recordings now.

Marco

To ensure that you don't have those odd freezing issues when you delete/remove the original file, and safer alternative might be to move the original file to a temporary location, so if the change happens mid-viewing, the file isn't lost and you aren't interrupted. Then, run a cronjob to remove the items from the temporary location early every morning. Something like:

# Replace the `rm` command at the end of the script with:
mkdir -p /tmp/tvh_converted
mv {_FILE_FULL} /tmp/tvh_converted/

# As a crontab entry (to run at 3:00am daily):
* 3 * * * rm -rf /tmp/tvh_converted

So,

now it's getting horrible complicated! :(

When two scheduled recordings are ending at the same time this ends up in a mess!

It seems the first recording is running the post processing script, but the second recording (with a different file name of course) seems to crash something.

Let's say, these two recordings are ending at the same time:

1) tagesschau-ARD.ts
2) heute-ZDF.ts

"tageschau" makes the run and starts the processing script. As I output each action into a log file I see that everything is copied and converted. Fine so far.
In the background I guess TVH wants to start the post process script for "heute" nearly at the same time, but that's not working yet - maybe only one script instance can run at the same time? I do not even see a log file entry that the conversion of "heute" did ever start.

Then the runninng "tagesschau" script wants to notify TVH about the change with the API call, here the script just DIES! The TVH webserver (I had this open in a browser window) got unresponding. But I'm catching when the CURL times out, in this case the script it's not working at all anymore. There is no entry in my log at all after this point (also failures are logged).

In addition, the second recording "heute" does never appear in the TVH database - it's just not there (although the .ts file is OK and in position).

So when TVH calls the post processing script by the ending of another recording while another one is still running this seems to end in a serious faulty situation. :-|

Does anyone know a workaround here? My only stupid idea is that the post process script just writes a kind of a "task file" into a directory and then exits immediately, and a cron job which is called every minute checks for new "tasks" and processes them (which will a be totally independent thread from TVH).

That idea implies a lot of reading on my side (I'm an embedded C guy and really not experienced in shell scripts ;) )

I'm probably closer to your situation than most others. The script I posted was a modified version of one I used as part of something to selectively repackage my TS files into MKV containers. I've had bad luck trying to get Tvheadend to properly and reliably run post-processing scripts. Furthermore, it seems that Tvheadend has problems with escaping/sanitizing the external commandline; arguments often get mangled.

I think running your own worker-process outside of Tvheadend might be the best way to handle your situation.

(Although, personally I have just been sticking with TS containers ... queueing through a recording may be a struggle at times, but it's manageable. Maybe someday Tvheadend will allow stream profiles to applied to recordings to allow them to stream at the requested profile instead of only at the stream profile used for the recording.)

Thanks, Robert!

I've tried to run the script in the background, so instead of postprocess.sh TVH now calls postrecord.sh which looks like this:

#!/bin/bash

./postprocess.sh $1 $2 &
exit 0

Even more bad luck ... records which end at the same time will now completely be ignored converting wise, I don't see even one single log line. Works perfect from shell and if just one single recording ends..

So I'll dive into that and realize my plan B: TVH will just create a "task" file for a recording and a cron job will check for new task files each minute and processes that independently. That should normally work (hopefully).

Done!

New strategy: TVH just calls the script "postrecord.sh" as post processing file. This does nothing else then creating a "taskfile" in my convert directory, it then immediately exits.

In the crontab of the user "hts" the script "convert.sh" is called once per minute. If it finds a taskfile it'll do the conversion, notify TVH about the filemane change and then exits.

Works in shell, I'll now schedule some recordings with identical end time tomorrow to see how that works in pratice :)

postrecord.sh:

#!/bin/sh

# variables
TSVIDEO=$1    #Full path to recording               /video/Recordings/%{ShowName}/%{FileName.ext}
error=$2      #error from tvheadend recorder        "OK" if no error

logfile="/home/hts/convert/conversions.log" 

filedate=`date '+%Y%m%d%H%M'`
filename=`basename $TSVIDEO`    #Nur der Filename
filepath=`dirname $TSVIDEO`     #Nur der Pfad
taskfile="/home/hts/convert/${filename%.*}-${filedate}.tsk"   #Nur der Name des Zielfeiles ohne Pfad

echo "\n\n$(date): *** Creating conversion task for ${TSVIDEO} ***" 
echo "\n\n$(date): *** Creating conversion task for ${TSVIDEO} ***" >> $logfile

if [ ! -d "/home/hts/convert" ]; then
  mkdir "/home/hts/convert" 
fi

# exit if not ok
if [ $error != "OK" ]; then
 echo "$(date): TVH returned error for recording of ${TSVIDEO}" 
 echo "$(date): TVH returned error for recording of ${TSVIDEO}" >> $logfile
 exit 1
fi

echo "${TSVIDEO}\n" >> $taskfile

exit 0

convert.sh:

#!/bin/bash

# Check for a new taskfile
taskfile="" 
for i in /home/hts/convert/*.tsk; do
    [ -f "$i" ] || break
    taskfile=$i
done

if [ -z "$taskfile" ]; then
  exit 0      #Raus, kein Taskfile da
fi

logfile="/home/hts/convert/conversions.log" 

# Holt die erste Zeile aus dem File in $TSVIDEO
TSVIDEO="" 
while IFS='' read -r TSVIDEO || [[ -n "$TSVIDEO" ]]; do
    break
done < $taskfile

if [ -n "$TSVIDEO" ]; then
  echo -e "$(date): Conversion file found: ${TSVIDEO}" 
  echo -e "\n$(date): Conversion file found: ${TSVIDEO}" >> $logfile
else
  rm -f $taskfile   # Taskfile löschen, da steht nix sinnvolles drin
  exit 0      #Raus, kein Taskfile da
fi

# Taskfile löschen
rm -f $taskfile

# Prüft, ob das Videofile aus dem Taskfile existiert
if [ ! -f $TSVIDEO ]; then
#  echo "Videofile not found!" 
  exit 0
fi

# Hier die Umwandlung beginnen
_TVH_USER="admin" 
_TVH_PASS="password" 
_TVH_IP="127.0.0.1" 
_TVH_PORT="9981" 
_API_ENDPOINT="/api/dvr/entry/filemoved" 
_BIN_CURL="/usr/bin/curl" 

#compactbase=$(echo "$basename" | sed "s/...$//")    # Remove file ext (3 char)
filedate=`date '+%Y%m%d%H%M'`
filename=`basename $TSVIDEO`    #Nur der Filename
filepath=`dirname $TSVIDEO`     #Nur der Pfad
mkfilename="${filename%.*}-${filedate}.mkv"    #Nur der Filename des Zielfiles
tempfile="/home/hts/convert/$filename"   #Name des temp-files
targetfile="/home/hts/convert/${filename%.*}-${filedate}.mkv"   #Nur der Name des Zielfeiles ohne Pfad
MKVIDEO="$filepath/${filename%.*}-${filedate}.mkv"   #Kompletter Pfad des neuen Files

copyok=0           #Fehler-Flag auf 0 initialisieren

echo -e "$(date): Starting conversion of ${TSVIDEO} to ${MKVIDEO}" 
echo -e "$(date): Starting conversion of ${TSVIDEO} to ${MKVIDEO}" >> $logfile

if [ ! -d "/home/hts/convert" ]; then
  mkdir "/home/hts/convert" 
fi

# Kopiert in lokales tmp-Directory, Konvertiert dort in MKV und kopiert dann auf TVH zurück.
echo -e "$(date): Copying ${filename} to /home/hts/convert/" 
echo -e "$(date): Copying ${filename} to /home/hts/convert/" >> $logfile
nice -20 cp -f "$TSVIDEO" "$tempfile" &&

echo -e "$(date): Converting ${filename} to ${mkfilename}" &&
echo -e "$(date): Converting ${filename} to ${mkfilename}" >> $logfile &&
nice -20 /usr/bin/ffmpeg -i "$tempfile" -sn -vcodec copy -acodec copy -flags +global_header -f matroska "$targetfile" &&

echo -e "$(date): Copying final file back to ${MKVIDEO}" &&
echo -e "$(date): Copying final file back to ${MKVIDEO}" >> $logfile &&
nice -20 cp -f "$targetfile" "$MKVIDEO" &&          #File erst mit falscher Endung (.ts) rüberkopieren, damit TVH die Database nicht löscht
# Uncomment if you wish to ensure group-level access 
#echo "$(date): Changing permissions on ${MKVIDEO}.mkv" 
#chmod 664 "$MKVIDEO" &&

copyok=1          #OK-Flag setzen wenn bis hierhin erfolgreich

if [ "$copyok" -eq 0 ]; then
  echo "File copy / conversion failed! Exiting" 
  echo "File copy / conversion failed! Exiting" >> $logfile
  rm -f "$tempfile" 
  rm -f "$targetfile"                              # Temp-File löschen
  exit 1
fi

# If you need user/pass for Tvheadend access, change the URL to:
# "http://${_TVH_USER}:${_TVH_PASS}@${_TVH_IP}:${_TVH_PORT}${_API_ENDPOINT}".
# Also, check that the user/pass variables are set
echo -e "$(date): Notifying Tvheadend of the change from ${filename} to ${mkfilename}" 
echo -e "$(date): Notifying Tvheadend of the change from ${filename} to ${mkfilename}" >> $logfile

success=0
repeat=0
while [ "$repeat" -lt 5 ]; do
  if ${_BIN_CURL} -s "http://${_TVH_USER}:${_TVH_PASS}@${_TVH_IP}:${_TVH_PORT}${_API_ENDPOINT}" \
    --data-urlencode "src=${TSVIDEO}"                           \
    --data-urlencode "dst=${MKVIDEO}" > /dev/null; then
    echo -e "$(date): Notification successful!" 
    echo -e "$(date): Notification successful!" >> $logfile
    success=1
    break
  else
    echo -e "$(date): Notification attempt failed ..." 
    echo -e "$(date): Notification attempt failed ..." >> $logfile
    sleep 30
  fi
  repeat=$((repeat+1))
done

if [ "$success" -gt 0 ]; then
  echo -e "$(date): All done, deleting ${TSVIDEO}" 
  echo -e "$(date): All done, deleting ${TSVIDEO}" >> $logfile
  rm -f "$TSVIDEO"                              # Original TS-File löschen
else
  echo -e "$(date): Could not notify TVH about the change - deleting ${MKVIDEO} and leaving original file" 
  echo -e "$(date): Could not notify TVH about the change - deleting ${MKVIDEO} and leaving original file" >> $logfile
  rm -f "$MKVIDEO"                              # Neues MKV-File löschen
fi

echo -e "$(date): Removing temporary files" 
echo -e "$(date): Removing temporary files" >> $logfile

rm -f "$tempfile" 
rm -f "$targetfile"                              # Temp-File löschen

if [ "$success" -gt 0 ]; then
  echo -e "$(date): Conversion task of ${filename} finished OK" 
  echo -e "$(date): Conversion task of ${filename} finished OK" >> $logfile
else
  echo -e "$(date): Conversion task of ${filename} FAILED" 
  echo -e "$(date): Conversion task of ${filename} FAILED" >> $logfile
fi

I have had issues recording some IPTV sources. Generally I find the local ones OK, but stuff on the other side of the world can be prone to errors. I've tried recording with MKV, but errors at the start of the stream can result in sound, but no picture, or vice-versa, or recordings with weird aspect ratios etc.

Using the 'pass' profile has its problems too, errors in a transport stream can result in a recording that can be played, but pausing, fast forwarding or skipping forward or back can result in the whole thing freezing, and the only option is to quit and play from the start again.

I've made modified version of the scripts above - I hope someone finds them useful.

The remuxing script can be used to manually remux previously recorded .ts files by passing the full path to the file[s] on the commandline:

tvhremux /path/to/file1.ts /path/to/file2.ts ...

When called with no parameters, it will process task files saved by postrecord.sh - one invocation will process all task files present. When post processing, a lock mechanism ensures only 1 tvhremux.sh runs in the case of 2 instances being started simultaneously.

postrecord.sh:

#!/bin/bash

# Modify to suit your server
HOME=/home/pi/.hts/convert/
LOGFILE="/tmp/conversions.log" 
BINDIR="/usr/local/bin" 
# variables
TSVIDEO=$1    #Full path to recording
error=$2      #error from tvheadend recorder        "OK" if no error

filedate=`date '+%Y%m%d%H%M'`
filename=`basename $TSVIDEO`
filepath=`dirname $TSVIDEO`
taskfile="$HOME${filename%.*}-${filedate}.tsk" # Full path to task file

echo "\n\n$(date): *** Creating conversion task for ${TSVIDEO} ***"  | tee -a $LOGFILE

if [ ! -d "$HOME" ]; then
    mkdir "$HOME" 
fi

if [ $error = "OK" ]; then
    echo "${TSVIDEO}" >> $taskfile
    # Kick off remuxer
    $BINDIR/tvhremux.sh &
else
    echo "$(date): TVH returned error for recording of ${TSVIDEO}"  | tee -a $LOGFILE
    exit 1
fi

exit 0

tvhremux.sh

#!/bin/bash

# Modify to suit your server
HOME="/home/pi/.hts/convert" 
TVH_USER="admin" 
TVH_PASS="secret" 
TVH_IP="127.0.0.1" 
TVH_PORT="9981" 
API_ENDPOINT="/api/dvr/entry/filemoved" 
BIN_CURL="/usr/bin/curl" 
LOGFILE="/tmp/conversions.log" 

# remux $1 to $2 & notify TVH
remux()
{
SRC=$1
DST=$2
if [ -e $SRC ] && [ ! -e $DST ] ; then
    echo -e "$(date): Starting conversion of \n${SRC} to \n${DST}" | tee -a $LOGFILE
    ffmpeg -i $SRC -c copy $DST
    if [ $? -eq 0 -a -e $DST ] ; then
        echo -e "$(date): Notifying Tvheadend of the change from\n$(basename $SRC) to \n$(basename $DST)" | tee -a $LOGFILE
        SUCCESS=0
        REPEAT=0
        while [ "$REPEAT" -lt 5 ]; do
            if ${BIN_CURL} -s "http://${TVH_USER}:${TVH_PASS}@${TVH_IP}:${TVH_PORT}${API_ENDPOINT}" \
                --data-urlencode "src=${SRC}" \
                --data-urlencode "dst=${DST}" > /dev/null; then
                echo -e "$(date): Notification successful!" | tee -a $LOGFILE
                    SUCCESS=1
                    break
              else
                let REPEAT++
                    echo -e "$(date): Notification attempt $REPEAT failed ..." | tee -a $LOGFILE
                sleep 30
            fi
        done
        if [ "$SUCCESS" -gt 0 ]; then
            echo -e "$(date): All done, deleting ${SRC}" | tee -a $LOGFILE
            rm $SRC
        else
            echo -e "$(date): Could not notify TVH about the change - deleting ${DST} and leaving original file" | tee -a $LOGFILE
            rm -f $DST
        fi
    else
        echo "$(date): Conversion task of ${SRC} FAILED" | tee -a $LOGFILE
        if [  -e $DST ] ; then
            rm $DST
        fi
    fi
else
    if [ ! -e $SRC ] ; then
        echo -e "$(date): ${SRC} not found" | tee -a $LOGFILE
    fi
    if [ -e $DST ] ; then
        echo -e "$(date): ${DST} already exits" | tee -a $LOGFILE
    fi
fi
}

# Do conversion of $1, handling multiple files if -n.ts suffix is present
convert()
{
FTYPE=mkv
INFILE=$1
NAME=$(basename $INFILE)
FILENAME=${NAME%.ts}
PATHNAME=$(dirname $INFILE)
VID=$PATHNAME/$FILENAME.$FTYPE
TERM=${FILENAME:(-2)}
TERM2=${TERM:0:1}
if [ "$TERM2" = "-" ] ; then
    SUFFIX=${TERM:1}
else
    SUFFIX="" 
fi
# If we have a suffix of say, -3 - then we need to account for:
# PROG-NAME-YYYY-MM-DDHH-MM-3.xyz
# PROG-NAME-YYYY-MM-DDHH-MM-2.xyz
# PROG-NAME-YYYY-MM-DDHH-MM-1.xyz
# PROG-NAME-YYYY-MM-DDHH-MM.xyz
if [ -n "$SUFFIX" ] ; then
    FNAME=${FILENAME%%??}
    while [ $SUFFIX -gt 0 ]
    do
        remux $PATHNAME/$FNAME-$SUFFIX.ts $PATHNAME/$FNAME-$SUFFIX.$FTYPE
        let SUFFIX--
    done
    remux $PATHNAME/$FNAME.ts $PATHNAME/$FNAME.$FTYPE
else
    remux $INFILE $VID
fi
}

if [ $# -gt 0 ]; then
     # Commandline usage - convert each parameter [full path to file].
    for FILE in $@ ; do
        convert $FILE
    done
else
    # Postprocessing - read all task files & convert
    #
    # Random delay of up to 10 sec to ensure when two
    # instances are started simultaneously, only 1 will continue
    #
    LOCK=/var/lock/$(basename $0).lock
    RANDOM=$$
    DELAY=$(expr $RANDOM % 10)
    echo Waiting for $DELAY sec
    sleep $DELAY
    if [ -e $LOCK ]; then
        echo exiting as $LOCK exists
        exit 0
    fi
    touch $LOCK

    # Check for new taskfiles
    for TASK in $HOME/*.tsk; do
        if [ -f "$TASK" ] ; then
                # Get the first line from the file in $TSVIDEO
            TSVIDEO="" 
            while IFS='' read -r TSVIDEO || [[ -n "$TSVIDEO" ]]; do
                break
            done < $TASK

            rm -f $TASK
            if [ -n "$TSVIDEO" -a -f "$TSVIDEO" ]; then
                echo -e "$(date): Conversion file found: ${TSVIDEO}" | tee -a $LOGFILE
                convert $TSVIDEO
            fi
        fi
    done
    rm -f $LOCK
fi

An update on tvhremux.sh above.

Sometimes the conversion fails with an ffmpeg error 'dimensions not set'. This can happen when you get corrupted data at the start of a stream.

This can be fixed by adding -analyzeduration to the ffmpeg commandline, e.g.

        ffmpeg -fflags +genpts -analyzeduration 15M -i $SRC -c copy $DST