Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help)
Added by Andrew Blank almost 10 years ago
So let me start with what "Working but needs help" means.
It means this script is
- flagging for commercials and outputting and .edl file
- Mencoder is reading the edl file and making cuts
- HandBrake CLI is reading the cut file and transcoding
- The File name also includes the original air date as grabbed from the xmltv file generated by mc2xml
What's not working is
- After being cut the audio and video are out of sync and the video gets jumpy where the original was not.
If someone knows a terminal video editor that can use an .edl aside from Mencoder I'm all ears.
Notes:- If I don't run the recording through Mencoder then HandbrakeCLI transcodes the original file with commercials just fine with no sync issues. So there's something wrong with the Mencoder stage. I've also tried (without luck) to convert the Mpeg-TS to Mpeg-PS first with Mencoder and FFMpeg and rebuilding the index before comskip scanning to no avail.
- Some of the commands use sudo for permissions issues (bad I know). I setup auto sudo privilege for hts using visudo. Example can be found here: https://tvheadend.org/boards/5/topics/11932
- Ubuntu Server 14.04
- Intel Core 2 Duo
- Hauppauge HVR 1600
- Time Warner Cable QAM service
Pre-requisites to this script:
- Comskip Linux port from here: http://forum.kodi.tv/showthread.php?tid=150084
- HandBrake CLI
- xmllint (libxml2)
- perl
- mencoder
- ffmpeg
- mc2xml setup with TVHeadend as instructed here (https://tvheadend.org/boards/4/topics/10322)
TVHeadend Post Processing Command:
/home/hts/Convert %f %b %c %C %t %d %e %S %E
#!/bin/bash # $1 Full path to recording (%f) /home/user/Videos/News.mkv # $2 Basename of recording (%b) News.mkv # $3 Channel name (%c) BBC world # $4 Who created this recording (%C) user # $5 Program title (%t) News # $6 Program description (%d) News and stories... # $7 Error message (%e) Aborted by user # $8 Start time stamp of recording, UNIX epoch (%S) # $9 Stop time stamp of recording, UNIX epoch (%E) #My paths for example## #WorkPath=/home/hts/postprocessing #OutputPath includes show name #with the date in the filename (part of this script) #Plex can scrape thetvdb.com info :) #OutputPath=/media/seagate/DVR/$5 #Log=/home/hts/logs/ConvertLog.txt #XmlPath=/home/hts/mc2xml #File Name and path without extension #BaseFileName=${1%.*} #end my paths example## WorkPath=/path/for/handbrake/output OutputPath=/path/to/final/output Log=/home/path/to/log/file XmlPath=/path/to/xmltv.xml #File Name and path without extension BaseFileName=${1%.*} echo " " >> $Log date >> $Log echo "Input Path and File: " $1 >> $Log echo "Channel: " $3 >> $Log echo "Recording User: " $4 >> $Log echo "Program title: " $5 >> $Log echo "Program Description: " $6 >> $Log echo "Error Message: " $7 >> $Log echo "Input Start Time: " $8 >> $Log echo "Input End Time: " $9 >> $Log who = whoami echo "Who Am I: " $who >> $Log #Convert Unix time-stamp to YYYY-MM-DD.HH-MM-SS set to variable $t t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y%m%d%H%M%S', localtime($8))"); StartTimeLog=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d.%H-%M-%S', localtime($8))"); FileNameTime=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d.%H-%M', localtime($8))"); echo "Readable Start Time: " $StartTimeLog >> $Log echo "Converted Start Time: " $t >> $Log #query original air date from xmltv.xml (Generated by MC2XML) AirDate=$(xmllint --xpath "string(/tv/programme[title=\"$5\" and desc=\"$6\" and contains(@start, \"$t\")]/previously-shown/@start)" $XmlPath/xmltv.xml); echo "AirDate Query: xmllint --xpath 'string(/tv/programme[title=\"'"$5"'\" and desc=\"'"$6"'\" and contains(@start, \"'"$t"'\")]/previously-shown/@start)' $XmlPath/xmltv.xml" >> $Log echo "AirDate: " $AirDate >> $Log #Trim whitespace from Air Date Converted AirDate = "$AirDate" | tr -d ' ' #flag for commercials echo "***Comercial Flagging***" >> $Log echo "comskip" $1 >> $Log #attempts to convert to Mpeg-PS before comskip scan #MpegFile=$BaseFileName.mpg #ffmpeg -i "$1" -map 0:a -c:a copy -map 0:v -c:v copy "$MpegFile" #sudo mencoder "$1" -forceidx -mc 0 -noskip -oac copy -ovc copy -of mpeg -o "$MpegFile" #comskip "$MpegFile" # 2>&1 </dev/null >> $Log #end attempts to convert to Mpeg-PS before comskip scan comskip "$1" # 2>&1 </dev/null >> $Log echo "comskip Exit code: " $? >> $Log echo "***End Comercial Flagging***" >> $Log EdlFile=$BaseFileName.edl echo "EDL File Name Used:" $EdlFile >> $Log #Check if edl file was generated if [ -f $EdlFile ]; then #Cut Commercials echo "***Comercial Cutting***" >> $Log CommercialCutFile=$BaseFileName.CommCut.mpg sudo mencoder "$1" -edl "$EdlFile" -oac copy -ovc copy -of mpeg -o "$CommercialCutFile" echo "Commercial Output File: " $CommercialCutFile >> $Log echo "mencoder exit code: " $? >> $Log echo "***End Comercial Cutting***" >> $Log HandBrakeInput=$CommercialCutFile else #edl file is missing, encode original and output to log echo "Commercials not cut!" >> $Log HandBrakeInput=$1 fi echo "HandBrake Input File: " $HandBrakeInput >> $Log #Check if original Airdate was found in xmltv.xml if [[ -z "$AirDate" ]]; then #Airdate not found use today's date from recording start time. echo "Start Time of recording used in name" >> $Log #Convert Start Date of Recording for Plex/Kodi(XBMC) Naming convention. t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d', localtime($8))"); #Set Handbrake Output File HandBrakeOutput=$workPath/$5.$t.MP4 else #Original Airdate was found echo "Previously Aired date from xmltv.xml used in name" >> $Log #Convert AirDate to date format for Plex/Kodi(XBMC) Naming convention. AirDateConverted=$(echo ${AirDate:0:4}-${AirDate:4:2}-${AirDate:6:2}); #Set Handbrake Output File HandBrakeOutput=$WorkPath/$5.$AirDateConverted.MP4 fi #Transcode to h264 file. echo "HandBrake Output Path: " $HandBrakeOutput >> $Log sudo HandBrakeCLI -i "$HandBrakeInput" -o "$HandBrakeOutput" -X 720 --preset="Normal" echo "HandBrake Exit Code: " $? >> $Log #If HandBrake was successful if [ $? == 0 ]; then echo "Checking for Output Folder: " $OutputPath >> $Log #Check if folder exists if [ ! -d "$OutputPath" ]; then echo "Output Folder Created: " $OutputPath >> $Log #if not make new folder mkdir "$OutputPath" fi #move transcoded file to Output mv "$HandBrakeOutput" "$OutputPath/" fi #delete files after transcoding. rm -f $BaseFileName*
Replies (9)
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Prof Yaffle almost 10 years ago
Without digging into details, I'd guess that your cuts aren't clean so you're losing sync as a result... a slightly ill-informed opinion, but it's the first thing that pops into my head.
From your comment, the sync issue is there immediately after the mencoder stage, yes?
Thoughts:
1. Do you need to cut - can you use 'skip' instead (depends on what you're playing it on, of course)
2. Could you transcode first, perhaps upping the number of I-frames (you may not have any in the TS source) which are the only safe things to cut on (--keyint, I think)
(You could probably also use ffmpeg to cut instead of mencoder - I don't know how many segments it'd accept in one go, though, so it may be iterative - mkvtools might also work, as mkvmergegui certainly allows you to chunk things up)
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Andrew Blank almost 10 years ago
I will have to look into those options. My output folder feeds into a Plex Server that scrapes for the TV info and plays on my Roku. That being said, unlike the XBMC system on my RPi, Plex can't use the edl to skip so I would need to cut. Cutting in combination with transcoding also has the side benefit of a much smaller file size.
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Andrew Blank almost 10 years ago
Update!: Running some tests to see but it looks like comskip and mencoder's edl function can actually run on the h264 mp4 file generated by Handbrake. This may change everything with the sync issue. fingers crossed
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Andrew Blank almost 10 years ago
Executables have been tested to work. Script has been written. Recording has been set. Just waiting for recording to finish to make sure it all works and I'll post an update :).
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Andrew Blank almost 10 years ago
More testing is required to make sure this is working consistently but this does appear to work for now:
#!/bin/bash # $1 Full path to recording (%f) /home/user/Videos/News.mkv # $2 Basename of recording (%b) News.mkv # $3 Channel name (%c) BBC world # $4 Who created this recording (%C) user # $5 Program title (%t) News # $6 Program description (%d) News and stories... # $7 Error message (%e) Aborted by user # $8 Start time stamp of recording, UNIX epoch (%S) # $9 Stop time stamp of recording, UNIX epoch (%E) OutputPath=/media/seagate/DVR/$5 Log=/home/hts/logs/ConvertLog.txt XmlPath=/home/hts/mc2xml #File Name and path without extension BaseFileName=${1%.*} echo " " >> $Log date >> $Log echo "Input Path and File: " $1 >> $Log echo "Channel: " $3 >> $Log echo "Recording User: " $4 >> $Log echo "Program title: " $5 >> $Log echo "Program Description: " $6 >> $Log echo "Error Message: " $7 >> $Log echo "Input Start Time: " $8 >> $Log echo "Input End Time: " $9 >> $Log who = whoami echo "Who Am I: " $who >> $Log #Convert Unix time-stamp to YYYY-MM-DD.HH-MM-SS set to variable $t t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y%m%d%H%M%S', localtime($8))"); StartTimeLog=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d.%H-%M-%S', localtime($8))"); echo "Readable Start Time: " $StartTimeLog >> $Log echo "Converted Start Time: " $t >> $Log #query original air date from xmltv.xml (Generated by MC2XML) AirDate=$(xmllint --xpath "string(/tv/programme[title=\"$5\" and desc=\"$6\" and contains(@start, \"$t\")]/previously-shown/@start)" $XmlPath/xmltv.xml); echo "AirDate Query: xmllint --xpath 'string(/tv/programme[title=\"'"$5"'\" and desc=\"'"$6"'\" and contains(@start, \"'"$t"'\")]/previously-shown/@start)' $XmlPath/xmltv.xml" >> $Log echo "AirDate: " $AirDate >> $Log #Trim whitespace from Air Date Converted AirDate = "$AirDate" | tr -d ' ' #Set Handbrake Input File echo "HandBrake Input File: " $1 >> $Log #Set Handbrake Output File HandBrakeOutput="$BaseFileName.mp4" #Transcode to h264 file. echo "HandBrake Output Path: " $HandBrakeOutput >> $Log sudo HandBrakeCLI -i "$1" -o "$HandBrakeOutput" -e x264 --x264-preset ultrafast --x264-profile high --vfr -X 720 -q 20 -a 1,2,3 -E copy --audio-fallback mp3 -5 echo "HandBrake Exit Code: " $? >> $Log #flag for commercials echo "***Comercial Flagging***" >> $Log echo "comskip" $1 >> $Log comskip "$1" # 2>&1 </dev/null >> $Log echo "comskip Exit code: " $? >> $Log echo "***End Commercial Flagging***" >> $Log #Set edl File Name EdlFile="$BaseFileName.edl" #allow time for edl to write. sleep 5 echo "EDL File Name Used:" $EdlFile >> $Log #Check if edl file was generated by comskip. if [ -f "$EdlFile" ]; then #Cut Commercials echo "***Comercial Cutting***" >> $Log CommercialCutFile="$BaseFileName.m4v" sudo mencoder "$HandBrakeOutput" -edl "$EdlFile" -oac mp3lame -ovc copy -of avi -o "$CommercialCutFile" echo "Commercial Output File: " $CommercialCutFile >> $Log echo "mencoder exit code: " $? >> $Log echo "***End Comercial Cutting***" >> $Log FileToMove="$CommercialCutFile" else echo "Commercials not cut!" >> $Log FileToMove="$HandBrakeOutput" fi if [[ -z "$AirDate" ]]; then echo "Start Time of recording used in name" >> $Log #Convert Start Date of Recording for Plex/Kodi(XBMC) Naming convention. t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d', localtime($8))"); #Set Output File OutputName="$5.$t.MP4" else echo "Previously Aired date from xmltv.xml used in name" >> $Log #Convert AirDate to date format for Plex/Kodi(XBMC) Naming convention. AirDateConverted=$(echo ${AirDate:0:4}-${AirDate:4:2}-${AirDate:6:2}); #Set Output File OutputName="$5.$AirDateConverted.MP4" fi echo "Checking for Output Folder: " $OutputPath >> $Log #Check if folder exists if [ ! -d "$OutputPath" ]; then echo "Output Folder Created: " $OutputPath >> $Log #if not make new folder mkdir "$OutputPath" fi #move transcoded file to Output mv "$FileToMove" "$OutputPath/$OutputName" #delete files after transcoding. #rm -f $BaseFileName*
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Lee Pru almost 10 years ago
You might not like the following but it works very very reliably well!
Write a script that plays the files one after the other in 2 pipes, feed the output to VLC and let VLC stream it out.
So:
YOUR SCRIPT: feed files sequentially to pipe.1 and pipe.2 -> vlc "stdin from scrpt"
Hope this helps out. Good luck!
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Andrew Blank almost 10 years ago
@Lee What will that help me do and at what step in the script above would you suggest it?
Do you have an example of what that looks like in a bash script?
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Lee Pru almost 10 years ago
This whole operation would be in its own script. The way I have it setup (and I didn't write the script myself, I paid someone to do it), I have a Python script that feeds these recorded files to VLC and VLC is the one that outputs. In your setup, I would assume it goes last in the chain when you want to view the recordings.
I don't know Python very well but this is how the script goes:
- find all files and sort them accordingly into array
- create/reuse 2 named pipes (let's say pipe1.fifo and pipe2.fifo)
- launch VLC that has the 2 pipes as input and "loop" enabled
- start piping the files to the pipes: first pipe file#1 to pipe1.fifo and when near end of file#1 start piping file#2 to pipe2.fifo, etc.
This whole feeding business goes on in a infinite loop or until no files exist.
I think this is beyond what TV Headend is for, though, so I guess we shouldn't discuss it here. I just wanted to let people know that this is possible because I was getting hopeless until I hired a professional Python programmer to make this thing a reality.
Hope it helps
RE: Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help) - Added by Andrew Blank almost 10 years ago
Ok so this script is working but it has some drawbacks and I have some more cool ideas but I won't have time to keep coming back to this forum so from here if you're interested in the progress I now have a GitHub for it here:
https://github.com/andyrblank/Commercial-Clapboard
And I'll probably blog about it here when I have time:
https://blankstechblog.wordpress.com/
Thanks to everyone for your input in this!