Project

General

Profile

Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display

Added by Marco Jakobs about 7 years ago

Hi,

for months I was fighting with some issues on recorded files, mostly:

  • The length of the recording was displayed totally wrong in Kodi (i.e: A 50 minutes show was displayed with a runtime like 20:58:17)
  • In some recordings I was not able to skip forward - in that case Kodi immediately ends the playback

As it seems that these issues are not going to be fixed within short term in TVH and I read also about people with the same (unresolved) issue, I took some time today, read through forums and "merged" what I've found into a post recording script.

I did play around with some other recording formats than "pass" before. Recording as MKV seems to be more reliable in term of the two bugs mentioned above, but the big issue is that you can't start watching what is currently recorded. Or better: You can start watching, but that only goes to a certain point and then breaks - it does not recognize that the file "grows" on the other end while still being recorded. But that's what we often do: Record a show and start watching some 20-30 mins later to be able to skip over commercials.

The only solution for this seems to stay in the "pass" recording format.

The wrong time display and the issue with the non-working skip forward seems to be errors in the recorded TS container. It seems to happen more often on recordings which are descrambled (German HD+ via OSCAM), but that's more a statistics feeling - I've at least had wrong time displays also with normal FTA recordings.

But finally I found that rerunning ffmpeg over the file can fix these two issues! :-))

So I did a script like this:

#!/bin/sh

# variables
TSVIDEO=$1    #Full path to recording               /video/Recordings/%{ShowName}/%{FileName.ext}
error=$2      #error from tvheadend recorder        "OK" if no error

# exit if not ok
if [ $error != "OK" ]; then
   echo " Not OK" 
   exit 1
fi

/usr/bin/ffmpeg -i $TSVIDEO -c copy -f mpegts $TSVIDEO.tmp
mv -f $TSVIDEO.tmp $TSVIDEO

... and put it into /home/hts. You need to make it executable!

In the Post-processor-command field in the DVR profile in TVH you then need to enter:

/home/hts/postprocess.sh %f %e

So after finishing the recording TVH will start the ffmpeg process which fixes the TS error but does not recode the streams. That is pretty fast (takes around a minute for a 60h recording) and uses only around 30-40% of a single CPU core. So the load on my ESXi VM is basically around 7-8% when that is processing a file.

As the temporary file is just doing a rename operation over the original file, the original file is not missing at any point - so TVH will not put the recording in "Removed Recordings" by mistake.

And: When you are already watching the recording "from the start" while this process happens, it does not break the playback when the file is replaced (which is curious, but I'm happy about this).

Finally I hope that one day the TS issues are solved at the root in TVH, but this may help some of you guys until that is done.

VBR,
Marco


Replies (7)

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by Em Smith about 7 years ago

May I suggest a couple of simple modifications?

At the end of the "ffmpeg" if you add "&&" then the next line (mv) would only be run if the ffmpeg is successful (in case you run out of disk space or ffmpeg fails half-way through). Since it would leave the tmp file in that case, maybe force remove it at the end for simplicity.

If you add double quotation marks around $TSVIDEO then it will also handle filenames with spaces.

So something like this:

/usr/bin/ffmpeg -i "$TSVIDEO" -c copy -f mpegts "$TSVIDEO.tmp" &&
mv -f "$TSVIDEO.tmp" "$TSVIDEO" 
rm -f "$TSVIDEO.tmp" 

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by Marco Jakobs about 7 years ago

Good points, @Em!

I've made the changes accordingly, thanks for mentioning!

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by Em Smith almost 7 years ago

Couple of other minor points that caused me some problems today.

If the filename starts with "Live:" then ffmpeg treats it as a protocol (like http: ) and aborts. So it needs "file:" in front of the filename to force it to always read the file.

The other minor change I made was to ensure the written file has the same timestamp as the original file (since I don't run it as post-proc, only when needed).

So:

ffmpeg -i file:"$TSVIDEO" ... && touch -r "$TSVIDEO" "$TSVIDEO.tmp" && mv ...

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by K Shea almost 7 years ago

You may be able to stop these bad recordings from occurring in the first place. See this article:

Getting bad satellite recordings in TVHeadEnd? Here’s a possible fix - https://freetoairamerica.wordpress.com/2015/08/01/getting-bad-satellite-recordings-in-tvheadend-heres-a-possible-fix/

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by Marco Jakobs almost 7 years ago

Hi,

it does not seem that's the root cause of the issue, but it seems I've fixed that last weekend ;-)

I did a "netstat -us" on my TVH VM and saw a lot of missed UDP packets due to receiver buffer overflows. It seems that the UDP input buffers are not sufficient in Ubuntu 16.04, so I've added this to my /etc/sysctl.conf:

# Increase size of file handles and inode cache
fs.file-max = 2097152

# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

# Sets the time before the kernel considers migrating a proccess to another core
kernel.sched_migration_cost_ns = 5000000

# Group tasks by TTY
#kernel.sched_autogroup_enabled = 0

### GENERAL NETWORK SECURITY OPTIONS ###

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535

# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1

# Control Syncookies
net.ipv4.tcp_syncookies = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

### TUNING NETWORK PERFORMANCE ###

# Default Socket Receive Buffer
net.core.rmem_default = 31457280

# Maximum Socket Receive Buffer
net.core.rmem_max = 33554432

# Default Socket Send Buffer
net.core.wmem_default = 31457280

# Maximum Socket Send Buffer
net.core.wmem_max = 33554432

# Increase number of incoming connections
net.core.somaxconn = 65535

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65536

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.udp_mem = 65536 131072 262144

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 33554432
net.ipv4.udp_rmem_min = 16384

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65536 33554432
net.ipv4.udp_wmem_min = 16384

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

Then reboot the TVH server. From that point on my UDP RX errors are stable at 0 and I don't see any recording issues.

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by Em Smith almost 7 years ago

Thanks, I'll give those a try.

Recordings were working fine 99% of the time until my machine had major problems recently, now I have a number of files that don't fast forward correctly.

I'll keep an eye on the stats and see if they are increasing before I make the change, but it looks like it could be the network since I reset the tvh stats and my tvh is already saying 4000 continuity errors but my sat>ip server is saying (I believe) 8 errors.

RE: Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display - Added by Em Smith almost 7 years ago

It was a bad network. My sat>ip machine had (suddenly/unexpectedly) decided wi-fi should have a higher priority for sending data instead of the Ethernet cable. Since the cable data light was still flashing, I thought it was all working fine and I hadn't realized there was a problem with that machine.

The "rfkill" command can be used to permanently disable wifi.

    (1-7/7)