Tvheadend

Have you looked at ffmpeg? Use "-c copy" to prevent transcoding, "-ss [time specifier]" before input (-i) to start reading the input at the specified time, and "-t [time specifier]" to limit the new file to the specified length.

For instance, if you have a recording ("recording.ts") that had a 30 second padding before and 90 second padding after that you wanted to remove:

ffmpeg -ss 30 -i recording.ts -c copy -t -90 -f mpegts new_file.ts

ffmpeg can also cut out segments with the "between" filter. An internet search of "ffmpeg edl" should give a good starting point. Also, a general search for "edit decision list" will also give you different pieces of software that can use EDLs for editing.

Are the channels FTA or encrypted? Maybe there is some problem with decryption?

In VLC, under tools->messages set verbosity to 1 (warning) or 2 (debug) and see if it gives any hints as to the problems.

For mpeg files(no mp4 (HD)) , ProjectX is a very good free java program which runs on linux/windows. It repairs most errors and lets you set multiple cut points. But you should demux the stream (and then remux) for best results.

A non-free program is TS-Doctor, But I am not sure what the trial version limits you to.

Take a look at avidemux. I use it for cutting bits out (ie
commercials) of .ts files without having to re-encode.

@ Robert Cameron and Bobby Lockwood:
thanks for the suggestion. I will take a look at both ffmpeg and avidemux and see if it can help me out

• some of the problem files are episodes of a 10-part series, all of the same length (in time)
• the file length (size) for the problem files is comparable to the other episodes. I think when decryption fails, the resulting file size is much smaller than a good file.
• I have checked the oscam logs and found no errors

I'll see what VLC can tell me. I'll get back with some results.

IIRC ProjectX was one of the dioscontinued projects , which couldn't handle HD files becuase that was never implemented? I'll have a more detailed look at it though.
I'll check out TS-Doctor, let's hope the trial version is useful enough to let me judge if the program can help me or not.

Thanks all for your comments so far

Just tested with VLC. Got lots of 'invalid headers'

ts warning: invalid header [0xf6:30:62:ee] (pid: 518)
ts warning: invalid header [0x10:e1:92:34] (pid: 518)
ts warning: invalid header [0x47:50:2f:fb] (pid: 518)
ts warning: invalid header [0x7b:d9:a4:cf] (pid: 518)
ts warning: invalid header [0x8e:43:3c:ab] (pid: 518)
ts warning: invalid header [0x22:8e:d8:42] (pid: 518)
ts warning: invalid header [0x7b:40:26:73] (pid: 518)
a52 debug: emulated sync word (no sync on following frame)
ts warning: invalid header [0x97:b7:76:53] (pid: 518)
ts warning: invalid header [0x37:33:4b:b6] (pid: 518)
ts warning: invalid header [0x1f:36:cb:c3] (pid: 518)
ts warning: invalid header [0x0a:56:8b:1e] (pid: 518)
ts warning: invalid header [0x4c:78:29:5c] (pid: 106)
ts warning: invalid header [0x27:fe:d9:48] (pid: 518)
ts warning: invalid header [0x74:04:5f:c3] (pid: 518)
ts warning: invalid header [0x32:94:d7:63] (pid: 518)
ts warning: invalid header [0xde:c7:27:70] (pid: 518)
ts warning: invalid header [0xb6:2f:96:bc] (pid: 518)
ts warning: invalid header [0xfd:0c:5a:c2] (pid: 518)
ts warning: invalid header [0x45:5f:ee:44] (pid: 106)
ts warning: invalid header [0xe6:4a:34:df] (pid: 518)
...
core warning: picture is too late to be displayed (missing 8139 ms)
core warning: picture is too late to be displayed (missing 8149 ms)
core warning: buffer too late (-9013133 us): dropped
core debug: inserting 49241 zeroes
core debug: auto hiding mouse cursor
core warning: picture is too late to be displayed (missing 10273 ms)
core warning: picture is too late to be displayed (missing 33 ms)
core warning: picture is too late to be displayed (missing 10233 ms)
core debug: auto hiding mouse cursor
core debug: auto hiding mouse cursor
ts warning: invalid header [0xd9:ce:28:b5] (pid: 518)
ts warning: invalid header [0x1b:29:ea:e3] (pid: 518)
ts warning: invalid header [0xe2:20:ab:26] (pid: 518)
ts warning: invalid header [0xb5:6c:a5:be] (pid: 518)
ts warning: invalid header [0x5d:53:b9:a6] (pid: 518)
ts warning: invalid header [0x72:c1:4d:69] (pid: 518)
ts warning: invalid header [0x3b:d2:f8:f9] (pid: 518)
ts warning: invalid header [0x79:ad:cc:ae] (pid: 518)
a52 debug: emulated sync word (no sync on following frame)

Does that indicate the file is lost? Or is there anything I can do to repair it (best effort, I guess it can't be made perfect) so I can at least watch it?

Meanwhile ffmpeg is running on a test file.

Either I did not put the correct parameters and it IS transcoding everything, either I am naively thinking it should be faster than it is.
Right now, it is running at 1.11x speed,and processing the entire file. Is that what I should expect or did I miss something?

The file I'm testing on is a 3.30 GB recording (duration 1:49:59) of which I want to remove the first 6:10 and the last 15:58
From what I can see now, it seems like it will take about 1:50:00 to process.
I'll let it run, and see the result tomorrow morning.

Sounds like you definitely did something odd with your command. I'm running a similar test, and mine is progressing at roughly 40x speed, or about 1100fps. For your test, the command should be something like:

ffmpeg -ss 6:10 -i "input_recording.ts" -c copy -t 1:27:51 -f mpegts "output_recording.ts"

• -ss 6:10 – seek the input file to 6:10 before decoding the input
• -i "input_recording.ts" – use the file "input_recording.ts" as your input
• -c copy – copy the media in the TS file without transcoding (both video and audio)
• -t 1:27:51 – create an output file with a duration of 1:27:51 – 1:49:59 - 6:10 - 15:58 = 1:27:51
• -f mpegts – ensure the output format is MPEG-TS container
• "output_recording.ts" – create a new file "output_recording.ts" as a result.

Haha, I already had a suspicion I had done something wrong. However, I decided to let it proceed and see what happened.

The results of tonight's first try:

• the pre-pad and post-pad is removed at exactly the right moment

• it took a long time
• the quality of the video is not as good as it was, It's pretty bad, actually, instead of HD
• the file size decreased from 3.3GB to 1.4GB

The command I used was:
C:\progs\ffmpeg\ffmpeg-20160913-c19da0c-win64-static\bin>
ffmpeg -ss 370 -i "d:\_ffmpeg\Varg-Veum-NPO3-HD-2016-07-02-20-50.ts" -t 4971 -f mpegts vv02_04.ts

So indeed I forgot the "-c copy" part.
And converted all times to seconds (for -ss and -t) which seems unnecessary.

I saw there was a Linux, Apple and Windows-version available.
I used the Windows version because I had an old Windows-laptop(*) around that I could use

I'll try again with the "-c copy" option this time, and if the results are still not really satisfying, I'll get a spare hard drive and put Linux on that laptop first.
I assume you are using the Linux version and not Windows?

--
(*) You know, one with a dead battery, broken keyboard etc... Written off at work, useless as a laptop in its current state, but still handy for certain tasks

Invalid header is a good indication that there was a decryption problem.

Another reason the size can change is ffmpeg will ignore all those "invalid header" packets...

Joe User wrote:

Invalid header is a good indication that there was a decryption problem.

That would mean: ****ed up beyond repair?

Joe User wrote:

Another reason the size can change is ffmpeg will ignore all those "invalid header" packets...

• the file I used ffmpeg on is a file that is OK (a recording I already watched) just to cut off the pre- and post-padding.
  The size change is probably caused by myself: I missed one of ffmpeg's options in the command line.
• The files that are broken do seem OK in regards to file size and duration when compared to other recordings from the same series.

Yes, not repairable.

The files will show the same size, since all the packets are still saved, just some are not decrypted (or more accurately, wrongly decrypted.)

You will have to look at logs from oscam and tvheadend to try to diagnose why. Does it happen often?

Gert Jansen wrote:

Haha, I already had a suspicion I had done something wrong. However, I decided to let it proceed and see what happened.

The results of tonight's first try:

The good thing:

• the pre-pad and post-pad is removed at exactly the right moment

The bad things:

• it took a long time
• the quality of the video is not as good as it was, It's pretty bad, actually, instead of HD
• the file size decreased from 3.3GB to 1.4GB

The command I used was:
C:\progs\ffmpeg\ffmpeg-20160913-c19da0c-win64-static\bin>
ffmpeg -ss 370 -i "d:\_ffmpeg\Varg-Veum-NPO3-HD-2016-07-02-20-50.ts" -t 4971 -f mpegts vv02_04.ts

So indeed I forgot the "-c copy" part.
And converted all times to seconds (for -ss and -t) which seems unnecessary.

I assume you are using the Linux version and not Windows?

Correct, I am using Linux, but that doesn't really matter as long as the ffmpeg versions are the same. I assume you were using a current/3.x version of ffmpeg.

A note regarding file size: by default, ffmpeg will only select the first audio and video stream in a file. If your TS had multiple video streams (such as a full mux) or multiple audio streams (such as secondary audio tracks), those would not have been included. If you want to keep them, then you need to explicitly state which streams you want included with the "-map" option.

In regards to time durations:
From ffmpeg-utils(1):

There are two accepted syntaxes for expressing time duration.

[-][HH:]MM:SS[.m...]
HH expresses the number of hours, MM the number of minutes for a maximum of 2 digits, and SS the number of seconds for a maximum of 2 digits. The m at the end expresses decimal value for SS.

or

[-]S+[.m...]
S expresses the number of seconds, with the optional decimal part m.

In both expressions, the optional ‘-’ indicates negative duration.

Hopefully your second test goes better.

I should also mention, the documentation for ffmpeg is quite good, and ffmpeg.org has current and extensive man pages posted.

Alright, I have succesfully used both ffmpeg and avidemux (both on Windows)

Both did exactly what I asked of them, and the results are perfectly fine.

Avidemux has a graphical interface where you can have a preview of where you cut, that can be an advantage as well as a disadvantage. It took quite some time before it loaded my file (it was creating a index file I think) and while loading it, avidemux was "not responding" according to Windows.
I gave it enough time, and after a few minutes I could select start and end to cut off, and all the rest went fine.

ffmpeg is faster IMO, both because it does not have to wait for the graphical preview, and in the processing of the file.

But it is nice to have 2 alternatives

Thanks for the detailed explanation of the options Robert.
It would have worked fine after your first post, if only I had copied all of the options you had provided

I will take a look at the more advanced options (and the man page) later.
For now I know this works, and I will switch my attention to the broken files and see what I can do there.

I hope to have some time in the upcoming week to test TS-Doctor and ProjectX

Joe User wrote:

Are the channels FTA or encrypted? Maybe there is some problem with decryption?

Joe User wrote:

Invalid header is a good indication that there was a decryption problem.

I finally found some time to check out some of the repair tools.

• I could not get ProjectX working, I think it does not support HD files, and 98% of what I have are HD files.
  It doesn't work on a perfectly good file either.
  Of course, that can still be a user error instead of something with the program, it happened before ;)

• I tried 2 different versions of TS-Doctor (v 1.2.184 and v 2.0.51)
  Both are fully functional for 30 days, after which a license must be bought.
• I also tried MPEG2Repair (old version from 2007)

Sadly, all 3 found lots of errors in the same file I had checked with VLC before.

Part of the log of TSDoctor 1.2

Scrambled video packets found at the beginning 0
Scrambled video packets found at the end 17013893
Ignore scrambled areas!
First video PTS is 3424627515 10:34:11.417
Last video PTS  is -1 -00:00:00.000

First PCR  is 1027364591700 10:34:10.540
Last PCR  is 1205562885366 12:24:10.477
Duration of video stream is 593994312 01:49:59.937
Video PCR to PTS difference -1116 ms
Patch time base for PID 0026 = 26763312 ms
$006A: Delay to video stream = -660ms
$0026: Delay to video stream = -26762195ms

Part of the log of TSDoctor 2.0

Scrambled parts found in stream!
Cut out scrambled parts
PCR PID is 518 (0206)

searching for channel: SID=21040, TID=3225, VPID=518
                       found channel 2497.  NPO3 HD, M7 Group [Astra 23.5°E]

service_id $5230 = NPO3 HD

First video PTS is 3424627515 10:34:11.417
Last video PTS  is 3477138915 10:43:54.877

First PCR  is 1027364591700 10:34:10.540
Last PCR  is 1043116248850 10:43:53.935
Duration of video stream is 52505524 00:09:43.395
Video PCR to PTS difference -1116 ms
Patch time base for PID 0026 = 26763312 ms
106   (006A): Delay to video stream = -660ms
38    (0026): Delay to video stream = -26762195ms

Part of the log from MPEG2Repair

Sequence Frame 17676(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17677(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17678(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17679(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17680(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17681(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17682(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17683(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header
Error: TS Packet has payload_unit_start_indicator but no PES header
Sequence Frame 17684(0-X) / Time 0:05:50 :
Error: TS Packet has payload_unit_start_indicator but no PES header

• If I do, the remaining file is just about 10 minutes instead of 1:49:59
• If I don't, the error parts are just black (no sound, no vision)

I think I'll have to accept the files are f****ed up beyond repair.

Joe User wrote:

You will have to look at logs from oscam and tvheadend to try to diagnose why. Does it happen often?

• name
• channel
• date
• start and stop time
• which of the 4 recorders I used
• number of errors as reported by TVHeadend

The number of errors as reported by TVHeadend is mostly 0 or very high (due to heavy rain or snow). When it is very high, I know why that's the case.
Sometimes it is non-zero, ranging from 3 to 30, mostly continuity errors. I have one recorder where that happens frequently, can't get that solved.

All of the recordings that now give errors, have 0 TVHeadend Errors.

--

What I would like to do now is check all of my recordings for errors, to see if I can find a pattern in the details in my spreadsheet, and to check if it is still happening.
In case it was a config error, it may or may not be solved by now. I would really like to find out, of course.

Is there an easy way to check 1000 files? Not all files are important, some are just tests (e.g. to check how many simultaneous recordings I could make) but to get a good overview of what is/was happening I need to check those as well.
I can open them with MPEG2Repair (Step 1: Find PIDS, Step 2: Log Errors) or with TS-Doctor, but in both cases that's on a file-per-file basis.

Do you perhaps know a faster way to check the files in batch - and preferably create a log file per recording?

--
--

EDIT: TS-Doctor has a command-line mode.

C:\>"c:\Program Files (x86)\Cypheros\TSDoctor2\TSDoctor.exe" "N:\TVHeadend\Little Britain\*.TS" autocheck log

Unfortunately I have created a directory for every series I have been recording. That's a bit incovenient, but that can of course be overcome.

Gert Jansen wrote:

TS-Doctor has a command-line mode.

C:\>"c:\Program Files (x86)\Cypheros\TSDoctor2\TSDoctor.exe" "N:\TVHeadend\Little Britain\*.TS" autocheck log

Unfortunately I have created a directory for every series I have been recording. That's a bit incovenient, but that can of course be overcome.

I think this will do what I want:

SET TSCOMMAND="c:\Program Files (x86)\Cypheros\TSDoctor2\TSDoctor.exe" 
SET TSOPTIONS=autocheck log

N:
cd \tvheadend

FOR /D /r %%G in ("*") DO %TSCOMMAND% "%%G\*.TS" %TSOPTIONS%

After examination of the logs generated by TSDoctor (see above) I think the problem is caused by one my Oscam servers.
I have 2 Oscam servers, not all of my recorders connect of them, that's probably why the corrupted files seem to come from 2 recorders only.

It remains to be seen if the problem is caused by hardware, software or configuration.
I would like to get to the bottom of it and eliminate the problem, or work around it as good as I can.
I'll open a new topic for that. (http://tvheadend.org/boards/5/topics/22811)