Tvheadend

As far as I can tell there is no entry in the FAQ that states that old recordings are automatically deleted.

There is a project proposal to implement such a feature, but that is not yet implemented.
So for now you manually need to monitor the available disc space and manually delete recordings that are no longer needed.

Hein

Hi,
You are right! I didn't see that in the FAQ but in the basic guidelines: Tvheadend will clear enough room to store the expected recording size at the start of the recording. It will take into account already running recordings.

Do you guys always delete your recordigs manually? I happen to record more than 10 programs per day so I would like an automated behaviour. If it does not exist at this time, I will now write a script which can be put inside the crontab in order to automatically delete the oldest .mkv files recursively under a given directory, provided that the free space is below a given threshold.

Regards

like you said in an other thread: With 1TByte of storage you have enough room for 1 month of recordings. Why keep a programs after you have seen it? How often do you watch it again? And if you record more then you can watch, then why record it at all.

So yes ... I delete my recordings manually. And actually, I move them away from the server to a central location so that my recording directory always has enough room.

Hein

Hello,
The fact is that I don't see all of them! I only want to be able to choose between 20 movies when I want to see one. Thee oldest programs should be deleted when space is needed.
I record all these programs to have the choice (in order to never have to go through Live TV!), not to see all of them. This kind of feature is available in the other PVRs.
I will now write an autodelete script and share it with the community.

1. Save the script inside any folder (e.g. $HOME/scripts/autodelete)
2. Give it the executable access rights:
   

   chmod a+x <path to autodelete>
   

3. By default, the script runs in simulation mode and will not delete any file. It allows you to check its behaviour before you can use it in production.
   Try a test run using:
   

   /path/to/autodelete --space 20 <recordings_path>
   

   
   where 20 is the amount (in GB) of space you want to free and recordings_path is the path where you keep your recordings.
4. Sample output:
   

   2012-05-18 22:51:00,275 INFO Cleaning 20.0 GB of .mkv files in folder /pchc_f/Recorded TV/
   2012-05-18 22:51:00,348 INFO Current free space in /pchc_f/Recorded TV/ is 12.45 GB
   2012-05-18 22:51:00,348 INFO At least 7.55 GB need to be cleaned
   2012-05-18 22:51:00,348 INFO The following files need to be deleted:
   2012-05-18 22:51:00,348 INFO   * 01.33 GB - Sun Apr  1 10:05:00 2012 - /pchc_f/Recorded TV/Karambolage/ARTE-HD-Karambolage.2012-04-01.09-45.mkv
   2012-05-18 22:51:00,349 INFO   * 02.91 GB - Sun Apr  1 13:05:00 2012 - /pchc_f/Recorded TV/Philosophie/ARTE-HD-Philosophie.2012-04-01.12-30.mkv
   2012-05-18 22:51:00,349 INFO   * 02.37 GB - Sun Apr  1 15:10:00 2012 - /pchc_f/Recorded TV/Sherlock/France-4-Sherlock.2012-04-01.13-30.mkv
   2012-05-18 22:51:00,350 INFO   * 00.91 GB - Sun Apr  1 20:15:00 2012 - /pchc_f/Recorded TV/Karambolage/ARTE-HD-KARAMBOLAGE.2012-04-01.20-01.mkv
   2012-05-18 22:51:00,350 INFO   * 01.01 GB - Mon Apr  2 00:35:00 2012 - /pchc_f/Recorded TV/L'Øcho-des-lois/LCP-L'écho-des-lois.2012-04-02.00-00.mkv
   

5. When you are sure that you have found the right settings, you can use the script in production by adding the --production switch. The files which were found in simulation mode will REALLY be deleted!
   

   /path/to/autodelete --space 20 <recordings_path> --production
   

6. You can then automate the execution of this script after each recording. Since I could not configure Tvheadend to run a post-processing script with arguments, you will have to create a script (e.g. my_autodelete) by copying and pasting the line you have come up with:
   

   #!/usr/bin/env sh
   /path/to/autodelete --space 20 <recordings_path> --production
   

7. Give it the executable rights:
   

   chmod a+x /path/to/my_autodelete
   

8. Ask Tvheadend to run it after each recording by using the web interface, going to Configuration / DVR and set "Post-processor command" as /path/to/my_autodelete

You can then regularly check the work performed by the script by looking at the /tmp/autodelete.log file.

A handy script, but beware about tvheadend directory permissions if you decide to delete an empty directory. If someone is recording in, say, /data/recordings, and records all programmes "flat" (no date or channel sub-directories), then if all the recordings are deleted in that dir (e.g. manually or by the autodelete script), then the next time autodelete runs, it will actually delete the /data/recordings empty dir itself.

Now, it may be that /data is writable by the tvheadhead user and it can just create /data/recordings again and carry on, but there's a decent chance it's a root-owned mount point and therefore future recordings will fail to start because /data/recordings isn't creatable.

What the script probably needs is to test that the parent directory of the empty dir to be deleted is writable by the tvheadend user (if it isn't, don't delete the empty dir).

Hi,
This new version (1.1) checks, before deleting an empty directory:
- that it has the write access rights on this directory
- that it is not the root directory

Moreover, the logs (including the exceptions) are now written to ~/.config/autodelete, using automatic rolling logs (like syslog).

I have been using this script in production for 3 weeks now, and I am extremely happy: I now record about 20 programs per day using Tvheadend, which are automatically deleted after about one month. I never run out of free space, which was a major cause of bugs when I did not have this script.

I use XBMC as a front-end with tvheadend as the back-end and XBMC does offer the option to manually delete recordings (one at a time, ho hum), though there is an option in XBMC's System -> Settings -> Appearance -> File Lists to toggle this on or off. However, if I write click on a finished recording (Live TV -> Recordings), I'm presented with a "Delete" option even when I've turned off the rename/delete option in the XBMC Settings (duh! Bug?!).

What's a double whammy, though, is that if I attempt to delete a TV recording via XBMC as detailed above (and with the XBMC rename/delete option enabled), it will indeed remove it from the recordings list on XBMC, but does not delete it from the tvheadend backend, leaving potentially large files hanging around on the tvheadend server that - as people here seem to do - have to be deleted "again" from outside of XBMC.

As I've said in another thread, tvheadend leaves a lot to be desired when it comes to space management - there needs to be some status page with filestore stats and a warning appearing if it's getting now. As my above experience shows, deletion requests from the front-end should be passed back to the backend and acted upon (assuming permissions are OK), rather than giving the impression a programme has been deleted on the tvheadend filestore when it clearly hasn't.

Hi,
I just improved the "autodelete" script in order to allow to only keep the last 3 episodes of some shows (e.g. TV series, daily or frequent shows).
In combination with #1083 we will now get a much better space management, until #1398 is implemented.

Here is how I use it: I added a post-processor command to a script containing:

autodelete /pchc_f/Recorded\ TV --space 30 --production --limit_folders "Philosophie;Le-dessous-des-cartes;Yourope;Metropolis;Ludo;Le-petit-journal;Entrée-libre;Envoyé-spécial;Suspect-n°1;Silence,-ça-pousse;Personne-ne-bouge;Mystères-d'archives;Des-mots-de-minuit;Un-livre,-un-jour;Karambolage;La-grande-librairie;Ce-soir-(ou-jamais-\!);Le-blogueur;Les-aventuriers-d'Explô;Archipels;28-minutes" 

the --limit_folders argument will only keep the last 3 recordings of the shows passed as an argument.

How does this clean the tvheadend DB of the recording so it is not under failed "File Missing" ?

Hi,
It does not clean the database automatically, but to my knowledge it is not needed anymore since the latest version of Tvheadend seems to notice that the recording was removed and to update the database automatically. Under XBMC I do not get any "file missing" issues anymore since a few versions of Tvheadend.

I still see them in tvheadend with a git build from 11 days ago. I wound up building a python script from the samples to delete the recordings from tvheadend. Some initial tests worked fine, I have not had time to use it in my automated scripts ( update the perl script that does comskip, CC capturing and handbrake re-encoding at very high compression 60min ~ 220MB ish ) But I think this fix will do the trick and I will have a clean process for comercial skipping -> transcoding -> and moving into xbmc that will run unattended for many months. Now if only I could get the episode data from the free listings program to match I would have a near perfect FREE TV recording system.

• Last file deleted by my script:
  

  2013-10-29 07:25:00,168 INFO Cleaning 30.0 GB of ['mkv', 'ts'] files in folder /
  pchc_f/Recorded TV
  2013-10-29 07:25:00,223 INFO Current free space in /pchc_f/Recorded TV is 27.87 
  GB
  2013-10-29 07:25:00,223 INFO At least 2.13 GB need to be cleaned
  2013-10-29 07:25:00,223 INFO Now deleting the following files:
  2013-10-29 07:25:00,226 INFO   * 02.59 GB - Tue Oct 22 07:50:00 2013 - /pchc_f/R
  ecorded TV/Yourope/ARTE-HD-Yourope.2013-10-22.07-15.mkv
  

• In the web interface > Finished recordings: I don't see it anymore!
• In XBMC > Live TV > Recordings: I don't see it anymore!

Hi,
As a matter of fact, you are right, it is now filed under "Failed recordings > File missing" in the web interface.
But it is not annoying since it is not listed anymore under XBMC. It is even a good thing that Tvheadend notices that the file is missing and does not list it to the clients!

This said, it is not a Failed recording. It could be an improvement in Tvheadend: the "Failed recordings" tab should be renamed for instance: "Failed or deleted recordings". Apart from that, I cannot see why you want to delete the recording from the database: it is an interesting fact, from a logging point of view, to remember that there used to be such a recording, and that it was deleted by the user. I can file a new demand if you think it would have an interest.

In my case the database is just full of old inaccurate junk after a few months since "all" the recordings are marked as failed and the original files *.mpg no longer exist. That is not tvheadends fault I am snatching files it created and transcoding/renaming/relocating them. Hence my original desire to have a command line recording delete. Since I have figured out how to do this with the python library and examples I am good to go. tvheadend is doing the correct thing since the file IS missing and no one told it to stop managing the recording.

I take the nice and small mkv files with the edl comercial cuts and move to my XBMC TV library then tell xbmc to refresh the library and all is good. I don't manage recordings from the tvheadend plugin because there are none (Since they all get marked as failed ). Only Live TV and EPG and scheduling of recordings.

Hi,
After a given number of days set up in the web interface (30 days for me), Tvheadend should not keep on managing the recordings. Doesn't it then delete the database record?
Do you lose any quality after transcoding with Handbrake? Can you share your scripts?
With my autodelete script, I do not have the problem you mention. The database is perhaps full of junk but it does not have any impact on my script nor on my XBMC setup (same as you: Live TV, EPG, recordings). I still don't get it why you want to delete these junk records.
Regards

I had not noticed it deleting recordings, I'll pay closer attention but there were a huge number.

Quality -- technically allways loose some but practically speaking none and from 5 GB -> 220MB is a no brainer, and you can allways adjust to suit your taste. You can get the commands from the scripts

Yes this is about keeping tvheadend clean not XBMC I understand that would not be an issue.

I guess the reason is I don't like the idea of endless litter accumnulating in the tvheadend configs. If the 30 days is true then it's probably not worth chasing.

I'll attach the scripts but i have never had much luck with forum restrictions. I hope they go thru. I've taken no time to make them "pretty".

Cheers

Mark

( mk_mp4_tv.pl is the postprocessor run from tvheadend the other mk* are just more conversion/transcode examples )

I also feed tvheadend EPG data with a simple script running two commands from cron :
./mc2xml -c us -g <zipcode>
cat xmltv.xml | tv_sort --by-channel | socat - UNIX-CONNECT:/root/.hts/tvheadend/epggrab/xmltv.sock

some screenshots to tie it together.

• I check my ~/.hts/tvheadend/dvr/log directory and the oldest file is from October 12, about 30 days ago. So even when I manually delete the recordings, TvHeadend still deletes the log entries afterwards, which is a good thing!
• About htspdel: I just saw that it imports the tvh Python package.
  I guess it refers to this package
  I currently use the "packaged" version of TvHeadend 3.4.27 packaged for XBMCBuntu Frodo (Quantal). Is this Python API installed anywhere? How can I install it? Do I have to extract the source from the Git repository for that?
• I tried your script manually. I did not have ffmpeg installed so I installed it from the Ubuntu repositories:
  

  Unpacking ffmpeg (from .../ffmpeg_6%3a0.8.9-0ubuntu0.12.10.1_i386.deb) ...
  

  
  Then I got these errors:
  

  spontex@pchc:~/scripts/tvheadend$ ./mk_mkv_tv.pl -f CANAL+-Le-petit-journal.2013-11-09.07-30.mkv
  ./mk_mkv_tv.pl:18414:comskip 
  ./mk_mkv_tv.pl:18414:Logging to mk_mkv_tv.log with 1d archive interval
  CANAL+-Le-petit-journal.2013-11-09.07-30.mkv not in use .. start encode to CANAL+-Le-petit-journal.2013-11-09.07-30.mkv
  ffmpeg version 0.8.9-6:0.8.9-0ubuntu0.12.10.1, Copyright (c) 2000-2013 the Libav developers
    built on Nov  9 2013 19:15:53 with gcc 4.7.2
  *** THIS PROGRAM IS DEPRECATED ***
  This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
  [matroska,webm @ 0x9b5f720] Unknown/unsupported CodecID S_DVBSUB.
  [matroska,webm @ 0x9b5f720] max_analyze_duration reached
  [matroska,webm @ 0x9b5f720] Estimating duration from bitrate, this may be inaccurate
  Input #0, matroska,webm, from 'CANAL+-Le-petit-journal.2013-11-09.07-30.mkv':
    Metadata:
      title           : Le petit journal
      DATE_BROADCASTED: 2013-11-09 07:30:00
      ORIGINAL_MEDIA_TYPE: TV
      TVCHANNEL       : CANAL+
      SUMMARY         : Yann Barthès et son équipe dévoilent un décryptage décalé et humoristique de la communication des politiques, la vie des médias et des vedettes à travers des interviews et des reportages.
      SUMMARY-fre     : Yann Barthès et son équipe dévoilent un décryptage décalé et humoristique de la communication des politiques, la vie des médias et des vedettes à travers des interviews et des reportages.
    Duration: 00:35:00.68, start: 0.000000, bitrate: 16472 kb/s
      Stream #0.0(eng): Video: mpeg2video (Main), yuv420p, 720x576 [PAR 64:45 DAR 16:9], 15000 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
      Stream #0.1(fre): Audio: mp2, 48000 Hz, stereo, s16, 256 kb/s (default)
      Stream #0.2(eng): Audio: mp2, 48000 Hz, stereo, s16, 256 kb/s (default)
      Stream #0.3(fre): Audio: ac3, 48000 Hz, stereo, s16, 384 kb/s (default)
      Stream #0.4(eng): Audio: ac3, 48000 Hz, stereo, s16, 384 kb/s (default)
      Stream #0.5(qaa): Audio: mp2, 48000 Hz, stereo, s16, 192 kb/s (default)
      Stream #0.6(fre): Subtitle: [0][0][0][0] / 0x0000 (default)
      Stream #0.7(fre): Subtitle: [0][0][0][0] / 0x0000 (default)
      Stream #0.8(fre): Subtitle: [0][0][0][0] / 0x0000 (default)
  Unrecognized option 'c:v'
  Failed to set value 'libx264' for option 'c:v'

  
  I suspect a version mismatch. Do you use an older version of ffmpeg?
• Another question: After the conversion, does TvHeadend lose the link with its recording files? Are the converted recordings listed under XBMC > Live TV > Recordings?

Thanks!

I use git versions for ffmpeg/xbmc/tvheadend and some others. That is the library but good question I had to copy the lib to /usr/local/lib/py so it would run.

I also had to add the delete command to the library htsp.py from the manual pages sorry forgot about that:
mediacenter tvh # ls
dvb_charset_tables.py htsmsg.py htsmsg.pyc htsp.py htsp.pyc init.py init.pyc log.py log.pyc tsreader.py
mediacenter tvh # tail -10 htsp.py

1. Enable delete command
   def deleteDvrEntry( self, args = {} ):
   self.send('deleteDvrEntry', args)

1. ############################################################################
2. Editor Configuration #
3. vim:sts=2:ts=2:sw=2:et
4. ############################################################################
   mediacenter tvh # pwd
   /usr/local/lib/py/tvh
   mediacenter tvh #

Yes the link is lost because the name changes from mpg to mkv that is why I wanted to clean up. The script could easily relocate them but I have the current dirs defined in xbmc as TV type.

They do not show up in recordings because of this name change they show up in TV Shows since I do an auto refresh/update nightly and they are discovered and episode info is downloaded. ( If you can get the right sXXeYY into the name)

FYI I have a very new ffmpeg but the script is using handbrake for some time now the ffmpeg were commented out:

1. $enc_opts = '-map 0 -map -0:a:1 -c:v libx264 -s:v:0 640x480 -c:a:0 libfdk_aac -flags +qscale -global_quality 3 -afterburner 1 -ac:0 2 -c:s copy -crf 24';
2. $hbret = `/usr/bin/ffmpeg -i "${mfile}" -preset veryslow ${enc_opts} "${mediatitle}"`;
   $enc_opts = 'bframes=8:b-adapt=2:direct=auto:me=umh:merange=24:partitions=all:rc-lookahead=60:ref=16:subme=10:trellis=2:psy-rd=1.0,0.2';
   $hbret = `/usr/bin/HandBrakeCLI -i ${mfile} -I -v -e x264 -q 24 -w 640 --loose-anamorphic -a 1 -E faac -B 128 -f mkv -N eng -s 1,2 -m -x ${enc_opts} -o "${mediatitle}"`;

( so it is possible that the c:v syntax is no longer valid. handbrake runs the same libx264 but has easier subtitle inclusion )

Mark Kirkpatrick wrote:

They do not show up in recordings because of this name change they show up in TV Shows since I do an auto refresh/update nightly and they are discovered and episode info is downloaded. ( If you can get the right sXXeYY into the name)

Hi,
Yes but is it possible to replace the recordings? Cause I do not record only TV shows, also movies and various programs which won't have any metadata if I lose the TvHeadend EPG metadata.
Thanks

As in delete them then add in an new recording record or update an existing recording record ?? I have never looked at the api to see. I am certain you could search and replace the data file with he new filename but I always worried about syncing web server memory with behind the scene file system changes.

I was going to pass the metadata right to the script at one point since ffmpeg can store tags right in the files like iTunes ( atomicparsley ) and you could bundle it all in self-contained. Never got to it. Not hard to do since the meta is available in the call to the script.

I was just thinking about keeping the same file name. But your script do not produce .mkv files, do they?
Will the tags stored by ffmpeg be retrieved by XBMC? We will still miss some PVR information such as recording date, etc. This is why I would rather only replace the file with a smaller one or update the TVHeadend metadata.