Project

General

Profile

Tvheadend 4.0 how to import existing recordings from version 3.4

Added by Xavier S over 8 years ago

Hello,
I have been using tvh 3.4.27-6 on my Synology NAS for a while (running DSM 6.0), and have accumulated a lot of recordings.
I just upgraded to tvh 4.0.8-1, which I had to entirely reconfigure. Unfortunately I was not able to recover my previous TV recordings.
Here is what I did:
1- backup my existing tvheadend/var directory
2- uninstall tvh 3.4.27-6
3- install tvh 4.0.8-1
4- configure tvh 4.0.8-1

I then made a sample recording with my freshly installed tvh 4.0.8, just to see that the structure of the recording logs in tvheadend-4.0/var/dvr/log/ was much different from the old ones in tvheadend/var/dvr/log/. Just to make sure, I copied and old log from my var/dvr/log/ backup into tvheadend-4.0/var/dvr/log/ and restarted Tvheadend, but as expected only the new recording was listed in the Recordings panel, not the old one.

Hence my question: Is there a method or tool to import all previous recordings from Tvheadend version 3.4 into the 4.0 database?

Thanks for help !


Replies (11)

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Jonathan Thomson over 8 years ago

Someone may well know the specific answer but from what I've read the changes between v3 and v4 were so great that it was deemed necessary to have a brand new format for the configuration and I think I read that the two are not compatible with each other.

That being said, they are simply text files, so if you can work out the differences between a v3 log file and a v4 log file you might be able to get TVHeadend to recognise them - the most important part is obviously the file path, converting the metadata may be more involved.

Just for curiosity, can you post (to Pastebin or similar) a v3 log and a v4 log?

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Xavier S over 8 years ago

V3 log files are named by mere numbers, in the sequence where they are produced (i.e. 1, 2, 3, and so on). Here is one of them (file "42"):

{
    "channel": "ARTE HD",
    "start": 1438378200,
    "stop": 1438381500,
    "start_extra": 2,
    "stop_extra": 10,
    "config_name": "",
    "creator": "XBMC",
    "filename": "/volume4/Enregistrements/Uranium, si puissant et si dangereux ?-1.mkv",
    "title": {
        "und": "Uranium, si puissant et si dangereux ?" 
    },
    "description": {
        "und": "Blah blah blah " 
    },
    "pri": "normal",
    "noresched": 0,
    "contenttype": 35,
    "broadcast": 25493,
    "container": 1
}

V4 logs get randomly generated filenames. For example here are the contents of log file "5199e460d1985a5de62c0dcf56fc1009":

{
    "start": 1470434700,
    "start_extra": 2,
    "stop": 1470441600,
    "stop_extra": 10,
    "channel": "900749ebe81f94b2267cdc9696edb21f",
    "channelname": "W9",
    "title": {
        "fre": "Enquête d'action" 
    },
    "subtitle": {
        "fre": "Autoroutes : aux frontières de tous les trafics" 
    },
    "description": {
        "fre": "Blah blah blah" 
    },
    "pri": 2,
    "retention": 99,
    "container": -1,
    "config_name": "24b1ef28e7b6838b49c07bf1f6fe8f1a",
    "owner": "XBMC",
    "creator": "XBMC",
    "filename": "/volume4/Enregistrements/Enquête d'action.ts",
    "errorcode": 0,
    "errors": 0,
    "data_errors": 0,
    "dvb_eid": 0,
    "noresched": false,
    "autorec": "",
    "timerec": "",
    "content_type": 8,
    "broadcast": 5497
}

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Doctor Eggs almost 8 years ago

Were you ever able to figure this out? I am looking to import a bunch of recordings but can't seem to figure it out.

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Xavier S almost 8 years ago

Yes, here is what I did:

1- I batch edited the contents of all my v3 recording logs to make them look like v4 recordings as much as possible. As I was due to do this only once, I didn't bother to write a script for that. I used sed from the command line, once for each tag that had to be adjusted in the logs. The "config name" must be the same for all logs: it is the name of the file that you find in your tvheadend-4.0/var/dvr/config/ directory. The most cumbersome part of this process is about the channel names. These used to appear after the "channel" tag in the v3 logs, and now come after the "channelname" tag. The "channel" tag must be followed by the channel's config file name that you find in tvheadend-4.0/var/channel/config/. As these filenames are md5 hashes hence do not reference the channel's names, you have to cat them all to see which is which. From there on you can either make a script to associate each channel name to its config filename and batch edit the contents of your recording log files, or do it by hand and copy/paste what is needed in your sed command line to edit the "channel" - "channelname" thing in your log files, once for each different channel that you have in your recordings.

2- Once all your log files have been properly modified, you have to rename them all with their md5 hash. Once again there are different ways to do this. I moved my log files from my NAS to my Mac, batch renamed them in Terminal with md5 and put them back in place in my NAS.

3- Restart Tvheadend and voilà, it worked for me and I could enjoy all my recordings!

In the recording logs you have to pay special attention to the "retention" tag that appeared in v4. It specifies the number of days that each particular log file will be retained by Tvheadend after the recording date. Once this time has elapsed Tvh deletes the log file. The recording is still there, but it does not appear in your frontend listings any more (Kodi as far as I am concerned). As you have to create this tag yourself while converting your logs, do not hesitate to put a value large enough to suit your needs. I don't know what is the maximum value that Tvh supports; I put a value of 10000 and Tvh didn't complain. I don't know if there is a value that means "forever", but the main problem actually comes from my Kodi's PVR client, which is not able to specify a max. retention time greater than 1 year for the new recordings. If you face the same problem and want to keep your new recordings for longer than 1 year, this means that you'll have to remember to edit your new logs at least once a year to give them a longer retention time. Not very user-friendly...

Hope this helps, good luck!

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Doctor Eggs almost 8 years ago

Wow. This is great information. I'll be working on this this weekend but thanks for getting back so quickly.

For #2, does it matter what the log file was named before I rename them all with their md5 hash? I know that V3 numbered them in order (1,2,3, etc.), can I just take the md5 hash of 1, then 2, then 3? I have other recordings from a different PVR that I would like to put in as well. Would I just md5 hash the next available number?

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Xavier S almost 8 years ago

No, it doesn't matter, as the md5 hash is just about the file contents, not the original filename. So you can rename them in any order and at any time you want.

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Doctor Eggs almost 8 years ago

I guess I'm confused with what I need to hash to get the filename.

From your example above, how did you get "5199e460d1985a5de62c0dcf56fc1009" from:

{
    "start": 1470434700,
    "start_extra": 2,
    "stop": 1470441600,
    "stop_extra": 10,
    "channel": "900749ebe81f94b2267cdc9696edb21f",
    "channelname": "W9",
    "title": {
        "fre": "Enquête d'action" 
    },
    "subtitle": {
        "fre": "Autoroutes : aux frontières de tous les trafics" 
    },
    "description": {
        "fre": "Blah blah blah" 
    },
    "pri": 2,
    "retention": 99,
    "container": -1,
    "config_name": "24b1ef28e7b6838b49c07bf1f6fe8f1a",
    "owner": "XBMC",
    "creator": "XBMC",
    "filename": "/volume4/Enregistrements/Enquête d'action.ts",
    "errorcode": 0,
    "errors": 0,
    "data_errors": 0,
    "dvb_eid": 0,
    "noresched": false,
    "autorec": "",
    "timerec": "",
    "content_type": 8,
    "broadcast": 5497
}

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Xavier S almost 8 years ago

You can't take my example "as is", because if you look at the "description" carefully you'll notice that I replaced the actual (and very long) original description with "Blah blah blah" before posting. And by the time I opened this thread I didn't even know that the filename was an md5 hash, which is why I originally asked for help...

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Hissing Shark almost 8 years ago

I'm trying to do the same thing and I too can't seem to get past the filenames stage.

For example, I have a dvr/log called b4b6e0c49dbdbb1f0015b1001a43ef56 containing:

{
        "enabled": true,
        "start": 1477704900,
        "start_extra": 0,
        "stop": 1477708200,
        "stop_extra": 0,
        "channel": "f2064a5b472cebc2d95d1292b67f1fc7",
        "channelname": "Channel 4 HD",
        "title": {
                "eng": "Humans" 
        },
        "subtitle": {
                "eng": "Rejected and alone, Karen sets out to George's house to find Niska. Meanwhile Pete is determined to find out who Karen really is. The hunted Synths take refuge at the Hawkins. (S1 Ep7/8)  [AD,S,SL]" 
        },
        "description": {
                "eng": "Rejected and alone, Karen sets out to George's house to find Niska. Meanwhile Pete is determined to find out who Karen really is. The hunted Synths take refuge at the Hawkins. (S1 Ep7/8)  [AD,S,SL]" 
        },
        "pri": 2,
        "retention": 0,
        "removal": 0,
        "config_name": "6c91b0f9d5ee1bc37b82b0a0b8c47061",
        "owner": "admin",
        "creator": "admin",
        "errorcode": 0,
        "errors": 0,
        "data_errors": 0,
        "dvb_eid": 0,
        "noresched": true,
        "norerecord": false,
        "autorec": "9549fc0eb9ae11a870f317b6b8c3bc56",
        "timerec": "",
        "parent": "",
        "child": "",
        "content_type": 0,
        "broadcast": 0,
        "comment": "Auto recording",
        "files": [
                {
                        "filename": "/mnt/nas/pvr/Humans/Humans-4.ts",
                        "info": [
                                {
                                        "type": "H264",
                                        "width": 0,
                                        "height": 0,
                                        "duration": 0,
                                        "aspect_num": 0,
                                        "aspect_den": 0
                                },
                                {
                                        "type": "AAC",
                                        "language": "eng",
                                        "audio_type": 0
                                },
                                {
                                        "type": "AAC",
                                        "language": "eng",
                                        "audio_type": 3
                                },
                                {
                                        "type": "DVBSUB",
                                        "language": "eng",
                                        "composition_id": 1,
                                        "ancillary_id": 1
                                }
                        ],
                        "start": 1477704900,
                        "stop": 1477708200
                }
        ]
}

But md5sum b4b6e0c49dbdbb1f0015b1001a43ef56 yields:
8170d3f849672275c78ce95c8d55c252

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Xavier S almost 8 years ago

Right now I can't answer on your specific example, however I suggest you try converting an old log and see the result first. It may be after all that Tvh is "tolerant" about the accuracy of the md5 hash (meaning it doesn't care at all ?). I'm not sure of this but what leads me to this conclusion is that I recently modified the retention times in several of my log files, which of course changed their md5 signatures; and although I didn't change their filenames the modified logs were recognized by Tvh as before.

RE: Tvheadend 4.0 how to import existing recordings from version 3.4 - Added by Hissing Shark almost 8 years ago

Indeed, it's a thought that perhaps the file has been modified since it was originally named. The use of a hash may only serve to provide an initial unique numerical filename, rather than verify its integrity.
The source would reveal all, but I've get to get to that part.

    (1-11/11)