Project

General

Profile

Help with unusual setup needed - Two TH Servers sharing recordings but NO settings

Added by No Name about 6 years ago

Hello there,

I have a problem because I want to realize an extraordinary configuration. I have two satellite receivers (one Wetek Play and one Wetek Play2). With the Wetek Play I make recordings to a network drive. I would like to see these recordings on the Play2 in Kodi under -> TV -> recordings. I want to put the Play in the basement and this should only be responsible for the recordings from satellite. The Play2 should then stand in the living room and there I want to play the recordings directly from the network drive and watch liveTV. Both boxes mount the same recording directory from the network drive. Because of the very important WAF :D it would be great if the recordings could be displayed and played back under -> TV -> recordings (so network drive -> Play2 -> TV)). Does anyone know which files or directories I have to share between the two boxes?
I tried it with /dvr/log but the server gets autorec, finished recordings, ... This is not good because it will lead to collisions. I also tried it with a second client. There I see the recordings under -> TV -> recordings but the server on the Play 2 again gets too much. There is also the problem that if I want to play a recording he does it via the Play (so network drive -> Play -> Play2 -> TV) and not directly from the network drive.
I know it's a bit unusual but I hope I can do it with your help. For every tip I am very grateful.

Greetings,
count0

P.S.: I'm using LibreELEC 17.5 Krypton on both boxes with the latest version of TVHeadend available for both boxes (HTS Tvheadend 4.3-1288 ~ LibreELEC Tvh-addon v8.2.444).

P.P.S.: It is quite similar to the problem described here (https://tvheadend.org/boards/5/topics/29489). Only that I would like to have "Two TH Servers sharing recordings but NO settings". (In title added)


Replies (21)

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by jody gugelhop about 6 years ago

as you don't want to share settings why not edit the pvr menu in kodi that lists the recordings.
set up a share e.g. samba share or similar where all the recordings will be stored and edit/hack the PVR kodi menu to point to that share.
Alternatively, which you probably know you could simply add the recording directory share in kodi under Videos->files which is even less hassle but not sure if that's what you want...

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

Hello, there,

that I don't want to share the settings I didn't say but I just don't want to share all settings but only the needed ones.
I didn't quite understand what I meant with "Videos->Files". But that actually gave me the idea to try the library under "TV Shows".
I will try it the days. Thanks for the tip!

But if someone has an idea I am in for it.

Greetings,
count0

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

Your setup is not so "extraordinary", many people have multiple tvheadend servers and would like to share recordings. I have two servers, each at a different location. I am not in a big hurry for most of the recordings, so up until recently I just rsynced the files about once a week and then copied over the corresponding dvr/log files manually. Now I have setup a more automatic process.
This thread and Dave P's wiki were helpful.
[[https://tvheadend.org/boards/5/topics/28252]]
[[https://github.com/dave-p/TVH-API-docs/wiki/]] specifically: [[https://github.com/dave-p/TVH-API-docs/wiki/Dvr]]

First, I created a script to be run on the recording tvheadend server after each recording completes, here are probably the interesting parts for you:
/home/hts/export_json.sh

#!/bin/bash

file=$1
uuid=$2

printf "conf=" | cat - "/home/hts/.hts/tvheadend/dvr/log/$uuid" >"$file".import.json 


Then call it in with "Post-processor command:"
/home/hts/export_json.sh "%f" %U

This uses the uuid of the new recording to get the dvr/log file and prepends the needed "conf=" to it and saves it in the recording folder.

Then once a day a cron job rsyncs the recordings and json files to my other server. (after rsyncing I remove the local json file)

Another cron job on my receiving server runs a few hours later and "imports" the json files into tvheadend using the dvr/create web api.
For each json file it executes:

curl -q  --data @file.import.json 'http://user:pass@localhost:9981/api/dvr/entry/create'

Then, on success I remove the json file. (on error, I move it to a tmp folder to be examined manually later.)

For your setup, since you do not need to rsync files, your export script could include the curl command, connecting to the play2.

A few things to note:
1. Both my servers use the exact same path to the recordings directory, so I do not have to modify the filename. If your paths are different, you could use something like sed to fix the filename in the json file before importing.
2. I used to grep through all the dvr/log files to find the one with the filename (%f), but the brand new option (%U) makes it much easier.
(commit from sept. 26 [[https://github.com/tvheadend/tvheadend/commit/6e8c4691993e09d43a7e894f2a023bc390512022]] )
So, if you have an older version, you would have to resort back to grepping the dvr/log files to find the one you need.
something like:

logfile=$(grep -l "$file" /home/hts/.hts/tvheadend/dvr/log/*)
printf "conf=" | cat - "$logfile" >"$file".import.json

3. As noted in the wiki, the dvr/create function is there to create new scheduled recordings, using it this way is not supported and could change at any time!!!
4. I end up with a copy of the recording on each server, so later moving/deleting is not a problem. Since both of your servers will point to the same file, if you delete it on one, it will affect the other server...
5. While testing the dvr/create api I managed to crash my server many times with bad data, so beware. If I find some time this week, I will try to fix (hack...) the tvheadend code to make some better checks on the submitted data, or at least write a detailed bug report.

Again, big thanks to Dave Pickles for his wiki and "ullix tv" for his example!

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

Sorry, I forgot I had to add the "-F" option to grep to avoid special characters in the filename.

logfile=$(grep -F -l "$file" /home/hts/.hts/tvheadend/dvr/log/*)

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

Hello Joe User,

thank you for the detailed description. That's not so trivial but it's just the solution I was looking for. I will try to implement it in the coming days. I will certainly have one or the other question. If I do not get further I will announce it again here.

Thank you & greetings!
count0

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

Maybe all the explanation of my setup makes it look more complicated than it really is. :)

For you, just two steps:

On the wetek play, create script, not exactly sure of best location, but probably:

/storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/export_json.sh

#!/bin/bash

file=$1

# Find logfile for recording
logfile=$(grep -l "$file" /storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/dvr/log/*)

# Use logfile to create json data for import
printf "conf=" | cat - "$logfile" >"$file".import.json

# Create recording entry on play2
curl -q  --data @"$file".import.json 'http://user:pass@wetek_play2:9981/api/dvr/entry/create'

Then, in your recording profile set "Post-processor command:"

/storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/export_json.sh "%f" 

Hopefully the wetek supports the "-F" option for grep...

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

I can't believe it! That simple? That would be simply great. I will try it immediately. Tomorrow I can tell you if it worked or not.

Many, many thanks!

P.S.: I added the "-F" in the last version of you. :-) Only the question whether the UUID is no longer needed. "%U"?

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

It depends on your filenames - for me it sometimes worked and sometimes not without the "-F". Some versions of grep do not support the "-F" option.
The "%U" option is only available on the latest development version, which I am pretty sure you do not have on your weteks.

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

I'll try the "-F" option because grep supports it on Play1.

Thanks for your help. I will report tomorrow.

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

So far so good. I can now see the recording I made with the Play1 in the Tvheadend-WebIf of the Play2 but in Kodi under "TV -> recordings" is nothing.
Do you have an idea?

another problem has arisen. if the title of the recording contains blank spaces then it does not work either.

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

Sorry, I do not use kodi, so not sure why recordings are not showing up.

The quotes around the filename ( "$file" ) should account for spaces.

If you look at the .import.json files, do they contain data or just "conf=" ?

You can try these modifications then look at the log to see what is going on:

#!/bin/bash

# print trace of commands and variables
set -x

file=$1

# wait to make sure the log file is finished being updated before checking
sleep 5

# don't trace grep because it will echo every file in dvr/log directory
set +x  

# Find logfile for recording
logfile=$(grep -l "$file" /storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/dvr/log/*)

# trace again
set -x 

# Use logfile to create json data for import
printf "conf=" | cat - "$logfile" >"$file".import.json

#wait to make sure the network file is written
sleep 5

# Create recording entry on play2
curl -q  --data @"$file".import.json 'http://user:pass@wetek_play2:9981/api/dvr/entry/create'

I added the sleep commands because while I have a fast server and write to local drive, the weteks may be slower...

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

Hi, Joe User,

the script now runs like clockwork. Thanks a lot for that. Now I just have to find out what information the HTSP client needs to display the recordings under TV -> Recording. You have brought me however good piece further.

Greetings!
count0

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

This is old, but maybe is the problem?

[[https://tvheadend.org/boards/5/topics/21766]]

and the pages it links to:
[[https://github.com/kodi-pvr/pvr.hts/issues/311]]

error id kodi log??

ERROR: AddOnLog: Tvheadend HTSP Client: pvr.hts - malformed dvrEntryAdd/dvrEntryUpdate: unknown priority value 6

you can look in the .import.json files and see what the priority is set to

"pri": 6

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

I looked it up. The error message doesn't occur in my normal log. Do i have to switch something to debug mode?
i have seen that a few with "pri:6" the most are with "pri:2".

After further investigation I almost think that this is a rights problem. If I want to play a recording in the WebIf of the Play2 which the Play1 has made then I will be asked (with VLC) for username and password. I don't understand why but it is so. Unfortunately it doesn't work either. Funny.

I don't know why but I changed the "sync"-user and now I can play it at least without a password. Tonight I have to see if it finally appears in the list under TV -> Recordings.

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

Who is the owner of the recordings? You can see column in finished recordings.
If not using the same owner on play2, does the user must have "View All" permissions under "video recorder"?

Can you play recordings as an admin user?

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

See above. Changed to admin useer and now I can play the recordings.

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by No Name about 6 years ago

Hello Joe User,

it was the users rights. Now everything is running. I apologize for my stupidity. I don't know how I can thank you for your effort. If you feel like it & have time then I would like to optimize it for my ideas (now it works if both boxes are always on. My dream would be if I could turn the Play2 off and if I turned it on that the sync process would start automatically). But this is not so urgent.
Many, many thanks for your help!

Yours sincerely!
count0

P.S.: Many thanks also from my wife. :-)

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

If successful, the curl command will return the uuid of the newly created dvr_entry. You could check for this and if it has succeeded, you can either delete or rename the .import.json file (.import.json.done). Then on the play2, you could run some startup script that would look for any still existing .import.json files and use them with the curl command. (if the .import.json files still exist, it would mean they were not added yet.)
If you need help with the scripts, let me know...

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Em Smith about 6 years ago

My guess is that you could change the curl line to:

curl -q  --data @"$file".import.json 'http://user:pass@wetek_play2:9981/api/dvr/entry/create' && rm -f "$file.import.json" 

So after a successful import, the json file is no longer needed and can be removed.

Maybe also add the "--max-time 60" to the curl to timeout after 60 seconds in case you turn off the other machine at the wrong time.

So, then you need to find all the json files that exist (since they failed to import) and try to import them again:

#! /bin/sh
cd /storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/dvr/log/ || exit 1
for file in *.import.json
do
  curl -q --max-time 60  --data @"$file" 'http://user:pass@wetek_play2:9981/api/dvr/entry/create' && rm -f "$file" 
done

You then need to run that script somehow. If you do frequent recordings, you could just add it to the end of your existing post-record script. Effectively, once a recording is completed it will try and resend failed entries to the other play if it can. But, you might end up with two post-recordings happening at the same time (not sure if tvh handles two recordings created with same uuid), so best to try the alternative approach below first.

Perhaps your other box (the one your turn off) has cron? Some of these boxes have limited software, so I don't know.

Try: "crontab -e", if it pops you in to an editor then add:

@reboot root sleep 120; /path/to/retry/file/somewhere.sh > /tmp/tvhretry.log 2>&1

That should run the script at startup, and output to a file in /tmp so you know if it ran at all. We need the sleep to give time for tvheadend to start.

However, if you run it on the box that is turned off, that means the "cd /storage..." needs to be changed to go in to the mount of the other box (the one that is always on and has the json files ready to import). So a "cd /mnt/play_server/dvr/log", etc.

Finally, before running the first time, you need to remove the json files from your old imports (otherwise they'd import again as failed imports).

cd /storage/.kodi/userdata/addon_data/service.multimedia.tvheadend/dvr/log/
rm *.import.json

(All this is untested, but should give you an idea).

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Em Smith about 6 years ago

If you can't run the re-import on your box that you turn off (perhaps it has no easy access to the other box's dvr/log), then you could run it on the box you leave on using crontab to perhaps run every hour by replacing the "@reboot" with "@hourly".

RE: Help with unusual setup needed - Two TH Servers sharing recordings but NO settings - Added by Joe User about 6 years ago

Em Smith wrote:

My guess is that you could change the curl line to:

I found that curl needs the "-f" option to get more reliable return codes... (i.e. unauthorized returns 0)

So after a successful import, the json file is no longer needed and can be removed.

Yes, eventually but for testing it may be better to save for awhile, I move them to a separate dir and mark them as done.

Maybe also add the "--max-time 60" to the curl to timeout after 60 seconds in case you turn off the other machine at the wrong time.

Yes, even 30 would probably be enough for a local net.

So, then you need to find all the json files that exist (since they failed to import) and try to import them again:

Instead of retrying from the recording play, I would rather just run from the play2 when it starts, see below.

Finally, before running the first time, you need to remove the json files from your old imports (otherwise they'd import again as failed imports).

Actually it is worse than that, they would import just fine since there is no duplicate checking. Then you have the problem of 2(or more) duplicate entries pointing to the same recording file. And if you remove one, the recording gets removed (deleted!!!).... I found this out with my trial and error testing :) At that point, you could move the duplicate entries to failed recordings, or you would have to stop tvheadend and remove all but one of the duplicate dvr/log files, then restart tvheadend. (helpful to have the uuids - again see below.)

In my case, I use a fixed "json" directory to put all the import related files. (/record/tvh/json)

Then my export script creates the .import.json files there and the rsync copies them so my other server where my import script sees them.

In your case, you can make a "json" directory in the base directory of your recordings, then you can modify the end of your export script to something like:

# Use logfile to create json data for import

dir="/path/to/recordings/json/"  # note important trailing /  
filename=${file##*/}  # get just filename

printf "conf=" | cat - "$logfile" > "$dir$filename".import.json

#wait to make sure the network file is written
sleep 5

# Create recording entry on play2
curl -q  --data @"$dir$filename".import.json 'http://user:pass@wetek_play2:9981/api/dvr/entry/create' > "$dir$filename".reply

if grep -q uuid "$dir$filename".reply ;then
        echo "worked" 
        mv "$dir$filename".import.json "$dir"$filename"import.json.done
else
        echo "failed" 
fi


If successful, in the .reply file you will have the uuid of the NEWLY created entry on the plays (will be different than the uuid of the original recording on the play) (Also, if accidentally run again, it will have a new uuid...)

Looks like you can have an "autostart.sh" script [[https://www.wetekforums.com/v/discussion/20882/startup-boot-script]]

In that script add a line at the end with something like:

nohup /storage/.config/import.sh &

And make an import.sh script in /storage/.config:

#! /bin/bash

# wait to give time for tvheadend to start.
sleep 180  # 3min - not sure how much time is really needed???

cd /path/to/recordings/json

for json_file in *.import.json 
do
     curl -q  --silent --data @"$json_filename" 'http://user:pass@wetek_play2:9981/api/dvr/entry/create' > "$json_filename".reply

     if grep -q uuid "$json_filename".reply ;then
        #echo "worked" 
        mv "$json_file" "$json_file".done
     else
        #echo "failed" 
        mv "$json_file" "$json_file".failed   #so not tried again
     fi
done

Last part is optional, but if it fails again, there may be something wrong with it so best not to try again.

Again, untested as my scripts are different due to different setup.
Also, I wrote post in pieces as I was working on a few different things, so quite possibly there are some errors. :)
But in addition to Em Smith's post it should be a good start.

    (1-21/21)