Bug #4682
TVH not grouping recordings after reboot following filemoved API call
0%
Description
Introduction
This bug is associated with how recordings are shown in the kodi THEClient addon, in particular how they are grouped.
However, I think the groupings come from TVH server so I am reporting the bug here. If I am wrong then happy to report it to kodi instead.
Normally (if kodi is set to group items) it shows folders corresponding to each programme. Selecting a folder then shows all the episides available for that programme.
However, there appears to be a bug that results in the recordings being shown in a flat list (i.e. not grouped by folders) if the file is moved and there is then a reboot.
Steps to reproduce the bug
In this case TVH is configured with:
Recording system path: /storage/tvshows'
Full pathname specification format string: $t/$t$_e_$c_%F_%R$n.$x
Kodi & TVH are running on the same box (Raspberry PI3 with LibreElec 8.1.2; kodi 17.4 Git:7fc6da0; HTS Tvheadend 4.2.3-20 ~ LibreELEC Tvh-addon v8.2.112)
Step 1 - Have THR record a tv programme. TVH will store the file here: /storage/tvshows/#title#/#filename.ts#
At this point kodi shows the the recording grouped within a folder '#title#' (correct)
Step 2 - The file is then moved (overnight using a script with rsync) to /storage/nas/tv/#title#/#filename.ts#
Note the script also uses the API call dvr/entry/filemoved call to update TVH with the new location.
At this point everything is still working as expected, i.e. kodi shows the the recording grouped within a folder '#title#'
Step 3 - At some time later reboot
Following the reboot, the recordings are no longer grouped within kodi, instead appearing as a flat list.
Interestingly it doesn't need to be a complete reboot - just a restart of kodi (systemctl stop kodi) will cause this behavior - presumably because it re-queries TVH server when it restarts.
Conclusions
TVH presents the grouping to kodi. However the necessary information is not held in the pvr/log file to reconsruct this table correctly following a reboot / restart.
Possible Solutions?
Just my thoughts...
In the short term (v4.2 fix?) TVH could try to reconstruct the grouping from the pvr/log following reboot. I.e. in the above case the #title# is still part of the path. This will solve most cases where users are simply moving files from one location to another.
Longer term it might be better to have two fields within the dvr/log file rather than just filename. I.e. a field for 'system path' and a field for 'file path' File path would be used for grouping. E.g.
system path: /storage/tvshows
file path: /#title#/#filename.ts#
This would give maximum flexibility but need the api and dvr/log structure to be changed
I'd be happy to help with testing if I can be of assistance.
Thanks
History
Updated by Em Smith about 7 years ago
I don't use pvr.hts, however, Kodi (probably 15) used to always forget the "group" option in the menu on restart since it did not persist the option. So, is it the case that the moving of files is incidental? So even if you don't move a file, Kodi still forgets grouping on reboot?
I took a quick look at the pvr.hts code and I could see no logic for grouping. My understanding is that Kodi itself does that and not the pvr.hts or Tvheadend.
I also noticed that Kodi recordings can have an image path so I'll look in to patching tvheadend to send that in a few days since it looks like it currently just uses the channel icon.
I'd recommend restarting Kodi without moving files and see if the problem happens.
If it doesn't, then take a copy of the dvr/log for the file before the update and after the update and do a "diff" on the two files just to confirm only the filename is changing.
Updated by Jaroslav Kysela about 7 years ago
TVH does not do any file grouping (except the filename generation when the recordings starts). It looks like a Kodi issue.
Updated by Nic Butcher about 7 years ago
Thanks for the quick response Em.
I have checked a few things as requested:
Em Smith wrote:
I don't use pvr.hts, however, Kodi (probably 15) used to always forget the "group" option in the menu on restart since it did not persist the option.
I did a quick check and Kodi (17.4) does seem to persist the 'group items' option following a reboot.
So, is it the case that the moving of files is incidental? So even if you don't move a file, Kodi still forgets grouping on reboot?
If I reboot before moving the files then the grouping remains - i.e. this definitely seems to be something to do with moving the files.
I took a quick look at the pvr.hts code and I could see no logic for grouping. My understanding is that Kodi itself does that and not the pvr.hts or Tvheadend.
OK - but kodi must be basing this upon some data that is coming from pvr.hts (which gets data from TVH server). For some reason it is able to group before the file move but not after. Do you know what data is exposed to kodi from pvr.hts? For example - could it be something to do with the 'depth' of the path.
I also noticed that Kodi recordings can have an image path so I'll look in to patching tvheadend to send that in a few days since it looks like it currently just uses the channel icon.
Great - this will be a good improvement - programme specific icons are available from my grabber.
I'd recommend restarting Kodi without moving files and see if the problem happens.
As discussed above - the problem does not happen if the files are not moved
If it doesn't, then take a copy of the dvr/log for the file before the update and after the update and do a "diff" on the two files just to confirm only the filename is changing.
Not sure that we need to do this - it is sounding like this is not a TVH issue? Should I report it on the kodi forums instead? Although it would be useful to understand what info is passed to kodi first to help them understand?
Updated by Nic Butcher about 7 years ago
Further to my message above - see this :
https://forum.kodi.tv/showthread.php?tid=293283
It suggests that
Tvheadend sends the path without the Recording Profile directory in the payload for each recording
This suggests that TVH strips off the directory (in the example above this would remove '/storage/tvshows/' and then just leave the remaining path, i.e. /#title#/#filename.ts)
Is it possible that TVH is having a problem doing this once the path has changed?
Is there anyway of looking at what data is provided to kodi?
Updated by Em Smith about 7 years ago
It seems from the Kodi thread that the issue is solved by using the unstable branch and specifying a pathname of "KidsTV/$t/$t%F%R$-e$n.$x"?
To monitor messages you can run support/htspmon from the source tree.
The pre-move filename could check the path against the profile path and strip it, but post-move the path does not match the profile path so could not be stripped. So, basically, you're saying "these files in this config are stored under /a/rec", later you move your files under "/b/otherrec" but tvheadend doesn't know that so can't guess that before it would strip /a/rec and now it has to strip /b/otherrec. You'd need to change the config profile for the dvr entry to one that has that filename hierarchy set. I'm not sure if that's easy to do via some idnode callback.
I checked and that logic would appear to be what happens in tvh's htsp_build_dvrentry.
Updated by Nic Butcher about 7 years ago
Em Smith wrote:
It seems from the Kodi thread that the issue is solved by using the unstable branch and specifying a pathname of "KidsTV/$t/$t%F%R$-e$n.$x"?
I'm not sure what the unstable branch is but I'm using HTS Tvheadend 4.2.3-20 ~ LibreELEC Tvh-addon v8.2.112 which is somewhat newer than that discussion and does have the fullname specification. So I suspect that I'm on that version?
To monitor messages you can run support/htspmon from the source tree.
I don't know how to do this - sorry - I'm pretty new to all this. Happy to learn though if you point me in the right direction and if this is required.
The pre-move filename could check the path against the profile path and strip it, but post-move the path does not match the profile path so could not be stripped. So, basically, you're saying "these files in this config are stored under /a/rec", later you move your files under "/b/otherrec" but tvheadend doesn't know that so can't guess that before it would strip /a/rec and now it has to strip /b/otherrec. You'd need to change the config profile for the dvr entry to one that has that filename hierarchy set. I'm not sure if that's easy to do via some idnode callback.
I checked and that logic would appear to be what happens in tvh's htsp_build_dvrentry.
Yes - I suspected that to deal with all possible file moves the dvr/log files will have to include two fields - i.e. the source path (not exposed to kodi) and then the rest of the path and filename (that will be exposed to kodi) - see my original post (long term solution?). I'm guessing this would need more substantial changes to the dvr/log file structure and the associated APIs etc. but would be the most flexible.
In the short term would it be possible to improve the way TVH handles these sort of situations when reading the dvr/log file?
For example if it can't find the expected /a/rec/ to strip - it could strip off the same 'depth' (in this case 2). So if the new filename was /b/nas/tv/title/episode1.ts it would remove the first two, i.e. /b/nas/
This would be better than the current behaviour which strips off everything?
Another option would be to have a new option in TCH config where the user could specify an 'alternative path(s)' that TVH could try to strip when reading the dvr/log file.
Just ideas...
Updated by Em Smith about 7 years ago
OK, I did some testing on 4.3.
Since the changed filename is not under the config directory it is not sent (at all) for those files.
So Kodi appears to do two different types of grouping. Where a filename exists it groups by that. Where a filename does not exist then it leave the title in the list and so it appears ungrouped.
'method': 'dvrEntryUpdate', 'files': [],
So Kodi's grouping is by directory rather than by Title.
I don't think it's wise to assume that path components are the same on two separate mounts since frequently you might have different levels of hierarchy.
Why not just record direct to the NAS? Or mount the NAS underneath the config directory?
Perhaps Tvheadend should just reject a rename that goes beyond the config path boundary.
Updated by Mark Clarkstone about 7 years ago
Em Smith wrote:
OK, I did some testing on 4.3.
Since the changed filename is not under the config directory it is not sent (at all) for those files.
So Kodi appears to do two different types of grouping. Where a filename exists it groups by that. Where a filename does not exist then it leave the title in the list and so it appears ungrouped.
[...]
So Kodi's grouping is by directory rather than by Title.
I don't think it's wise to assume that path components are the same on two separate mounts since frequently you might have different levels of hierarchy.
Why not just record direct to the NAS? Or mount the NAS underneath the config directory?
Perhaps Tvheadend should just reject a rename that goes beyond the config path boundary.
I like that idea, not sure whether others will though!
Updated by Nic Butcher about 7 years ago
Em,
Thanks for digging into this - the description of kodi makes sense and I agree that it would probably not be sensible to assume the path components were the same - it could cause all sorts of issues in other cases!
For info: Originally I did record directly to the NAS but it turned out to be a pain because it stopped me rebooting or messing with the NAS (or the network) whilst a recording was in progress. Also recordings would occasionally fail which I think was due to network failures (wifi is a bit temperamental in our house - thick walls!)
Also means that I can use rsync across the network which is good for moving large files, tolerant of failures and can be bandwidth limited.
I never thought to mount the nas drive within the TVH config boundary - great idea, I'll give it a try. It'll just take me a little time to change the scripts etc. but I'll report back soon.
And if it works - much easier than some of the other ideas I was proposing!
Thanks again.
Updated by Em Smith about 7 years ago
For others, the idea would be to set your recording system path in Configuration->Recording->Recorder Profiles to "/Videos" (or wherever) and prefix the format string (in advanced) with "rec" such as "rec/$t-$n.$x". Mount the nas under "/Videos/nas". That way when you move your videos from "/Videos/rec/myrec.ts" to "/Videos/nas/myrec.ts", it will still all be under "/Videos" so should all work for Kodi grouping.
If wi-fi is bad, perhaps try powerline?
Mark Hunting
Yes, you're probably right. I thought maybe add a force flag for cases where people really wanted to move outside the profile, but then everyone would always set force=true.
Updated by Nic Butcher about 7 years ago
Em,
I changed the script to place the destination directory within the TVH config boundary as you suggested and this works fine - thanks for all your help. Kodi now has two groups (Recent and Stored) under which all the programmes are grouped by title.
I appreciate that you might want to have TVH reject a rename that goes beyond the config boundary but I would prefer you didn't - better to let people do what they want but just warn them of the consequences? I've added an item to forum on what I've done that will hopefully help others - http://tvheadend.org/boards/4/topics/29110
Thanks again.
Updated by Mario Fenech almost 4 years ago
I know this is an old discussion but I was trying to do my own script and got stuck with Kodi not showing recordings in groups like before...
What is discussed here works fine...you just have to live with another subdirectory in recordings directory...
I sometimes get even the source directory showing in recordings but that is only when a recording did not move properly to destination or a recording is in place..
Something I would like to add and I think it's important to note that when you set your own directory in auto recordings you have to add your recordings directory to path otherwise all your recordings end up in the home directory..
example if you set your recording path to "/b/nas" and the format string as "KidsTV/$t/$t%F%R$-e$n.$x" to get your recordings at "/b/nas/KidsTV/$t/$t%F%R$-e$n.$x" then when you set your own directories to your auto recordings it has to be nas/KidsTV not just KidsTV..
Hope this helps someone..