Tvheadend

Joe User wrote:

Since many people would like to be able to read (and often modify) the config files, it would be nice if compression would be optional.
[[http://tvheadend.org/boards/13/topics/13421?r=27033#message-27033]]

I think compression should be by-default as it reduces start-up time, but a config/cmd line option to disable it would be nice.

What I'd like to see though is these config files completely gone & SQLite3 used.

Another option would be to add an export/import option for the config files. (I know it had been requested before.) But more specifically, the export should be a tar/zip of uncrompessd (and not banary json) files which could be modified and imported back.

SQLite3 would be nice because it would also be very useful for people to use sql statements to do searches and bulk update/changes to the database(config).

Mark Clarkstone wrote:

I think compression should be by-default as it reduces start-up time, but a config/cmd line option to disable it would be nice.

Two times "-1" from me. Why make it needlessly complicated?! Just because the systemd-locusts seems to be hell-bent to break every unix-concept that worked for 40+ years, does not mean we would want the same for tvheadend. I haven't seen a corrupted on-disk-text-file since the nineties, but databases? Even today it happens every now and then, that kodi b0rkes its database-files (which are not used for configuration!), that would be a lot of fun on tvheadend...

If you want an optional compression (for a few megabytes of config-files, that every 20 year old disk reads in under a second), fine, but default-on, oh hell no!

"Leave ... britney ...errr, myconfigfiles ... alone!"

my 2 cents...

/hp

Hanspeter Müller wrote:

Mark Clarkstone wrote:

I think compression should be by-default as it reduces start-up time, but a config/cmd line option to disable it would be nice.

Two times "-1" from me. Why make it needlessly complicated?!

It's not about making it "needlessly complicated" it's to help people who have loads of config files, I myself have experienced it with an old config of mine, not everyone is storing configs on fast drives like yourself.

Just because the systemd-locusts seems to be hell-bent to break every unix-concept that worked for 40+ years, does not mean we would want the same for tvheadend.

I'm not sure why you've mentioned this here, I'm honestly not a fan of systemd myself, but it's here to stay so may as well get used to it, and yes, I have removed it from a few of my systems..

I haven't seen a corrupted on-disk-text-file since the nineties, but databases? Even today it happens every now and then, that kodi b0rkes its database-files (which are not used for configuration!), that would be a lot of fun on tvheadend...

If you want an optional compression (for a few megabytes of config-files, that every 20 year old disk reads in under a second), fine, but default-on, oh hell no!

A few megabytes, really? I've seen one config folder [without backups] hit 1GB before, on a very old config.. I think your objection is to the fact that it makes editing the config files harder more than anything, which you have every right to.

"Leave ... britney ...errr, myconfigfiles ... alone!"

Chill.. Nothing is set in stone. There are many options to consider, for example.

• Plain text/ini - remember those?
• XML.
• JSON (as it is currently).
• SQL
• Something else that I can't think of..

my 2 cents...

/hp

I think one solution would be to have the best of both worlds..

• Import/Export json?
• Import/Export SQLite3?
• Something else? ideas people?!

As far as the default for compression option, I really don't care as long as there is an option. :)

While plain config files are simplest, they become not so simple when it comes to complex configurations. You have channels which can be connected to multiple services which are connected to muxes which are connected to networks which can be connected to multiple tuners. A relational database is far better suited to storing that kind of information.

As far as corruption, modifying a database which has constraints is far safer than modifying multiple inter-related text files.

Granted most users are probably more adept at using find/grep/sed/awk than sql, but learning is fun. :)

One downside of modifying config files outside of Tvheadend is having to stop and restart for changes to be seen. Having an export/import function would help with that. And if there is a good export/import mechanism, then the internal way Tvheadend stores the config can be based more on performance than having to provide human readability.

Another thought is that having an export function can also be helpful when people have problems and need to post their config here for support. Currently people end up having to do a lot of screenshots. Of course the export would have to somehow protect any sensitive information the user might have. Maybe a selection of which parts of the config to export???

Lastly, I have over 3000 recordings - certainly reading that info from a database file would be far faster than reading 3000 individual log files.

Tvheadend runs on many different platforms and for most, the cpu overhead is trivial compared to the file reading time. Consider a device which uses an SD card...

Those files you are referring to are 290 bytes to 2.5K ... hardly a reason to compress them, especially when they are all on their own and not bundled.
The only large amounts of file are the muxes. Here is an ls:

/home/hts/.hts/tvheadend/input/dvb/networks/c16ed10eef0a2ffbc6b145a94f5ec4ad/muxes la
total 608K
drwx------ 2 hts  12K Jul 25 23:21 ./
drwx------ 3 hts 4.0K Jul 25 23:21 ../
-rw------- 1 hts 2.1K Jul 25 23:20 01a147faa3412e038133df4de62a7833
-rw------- 1 hts  286 Jul 25 17:33 05236414a40516124cd255b7c9b6dbac
-rw------- 1 hts  291 Jul 25 17:33 0b0659b9edb27679160cde8790798843
-rw------- 1 hts  291 Jul 25 17:33 0c6e7361531d71f4904b0bd1c873965f
-rw------- 1 hts 2.4K Jul 25 23:19 0fa4cdd8bba1e57d46afc9980fec1ba1
-rw------- 1 hts 3.1K Jul 25 23:19 0fcf95e48dfcda8b1d3ce42e7c07a946
-rw------- 1 hts  292 Jul 25 17:33 123f10f83c4998a329a37168a2715e15
-rw------- 1 hts  298 Jul 25 23:15 12a0baeddcffc032888dc7c221b9a5d4
-rw------- 1 hts  289 Jul 25 17:33 12b041299686b3424dd878cb889b4260
-rw------- 1 hts  293 Jul 25 17:33 14a4fb23228548a7da9ed7aae274c136
-rw------- 1 hts  294 Jul 25 17:32 17e6acd1f5b79f9ac393d35f5c772559
-rw------- 1 hts 2.5K Jul 25 23:20 1815f078f25894c91b8a3f87f8407651
-rw------- 1 hts  296 Jul 25 23:15 19b3f3fc566fbdc49658ca5fd73714dd
-rw------- 1 hts  292 Jul 25 17:33 1d2746daa79075f89d581c41c370b961
-rw------- 1 hts 2.8K Jul 25 23:21 1fbdc76acdc772435f8cf220aa7d071c
-rw------- 1 hts 2.6K Jul 25 23:18 21981970351481e243f84fa6dd77e3a9
-rw------- 1 hts  297 Jul 25 23:15 26aef6da72a42a23fb3d133bf4dc843f
-rw------- 1 hts  287 Jul 25 17:33 276844f4c915e1dfa63f9b01a98a7aed
-rw------- 1 hts  291 Jul 25 17:33 2b61934a7ddeec3e828904fd50278b8a
-rw------- 1 hts  296 Jul 25 23:15 2c572ad81417b38f1740a6b2723df553
-rw------- 1 hts 4.1K Jul 25 23:20 2c89d60cf97ee5ca692b183bcb84640d
-rw------- 1 hts  293 Jul 25 17:30 2d809fc219c0d26af7e3a2f06f838562
-rw------- 1 hts 2.4K Jul 25 23:21 2dc3aa43b87acf674c6830ad4a7a5505
-rw------- 1 hts 2.4K Jul 25 23:21 304685e52f4106bd74cdf37d5042b982
-rw------- 1 hts  292 Jul 25 17:33 3053aa7b4f6568c8061c15dcb0fa8a1f
-rw------- 1 hts  292 Jul 25 17:33 323a64b639462deeafda408d298c85a8
-rw------- 1 hts 4.1K Jul 25 23:21 337c0ccf2f7794f149bb870c4a7223eb
-rw------- 1 hts 2.7K Jul 25 23:19 39dd6c47b7fb79c8968dce45c07b3a6c
-rw------- 1 hts  294 Jul 25 23:15 3f517886a2ddc2de2abb2c8ba411c8a4
-rw------- 1 hts 3.2K Jul 25 23:21 3f538bb415055c9a7336acac3e2eca98
-rw------- 1 hts  293 Jul 25 17:31 40613bf0e41d995bd92335298930af24
-rw------- 1 hts  289 Jul 25 23:15 40aaa317cd2a4997285682fca9e9db35
-rw------- 1 hts  288 Jul 25 17:33 4362ed6d88c0ec4c994fa944d6823c4
.....................

You think that's reason enough to compress them and impose a lot of complexity when it comes to configuration?

The argument here is that the SD card reading of each of those files would be greatly impaired if they where not gzipped?

Talk about premature optimization, and for whom? The 0.000001% running from an SD card.

Anyhow, due to this design optimization decision it's costed me about 2 days of work. Now multiply that with a thousand other people running tvheadend on various system.

You'll come to the realization that the amount of time spent on circumventing this could have been better used serving humanity, building schools for african kids, or volunteer at a homeless shelter.

Yes, all for the sd card users.

Sorry for sounding arrogant, but this is the internet and what better way to argue your case then harsh language. Especially when there is still a guy on the issue continuing to argue for it's usefulness while killing it for the rest of us looking for using this in a professional environment rather on the memory card because his case is so important, and even if that where the case,

there is still no reason to compress them, because the files are fucking 240 bytes each. Maybe it was useful in 1995 when tvheadend used to run on floppy disks.

*Sorry for language, i just like to argue this way and I have nothing better to do than complain right now. *

Also, I don't think introducing additional complexity by moving to SQLlite is necessary either as that again introduces complexity when it comes to generating the configuration files, and really not needed since the amount of data is usually relatively small, even for a complex use of up to 10 networks and 20 000 files.

Rather than generating a file for each mux, you should consider generating a large one per network instead. At most that will be 4000 entries and when adding you could simply just add a line to the bottom of the file. When editing you could target that exact line by having a reference to the linenumber in the UI. When removing you could just simply replace it with an empty line, or at some point like reading all of the muxes on refresh the config file could be read and a new one while reading without all the empty newlines if that bothers someone.

Also, as mentioned in the thread: https://tvheadend.org/boards/5/topics/27367?r=27436

One folder containing configuration files should be created, and they should all lie there, rather than spread across various directories. The tvheadend hts folder is so small anyway and just introduces a lot of complexity with regards to figuring out what is a config file and what is not. /configs/{networks.json, muxes.json, tuners.json, settings.json, epg.json} and these can all reference each other within.

As the main developer, I don't like that users touch the config files. The files should be modified only if tvh is not running.

The json API (or HTSP api method) should be used to modify the tvh's database.

If you like to turn off compression anyway, it's one line change:

diff --git a/src/settings.c b/src/settings.c
index 3718c48..1ae9fc4 100644
--- a/src/settings.c
+++ b/src/settings.c
@@ -158,7 +158,7 @@ hts_settings_save(htsmsg_t *record, const char *pathfmt, ...)
   }

   /* Store data */
-#if ENABLE_ZLIB
+#if ENABLE_ZLIB && 0
   pack = strstr(path, "/muxes/") != NULL && /* ugly, redesign API */
          strstr(path, "/networks/") != NULL &&
          strstr(path, "/input/") != NULL;

Peter Wislander wrote:

.. snip ..

Those files you are referring to are 290 bytes to 2.5K ... hardly a reason to compress them, especially when they are all on their own and not bundled.
The only large amounts of file are the muxes. Here is an ls:

Who's talking about an SD card? I certainly wasn't & I don't see it mentioned anywhere in this issue? Posting an example of your config only explains your use case & not everyone else's.

The argument here is that the SD card reading of each of those files would be greatly impaired if they where not gzipped?

Again, I don't know where you're getting SD card from :s.

Talk about premature optimization, and for whom? The 0.000001% running from an SD card.

What's with the percentage, can I please see your survey data? I hate percentages with a passion as they can be easily skewed, or like in this case totally made up.

Anyhow, due to this design optimization decision it's costed me about 2 days of work. Now multiply that with a thousand other people running tvheadend on various system.

You'll come to the realization that the amount of time spent on circumventing this could have been better used serving humanity, building schools for african kids, or volunteer at a homeless shelter.

This was a really bad example.. You talk about how it's cost you 2 days work, then go on to say how the time could've been spent helping others in need? Wouldn't it have made more sense to help those people first? Or maybe that's just me?

Sorry for sounding arrogant, but this is the internet and what better way to argue your case then harsh language. Especially when there is still a guy on the issue continuing to argue for it's usefulness while killing it for the rest of us looking for using this in a professional environment rather on the memory card because his case is so important, and even if that where the case,

Who is arguing, and again with the SD card? Where is this coming from?

there is still no reason to compress them, because the files are fucking 240 bytes each. Maybe it was useful in 1995 when tvheadend used to run on floppy disks.

Chill! Please?

The problem comes down to this..

1. Should the config files be compressed by default, for the average user? I'm talking about those who don't fiddle with the config files.
2. How should they be stored? What format?
3. Those who do alter the configs want an easy way to export/import.

For me personally? Honestly, I'm not for or against compression, as long as it works!

1. Yes, but with the option to disable it. If the option is passed tvh should decompress/compress when asked. Again, remember this is for the average user!
2. I'd prefer SQLite3, but it's not just about me. While I don't like plain text configs, many do & I have no problem with that.
3. There needs to be an easy way to import/export configs in a number of formats, but which should be default?

Maybe we should have a survey? :)

This issue is to be merge with import/export branch of tickets ;)

Why not to have option --disable-gzip-config in compilation?
Any one who want to test to compile binary without config compression.

Jaroslav Kysela wrote:

As the main developer, I don't like that users touch the config files. The files should be modified only if tvh is not running.

Yes, it should not be the norm, but certainly you must recognize that there are times when it is far easier and faster to modify the config files.

The json API (or HTSP api method) should be used to modify the tvh's database.

Again, I agree, but until the API is better documented, users will use the more simple method of working directly with the config files.

Mark Clarkstone wrote:

Who's talking about an SD card? I certainly wasn't & I don't see it mentioned anywhere in this issue?

I mentioned it in another thread where config file editing was discussed.

Mark pretty much covered everything well, I will just add:

Peter Wislander wrote:

Sorry for sounding arrogant, but this is the internet and what better way to argue your case then harsh language.

Hopefully your attitude will change one day when you learn that the internet is just like real life - the more mature you act, the more seriously you will be taken. In fact it is far easier to ignore people's rants on the internet than in real life. While it may feel therapeutic to you, you should at least realize that it hurts you in the long run.

For the near term, as far as compressing config files, I would be happy with just a build option to enable/disable. It would be easier than having to modify code after each pull. I even don't care which is the default - I would leave that decision to someone who knows better about what the average user needs.

For the long term I also agree a database approach would be best. Saying it is too complex ignores the fact that the configuration of Tvheadend IS complex. Arguing for plain text files and then complaining about the hash id's is a bit hypocritical. Plain text config files are fine most of the time, but when it comes to configs with multiple inter-related settings, a database actually makes things simpler, not more complex.

Mark Clarkstone wrote:

1. Should the config files be compressed by default, for the average user? I'm talking about those who don't fiddle with the config files.

"No. Under no circumstances this shall be a default." Non-negotiable, punishable by forking ;)

I totally understand Jaroslavs point-of-view as a dev that he does not like users tinkering with config-files, but this is not the normal case, i only do this, when something goes really wrong, with broken git-commits or just SNAFU (which happened two or three times last year, in all cases i would have been majorly screwed without the ability to grep/sed/etc and would have to restore a multi-month-old-backup)

1. How should they be stored? What format?

"Same as now." I still see no reason for compression except to make it more complicated, but an OPTION (compile-time or startoption) for experts to ENABLE compression would be fine, as long as it defaults to OFF.

THIS is exactly why i referenced systemd a posting earlier, they have an obsessive compulsion to get rid of every configfile and move to their view of a statleless system with a static /etc, how the entire computerworld should work in their view. Stop it!

Again, remember this is for the average user!

What's this average user you're keep referencing?! You're talking about a system to view and record tv-programs, which produces 5-10 GIGABYTES per hour on a single Cable-HD-Channel, if your storage is good enough for that, it's also good enough to read a few kilobytes on startup!

My oldest SATA Drive that is still alive (a Samsung 160GB drive, should be about 15 Years old by now!) does 80MB/s, so assuming that Peter is wrong and you're not talking about SD-Cards, what ARE you using that seems to suck trump so hard, that you want to compress config-files in a range from 10 to 3000 bytes each?! If it where the case that this is one bigass json/xml file, compression might actually be usefull, but compressing files that small only produces overhead, with the additional downside of deniing the average-user™ editing access, when he REALLY needs it.

Also: If you for-whatever-reason, you want this, use btrfs, zfs or a fuse-compression filesystem, then you have what you want. Dont change defaults in a project as complex as this one. It's like the advertisment-mafia wanting opt-out on everything (because no fu***ng idiot actually whants ads and would opt-in)...

My 2 cents. Really, no personal offence intended, i just get really testy when someone starts to screw with essential infrastructure. If you have a Family on your tvheadend, you can probably relate ;)

Joe User wrote:

... snip ..
Mark Clarkstone wrote:

Who's talking about an SD card? I certainly wasn't & I don't see it mentioned anywhere in this issue?

I mentioned it in another thread where config file editing was discussed.

I see, in that case, I apologise to Peter for not noticing this!

Hanspeter Müller wrote:

Mark Clarkstone wrote:

1. Should the config files be compressed by default, for the average user? I'm talking about those who don't fiddle with the config files.

"No. Under no circumstances this shall be a default." Non-negotiable, punishable by forking ;)

I totally understand Jaroslavs point-of-view as a dev that he does not like users tinkering with config-files, but this is not the normal case, i only do this, when something goes really wrong, with broken git-commits or just SNAFU (which happened two or three times last year, in all cases i would have been majorly screwed without the ability to grep/sed/etc and would have to restore a multi-month-old-backup)

1. How should they be stored? What format?

"Same as now." I still see no reason for compression except to make it more complicated, but an OPTION (compile-time or startoption) for experts to ENABLE compression would be fine, as long as it defaults to OFF.

THIS is exactly why i referenced systemd a posting earlier, they have an obsessive compulsion to get rid of every configfile and move to their view of a statleless system with a static /etc, how the entire computerworld should work in their view. Stop it!

Again, remember this is for the average user!

What's this average user you're keep referencing?!

I've already answered that.. "I'm talking about those who don't fiddle with the config files."

You're talking about a system to view and record tv-programs, which produces 5-10 GIGABYTES per hour on a single Cable-HD-Channel, if your storage is good enough for that, it's also good enough to read a few kilobytes on startup!

Not everyone stores their config on the same drives as recordings, and not all storage is created equal, sometimes it's dog slow, or has limited r/w. Yes, it's crappy, but shall we ignore it?

My oldest SATA Drive that is still alive (a Samsung 160GB drive, should be about 15 Years old by now!) does 80MB/s, so assuming that Peter is wrong and you're not talking about SD-Cards, what ARE you using that seems to suck trump so hard, that you want to compress config-files in a range from 10 to 3000 bytes each?!

For what it's worth, I use a load of different storage methods, SSD, standard sata (and esata), eMMC, Compact Flash, USB etc.. None of them suck trump to use your words.

If it where the case that this is one bigass json/xml file, compression might actually be usefull, but compressing files that small only produces overhead, with the additional downside of deniing the average-user™ editing access, when he REALLY needs it.

Also: If you for-whatever-reason, you want this, use btrfs, zfs or a fuse-compression filesystem, then you have what you want. Dont change defaults in a project as complex as this one. It's like the advertisment-mafia wanting opt-out on everything (because no fu***ng idiot actually whants ads and would opt-in)...

My 2 cents. Really, no personal offence intended, i just get really testy when someone starts to screw with essential infrastructure. If you have a Family on your tvheadend, you can probably relate ;)

None taken, at least not from me, and I hope I've not offended anyone also.

Being fairly new to tvheadend I think it is a great product - but immediately screwed me with the update which started compressing the services files!

Protecting configuration files from fiddlers may help prevent some problems but when there is no "official" way to make changes in a controlled manner this just causes real hassle to the (small?) percentage of users who have a real need to incorporate a change.

The Network & Services files are not true "config" files and nobody will even find them unless they need to (and probably know what they are doing).
The files are not large so compression is not, IMHO, a relevant issue.

The current compression is more like encryption because it is really not an easy task to edit the contents - I don't want to load up a "foreign" development environment on my PC, download the whole of tvheadend, edit the source code (with a simple change), compile and then have to upload to a small box which only has a remote control just to change an entry.

Far, far, better to either have a tvheadend config file with a "Compress info files or not" checkbox OR use simple compression if it is likely to be of significant value

Is someone able to write SQLITE fork of TVH ;)

saen acro wrote:

Is someone able to write SQLITE fork of TVH ;)

I very much doubt that'll happen, unless you do it ;).

How about the idea of a single import/export method using various formats. But I think that it should only work locally, by that I mean no upload/download via the webui. There could be trigger buttons on the webui to import/export from/to a local tar file.

For the import tvh could unpack the config, making a backup of the current, and restart?

Although all this really does is what the user would do anyway?

How do we "decrypt" files from the console??

Again, I'd like that you forget about the possibility to modify the database files directly. I created a small python script which allows to export/import the idnode contents: https://github.com/tvheadend/tvheadend/tree/master/lib/api/python . It's WiP, but there are some example commands:

$ TVH_API_URL="http://localhost:9981/api" TVH_USER=admin TVH_PASS=admin ./tvh-json.py get pathlist

$ TVH_USER=admin TVH_PASS=admin ./tvh-json.py --debug get raw/export '{"class":"mpegts_mux"}'

Basically, these data are IDENTICAL what you find in the configuration files, but all operations are safe and you can do them when tvh is running.

Jaroslav Kysela wrote:

Again, I'd like that you forget about the possibility to modify the database files directly. I created a small python script which allows to export/import the idnode contents: https://github.com/tvheadend/tvheadend/tree/master/lib/api/python . It's WiP, but there are some example commands:

[...]

Basically, these data are IDENTICAL what you find in the configuration files, but all operations are safe and you can do them when tvh is running.

Sorry but all I get is:
[[ File "./tvh-json.py", line 126
main(sys.argv)
^
IndentationError: expected an indented block
]]

Best regards!

Use the file from repo (git clone) or download the raw file. You probably did something wrong with copy-n-paste.

Jaroslav Kysela wrote:

I created a small python script which allows to export/import the idnode contents: https://github.com/tvheadend/tvheadend/tree/master/lib/api/python . It's WiP, but there are some example commands:

[...]

Basically, these data are IDENTICAL what you find in the configuration files, but all operations are safe and you can do them when tvh is running.

Hi,

This is a very good idea!
However, please consider also if you can expand this functionality to this:

- Start the THV with autoloading (importing) one "tvh-conf.json" file passed in the commandline.

What is the objective of this?
Let the user the option for bootstrapping a new install with some pre-configuration.
The idea is quite simple: you can configure manually the "tvh-conf.json" and load it at the first run.

You agree to support this?

Mono Polimorph wrote:

This is a very good idea!
However, please consider also if you can expand this functionality to this:

- Start the THV with autoloading (importing) one "tvh-conf.json" file passed in the commandline.

We need an useful UI oriented export (download a config file) as first and probably import too (upload a config file). The same file might be probably specifed on command-line, but I'm not sure if it's worth to add this new option.

I improved tvh-json.py and added README.md : https://github.com/tvheadend/tvheadend/tree/master/lib/api/python

Jaroslav Kysela wrote:

Mono Polimorph wrote:

This is a very good idea!
However, please consider also if you can expand this functionality to this:

- Start the THV with autoloading (importing) one "tvh-conf.json" file passed in the commandline.

We need an useful UI oriented export (download a config file) as first and probably import too (upload a config file). The same file might be probably specifed on command-line, but I'm not sure if it's worth to add this new option.

I'm not a fan of downloading/uploading config files via the webui, this can lead to a whole host of problems. I suppose it'll be fine it it can be disabled via the command line.

The main reasons are:

• Whats to stop the upload of mangled config files? Usually with the intent to crash.
• People could upload some nasty scripts that they'd be able to launch using the pipe command!
• Configs could contain some personal identifiable info.

Hence for these reasons that I suggested an upload/download trigger buttons that requires the user have file system access to pull/push the files.