Bug #3134
configuration save/load - improve throughput - save space (was: big starting time)
0%
Description
Setup: RPi2 with OpenElec + Tvh latest stable/master (no difference) + "quick" SD card + Sat>IP (network is ready before tvh starts)
At start time tvh takes a "break" for 30sec to 2min. I got no idea where this comes from.
I thought that it tries to load the whole epg.db (~100mb) but even with 0 size db it didn't makes an difference.
Trace, debug ... shows nothing (maybe wrong values?). I disable all what I can think of (initial epg scan ...) at gui but no change.
12:19:36.297 [ INFO] main: Log started
12:19:36.356 [ INFO] config: loaded
12:19:40.840 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
12:19:40.840 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
12:19:40.840 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
12:19:40.840 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
12:21:03.893 [ INFO] dvr: Creating new configuration ''
12:21:03.902 [ INFO] cwc: Attemping to connect to 192.168.0.244:24241
12:21:03.902 [ INFO] descrambler: adding CAID 0963 as quick ECM (Sky UK)
12:21:03.902 [ INFO] descrambler: adding CAID 098C as quick ECM (Sky DE)
12:21:03.902 [ INFO] descrambler: adding CAID 09C4 as quick ECM (Sky DE)
12:21:03.902 [ INFO] descrambler: adding CAID 09CD as quick ECM (Sky IT)
12:21:03.902 [ INFO] epggrab: module eit created
12:21:03.903 [ INFO] epggrab: module uk_freesat created
12:21:03.903 [ INFO] epggrab: module uk_freeview created
12:21:03.903 [ INFO] epggrab: module viasat_baltic created
12:21:03.903 [ INFO] epggrab: module Bulsatcom_39E created
12:21:03.903 [ INFO] cwc: Connected to 192.168.0.244:24241
what is he doing between
scanfile: ATSC - loaded 2 regions with 14 networks and dvr: Creating new configuration ''
History
Updated by C vH about 9 years ago
At an 1 month old (oldest I have) log I saw a value of 20sec "break" - that was even slow in comparison to the expected (other RPi Tvh users) values of about 1-2sec.
Updated by Jaroslav Kysela about 9 years ago
Hmm. It's on my laptop:
2015-10-06 13:05:41.950 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 2015-10-06 13:05:41.950 [ INFO] mpegts: Loading input/dvb/networks 2015-10-06 13:05:43.785 [ INFO] mpegts: ... done 2015-10-06 13:05:43.871 [ INFO] linuxdvb: adapter added /dev/dvb/adapter1
The <cfg>/input/dvb/networks files (all) are parsed and networks, muxes and services are created in that time.
iff --git a/src/input/mpegts/mpegts_network_dvb.c b/src/input/mpegts/mpegts_network_dvb.c index 47007c3..f65ba8b 100644 --- a/src/input/mpegts/mpegts_network_dvb.c +++ b/src/input/mpegts/mpegts_network_dvb.c @@ -658,6 +658,7 @@ void dvb_network_init ( void ) dvb_network_builder); /* Load settings */ + tvhinfo("mpegts", "Loading input/dvb/networks"); if (!(c = hts_settings_load_r(1, "input/dvb/networks"))) return; @@ -673,6 +674,7 @@ void dvb_network_init ( void ) } } htsmsg_destroy(c); + tvhinfo("mpegts", "... done"); } void dvb_network_done ( void )
Updated by C vH about 9 years ago
13:54:53.513 [ INFO] main: Log started
13:54:53.576 [ INFO] config: loaded
13:54:58.214 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
13:54:58.214 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
13:54:58.214 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
13:54:58.214 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
13:54:58.214 [ INFO] mpegts: Loading input/dvb/networks
13:56:05.405 [ INFO] mpegts: ... done
Updated by Jaroslav Kysela about 9 years ago
Cached ? (You need to clear page cache to simulate the boot process: "echo 1 > /proc/sys/vm/drop_caches")
Updated by Jaroslav Kysela about 9 years ago
- Subject changed from huge timeouts at starting time to big starting time
Updated by Jaroslav Kysela about 9 years ago
Yeah. I overlooked difference between 54/56 (minutes). I though that the timediff is only 7 seconds. You may compare, if the start time is lower when the files are cached (start tvh multiple times after boot). Just for a curiosity to check the i/o delays at boot.
Updated by C vH about 9 years ago
I had tested this at the mean time,
looks like it works "quick - 20sec" if it is cached (new profile is a tvh profile from an x64 htpc)
my old very slow (1-2min) profile has ~14000 files at tvh/input/dvb/networks/id the other one from my x64 (20-30sec) has only ~6000 but both profiles have same network, same channels and same config O_o
Maybe it is purely overwhelmed by the number of files that the i/o is breaking down (some cache is full?)?
I don't know if Tvh has to open every file at input/dvb/... or if it is just on demand? The pure read speed should not be a problem, but to look into 14000 files is a problem ;)
If it has to open every file (design fault?) it would explain it. Is there no sort of cache which prevents to load the settings at every start up ? For example throw the settings at runtime into a settings.xmlz (or at shut-down) and let the ~1mb xmlz been read instead of the ~k files at startup ?
cached
new profile
20:06:14.049 [ INFO] main: Log started
20:06:14.099 [ INFO] config: loaded
20:06:15.040 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:06:15.040 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:06:15.040 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:06:15.040 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:06:28.769 [ INFO] dvr: Creating new configuration ''20:09:32.826 [ INFO] main: Log started
20:09:32.878 [ INFO] config: loaded
20:09:33.785 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:09:33.785 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:09:33.785 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:09:33.785 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:09:47.196 [ INFO] dvr: Creating new configuration ''20:15:55.918 [ INFO] main: Log started
20:15:55.965 [ INFO] config: loaded
20:15:56.878 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:15:56.878 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:15:56.878 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:15:56.878 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:16:10.481 [ INFO] dvr: Creating new configuration ''old profile
20:19:20.629 [ INFO] main: Log started
20:19:20.680 [ INFO] config: loaded
20:19:22.269 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:19:22.270 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:19:22.270 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:19:22.270 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:19:39.562 [ INFO] dvr: Creating new configuration ''20:33:57.306 [ INFO] main: Log started
20:33:57.355 [ INFO] config: loaded
20:33:58.276 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
20:33:58.276 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
20:33:58.276 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
20:33:58.276 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
20:34:12.061 [ INFO] dvr: Creating new configuration ''21:04:53.319 [ INFO] main: Log started
21:04:53.366 [ INFO] config: loaded
21:04:54.283 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks
21:04:54.284 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks
21:04:54.284 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks
21:04:54.284 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks
21:05:08.165 [ INFO] dvr: Creating new configuration ''
Updated by Jaroslav Kysela about 9 years ago
- Subject changed from big starting time to configuration save/load - improve throughput - save space (was: big starting time)
- Target version set to 4.4
You should probably clean the dead services / muxes in the config with more files. They're not remove automatically. Anyway, the configuration storage is not optimal. I'm playing with some ideas like to create "snapshots" to a compressed file (like EPG) in a binary format (instead text one). But it's something for 4.4.
Updated by C vH about 9 years ago
Maybe some kind of xml file? I get a ~4mb file which could be gziped (fastest) to 300kb. Compression + file size shouldn't be a problem for every platform.
In the mean time tx for your quick help (as always)
Updated by C K about 9 years ago
C vH wrote:
Maybe some kind of xml file? I get a ~4mb file which could be gziped (fastest) to 300kb. Compression + file size shouldn't be a problem for every platform.
In the mean time tx for your quick help (as always)
What about using a database like mysql or postgresql for all the storage? this could allow - in a case of a re-installation - to keep your channel nummbers :-)
Updated by C vH about 9 years ago
sqlite/libsqlite would do it also (no need for heavy mysql server), would fix the i/o problem also including simple import/backup possibility also sounds like easy implementation
downside, it would break the config again :/ -> easy downgrade isn't possible
Updated by Jaroslav Kysela over 8 years ago
There are two changes in latest master to reduce disk I/O in the start phase:
1) EPG data can be gziped now (it's on by default - global config)
2) mux config / service config data are merged together and gziped, too (forced - updated with the new data save)
It should reduce significantly the loading times when the disk I/O is an issue.
Note: You should rescan all networks/muxes to trigger the new save functions. Otherwise, the old configs are not touched.
Updated by Volker Schmidt over 8 years ago
Just for the record: Savings at my installation was approx. 10 MBytes.
Feeds: Astra 19.2° E plus a local DVB-C cable provider.
Updated by Joe User about 5 years ago
I think this can be closed after the gzip config change???
Updated by Pablo R. about 5 years ago
Joe User wrote:
I think this can be closed after the gzip config change???
And cookie files already (auto)cleared after 7 days. I think that is not longer a problem.
(c73e4248a9072be57d5ede3a510773ba06bebc09)
Updated by Mark Clarkstone about 5 years ago
- Status changed from New to Fixed
Pablo R. wrote:
Joe User wrote:
I think this can be closed after the gzip config change???
And cookie files already (auto)cleared after 7 days. I think that is not longer a problem.
(c73e4248a9072be57d5ede3a510773ba06bebc09)
Joe User wrote:
I think this can be closed after the gzip config change???