Tvheadend

Here's a (perhaps dumb!) question - what is this file actually used for?

As noted on issue 996, I had this problem and deleted the epgdb file, and the system has worked fine since. However, this file has not been recreated, despite a twice-daily XMLTV update.

My only thought is that I haven't restarted the system since - so it may be something that's only created on startup, but then doesn't seem to be needed (not in my setup, anyway).

the EPGDB is created when tvheadend is shutdown and contains the epg database that tvheadend maintains in memory at that time.
This was done so that when tvheadend restarts it can reload the epg from that file, instead of needing a full xmltv load and/or loading the epg from EIT information.

It is done in /src/epg.c using the epg_save and epg_load functions.

Is it possible that when restarting the computer, Tvheadend does not have the time to create this file so it gets corrupt?
Wouldn't it be better to create this file just after the epg has changed?
Thanks

well .. we do not want to continually write the epg to disc on each change there can be lot's of changes when doing continues mux scans and eit changes
... so a better solution would be to check if the file is valid when it gets loaded.

so now the question is .. how can we detect a corrupt epgdb?
write some checksum to the end of the epgdb so we can check of the complete file was written?

and .... is that the real cause of these problems?

A normal checksum (e.g. MD5) would only tell you that the file is as it was written, i.e. nothing has changed it since it was written, surely? That doesn't tell you if it was written in a way that could be read back again, though...

We need to find a way to say "yes, this is a valid file that we'd expect to be readable again later", then test that against a file that is "problematic" (i.e. the deletion of which seems to solve the problem - Sébastien seems to be able to create them on demand, and has provided a copy to start with).

If the "problematic" file is actually syntactically valid, then the problem lies elsewhere - maybe it's the file size that's overflowing something, or a timing issue where the file is still being read when something tries to access the data "because we know we've got a full EPG".

If it's not valid, then the file is being created as corrupt, and the next step is probably finding either

(a) why it's being written badly [is tvheadend dying too soon on shutdown - before the file is complete, for example?] or

(b) where in the 'reading it back in' code it chokes.

The checksum I was thinking about is a checksum that is calculated as the last step of writing the epgdb file. So it would be a checksum of the epgdb file WITHOUT that checksum. If the writing of the epgdb file was interrupted, it would not contain this checksum. When loading the file you can check if the last x bytes of the file are the checksum of the rest of the file ... if not .. reject the epgdb file as invalid.

But .... we first need to determine if the writing of the epgdb file is causing these errors, or if it is somewhere else......

Hein

Mmm, understand that - but if the file is written fully, but is faulty, the checksum would be valid because it's derived from the broken file.

I don't know if this is because the file format doesn't read back in, or because the files in question are broken by something while being written. That'd be my first thing to check...

Hein Rigolo wrote:

well .. we do not want to continually write the epg to disc on each change there can be lot's of changes when doing continues mux scans and eit changes
... so a better solution would be to check if the file is valid when it gets loaded.

At least in "XMLTV" mode, this update occurs for instance twice a day so the writing could be done at this time, couldn't it? I am not sure if writing a file of several MB at shutdown is a viable solution.

Ian Harper wrote:

(a) why it's being written badly [is tvheadend dying too soon on shutdown - before the file is complete, for example?] or
(b) where in the 'reading it back in' code it chokes.

You are right, we should even try to fix both these problems.
- Without any cheksum, Tvheadend could, like Firefox does, write an "epgdb.part" file and then rename it to "epgdb" when writing is complete. Using this method, we would never have incomplete epgdb file (think about a power outage at this time...)
- Could the reading code be "try/catched" so that a faulty epgdb never makes Tvheadend crash (in the worst case, let us re-read the XMLTV file or reload the data!)?

How big is the epgdb file created on your system, Sébastien? Mine's over 11Mb, which is far too large to get written out during a shutdown event... I'm going to try manually stopping tvheadend when this XMLTV grab is finished and see how big the file is then, and whether it can be read back in again.

My suspicion is simply that the file is corrupt if it's too big to get written properly, which would be the case if you have tens of thousands of EPG events in it (as I do).

Okay, I didn't let the XMLTV update run its course - life's too short! - but I stopped tvheadend with sudo service tvheadend stop - checked that it had created epgdb (it had), and then restarted with sudo service tvheadend start.

A quick ps -eaf | grep tv showed that tvheadend wasn't running.

Dmesg | grep tvheadend revealed:

[11231.301606] tvheadend[1987]: segfault at 0 ip 00656228 sp bfe51c4c error 4 in libc-2.11.1.so[53e000+153000]

Soooo.... it's not the fact that the EPG is being written (and corrupted) during shutdown, since I didn't shut down the machine. This was a controlled stop/start.

And also, it's not the size of the file, since it wasn't full-sized ... a mere 2.6Mb. So if it's an overflow of some kind, it's a very small limit.

For what it's worth, in case Séb or someone else can use this information to find common ground...

System:

Linux XBMCLive 2.6.32-41-generic #89-Ubuntu SMP Fri Apr 27 22:22:09 UTC 2012 i686 GNU/Linux

Recent packages (from zcat -f /var/log/dpkg.log* | grep "\ install\ " | sort):

_some packages from March ... I don't know when this issue started ..._
2012-05-04 17:50:58 install libcommon-sense-perl <none> 3.00-1
2012-05-04 17:50:59 install libjson-perl <none> 2.17-1
2012-05-04 17:50:59 install libjson-xs-perl <none> 2.270-1
2012-05-21 20:40:29 install liblockdev1 <none> 1.0.3-1.3
2012-05-21 20:40:30 install libcec1 <none> 1.6.1-5
2012-05-21 20:54:21 install linux-image-2.6.32-41-generic <none> 2.6.32-41.89
2012-05-21 20:54:48 install linux-backports-modules-alsa-2.6.32-41-generic <none> 2.6.32-41.43
2012-05-21 20:54:52 install python-pycurl <none> 7.19.0-3
2012-05-21 21:28:32 install libcec1 1.6.1-5 1.6.1-5
2012-05-21 21:28:32 install liblockdev1 1.0.3-1.3 1.0.3-1.3
2012-05-25 14:25:28 install libcec-dev <none> 1.6.1-5
2012-06-04 12:31:29 install linux-headers-2.6.32-41 <none> 2.6.32-41.89
2012-06-04 12:31:49 install linux-headers-2.6.32-41-generic <none> 2.6.32-41.89
2012-06-04 17:31:01 install foremost <none> 1.5.6-1
2012-06-04 18:04:51 install testdisk <none> 6.11-1

libc version as above.

Sorry not to stop my Tvheadend instance which is currently recording some movies. Its uptime is now more than 5 days and I notice that except this bug at shutdown (and sometimes macroblocking when reading HD files with XBMC) it is the perfect system for me.

This said, below is the information about my system:
I am using XBMCBuntu 11.0 "Eden"

Linux pchc 3.0.0-20-generic-pae #34-Ubuntu SMP Tue May 1 18:35:40 UTC 2012 i686 i686 i386 GNU/Linux

Recent packages:

2012-05-08 15:49:58 install tvheadend 2.13~git20120305.0664329-ppa1~oneiric 2.12.99~pulse-oneiric-6
2012-05-13 19:16:08 install python-gtk2 <none> 2.24.0-2
2012-05-13 19:16:09 install python-glade2 <none> 2.24.0-2
2012-05-13 19:16:10 install python-vte <none> 1:0.28.2-0ubuntu2
2012-05-13 19:16:11 install par2 <none> 0.4-11
2012-05-13 19:16:12 install pypar2 <none> 1.4-6
2012-05-15 22:50:19 install nvidia-cg-toolkit <none> 3.0.0007-0ubuntu1
2012-05-15 22:50:22 install libao-common <none> 1.1.0-1ubuntu1
2012-05-15 22:50:23 install libao4 <none> 1.1.0-1ubuntu1
2012-05-15 22:50:24 install libglew1.5 <none> 1.5.7.is.1.5.2-1ubuntu4
2012-05-15 22:50:25 install libopenal1 <none> 1:1.13-2
2012-05-15 22:50:26 install libsfml-network1.6 <none> 1.6+dfsg1-2ubuntu1
2012-05-15 22:50:27 install libsoil1 <none> 1.07~20080707.dfsg-2
2012-05-15 22:50:27 install libwxbase2.8-0 <none> 2.8.11.0-0ubuntu10
2012-05-15 22:50:28 install libwxgtk2.8-0 <none> 2.8.11.0-0ubuntu10
2012-05-15 22:50:29 install dolphin-emu <none> 3.0-0ubuntu1~oneiric
2012-05-16 20:03:14 install zip <none> 3.0-4
2012-05-16 20:03:15 install genisoimage <none> 9:1.1.11-1ubuntu2
2012-05-16 20:03:16 install file-roller <none> 3.2.1-0ubuntu2
2012-05-16 20:03:18 install python-gst0.10 <none> 0.10.21-2ubuntu1
2012-05-16 20:03:19 install python-aptdaemon.gtkwidgets <none> 0.43+bzr697-0ubuntu1.2
2012-05-16 20:03:19 install python-xapian <none> 1.2.5-2ubuntu1
2012-05-16 20:03:20 install python-aptdaemon-gtk <none> 0.43+bzr697-0ubuntu1.2
2012-05-16 20:03:21 install sessioninstaller <none> 0.20+bzr120-0ubuntu2
2012-05-16 20:04:06 install rar <none> 2:4.0.b3-1
2012-05-16 20:04:07 install unrar <none> 1:4.0.3-1
2012-05-18 22:27:39 install tcl8.5 <none> 8.5.10-1ubuntu1
2012-05-18 22:27:41 install tk8.5 <none> 8.5.10-1
2012-05-18 22:27:42 install blt <none> 2.4z-4.2ubuntu1
2012-05-18 22:27:44 install python-logilab-common <none> 0.55.2-1
2012-05-18 22:27:45 install python-logilab-astng <none> 0.21.1-1
2012-05-18 22:27:46 install pylint <none> 0.23.0-1
2012-05-18 22:27:47 install python-egenix-mxdatetime <none> 3.2.0-1
2012-05-18 22:27:47 install python-egenix-mxtools <none> 3.2.0-1
2012-05-18 22:27:48 install python-tk <none> 2.7.1-3
2012-05-19 20:18:05 install liblockdev1 <none> 1.0.3-1.4build1
2012-05-19 20:18:06 install libcec1 <none> 1.6.1-oneiric-5
2012-05-20 21:35:42 install gkrellm <none> 2.3.4-1
2012-05-20 21:35:42 install libntlm0 <none> 1.2-1
2012-05-21 23:07:06 install gstreamer0.10-ffmpeg <none> 0.10.12-1ubuntu1
2012-05-21 23:17:23 install linux-firmware-nonfree <none> 1.11
2012-05-22 15:57:25 install libaccess-bridge-java-jni <none> 1.26.2-6
2012-05-22 15:57:25 install libaccess-bridge-java <none> 1.26.2-6
2012-05-22 15:57:26 install openjdk-6-jre <none> 6b23~pre11-0ubuntu1.11.10.2
2012-05-22 15:57:27 install default-jre <none> 1:1.6-42ubuntu2
2012-05-22 15:57:27 install ttf-dejavu-extra <none> 2.33-1ubuntu1
2012-05-22 15:57:28 install dumphd <none> 0.61-0.4ubuntu1
2012-05-22 15:57:29 install icedtea-netx <none> 1.1.3-1ubuntu1.1
2012-05-25 21:30:01 install linux-image-3.0.0-20-generic <none> 3.0.0-20.34
2012-05-25 21:30:10 install linux-image-3.0.0-20-generic-pae <none> 3.0.0-20.34
2012-05-25 21:30:22 install linux-headers-3.0.0-20 <none> 3.0.0-20.34
2012-05-25 21:30:27 install linux-headers-3.0.0-20-generic <none> 3.0.0-20.34
2012-05-25 21:30:29 install linux-headers-3.0.0-20-generic-pae <none> 3.0.0-20.34

This bug did not occur anymore the last few times I started my computer. My installation of TvHeadend has now been updated to version 2.99.37.1332f9f
I just tried to stop TvHeadend manually by typing killall tvheadend and the epgdb file has a size of about 2 MB:

user@pchc:~$ ls -l ~/.hts/tvheadend/epgdb 
-rwx------ 1 user video 2168198 2012-06-22 19:39 /home/user/.hts/tvheadend/epgdb

We can now close this bug if no one else experiences it.

I haven't rebooted my system in a while, but you're right - I have updated tvheadend and it's restarted with no problem.

I'm currently running 2.99-7~lucid (dpkg log)/2.99.37.1332f9f (from the tvheadend web "about" tab). I've just manually shut it down and restarted (service tvheadend stop/service tvheadend start) and it's cycled with no issues.

The epgdb file is there and is 12M in size - but then, with a twin DVB-T and DVB-S tuner set up, and data from multiple grabbers, I do have a lot of channels and events, so that's not necessarily a problem.

As a final test, I've also just rebooted the PC, and tvheadend has come back up with no concerns (well, not because of the epgdb file - I still have timing problems with one of my tuners, but that's a different story!).