Tvheadend

I have set 'EPG scan timeout in seconds (30-7200):' to 3600 and I remember that after restarting TVH it took a lot of time to grab EPG by EIT. I have seen epggrab in Status tab for a long time.
Actually, all epggrab subscriptions are gone after a while, even timeout set has been not reached. I think this might be related to presented 'holes' in EPG. Looks like it is not obtaining whole EPG and misses some data now.

I seem to have EPG problems as well - I'm not sure that they're 'holes', but I'm certainly missing most events in the WebUI. It may or may not be related.

Trying to bisect the commits:

Build 2004 commit 77d4d39 - gives working EPG
Build 2013 0d05c29 crashes
Build 2020 1453747 crashes
Build 2038 deb9473 crashes
Build 2043 0b2e072 crashes
Build 2048 4c1f0fa runs - but has missing EPG data

My normal EPG is c. 70,000 events, +/- a bit.

My 'truncated' version is < 400. And it stays there from what I can see. I've made no changes to the EPG timeout.

I'm using DVB-S2 and DVB-T/T2 with OTA grabbers.

Yes, I have holes... if I look at a clean build (3.9.2111~ga0e69f8), I have some data for 18/11... one event on 19/11... one on 22/11... and then a few hundred on 25/11.

Like a good Swiss cheese.

Trying here, removed EPG but got grabbed all events from DVB-T and DVB-S seems OK too.. But running more tests here.. Another strange bug.

Tested with 1000 Channels DVB-T and DVB-S from 3 different SAT bouquets (Czech, German) and I don't see any missing EPG or gaps:

[   INFO] epgdb:   episodes   109138
[   INFO] epgdb:   broadcasts 109138

rwx------ 1 perexg perexg 71748931 Nov 18 17.48 cfg3/epgdb.v2

I can confirm, that reverting back to mentioned commit fixes this issue.
As i have stated in ticket description - this had to be introduced in one of latest commit...

Prof Yaffle wrote:

I seem to have EPG problems as well - I'm not sure that they're 'holes', but I'm certainly missing most events in the WebUI. It may or may not be related.

Could you try latest and provide "--trace epggrab" ?

Running now - my EPG survived the upgrade for the first time since build 2000ish, so instead of starting with < 400 events I started with > 66,000. That's good.

epggrab kicked in immediately as well, and appears to be running, which is also good. I'll let it run for a bit, see if it finishes and PM you the logfile.

Looks good, though, for the moment... let's see if new events appear. I'll leave it overnight if necessary.

Thanks for the outstanding work here - děkuji :-)

Okay, EPG deleted events as they expired, and new events appeared on the end of the list. It appeared to be functional. I'm getting too many crashes to really test it further, so need to revert to my stable build, but it's looking promising.

Let's see what Rafal has to say tomorrow...

Steps done:

1) Updated to 3.9.2130
2) Restarted tvheadend
3) Enabled initial EPG scan on startup
4) Set EPG grab time limit to 15 minutes
5) Stopped tvheadend
6) Moved epgdb.v2 file
7) Started tvheadend
8) Waited 15 minutes
9) Checked EPG for 'holes'
10) Stopped tvheadend
11) Compared new epgdb.v2 file with the old one

Results:
) I have found missing EPG data in WebUI on few channels (both DVB-T and DVB-S)
) I have found missing EPG data in XBMC (after clearing all PVR data)
*) New epgdb.v2 file is about ~1MB smaller:

rwx----- 1 tvheadend video 21994043 Nov 19 23:58 epgdb.v2
rwx----- 1 tvheadend video 22730502 Nov 19 23:36 epgdb.v2_BACKUP

I deleted my EPG db and restarted tvh... no EPG data was grabbed despite epggrab running.

On Jaroslav's advice, I tried manually tuning to the Freesat EPG mux - tvh immediately crashed. I don't know if the two things are related.

gdb backtrace (gdb / set args -u hts -g video -D / file /usr/bin/tvheadend / run):

[New Thread 0x7fffadffb700 (LWP 13849)]
tcp_server: tvhpoll_wait: Interrupted system call
2014-11-19 11:53:48.103 [  ERROR] iptv: poll() error Interrupted system call, sleeping 1 second
2014-11-19 11:53:58.050 [   INFO] mpegts: 12324V in ASTRA - scan no data, failed

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc5ffb700 (LWP 4494)]
strlen () at ../sysdeps/x86_64/strlen.S:106
106    ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) bt
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x00007ffff60a02ae in __GI___strdup (
    s=0x6433383661630202 <error: Cannot access memory at address 0x6433383661630202>) at strdup.c:41
#2  0x000000000045802b in htsmsg_add_str (msg=<optimised out>, 
    msg@entry=0x7fffb00210a0, name=<optimised out>, 
    name@entry=0xa6c0ee "title", 
    str=0x6433383661630202 <error: Cannot access memory at address 0x6433383661630202>) at src/htsmsg.c:241
#3  0x0000000000454970 in htsp_build_event (e=e@entry=0x12b7bf0, 
    method=method@entry=0x0, lang=lang@entry=0x0, update=update@entry=0, 
    htsp=0x7fffc5ffa6b0) at src/htsp_server.c:789
#4  0x0000000000454e5c in htsp_method_getEvents (htsp=0x7fffc5ffa6b0, 
    in=<optimised out>) at src/htsp_server.c:1107
#5  0x0000000000456bdc in htsp_read_loop (htsp=0x7fffc5ffa6b0)
    at src/htsp_server.c:2382
#6  htsp_serve (fd=34, opaque=0x7fffd0000c08, source=<optimised out>, 
    self=<optimised out>) at src/htsp_server.c:2518
#7  0x00000000004390bf in tcp_server_start (aux=0x7fffd0000be0)
    at src/tcp.c:540
#8  0x0000000000435f41 in thread_wrapper (p=0x7fffd0000d60)
    at src/wrappers.c:140
#9  0x00007ffff68f3182 in start_thread (arg=0x7fffc5ffb700)
---Type <return> to continue, or q <return> to quit---
    at pthread_create.c:312
#10 0x00007ffff6111fbd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) 
</pe>

Valgrind output:

<pre>
$ valgrind --leak-check=full --show-reachable=yes /usr/bin/tvheadend -u hts -g video -D

    ==28528== Thread 41 tcp_server_start:
    ==28528== Invalid read of size 8
    ==28528==    at 0x499839: descrambler_service_start (descrambler.c:108)
    ==28528==    by 0x44EA5F: service_start (service.c:623)
    ==28528==    by 0x450049: service_find_instance (service.c:734)
    ==28528==    by 0x44C24B: subscription_reschedule (subscriptions.c:314)
    ==28528==    by 0x44CA43: subscription_create_from_channel_or_service (subscriptions.c:677)
    ==28528==    by 0x44CB39: subscription_create_from_channel (subscriptions.c:688)
    ==28528==    by 0x48EF94: http_stream_channel (webui.c:863)
    ==28528==    by 0x4904E8: http_stream (webui.c:938)
    ==28528==    by 0x43C81F: http_exec (http.c:489)
    ==28528==    by 0x43CD06: http_cmd_get (http.c:555)
    ==28528==    by 0x43CF81: process_request (http.c:635)
    ==28528==    by 0x43D2BC: http_serve (http.c:946)
    ==28528==  Address 0xd8317a8 is 24 bytes before a block of size 48 free'd
    ==28528==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==28528==    by 0x4A71EE: dvb_bat_destroy_lists.isra.5 (dvb_psi.c:1163)
    ==28528==    by 0x4A96E3: dvb_nit_callback (dvb_psi.c:1290)
    ==28528==    by 0x4A42DF: mpegts_table_dispatch (mpegts_table.c:124)
    ==28528==    by 0x4A3EB1: mpegts_psi_section_reassemble0 (mpegts_table.c:338)
    ==28528==    by 0x4A4FA9: mpegts_psi_section_reassemble (mpegts_table.c:376)
    ==28528==    by 0x49E0B5: mpegts_input_table_dispatch (mpegts_input.c:683)
    ==28528==    by 0x49EA5A: mpegts_input_table_thread (mpegts_input.c:924)
    ==28528==    by 0x435F40: thread_wrapper (wrappers.c:140)
    ==28528==    by 0x6110181: start_thread (pthread_create.c:312)
    ==28528==    by 0x692EFBC: clone (clone.S:111)
    ==28528==
</pre>

I hate it I hate it I hate it I hate it.

After watching various versions crash on demand yesterday, and having holes all over the place on my EPG, I started with a more systematic approach. Last night and this morning, I've built, run and debugged 40-odd versions of tvheadend between 3.9.2004~g77d4d39 and 3.9.2144~gda3d9ec. Would they crash? Would they hell. Would they eat my EPG? Nope, I currently have 75,000 events and climbing.

I've left tvh running in gdb (vs as a service) for the moment to see if anything changes. Otherwise, my conclusion is a config corruption - I can't explain it otherwise without randomly blaming an OTA packet corruption or a driver error.

I suppose it's possible - though unlikely - that there's an issue in config migration, and jumping from too early a version to too late a version is leaving somethign out of the config that's later relied upon? Compare that to my step-by-step upgrades over the past 24 hours - either commit-by-commit or in groups of five or ten commits as I lost patience with everything resolutely working. If there is a migration issue, chances are high I'd have avoided it like that - or if anyone running from an unstable repo or rebuilding every 48 hours or so.

Guesswork, though. Let's see.

Confirmed with a 'clean' grab as well, after removing epgdb.v2 - events were successfully re-grabbed OTA for both DVB-T and -S. Back up to 75k events in < 30s or so.

Prof Yaffle wrote:

I hate it I hate it I hate it I hate it.

After watching various versions crash on demand yesterday, and having holes all over the place on my EPG, I started with a more systematic approach. Last night and this morning, I've built, run and debugged 40-odd versions of tvheadend between 3.9.2004~g77d4d39 and 3.9.2144~gda3d9ec. Would they crash? Would they hell. Would they eat my EPG? Nope, I currently have 75,000 events and climbing.

I've left tvh running in gdb (vs as a service) for the moment to see if anything changes. Otherwise, my conclusion is a config corruption - I can't explain it otherwise without randomly blaming an OTA packet corruption or a driver error.

I suppose it's possible - though unlikely - that there's an issue in config migration, and jumping from too early a version to too late a version is leaving somethign out of the config that's later relied upon? Compare that to my step-by-step upgrades over the past 24 hours - either commit-by-commit or in groups of five or ten commits as I lost patience with everything resolutely working. If there is a migration issue, chances are high I'd have avoided it like that - or if anyone running from an unstable repo or rebuilding every 48 hours or so.

Guesswork, though. Let's see.

Maybe it is safe to check latest commit with clean config?
In this case i would move /etc/tvheadend somewhere and start from scratch, because I dont encouter any crashes.

Prof Yaffle wrote:

I suppose it's possible - though unlikely - that there's an issue in config migration, and jumping from too early a version to too late a version is leaving somethign out of the config that's later relied upon? Compare that to my step-by-step upgrades over the past 24 hours - either commit-by-commit or in groups of five or ten commits as I lost patience with everything resolutely working. If there is a migration issue, chances are high I'd have avoided it like that - or if anyone running from an unstable repo or rebuilding every 48 hours or so.

Guesswork, though. Let's see.

At least consistent with the crashes I'm seeing now after skipping several hundred commits when updating (#2498), but might not be related either.

Rafal Kupiec wrote:

Maybe it is safe to check latest commit with clean config?
In this case i would move /etc/tvheadend somewhere and start from scratch, because I dont encouter any crashes.

Config removed, dpkg-reconfigure to recreate the 'base' config, networks and muxes added, scanning running for an hour and a half or so now. DVB-T stuff completed quickly as always, and generated > 14k events; DVB-S stuff is still discovering muxes and adding services, so I've a lot of holes on specific channels (i.e. no events), partial holes on others (i.e. events every day, but only in the afternoon), and complete gaps elsewhere (i.e. channels that don't yet exist because the underlying services and muxes haven't been discovered). Nothing deeply worrying at this stage... given the discovery of events when I simply deleted the EPG DB, I'd put all of these holes down to time and discovery.

I'll leave it run for another hour or so and see if anything changes. @Rafal, maybe you want to try as well and see where you land?

@Martin Walter - re: your crashes - all I can suggest is a clean config to see if it helps. It's a random hope to grab at...!

Prof Yaffle wrote:

Rafal Kupiec wrote:

Maybe it is safe to check latest commit with clean config?
In this case i would move /etc/tvheadend somewhere and start from scratch, because I dont encouter any crashes.

Config removed, dpkg-reconfigure to recreate the 'base' config, networks and muxes added, scanning running for an hour and a half or so now. DVB-T stuff completed quickly as always, and generated > 14k events; DVB-S stuff is still discovering muxes and adding services, so I've a lot of holes on specific channels (i.e. no events), partial holes on others (i.e. events every day, but only in the afternoon), and complete gaps elsewhere (i.e. channels that don't yet exist because the underlying services and muxes haven't been discovered). Nothing deeply worrying at this stage... given the discovery of events when I simply deleted the EPG DB, I'd put all of these holes down to time and discovery.

I'll leave it run for another hour or so and see if anything changes. @Rafal, maybe you want to try as well and see where you land?

@Martin Walter - re: your crashes - all I can suggest is a clean config to see if it helps. It's a random hope to grab at...!

I have written this regarding your crashes.
About EPG: Please take a look at epg.png (6.36 KB). Nothing changed since that time.

What about this bug.. Can it be reproduced with the current tvh code ?

Jaroslav Kysela wrote:

What about this bug.. Can it be reproduced with the current tvh code ?

Need to check thoroughly.
Will keep you updated within next 2-3 days.

Yes, problem still exists

Any update here?

Also interested. Have the same problem since a couple of updates. Some channels miss data in the EGP.

Can we do anything with this, please?
Im pretty sure its not dependent on drivers or device, because It is working great with TVH 3.4.

Agree with Rafal. Did a clean install a few weeks ago. All EGP data appeared again. Now I have one channel missing OTA EPG data again. I have a feeling that something is happening during upgrades of TVH. I upgrade frequently...almost as soon as a new version is avaliable. Just a guess...might be totally wrong.

I've not seen this in my setup. Provide some logs.

1) remove the <cfgdir>/epgdb.* file
2) start tvh
3) trigger EPG scan (in the config/channels/epg dialog - there's a button on top)
4) wait until this scan finishes
5) wait for EPG holes (if they're not visible immediatelly after scan)
6) provide the logs and describe them (channel, time)

Subsystem to trace (--trace epg,eit), tvh binary must be compiled with --enable-trace.

THE LOGS WILL BE HUGE!!!! You need a lot of disc space.

Anders Falk wrote:

Agree with Rafal. Did a clean install a few weeks ago. All EGP data appeared again. Now I have one channel missing OTA EPG data again. I have a feeling that something is happening during upgrades of TVH. I upgrade frequently...almost as soon as a new version is avaliable. Just a guess...might be totally wrong.

or during restart

@Rafal
@Andres

Can you please try with tvh version 4.1-467-ge2ee2a7 or newer?

Still some holes with 4.1-467~ge2ee2a7

Rafal Kupiec wrote:

Still some holes with 4.1-467~ge2ee2a7

Can you post a screenshot of the epg grabber settings page?

Do you use kodi?, can you post a screenshot of the epg page with holes?

Ok, i asked you screenshots because something similar happened to me.

I can speak only for the eit grabber, i don't use xmltv, and this is mi thinking (maybe wrong anyway):

- tvheadend grabs eit epg data every 12 hours (default config at 02:04 and 14:04);
- You set timeout for muxes scan to 900 = 15min so, in case a mux reach timeout, the receiver is busy for 15 minutes, this for each mux reaces timeout.
- i don't know your equipment (number of tuners), the number of muxes that reaches timeout and also, in your screenshot (kodi one), i cannot understand if the affected channel epg is now/next or scheduled.

Then my thought is that your timeuot is too big. In my experience, here in Italy, problematic muxes, despite the timeout, send quickly the full epg data, so, for me, a timeout of 120 secs is enough. This lock tuners for only two minutes per problematic mux. Also the refresh period (12 hours) in my case was too long, i have many channels with now/next epg data, so, in short programs, cartoons for example, i had holes. In my case, the optimal scan period is 45 minutes with really few holes in case of really short programs, like 3 min news

Maybe not the solution, but can you give a try?

Useful also a log with timings for muxes epg scan. Can you try to launch in a console this:

tvheadend -l tvheadend.log -C

so the normal (not trace) log is wrote in tvheadend.log and is possible - at least - check the number of problematic muxes and the time tvheadend takes to scan the whole network.

Well, nice investigation, but here are my thoughts:

1) I need longer timeout because i have several networks. I don't see such option per network/mux and all my DVB-S requires that
2) This hole is in EPG for Sunday (tomorrow), but sometimes happens that I am watching channel and there is no current EPG for it
3) I got 4 DVB-T tuners with 3 MUXes and 4 DVB-S tuners with about 20 transponders
4) I don't experience this issue with TVH 3.4. Perex told me its related to bug in driver, but I think there might be sth wrong after rewrite.

1) I need longer timeout because i have several networks. I don't see such option per network/mux and all my DVB-S >requires that

No, timeout isn't related to network(s), this is the reason i asked you to lower it. Timeout is related to each single mux, no matter the network number. Tvheadend scans all services registered in otamux directory, no matter the amount of time this implies, but if it meets a problematic mux, related tuner is locked until timeout.
In your case is relevant the cron refresh period, this yes, if is too low, tvheadend cannot start new epg scan cycle before the previus isn't finished (and this is the case for holes).

The log tells us exactly the time tvheadend takes to scan all your muxes/services, this to calibrate the cron multiline command.

This i my thought, Skilled guys can correct me if i'm wrong, but:

1) the timeout parameter is related to each mux (the option you don't find per mux is for "all" single mux).
2) the per network(s) (DVBT/satellites) time is influenced by the cron multiline (if intensive scans may be climbed over, if rare, previously registeed epg may end before a new rescan)

4) I don't experience this issue with TVH 3.4. Perex told me its related to bug in driver, but I think there might be >sth wrong after rewrite.

For me too, but the otamux structure has changed. previously there was a file per service, and i subspect that the epg scan was for all services. Now there is logic and filters, so it act more efficiently, but problematic muxes cause troubles (the reason i asked you to try with the specific version of tvh is to exclude bug 1942). This, in my though, isn't a driver problem, neither a tvheadend problem. I think this is a configuration problem to be optimized according broadcasters parameters.

Ah, for the record, i got 2 dvbt tuners (57 muxes), a dvbs2 tuner (hotbird) and another dvbs2 tuner connected to a 180 cm H-H dish (~20 satellites if i remember well), so the full satellite epg scan takes about an hour, but my tuners don't need timeouts (tevii S480)