Project

General

Profile

Bug #2328

tvheadend crashes ( psi: tables count inconsistency (num 37, list 36) )

Added by Jouk Hettema - about 10 years ago. Updated almost 10 years ago.

Status:
Fixed
Priority:
High
Assignee:
Jaroslav Kysela
Category:
Crashes
Target version:
-
Start date:
2014-09-23
Due date:
% Done:

0%

Estimated time:
Found in version:
3.9.1567~gde634dd
Affected Versions:

Description

hi all,

tvheadend crashes while recording multiple channels. It also occurs while recording and watching, but it's easier to reproduce with just recording.

small config description:
- DN2800MT + 8GB memory
- PLEXTOR PX-128M5S as storage (and a bigger disk, but that's left out of this test)
- Digital Devices GmbH Mystique SaTiX-S2 V3 DVB adapter dual tuner (DVB-S2)
- Hauppauge Nova-TD dual tuner usb stick (DVB-T)

For this tvheadend setup I configured one dvb-s2 tuner and 1 dvb-t tuner. I also configured the cardserver specifics of course (and AFAIK it's not directly related to the cwc's).

I configured the muxes and services and some other stuff (can upload the config if needed) and I can record and watch tv.

One of the stress tests I always try to do is running as much recordings and watching as much channels as possible (for the moment the cardserver and the amount of tuners are the limit).
When I start recording on multiple channels (both dvb-s2 and dvb-t) it runs well for a while, but then it crashes. I had this issue for a while but didn't have the time yet to run it in a debugger.

I attach the debug log file and the gdb logging output. If necessary I can upload the tvheadend binary and the coredump as well, but the coredump is around 750MB in size).

kind regards

Jouk

Files

debug.log (124 KB) debug.log Jouk Hettema -, 2014-09-23 11:22
gdb.txt (3.08 KB) gdb.txt Jouk Hettema -, 2014-09-23 11:22
gdb.txt (1.98 KB) gdb.txt Crazy Fin, 2014-10-22 07:23
tvh.log.crash (3.92 KB) tvh.log.crash Crazy Fin, 2014-10-22 07:25
TVH crash log 20141023 (2.41 KB) TVH crash log 20141023 Crazy Fin, 2014-10-23 18:24
gdb.txt (2 KB) gdb.txt Crazy Fin, 2014-10-23 18:24
tvh.crash.log (3.52 KB) tvh.crash.log Crazy Fin, 2014-10-23 22:10
gdb.txt (1.93 KB) gdb.txt Crazy Fin, 2014-10-23 22:10
gdb.txt (1.59 KB) gdb.txt Crazy Fin, 2014-10-26 00:44
tvh.log (607 KB) tvh.log Crazy Fin, 2014-10-26 00:44
tvh.log (279 KB) tvh.log Crazy Fin, 2014-10-26 09:35
gdb.txt (8.09 KB) gdb.txt Crazy Fin, 2014-10-26 09:35
gdb.txt (8.08 KB) gdb.txt Crazy Fin, 2014-10-26 13:37
tvh.log (72.2 KB) tvh.log Crazy Fin, 2014-10-26 13:37
tvh.crash.log (335 KB) tvh.crash.log Crazy Fin, 2014-10-28 10:07
gdb.txt (3.69 KB) gdb.txt Crazy Fin, 2014-10-28 10:07
gdb.txt (1.95 KB) gdb.txt Crazy Fin, 2014-10-30 07:08
tvh.crash.log (1.29 MB) tvh.crash.log Crazy Fin, 2014-10-30 07:08
gdb.txt (1.79 KB) gdb.txt Crazy Fin, 2014-11-01 02:17
tvh.crash.log (300 KB) tvh.crash.log Crazy Fin, 2014-11-01 02:17
gdb.txt (2.54 KB) gdb.txt Crazy Fin, 2014-11-01 18:42
tvh.crash.log (319 KB) tvh.crash.log Crazy Fin, 2014-11-01 18:42
gdb.txt (2.37 KB) gdb.txt Crazy Fin, 2014-11-05 10:40
tvh.crash.log (25.5 MB) tvh.crash.log Crazy Fin, 2014-11-05 10:40
gdb.txt (1.94 KB) gdb.txt Crazy Fin, 2014-11-05 21:50
tvh.crash.log (149 KB) tvh.crash.log Crazy Fin, 2014-11-05 21:50
gdb.txt (3.18 KB) gdb.txt Crazy Fin, 2014-11-06 07:01
tvh.crash.log (199 KB) tvh.crash.log Crazy Fin, 2014-11-06 07:01
gdb.txt (3.18 KB) gdb.txt Crazy Fin, 2014-11-06 19:27
tvh.crash.log (498 KB) tvh.crash.log Crazy Fin, 2014-11-06 19:27
gdb.txt (3.18 KB) gdb.txt Crazy Fin, 2014-11-07 12:03
tvh.crash.log (2.41 MB) tvh.crash.log Crazy Fin, 2014-11-07 12:03
gdb.txt (3.32 KB) gdb.txt Crazy Fin, 2014-11-10 06:57
tvh.crash.log (266 KB) tvh.crash.log Crazy Fin, 2014-11-10 06:57
gdb.txt (3.34 KB) gdb.txt Crazy Fin, 2014-11-16 00:42
tvh.crash.log (155 KB) tvh.crash.log Crazy Fin, 2014-11-16 00:42

Related issues

Related to Bug #2296: TVH 3.9.1454 crashes with psi: tables count inconsistency (num 18, list 2)Fixed2014-08-21

Actions

History

#1

Updated by Rob vh about 10 years ago

last message before crash: psi: tables count inconsistency (num 37, list 36)
that's a message I've seen a few times.
I got (almost) rid of them by disabling "over the air EPG" at the adapter level. I grab EPG with a timer at a quiet moment.

#2

Updated by Jouk Hettema - about 10 years ago

will certainly try that. Got another crash this morning (test rig). As far as I could find it isn't on the dvb-t tuner but only on the dvb-s2 tuner. Will try to narrow it down to see if it has to do with a certain channel/mux.

will also try with and withou over the air EPG.

#3

Updated by Jaroslav Kysela about 10 years ago

  • Related to Bug #2296: TVH 3.9.1454 crashes with psi: tables count inconsistency (num 18, list 2) added
#4

Updated by Jaroslav Kysela about 10 years ago

  • Subject changed from tvheadend crashes while recording multiple channels to tvheadend crashes ( psi: tables count inconsistency (num 37, list 36) )
#5

Updated by Crazy Fin about 10 years ago

I am sorry to say that I got these crashes as well. I thought the memory corruption problem/bug was gone (see my case no 2296, https://tvheadend.org/issues/2328) but I recently got two crashes with the same line in the beginning of the crash report:

tvheadend: src/input/mpegts/mpegts_input.c:625: mpegts_input_table_dispatch: Assertion `0' failed.
2014-09-28 23:02:22.435 [ ERROR] psi: tables count inconsistency (num 12, list 5)
2014-09-28 23:02:22.435 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1576~g74794ab) [32982e45741d9bfce602a591b2758fe402867af3] CWD: /home/tvserver/tvheadend
2014-09-28 23:02:22.435 [ ALERT] CRASH: Fault address 0x55da (N/A)
2014-09-28 23:02:22.435 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /usr/lib/x86_64-linux-gnu/libavcodec.so.54 /usr/lib/x86_64-linux-gnu/libavutil.so.52 /usr/lib/x86_64-linux-gnu/libavformat.so.54 /usr/lib/x86_64-linux-gnu/libswscale.so.2 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libxvidcore.so.4 /usr/lib/x86_64-linux-gnu/libx264.so.142 /usr/lib/x86_64-linux-gnu/libvpx.so.1 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 /usr/lib/x86_64-linux-gnu/libvorbis.so.0 /usr/lib/x86_64-linux-gnu/libtheoraenc.so.1 /usr/lib/x86_64-linux-gnu/libtheoradec.so.1 /usr/lib/x86_64-linux-gnu/libspeex.so.1 /usr/lib/x86_64-linux-gnu/libschroedinger-1.0.so.0 /usr/lib/x86_64-linux-gnu/libop
2014-09-28 23:02:22.435 [ ALERT] CRASH: Register dump [23]: fefefefefefefe0000007f7d680082700000000000000008000000000000020200000000004d26b100000000004cf0800000000000000001000000000000000c00000000000055da00000000000055e900007f7de3b6837000007f7de6c8900000000000000000060000000000000000ffffffffffffffff00007f7dd57f95d800007f7de3a1dbb90000000000000202000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-09-28 23:02:22.435 [ ALERT] CRASH: STACKTRACE
2014-09-28 23:02:22.445 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x436989
2014-09-28 23:02:22.451 [ ALERT] CRASH: ??:0 0x7f7de42cb340
2014-09-28 23:02:22.451 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-09-28 23:02:22.451 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-09-28 23:02:22.460 [ ALERT] CRASH: ??:0 0x7f7de3a16a76
2014-09-28 23:02:22.468 [ ALERT] CRASH: ??:0 0x7f7de3a16b22
2014-09-28 23:02:22.479 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:625 0x46f62a
2014-09-28 23:02:22.491 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:881 0x46f7d8
2014-09-28 23:02:22.502 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x4119c1
2014-09-28 23:02:22.511 [ ALERT] CRASH: ??:0 0x7f7de42c3182

It seems that they are related to timer recordings. I did not have a crash for 7 days and as soon as I had one or more recordings starting the crashes also started.

#6

Updated by Jouk Hettema - about 10 years ago

hi,

I don't think it's just related to DVR. I just found out that while my wife was watching (via XBMC) a channel tvheadend crashed again. When I looked at the logging there was no recording scheduled.

Below the logging from that moment in time:

Sep 29 12:25:04 server tvheadend7525: htsp: Got connection from 192.168.11.58
Sep 29 12:25:04 server tvheadend7525: htsp: 192.168.11.58: Welcomed client software: XBMC Media Center (HTSPv8)
Sep 29 12:58:43 server tvheadend7525: mpegts: 11739V in 23e5 - tuning on STV090x Multistandard : DVB-S #1
Sep 29 12:58:44 server tvheadend7525: subscription: "192.168.11.58 [ XBMC Media Center ]" subscribing on "Investigation Discovery", weight: 150, adapter: "STV090x Multistandard : DVB-S #1", network: "23e5", mux: "11739V", provider: "M7 Group", service: "Investigation Discovery", hostname="192.168.11.58", username="<N/A>", client="XBMC Media Center"
Sep 29 13:02:16 server tvheadend7525: psi: tables count inconsistency (num 32, list 31)
Sep 29 13:02:16 server tvheadend7525: CRASH: Signal: 6 in PRG: /usr/bin/tvheadend (3.9.1573~gba136c5) [811d64693c5cc6d75b76c3c74dd9dde3dc2748b4] CWD: /
Sep 29 13:02:16 server tvheadend7525: CRASH: Fault address 0x1d65 (N/A)
Sep 29 13:02:16 server tvheadend7525: CRASH: Loaded libraries: /usr/lib64/libdvbcsa.so.1 /usr/lib64/libssl.so.10 /usr/lib64/libcrypto.so.10 /lib64/libz.so.1 /usr/lib64/libavahi-common.so.3 /usr/lib64/libavahi-client.so.3 /lib64/libdbus-1.so.3 /lib64/libpthread.so.0 /lib64/librt.so.1 /lib64/libdl.so.2 /lib64/libm.so.6 /lib64/libc.so.6 /lib64/libgssapi_krb5.so.2 /lib64/libkrb5.so.3 /lib64/libcom_err.so.2 /lib64/libk5crypto.so.3 /lib64/ld-linux-x86-64.so.2 /lib64/libkrb5support.so.0 /lib64/libkeyutils.so.1 /lib64/libresolv.so.2 /lib64/libselinux.so.1 /lib64/libnss_files.so.2
Sep 29 13:02:16 server tvheadend7525: CRASH: Register dump [23]: 00000000ffffffffffffffff000000000000000000000008000000000000020600000000004c06b100000000004bd080000000000000000900000000000000200000000000001d650000000000001d7400000000004bcfb000007f94edfc300000000000000000060000000000000000ffffffffffffffff00007f94db7f53e800007f94ec0ce6350000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
Sep 29 13:02:16 server tvheadend7525: CRASH: STACKTRACE
Sep 29 13:02:16 server tvheadend7525: CRASH: /mnt/storage0/workspace/tvheadend/src/trap.c:148 0x4338f6
Sep 29 13:02:16 server tvheadend7525: CRASH: ??:0 0x7f94ecacf710
Sep 29 13:02:16 server tvheadend7525: CRASH: gsignal+0x35 (/lib64/libc.so.6)
Sep 29 13:02:16 server tvheadend7525: CRASH: abort+0x175 (/lib64/libc.so.6)
Sep 29 13:02:16 server tvheadend7525: CRASH: ??:0 0x7f94ec0c775e
Sep 29 13:02:16 server tvheadend7525: CRASH: __assert_perror_fail+0x0 (/lib64/libc.so.6)
Sep 29 13:02:16 server tvheadend7525: CRASH: /mnt/storage0/workspace/tvheadend/src/input/mpegts/mpegts_input.c:625 0x46a69b
Sep 29 13:02:16 server tvheadend7525: CRASH: /mnt/storage0/workspace/tvheadend/src/input/mpegts/mpegts_input.c:769 0x46aef3
Sep 29 13:02:16 server tvheadend7525: CRASH: /mnt/storage0/workspace/tvheadend/src/wrappers.c:125 0x410551
Sep 29 13:02:16 server tvheadend7525: CRASH: ??:0 0x7f94ecac79d1
Sep 29 13:02:16 server tvheadend7525: CRASH: clone+0x6d (/lib64/libc.so.6)
Sep 29 13:02:21 server tvheadend10291: main: Log started
Sep 29 13:02:22 server tvheadend10291: charset: 138 entries loaded

Because I have monit running for this specific situation there was only a 15 second gap between the crash and the moment she could continue watching. For now that'll work, but I think it needs to be fixed because AFAIK it's a rather fundamental part where it goes wrong (gut feeling...).

If I need to setup a test environment for collecting coredumps (or upload the coredumps I already have) and other information, please let me know.

kind regards

Jouk

#7

Updated by Jaroslav Kysela about 10 years ago

The bad thing is that it is really difficult to debug this. Do you use any other EPG source ? Timeshift ? Other tvh features ?

#8

Updated by Crazy Fin about 10 years ago

I understand the difficulty of debugging this... :-(

I use ONLY "EIT:DVB Grabber" under the tab "Configuration" -> "Channel/EPG" -> "EPG Grabber". No other EPG Grabbers at all.

I have even disabled "Over-The-Air EPG" for each of my 8 tuners under "Configuration" -> "DVB Inputs" -> "TV adapters".

Under "Configuration" -> "DVB Inputs" -> "Networks" I have "Network Discovery" and "Idle Scan unchecked" and "Skip Initial Scan" checked.

Under "Configuration" -> "DVB Inputs" -> "Muxes" I have set all my enabled Muxes to "Only EIT" on "EPG Scan"

Under "Configuration" -> "Channel/EPG" -> "EPG Grabber" I have set "Force initial EPG scan at startup:" to disabled and the Cron multi-line has the default setting:

"_# Default config (02:04 and 14:04 everyday)
4 2 * * *
4 14 * * *_"

"EPG scan timeout in seconds (30-7200):" is set to 900

"Periodic save EPG to disk" is set to 4 and the 3 tick boxes about updating channel name, number and icon is set to ticked.

#9

Updated by Crazy Fin about 10 years ago

Forgot to mention that I have Time Shift disabled.

#10

Updated by Crazy Fin about 10 years ago

As I mentioned above I have DISABLED all EPG modules except the EIT: DVB Grabber. That is the ONLY tick box I have enabled.

However, I can see in the startup log that TVH still creates all EPG modules:

2014-09-29 22:32:02.519 [ INFO] epggrab: module eit created
2014-09-29 22:32:02.519 [ INFO] epggrab: module uk_freesat created
2014-09-29 22:32:02.519 [ INFO] epggrab: module uk_freeview created
2014-09-29 22:32:02.519 [ INFO] epggrab: module viasat_baltic created
2014-09-29 22:32:02.522 [ INFO] epggrab: module opentv-ausat created
2014-09-29 22:32:02.522 [ INFO] epggrab: module opentv-skyuk created
2014-09-29 22:32:02.522 [ INFO] epggrab: module opentv-skyit created
2014-09-29 22:32:02.524 [ INFO] epggrab: module pyepg created
2014-09-29 22:32:02.524 [ INFO] epggrab: module xmltv created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_ch_search created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_fi created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_uk_rt created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_hr created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_is created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_pt_meo created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_es_laguiatv created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_eu_egon created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_se_swedb created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_in created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_na_dtv created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_dk_dr created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_no_gfeed created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_uk_bleb created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_it created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_il created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_fr_kazer created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_fr created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_se_tvzon created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_eu_epgdata created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_na_dd created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_huro created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_za created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_ar created
2014-09-29 22:32:03.445 [ INFO] epggrab: module /usr/bin/tv_grab_combiner created
_

#11

Updated by Lee Pru about 10 years ago

Just wanted to say that I'm also getting this only in last update today. I updated from version 3.9.383 to 3.9.1588 and tvheadend does not stay up for more than 30-60 minutes. The syslog is showing:

Sep 30 04:08:18 mysrv tvheadend[24601]: psi: tables count inconsistency (num 44, list 43)
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: Signal: 6 in PRG: /usr/bin/tvheadend (3.9.1588~g8602487) [7eb952197237c307de4b5a3334213a7870a0710f] CWD: /  
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: Fault address 0x6e00006019 (N/A)
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: Loaded libraries: /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/liburiparser.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libnss_compat.so.2 /lib/x86_64-linux-gnu/libnsl.so.1 /lib/x86_64-linux-gnu/libnss_nis.so.2 /lib/x86_64-linux-gnu/libnss_files.so.2  
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: Register dump [23]: 00007f74e6ff570060206e6f697472650000000000000008000000000000020200000000004c7b4500000000004c57d000007f74f612afe700000000000002710000000000006019000000000000604900007f74f612afe700007fff05791f8300000000000000060000000000000000ffffffffffffffff00007f74e6ff445800007f74f60111a50000000000000202000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: STACKTRACE
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: ??:0 0x435666
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: ??:0 0x7f74f6803030
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: gsignal+0x35  (/lib/x86_64-linux-gnu/libc.so.6)
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: abort+0x180  (/lib/x86_64-linux-gnu/libc.so.6)
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: __assert_fail+0xf1  (/lib/x86_64-linux-gnu/libc.so.6)
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: ??:0 0x46c8f3
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: ??:0 0x46e02c
Sep 30 04:08:18 mysrv tvheadend[24601]: CRASH: ??:0 0x411121
Sep 30 04:08:19 mysrv tvheadend[24601]: CRASH: ??:0 0x7f74f67fab50
Sep 30 04:08:19 mysrv tvheadend[24601]: CRASH: clone+0x6d  (/lib/x86_64-linux-gnu/libc.so.6)

I am running Debian 7.6 (x86_64).

Hope this helps in making this a higher priority. Let me know how else I can help.

Thanks!

#12

Updated by Jaroslav Kysela about 10 years ago

I have a good experience with valgrind to detect the wrong memory accesses, so if you can trigger this bug in this short time and you're on fast machine, you may try to run the tvh under valgrind (compliled from source using all debug symbols). I'm using this cmd line:

valgrind --leak-check=full --show-reachable=yes ./build.linux/tvheadend <your tvh options>
#13

Updated by Mark Clarkstone about 10 years ago

  • Priority changed from Normal to High

- Changing this to high as it affects more than a few people.

Been running tvh in valgrind with a single tuner for a good 17+ hours now while running another script (perexg gave me) that switches channels and it has yet to crash! :s

On my second machine, I've set all 7 tuners to use and it appears to be working fine here too.
I've only ever experienced this crash when using xbmc.

It's a strange one!

#14

Updated by Jouk Hettema - about 10 years ago

hi all,

got a crash this morning with the 1600 version of tvheadend. It didn't generate a core file and I just have the log message.

When I tried to setup a valgrind/debug configuration I found out there were quite some continuity errors on the interface. If possible I will replace the LNB tomorrow which is the most likely cause of these errors. When that's done I'll try to setup a cleaner solution to test tvheadend in valgrind.

regards

Jouk

ps: it's a strange one indeed. But, I have the feeling it only happens on DVB-S2 interfaces. Would it be helpfull if we describe the different setups in some detail?

#15

Updated by Jaroslav Kysela about 10 years ago

The commit c72cadbd5da151b748f58bd4eb9f6de1d111489a might make a difference.. But the whole coverity fixes can play a role to fix this bug.

#16

Updated by Jouk Hettema - about 10 years ago

didn't replace my lnb, but giving the newest version on git a go. Just compiled it and now it's recording 4 channels which gave me the most psi crashes.

#17

Updated by Jouk Hettema - about 10 years ago

afraid it isn't fixed :( got the error after 30 minutes of recording.

Oct 4 11:24:39 server tvheadend20504: psi: tables count inconsistency (num 40, list 2)
Oct 4 11:24:39 server tvheadend20504: CRASH: Signal: 6 in PRG: /usr/bin/tvheadend (3.9.1721~g9734af1) [2a69b28aaab670af265e8fb47d9e9af63d1755da] CWD: /
Oct 4 11:24:39 server tvheadend20504: CRASH: Fault address 0x5018 (N/A)
Oct 4 11:24:39 server tvheadend20504: CRASH: Loaded libraries: /usr/lib64/libdvbcsa.so.1 /usr/lib64/libssl.so.10 /usr/lib64/libcrypto.so.10 /lib64/libz.so.1 /usr/lib64/libavahi-common.so.3 /usr/lib64/libavahi-client.so.3 /lib64/libdbus-1.so.3 /lib64/libpthread.so.0 /lib64/librt.so.1 /lib64/libdl.so.2 /lib64/libm.so.6 /lib64/libc.so.6 /lib64/libgssapi_krb5.so.2 /lib64/libkrb5.so.3 /lib64/libcom_err.so.2 /lib64/libk5crypto.so.3 /lib64/ld-linux-x86-64.so.2 /lib64/libkrb5support.so.0 /lib64/libkeyutils.so.1 /lib64/libresolv.so.2 /lib64/libselinux.so.1 /lib64/libnss_files.so.2
Oct 4 11:24:39 server tvheadend20504: CRASH: Register dump [23]: 00000000ffffffffffffffff000000000000000000000008000000000000020600000000004c44d100000000004c0ea0000000000000000200000000000000280000000000005018000000000000503b00000000004c0db800007fe0918dc00000000000000000060000000000000000ffffffffffffffff00007fe07effbf2800007fe08f9e76350000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
Oct 4 11:24:39 server tvheadend20504: CRASH: STACKTRACE
Oct 4 11:24:39 server tvheadend20504: CRASH: /mnt/storage0/workspace/tvheadend/src/trap.c:148 0x434926
Oct 4 11:24:39 server tvheadend20504: CRASH: ??:0 0x7fe0903e8710
Oct 4 11:24:39 server tvheadend20504: CRASH: gsignal+0x35 (/lib64/libc.so.6)
Oct 4 11:24:39 server tvheadend20504: CRASH: abort+0x175 (/lib64/libc.so.6)
Oct 4 11:24:39 server tvheadend20504: CRASH: ??:0 0x7fe08f9e075e
Oct 4 11:24:39 server tvheadend20504: CRASH: __assert_perror_fail+0x0 (/lib64/libc.so.6)
Oct 4 11:24:39 server tvheadend20504: CRASH: /mnt/storage0/workspace/tvheadend/src/input/mpegts/mpegts_input.c:625 0x46d8db
Oct 4 11:24:39 server tvheadend20504: CRASH: /mnt/storage0/workspace/tvheadend/src/input/mpegts/mpegts_input.c:773 0x46e133
Oct 4 11:24:39 server tvheadend20504: CRASH: /mnt/storage0/workspace/tvheadend/src/wrappers.c:125 0x410731
Oct 4 11:24:39 server tvheadend20504: CRASH: ??:0 0x7fe0903e09d1
Oct 4 11:24:39 server tvheadend20504: CRASH: clone+0x6d (/lib64/libc.so.6)

#18

Updated by Rob vh about 10 years ago

Oct 4 19:13:54 sat tvheadend1056: psi: tables count inconsistency (num 24, list 15)
Oct 4 19:13:54 sat tvheadend1056: CRASH: Signal: 6 in PRG: tvheadend (3.9.1721~g9734af1) [567c6ad7fb2e0ea8ce152d0cea9ca0ab0c1e0
4b8] CWD: /
Oct 4 19:13:54 sat tvheadend1056: CRASH: Fault address 0x7000000420 (N/A)
Oct 4 19:13:54 sat tvheadend1056: CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypt
o.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/liburiparser.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_
64-linux-gnu/libavahi-client.so.3 /usr/lib/x86_64-linux-gnu/libavcodec.so.53 /usr/lib/x86_64-linux-gnu/libavutil.so.51 /usr/lib/x8
6_64-linux-gnu/libavformat.so.53 /usr/lib/x86_64-linux-gnu/libswscale.so.2 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-
gnu/libpthread.so.0 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-
linux-gnu/libc.so.6 /usr/lib/libvpx.so.1 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 /usr/lib/x86_64-linux-gnu/libvorbis.so.0 /usr
/lib/x86_64-linux-gnu/libtheoraenc.so.1 /usr/lib/x86_64-linux-gnu/libtheoradec.so.1 /usr/lib/x86_64-linux-gnu/libspeex.so.1 /usr/l
ib/libschroedinger-1.0.so.0 /usr/lib/libgsm.so.1 /usr/lib/x86_64-linux-gnu/libva.so.1 /lib/x86_64-linux-gnu/libbz2.so.1.0 /lib64/l
d-linux-x86-64.so.2
Oct 4 19:13:54 sat tvheadend1056: CRASH: Register dump [23]: fefefefefefefefffeff8fa0ff092d630000000000000008000000000000020200
000000004d0dc500000000004cea5000007f8e0b7fd2c000000000000000b6000000000000042000000000000007f600000000004ce89800007f8e219570000000
0000000000060000000000000000ffffffffffffffff00007f8e0b7fd0d800007f8e1e40e0d5000000000000020200000000000000330000000000000000000000
0000000000fffffffe7ffbba170000000000000000
Oct 4 19:13:54 sat tvheadend1056: CRASH: STACKTRACE
Oct 4 19:13:54 sat tvheadend1056: psi: tables count inconsistency (num 24, list 15)
Oct 4 19:13:54 sat kernel: [22537.676298] init: tvheadend main process (1056) killed by ABRT signal
Oct 4 19:13:54 sat kernel: [22537.676361] init: tvheadend main process ended, respawning
Oct 4 19:13:54 sat tvheadend4325: main: Log started

#19

Updated by Mark Clarkstone about 10 years ago

I'm still running tvheadend from within valgrind trying to get it to crash, but it just wont!

Seeing as you guys have the crash more frequent than I do, do you think you could run valgrind with your setup and post the results when it crashes?

Mine crashed again last night after running for about 24 hours, so I set it up through valgrind again, but it could be a while before it finally does bork out!

#20

Updated by Jouk Hettema - about 10 years ago

my server is to small in cpu resources for running tvheadend in valgrind. I can start it and it runs, but it's hardly useable.

As I haven't seen this error in a couple of days I would like to lower the priority if that's ok with you guys?

#21

Updated by Crazy Fin about 10 years ago

Please do not lower the priority. I got a crash like this one (psi: tables count inconsistency (num 24, list 15)) last night after 18 hours of running latest build 3.9.1743.

During the coming weekend I will try to setup to run TVH in Valgrind. Would be great to "nail" this memory bug, except for this bug, TVH is now running really nice (at least for me. :-) )

#22

Updated by Mark Clarkstone about 10 years ago

I'd just like to add, I'm been running build 3.9.1738 now for just over 20 hours without any errors so far. Newer builds (1741 in this case) keep crashing for me not sure if related to this bug.. gdb log here. http://sprunge.us/aUME

#23

Updated by Jaroslav Kysela about 10 years ago

I added in v3.9-1901-g43e805d the table number / list inconsitency check to more subroutines. I hope that we can do different crashes with this patch to locate the bug . Please, compile with --enable-trace (as always) and provide crash reports..

#24

Updated by Crazy Fin about 10 years ago

Ok great since I am still getting these errors until the version 3.9.1889.

To compile with --enable-trace:

Do I enable the trace functionality with "./configure --enable-trace" before I do the actual compilation with "make -j5"?

#25

Updated by Crazy Fin about 10 years ago

Sorry I also forgot to ask that when I start TVH with

"./build.linux/tvheadend -l tvh.log"

should I still start it with the debug and trace parameters as well?

Like
"./build.linux/tvheadend -l tvh.log --debug all --trace all" or what debug settings should I start TVH with?

#26

Updated by Jaroslav Kysela about 10 years ago

Crazy Fin wrote:

Sorry I also forgot to ask that when I start TVH with

"./build.linux/tvheadend -l tvh.log"

should I still start it with the debug and trace parameters as well?

Like
"./build.linux/tvheadend -l tvh.log --debug all --trace all" or what debug settings should I start TVH with?

It's sufficient to configure the tvh build with --enable-trace otherwise the extra checks are not included to the code. It's not required to enable trace at runtime.

#27

Updated by Crazy Fin about 10 years ago

OK thanks so I´ll do

./configure --enable-trace

and then start TVH with

./build.linux/tvheadend -l tvh.log --debug all

or is even sufficient with

./build.linux/tvheadend -l tvh.log

I had a crash again last night (running 3.9-1901) and I will collect the gdb coredump tonight when I´m back at home. 9 out of 10 times these crashes has happened during midnight and approx 6 am in the morning.

#28

Updated by Jaroslav Kysela about 10 years ago

Note I fixed another issues (which looks very promising regarding this bug). Please, upgrade to v3.9-1911-ge1d8c2a .

#29

Updated by Crazy Fin about 10 years ago

Ok great! I have now pulled in v3.9-1911-ge1d8c2a and I am running it now with

./build.linux/tvheadend -l tvh.log --debug all

as well as core dump enabled as per https://tvheadend.org/projects/tvheadend/wiki/Debugging

Let's keep fingers crossed that you have been able to fix this annoying bug. :-)

Will report back tomorrow morning.

#30

Updated by Crazy Fin about 10 years ago

Sorry but still crashing... :-(

Again at midnight. This is so strange since TVH is running during the day pretty nice. 99% of the crashes I get happens between midnight and 6 am.

tvh.log (cut down the log to few minutes before crash) and core dump (gdb.txt) attached.

#31

Updated by Jaroslav Kysela about 10 years ago

Could you enable 'mpegts' trace to get more debug logs ? --trace mpegts

#32

Updated by Crazy Fin about 10 years ago

Sure, will do that tonight when back at home from work.

./build.linux/tvheadend -l tvh.log --debug all --trace mpegts

or is

./build.linux/tvheadend -l tvh.log --trace mpegts

enough?

#33

Updated by Mark Clarkstone about 10 years ago

Crazy Fin wrote:

Sorry but still crashing... :-(

Again at midnight. This is so strange since TVH is running during the day pretty nice. 99% of the crashes I get happens between midnight and 6 am.

tvh.log (cut down the log to few minutes before crash) and core dump (gdb.txt) attached.

Funny you should mention that this is what was happening to me as well, always in the morning.

#34

Updated by Crazy Fin about 10 years ago

Strange, now when I have been testing TVH to catch the crashes I have started TVH with either

./build.linux/tvheadend -l tvh.log --debug all --trace mpegts
./build.linux/tvheadend -l tvh.log --debug all
or just
./build.linux/tvheadend -l tvh.log

and now I get a crash everytime I stop TVH from running with CTRL-C.

2014-10-22 18:34:09.616 [ ALERT] CRASH: Signal: 11 in PRG: ./build.linux/tvheadend (3.9.1911~ge1d8c2a) [b4c36d083ea283617303afdf6e4a537d93c2852f] CWD: /home/tvserver/tvheadend
2014-10-22 18:34:09.616 [ ALERT] CRASH: Fault address (nil) (Address not mapped)

I guess this is not a problem to bother about at this stage.

I am now running latest TVH build 3.9.1926~g917fb80 --trace mpegts enabled. Lets see if we can get usable crash data tonight.

#35

Updated by Crazy Fin about 10 years ago

Alright got another crash after 2 hours of running this morning.

Final part of tvh.log attached (let me know if you need the full log file) and gdb.txt attached.

Hope this gives you some idea on where the memory corruption happens.

Let me know if you need further debugging.

#36

Updated by Crazy Fin about 10 years ago

2 hours more and another crash.

New tvh.crash.log and gdb.txt attached.

#37

Updated by Jaroslav Kysela about 10 years ago

Thanks. I added some more mpegts traces to the source. I know that this debugging is annoying, but if you can trigger this bug, it will help to others..

Also, please, provide the more complete log (if possible) - the grep for :mpegts: lines (say last 10 minutes) and complete last lines from log (like now) might be sufficient. Thanks.

#38

Updated by Crazy Fin about 10 years ago

I am on 3.9.1950.

Got a crash now after almost 2 days without any crashes. This crash looks different from the previous crashes though.

2014-10-26 00:31:16.812 [ ALERT] CRASH: Signal: 11 in PRG: ./build.linux/tvheadend (3.9.1950~g596b4e0) [43f7b8123372a07b0a6b4600a6fb718fca84a9f4] CWD: /home/tvserver/tvheadend
2014-10-26 00:31:16.812 [ ALERT] CRASH: Fault address 0x90 (Address not mapped)
2014-10-26 00:31:16.812 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-10-26 00:31:16.812 [ ALERT] CRASH: Register dump [23]: 000000000000000000007fb0f000007800000000000000050000000000000006000000000000000000007fb0c801c3e000000000019d11e00000000001ace4c00000000001ace4c000007fb0c8020d6c000000000000001200007fb0c8020d7000007fb0f0000af000007fb0f0000bd0000000000000000000007fb135ffa7900000000000474497000000000001028700000000000000330000000000000004000000000000000efffffffe7ffbba170000000000000090
2014-10-26 00:31:16.812 [ ALERT] CRASH: STACKTRACE
2014-10-26 00:31:16.832 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x437ab9
2014-10-26 00:31:16.838 [ ALERT] CRASH: ??:0 0x7fb148adf340
2014-10-26 00:31:16.849 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:810 0x474497
2014-10-26 00:31:16.860 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411121
Segmentation fault (core dumped)

gdb.txt as well as tvh.log attached (15 last minutes of log)

Let me know if the log is not enough.

#39

Updated by Crazy Fin about 10 years ago

Another crash during the night.
The crashes definately looks very different now:

2014-10-26 02:56:00.447 [ ALERT]:CRASH: Signal: 11 in PRG: ./build.linux/tvheadend (3.9.1952~gb80c232) [e7fc8b76fe892429e1a3e37523b393162ece9b48] CWD: /home/tvserver/tvheadend
2014-10-26 02:56:00.447 [ ALERT]:CRASH: Fault address 0x1c5bacba8 (Address not mapped)
2014-10-26 02:56:00.447 [ ALERT]:CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-10-26 02:56:00.447 [ ALERT]:CRASH: Register dump [23]: 000000000000000000000000022da800000000000005d0000000000000000246000000000000000000000000000003e800007fff83ba703000000000004b7a8900000001c5bacba800000000ffffffff000000000000000000000001c5bacba800007fc37757b78800000000024a576000000000024a576000007fff83ba674000000000004231e9000000000001020200000000000000330000000000000006000000000000000efffffffe7ffbba1500000001c5bacba8
2014-10-26 02:56:00.447 [ ALERT]:CRASH: STACKTRACE
2014-10-26 02:56:00.463 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4380d9
2014-10-26 02:56:00.478 [ ALERT]:CRASH: ??:0 0x7fc377aa0340
2014-10-26 02:56:00.495 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/streaming.c:256 0x4231e9
2014-10-26 02:56:00.510 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/plumbing/globalheaders.c:48 0x45b9f2
2014-10-26 02:56:00.526 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/plumbing/globalheaders.c:354 0x45bfa9
2014-10-26 02:56:00.543 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/profile.c:731 0x44c515
2014-10-26 02:56:00.559 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/dvr/dvr_rec.c:124 0x461257
2014-10-26 02:56:00.577 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/dvr/dvr_db.c:929 0x45f1d9
2014-10-26 02:56:00.593 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/main.c:410 0x407ecc
2014-10-26 02:56:00.593 [ ALERT]:CRASH: __libc_start_main+0xf5 (/lib/x86_64-linux-gnu/libc.so.6)

I also see a LOT of errors like this:

2014-10-26 02:40:53.677 [ ERROR]:parser: transport stream TEXTSUB, DTS discontinuity. DTS = 361254463, last = 360826782

Last 10-15 mins of tvh.log attached as well as the gdb.txt output.

#40

Updated by Crazy Fin about 10 years ago

Crashed again after few hours of running.

The crashes do indeed look different now. Should I open a new bug ticket or shall we stay on this one?

===================

2014-10-26 11:46:00.439 [ ALERT] CRASH: Signal: 11 in PRG: ./build.linux/tvheadend (3.9.1952~gb80c232) [e7fc8b76fe892429e1a3e37523b393162ece9b48] CWD: /home/tvserver/tvheadend
2014-10-26 11:46:00.439 [ ALERT] CRASH: Fault address 0x124c4d17b (Address not mapped)
2014-10-26 11:46:00.439 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-10-26 11:46:00.439 [ ALERT] CRASH: Register dump [23]: 0000000000000000000000000166f2f000007fe9d000007800007fe9d0000070000000000000000000000000000003e800007fff389a0ca000000000004b7a890000000124c4d17b00000000ffffffff00000000000000000000000124c4d17b00007fea4dbdf78800000000022c3a6000000000022c3a6000007fff389a03b000000000004231e9000000000001020200000000000000330000000000000006000000000000000efffffffe7ffbba150000000124c4d17b
2014-10-26 11:46:00.439 [ ALERT] CRASH: STACKTRACE
2014-10-26 11:46:00.467 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4380d9
2014-10-26 11:46:00.473 [ ALERT] CRASH: ??:0 0x7fea4e104340
2014-10-26 11:46:00.482 [ ALERT] CRASH: /home/tvserver/tvheadend/src/streaming.c:256 0x4231e9
2014-10-26 11:46:00.492 [ ALERT] CRASH: /home/tvserver/tvheadend/src/plumbing/globalheaders.c:48 0x45b9f2
2014-10-26 11:46:00.500 [ ALERT] CRASH: /home/tvserver/tvheadend/src/plumbing/globalheaders.c:354 0x45bfa9
2014-10-26 11:46:00.509 [ ALERT] CRASH: /home/tvserver/tvheadend/src/profile.c:731 0x44c515
2014-10-26 11:46:00.518 [ ALERT] CRASH: /home/tvserver/tvheadend/src/dvr/dvr_rec.c:124 0x461257
2014-10-26 11:46:00.527 [ ALERT] CRASH: /home/tvserver/tvheadend/src/dvr/dvr_db.c:929 0x45f1d9
2014-10-26 11:46:00.535 [ ALERT] CRASH: /home/tvserver/tvheadend/src/main.c:410 0x407ecc
2014-10-26 11:46:00.535 [ ALERT] CRASH: __libc_start_main+0xf5 (/lib/x86_64-linux-gnu/libc.so.6)
Segmentation fault (core dumped)

=========================

I've started to see a LOT of these errors after upgrading to 3.9.1952:

2014-10-26 11:43:22.496 [ ERROR] parser: transport stream TEXTSUB, DTS discontinuity. DTS = 128581718, last = 128300918

Core dump (bt full into gdb.txt9 attached as well as the last 10-15 minutes of debug and trace log from tvh.log.

#41

Updated by Jaroslav Kysela about 10 years ago

The above three patches are unrelated to original issue. I fixed it in v3.9-1953-gaf9cb5b . Thanks.

#42

Updated by Jaroslav Kysela about 10 years ago

Above three crashes not patches - sorry..

#43

Updated by Jaroslav Kysela about 10 years ago

  • Assignee set to Jaroslav Kysela
#44

Updated by Crazy Fin about 10 years ago

Alright, got a crash this morning again with your new tracing functionality. I hope you can see something else now that will help you nail this memory corruption bug. :-)

Attached is the last hour of the tvh.log and the gdb.txt output.

Let me know if you need any more logging. It seems to me that these crashes happens at least 1 or 2 times per day.

============
2014-10-28 08:05:01.290 [ TRACE]:mpegts: table: mux 0x1752450 free bat 48/F8 (72) pid 0011 (17)
2014-10-28 08:05:01.290 [ TRACE]:mpegts: table: mux 0x1752450 free क 48/F8 (72) pid 0011 (17)
2014-10-28 08:05:01.291 [ ALERT]:CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1968~g6b5c0cd) [33863255be35f630f28ebd7a91027915b6105ddd] CWD: /home/tvserver/tvheadend
2014-10-28 08:05:01.291 [ ALERT]:CRASH: Fault address 0x3e800003f96 (N/A)
2014-10-28 08:05:01.291 [ ALERT]:CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-10-28 08:05:01.291 [ ALERT]:CRASH: Register dump [23]: 30363033623731306f6974707572726f0000000000000008000000000000020600007f8ddbffe4a00000000000000007000000000000006400000000000000070000000000003f960000000000003fa500007f8ddbffe690000000000000006400000000000000060000000000000000ffffffffffffffff00007f8ddbffe2f800007f8deb063bb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-10-28 08:05:01.291 [ ALERT]:CRASH: STACKTRACE
2014-10-28 08:05:01.304 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x438379
2014-10-28 08:05:01.313 [ ALERT]:CRASH: ??:0 0x7f8deb911340
2014-10-28 08:05:01.313 [ ALERT]:CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-10-28 08:05:01.313 [ ALERT]:CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-10-28 08:05:01.323 [ ALERT]:CRASH: ??:0 0x7f8deb0a0e14
2014-10-28 08:05:01.334 [ ALERT]:CRASH: ??:0 0x7f8deb0ad0ee
2014-10-28 08:05:01.349 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_table.c:147 0x47b159
2014-10-28 08:05:01.362 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:793 0x474fc5
2014-10-28 08:05:01.380 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:895 0x47515a
2014-10-28 08:05:01.392 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411121
2014-10-28 08:05:01.401 [ TRACE]:mpegts: table: mux 0x1752450 free क 48/F8 (72) pid 0011 (17)

#45

Updated by Jaroslav Kysela about 10 years ago

Another cleanup is in v3.9-1970-g863ea6f which may improve something. Unfortunately, nor the last trace helps me to identify the issue.

#46

Updated by Crazy Fin about 10 years ago

Sorry but after 12 hours of running I got a new crash...

Same type of crash as previously and gdb.txt as well as last 2 hours of tvh.log attached.

==============
tvheadend: src/input/mpegts/mpegts_input.c:644: mpegts_input_table_dispatch: Assertion `0' failed.
2014-10-30 05:09:27.290 [ ERROR] psi: tables count inconsistency (num 9, list 7)
2014-10-30 05:09:27.291 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1972~g9bb73c1) [3d7f51881d56229aa6d86e038bb5ff368c3b2962] CWD: /home/tvserver/tvheadend
2014-10-30 05:09:27.291 [ ALERT] CRASH: Fault address 0x3e8000055ff (N/A)
2014-10-30 05:09:27.291 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-10-30 05:09:27.291 [ ALERT] CRASH: Register dump [23]: fefefefefefefe0000007fabe8024d900000000000000008000000000000020600000000004d77b900000000004d3d600000000000000007000000000000000900000000000055ff000000000000561b00007fac837c937000007fac8540800000000000000000060000000000000000ffffffffffffffff00007fac717f956800007fac8367ebb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-10-30 05:09:27.291 [ ALERT] CRASH: STACKTRACE
2014-10-30 05:09:27.302 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4383b9
Aborted (core dumped)

#47

Updated by Crazy Fin about 10 years ago

Another crash again but this time with a different crash signal?

==========================

2014-10-31 20:54:29.831 [ ERROR] parser: transport stream TEXTSUB, DTS discontinuity. DTS = 1715572851, last = 1715572852
2014-10-31 21:58:17.289 [ ERROR] mpegts: table: mux 0x23d9490 count inconsistency (num 15, list 8)
2014-10-31 21:58:17.289 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1977~gd347c3d) [d33d04205cd91dd1a6d51e014c8711997eff57f7] CWD: /home/tvserver/tvheadend
2014-10-31 21:58:17.289 [ ALERT] CRASH: Fault address 0x3e800000c5a (N/A)
2014-10-31 21:58:17.289 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-10-31 21:58:17.289 [ ALERT] CRASH: Register dump [23]: 0000000000000000000000000400000100000000000000080000000000000206000000000000000000007f925c00c27000000000023d949000000000021fa4d00000000000000c5a0000000000000c6600000000023d95e000000000023d949000000000000000060000000000000000ffffffffffffffff00007f92dfffe62800007f92ee21bbb90000000000000206a7e300000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-10-31 21:58:17.289 [ ALERT] CRASH: STACKTRACE
2014-10-31 21:58:17.299 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4383b9
2014-10-31 21:58:17.308 [ ALERT] CRASH: ??:0 0x7f92eeac9340
2014-10-31 21:58:17.308 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-10-31 21:58:17.308 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-10-31 21:58:17.322 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_table.c:39 0x47ad56
2014-10-31 21:58:17.334 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:701 0x475629
2014-10-31 21:58:17.345 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411121
2014-10-31 21:58:17.354 [ ALERT] CRASH: ??:0 0x7f92eeac1182
Aborted (core dumped) ==========================

Attached: BT FULL output of coredump in gdb.txt and last hour of tvh.log attached.

#48

Updated by Crazy Fin about 10 years ago

Crashed again with same crash signal as before.
Logs attached.

I see a LOT of these errors in the log today:
2014-11-01 16:18:12.433 [ DEBUG]:psi: PID 0012 CC error 1 != 15
2014-11-01 16:18:12.433 [ DEBUG]:psi: PID 0012 CC error 13 != 2
2014-11-01 16:18:12.433 [ DEBUG]:psi: PID 0012 CC error 10 != 14
2014-11-01 16:18:12.949 [ TRACE]:mpegts: pid 0012 cc err 3 != 11
2014-11-01 16:18:12.949 [ TRACE]:mpegts: pid 0012 cc err 13 != 4

==========================

tvheadend: src/input/mpegts/mpegts_input.c:644: mpegts_input_table_dispatch: Assertion `0' failed.
2014-11-01 17:10:55.798 [ ERROR] psi: tables count inconsistency (num 27, list 24)
2014-11-01 17:10:55.799 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1987~gc053acd) [5f63503b8989ef1d18b18c8021adbf0f5b1c1b14] CWD: /home/tvserver/tvheadend
2014-11-01 17:10:55.799 [ ALERT] CRASH: Fault address 0x3e800003c4a (N/A)
2014-11-01 17:10:55.799 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-01 17:10:55.799 [ ALERT] CRASH: Register dump [23]: fefefefefefefe000000000001bc69100000000000000008000000000000020200000000004d77d900000000004d3d800000000000000018000000000000001b0000000000003c4a0000000000003c5d00007fd2afa1637000007fd2b165500000000000000000060000000000000000ffffffffffffffff00007fd2a4ff84d800007fd2af8cbbb90000000000000202000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-01 17:10:55.799 [ ALERT] CRASH: STACKTRACE
2014-11-01 17:10:55.809 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4383b9
2014-11-01 17:10:55.816 [ ALERT] CRASH: ??:0 0x7fd2b0179340
2014-11-01 17:10:55.816 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-01 17:10:55.816 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-01 17:10:55.828 [ ALERT] CRASH: ??:0 0x7fd2af8c4a76
2014-11-01 17:10:55.836 [ ALERT] CRASH: ??:0 0x7fd2af8c4b22
2014-11-01 17:10:55.850 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:644 0x47512a
2014-11-01 17:10:55.862 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:899 0x47519a
2014-11-01 17:10:55.872 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411121
2014-11-01 17:10:55.883 [ ALERT] CRASH: ??:0 0x7fd2b0171182
2014-11-01 17:10:55.884 [ ALERT] CRASH: clone+0x6d (/lib/x86_64-linux-gnu/libc.so.6)
Aborted (core dumped)

==========================

#49

Updated by Crazy Fin about 10 years ago

This obsveration is quite interesting!

I have had the "Idle Scan" turned off for 2 days now and NOT a crash yet...

#50

Updated by Jouk Hettema - about 10 years ago

That's the case with me as well.

Haven't seen a crash in days/weeks, so my wife and kids are happy campers. However, afaik I have never enabled the idle scan feature on my cards.

Soon I'll have a third sat tuner in my system so I can do some more testing.

#51

Updated by Mark Clarkstone about 10 years ago

Crazy Fin wrote:

This obsveration is quite interesting!

I have had the "Idle Scan" turned off for 2 days now and NOT a crash yet...

Interesting, I have "Idle Scan Muxes" enabled on all networks and no crashes recently (currently running 1992).

I'm currently using the following:-

PCTV Nanostick 290e single tuner DVB-T2 (USB)
Hauppauge Nova-TD dual tuner DVB-T (USB)
Tevii S471 single tuner DVB-S2 (Pcie)
Tmax TAS2101 (A.K.A TBS 5922) single tuner DVB-S2 (USB)

Running kernel 3.10.38 with lja's linuxtv media tree from github.

I was previously using the above with a Hauppauge stick (same chipset as the TD), an RTL DVB stick (you know those Chinese cheapies with FM/DAB) all DVB-T and a Technisat Skystar DVB-S2 USB HD, when I experienced the crashes, I may connect these back up at some point and see if I get any crashes again.

Seems really weird how you're getting crashes frequently and I'm not anymore..

#52

Updated by Crazy Fin about 10 years ago

Sorry but crashed now after 3 days of running without idle scan enabled.

This time the crash message is a little bit different:

"double free or corruption (top)" but still in the same module "src/input/mpegts/mpegts_input.c"

Should I create this crash as a new bug ticket or shall we keep on adding into this ticket?

gdb.txt is attached as well as the last 30 minutes of tvh.log.

=================================

2014-11-05 04:18:56.288 [ TRACE]:mpegts: table: mux 0x248fd60 free bat 48/F8 (72) pid 0011 (17)
2014-11-05 04:18:56.288 [ TRACE]:mpegts: table: mux 0x248fd60 free Pз 48/F8 (72) pid 0011 (17)
2014-11-05 04:18:56.288 [ TRACE]:mpegts: table: mux 0x248fd60 free Pз 48/F8 (72) pid 0011 (17)
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1987~gc053acd) [5f63503b8989ef1d18b18c8021adbf0f5b1c1b14] CWD: /home/tvserver/tvheadend
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Fault address 0x3e800005866 (N/A)
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Register dump [23]: 306433376661333000000000000000000000000000000008000000000000020600007ff4577fd4a00000000000000007000000000000006000000000000000070000000000005866000000000000587500007ff4577fd690000000000000006000000000000000060000000000000000ffffffffffffffff00007ff4577fd2f800007ff466596bb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-05 04:18:56.289 [ ALERT]:CRASH: STACKTRACE
2014-11-05 04:18:56.308 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4383b9
2014-11-05 04:18:56.315 [ ALERT]:CRASH: ??:0 0x7ff466e44340
2014-11-05 04:18:56.315 [ ALERT]:CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-05 04:18:56.315 [ ALERT]:CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-05 04:18:56.323 [ ALERT]:CRASH: ??:0 0x7ff4665d3e14
2014-11-05 04:18:56.331 [ ALERT]:CRASH: ??:0 0x7ff4665e00ee
2014-11-05 04:18:56.340 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:793 0x475005
2014-11-05 04:18:56.352 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:899 0x47519a
2014-11-05 04:18:56.361 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411121

=======================

#53

Updated by Crazy Fin about 10 years ago

Another crash today again and since I am not using "Idle Mux Scan" anymore it seems to me that these crashes are not dependent on that setting.

As always, gdb.txt and last 30 minutes of tvh.log attached.

==========================================

tvheadend: src/input/mpegts/mpegts_input.c:644: mpegts_input_table_dispatch: Assertion `0' failed.
2014-11-05 17:31:08.487 [ ERROR] psi: tables count inconsistency (num 14, list 7)
2014-11-05 17:31:08.488 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1992~gf1da79f) [182924a240519ec519a03b09e5b7bd0f9a3cc722] CWD: /home/tvserver/tvheadend
2014-11-05 17:31:08.488 [ ALERT] CRASH: Fault address 0x3e8000036c3 (N/A)
2014-11-05 17:31:08.488 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-05 17:31:08.488 [ ALERT] CRASH: Register dump [23]: fefefefefefefe0000007f3e600142000000000000000008000000000000020600000000004d791900000000004d3ec00000000000000002000000000000000e00000000000036c300000000000036d100007f3e92fa337000007f3e94be300000000000000000060000000000000000ffffffffffffffff00007f3e897f954800007f3e92e58bb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-05 17:31:08.488 [ ALERT] CRASH: STACKTRACE
2014-11-05 17:31:08.497 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x438459
2014-11-05 17:31:08.503 [ ALERT] CRASH: ??:0 0x7f3e93706340
2014-11-05 17:31:08.504 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-05 17:31:08.504 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)

==========================================

#54

Updated by Crazy Fin about 10 years ago

And now just approx 12 hours another crash but this time the crash message says

* Error in `./build.linux/tvheadend': double free or corruption (fasttop): 0x00007f9b4c001380 *
2014-11-06 06:57:14.437 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1992~gf1da79f) [182924a240519ec519a03b09e5b7bd0f9a3cc722] CWD: /home/tvserver/tvheadend
2014-11-06 06:57:14.437 [ ALERT] CRASH: Fault address 0x3e800002fed (N/A)
2014-11-06 06:57:14.437 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-06 06:57:14.437 [ ALERT] CRASH: Register dump [23]: 30383331303063346f6974707572726f0000000000000008000000000000020600007f9bbbff64100000000000000007000000000000006400000000000000070000000000002fed000000000000300600007f9bbbff6600000000000000006400000000000000060000000000000000ffffffffffffffff00007f9bbbff626800007f9bcdcc8bb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-06 06:57:14.437 [ ALERT] CRASH: STACKTRACE
2014-11-06 06:57:14.444 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x438459
2014-11-06 06:57:14.452 [ ALERT] CRASH: ??:0 0x7f9bce576340
2014-11-06 06:57:14.452 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-06 06:57:14.452 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-06 06:57:14.458 [ ALERT] CRASH: ??:0 0x7f9bcdd05e14
2014-11-06 06:57:14.466 [ ALERT] CRASH: ??:0 0x7f9bcdd120ee
2014-11-06 06:57:14.478 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_table.c:147 0x47b309
2014-11-06 06:57:14.487 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:793 0x475125
2014-11-06 06:57:14.499 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:899 0x4752ba
2014-11-06 06:57:14.507 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411151
2014-11-06 06:57:14.515 [ ALERT] CRASH: ??:0 0x7f9bce56e182

gdb.txt and last 30 minutes of log attached.

#55

Updated by Crazy Fin about 10 years ago

Another crash with the second type of error message:

Error in `./build.linux/tvheadend': double free or corruption (fasttop): 0x00007f9730000ba0

==========================================

2014-11-06 14:52:09.288 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1992~gf1da79f) [182924a240519ec519a03b09e5b7bd0f9a3cc722] CWD: /home/tvserver/tvheadend
2014-11-06 14:52:09.288 [ ALERT] CRASH: Fault address 0x3e8000048f0 (N/A)
2014-11-06 14:52:09.288 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-06 14:52:09.288 [ ALERT] CRASH: Register dump [23]: 30616230303030336f6974707572726f0000000000000008000000000000020600007f975fffe44000000000000000070000000000000064000000000000000700000000000048f000000000000048fb00007f975fffe630000000000000006400000000000000060000000000000000ffffffffffffffff00007f975fffe29800007f976df63bb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-06 14:52:09.288 [ ALERT] CRASH: STACKTRACE
2014-11-06 14:52:09.296 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x438459
2014-11-06 14:52:09.308 [ ALERT] CRASH: ??:0 0x7f976e811340
2014-11-06 14:52:09.308 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-06 14:52:09.308 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-06 14:52:09.313 [ ALERT] CRASH: ??:0 0x7f976dfa0e14
2014-11-06 14:52:09.317 [ ALERT] CRASH: ??:0 0x7f976dfad0ee
2014-11-06 14:52:09.326 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_table.c:147 0x47b309
2014-11-06 14:52:09.334 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:793 0x475125
2014-11-06 14:52:09.342 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:899 0x4752ba
2014-11-06 14:52:09.347 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411151

==========================================

gdb.txt and tvh.log (last 30 minutes) attached.

#56

Updated by Crazy Fin about 10 years ago

And a crash this morning again:

2014-11-05 04:18:56.288 [ TRACE]:mpegts: table: mux 0x248fd60 free bat 48/F8 (72) pid 0011 (17)
2014-11-05 04:18:56.288 [ TRACE]:mpegts: table: mux 0x248fd60 free Pз 48/F8 (72) pid 0011 (17)
2014-11-05 04:18:56.288 [ TRACE]:mpegts: table: mux 0x248fd60 free Pз 48/F8 (72) pid 0011 (17)
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.1987~gc053acd) [5f63503b8989ef1d18b18c8021adbf0f5b1c1b14] CWD: /home/tvserver/tvheadend
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Fault address 0x3e800005866 (N/A)
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-05 04:18:56.289 [ ALERT]:CRASH: Register dump [23]: 306433376661333000000000000000000000000000000008000000000000020600007ff4577fd4a00000000000000007000000000000006000000000000000070000000000005866000000000000587500007ff4577fd690000000000000006000000000000000060000000000000000ffffffffffffffff00007ff4577fd2f800007ff466596bb90000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-05 04:18:56.289 [ ALERT]:CRASH: STACKTRACE
2014-11-05 04:18:56.308 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x4383b9
2014-11-05 04:18:56.315 [ ALERT]:CRASH: ??:0 0x7ff466e44340
2014-11-05 04:18:56.315 [ ALERT]:CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-05 04:18:56.315 [ ALERT]:CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-05 04:18:56.323 [ ALERT]:CRASH: ??:0 0x7ff4665d3e14
2014-11-05 04:18:56.331 [ ALERT]:CRASH: ??:0 0x7ff4665e00ee
2014-11-05 04:18:56.340 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:793 0x475005
2014-11-05 04:18:56.352 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:899 0x47519a
2014-11-05 04:18:56.361 [ ALERT]:CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411121

#57

Updated by Jaroslav Kysela almost 10 years ago

Another table fix is in v3.9-2043-g0b2e072 , please, upgrade..

#58

Updated by Crazy Fin almost 10 years ago

Sorry but a crash again - with a different crash message though:

tvheadend: /home/tvserver/tvheadend/src/input/mpegts.h:806: mpegts_table_release: Assertion `mt->mt_destroyed == 1' failed.
2014-11-10 06:40:45.438 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.2046~g1a9f8d4) [9121457becd5967fa567ea87573ca11d0686056c] CWD: /home/tvserver/tvheadend
2014-11-10 06:40:45.438 [ ALERT] CRASH: Fault address 0x3e80000729e (N/A)
2014-11-10 06:40:45.438 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-10 06:40:45.438 [ ALERT] CRASH: Register dump [23]: fefefefefefefeffffffff00000000000000000000000008000000000000020200000000004dacec00000000004daf40000000000000000500007f0749ffa6b8000000000000729e00000000000072b100007f0754d3137000007f075697100000000000000000060000000000000000ffffffffffffffff00007f0749ffa49800007f0754be6bb90000000000000202d24200000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-10 06:40:45.438 [ ALERT] CRASH: STACKTRACE
2014-11-10 06:40:45.449 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x438849
2014-11-10 06:40:45.458 [ ALERT] CRASH: ??:0 0x7f0755494340
2014-11-10 06:40:45.458 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-10 06:40:45.458 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-10 06:40:45.469 [ ALERT] CRASH: ??:0 0x7f0754bdfa76
2014-11-10 06:40:45.478 [ ALERT] CRASH: ??:0 0x7f0754bdfb22
2014-11-10 06:40:45.490 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:806 0x4067a3
2014-11-10 06:40:45.502 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:804 0x477c20
2014-11-10 06:40:45.514 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:796 0x478483
2014-11-10 06:40:45.526 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411371

gdb.txt and tvh.log attached

#59

Updated by Jaroslav Kysela almost 10 years ago

OK, I think that I found the wrong code.

https://tvheadend.org/projects/tvheadend/repository/revisions/7f1d851a404a84cbe266923cfeecb92995291b8b/diff/

Please, upgrade to v3.9-2081-g7f1d851 and let me know the result.

#60

Updated by Crazy Fin almost 10 years ago

Waoh! 6 days without a crash but now I got one again... :-(

Again, the recent crash messages has been somewhat different but still related to the same part of the source code? I am not sure if I need to create a new bug ticket for this error:

tvheadend: /home/tvserver/tvheadend/src/input/mpegts.h:806: mpegts_table_release: Assertion `mt->mt_destroyed == 1' failed.
2014-11-16 00:37:14.289 [ ALERT] CRASH: Signal: 6 in PRG: ./build.linux/tvheadend (3.9.2064~g3cc291c) [847482646016165cbdd69b1dd0934f26bf8d96a5] CWD: /home/tvserver/tvheadend
2014-11-16 00:37:14.289 [ ALERT] CRASH: Fault address 0x3e80000256e (N/A)
2014-11-16 00:37:14.289 [ ALERT] CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
2014-11-16 00:37:14.289 [ ALERT] CRASH: Register dump [23]: fefefefefefefefffeff7dff092d63640000000000000008000000000000020200000000004daf9700000000004db200000000000000000b00007fb743ff6730000000000000256e000000000000258b00007fb7568cc37000007fb75850c00000000000000000060000000000000000ffffffffffffffff00007fb743ff651800007fb756781bb900000000000002022b3f00000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
2014-11-16 00:37:14.289 [ ALERT] CRASH: STACKTRACE
2014-11-16 00:37:14.314 [ ALERT] CRASH: /home/tvserver/tvheadend/src/trap.c:148 0x438919
2014-11-16 00:37:14.330 [ ALERT] CRASH: ??:0 0x7fb75702f340
2014-11-16 00:37:14.330 [ ALERT] CRASH: gsignal+0x39 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-16 00:37:14.330 [ ALERT] CRASH: abort+0x148 (/lib/x86_64-linux-gnu/libc.so.6)
2014-11-16 00:37:14.346 [ ALERT] CRASH: ??:0 0x7fb75677aa76
2014-11-16 00:37:14.362 [ ALERT] CRASH: ??:0 0x7fb75677ab22
2014-11-16 00:37:14.378 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:806 0x4067a3
2014-11-16 00:37:14.397 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts.h:804 0x477da0
2014-11-16 00:37:14.413 [ ALERT] CRASH: /home/tvserver/tvheadend/src/input/mpegts/mpegts_input.c:798 0x478603
2014-11-16 00:37:14.431 [ ALERT] CRASH: /home/tvserver/tvheadend/src/wrappers.c:125 0x411391

gdb.txt and tvh.log attached as usually.

#61

Updated by Jaroslav Kysela almost 10 years ago

Pleasse, try to upgrade to latest. I believe that this bug was fixed there.

#62

Updated by Jouk Hettema - almost 10 years ago

Upgraded yesterday to 3.9.2089. Didn't see any issues so far.

#63

Updated by Crazy Fin almost 10 years ago

I think we can close this ticket now. System has been stable since 3.9.2089 without any crashes similar to this ticket.

Well done Jaroslav and I am glad that you finally found the issue. :-)

#64

Updated by Jouk Hettema - almost 10 years ago

Yes indeed, well done Jaroslav!

#65

Updated by Jaroslav Kysela almost 10 years ago

  • Status changed from New to Fixed

Great. This one was tought..

Also available in: Atom PDF