Bug #2156
TVHeadend crashes since some builds back
0%
Description
I tested the latest build 3.9.967~gd924c84~precise and I notice that TVHeadend crashes every 30-60 minute. I can not see any specifically similar in the debug log that gives me a clue of what is happening.
I am not sure how many builds back this started but it was working fine with the builds back from last weeks of May.
Here is a piece of the syslog with debug -all enabld (I had no trace enabled though):
==========================
Jun 29 20:30:56 tvserver tvheadend7315: cwc: Sending ECM (PID 7093) section=0/0, for service "SVT1 HD" (seqno: 1277)
Jun 29 20:30:56 tvserver tvheadend7315: cwc: Received ECM reply (PID 7093) for service "SVT1 HD" even: af.19.90.58.f6.8c.a4.26 odd: 8f.a2.a8.d9.3e.b8.62.58 (seqno: 1277 Req delay: 332 ms)
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: Signal: 6 in PRG: tvheadend (3.9.967~gd924c84~precise) [19dc61acdf5674894de2298eec397423f45001c5] CWD: /
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: Fault address 0x7300001c93 (N/A)
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: Loaded libraries: /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/liburiparser.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libnss_compat.so.2 /lib/x86_64-linux-gnu/libnsl.so.1 /lib/x86_64-linux-gnu/libnss_nis.so.2 /lib/x86_64-linux-gnu/libnss_files.so.2
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: Register dump [23]: fefefefefefefeffffffffffff0000000000000000000008000000000000020600000000004b2fe500000000004b31e0000000000000000f00000000000000120000000000001c930000000000001ca700000000004b30a000007f918d84800000000000000000060000000000000000ffffffffffffffff00007f917bffe8e800007f918bafb4250000000000000206000000000000003300000000000000000000000000000000fffffffe7ffbba170000000000000000
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: STACKTRACE
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x42f37c
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x7f918c398cb0
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: gsignal+0x35 (/lib/x86_64-linux-gnu/libc.so.6)
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: abort+0x17b (/lib/x86_64-linux-gnu/libc.so.6)
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x7f918baf40ee
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x7f918baf4192
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x4630ba
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x463134
Jun 29 20:30:58 tvserver tvheadend7315: CRASH: ??:0 0x40dfe1
Jun 29 20:30:59 tvserver tvheadend30100: charset: 138 entries loaded
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter7
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter6
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter5
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter4
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter3
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter2
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter1
Jun 29 20:30:59 tvserver tvheadend30100: linuxdvb: adapter added /dev/dvb/adapter0
==========================
Is there anything more I can debug or trace to help you investigate his crashing?
Files
History
Updated by Crazy Fin over 10 years ago
I tested 3.9.818~gfda1a03 and that one has not yet crashed after 3 hours of running (2 tuners watching, 3rd tuner recording).
Updated by Crazy Fin over 10 years ago
I see this line in the log just before it crashes:
tvheadend: src/input/mpegts/mpegts_input.c:540: mpegts_input_table_dispatch: Assertion `i == len' failed.
3.9.871~ga70a3f5 crashes as well
Updated by Jozef M over 10 years ago
Don't know if related to my issue, but I can ACK this.
Latest stable version I have is from Jun 22nd - 3.9.692~gf36be10-dirty. Feed to TVH is in my case IPTV via MC. Any version up to couple of commits back is segfaulting, at least once a day. I will be able to provide a debug/gdb outputs hopefully this weekend.
Updated by Crazy Fin over 10 years ago
I am back down to release 3.9.821~g8af2590 which now has been up and running for approx 24 hours without a glitch.
I was having crashes from 3.9.848~g2be0f84 and up to 3.9.967~gd924c84.
I will try and test other release between 3.9.848~g2be0f84 and the current one 3.9.821~g8af2590 that I now have installed and is working fine.
Maybe I will find out between which releases the break causing the crashes is.
Updated by Crazy Fin over 10 years ago
Ok I tested all versions from
3.9.848~g2be0f84
down to
3.9.821~g8af2590 (working fine)
and ALL releaseas between those two does not compile without errors. They all generate the following error when compiling "descrambler.o":
==========================
CC src/descrambler/descrambler.o
src/descrambler/descrambler.c: In function ‘descrambler_descramble’:
src/descrambler/descrambler.c:167:31: error: ‘failed’ may be used uninitialized in this function [-Werror=uninitialized]
src/descrambler/descrambler.c:167:31: error: ‘count’ may be used uninitialized in this function [-Werror=uninitialized]
cc1: all warnings being treated as errors
make: *** [/home/tvserver/tvheadend/build.linux/src/descrambler/descrambler.o] Error 1
==========================
I know tried to compile the latest commit (Build #5474 - v3.9.976 - precise-amd64) but when I do
"git checkout bb181b06c336ae074fe219a982a2d1da8c07039f"
I get error
"fatal: reference is not a tree: bb181b06c336ae074fe219a982a2d1da8c07039f"
So I will stay on 3.9.821~g8af2590 until the crash bug has been fixed.
Let me know if there is any deugging I can do to help solving this issue.
Updated by Crazy Fin over 10 years ago
I tested now with (in the TVHeadend directory)
"git checkout master"
and then
"git pull"
I was now able to do "./configure" and then "make" and the compilation succeeded without any errors and when I started with
"./build.linux/tvheadend -l tvh.log --debug all"
it started fine and I could see that I now was on the latest build "Build #5478 - v3.9.980 - precise-amd64". Cool!
However, crashes still the same way as all versions from 3.9.822 and up so I am still staying on "3.9.821~g8af2590".
Updated by Crazy Fin over 10 years ago
I tested the latest build "3.9.1050~g388b4d7" yesterday and it still crashes after approx 1 hour of running and the log looks identical as in previous crashes:
"tvheadend: src/input/mpegts/mpegts_input.c:540: mpegts_input_table_dispatch: Assertion `i == len' failed."
in the log right before the crash ("segmentation fault")
Going back to "3.9.821~g8af2590" again and it works fine without crashes. However, this version has some "counter continuity errors" that I do not see in the newer versions that crashes after approx 1 hour.
Updated by Crazy Fin over 10 years ago
Ok, version 3.9.1064~gd81a612 has not crashed (yet) for 5 hours so this release is promising!
"Continuity counter errors" that I see in the version that was my latest working release (3.9.821~g8af2590) are also gone.
I´ll keep on running this version for about 24 hours more before finally confirming that it is "stable" enough compared to the releases 2 months back and up until now.
Updated by Crazy Fin over 10 years ago
Nooooo... crashed again but it 9 hours.
Same message in the debug log just before it crashes:
"tvheadend: src/input/mpegts/mpegts_input.c:540: mpegts_input_table_dispatch: Assertion `i == len' failed."
Updated by Mario D over 10 years ago
I think a 'bt full' from the core dump image could be useful to the developers: [[https://tvheadend.org/projects/tvheadend/wiki/Debugging]]
Updated by Crazy Fin over 10 years ago
I did the following (as per https://tvheadend.org/projects/tvheadend/wiki/Debugging):
1. Enabled coredumps with
ulimit -c unlimited
2. Since I am using Ubuntu 12.0.4 LTS I did
core_uses_pid=1 (as suggested in the debugging info page)
3. Once I had the core dump file I did
gdb tvheadend core
(gdb) set logging on
(gdb) set pagination off
(gdb) bt full
but the only output I get into gdb.txt is
[New LWP 3833]
[New LWP 3840]
[New LWP 3837]
[New LWP 3836]
[New LWP 3841]
[New LWP 3844]
[New LWP 3845]
[New LWP 3851]
[New LWP 3848]
[New LWP 3861]
[New LWP 3849]
[New LWP 3852]
[New LWP 3862]
[New LWP 3855]
[New LWP 3866]
[New LWP 3856]
[New LWP 3817]
[New LWP 3858]
[New LWP 3881]
[New LWP 3859]
[New LWP 3891]
[New LWP 3860]
[New LWP 3864]
[New LWP 3877]
[New LWP 3878]
[New LWP 3880]
[New LWP 3887]
[New LWP 3888]
[New LWP 6837]
[New LWP 6840]
[New LWP 3818]
[New LWP 3819]
[New LWP 3820]
[New LWP 3821]
[New LWP 3822]
[New LWP 3823]
[New LWP 3824]
[New LWP 3825]
[New LWP 3826]
[New LWP 3828]
[New LWP 3885]
[New LWP 3832]
[New LWP 3829]
[New LWP 3876]
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff05fe000
Core was generated by `./build.linux/tvheadend -c test-conf -l tvh.log --debug all'.
Program terminated with signal 6, Aborted.
#0 0x00007f03d7d3d425 in ?? ()
and the output from
set logging on
(gdb) set pagination off
(gdb) bt full
shows
#0 0x00007f03d7d3d425 in ?? ()
No symbol table info available.
#1 0x00007f03d7d40b8b in ?? ()
No symbol table info available.
#2 0x00000000004ba2a5 in ?? ()
No symbol table info available
and so on up to
#23 0x0000000000000000 in ?? ()
No symbol table info available.
I guess I am doing something wrong with these core dumps?
Updated by Crazy Fin over 10 years ago
Ok I think I have now been able to get a proper core dump file.
When running
gdb tvheadend core
I had instead of course to do
./build.linux/tvheadend core
to get it work properly (since I was in tvheadend directory)
Attached is the tvh.log (debug log) and the gdb.txt file from the core dump.
Let me know if there is some other files that I should attach.
Updated by Crazy Fin over 10 years ago
Ok, I thiiiiink that we can close this bug ticket now. :-)
I am currently testing 3.9.1083 and it does not crash anymore within an hour or so.
I have been able to run TVHeadend for 12-24 hours until it crashes and the crash is now quite different than this one.
I will report the new type of crash in a separate ticket with corresponing debug log and core dump.