Bug #1029
TVHeadend IPTV handling crash
100%
Description
I am using latest 79c1b4904be7ceac4c10a92f6145b1cf18e7e967.
When switching channels, system crashes at iptv_input.c:
Jun 7 21:55:29 fsiptv tvheadend[21838]: subscription: "172.30.108.119 [ XBMC Media Center ]" unsubscribing from "CNBC Europe" Jun 7 21:55:29 fsiptv tvheadend[21838]: subscription: "172.30.108.119 [ XBMC Media Center ]" subscribing on "RTL Nitro", weight: 150, adapter: "eth0", network: "<N/A>", mux: "224.0.9.7", provider: "<N/A>", service: "<N/A>", quality: 100 Jun 7 21:55:29 fsiptv tvheadend[21838]: Service: eth0/224.0.9.7: Status changed to [Hardware input] Jun 7 21:55:29 fsiptv tvheadend[21838]: Service: eth0/224.0.9.7: Status changed to [Hardware input] [Input on service] Jun 7 21:55:29 fsiptv tvheadend[21838]: Service: eth0/224.0.9.7: Status changed to [Hardware input] [Input on service] [Demuxed packets] Jun 7 21:55:29 fsiptv tvheadend[21838]: Service: eth0/224.0.9.7: Status changed to [Hardware input] [Input on service] [Demuxed packets] [Reassembled packets] Jun 7 21:55:38 fsiptv tvheadend[21838]: subscription: "172.30.108.119 [ XBMC Media Center ]" unsubscribing from "RTL Nitro" Jun 7 21:55:38 fsiptv tvheadend[21838]: subscription: "172.30.108.119 [ XBMC Media Center ]" subscribing on "N-TV", weight: 150, adapter: "eth0", network: "<N/A>", mux: "224.0.9.5", provider: "<N/A>", service: "<N/A>", quality: 100 Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: Signal: 11 in PRG: /usr/local/bin/tvheadend (2.99.26.gfc10a) [da89eeee553903800ec294b230c365b42f3da09f] CWD: / Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: Fault address 0x48 (Address not mapped) Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: Loaded libraries: /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libnss_compat.so.2 /lib/x86_64-linux-gnu/libnsl.so.1 /lib/x86_64-linux-gnu/libnss_nis.so.2 /lib/x86_64-linux-gnu/libnss_files.so.2 Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: Register dump [23]: 00007f2cc2ffcd08 000000000000555b 0000000000000001 0000000000000206 0000000001ff5e60 0000000001ff5b50 00007f2cc2ffcce0 00007f2cdc0343a8 00007f2cc2ffcce0 00007f2cda7ebb30 00007f2cd4039120 0000000001ff69d0 00007f2cc2ffcce0 0000000000000000 00007f2ce4917d75 00007f2cda7ebb10 000000000041a26d 0000000000010202 0000000000000033 0000000000000004 000000000000000e fffffffe7ffbfa17 0000000000000048 Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: STACKTRACE Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/trap.c:139 0x42096b Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: ??:0 0x7f2ce4919060 Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/htsp.c:1619 0x41a26d Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/streaming.c:288 0x40d50e Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/parsers.c:1350 0x4162fd Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/parsers.c:1092 0x417785 Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/parsers.c:332 0x4160b1 Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/tsdemux.c:122 0x418b2c Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/tsdemux.c:256 0x418f6e Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: /usr/src/tvheadend/src/iptv_input.c:116 0x4221ad Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: ??:0 0x7f2ce4910efc Jun 7 21:55:38 fsiptv tvheadend[21838]: CRASH: clone+0x6d (/lib/x86_64-linux-gnu/libc.so.6) Jun 7 21:55:38 fsiptv kernel: [1690680.069588] tvheadend[21851]: segfault at 48 ip 000000000041a26d sp 00007f2cda7ebb10 error 4 in tvheadend[400000+8f000] Jun 7 21:55:38 fsiptv kernel: [1690680.095051] init: tvheadend main process (21838) killed by SEGV signal Jun 7 21:55:38 fsiptv kernel: [1690680.095161] init: tvheadend main process ended, respawning
There is 60 devices connected to this tvheadend, around 20 of using it simultaneously each day. The crash happens randomnly, sometimes once per day two, sometimes more times a day.
TvHeadend is run on dedicated AtomD252 server, and it is fed by 4 DVBS/IPTV streamers, each having TBS6984 and multicast streaming our 4 multiplexes by dvblast. Some of the channels are also processes by tsdecrypt/oscam and then restreamed to multicast as well, where tvheadend pick them up. Traffic at tvheadend server is around 200Mbps multicast in and peaking 100Mbps htsp out, server is not loaded (90+ % idle).
Let me know, if i can provide more info.
Related issues
History
Updated by Andreas Smas over 12 years ago
The reason for this crash is that the code that sends signal status updates in the HTSP stream is broken (by design). It violates the lock domains. So i'll just remove that
Updated by Andreas Smas over 12 years ago
- Status changed from New to Fixed
- % Done changed from 0 to 100
Applied in changeset commit:8f9ac9040e4128915d6d66faf82bf23ebc9f14f3.
Updated by Zdenek Janda over 12 years ago
Thanks much, updated to 8f9ac9040e4128915d6d66faf82bf23ebc9f14f3 and will report stability.
Updated by Andreas Smas over 12 years ago
Out of curiosity. Where do you have 60 clients installed? A Hotel?
Updated by Hein Rigolo over 12 years ago
and related to that ... could you describe your setup in some more detail, I curious about how it is all connected together. Maybe write a wiki page about it?
Updated by Zdenek Janda over 12 years ago
Agreed, will produce some in detail documentation to the wiki. It has been quite a struggle to put it all together, but its worth of effort.
Updated by Anonymous over 12 years ago
Thanks for the update related to our bug #1032. We have updated to the latest git, will test and report back in a few days...
Updated by Krzychu Noski over 12 years ago
I still run onto this problem whenever IPTV channel is opened:
Jul 1 22:49:10 smeagoskan tvheadend6490: CRASH: Signal: 11 in PRG: /home/knienart/dvb/bin/tvheadend (2.99.44.g4ef72) [25a44d28bec6458e5577bf00987010c58210be1b] CWD: /
Jul 1 22:49:10 smeagoskan tvheadend6490: CRASH: Fault address 0x118 (Address not mapped)
Jul 1 22:49:10 smeagoskan tvheadend6490: CRASH: Loaded libraries: /lib/librt.so.1 /lib/libdl.so.2 /usr/lib/libavahi-common.so.3 /usr/lib/libavahi-client.so.3 /lib/libssl.so.0.9.8 /lib/libcrypto.so.0.9.8 /lib/libpthread.so.0 /lib/libm.so.6 /lib/libc.so.6 /lib64/ld-linux-x86-64.so.2 /lib/libdbus-1.so.3 /lib/libz.so.1 /lib/libnss_compat.so.2 /lib/libnsl.so.1 /lib/libnss_nis.so.2 /lib/libnss_files.so.2
Jul 1 22:49:10 smeagoskan tvheadend6490: CRASH: Register dump [23]: 0000000000000000 0000000002323ef0 207265746e654320 00000000ffffffdf 00007f6cbcdbfa60 00007f6cbcdbf9c0 0000000000000000 0000000000000001 0000000000000007 0000000000457835 00007f6cbcdbfa40 0000000002323f10 0000000000450f30 0000000000000000 00007f6cbcdbfa60 00007f6cbcdbf9b0 0000000000413b74 0000000000010202 0000000000000033 0000000000000004 000000000000000e fffffffe7ffbfa17 0000000000000118
Jul 1 22:49:10 smeagoskan tvheadend6490: CRASH: STACKTRACE
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/trap.c:139 0x421f3b
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: ??:0 0x7f6cc43a08f0
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/service.c:346 0x413b74
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/subscriptions.c:173 0x4126db
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/subscriptions.c:326 0x41289d
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/htsp.c:916 0x41b037
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/htsp.c:1210 0x41cdfd
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: /home/knienart/dvb/tvheadend.git/tvheadend.andoma/src/tcp.c:397 0x4071ab
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: ??:0 0x7f6cc43979ca
Jul 1 22:49:11 smeagoskan tvheadend6490: CRASH: clone+0x6d (/lib/libc.so.6)
Jul 1 22:49:11 smeagoskan kernel: [37906.701835] tvheadend6817: segfault at 118 ip 0000000000413b74 sp 00007f6cbcdbf9b0 error 4 in tvheadend[400000+8f000]
Updated by Andy Brown over 12 years ago
Just wanted to comment on this, so is the signalstatus code broken on all transports? As I've never had any problems on DVB-T and DVB-S cards, so is this purely an IPTV related crash, and so can we code around retrieving signal status when its an unsupported transport? (Or return a dummy status?)
Reason I ask is it seems a bit harsh to kill the code out completely as I'm unaware of any other crashes relating to this apart from this transport stream (I've reversed the patch on my local systems/test systems and not had stability problems as a result?)
Just curious and want to track this fault.