Tvheadendcrash » History » Revision 5
Revision 4 (Andreas Smas, 2010-08-19 20:08) → Revision 5/7 (mdd -, 2010-09-24 15:35)
[[PageOutline]]
This article aims to provide a couple of guidelines guidlines of how to debug Tvheadend with GDB.
= Running in gdb to file a bug report =
This method is by far the best one as it yields most info. Especially if you have compiled the source yourself.
It does assume that you are able to reproduce the bug since you probably won't be running Tvheadend inside a gdb session normally.
To start Tvheadend from a gdb session just execute this command when standing in the dir where tvheadend is checked out.
{{{
$ gdb --args build.Linux/tvheadend -d
}}}
Or, If you get repeatable crashes in the installed binary you can also run gdb on that binary. In order to that you must disable the running daemon and run the binary as the hts user (or it won't pick up the correct config, etc)
{{{
$ sudo /etc/init.d/hts-tvheadend stop
$ sudo -u hts -i
$ gdb --args /usr/bin/tvheadend -d
}}}
{{{
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/andoma/tvheadend/build.Linux/tvheadend...done.
}}}
{{{
(gdb) r
}}}
Eventually the program will crash. Then execute these three commands and paste in the bug report.
=== Register dump ===
{{{
(gdb) info reg
eax 0x0 0
ecx 0x10 16
edx 0x80cd818 135059480
ebx 0x8178d90 135761296
esp 0xb45ff140 0xb45ff140
ebp 0xb45ff168 0xb45ff168
esi 0x80f9f18 135241496
edi 0xb45ff220 -1268780512
eip 0x8057ec8 0x8057ec8 <subscription_unsubscribe+200>
eflags 0x10282 [ SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
}}}
=== Disassembly of code around the crash ===
{{{
(gdb) disassemble $pc-32,$pc+32
Dump of assembler code from 0x8057ea8 to 0x8057ee8:
0x08057ea8 <subscription_unsubscribe+168>: add $0x24,%al
0x08057eaa <subscription_unsubscribe+170>: call 0x804a814 <free@plt>
0x08057eaf <subscription_unsubscribe+175>: mov %ebx,(%esp)
0x08057eb2 <subscription_unsubscribe+178>: call 0x804a814 <free@plt>
0x08057eb7 <subscription_unsubscribe+183>: call 0x80579c0 <subscription_reschedule>
0x08057ebc <subscription_unsubscribe+188>: mov 0x80c8bc8,%eax
0x08057ec1 <subscription_unsubscribe+193>: lea 0x0(%esi,%eiz,1),%esi
=> 0x08057ec8 <subscription_unsubscribe+200>: mov (%eax),%edx
0x08057eca <subscription_unsubscribe+202>: test %edx,%edx
0x08057ecc <subscription_unsubscribe+204>: je 0x8057ed4 <subscription_unsubscribe+212>
0x08057ece <subscription_unsubscribe+206>: mov 0x4(%eax),%ecx
0x08057ed1 <subscription_unsubscribe+209>: mov %ecx,0x4(%edx)
0x08057ed4 <subscription_unsubscribe+212>: mov 0x4(%eax),%ecx
0x08057ed7 <subscription_unsubscribe+215>: mov %edx,(%ecx)
0x08057ed9 <subscription_unsubscribe+217>: mov (%eax),%eax
0x08057edb <subscription_unsubscribe+219>: jmp 0x8057ec8 <subscription_unsubscribe+200>
0x08057edd <subscription_unsubscribe+221>: mov 0x2c(%ebx),%eax
0x08057ee0 <subscription_unsubscribe+224>: movl $0x8094ad9,0x8(%esp)
}}}
=== Full backtrace ===
{{{
(gdb) bt full
#0 subscription_unsubscribe (s=0x0) at /home/andoma/tvheadend/src/subscriptions.c:221
t = 0x80f9f18
#1 0x08061609 in htsp_subscription_destroy (htsp=0xb45ff220, hs=0x81762b8) at /home/andoma/tvheadend/src/htsp.c:240
No locals.
#2 0x080616e9 in htsp_method_unsubscribe (htsp=0xb45ff220, in=0x817a468) at /home/andoma/tvheadend/src/htsp.c:806
s = 0x81762b8
sid = 1
#3 0x080632ac in htsp_read_loop (fd=38, opaque=0x0, source=0x80d5774, self=0x80d5784) at /home/andoma/tvheadend/src/htsp.c:1065
m = 0x817a468
i = 8
reply = <value optimized out>
r = <value optimized out>
method = <value optimized out>
#4 htsp_serve (fd=38, opaque=0x0, source=0x80d5774, self=0x80d5784) at /home/andoma/tvheadend/src/htsp.c:1188
htsp = {htsp_fd = 38, htsp_peer = 0x80d5774, htsp_version = 0, htsp_logname = 0x80d3420 "127.0.0.1 [ HTS Showtime ]",
htsp_peername = 0x80ccf50 "127.0.0.1", htsp_username = 0x0, htsp_clientname = 0x80d7dd0 "HTS Showtime", htsp_async_mode = 1,
htsp_async_link = {le_next = 0x0, le_prev = 0x80c7e30}, htsp_writer_thread = 3017796464, htsp_writer_run = 1, htsp_active_output_queues = {
tqh_first = 0x0, tqh_last = 0xb45ff250}, htsp_out_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __kind = 0, __nusers = 1, {
__spins = 0, __list = {__next = 0x0}}}, __size = '\000' <repeats 16 times>, "\001\000\000\000\000\000\000", __align = 0},
htsp_out_cond = {__data = {__lock = 0, __futex = 119, __total_seq = 60, __wakeup_seq = 59, __woken_seq = 59, __mutex = 0xb45ff258,
__nwaiters = 2, __broadcast_seq = 0},
__size = "\000\000\000\000w\000\000\000<\000\000\000\000\000\000\000;\000\000\000\000\000\000\000;\000\000\000\000\000\000\000X\362_\264\002\000\000\000\000\000\000\000\000\000\000", __align = 511101108224}, htsp_hmq_ctrl = {hmq_q = {tqh_first = 0x0, tqh_last = 0xb45ff2a0}, hmq_link = {
tqe_next = 0x0, tqe_prev = 0xb45ff250}, hmq_strict_prio = 0, hmq_length = 0, hmq_payload = 0}, htsp_hmq_epg = {hmq_q = {
tqh_first = 0x0, tqh_last = 0xb45ff2bc}, hmq_link = {tqe_next = 0x0, tqe_prev = 0x0}, hmq_strict_prio = 0, hmq_length = 0,
hmq_payload = 0}, htsp_hmq_qstatus = {hmq_q = {tqh_first = 0x0, tqh_last = 0xb45ff2d8}, hmq_link = {tqe_next = 0x81762e4,
tqe_prev = 0xb45ff250}, hmq_strict_prio = 1, hmq_length = 0, hmq_payload = 0}, htsp_subscriptions = {lh_first = 0x0},
htsp_granted_access = 15, htsp_challenge = "\033\231\367B\270ʙ\375q~\272r\305&vә\\aۑ9\332+IM\036\365K\241\067K"}
buf = "127.0.0.1", '\000' <repeats 20 times>
s = <value optimized out>
#5 0x0804d7a3 in tcp_server_start (aux=0x80d5768) at /home/andoma/tvheadend/src/tcp.c:397
val = 1
#6 0x0018b96e in start_thread (arg=0xb45ffb70) at pthread_create.c:300
---Type <return> to continue, or q <return> to quit---
__res = <value optimized out>
__ignore1 = <value optimized out>
__ignore2 = <value optimized out>
pd = 0xb45ffb70
now = <value optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {1687540, 0, 4001536, -1268779880, 1031645625, -1276594991}, mask_was_saved = 0}}, priv = {pad = {
0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#7 0x00292a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
No locals.
}}}
= Using the crash info from syslog =
Starting with r5182 the output in syslog (/var/log/daemon.log) from a crash looks like this:
{{{
[ALERT]:CRASH: Signal: 11 in PRG: build.Linux/tvheadend (SVN-r5179) [4c9746188f23f5e04863d2d8388833b71f45cbf4] CWD: /home/andoma/tvheadend
[ALERT]:CRASH: Fault address 0x3 (Address not mapped)
[ALERT]:CRASH: Loaded libraries: /lib/tls/i686/cmov/librt.so.1 /lib/tls/i686/cmov/libcrypt.so.1 /usr/lib/libavahi-common.so.3 /usr/lib/libavahi-client.so.3 /lib/tls/i686/cmov/libpthread.so.0 /lib/tls/i686/cmov/libm.so.6 /lib/tls/i686/cmov/libc.so.6 /lib/tls/i686/cmov/libdl.so.2 /lib/ld-linux.so.2 /lib/libdbus-1.so.3 /lib/tls/i686/cmov/libnss_compat.so.2 /lib/tls/i686/cmov/libnsl.so.1 /lib/tls/i686/cmov/libnss_nis.so.2 /lib/tls/i686/cmov/libnss_files.so.2
[ALERT]:CRASH: Register dump [19]: 00000033 00000000 0000007b 0000007b 00000001 000003e8 bfb2c948 bfb2c7d0 000003e8 0000009c 0028d36c 00000000 0000000e 00000006 0804bbc3 00000073 00010246 bfb2c7d0 0000007b
[ALERT]:CRASH: STACKTRACE
[ALERT]:CRASH: build.Linux/tvheadend 0x80678a9
[ALERT]:CRASH: __kernel_rt_sigreturn+0x0 ()
[ALERT]:CRASH: __libc_start_main+0xe6 (/lib/tls/i686/cmov/libc.so.6)
[ALERT]:CRASH: build.Linux/tvheadend 0x804aee1
}}}
Just paste this info in the bug report
This article aims to provide a couple of guidelines guidlines of how to debug Tvheadend with GDB.
= Running in gdb to file a bug report =
This method is by far the best one as it yields most info. Especially if you have compiled the source yourself.
It does assume that you are able to reproduce the bug since you probably won't be running Tvheadend inside a gdb session normally.
To start Tvheadend from a gdb session just execute this command when standing in the dir where tvheadend is checked out.
{{{
$ gdb --args build.Linux/tvheadend -d
}}}
Or, If you get repeatable crashes in the installed binary you can also run gdb on that binary. In order to that you must disable the running daemon and run the binary as the hts user (or it won't pick up the correct config, etc)
{{{
$ sudo /etc/init.d/hts-tvheadend stop
$ sudo -u hts -i
$ gdb --args /usr/bin/tvheadend -d
}}}
{{{
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/andoma/tvheadend/build.Linux/tvheadend...done.
}}}
{{{
(gdb) r
}}}
Eventually the program will crash. Then execute these three commands and paste in the bug report.
=== Register dump ===
{{{
(gdb) info reg
eax 0x0 0
ecx 0x10 16
edx 0x80cd818 135059480
ebx 0x8178d90 135761296
esp 0xb45ff140 0xb45ff140
ebp 0xb45ff168 0xb45ff168
esi 0x80f9f18 135241496
edi 0xb45ff220 -1268780512
eip 0x8057ec8 0x8057ec8 <subscription_unsubscribe+200>
eflags 0x10282 [ SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
}}}
=== Disassembly of code around the crash ===
{{{
(gdb) disassemble $pc-32,$pc+32
Dump of assembler code from 0x8057ea8 to 0x8057ee8:
0x08057ea8 <subscription_unsubscribe+168>: add $0x24,%al
0x08057eaa <subscription_unsubscribe+170>: call 0x804a814 <free@plt>
0x08057eaf <subscription_unsubscribe+175>: mov %ebx,(%esp)
0x08057eb2 <subscription_unsubscribe+178>: call 0x804a814 <free@plt>
0x08057eb7 <subscription_unsubscribe+183>: call 0x80579c0 <subscription_reschedule>
0x08057ebc <subscription_unsubscribe+188>: mov 0x80c8bc8,%eax
0x08057ec1 <subscription_unsubscribe+193>: lea 0x0(%esi,%eiz,1),%esi
=> 0x08057ec8 <subscription_unsubscribe+200>: mov (%eax),%edx
0x08057eca <subscription_unsubscribe+202>: test %edx,%edx
0x08057ecc <subscription_unsubscribe+204>: je 0x8057ed4 <subscription_unsubscribe+212>
0x08057ece <subscription_unsubscribe+206>: mov 0x4(%eax),%ecx
0x08057ed1 <subscription_unsubscribe+209>: mov %ecx,0x4(%edx)
0x08057ed4 <subscription_unsubscribe+212>: mov 0x4(%eax),%ecx
0x08057ed7 <subscription_unsubscribe+215>: mov %edx,(%ecx)
0x08057ed9 <subscription_unsubscribe+217>: mov (%eax),%eax
0x08057edb <subscription_unsubscribe+219>: jmp 0x8057ec8 <subscription_unsubscribe+200>
0x08057edd <subscription_unsubscribe+221>: mov 0x2c(%ebx),%eax
0x08057ee0 <subscription_unsubscribe+224>: movl $0x8094ad9,0x8(%esp)
}}}
=== Full backtrace ===
{{{
(gdb) bt full
#0 subscription_unsubscribe (s=0x0) at /home/andoma/tvheadend/src/subscriptions.c:221
t = 0x80f9f18
#1 0x08061609 in htsp_subscription_destroy (htsp=0xb45ff220, hs=0x81762b8) at /home/andoma/tvheadend/src/htsp.c:240
No locals.
#2 0x080616e9 in htsp_method_unsubscribe (htsp=0xb45ff220, in=0x817a468) at /home/andoma/tvheadend/src/htsp.c:806
s = 0x81762b8
sid = 1
#3 0x080632ac in htsp_read_loop (fd=38, opaque=0x0, source=0x80d5774, self=0x80d5784) at /home/andoma/tvheadend/src/htsp.c:1065
m = 0x817a468
i = 8
reply = <value optimized out>
r = <value optimized out>
method = <value optimized out>
#4 htsp_serve (fd=38, opaque=0x0, source=0x80d5774, self=0x80d5784) at /home/andoma/tvheadend/src/htsp.c:1188
htsp = {htsp_fd = 38, htsp_peer = 0x80d5774, htsp_version = 0, htsp_logname = 0x80d3420 "127.0.0.1 [ HTS Showtime ]",
htsp_peername = 0x80ccf50 "127.0.0.1", htsp_username = 0x0, htsp_clientname = 0x80d7dd0 "HTS Showtime", htsp_async_mode = 1,
htsp_async_link = {le_next = 0x0, le_prev = 0x80c7e30}, htsp_writer_thread = 3017796464, htsp_writer_run = 1, htsp_active_output_queues = {
tqh_first = 0x0, tqh_last = 0xb45ff250}, htsp_out_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __kind = 0, __nusers = 1, {
__spins = 0, __list = {__next = 0x0}}}, __size = '\000' <repeats 16 times>, "\001\000\000\000\000\000\000", __align = 0},
htsp_out_cond = {__data = {__lock = 0, __futex = 119, __total_seq = 60, __wakeup_seq = 59, __woken_seq = 59, __mutex = 0xb45ff258,
__nwaiters = 2, __broadcast_seq = 0},
__size = "\000\000\000\000w\000\000\000<\000\000\000\000\000\000\000;\000\000\000\000\000\000\000;\000\000\000\000\000\000\000X\362_\264\002\000\000\000\000\000\000\000\000\000\000", __align = 511101108224}, htsp_hmq_ctrl = {hmq_q = {tqh_first = 0x0, tqh_last = 0xb45ff2a0}, hmq_link = {
tqe_next = 0x0, tqe_prev = 0xb45ff250}, hmq_strict_prio = 0, hmq_length = 0, hmq_payload = 0}, htsp_hmq_epg = {hmq_q = {
tqh_first = 0x0, tqh_last = 0xb45ff2bc}, hmq_link = {tqe_next = 0x0, tqe_prev = 0x0}, hmq_strict_prio = 0, hmq_length = 0,
hmq_payload = 0}, htsp_hmq_qstatus = {hmq_q = {tqh_first = 0x0, tqh_last = 0xb45ff2d8}, hmq_link = {tqe_next = 0x81762e4,
tqe_prev = 0xb45ff250}, hmq_strict_prio = 1, hmq_length = 0, hmq_payload = 0}, htsp_subscriptions = {lh_first = 0x0},
htsp_granted_access = 15, htsp_challenge = "\033\231\367B\270ʙ\375q~\272r\305&vә\\aۑ9\332+IM\036\365K\241\067K"}
buf = "127.0.0.1", '\000' <repeats 20 times>
s = <value optimized out>
#5 0x0804d7a3 in tcp_server_start (aux=0x80d5768) at /home/andoma/tvheadend/src/tcp.c:397
val = 1
#6 0x0018b96e in start_thread (arg=0xb45ffb70) at pthread_create.c:300
---Type <return> to continue, or q <return> to quit---
__res = <value optimized out>
__ignore1 = <value optimized out>
__ignore2 = <value optimized out>
pd = 0xb45ffb70
now = <value optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {1687540, 0, 4001536, -1268779880, 1031645625, -1276594991}, mask_was_saved = 0}}, priv = {pad = {
0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
robust = <value optimized out>
freesize = <value optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#7 0x00292a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
No locals.
}}}
= Using the crash info from syslog =
Starting with r5182 the output in syslog (/var/log/daemon.log) from a crash looks like this:
{{{
[ALERT]:CRASH: Signal: 11 in PRG: build.Linux/tvheadend (SVN-r5179) [4c9746188f23f5e04863d2d8388833b71f45cbf4] CWD: /home/andoma/tvheadend
[ALERT]:CRASH: Fault address 0x3 (Address not mapped)
[ALERT]:CRASH: Loaded libraries: /lib/tls/i686/cmov/librt.so.1 /lib/tls/i686/cmov/libcrypt.so.1 /usr/lib/libavahi-common.so.3 /usr/lib/libavahi-client.so.3 /lib/tls/i686/cmov/libpthread.so.0 /lib/tls/i686/cmov/libm.so.6 /lib/tls/i686/cmov/libc.so.6 /lib/tls/i686/cmov/libdl.so.2 /lib/ld-linux.so.2 /lib/libdbus-1.so.3 /lib/tls/i686/cmov/libnss_compat.so.2 /lib/tls/i686/cmov/libnsl.so.1 /lib/tls/i686/cmov/libnss_nis.so.2 /lib/tls/i686/cmov/libnss_files.so.2
[ALERT]:CRASH: Register dump [19]: 00000033 00000000 0000007b 0000007b 00000001 000003e8 bfb2c948 bfb2c7d0 000003e8 0000009c 0028d36c 00000000 0000000e 00000006 0804bbc3 00000073 00010246 bfb2c7d0 0000007b
[ALERT]:CRASH: STACKTRACE
[ALERT]:CRASH: build.Linux/tvheadend 0x80678a9
[ALERT]:CRASH: __kernel_rt_sigreturn+0x0 ()
[ALERT]:CRASH: __libc_start_main+0xe6 (/lib/tls/i686/cmov/libc.so.6)
[ALERT]:CRASH: build.Linux/tvheadend 0x804aee1
}}}
Just paste this info in the bug report