Project

General

Profile

Debugging » History » Version 2

Jaroslav Kysela, 2014-10-27 19:31

1 1 Adam Sutton
h1. Debugging
2
3
If you're going to be regularly trying development versions of Tvheadend or need to report a crash or deadlock then you should really read this page!
4
5
If you are investigating problems within Tvheadend then its worth being familiar with tools such as gdb and valgrind, although these are not covered here.
6
7
However one thing that can be useful in investigating crashes within Tvheadend is to ensure that coredumps are generated, this will allow post analysis in gdb without having to actual run Tvheadend within gdb.
8
9
You can enable temporarily by running:
10
11
<pre>
12
ulimit -c unlimited
13
</pre>
14
15
To make this permanent put this somewhere in your shell environment setup (.bashrc, .profile, etc...)
16
Firstly I'd recommend that if you're specifically trying to investigate an issue then you should consider running Tvheadend manually, rather than as a service, as documented [[Development|here]].
17
18
h2. Logging
19
20
I'd strongly recommend that if you're specifically trying to investigate a crash or other problem in Tvheadend that you enable debugging:
21
22
* *-s* will output debug info to syslog
23
* *--debug* allows you to specify which subsystem to debug (TODO: add more info)
24
* *--trace* allows you to enable trace (more in-depth) logging on specific subsystems
25
26
You can also get Tvheadend to log to it's own file using:
27
28
<pre>
29
-l FILE
30
</pre>
31
32 2 Jaroslav Kysela
You may also modify the debug settings using WEB GUI as admin - Configuration/Debugging. Note that the information is not saved,
33
it is just set for run-time (current task).
34
35
* Debug log path - filename to store log
36
* Debug trace - enable traces
37
* Debug subsystems - comma separated list of subsystems
38
* Trace subsystems - comma separated list of subsystems
39
40
41 1 Adam Sutton
h2. Enabling coredumps
42
43
Although you can run Tvheadend within gdb, personally I never bother. If you need to investigate some running problem you can always attach (see below) later and if you need to trap crashes, then you can configure your system to generate a core file and then retrospectively analyse this with gdb.
44
45
If you're running manually you should enable coredumps in your environment:
46
47
<pre>
48
ulimit -c unlimited
49
</pre>
50
51
I'd recommend you enable this permanently by putting this command in your shell initialisation scripts (.bashrc etc..).
52
53
If you're running as a daemon then you should use the -D command line option, this will enable coredumps from the daemon. If you start using sysvinit, upstart etc... then you will need to put this in the configuration file, e.g.:
54
55
<pre>
56
TVH_ARGS="-D"
57
</pre>
58
59
Finally it's probably worth changing the coredump file format, personally I use the following configuration:
60
61
<pre>
62
echo core.%h.%e.%t | sudo tee /proc/sys/kernel/core_pattern
63
echo 0 | sudo tee /proc/sys/kernel/core_uses_pid
64
</pre>
65
66
Or put the following in /etc/sysctl.conf:
67
68
<pre>
69
kernel.core_pattern = core.%h.%e.%t
70
kernel.core_uses_pid = 0
71
</pre>
72
73
If you're using a system like Ubuntu that uses apport (and cripples the ability to change the core format) just set core_uses_pid=1 instead.
74
75
Note: coredumps are (by default) stored in the current working directory, to make it possible for the daemon to write files the current working directory is set to /tmp when using -D, so check there for core files.
76
77
To verify that you have everything configured properly you can use the -A option to force a crash on startup. Do this from the command line or add to /etc/default/tvheadend:
78
79
<pre>
80
TVH_ARGS="-D -A"
81
</pre>
82
83
Note: remember to remove the option after you've tested it!
84
85
h2. Processing core file.
86
87
Once you have a core file you can start up gdb with that coredump, just as if you'd caught the crash while running under gdb:
88
89
<pre>
90
gdb tvheadend core
91
</pre>
92
93
You may need to replace _tvheadend_ and _core_ above with the proper paths.
94
95
For most crashes the most useful information is the back trace, this will provide a stack trace showing where the code crashed and the stack information at the time of the crash:
96
97
<pre>
98
(gdb) set logging on
99
(gdb) set pagination off
100
(gdb) bt full
101
#0  0x00007f5b10cc1425 in __GI_raise (sig=<optimised out>)
102
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
103
        resultvar = 0
104
        pid = <optimised out>
105
        selftid = 7517
106
#1  0x00007f5b10cc4b10 in __GI_abort () at abort.c:120
107
        act = {__sigaction_handler = {sa_handler = 0, sa_sigaction = 0}, 
108
          sa_mask = {__val = {18446744073709551615 <repeats 16 times>}}, 
109
          sa_flags = 0, sa_restorer = 0}
110
        sigs = {__val = {32, 0 <repeats 15 times>}}
111
#2  0x000000000040744e in main (argc=<optimised out>, argv=<optimised out>)
112
    at src/main.c:810
113
        i = <optimised out>
114
        set = {__val = {16386, 0 <repeats 15 times>}}
115
        adapter_mask = <optimised out>
116
        log_level = <optimised out>
117
        log_options = <optimised out>
118
        log_debug = <optimised out>
119
        log_trace = <optimised out>
120
        buf = "/tmp\000\000\000\000\360\350\364\023[\177\000\000\000\320\365\023[\177\000\000t\n\327\023[\177\000\000\370\271\311\020[\177\000\000\017\000\000\000\000\000\000\000:\000\000\000\000\000\000\000h\344\364\023[\177\000\000.N=\366\000\000\000\000\236\022\327\023[\177\000\000\300\304S\205\377\177\000\000.\000\000\000\000\000\000\000 \305S\205\377\177\000\000\377\377\377\377\000\000\000\000\264\352\310\020[\177\000\000\250\354\310\020[\177\000\000\360\304S\205\377\177\000\000\360\350\364\023[\177\000\000@\256\311\020[\177", '\000' <repeats 18 times>"\340, \346\364\023[\177\000\000\000\320\365\023[\177\000\000\231,@\000\000\000\000\000\370\271\311\020[\177\000\000\340\033@\000\000\000\000\000\000\000\000\000\001\000\000\000\021\b\000\000\001", '\000' <repeats 11 times>, " \266\370\023[\177\000\000`\305S\205\377\177\000\000.N=\366\000\000\000\000\340\346\364\023[\177\000\000\200\305S\205\377\177\000\000"...
121
        opt_help = 0
122
        opt_version = 0
123
        opt_fork = 1
124
        opt_firstrun = 0
125
        opt_stderr = 0
126
        opt_syslog = 0
127
        opt_uidebug = 0
128
</pre>
129
130
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
131
132
However I'd strongly recommend that you keep a copy of tvheadend binary and core file in case further analysis is required.
133
134
h2. Dead or Live Lock
135
136
If Tvheadend appears to die but the process is still running, then its quite possible that the process is deadlocked (or possibly live locked). The best way to help investigate such a problem is to get a full stack trace from every thread in the system.
137
138
First attach gdb to the running process:
139
140
<pre>
141
gdb tvheadend pid
142
</pre>
143
144
You may need to replace _tvheadend_ with the full path to the binary and you will need to replace _pid_ with the PID of the running process. To find that run:
145
146
<pre>
147
ps -C tvheadend
148
</pre>
149
150
Once you have gdb attached grab a stack trace from every thread using the following command:
151
152
<pre>
153
(gdb) set logging on
154
(gdb) set pagination off
155
(gdb) thread apply all bt full
156
</pre>
157
158
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
159
160
It might also be useful to generate a core file for good measure:
161
162
<pre>
163
(gdb) generate-core-file
164
</pre>
165
166
This information may give an indication as to why things are locked, often 2 threads are stuck trying to lock a mutex (probably each holds the opposite lock).
167
168
h2. Reporting crash (or lock)
169
170
If you're going to report a crash (or lockup) then please try to provide the above information, including a debug log (or whatever logging you have), a core file and if you're not using a pre-built tvheadend package then the binary and basic information about the platform (distribution, version and architecture) you're running on.