Tvheadend

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Incorrect-not-useable-crash-reports

Going to state the obvious here.. Based on the paths I'm going to assume it's a NAS? This one is might be hard to debug especially as the crash lines give little away!

Yes, its on a Qnap NAS - have enabled core dumps to see if it dies but I've got a theory now about the cause. It seems the Mux it crashed on is empty. Crystal Palace seems to have a extra Mux with No channels in it. I disabled this Mux and so far its stayed running.

Jeff B wrote:

Yes, its on a Qnap NAS - have enabled core dumps to see if it dies but I've got a theory now about the cause. It seems the Mux it crashed on is empty. Crystal Palace seems to have a extra Mux with No channels in it. I disabled this Mux and so far its stayed running.

Are you able to capture a sample of the mux? (just wget the faulty mux's play link from the muxes tab).

Mark Clarkstone wrote:

Are you able to capture a sample of the mux? (just wget the faulty mux's play link from the muxes tab).

Yes, just did that and I get:

Connecting to 192.168.0.8:9981 (192.168.0.8:9981)
42767040ee1589356ee7 100% |*******************************| 156 0:00:00 ETA

and it pulls down:

#EXTM3U
#EXTINF:-1,714MHz / Freeview
http://192.168.0.8:9981/stream/mux/42767040ee1589356ee70b1d762365cc?ticket=2779334481f6a4cad56b83243cdb482ddc2994af

Install addr2line binary on your platform (with the tvh debug package). The addr2line tool is in the binutils package usually.

Jaroslav Kysela wrote:

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Incorrect-not-useable-crash-reports

What's the correct method to include the debugging symbols when building TVHeadend from source code (i.e. not using the "debug package")? The linked wiki page doesn't say how to do it.

I've tried with a "--enable-debug" configure switch, but the stacktrace when testing a crash with the "-A" command line gives for example:

2018-01-10 19:17:55.634 [  ALERT] CRASH: STACKTRACE
2018-01-10 19:17:55.637 [  ALERT] CRASH:  0x7ff6a921d30a 0x7ff6a8fc6000
2018-01-10 19:17:55.639 [  ALERT] CRASH:  0x7ff6a7e6b0c0 0x7ff6a7e5a000
2018-01-10 19:17:55.639 [  ALERT] CRASH: gsignal+0xcf  (/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
2018-01-10 19:17:55.639 [  ALERT] CRASH: abort+0x16a  (/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
2018-01-10 19:17:55.642 [  ALERT] CRASH:  0x7ff6a91d653a 0x7ff6a8fc6000
2018-01-10 19:17:55.642 [  ALERT] CRASH: __libc_start_main+0xf1  (/share/CACHEDEV1_DATA/.qpk/TVHeadend/bin/libc/libc.so.6)

virtual dj wrote:

I've tried with a "--enable-debug" configure switch, but the stacktrace when testing a crash with the "-A" command line gives for example:
[...]which doesn't seem to include the symbols, right?

It appears that only tvheadend binary has no symbols. The glibc symbols are resolved. You may ask the package maintainer to provide also debug version of the tvheadend binary with unstripped symbols.

Jaroslav Kysela wrote:

You may ask the package maintainer to provide also debug version of the tvheadend binary with unstripped symbols.

Actually I've compiled the tvheadend binary from source code myself, hence I'm asking how to create the debug version.
I've configured it with:

# cat .config.mk 
# Automatically generated by configure - DO NOT EDIT!
CONFIGURE_ARGS = --disable-pcre --disable-avahi --enable-bundle --disable-dbus_1 --enable-hdhomerun_client --enable-libfdkaac --enable-debugging
ROOTDIR  ?= /root/tvheadend/tvheadend
BUILDDIR ?= /root/tvheadend/tvheadend/build.linux
PLATFORM ?= linux
OSENV    ?= posix
ARCH     ?= x86_64
CPU      ?= generic
COMPILER ?= gcc
ifeq ($(origin CC),default)
CC        = cc
endif
PYTHON   ?= python
GZIPCMD  ?= gzip
BZIP2    ?= bzip2
XGETTEXT ?= xgettext
MSGMERGE ?= msgmerge
PKG_CONFIG ?= pkg-config
CFLAGS   += 
LDFLAGS  += 
prefix    = /usr/local
bindir    = ${prefix}/bin
mandir    = ${datadir}/man
datadir   = ${prefix}/share
libdir    = ${prefix}/lib
LDFLAGS += -lssl -lcrypto
CFLAGS  += 
LDFLAGS += -lz
CFLAGS  += 
LDFLAGS += -luriparser
CFLAGS  += 
CONFIG_PIE = yes
CONFIG_CCDEBUG = no
CONFIG_CARDCLIENT = yes
CONFIG_CWC = yes
CONFIG_CCCAM = yes
CONFIG_CAPMT = yes
CONFIG_CONSTCW = yes
CONFIG_LINUXDVB = yes
CONFIG_SATIP_SERVER = yes
CONFIG_SATIP_CLIENT = yes
CONFIG_HDHOMERUN_CLIENT = yes
CONFIG_HDHOMERUN_STATIC = yes
CONFIG_IPTV = yes
CONFIG_TSFILE = yes
CONFIG_DVBSCAN = yes
CONFIG_TIMESHIFT = yes
CONFIG_TRACE = yes
CONFIG_IMAGECACHE = yes
CONFIG_AVAHI = no
CONFIG_ZLIB = yes
CONFIG_LIBAV = yes
CONFIG_FFMPEG_STATIC = yes
CONFIG_LIBX264 = yes
CONFIG_LIBX264_STATIC = yes
CONFIG_LIBX265 = yes
CONFIG_LIBX265_STATIC = yes
CONFIG_LIBVPX = yes
CONFIG_LIBVPX_STATIC = yes
CONFIG_LIBTHEORA = yes
CONFIG_LIBTHEORA_STATIC = yes
CONFIG_LIBVORBIS = yes
CONFIG_LIBVORBIS_STATIC = yes
CONFIG_LIBFDKAAC = yes
CONFIG_LIBFDKAAC_STATIC = yes
CONFIG_LIBOPUS = yes
CONFIG_LIBOPUS_STATIC = yes
CONFIG_NVENC = no
CONFIG_VAAPI = no
CONFIG_MMAL = no
CONFIG_OMX = no
CONFIG_INOTIFY = yes
CONFIG_EPOLL = yes
CONFIG_PCRE = no
CONFIG_PCRE2 = no
CONFIG_URIPARSER = yes
CONFIG_CCACHE = no
CONFIG_TVHCSA = yes
CONFIG_BUNDLE = yes
CONFIG_PNGQUANT = no
CONFIG_DVBCSA = no
CONFIG_KQUEUE = no
CONFIG_DBUS_1 = no
CONFIG_ANDROID = no
CONFIG_TSDEBUG = no
CONFIG_GTIMER_CHECK = no
CONFIG_SLOW_MEMORYINFO = no
CONFIG_LIBSYSTEMD_DAEMON = no
CONFIG_BINTRAY_CACHE = yes
CONFIG_DDCI = no
CONFIG_EXECINFO = yes
CONFIG_MMX = yes
CONFIG_SSE2 = yes
CONFIG_W_UNUSED_RESULT = yes
CONFIG_GETLOADAVG = yes
CONFIG_ATOMIC32 = yes
CONFIG_ATOMIC64 = yes
CONFIG_ATOMIC_TIME_T = yes
CONFIG_ATOMIC_PTR = yes
CONFIG_BITOPS64 = yes
CONFIG_LOCKOWNER = yes
CONFIG_QSORT_R = yes
CONFIG_STIME = yes
CONFIG_GMTOFF = yes
CONFIG_RECVMMSG = yes
CONFIG_SENDMMSG = yes
CONFIG_IFNAMES = yes
CONFIG_PY_GZIP = yes
CONFIG_BIN_PKG_CONFIG = yes
CONFIG_BIN_XGETTEXT = yes
CONFIG_BIN_MSGMERGE = yes
CONFIG_BIN_GZIP = yes
CONFIG_BIN_BZIP2 = yes
CONFIG_SSL = yes
CONFIG_LINUXDVBAPI = yes
CONFIG_LINUXDVB_CA = yes
CONFIG_UPNP = yes
CONFIG_BIN_CMAKE = yes
CONFIG_STDCPP = yes
CONFIG_LIBOGG_STATIC = yes
CONFIG_INOTIFY_H = yes
CONFIG_INOTIFY_INIT1 = yes
CONFIG_EPOLL_CREATE1 = yes
CONFIG_MPEGTS = yes
CONFIG_MPEGTS_DVB = yes

Have you try adding "-g" to Cflags ?

Tip: I've lose a lot of time in an my friend's system because while debug symbols where correctly included, they were removed by "strip" by its distribution package maker... i.e. the default behavior of makepkg in archlinux is to strip debug symbols....

There's automatic '-g' in tvh's Makefile - see the first lines. But if you use 'install -s' utility for example, then symbols will be stripped. Example:

$ nm ./build.linux/tvheadend  | grep linuxdvb_init
000000000032923c T linuxdvb_init
$ nm /bin/ls
nm: /bin/ls: no symbols

In my compiled binary (post #10):

root@debian-x64:~/tvheadend/tvheadend# nm ./build.linux/tvheadend | grep linuxdvb_init
00000000002f8d30 T linuxdvb_init

Yes, it appears that used addr2line does not work on QNAP correctly for the binary. You can resolve addresses also on build host:

...
2018-01-12 08:39:17.532 [  ALERT] CRASH: /home/perex/git/tvheadend/src/main.c:1282 0x55c6dbdb9b4f 0x55c6dbbe0000
...
$ addr2line -e ./build.linux/tvheadend $(python -c 'print("%x" % (0x55c6dbdb9b4f-0x55c6dbbe0000-1))')
/home/perex/git/tvheadend/src/main.c:1282

But it would be probably nice to debug why addr2line does not work on QNAP for tvh's binary, too.

On the QNAP, by trying the addresses on post 10 (simulated crashing) I get:

[/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin] # ./addr2line -e tvheadend $(python -c 'print("%x" % (0x7ff6a921d30a-0x7ff6a8fc6000-1))')
/root/tvheadend/tvheadend/src/trap.c:148

Mmm, but the couple of addresses immediately after gives:

[/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin] # ./addr2line -e tvheadend $(python -c 'print("%x" % (0x7ff6a7e6b0c0-0x7ff6a7e5a000-1))')
??:0

Try remove -1 from the address expression.

Jaroslav Kysela wrote:

Try remove -1 from the address expression.

Just tried:

[/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin] # ./addr2line -e tvheadend $(python -c 'print("%x" % (0x7ff6a7e6b0c0-0x7ff6a7e5a000))')
??:0

Just FYI, This seems to be resolved now with newer builds so some other change must have fixed it.