Bug #1343: Crash in tvheadend - Tvheadend

Bug #1343

Crash in tvheadend

Added by Jeremy Bush about 12 years ago. Updated about 12 years ago.

Status:

Fixed

Priority:

Normal

Assignee:

Category:

EPG - Grabbers

Target version:

3.2

Start date:

2012-10-19

Due date:

% Done:

Estimated time:

Found in version:

release/3.2

Affected Versions:

Description

I'm getting crashes in the backend every so often. Here's a log entry of it:

Oct 18 22:34:32 htpc tvheadend[31684]: mkv: /media/Drobo01/dvr/6-1-WITIDT/Seinfeld.2012-10-18.22-35.mkv: Write failed -- No space left on device
Oct 18 22:34:32 htpc tvheadend[31684]: dvr: Recording error: "/media/Drobo01/dvr/6-1-WITIDT/Seinfeld.2012-10-18.22-35.mkv": Unable to init file
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: Signal: 11 in PRG: /usr/src/tvheadend/build.linux/tvheadend (3.1.769~g4303374-dirty) [7ebe00d49e654a5439f901b7395184165f1d9c97] CWD: /usr/src/tvheadend  
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: Fault address 0x4f18 (Address not mapped)
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: Loaded libraries: linux-gate.so.1 /usr/lib/libssl.so.1.0.0 /usr/lib/libcrypto.so.1.0.0 /lib/libz.so.1 /lib/librt.so.1 /lib/libdl.so.2 /lib/libpthread.so.0 /lib/libc.so.6 /lib/ld-linux.so.2  
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: Register dump [19]: 00000033 c1430000 ffff007b 0000007b ae492288 ae1be548 b09ff1d8 b09ff1c0 ae492288 ae3a4630 aeb00010 00004f10 0000000e 00000004 08057d52 00000073 00010246 b09ff1c0 0000007b 
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: STACKTRACE
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: /usr/src/tvheadend/src/trap.c:139 0x806c5ff
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: __kernel_rt_sigreturn+0x0  (linux-gate.so.1)
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: /usr/src/tvheadend/src/dvr/mkmux.c:898 0x8082031
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: /usr/src/tvheadend/src/muxer_tvh.c:124 0x808bafc
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: /usr/src/tvheadend/src/dvr/dvr_rec.c:445 0x807f1ae
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: ??:0 0xb76c6f07
Oct 18 22:36:42 htpc tvheadend[31684]: CRASH: clone+0x5e  (/lib/libc.so.6)
Oct 18 22:36:42 htpc kernel: [1678757.998492] tvheadend[19816]: segfault at 4f18 ip 08057d52 sp b09ff1c0 error 4 in tvheadend[8048000+72000]

Since my codebase is marked dirty, here's my changes. It's just some additional logging code:

diff --git a/src/htsmsg.c b/src/htsmsg.c
index 5dd67e3..35979fb 100644
--- a/src/htsmsg.c
+++ b/src/htsmsg.c
@@ -16,6 +16,7 @@
  *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */

+#include "tvheadend.h" 
 #include <assert.h>
 #include <sys/types.h>
 #include <stdio.h>
@@ -413,7 +414,10 @@ htsmsg_get_bin(htsmsg_t *msg, const char *name, const void **binp,
     return HTSMSG_ERR_FIELD_NOT_FOUND;

   if(f->hmf_type != HMF_BIN)
+  {
+    tvhlog(LOG_INFO, "htsp", "MSG Type: %i", f->hmf_type);
     return HTSMSG_ERR_CONVERSION_IMPOSSIBLE;
+  }

   *binp = f->hmf_bin;
   *lenp = f->hmf_binsize;
diff --git a/src/htsp.c b/src/htsp.c
index 789c036..90b009b 100644
--- a/src/htsp.c
+++ b/src/htsp.c
@@ -1249,8 +1249,14 @@ htsp_authenticate(htsp_connection_t *htsp, htsmsg_t *m)
     htsp_update_logname(htsp);
   }

-  if(htsmsg_get_bin(m, "digest", &digest, &digestlen))
+  int d = htsmsg_get_bin(m, "digest", &digest, &digestlen);
+  if(d)
+  {
+    tvhlog(LOG_INFO, "htsp", "%s: Digest is wrong: %i", htsp->htsp_logname, d);
     return;
+  }
+
+  tvhlog(LOG_INFO, "htsp", "%s: Trying digest...", htsp->htsp_logname);

   access = access_get_hashed(username, digest, htsp->htsp_challenge,
                             (struct sockaddr *)htsp->htsp_peer, &match);

History

Updated by Joakim Hernberg about 12 years ago

Here is some more info from the same type of crash:

(gdb) bt full
#0 opentv_parse_string (
buf=buf@entry=0x7ffff551b0bd "\304\331\366tv\216\366[7\335\032;dj\366d\344\036\260>\222\341\341\366\264\345U\343?\275>6v\037\377\266\201\203\243e\371m3@)\233\060\001\351\361\061\271\360%\n\304\331\366tv\216\366[7\335\032;d\205\252\323+>\345o\263\207\272I\264v\311\320p2\210W\247\302\236\373)\357\302\370\020?l\330\037Ip\360\373Zr\252\361\237\336\353lm+\367a\377\373h\030:6\226\323\064\002B\325\354\311\310=\235?\231\367\323{\377kx\317\357u\347\336_\357\310\376\344\344\275\311D<\223$\237\266+\232\275\305\305S\177\357I"..., len=len@entry=-2, prov=0x6a5aa0) at /home/jack/src/tvheadend-git/src/epggrab/module/opentv.c:222
ret = 0x0
tmp = 0x0
#1 0x0000000000431b1f in opentv_parse_event_record (mjd=1350777600, len=<optimized out>,
buf=0x7ffff551b0b4 "\265\005\261\031\246\273=%\n\304\331\366tv\216\366[7\335\032;dj\366d\344\036\260>\222\341\341\366\264\345U\343?\275>6v\037\377\266\201\203\243e\371m3@)\233\060\001\351\361\061\271\360%\n\304\331\366tv\216\366[7\335\032;d\205\252\323+>\345o\263\207\272I\264v\311\320p2\210W\247\302\236\373)\357\302\370\020?l\330\037Ip\360\373Zr\252\361\237\336\353lm+\367a\377\373h\030:6\226\323\064\002B\325\354\311\310=\235?\231\367\323{\377kx\317\357u\347\336_\357\310\376\344\344\275\311D<\223$\237\266"..., ev=0x7ffff551ac20, prov=0x6a5aa0) at /home/jack/src/tvheadend-git/src/epggrab/module/opentv.c:259
rtag = <optimized out>
rlen = 5 '\005'
#2 opentv_parse_event (type=<optimized out>, ev=0x7ffff551ac20, mjd=1350777600, cid=<optimized out>, len=925,
buf=0x7ffff551afc5 "\261\061\271\360%\n\304\331\366tv\216\366[7\335\032;d\205\252\323+>\345o\263\207\272I\264v\311\320p2\210W\247\302\236\373)\357\302\370\020?l\330\037Ip\360\373Zr\252\361\237\336\353lm+\367a\377\373h\030:6\226\323\064\002B\325\354\311\310=\235?\231\367\323{\377kx\317\357u\347\336_\357\310\376\344\344\275\311D<\223$\237\266+\232\275\305\305S\177\357I6v\364&\365.m;\376\213\357G\243\063\v\257/j\375\377G90<\034|\204-\233\375\265D\235I\237\070\215\347t\016D\356\216\f4\253\006V\336\375\255\062\207o\335\003\277"..., sta=0xf53c260, prov=0x6a5aa0) at /home/jack/src/tvheadend-git/src/epggrab/module/opentv.c:304
slen = <optimized out>
i = <optimized out>
e = <optimized out>
#3 _opentv_parse_event_section (mod=0x6a5aa0, sta=0xf53c260, buf=0x7ffff551ae73 "", len=1263, type=<optimized out>)
at /home/jack/src/tvheadend-git/src/epggrab/module/opentv.c:343
i = <optimized out>
cid = <optimized out>
---Type <return> to continue, or q <return> to quit---
save = 1
mjd = 1350777600
ec = 0x0
ebc = <optimized out>
ee = <optimized out>
es = <optimized out>
ev = {ev_link = {left = 0x0, right = 0x0, parent = 0x0, color = 0}, cid = 212, eid = 45361, start = 1350868274,
stop = 1350953640, title = 0x0, summary = 0x0, desc = 0x0, cat = 61 '=', serieslink = 0, type = 2 '\002'}
src = 0x6a5aa0
lang = 0x460c9b "it"
ecl = 0x7fffd4f108c0
PRETTY_FUNCTION = "_opentv_parse_event_section"
#4 0x0000000000445f0a in dvb_proc_table (r=<optimized out>, sec=0x7ffff551ae70 "\250\364", <incomplete sequence \363>,
tdt=0x12118e70, tdmi=0x1785fb0) at /home/jack/src/tvheadend-git/src/dvb/dvb_tables.c:171
chkcrc = <optimized out>
len = <optimized out>
ret = <optimized out>
tableid = <optimized out>
#5 dvb_table_input (aux=0xbe19b0) at /home/jack/src/tvheadend-git/src/dvb/dvb_tables.c:217
tda = 0xbe19b0
r = <optimized out>
i = <optimized out>
tid = 138
fd = 49
x = <optimized out>
ev = {{events = 1, data = {ptr = 0x310000008a, fd = 138, u32 = 138, u64 = 210453397642}}}
sec = "\250\364\363\000\324\341\000\000\001\350\361E\271\376%\n\304\317\035'\243\202M\263d\022m\035\262B\325\306j$\363\354\310\022\357K\207u\202\n\177\317\022p\203\242\351:\r\214\371\274\277\271A\347\373\036\177X1\374;as\030\253\241k\033\203\272~\235\376\300\266F\351\355};\020\355\273\063\r+\217?i\314M\214\273\356-W\277\265\005\274\355\"e\030\304-c\217<\333\no\374\325\354\027\221---Type <return> to continue, or q <return> to quit---
y\365\253t\373\207\071o\243\315>'8\232\022t\237\032\016J2`C+\025\277\006\"\212\257\321`A@\001'\360\301\064\323\066nVl\335o\277a\362\353\v\020\355\273#zwz(\260\215\262\233\227{O\336;\327x\bZ\245\n\304\334t\234\237\035''\247\376}\220z\376\274\202NO\223B\325@\347\311\326)\207)\277\034z\203\301\343\241X:\n\277m\210\340V\016\235{>,\351\370\273C%\n\304\317\035'\243\202M\263d\022m\035\262l\333N\377`lK+s\367\037\266o\263''\353\351\373\244\237N\344i\364\070:u\354{_N\304;n\340\342\324\017\016~\303KBl\302\001\061\271\360%\n\304\331\366tv\216\366[7\335\032;d\205\252\323+>\345o\263\207\272I\264v\311\320p2\210W\247\302\236\373)\357\302\370\020?l\330\037Ip\360\373Zr\252\361\237\336\353lm+\367a\377\373h\030:"...
tdmi = 0x1785fb0
tdt = 0x12118e70
cycle_barrier = 0
PRETTY_FUNCTION = "dvb_table_input"
#6 0x00007ffff6b13e0f in start_thread () from /usr/lib/libpthread.so.0
No symbol table info available.
#7 0x00007ffff684b45d in clone () from /usr/lib/libc.so.6
No symbol table info available.
(gdb)

Updated by Adam Sutton about 12 years ago

It really does look like the problem is that malloc() has returned NULL, I don't explicitly check for stuff like this since malloc() returning NULL should only happen if we've run out of memory and to be honest if that happens we're screwed!

The only reason this should happen is a) you've got very limited memory b) you've used it all up elsewhere or c) tvh has a memleak.

Maybe keep a watch on memory usage, use something like sar (pkg sysstat) to see whether mem usage is going up.

Also interesting is that you appear to have run out of disk space for the recordings, could this be linked somehow (storing stuff in tmpfs?).

Adam

Updated by Adam Sutton about 12 years ago

Category set to EPG - Grabbers
Status changed from New to Need feedback

Updated by Jeremy Bush about 12 years ago

I have plenty of memory, no other apps seem to run out of memory. I did paste the ran out of disk space in case it was relevant. I cleared up the hard drive and we'll see if it crashes again. Even so, I wouldn't expect the program to crash on no disk space (just generate lots of errors).

Updated by Adam Sutton about 12 years ago

The only reason I mentioned was in case you knew of a link (such as if you were storing files to memory etc..). If it was genuinely that you ran out of disk space I wouldn't expect them to be related.

Check mem usage on tvh and see if its obviously leaking, I certainly wouldn't be completely surprised. Though odd no one else has noticed it.

Adam

Updated by Adam Sutton about 12 years ago

Actually I've just realised something...

Didn't John T submit some fixes yesterday/today that fixed memory leaks in MKV muxer? And didn't these relate to error scenarios?

Could it simply be that by running out of disk space you were thus creating MKV recording errors and in turn exposing the memory leaks that John fixed?

If that's true then the clearing of disk space will remove the problem. If not it will continue to happen and we'll investigate further.

Adam

Updated by Adam Sutton about 12 years ago

Status changed from Need feedback to Fixed
Target version set to 3.2

I'm pretty sure this will have been a result of a memory leak caused by a bug in the MKV muxer which has now been fixed.

Updated by Jeremy Bush about 12 years ago

I haven't had any problems in the last few days, so if I do, i'll file a new issue.

Also available in: Atom PDF

Project

General

Profile

Tvheadend

Issues

Custom queries

Bug #1343

Crash in tvheadend

History

Updated by Joakim Hernberg about 12 years ago

Updated by Adam Sutton about 12 years ago

Updated by Adam Sutton about 12 years ago

Updated by Jeremy Bush about 12 years ago

Updated by Adam Sutton about 12 years ago

Updated by Adam Sutton about 12 years ago

Updated by Adam Sutton about 12 years ago

Updated by Jeremy Bush about 12 years ago