Bug #3381: Major crash !? - Tvheadend

Bug #3381

Major crash !?

Added by Bengt Madeberg almost 9 years ago. Updated almost 9 years ago.

Status:

Fixed

Priority:

Normal

Assignee:

Category:

Crashes

Target version:

Start date:

2015-12-05

Due date:

% Done:

100%

Estimated time:

Found in version:

HTS Tvheadend 4.1-1159~g4484874

Affected Versions:

Description

tvheadend is running but no client can reach it or the web interface. Top shows

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8733 hts       20   0 6120424 3.112g    164 S 100.5 83.2 645:57.08 tvheadend
 1035 nobody    20   0  709136   6908    264 S   0.7  0.2 149:10.32 nodejs

Cant fin anything more in syslog except this:

Dec  5 07:17:01 server1a CRON[11351]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  5 08:01:48 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5705094752, last = 2884839841
Dec  5 08:09:58 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5661009152, last = 2928925441
Dec  5 08:11:25 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5653200752, last = 2936733841
Dec  5 08:17:01 server1a CRON[11418]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  5 08:19:19 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5610486752, last = 2979447841
Dec  5 08:28:08 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5562955952, last = 3026978641
Dec  5 08:28:21 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5561757152, last = 3028177441
Dec  5 08:29:52 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5553538352, last = 3036396241
Dec  5 08:31:16 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5546032352, last = 3043902241
Dec  5 08:42:16 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5486632352, last = 3103302241
Dec  5 08:47:40 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5457418352, last = 3132516241
Dec  5 08:47:47 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5456791952, last = 3133142641
Dec  5 08:52:43 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5430137552, last = 3159797041
Dec  5 09:17:01 server1a CRON[11514]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  5 09:18:51 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5289046352, last = 3300888241
Dec  5 09:19:12 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5287221152, last = 3302713441
Dec  5 09:19:59 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5282987552, last = 3306947041
Dec  5 09:23:46 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5262564752, last = 3327369841
Dec  5 09:34:30 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5204536352, last = 3385398241
Dec  5 09:49:02 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5126063552, last = 3463871041
Dec  5 09:50:09 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5120015552, last = 3469919041
Dec  5 09:54:26 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5096892752, last = 3493041841
Dec  5 09:55:12 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5092831952, last = 3497102641
Dec  5 10:09:57 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -5013181952, last = 3576752641
Dec  5 10:12:30 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4999411952, last = 3590522641
Dec  5 10:17:01 server1a CRON[11582]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  5 10:37:53 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4862251952, last = 3727682641
Dec  5 10:38:48 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4857305552, last = 3732629041
Dec  5 10:39:06 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4855717952, last = 3734216641
Dec  5 10:39:29 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4853698352, last = 3736236241
Dec  5 10:40:27 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4848427952, last = 3741506641
Dec  5 10:49:58 server1a tvheadend[8733]: parser: transport stream TEXTSUB, DTS discontinuity. DTS = -4797127952, last = 3792806641

I did a restart and now it's ok. I will wait and see if this occurs again.

Files

gdb.txt (256 KB) gdb.txt		Bengt Madeberg, 2015-12-06 23:38
gdb.txt (91.6 KB) gdb.txt		B C, 2015-12-07 15:46
gdb.txt (341 KB) gdb.txt		Bengt Madeberg, 2015-12-28 23:51
gdb.txt (637 KB) gdb.txt		Bengt Madeberg, 2015-12-29 00:29

History

Updated by Mario D almost 9 years ago

If it happens again, don't stop the service. Follow section 'Dead or Live Lock' in https://tvheadend.org/projects/tvheadend/wiki/Debugging to report useful information to developers.

Updated by Bengt Madeberg almost 9 years ago

Ok. I will do that. If it happened once it will probably happen twice...

Updated by Bengt Madeberg almost 9 years ago

Once again.

Syslog

Dec  6 20:48:47 server1a tvheadend[11714]: TS: Teracom/570MHz/SVT1 HD Väst: AC3 @ #1417: Invalid start code 34:ea:f1
Dec  6 20:48:47 server1a tvheadend[11714]: TS: Teracom/570MHz/SVT1 HD Väst: H264 @ #1419: Invalid start code 2d:86:25
Dec  6 20:48:49 server1a tvheadend[11714]: parser: transport stream AAC, DTS discontinuity. DTS = 276923700, last = 276758580
Dec  6 20:49:00 server1a tvheadend[11714]: parser: transport stream AC3, DTS discontinuity. DTS = 277905600, last = 277801920
Dec  6 20:49:00 server1a tvheadend[11714]: parser: transport stream AAC, DTS discontinuity. DTS = 277925940, last = 277829940
Dec  6 21:13:10 server1a tvheadend[11714]: parser: transport stream AAC, DTS discontinuity. DTS = 408389940, last = 408190260
Dec  6 21:17:01 server1a CRON[18143]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  6 21:23:13 server1a kernel: [4485621.153172] CIFS VFS: Send error in SETFSUnixInfo = -11
Dec  6 21:23:24 server1a kernel: [4485631.219475] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.241183] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.251749] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.261900] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.272666] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.293384] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.304007] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.318823] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.334911] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.352553] CIFS VFS: Error -32 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.365906] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.400299] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.412675] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.428410] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.443055] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.453334] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.463671] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.473780] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.483842] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.514948] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.525044] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.545320] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.555648] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.565891] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.589792] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.610105] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.655456] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.671155] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.715632] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.780007] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.842856] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.860420] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.872414] CIFS VFS: Error -32 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.916510] CIFS VFS: Error -32 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.934964] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.956926] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.979705] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485631.993521] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.027514] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.078862] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.089659] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.101790] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.132253] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.152756] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.162835] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.173921] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:24 server1a kernel: [4485632.185127] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.196076] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.206704] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.217049] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.228914] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.262165] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.275083] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.292253] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.323639] CIFS VFS: Error -32 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.370658] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.411540] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:25 server1a kernel: [4485632.425636] CIFS VFS: Error -104 sending data on socket to server
Dec  6 21:23:39 server1a kernel: [4485646.248860] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.249413] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.249957] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.250558] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.251100] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.251640] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.252177] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.252720] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.253293] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.253838] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.254370] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.254902] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.255434] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.255964] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.256560] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.257098] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.257624] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.258151] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.258674] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.259250] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.259769] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.260287] CIFS VFS: No writable handles for inode
Dec  6 21:23:39 server1a kernel: [4485646.260800] CIFS VFS: No writable handles for inode

Dead lock log

Thread 48 (Thread 0x7f50050c9700 (LWP 11715)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000000434197 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50050c9700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50050c9700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981658822400, -7010849927254589483, 0, 0, 139981658823104, 139981658822400, 6923298422718644181, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 47 (Thread 0x7f50048c8700 (LWP 11717)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004461f7 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50048c8700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50048c8700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981650429696, -7010849927254589483, 0, 0, 139981650430400, 139981650429696, 6923295116130697173, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 46 (Thread 0x7f50040c7700 (LWP 11718)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x000000000045178d in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50040c7700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50040c7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981642036992, -7010849927254589483, 0, 0, 139981642037696, 139981642036992, 6923296216179195861, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 45 (Thread 0x7f50038c6700 (LWP 11719)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000000000043317f in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50038c6700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50038c6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981633644288, -7010849927254589483, 0, 0, 139981633644992, 139981633644288, 6923284122088161237, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 44 (Thread 0x7f50030c5700 (LWP 11720)):
#0  0x00007f50061e83bd in read () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000047e63d in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50030c5700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50030c5700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981625251584, -7010849927254589483, 0, 0, 139981625252288, 139981625251584, 6923285222136659925, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 43 (Thread 0x7f50028c4700 (LWP 11721)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000000477357 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50028c4700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50028c4700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981616858880, -7010849927254589483, 0, 0, 139981616859584, 139981616858880, 6923281924138647509, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 42 (Thread 0x7f50020c3700 (LWP 11722)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x000000000047cc83 in ?? ()
No symbol table info available.
#3  0x000000000043c37e in ?? ()
No symbol table info available.
#4  0x00007f50061e1182 in start_thread (arg=0x7f50020c3700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50020c3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981608466176, -7010849927254589483, 0, 0, 139981608466880, 139981608466176, 6923283024187146197, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#5  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 41 (Thread 0x7f50018c2700 (LWP 11723)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000000000045a85f in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50018c2700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50018c2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981600073472, -7010849927254589483, 0, 0, 139981600074176, 139981600073472, 6923288522282155989, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 40 (Thread 0x7f50010c1700 (LWP 11724)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50010c1700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50010c1700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981591680768, -7010849927254589483, 0, 0, 139981591681472, 139981591680768, 6923289622330654677, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 39 (Thread 0x7f50008c0700 (LWP 11725)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f50008c0700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f50008c0700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981583288064, -7010849927254589483, 0, 0, 139981583288768, 139981583288064, 6923286298562838485, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 38 (Thread 0x7f4ffbfff700 (LWP 11726)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x000000000050588f in ?? ()
No symbol table info available.
#3  0x000000000043c37e in ?? ()
No symbol table info available.
#4  0x00007f50061e1182 in start_thread (arg=0x7f4ffbfff700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ffbfff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981507000064, -7010849927254589483, 0, 0, 139981507000768, 139981507000064, 6929776338281349077, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#5  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 37 (Thread 0x7f4ffb7fe700 (LWP 11727)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f50061e3657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x00007f50061e3480 in __GI___pthread_mutex_lock (mutex=0x10b74a0) at ../nptl/pthread_mutex_lock.c:79
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock" 
        type = 4294966784
#3  0x00000000004cabdc in ?? ()
No symbol table info available.
#4  0x000000000043c37e in ?? ()
No symbol table info available.
#5  0x00007f50061e1182 in start_thread (arg=0x7f4ffb7fe700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ffb7fe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981498607360, -7010849927254589483, 0, 0, 139981498608064, 139981498607360, 6929777438329847765, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#6  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 36 (Thread 0x7f4ffaffd700 (LWP 11728)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4ffaffd700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ffaffd700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981490214656, -7010849927254589483, 0, 0, 139981490215360, 139981490214656, 6929774140331835349, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 35 (Thread 0x7f4ffa7fc700 (LWP 11729)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f50061e3657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x00007f50061e3480 in __GI___pthread_mutex_lock (mutex=0x10b74a0) at ../nptl/pthread_mutex_lock.c:79
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock" 
        type = 4294966784
#3  0x00000000004cabdc in ?? ()
No symbol table info available.
#4  0x000000000043c37e in ?? ()
No symbol table info available.
#5  0x00007f50061e1182 in start_thread (arg=0x7f4ffa7fc700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ffa7fc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981481821952, -7010849927254589483, 0, 0, 139981481822656, 139981481821952, 6929775240380334037, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#6  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 34 (Thread 0x7f4ff9ffb700 (LWP 11730)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4ff9ffb700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ff9ffb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981473429248, -7010849927254589483, 0, 0, 139981473429952, 139981473429248, 6929780738475343829, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 33 (Thread 0x7f4ff97fa700 (LWP 11731)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4ff97fa700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ff97fa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981465036544, -7010849927254589483, 0, 0, 139981465037248, 139981465036544, 6929781838523842517, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 32 (Thread 0x7f4ff8ff9700 (LWP 11732)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4ff8ff9700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4ff8ff9700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981456643840, -7010849927254589483, 0, 0, 139981456644544, 139981456643840, 6929778540525830101, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 31 (Thread 0x7f4feffff700 (LWP 11733)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4feffff700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4feffff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981305673472, -7010849927254589483, 0, 0, 139981305674176, 139981305673472, 6929820318746460117, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 30 (Thread 0x7f4fef7fe700 (LWP 11734)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fef7fe700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fef7fe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981297280768, -7010849927254589483, 0, 0, 139981297281472, 139981297280768, 6929821418794958805, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 29 (Thread 0x7f4feeffd700 (LWP 11735)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4feeffd700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4feeffd700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981288888064, -7010849927254589483, 0, 0, 139981288888768, 139981288888064, 6929818120796946389, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 28 (Thread 0x7f4fee7fc700 (LWP 11736)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fee7fc700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fee7fc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981280495360, -7010849927254589483, 0, 0, 139981280496064, 139981280495360, 6929819220845445077, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 27 (Thread 0x7f4fedffb700 (LWP 11737)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fedffb700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fedffb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981272102656, -7010849927254589483, 0, 0, 139981272103360, 139981272102656, 6929824718940454869, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 26 (Thread 0x7f4fed7fa700 (LWP 11738)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fed7fa700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fed7fa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981263709952, -7010849927254589483, 0, 0, 139981263710656, 139981263709952, 6929825818988953557, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 25 (Thread 0x7f4fecff9700 (LWP 11739)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fecff9700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fecff9700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981255317248, -7010849927254589483, 0, 0, 139981255317952, 139981255317248, 6929822520990941141, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 24 (Thread 0x7f4fec7f8700 (LWP 11740)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cd6e5 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fec7f8700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fec7f8700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981246924544, -7010849927254589483, 0, 0, 139981246925248, 139981246924544, 6929823612449505237, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 23 (Thread 0x7f4febff7700 (LWP 11741)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000000000050bf73 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4febff7700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4febff7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981238531840, -7010849927254589483, 0, 0, 139981238532544, 139981238531840, 6929811518358470613, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 22 (Thread 0x7f4feb7f6700 (LWP 11742)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f50061e3657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x00007f50061e3480 in __GI___pthread_mutex_lock (mutex=0x10b74a0) at ../nptl/pthread_mutex_lock.c:79
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock" 
        type = 4294966784
#3  0x00000000004401b2 in ?? ()
No symbol table info available.
#4  0x000000000043c37e in ?? ()
No symbol table info available.
#5  0x00007f50061e1182 in start_thread (arg=0x7f4feb7f6700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4feb7f6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981230139136, -7010849927254589483, 0, 0, 139981230139840, 139981230139136, 6929812618406969301, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#6  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 21 (Thread 0x7f4feaff5700 (LWP 11743)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x00000000004bd101 in ?? ()
No symbol table info available.
#3  0x000000000043c37e in ?? ()
No symbol table info available.
#4  0x00007f50061e1182 in start_thread (arg=0x7f4feaff5700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4feaff5700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981221746432, -7010849927254589483, 0, 0, 139981221747136, 139981221746432, 6929809320408956885, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#5  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 20 (Thread 0x7f4fea7f4700 (LWP 11744)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000000478fbf in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fea7f4700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fea7f4700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981213353728, -7010849927254589483, 0, 0, 139981213354432, 139981213353728, 6929810420457455573, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 19 (Thread 0x7f4fe9ff3700 (LWP 11745)):
#0  0x00007f50059f312d in poll () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x0000000000440b70 in ?? ()
No symbol table info available.
#2  0x00000000005182ab in ?? ()
No symbol table info available.
#3  0x0000000000518375 in ?? ()
No symbol table info available.
#4  0x0000000000519b28 in ?? ()
No symbol table info available.
#5  0x000000000043c37e in ?? ()
No symbol table info available.
#6  0x00007f50061e1182 in start_thread (arg=0x7f4fe9ff3700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe9ff3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981204961024, -7010849927254589483, 0, 0, 139981204961728, 139981204961024, 6929815918552465365, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#7  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 18 (Thread 0x7f4fe97f2700 (LWP 11746)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x0000000000450742 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fe97f2700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe97f2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981196568320, -7010849927254589483, 0, 0, 139981196569024, 139981196568320, 6929817018600964053, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 17 (Thread 0x7f4fe8ff1700 (LWP 11747)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x0000000000517471 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fe8ff1700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe8ff1700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981188175616, -7010849927254589483, 0, 0, 139981188176320, 139981188175616, 6929813720602951637, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 16 (Thread 0x7f4fe3fff700 (LWP 11748)):
#0  0x00007f50061e83bd in read () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000050fd9d in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fe3fff700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe3fff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981104346880, -7010849927254589483, 0, 0, 139981104347584, 139981104346880, 6929829114839482325, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 15 (Thread 0x7f4fe37fe700 (LWP 11749)):
#0  0x00007f50061e8b9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00000000004c3679 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fe37fe700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe37fe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981095954176, -7010849927254589483, 0, 0, 139981095954880, 139981095954176, 6929830214887981013, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 14 (Thread 0x7f4fe2ffd700 (LWP 11750)):
#0  0x00007f50059f312d in poll () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007f5006a5851b in avahi_simple_poll_run () from /usr/lib/x86_64-linux-gnu/libavahi-common.so.3
No symbol table info available.
#2  0x00007f5006a586f8 in avahi_simple_poll_iterate () from /usr/lib/x86_64-linux-gnu/libavahi-common.so.3
No symbol table info available.
#3  0x0000000000510541 in ?? ()
No symbol table info available.
#4  0x000000000043c37e in ?? ()
No symbol table info available.
#5  0x00007f50061e1182 in start_thread (arg=0x7f4fe2ffd700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe2ffd700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981087561472, -7010849927254589483, 0, 0, 139981087562176, 139981087561472, 6929826916889968597, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#6  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 13 (Thread 0x7f4fb7fff700 (LWP 11755)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004cad0b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fb7fff700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fb7fff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980366149376, -7010849927254589483, 0, 0, 139980366150080, 139980366149376, 6929661989072060373, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 12 (Thread 0x7f4fb77fe700 (LWP 11756)):
#0  0x0000000000493433 in ?? ()
No symbol table info available.
#1  0x00000000004ce78e in ?? ()
No symbol table info available.
#2  0x00000000004cea81 in ?? ()
No symbol table info available.
#3  0x00000000004cd196 in ?? ()
No symbol table info available.
#4  0x000000000043c37e in ?? ()
No symbol table info available.
#5  0x00007f50061e1182 in start_thread (arg=0x7f4fb77fe700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fb77fe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980357756672, -7010849927254589483, 0, 0, 139980357757376, 139980357756672, 6929663089120559061, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#6  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 11 (Thread 0x7f4fb6ffd700 (LWP 11757)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x0000000000500eb9 in ?? ()
No symbol table info available.
#3  0x000000000043c37e in ?? ()
No symbol table info available.
#4  0x00007f50061e1182 in start_thread (arg=0x7f4fb6ffd700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fb6ffd700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980349363968, -7010849927254589483, 0, 0, 139980349364672, 139980349363968, 6929659791122546645, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#5  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 10 (Thread 0x7f4fb67fc700 (LWP 17337)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f50061e3657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x00007f50061e3480 in __GI___pthread_mutex_lock (mutex=0x10b74a0) at ../nptl/pthread_mutex_lock.c:79
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock" 
        type = 4294966784
#3  0x00000000004611be in ?? ()
No symbol table info available.
#4  0x000000000044000d in ?? ()
No symbol table info available.
#5  0x000000000043c37e in ?? ()
No symbol table info available.
#6  0x00007f50061e1182 in start_thread (arg=0x7f4fb67fc700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fb67fc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980340971264, -7010849927254589483, 0, 0, 139980340971968, 139980340971264, 6929660891171045333, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#7  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 9 (Thread 0x7f4f6effd700 (LWP 17338)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000000000045e38b in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4f6effd700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4f6effd700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139979141404416, -7010849927254589483, 0, 0, 139979141405120, 139979141404416, 6929536645820235733, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 8 (Thread 0x7f4fe1ffb700 (LWP 17646)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004a98a7 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fe1ffb700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe1ffb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981070776064, -7010849927254589483, 0, 0, 139981070776768, 139981070776064, 6929833515033477077, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 7 (Thread 0x7f4fb5ffb700 (LWP 17647)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004a98a7 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fb5ffb700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fb5ffb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980332578560, -7010849927254589483, 0, 0, 139980332579264, 139980332578560, 6929666389266055125, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 6 (Thread 0x7f4fe17fa700 (LWP 17648)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00000000004a98a7 in ?? ()
No symbol table info available.
#2  0x000000000043c37e in ?? ()
No symbol table info available.
#3  0x00007f50061e1182 in start_thread (arg=0x7f4fe17fa700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe17fa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981062383360, -7010849927254589483, 0, 0, 139981062384064, 139981062383360, 6929834615081975765, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#4  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 5 (Thread 0x7f4fe27fc700 (LWP 17649)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x00000000004f5b7c in ?? ()
No symbol table info available.
#3  0x000000000043c37e in ?? ()
No symbol table info available.
#4  0x00007f50061e1182 in start_thread (arg=0x7f4fe27fc700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe27fc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981079168768, -7010849927254589483, 0, 0, 139981079169472, 139981079168768, 6929828016938467285, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#5  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 4 (Thread 0x7f4fa37fe700 (LWP 17652)):
#0  0x00007f5005a00b13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x000000000046dce0 in ?? ()
No symbol table info available.
#2  0x00000000004f5b7c in ?? ()
No symbol table info available.
#3  0x000000000043c37e in ?? ()
No symbol table info available.
#4  0x00007f50061e1182 in start_thread (arg=0x7f4fa37fe700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fa37fe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980022212352, -7010849927254589483, 0, 0, 139980022213056, 139980022212352, 6929689477399625685, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#5  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 3 (Thread 0x7f4fb4ff9700 (LWP 17692)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x00000000004b426b in ?? ()
No symbol table info available.
#2  0x0000000000444ff0 in ?? ()
No symbol table info available.
#3  0x0000000000445c21 in ?? ()
No symbol table info available.
#4  0x00000000004454f3 in ?? ()
No symbol table info available.
#5  0x0000000000445f0d in ?? ()
No symbol table info available.
#6  0x0000000000445fcd in ?? ()
No symbol table info available.
#7  0x000000000044000d in ?? ()
No symbol table info available.
#8  0x000000000043c37e in ?? ()
No symbol table info available.
#9  0x00007f50061e1182 in start_thread (arg=0x7f4fb4ff9700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fb4ff9700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139980315793152, -7010849927254589483, 0, 0, 139980315793856, 139980315793152, 6929664191316541397, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#10 0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 2 (Thread 0x7f4fe0ff9700 (LWP 17694)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f50061e3657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x00007f50061e3480 in __GI___pthread_mutex_lock (mutex=0x10b74a0) at ../nptl/pthread_mutex_lock.c:79
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock" 
        type = 4294966784
#3  0x0000000000445fde in ?? ()
No symbol table info available.
#4  0x000000000044000d in ?? ()
No symbol table info available.
#5  0x000000000043c37e in ?? ()
No symbol table info available.
#6  0x00007f50061e1182 in start_thread (arg=0x7f4fe0ff9700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f4fe0ff9700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139981053990656, -7010849927254589483, 0, 0, 139981053991360, 139981053990656, 6929831317083963349, 6923291700201832405}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#7  0x00007f5005a0047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 1 (Thread 0x7f5007f05a40 (LWP 11714)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f50061e3657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2  0x00007f50061e3480 in __GI___pthread_mutex_lock (mutex=0x7f4fb0002c38) at ../nptl/pthread_mutex_lock.c:79
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock" 
        type = 4294966784
#3  0x00000000004ca7ab in ?? ()
No symbol table info available.
#4  0x0000000000430ffc in ?? ()
No symbol table info available.
#5  0x00007f5005927ec5 in __libc_start_main (main=0x42f7a0, argc=8, argv=0x7fffdd8ca6a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffdd8ca698) at libc-start.c:287
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -7010849927254589483, 4401630, 140736910370464, 0, 0, 7010924000612892629, 6923297091889446869}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0xb43c00, 0x7fffdd8ca6a8}, data = {prev = 0x0, cleanup = 0x0, canceltype = 11811840}}}
        not_first_call = <optimized out>
#6  0x0000000000432a07 in ?? ()
No symbol table info available.
warning: target file /proc/11714/cmdline contained unexpected null characters
Saved corefile core.11714

Updated by Jaroslav Kysela almost 9 years ago

Use tvh with the debugging symbols. I cannot use this.

Updated by Bengt Madeberg almost 9 years ago

File gdb.txt gdb.txt added

It happend again. New gdb.txt. I will revert back to 589 due to 3 crashes within 3 days.

Updated by Jaroslav Kysela almost 9 years ago

Again, use the tvh binary with the debugging symbols. It seems that there's an endless loop somewhere with memory allocations.

No symbol table info available.

Updated by B C almost 9 years ago

File gdb.txt gdb.txt added

as I just run into the same error, here a log from my box
if the core dump is usefull aswell let me know

keep up the good work

Updated by Jaroslav Kysela almost 9 years ago

@B C: I'm not sure if it's same issue - this one is for IPTV HTTP input. Fixed in v4.1-1183-g637d1f3 .

Updated by B C almost 9 years ago

I didn't look at Bengts logs. Thanks for this fast fix

#10

Updated by Bengt Madeberg almost 9 years ago

Jaroslav Kysela wrote:

Again, use the tvh binary with the debugging symbols. It seems that there's an endless loop somewhere with memory allocations.

[...]

Sorry but due to lack of knowledge I don't know how to do it. I just followed the wiki/debugging info "dead or live lock"

#11

Updated by Jaroslav Kysela almost 9 years ago

The binary in ./build.linux should have all symbols in...

#12

Updated by Bengt Madeberg almost 9 years ago

File gdb.txt gdb.txt added

Another lock. Now running the ./build.linux version.

#13

Updated by Bengt Madeberg almost 9 years ago

File gdb.txt gdb.txt added

Once again.

#14

Updated by Bengt Madeberg almost 9 years ago

It also seems that all EPG data was gone after the last crash/lock.

Dec 29 00:31:40 server1a tvheadend[11519]: htsp: 192.168.0.65: Welcomed client software: Kodi Media Center (HTSPv24)
Dec 29 00:31:40 server1a tvheadend[11519]: htsp: 192.168.0.65 [ Kodi Media Center ]: Identified as user 'muy'
Dec 29 00:31:40 server1a tvheadend[11519]: htsp: 192.168.0.65 [ muy | Kodi Media Center ]: Privileges updated
Dec 29 00:31:41 server1a tvheadend[11519]: mpegts: 570MHz in Teracom_sat_ip - tuning on SAT>IP DVB-T Tuner #1 (192.168.0.93:9983)
Dec 29 00:31:41 server1a tvheadend[11519]: subscription: 0001: "DVR: Tinker, tailor, soldier, spy" subscribing on channel "TV4 HD Göteborg", weight: 300, adapter: "SAT>IP DVB-T Tuner #1 (192.168.0.93:9983)", network: "Teracom_sat_ip", mux: "570MHz", provider: "TV4 AB", service: "TV4 HD Göteborg", profile="matroska" 
Dec 29 00:31:45 server1a tvheadend[11519]: dvr: /disk1/hts/Tinker,-tailor,-soldier,-spy/Tinker,-tailor,-soldier,-spy-2.mkv from adapter: "SAT>IP DVB-T Tuner #1 (192.168.0.93:9983)", network: "Teracom_sat_ip", mux: "570MHz", provider: "TV4 AB", service: "TV4 HD Göteborg" 
Dec 29 00:31:45 server1a tvheadend[11519]: dvr:  #  type              lang  resolution  aspect ratio  sample rate  channels
Dec 29 00:31:45 server1a tvheadend[11519]: dvr:  1  H264                    1280x720    16:9
Dec 29 00:31:45 server1a tvheadend[11519]: dvr:  2  AAC               swe                             24000        5.1
Dec 29 00:31:56 server1a tvheadend[11519]: subscription: 0001: "DVR: Tinker, tailor, soldier, spy" unsubscribing from "TV4 HD Göteborg" 
Dec 29 00:31:56 server1a tvheadend[11519]: dvr: "Tinker, tailor, soldier, spy" on "TV4 HD Göteborg": End of program: Aborted by user
Dec 29 00:31:57 server1a tvheadend[11519]: mpegts: 570MHz in Teracom_sat_ip - tuning on SAT>IP DVB-T Tuner #1 (192.168.0.93:9983)
Dec 29 00:31:57 server1a tvheadend[11519]: subscription: 000C: "epggrab" subscribing to mux "570MHz", weight: 4, adapter: "SAT>IP DVB-T Tuner #1 (192.168.0.93:9983)", network: "Teracom_sat_ip", service: "Raw PID Subscription" 
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: Signal: 11 in PRG: tvheadend (4.1-1221~g66e5d27) [16a2c4515f54bd836eefac67e87eb05993b98762] CWD: /
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: Fault address 0x1 (Address not mapped)
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: Loaded libraries: /usr/lib/libdvben50221.so /usr/lib/libdvbapi.so /usr/lib/libucsi.so /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/liburiparser.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libnss_compat.so.2 /lib/x86_64-linux-gnu/libnsl.so.1 /lib/x86_64-linux-gnu/libnss_nis.so.2 /lib/x86_64-linux-gnu/libnss_files.so.2
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: Register dump [23]: 00000000000000000000000000000006fffffffffffff786000000000000020000000000000000010000000000b5058900000000030f447000000000030f3d800000000000000001000000000000000100007fa8157f92f000000000030f447000007fa804000f0000000000030f3d80000000000000000000007fa8157f91c000000000004744540000000000010202ffff0000000000330000000000000004000000000000000efffffffe7ffbba130000000000000001
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: STACKTRACE
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/trap.c:148 0x46cd69
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: ??:0 0x7fa89f5d3340
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/lang_codes.c:641 0x474454
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/lang_codes.c:612 0x474559
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/lang_str.c:84 0x4752c2
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/lang_str.c:106 0x475369
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/dvr/dvr_db.c:2478 (discriminator 3) 0x4a1dc0
Dec 29 00:32:03 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/idnode.c:373 0x435bab
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/idnode.c:714 (discriminator 3) 0x43629e
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: ??:0 0x7fa89ed2b4c9
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: ??:0 0x7fa89ed2b238
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: ??:0 0x7fa89ed2b222
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: ??:0 0x7fa89ed2b222
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: qsort_r+0x28c  (/lib/x86_64-linux-gnu/libc.so.6)
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/api/api_idnode.c:140 0x48c714
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/webui/webui_api.c:43 0x4b8721
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/http.c:618 0x4452a0
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/http.c:754 0x445ed1
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/http.c:863 0x4457a3
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/http.c:1151 0x4461bd
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/http.c:1194 0x44627d
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/tcp.c:642 0x4402bd
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: /home/muy/tvh_20151213/tvheadend/src/wrappers.c:177 0x43c4de
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: ??:0 0x7fa89f5cb182
Dec 29 00:32:04 server1a tvheadend[11519]: CRASH: clone+0x6d  (/lib/x86_64-linux-gnu/libc.so.6)
Dec 29 00:32:04 server1a kernel: [6396641.825499] tvh:tcp-start[11585]: segfault at 1 ip 0000000000474454 sp 00007fa8157f91c0 error 4 in tvheadend[400000+9ba000]
Dec 29 00:32:05 server1a kernel: [6396643.442025] init: tvheadend main process (11519) killed by SEGV signal
Dec 29 00:32:05 server1a kernel: [6396643.442050] init: tvheadend main process ended, respawning
Dec 29 00:32:05 server1a tvheadend[11615]: main: Log started
Dec 29 00:32:05 server1a tvheadend[11615]: http: Starting HTTP server 0.0.0.0:9981
Dec 29 00:32:05 server1a tvheadend[11615]: htsp: Starting HTSP server 0.0.0.0:9982
Dec 29 00:32:05 server1a tvheadend[11615]: config: loaded
Dec 29 00:32:06 server1a tvheadend[11615]: scanfile: DVB-T - loaded 44 regions with 1104 networks
Dec 29 00:32:06 server1a tvheadend[11615]: scanfile: DVB-S - loaded 1 regions with 92 networks
Dec 29 00:32:06 server1a tvheadend[11615]: scanfile: DVB-C - loaded 17 regions with 53 networks
Dec 29 00:32:06 server1a tvheadend[11615]: scanfile: ATSC-T - loaded 2 regions with 9 networks
Dec 29 00:32:06 server1a tvheadend[11615]: scanfile: ATSC-C - loaded 1 regions with 5 networks

#15

Updated by Jaroslav Kysela almost 9 years ago

Note to me: It looks like a deadlock in src/parsers/parsers.c:320 (AAC ADTS parser). The 'muxlen = 0', so 'p' index is not updated resulting in the endless loop.

#16

Updated by Jaroslav Kysela almost 9 years ago

Status changed from New to Fixed
% Done changed from 0 to 100

Applied in changeset commit:tvheadend|b480c92c0e0c78af3136ba6483b22c7e8b67d685.

#17

Updated by Jaroslav Kysela almost 9 years ago

Fixed in v4.1-1258-gb480c92 .

#18

Updated by Jaroslav Kysela almost 9 years ago

Note it seems related to #3383 .

Also available in: Atom PDF

Project

General

Profile

Tvheadend

Issues

Custom queries

Bug #3381

Major crash !?

History

Updated by Mario D almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by B C almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by B C almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Bengt Madeberg almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago

Updated by Jaroslav Kysela almost 9 years ago