Tvheadend

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Incorrect-not-useable-crash-reports

Jaroslav Kysela wrote:

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Incorrect-not-useable-crash-reports

Sorry! I'll prepare a correct one.
However, please check it in your environment as RTP-over-TCP always fails with the current staging.

Mono Polimorph wrote:

Hi,

I open a new issue, as the previous are very large... #4226 & #4517.

http://www.tvheadend.org/projects/tvheadend/repository/revisions/b829ab46c923f6487dee636b4d4cd1ef61c94826
After this commit the RTP-over-TCP crashes every time. You start the TVH and at the first SETUP request the server crashes. See the log:

[...]

I suggest to rollback the commit rb829ab46.

Sorry! The commit that fails is rebbbc3b1:
http://www.tvheadend.org/projects/tvheadend/repository/revisions/ebbbc3b1e9c954a19c4aa518d83ccd28d67e6d23

Hi Jaros,

You tested the current snapshot with RTP-over-TCP enabled between two TVH instances?
I don't think you did. In all test I do, all the time it crashes!
Any suggestion?

I also confirm that with tvheadend_4.3-448~g2f07ea0_armhf.deb my tvheadend SAT-IP server (192.168.1.24) crashes and cannot be reached from the TVH SAT-IP client (previous versions continue to respond):

2017-09-08 11:10:36.593 mpegts: 506MHz in DVB-T Italy - tuning on SAT>IP DVB-T Tuner #2 (192.168.1.24:9983)
2017-09-08 11:10:36.593 subscription: 6EF0: "scan" subscribing to mux "506MHz", weight: 6, adapter: "SAT>IP DVB-T Tuner #2 (192.168.1.24:9983)", network: "DVB-T Italy", service: "Raw PID Subscription" 
2017-09-08 11:10:36.594 mpegts: 722MHz in DVB-T Italy scan complete
2017-09-08 11:10:36.594 subscription: 6EE4: "scan" unsubscribing
2017-09-08 11:10:36.594 mpegts: 746MHz in DVB-T Italy - tuning on SAT>IP DVB-T Tuner #4 (192.168.1.22:9983)
2017-09-08 11:10:36.595 subscription: 6EF2: "scan" subscribing to mux "746MHz", weight: 6, adapter: "SAT>IP DVB-T Tuner #4 (192.168.1.22:9983)", network: "DVB-T Italy", service: "Raw PID Subscription" 
2017-09-08 11:10:36.596 satip: SAT>IP DVB-T Tuner #2 (192.168.1.24:9983) - RTSP error -111 (Connection refused) [6-0]
2017-09-08 11:10:41.592 mpegts: 506MHz in DVB-T Italy - scan no data, failed
2017-09-08 11:10:41.592 subscription: 6EF0: "scan" unsubscribing

Jaroslav Kysela wrote:

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Incorrect-not-useable-crash-reports

Hi Jaros,

Impossible to generate any other report that the one printed in the first post.
However, I do this: start tvhe, attach gdb to it and "c", generate the crash (easy, any RTSP-over-TCP call raises a crash), then inside the gdb "thread apply all where"

Here the stack-trace of the thread that raises the crash:

Thread 5 (Thread 0x7fce54d4e700 (LWP 16826)):
#0  0x00007fce57bf3165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fce57bf63e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fce57c2d39b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fce57c36be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007fce57c3b98c in free () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007fce5947fa43 in htsbuf_data_free (hq=<optimized out>, hd=0x7fce5aedaa60) at src/htsbuf.c:72
#6  0x00007fce59450db5 in http_extra_flush (hc=0x7fce54549960) at src/http.c:820
#7  0x00007fce594515de in http_extra_send_prealloc (hc=<optimized out>, data=data@entry=0x7fce54d4cf80, data_len=data_len@entry=68) at src/http.c:900
#8  0x00007fce594e3822 in satip_rtp_tcp_data (stream=stream@entry=1 '\001', data=data@entry=0x7fce54d4cf80 "$\001", data_len=data_len@entry=68,
    rtp=<error reading variable: Unhandled dwarf expression opcode 0xfa>) at src/satip/rtp.c:277
#9  0x00007fce594e53ea in satip_rtcp_thread (aux=<optimized out>) at src/satip/rtp.c:936
#10 0x00007fce59446bb3 in thread_wrapper (p=0x7fce5af0f8b0) at src/wrappers.c:161
#11 0x00007fce583dcb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007fce57c9c95d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()

Please, remember that any use of the RTP-over-TCP generates a crash!

Hi,

The crash seems to be created by this line of code:
http://www.tvheadend.org/projects/tvheadend/repository/revisions/ebbbc3b1e9c954a19c4aa518d83ccd28d67e6d23/entry/src/htsbuf.c#L70

And the change was introduced in this patch:
http://www.tvheadend.org/projects/tvheadend/repository/revisions/ebbbc3b1e9c954a19c4aa518d83ccd28d67e6d23/diff/src/htsbuf.c

But this patch has completely broken the RTP-over-TCP mode!

Try v4.3-453-g3391e1d99 ...

tvheadend_4.3-453~g3391e1d_armhf.deb still crashes.
On TVH SAT>IP client I just run a "force scan" on one of the networks and the server crashes immediately.

tvheadend_4.3-460~g3e2ac87_armhf.deb still crashes.

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Correct-crash-reports

Jaroslav Kysela wrote:

Try v4.3-453-g3391e1d99 ...

Hi,

Indentical behaviour with this release!

Thread 9 (Thread 0x7ff726059700 (LWP 18171)):
#0  0x00007ff728efe165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ff728f013e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ff728f3839b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ff728f41be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007ff728f4698c in free () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007ff72a78a9e3 in htsbuf_data_free (hq=<optimized out>, hd=0x7ff72cb74150) at src/htsbuf.c:72
#6  0x00007ff72a75beb3 in http_extra_flush (hc=0x7ff725050960) at src/http.c:826
#7  0x00007ff72a75c6ce in http_extra_send_prealloc (hc=<optimized out>, data=data@entry=0x7ff726057f80, data_len=data_len@entry=88)
    at src/http.c:904
#8  0x00007ff72a7ee302 in satip_rtp_tcp_data (stream=stream@entry=1 '\001', data=data@entry=0x7ff726057f80 "$\001",
    data_len=data_len@entry=88, rtp=<error reading variable: Unhandled dwarf expression opcode 0xfa>) at src/satip/rtp.c:277
#9  0x00007ff72a7efece in satip_rtcp_thread (aux=<optimized out>) at src/satip/rtp.c:935
#10 0x00007ff72a751c93 in thread_wrapper (p=0x7ff72cc1a7c0) at src/wrappers.c:161
#11 0x00007ff7296e7b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007ff728fa795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()

You see that the crash is caused by this line of code?
http://www.tvheadend.org/projects/tvheadend/repository/revisions/ebbbc3b1e9c954a19c4aa518d83ccd28d67e6d23/entry/src/htsbuf.c#L70

I appreciate your effort with the development of the TVH. But please take note that the RTP-over-TCP is now completly wroken. If you check the code in your environment... you see that any call using the the RTP-over-TCP crash! The crash starts when the first RTCP packet is sent, that is, when the first Interlaved packet is sent.

Thank you for your support!

I'm not very familiar with debugging.
I obtained the following info at the moment:

#0  0x75beaf70 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        _sys_result = <optimized out>
        pd = 0x70bfef60
        pid = <optimized out>
        selftid = 20926
#1  0x75bec324 in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 27 times>, 1891621600, 7, 75, 75, 1891621600}}, sa_flags = 7, sa_restorer = 0x4b}
        sigs = {__val = {32, 0 <repeats 31 times>}}
#2  0x75c26954 in __libc_message (do_abort=<optimized out>, fmt=0x75cdc6e8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {__ap = 0x70bfdc14}
        fd = 1
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x75c2cb80 in malloc_printerr (action=1, str=0x75cdc8ec "free(): invalid pointer", ptr=<optimized out>) at malloc.c:4996
        buf = "70bfdd24" 
        cp = <optimized out>
#4  0x75c2db24 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=56) at malloc.c:3840
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
        __func__ = "_int_free" 
#5  0x54c5d2d8 in htsbuf_data_free (hq=hq@entry=0x54c2d114 <http_extra_flush+400>, hd=hd@entry=0x72d004b8) at src/htsbuf.c:72
No locals.
#6  0x54c2d114 in http_extra_flush (hc=0x6bfd07a0) at src/http.c:826
        hd = 0x72d004b8
        r = <optimized out>
        serr = 0
#7  0x54c2d284 in http_extra_send_prealloc (hc=<optimized out>, data=data@entry=0x70bfdd24, data_len=data_len@entry=96) at src/http.c:904
No locals.
#8  0x54cc42f4 in satip_rtp_tcp_data (rtp=0x74b01428, data_len=96, data=0x70bfdd1c "", stream=1 '\001') at src/satip/rtp.c:277
No locals.
#9  satip_rtcp_thread (aux=<optimized out>) at src/satip/rtp.c:935
        rtp = 0x74b01428
        us = <optimized out>
        msg = "$\001\000\\\000\000\000\000SES1\000\000\000Over=1.1;tuner=2,220,0,15,754,8,dvbt,8k,qam64,14,,,,;pids=0,1,16,17,18,3002,3003", '\000' <repeats 1325 times>
        addrbuf = '\000' <repeats 49 times>
        r = <optimized out>
        len = 96
        err = <optimized out>
#10 0x54c22e6c in thread_wrapper (p=0x5c77ca90) at src/wrappers.c:161
        ts = 0x5c77ca90
        set = {__val = {16388, 0 <repeats 31 times>}}
        r = <optimized out>
#11 0x76ba6e90 in start_thread (arg=0x70bfef60) at pthread_create.c:311
        pd = 0x70bfef60
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {1891626952, 1, 2126513284, 338, 1431061564, 1963177152, 0, 1891625628, -60367753, -94151817, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#12 0x75c8e598 in ?? () at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Fixed in v4.3-471-g219dc8e5d.

I confirm that tvheadend_4.3-471~g219dc8e_armhf.deb is now working as expected (no more crashes) in my SAT>IP client/server configuration.
Thanks.

Jaroslav Kysela wrote:

Fixed in v4.3-471-g219dc8e5d.

Thank you! Now RTP-over-TCP doesn't crash. :)

However, another problem appears:

2017-09-15 16:19:03.239 satip: SAT>IP DVB-S Tuner #1 (192.168.1.22) - RTSP error -90 (Message too long) [5-0]
2017-09-15 16:19:03.721 subscription: 0009: service instance is bad, reason: Tuning failed

When the TCP socket collapses, this problem appears.
I'll open another issue.

Regards.