Tvheadend

sync && echo 3 > /proc/sys/vm/drop_caches

I run TVH on 2Gb system without problems.

Just now this gives no result. But now the problem don't exist, I don't run the OTA-EPG now.
I have to add the information: I run this constellation since about 4 years without problem. Only since the update from the QNAP-fw the problem occures. I'm not sure that the reason is only TVH. Sometime everything runes for days without promlems. Bus for example last week I started the OTA-EPG end the problem occures direct in this moment.
I made a lot of tests. Fist I thought that my virtualization system is the problem, I run some virtual machines on this QNAP. I opendes a ticket with QNAP without result. I changed my RAM. But then I see that everytime I run in this problem it is solved by restart TVH.
Another point: My QNAP has now 64 GB RAM. In the moment that TVH usage of memory increases to high values like 10 GB and more the QNAP shows me a usage of nearly 100% of the 64 GB. I'm not a specialist, so I absolutly don't understand this.

There have been several memory leaks fixed in the latest version. Some in the opentv-grabber which could be the cause for this.

The logs indicate some kind of lock, so you might want to check with gdb to see why it's stalled and not processing any more input data.

Flole Systems wrote:

There have been several memory leaks fixed in the latest version. Some in the opentv-grabber which could be the cause for this.

I could not compile the latest versions for Harald K to try. The last version that builds correctly is bbf76ca, then with c5d4d7d and onwards I always get this error:

CC              src/input/mpegts/scanfile.o
src/input/mpegts/scanfile.c: In function ‘scanfile_create_network’:
src/input/mpegts/scanfile.c:367:46: error: ‘%s’ directive output may be truncated writing up to 269 bytes into a region of size between 256 and 262 [-Werror=format-truncation=]
     snprintf(buf3, sizeof(buf3), "%c%3i.%i%c:%s", opos < 0 ? '<' : '>',
                                              ^~
src/input/mpegts/scanfile.c:369:73:
                                                    opos < 0 ? 'W' :'E', buf);
                                                                         ~~~
In file included from /usr/include/stdio.h:873,
                 from /root/tvheadend/tvheadend/src/tvheadend.h:24,
                 from src/input/mpegts/scanfile.c:19:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: ‘__builtin___snprintf_chk’ output between 9 and 284 bytes into a destination of size 270
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/input/mpegts/scanfile.c:372:36: error: ‘%s’ directive output may be truncated writing up to 269 bytes into a region of size between 256 and 262 [-Werror=format-truncation=]
   snprintf(buf2, sizeof(buf2), "%s_%s", type, buf);
                                    ^~         ~~~
In file included from /usr/include/stdio.h:873,
                 from /root/tvheadend/tvheadend/src/tvheadend.h:24,
                 from src/input/mpegts/scanfile.c:19:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: ‘__builtin___snprintf_chk’ output 2 or more bytes (assuming 277) into a destination of size 263
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CC              src/epggrab/otamux.o
CC              src/epggrab/module/eit.o
cc1: all warnings being treated as errors
make: *** [Makefile:717: /root/tvheadend/tvheadend/build.linux/src/input/mpegts/scanfile.o] Error 1
make: *** Waiting for unfinished jobs....

Flole Systems wrote:

The logs indicate some kind of lock, so you might want to check with gdb to see why it's stalled and not processing any more input data.

Harald K, maybe you should post the zipped trace log, as running gdb on the QNAP (an embedded system) is not feasible.

That compile issue will be fixed soon.

Why is running gdb on an embedded system not feasible? There is no point on posting logs, you need to figure out the cause of it. The logs are just showing the consequences, you need to find the cause. Maybe you can try to use the deadlock detector in Tvheadend (see the Wiki for details about that) but other than that there is not much which can be done without using gdb.

I must try to reproduce the problem with running debug. Then I'll post the trace log. Unfortunately I'm a stupid user, things like gdb are very strange for me.

Again: A trace log does not help. It will only show the consequences and not the cause. No need to waste time with that.

Again: A trace log does not help. It will only show the consequences and not the cause. No need to waste time with that.

OK, sorry, didn't know about that.

Why is running gdb on an embedded system not feasible?

Probably I've overstated, anyway to have TVHeadend run on the QNAP (which uses ancient glibc 2.21) I compile it on a Debian VM and then move all the required libraries (including the Linux dynamic loader ld.so) to the QNAP.

For gdb it's the same: it is not included and must be compiled from sources. Or maybe, if we're lucky, the version shipped with Entware might work, but I've never tried before.

Sounds like you're talking about a common architecture like x86 or i386, so use your preferred search engine and find a statically compiled version of gdb and you can just run it. As you only need it a few times (and have plenty of diskspace available) that is probably the easiest solution. But be aware: You need Tvheadend with debug symbols, so that's something you need to keep in mind while compiling.

The fix for the compile issue is in now, so you should be able to compile again.

So you should be able to compile again.

Yeah, confirmed working, thanks!

But be aware: You need Tvheadend with debug symbols, so that's something you need to keep in mind while compiling.

Just to be sure, is it the --enable-ccdebug option, right? The wiki doesn't mention when compiling from sources, just to use the "-dbg" packages...
Bear in mind that I'm using --enable-bundle to have a single TVHeadend binary, are the debug symbols included inside the binary itself?

@Harald K
You can download the compiled binary (4.3-1936~g0046c96d8) from here:

https://www.dropbox.com/s/0b1oaucnfkh65xb/tvheadend.zip?dl=1

Extract the file and overwrite the one you have in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/ (replace CACHEDEV1_DATA with the one correct for your NAS).

What you compiled contains the debug symbols, so it's all good.

If you look at the Wiki you can also try to use the thrdebug-Flag when starting Tvheadend. But gdb is much more promising. If you write a coredump it will write all memory used by Tvheadend to a file and you can also later on restore that state with gdb and pull more information from it if needed.

@virtual dj: I have done an now 4.3-1936~g0046c96d is running on my QNAP.

Thank you all very much.

What you compiled contains the debug symbols, so it's all good.

I've tried on my NAS, to be sure that I'm doing the things right. I'm writing the complete procedure here, so Harald may reproduce it if/when he will have the thread lock.

1. I've installed gdb from Entware (available at Qnapclub.eu) with:
   

   [/share/Public] # opkg update
   [/share/Public] # opkg install gdb
   [/share/Public] # gdb -v | head -n 1
   GNU gdb (GDB) 10.1
   

2. I've copied the libthread_db.so.1 from my VM into /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/ (where the dynamic loader libs reside) to avoid the following error:
   

   warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

   You can download it from here: https://www.dropbox.com/s/6vfadtmiyi2gk6o/libthread_db.so.zip?dl=1
3. I've created a .gdbinit file like so:
   

   [/share/Public] # echo "set auto-load safe-path /" > /root/.gdbinit
   

   ... to avoid the error:
   

   warning: File "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

4. With TVHeadend running, I run gdb like so:
   

   [/share/Public] # gdb tv $(pidof tvh-loader)
   ... cut ...
   [New LWP 20009]
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/--Type <RET> for more, q to quit, c to continue without paging--
   libthread_db.so.1".
   0x00007effefe4035b in pthread_cond_timedwait@@GLIBC_2.3.2 ()
      from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
   

At first sight it seems correct, but what doesn't convince me is that if I type:

(gdb) bt
#0  0x00007effefe4035b in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x00007efff0b1798f in ?? ()
#2  0x00007ffc15d26c60 in ?? ()
#3  0x00007ffc15d26c90 in ?? ()
#4  0x00007efff2b8fbe0 in ?? ()
#5  0x00007efff2a2ef80 in ?? ()
#6  0x00007ffc15d26c80 in ?? ()
#7  0xfffffffff0afe40f in ?? ()
#8  0x0000000000000000 in ?? ()

(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
0x00007efff08bf140  0x00007efff08c6fe0  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libdvbcsa.so.1
0x00007efff084c450  0x00007efff0896916  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libssl.so.1.1
0x00007efff05d3000  0x00007efff0769bbc  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libcrypto.so.1.1
0x00007efff03283d0  0x00007efff033ba70  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libz.so.1
0x00007efff02a3260  0x00007efff02fee63  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpcre2-8.so.0
0x00007efff02825a0  0x00007efff0294f78  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/liburiparser.so.1
0x00007efff005dab0  0x00007efff0070382  Yes         /share/CACHEDEV1_DATA/.qpkg/CodexPack/opt/cdx/lib/libva.so.2
0x00007effefe58a80  0x00007effefe58e7b  Yes         /share/CACHEDEV1_DATA/.qpkg/CodexPack/opt/cdx/lib/libva-drm.so.2
0x00007effefe54130  0x00007effefe54e75  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libdl.so.2
0x00007effefe385b0  0x00007effefe46641  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
0x00007effefcbc270  0x00007effefd5a4f2  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libm.so.6
0x00007effefca53b0  0x00007effefca849c  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/librt.so.1
0x00007effefc79240  0x00007effefc858e0  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libmvec.so.1
0x00007effefb7f4b0  0x00007effefc2704e  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libstdc++.so.6
0x00007effefadc2e0  0x00007effefaecc2d  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libgcc_s.so.1
0x00007effef93a320  0x00007effefa8039b  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
0x00007effef907570  0x00007effef90febb  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libdrm.so.2
0x00007efff2cc0090  0x00007efff2cddb20  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
0x00007effef76a220  0x00007effef76eaec  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnss_compat.so.2
0x00007effef75d2e0  0x00007effef763d73  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnss_nis.so.2
0x00007effef7466b0  0x00007effef751f01  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnsl.so.1
0x00007effef730300  0x00007effef736578  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnss_files.so.2
0x00007effed998ac0  0x00007effedee7e19  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/CodexPack/opt/cdx/lib/libx265.so
(*): Shared library is missing debugging information.

I also tried a different gdb, a static version that you suggested, such as this:
https://github.com/hugsy/gdb-static/blob/master/gdb-7.10.1-x64

but it's even worse and it still has the ??.

I am not entirely sure I fully understand what you are doing there, but you need to attach directly to Tvheadend. I have no clue what that magic loader is doing and how it works, but your result should be a proper stacktrace and not just question marks. Also is your tvheadend binary really called tv? Otherwise the gdb command to attach isn't correct.

Hmmm, I'm not really sure what I have to/can do now. Now 4.3-1936~g0046c96d8 is running some days without any problem, but also without ota-epg. I thought next step can be to try to activate ota-epg with debug trace all? Of cause I'll wait to do this for a less critial time, when there are no recordings planned and no other things running and I have the time to take care for the NAS.

Harald K wrote:

I thought next step can be to try to activate ota-epg with debug trace all? Of cause I'll wait to do this for a less critial time, when there are no recordings planned and no other things running and I have the time to take care for the NAS.

You should activate ota-epg, but as Flole Systems said, you should NOT trace but instead, when you see the RAM increasing and the [WARNING]:mpegts: too much queued table input data (over 2MB) message on the log, attach gdb to it and create a core dump file.

Flole Systems wrote:

but your result should be a proper stacktrace and not just question marks.

Mmm, I suspected that.

I am not entirely sure I fully understand what you are doing there, but you need to attach directly to Tvheadend.

I cannot run tvheadend directly on the NAS: if I do, it won't run because it has an ancient libc. So I run:

[/the/copied/libc/folder] # ./ld.so --library-path /the/copied/libc/folder /my/application/path/tvheadend --fork

Flole Systems wrote:

Also is your tvheadend binary really called tv? Otherwise the gdb command to attach isn't correct.

No, but if I run (I renamed the file to stick with the example):

[~] # gdb /my/application/path/tvheadend $(pidof ld.so)
... cut ...
[New LWP 19956]
[New LWP 19957]
[New LWP 19958]

warning: Build ID mismatch between current exec-file /my/application/path/tvheadend
and automatically determined exec-file /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
exec-file-mismatch handling is currently "ask" 
Load new symbol table from "/the/copied/libc/folder/ld.so"? (y or n)

[~] # gdb /my/application/path/tvheadend $(pidof ld.so)
... cut ...
[New LWP 19956]
[New LWP 19957]
[New LWP 19958]

warning: Build ID mismatch between current exec-file /my/application/path/tvheadend
and automatically determined exec-file /the/copied/libc/folder/ld.so
exec-file-mismatch handling is currently "ask" 
Load new symbol table from "/the/copied/libc/folder/ld.so"? (y or n)

• tvheadend binary (that one that I've compiled above) resides in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend
• the ld.so has been renamed (to avoid clashes) as /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader

Check this... It could help you, about how to use gdb.

https://tvheadend.org/issues/5295#note-10

virtual dj wrote:

You should activate ota-epg, but as Flole Systems said, you should NOT trace but instead, when you see the RAM increasing and the [WARNING]:mpegts: too much queued table input data (over 2MB) message on the log, attach gdb to it and create a core dump file.

Ok, seems I have to do this. Please be patient, I'm very stupid in this topic. "attach gdb to it and create core dump file" means first to study what that will say to me.
Exactly at the moment I started manual the ota-epg (22:49) the memory used by TVH increases from about 200 MB to about 2.2 GB, after the end of ota-epg it falls to about 1.9 GB. Parallel running recordings seems to have no real influence on the memory, but while the running of the automtic ota-epg (02:04) I get some messages for about 5 minutes like this (but no increasing use of memory)

2021-02-23 02:10:54.330 subscription: 0082: "epggrab" unsubscribing
2021-02-23 02:10:55.330 mpegts: 12480V in Astra-19.2E - tuning on Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0
2021-02-23 02:10:56.025 subscription: 0084: "epggrab" subscribing to mux "12480V", weight: 4, adapter: "Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0", network: "Astra-19.2E", service: "Raw PID Subscription" 
2021-02-23 02:11:06.582 subscription: 0084: "epggrab" unsubscribing
2021-02-23 02:11:58.621 subscription: 007C: "epggrab" unsubscribing
2021-02-23 02:20:05.049 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:15.058 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:25.043 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:20:25.066 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:35.093 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:35.270 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:20:45.119 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:45.285 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:20:55.118 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:55.476 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:05.135 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:05.683 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:15.162 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:15.705 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:25.154 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:25.887 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:35.141 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:36.103 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:45.156 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:46.149 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:55.157 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:56.324 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:05.163 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:06.464 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:15.199 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:16.555 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:25.205 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:26.749 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:35.194 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:36.863 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:45.239 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:46.961 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:55.241 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:57.173 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:05.228 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:07.287 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:15.236 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:17.428 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:25.291 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:27.543 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:35.261 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:37.710 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:45.297 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:47.752 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:55.287 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:57.927 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:05.310 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:08.133 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:15.308 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:18.159 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:25.316 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:28.386 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:35.345 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:38.521 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:42.243 subscription: 005D: "DVR: Young Sheldon" unsubscribing from "ProSieben", username="ypthk" 
2021-02-23 02:24:42.306 dvr: "Young Sheldon" on "ProSieben": End of program: Completed OK
2021-02-23 02:27:11.947 subscription: 007E: "epggrab" unsubscribing
2021-02-23 02:35:44.644 epggrab: EIT: DVB Grabber - data completion timeout for 11523.25H in Astra-19.2E
2021-02-23 02:35:44.644 subscription: 006C: "epggrab" unsubscribing

It looks like you are receiving only Astra 19.2E, right? I can receive that one aswell, I'll see if I can somehow reproduce this behaviour.

Before using GDB it's necessary to figure out why it's not working properly and only outputting questionmarks. I think this might be related to the way it is executed. @virtual dj might be able to check if it works properly on the host that built this binary. I am not sure about this but it might be possible to simply replace the path to the ld.so in the binary file (as long as the new name has the same length!!!) and that should get around this issue.

right, I'm receiving only Astra 19.2E.

Okay, new situation: now there started a recording some minutes ago, no ota-epg is running but the message

2021-02-23 13:37:25.128 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0, discarding new
2021-02-23 13:37:35.135 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0, discarding new

seems to come endless, TVH don't react, memory usage increases.
So I'll stop the ota-epg and wait for news.

I can assure you this is driving me nuts! Sorry for the long post, but I would like to prove that I'm not just sitting on my hands ;-)

Flole Systems wrote:

Before using GDB it's necessary to figure out why it's not working properly and only outputting questionmarks. I think this might be related to the way it is executed.

You're right, of course it is . I've tried to reproduce the issue with this simple file:

root@debian-x64:~# cat pid.c 
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>

int main() 
{
        pid_t pid;

        /* get the process id */
        if ((pid = getpid()) > 0) {
          printf("The process id is %d\n", pid);
        }

        printf("Type 'q' to exit.\n");
        char exit;
        do{
                exit = getchar();
        }while(exit != 'q');
        return(0);
}

root@debian-x64:~# gcc -O2 -g -o mypid pid.c

[/share/Public/gdb] # ./mypid
The process id is 7344
Type 'q' to exit.
^Z
[3]+  Stopped(SIGTSTP)        ./mypid

[/share/Public/gdb] # gdb ./mypid 7344
GNU gdb (GDB) 10.1
... cut ...
Reading symbols from ./mypid...
Attaching to program: /share/CACHEDEV1_DATA/Public/gdb/mypid, process 7344
Reading symbols from /lib/libc.so.6...
(No debugging symbols found in /lib/libc.so.6)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)

Program received signal SIGTSTP, Stopped (user).
0x00007f5965368570 in read () from /lib/libc.so.6
(gdb) bt
#0  0x00007f5965368570 in read () from /lib/libc.so.6
#1  0x00007f5965300c90 in _IO_file_underflow () from /lib/libc.so.6
#2  0x00007f5965301b7e in _IO_default_uflow () from /lib/libc.so.6
#3  0x00007f59652fd2e8 in getc () from /lib/libc.so.6
#4  0x000055c3e97870bc in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#5  main () at pid.c:18

Then I've tried to run the small program under the Linux loader (just as TVHeadend):

[/share/Public/gdb] # /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader --library-path /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc ./mypid
The process id is 12766
Type 'q' to exit.
^Z
[1]+  Stopped(SIGTSTP)        /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader --library-path /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc ./mypid

[/share/Public/gdb] # gdb ./mypid 12766
GNU gdb (GDB) 10.1
... cut ...
Reading symbols from ./mypid...
Attaching to program: /share/CACHEDEV1_DATA/Public/gdb/mypid, process 12766

warning: Build ID mismatch between current exec-file /share/CACHEDEV1_DATA/Public/gdb/mypid
and automatically determined exec-file /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
exec-file-mismatch handling is currently "ask" 
Load new symbol table from "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader"? (y or n) y
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader)
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader)

Program received signal SIGTSTP, Stopped (user).
0x00007f479c474461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
(gdb) bt
#0  0x00007f479c474461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#1  0x00007f479c406670 in _IO_file_underflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#2  0x00007f479c4077b2 in _IO_default_uflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#3  0x00007f479c54e0bc in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) 

... cut ...
warning: loading /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader Not confirmed.
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)

Program received signal SIGTTIN, Stopped (tty input).
0x00007f479c474461 in ?? ()
(gdb) bt
#0  0x00007f479c474461 in ?? ()
#1  0x00007f479c406670 in ?? ()
#2  0x00007f479c546760 in ?? ()
#3  0x00007f479c545a00 in ?? ()
#4  0x00007f479c5422a0 in ?? ()
#5  0x00007f479c54e0d0 in ?? ()
#6  0x00007ffd2b7272a8 in ?? ()
#7  0x0000000000000000 in ?? ()

The only way that I was able to do it is compiling the source like so:

root@debian-x64:~# gcc -O2 -g -o mypid2 -Wl,-rpath /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc -Wl,--dynamic-linker=/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader pid.c

[/share/Public/gdb] # ./mypid2 
The process id is 23844
Type 'q' to exit.
^Z
[1]+  Stopped(SIGTSTP)        ./mypid2

[/share/Public/gdb] # gdb ./mypid2 23844  
GNU gdb (GDB) 10.1
.. cut ...
Reading symbols from ./mypid2...
Attaching to program: /share/CACHEDEV1_DATA/Public/gdb/mypid2, process 23844
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader)

Program received signal SIGTSTP, Stopped (user).
0x00007f2b1b357461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
(gdb) bt
#0  0x00007f2b1b357461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#1  0x00007f2b1b2e9670 in _IO_file_underflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#2  0x00007f2b1b2ea7b2 in _IO_default_uflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#3  0x000055ed67e8c0bc in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#4  main () at pid.c:18

I also tried the method described here to attach the symbols afterwards:

https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging#Debugging_With_an_Alternate_Loader

but I was unable to... probably I'm doing something wrong. I used the add-symbol-file command but in my opinion the address that I must specify as argument is not correct (according to the linked article) and still results either in ?? or in wrong symbols.

I won't give up, but I need some fresh ideas... believe me that this is very new (and hard) for me to understand only by Googling and trial & error.

P.S.: Another idea that I thought is to edit the TVHeadend sources to trigger a crash (just like --abort) when the error mpegts: too much queued table input data (over 2MB) is printed and then launch TVHeadend on the QNAP using the --dump switch. Might it produce anything useful, in this way?

Thanks for your patience.

I think I've already mentioned the solution: Use a hex editor and edit the path to the ld.so ;) The new path has to be the same length, so you might have to give it a weird name but other than that it should work.

You don't necessarily need to trigger a crash, attaching should usually show you something inside Tvheadend as a stack trace.

I just ran an epg scan and used valgrind to watch out for memory leaks and I didn't see any extreme RAM usage and valgrind also didn't catch anything. Unfortunately a NAS is one of the worst platforms to run valgrind, so that is most likely not an option for you.

Just to rule out a driver issue: Can you please try to disable the Sundtek device and use only SAT>IP and see if that changes anything? Also another idea might be to use the Lock-debug feature, but don't expect too much from it: https://tvheadend.org/projects/tvheadend/wiki/Debugging#Mutex-profiling
Once it's slow to react you should see long mutex long times being shown.

It looks a little like you were running the epg grabber while a recording was running and then the issues started appearing? Maybe that is what triggers in on your machine?

I had running this configuration for years without any problem wit one single tuner direct at the QNAP (sundtek) and another one as sat->ip on a raspberry pi. I always run only the ota-epg. Some months ago (October or November) QNAP roll out a new firmware. Exactly at this time ma single sundtek was killed. So I changed it to the currend 2xdual tuner. All in all the same range time I changed the tuner, firmware and some other Apps on the QNAP (especially upgrading the virtualization station) but not TVH. Since then I have tis problem. Only same weeks later I upgraded TVH, without any effect since now. I also changed the RAM without effect.

I don't know what is the reason of all, but I see that the problem always occures togeteher with TVH. If I restart TVH everything is ok. But not all RAM is used by TVH. My VM uses about 16 GB, QTS less the 1 GB, some others also some GB and all in all about 24 GB and not more of 64 GB. But if the problem starts and TVH uses 10-20 GB the system complete uses 64 GB. I knoew about the cache and such things, that's not the problem.

This issue appers independent of recordings. I can run more then 5 recordings same time without problem.

Now I run TVH with only the SAT-IP-tuner activated and runnung ota-epg. And also now the usage of RAM of TVH increases up to about 1 GB in this time and stays there. Second start with the same conditions yields nothing strange.

OK, I succeeded it eventually, I had to edit the interpreter as you said (no luck with gdb's add-symbol-file).

@Harald K

Remember that this will change your TVHeadend directory to allow debugging, so when we'll finish you'd better remove this stuff and rollback by reinstalling the original QPKG. That's another story.

1. Install Entware QPKG from the Qnapclub.eu repository to have gdb (I tried with a static build but it complains for libthread_db, while Entware's doesn't).
2. Download this file and copy it into /share/Public:
3. Stop TVHeadend and expand the contents on a new folder (remember that gdb will write core files, which are very big, here):
   

   [/share/Public] # tvheadend status
   TVHeadend status: STOPPED.
   [/share/Public] # tar zxf package.tar.gz -C ./package/
   [/share/Public] # cd package
   [/share/Public/package] #
   

4. Install gdb using Entware:
   

   [/share/Public/package] # opkg update
   [/share/Public/package] # opkg install nano patch gdb
   [/share/Public/package] # echo "set auto-load safe-path /" > ~/.gdbinit
   [/share/Public/package] # echo "set pagination off" >> ~/.gdbinit
   

5. Copy the required files to the TVHeadend installation directory, and patch the script (pay a lot of attention to the correct PATH if it's different from yours):
   

   [/share/Public/package] # cp ./tvheadend /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/
   [/share/Public/package] # cp ./libthread_db.so.1 /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/
   [/share/Public/package] # patch /share/CACHEDEV1_DATA/.qpkg/TVHeadend/TVHeadend.sh ./script.patch           
   patching file /share/CACHEDEV1_DATA/.qpkg/TVHeadend/TVHeadend.sh
   

6. Start TVHeadend, check that there are no issues (I've tried in my system and it runs):
   

   [/share/Public/package] # tvheadend start
   ... cut ...
   TVHeadend started.
   

7. Now, as a first test, try to attach gdb to the running TVHeadend:
   

   [/share/Public/package] # ./gdb /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend $(pidof tvheadend)
   Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend...
   Attaching to program: /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend, process 10238
   [New LWP 10239]
   .. cut ..
   [New LWP 10331]
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
   0x00007fa88523135b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
   (gdb) bt
   #0  0x00007fa88523135b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
   #1  0x000055852973e98f in tvh_cond_timedwait_ts (cond=0x55852b655f80 <gtimer_cond>, mutex=0x55852b7b6be0 <gtimer_lock>, ts=0x7ffc5c20bbe0) at src/tvh_thread.c:425
   #2  0x000055852972712e in mainloop () at src/main.c:775
   #3  0x000055852972a5fa in main (argc=8, argv=0x7ffc5c20cab8) at src/main.c:1364
   (gdb)
   

   The question marks finally disappeared! :-)
8. If this is working like me, detach the debugger by quitting (q and then y):
   

   (gdb) q
   A debugging session is active.
   
           Inferior 1 [process 10238] will be detached.
   
   Quit anyway? (y or n) y
   Detaching from program: /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend, process 10238
   [Inferior 1 (process 10238) detached]
   

9. Let TVHeadend run and wait until the problem arises. When it does, attach gdb again like on the previous point (pay attention to be in the /share/Public/package folder because the files will be created here) and then issue this commands:
   

   (gdb) set logging on
   Copying output to gdb.txt.
   Copying debug output to gdb.txt.
   (gdb) thread apply all bt full
   ... cut ...
   (gdb) generate-core-file
   warning: target file /proc/10238/cmdline contained unexpected null characters
   Saved corefile core.10238
   (gdb) q
   

10. The gdb.txt file and the core.10238 file (the latter is huge) are those Flole System needs (please correct me if I'm wrong). Zipped, of course.

Please don't upload or send me the core file, it contains a snapshot of the entire Tvheadend memory (including the credentials of the users if they were in memory). In addition to that it's useless without the binary and the ability to execute it, so I couldn't even look at it. Once you have a core file you can do the debugging and see what was going on at that point. It's really not difficult, but let's wait for the issue to appear again :)

Now I reached step 8, but step 7 yields not the right thing. It seems something is not correct installed up to here.
"./gdb" I have replaced by "gdb".
(Btw: What is the correct form to write the following code here?)

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend 5349
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend...
Attaching to program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 5349
[New LWP 5350]
...cut...
[New LWP 7708]

warning: File "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
    add-auto-load-safe-path /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path" 

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
0x00007fb058a5935b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) q
A debugging session is active.

    Inferior 1 [process 5349] will be detached.

Quit anyway? (y or n) y
Detaching from program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 5349
[Inferior 1 (process 5349) detached]
[/share/Public/package] #

After adding

To enable execution of this file add
    add-auto-load-safe-path /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1
line to your configuration file "/root/.gdbinit".

like written in the warning I get at the end

...cut...
[New LWP 7708]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
0x00007fb058a5935b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007fb058a5935b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x0000565139f7e98f in tvh_cond_timedwait_ts (cond=0x56513be95f80 <gtimer_cond>, mutex=0x56513bff6be0 <gtimer_lock>, ts=0x7ffdf0ed1680) at src/tvh_thread.c:425
#2  0x0000565139f6712e in mainloop () at src/main.c:775
#3  0x0000565139f6a5fa in main (argc=8, argv=0x7ffdf0ed2558) at src/main.c:1364
(gdb)

And now waiting for the issue.

One stupid question - just now comes to my sense: I have activated ota-epg on all my 5 tuners. Does this make sense?

Harald K wrote:

"./gdb" I have replaced by "gdb".

Yeah, sorry, it was my fault due to copy and paste. The correct command (as you discovered) is:

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend $(pidof tvheadend)

Harald K wrote:

After adding
[...]

That's because you probably missed these two:

[/share/Public/package] # echo "set auto-load safe-path /" > ~/.gdbinit
[/share/Public/package] # echo "set pagination off" >> ~/.gdbinit

[/share/Public/package] # cat ~/.gdbinit
set auto-load safe-path /
set pagination off
[/share/Public/package] #

Harald K wrote:

And now waiting for the issue.

When it appears, capture the log and core file but as Flole has written, take them apart without uploading and wait for his instructions.

Harald K wrote:

One stupid question - just now comes to my sense: I have activated ota-epg on all my 5 tuners. Does this make sense?

I think it's correct, you should do the same that you did previously. The scope of gdb is to inspect what component causes the lock.

You are right, my failure. In the second echo I use the ">" instead of ">>".

The reason of my "stupid question". Since I use the two dual tuner my intension was to use only the 5th tuner (sat->ip) for the ota-epg, otherwise it is very boring and only very seldom is used. So last days I do so because the idea of Flole and then the use of RAM is high, about 1 GB, but don't increase up to now. I thought it can be a reason that all tuners do the same while ota-epg.

But of cause, you are right, I have to run now the same situation like before to find the real reason. I hope I have a look on my system in the right moment - some time I have to earn some money....

@virtual dj

Just now I saw that in my tvheadend tme internal xmltv is vanished since this last steps - is that ok?

Harald K wrote:

Just now I saw that in my tvheadend tme internal xmltv is vanished since this last steps - is that ok?

Yes, I think it's due to the different loader (used to allow debug with gdb, as written above) that interfere with XMLTV, because I have the messages in the log:

2021-02-26 15:04:11.862 [   INFO]:spawn: Executing "/usr/bin/tv_grab_zz_sdjson" 
2021-02-26 15:04:11.879 [   INFO]:spawn: Executing "/usr/bin/tv_grab_uk_tvguide" 
2021-02-26 15:04:11.895 [   INFO]:spawn: Executing "/usr/bin/tv_grab_uk_bleb" 
2021-02-26 15:04:11.911 [   INFO]:spawn: Executing "/usr/bin/tv_grab_tr" 
2021-02-26 15:04:11.926 [   INFO]:spawn: Executing "/usr/bin/tv_grab_script" 
2021-02-26 15:04:11.942 [   INFO]:spawn: Executing "/usr/bin/tv_grab_na_tvmedia" 
2021-02-26 15:04:11.958 [   INFO]:spawn: Executing "/usr/bin/tv_grab_na_dtv" 
2021-02-26 15:04:11.974 [   INFO]:spawn: Executing "/usr/bin/tv_grab_na_dd" 
2021-02-26 15:04:11.989 [   INFO]:spawn: Executing "/usr/bin/tv_grab_it" 
2021-02-26 15:04:12.006 [   INFO]:spawn: Executing "/usr/bin/tv_grab_is" 
2021-02-26 15:04:12.020 [   INFO]:spawn: Executing "/usr/bin/tv_grab_il" 
2021-02-26 15:04:12.036 [   INFO]:spawn: Executing "/usr/bin/tv_grab_huro" 
2021-02-26 15:04:12.051 [   INFO]:spawn: Executing "/usr/bin/tv_grab_fr" 
2021-02-26 15:04:12.067 [   INFO]:spawn: Executing "/usr/bin/tv_grab_fi_sv" 
2021-02-26 15:04:12.082 [   INFO]:spawn: Executing "/usr/bin/tv_grab_file" 
2021-02-26 15:04:12.098 [   INFO]:spawn: Executing "/usr/bin/tv_grab_fi" 
2021-02-26 15:04:12.113 [   INFO]:spawn: Executing "/usr/bin/tv_grab_eu_xmltvse" 
2021-02-26 15:04:12.129 [   INFO]:spawn: Executing "/usr/bin/tv_grab_eu_epgdata" 
2021-02-26 15:04:12.145 [   INFO]:spawn: Executing "/usr/bin/tv_grab_dk_dr" 
2021-02-26 15:04:12.161 [   INFO]:spawn: Executing "/usr/bin/tv_grab_combiner" 
2021-02-26 15:04:12.177 [   INFO]:spawn: Executing "/usr/bin/tv_grab_ch_search" 
2021-02-26 15:04:12.194 [   INFO]:spawn: Executing "/usr/bin/tv_grab_ar" 

When TVHeadend spawns those scripts with --description probably they fail (due to the different libc) and so they don't return any data and are thus excluded from the candidates.
Until you're using this "debug version" of TVHeadend you won't have access to the XMLTV automatically.

However, in the mean time, you can invoke your grabber manually (from SSH) and post the data to TVHeadend.
Enable the External: XMLTV grabber module in TVHeadend and look at the sock file; you can use it like this (adapth path and XMLTV grabber):

# opkg install socat
# tv_grab_?? --days 7 | socat - UNIX-CONNECT:/share/CACHEDEV1_DATA/.qpkg/TVHeadend/config/epggrab/xmltv.sock

Yes, it's the same here: No XMLTV grappers listed in EPG Grabber modules. I'll try the External XMLTV. But now the main point of view is the ota-epg.

Is it right, I start the gdb only when the problem occures. Is it not to late?
This moment everything runs fine, no problem with ota-epg, recording and nothing. Strange to hope now the problem occures soon.

Harald K wrote:

Is it right, I start the gdb only when the problem occures. Is it not to late?

No, it is not. If I've understood well, when you see the [WARNING]:mpegts: too much queued table input data... message on the TVHeadend log it means it is (actually one of its thread I believe) locked into something and it doesn't have "time" to process the input data (= too much queue).

So when you see these message, you attach with gdb and create a log file of the current stack (with thread apply all bt full) and a core file (with generate-core-file). These two files are a snapshot of the state of TVHeadend, including its memory.

Then it should be possible to load this core file in gdb and inspect where the program is "stuck" (the stack should reveal the order of the operations that led to that situation). I don't know how to do this, but Flole Systems will certainly help.

This moment everything runs fine, no problem with ota-epg, recording and nothing. Strange to hope now the problem occures soon.

After all this work, it's better for it to occur!!! ;-)

Now I have the files gdb.txt and core.18346 (which is about 15 GB) - what to do now?
Something is different against the time before: The total used RAM of the QNAP didn't increase up to the limit this time.

You can publish the gdb.txt file, that shouldn't contain any personal data. Make a search for your username and password in it though to be on the safe side.

Just to confirm: The Webinterface was completely unresponsive and Tvheadend was completely locked up?

Now you can go through the steps described here. Your Thread IDs will obviously be different, so you need to see where they are coming from (usually from the previous line) and what your IDs are.

Pablo R. wrote:

Check this... It could help you, about how to use gdb.

https://tvheadend.org/issues/5295#note-10

As you have a coredump you can use gdb to load that coredump (with the -c parameter instead of specifying the PID) and then you can pull the information from Tvheadend while it was locked up.

Excuse me, I have to add: It seems that free RAM by stopping the VMs helped at the moment: Nor I have access to TVH without restart it. Two running recordings this time seem to be continued with only some failures (one recording 13 failures and the other 14 failures).

Okay , so I'll try to go through the core file. Give me some time - I repead: I'm very stupid in this topic. Thank you very much for your patience. I'll write here if I get some information - ore questions.

I'm afraid something is not yet going as intended. What am I doing wrong?

[/share/Public/package] # gdb --core=core.18346
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".

warning: Can't open file /SYSV008a6d31 (deleted) during file-backed mapping note processing

warning: Can't open file /SYSV008a6d33 (deleted) during file-backed mapping note processing
[New LWP 18346]
....cut....
[New LWP 2456]
Core was generated by `/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend'.
#0  0x00007f33c6075495 in ?? ()
[Current thread is 1 (LWP 18346)]
(gdb) print global_lock
No symbol table is loaded.  Use the "file" command.
(gdb) file /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend
Reading symbols from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend...
(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 18346, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\252G\000\000\003", '\000' <repeats 26 times>, __align = 2}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}
(gdb) thread find 18346
Thread 1 has target id 'LWP 18346'
(gdb) info threads 1
  Id   Target Id         Frame 
* 1    LWP 18346         0x00007f33c6075495 in ?? ()
(gdb) thread 1
[Switching to thread 1 (LWP 18346)]
#0  0x00007f33c6075495 in ?? ()
(gdb) bt
#0  0x00007f33c6075495 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x00005650ebbcaea0 in ?? ()
#3  0x00007f33c60752a0 in ?? ()
#4  0x00007f339d7ebd28 in ?? ()
#5  0x00005650ea431fa0 in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb) 

I can confirm. I have exactly the same issue.

Mar 01 16:01:50 thinHP-ubuntu tvheadend[4110588]: mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:01:50 thinHP-ubuntu tvheadend[4110588]: 2021-03-01 16:01:50.446 [WARNING] mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:00 thinHP-ubuntu tvheadend[4110588]: mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:00 thinHP-ubuntu tvheadend[4110588]: 2021-03-01 16:02:00.978 [WARNING] mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:11 thinHP-ubuntu tvheadend[4110588]: mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:11 thinHP-ubuntu tvheadend[4110588]: 2021-03-01 16:02:11.481 [WARNING] mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:21 thinHP-ubuntu tvheadend[4110588]: mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:21 thinHP-ubuntu tvheadend[4110588]: 2021-03-01 16:02:21.897 [WARNING] mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:32 thinHP-ubuntu tvheadend[4110588]: mpegts: too much queued table input data (over 2MB), discarding new
Mar 01 16:02:32 thinHP-ubuntu tvheadend[4110588]: 2021-03-01 16:02:32.429 [WARNING] mpegts: too much queued table input data (over 2MB), discarding new

What version on what system? Looks like it's not a NAS, so you can most likely do better debugging and even use valgrind to see if there is indeed a memory leak.

Harald K wrote:

Excuse me, I have to add: It seems that free RAM by stopping the VMs helped at the moment: Nor I have access to TVH without restart it. Two running recordings this time seem to be continued with only some failures (one recording 13 failures and the other 14 failures).

That sounds like Tvheadend is simply out of memory and becomes unresponsive because of that. In that case there's no real point in continuing with the gdb analysis, it is necessary to figure out why the memory usage is so high.

Flole Systems wrote:

What version on what system? Looks like it's not a NAS, so you can most likely do better debugging and even use valgrind to see if there is indeed a memory leak.

No it is thin client:
OS:
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Linux thinHP-ubuntu 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build:
tvheadend: version 4.2.9~pre+201911151752-0~built202011131255~git5bdcfd8ac~ubuntu20.04.1

I happens randomly, I can't predict when will it happen again.

Szymon Życińśki wrote:

Flole Systems wrote:

What version on what system? Looks like it's not a NAS, so you can most likely do better debugging and even use valgrind to see if there is indeed a memory leak.

No it is thin client:
OS:
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Linux thinHP-ubuntu 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build:
tvheadend: version 4.2.9~pre+201911151752-0~built202011131255~git5bdcfd8ac~ubuntu20.04.1

I happens randomly, I can't predict when will it happen again.

this package is not build by official source
build you package
https://tvheadend.org/boards/4/topics/24116

Exactly, and it's super old.

Harald K wrote:

I'm afraid something is not yet going as intended. What am I doing wrong?

I've tried to reproduce loading a core file in gdb (even though I don't have your issue so all is going well in my case) and I get the ??? in the stack trace, too .

So probably it's better to attach gdb during the issue, without relying into the core file? I mean, when you have the error, invoking gdb as we did before (using the PID) and then typing print global_lock and so on...

Ok, I'll try to force this state again and then use this way.

Compiled and setup as service. I will monitor if this new build will solve issues.

Now there is not more RAM used than the last days but TVH don't react. So I'm doing again:

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend 18346
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend...
Attaching to program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 18346
[New LWP 18347]
... cut ...
[New LWP 9172]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x00005650e4e04c1c in dvr_rec_unsubscribe (de=0x7f3380173570) at src/dvr/dvr_rec.c:198
#2  0x00005650e4dffd44 in dvr_stop_recording (de=0x7f3380173570, stopcode=0, saveconf=1, clone=0) at src/dvr/dvr_db.c:2868
#3  0x00005650e4e0006d in dvr_timer_stop_recording (aux=0x7f3380173570) at src/dvr/dvr_db.c:2924
#4  0x00005650e4d300d1 in mainloop () at src/main.c:767
#5  0x00005650e4d335fa in main (argc=8, argv=0x7fffc74af498) at src/main.c:1364
(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 18346, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\252G\000\000\003", '\000' <repeats 26 times>, __align = 2}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}
(gdb) tread find 18346
Undefined command: "tread".  Try "help".
(gdb) thread find 18346
Thread 1 has target id 'Thread 0x7f33c5b38b40 (LWP 18346)'
(gdb) info threads 1
  Id   Target Id                                     Frame 
* 1    Thread 0x7f33c5b38b40 (LWP 18346) "tvheadend" 0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f33c5b38b40 (LWP 18346))]
#0  0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x00005650e4e04c1c in dvr_rec_unsubscribe (de=0x7f3380173570) at src/dvr/dvr_rec.c:198
#2  0x00005650e4dffd44 in dvr_stop_recording (de=0x7f3380173570, stopcode=0, saveconf=1, clone=0) at src/dvr/dvr_db.c:2868
#3  0x00005650e4e0006d in dvr_timer_stop_recording (aux=0x7f3380173570) at src/dvr/dvr_db.c:2924
#4  0x00005650e4d300d1 in mainloop () at src/main.c:767
#5  0x00005650e4d335fa in main (argc=8, argv=0x7fffc74af498) at src/main.c:1364
(gdb) 

I got the problem back very quickly. This time the RAM requirement increases almost immediately when restarting TVH and then OTA EPG within 6 hours continuously to about 20 GB, of course again with the above warning. Unfortunately the complete issue is now so long that I could not copy everything from the terminal. I hope the most important is included. My QNAP now also needs the complete 64 GB RAM. I need again to restart TVH

Translated with www.DeepL.com/Translator (free version)

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend 19080
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend...
Attaching to program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 19080
[New LWP 19084]
[New LWP 19139]
[New LWP 19140]
[New LWP 19141]
[New LWP 19142]
[New LWP 19143]
[New LWP 19144]
[New LWP 19164]
[New LWP 19290]
[New LWP 19291]
[New LWP 19666]
[New LWP 20231]
[New LWP 20232]
[New LWP 20496]
[New LWP 20497]
[New LWP 20645]
[New LWP 20646]
[New LWP 20647]
[New LWP 20648]
[New LWP 21123]
[New LWP 21216]
[New LWP 21217]
[New LWP 21231]
[New LWP 21601]
[New LWP 21609]
[New LWP 21610]
[New LWP 21611]
[New LWP 21612]
[New LWP 21613]
[New LWP 21614]
[New LWP 21615]
[New LWP 21616]
[New LWP 21617]
[New LWP 21618]
[New LWP 21619]
[New LWP 21620]
[New LWP 21657]
[New LWP 21658]
[New LWP 21659]
[New LWP 13965]
[New LWP 1079]
[New LWP 16084]
[New LWP 23713]
[New LWP 2100]
[New LWP 2102]
[New LWP 2317]
[New LWP 2341]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) thread apply all bt full

... cut (outside my terminal) ...

06U\000\000\004", '\000' <repeats 130 times>, "\020", '\000' <repeats 20 times>...
#3  0x0000558632bf3f09 in thread_wrapper (p=0x7fc7b8000c90) at src/tvh_thread.c:91
        ts = 0x7fc7b8000c90
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 26 (Thread 0x7fc7c16f9700 (LWP 21609) "tvh:mi-table"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558637d25748, mutex=0x558637d256c8) at src/tvh_thread.c:363
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632cf7104 in mpegts_input_table_thread (aux=0x558637d25420) at src/input/mpegts/mpegts_input.c:1798
        mtf = 0x0
        mi = 0x558637d25420
        mm = 0x0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x7fc7b8000b20) at src/tvh_thread.c:91
        ts = 0x7fc7b8000b20
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 25 (Thread 0x7fc7c18fa700 (LWP 21601) "tvh:satip-rtcp"):
#0  0x00007fc7db499123 in clock_nanosleep () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
#1  0x0000558632bf34cd in tvh_usleep (us=150000) at src/wrappers.c:158
        ts = {tv_sec = 0, tv_nsec = 31086901}
        val = 94034866352592
        r = 21894
#2  0x0000558632ce643c in satip_rtcp_thread (aux=0x0) at src/satip/rtp.c:928
        rtp = 0x0
        us = 150000
        msg = '\000' <repeats 832 times>...
        msg1 = 0x0
        addrbuf = '\000' <repeats 49 times>
        r = 0
        len = 0
        err = 0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x558638e97140) at src/tvh_thread.c:91
        ts = 0x558638e97140
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 24 (Thread 0x7fc7c31fb700 (LWP 21231) "tvh:dvr-inotify"):
#0  0x00007fc7db8bd544 in read () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7dbf4dec2 in ?? () from /share/CACHEDEV7_DATA/.qpkg/TVH_Dev_Sundtek/sundtek/opt/lib/libmediaclient.so
No symbol table info available.
#2  0x00007fc7dbf4f1e8 in read () from /share/CACHEDEV7_DATA/.qpkg/TVH_Dev_Sundtek/sundtek/opt/lib/libmediaclient.so
No symbol table info available.
#3  0x0000558632d6f66b in _dvr_inotify_thread (p=0x0) at src/dvr/dvr_inotify.c:391
        fd = 31
        i = 0
        len = 80
        buf = "\001\000\000\000\000\002\000\000\000\000\000\000@\000\000\000.ebastian Pufpaff_ Noch nicht Schicht!.ts.removing", '\000' <repeats 14 times>, " Taiwan mit Valerie Niehaus-3.ts.removing\000\000\000\000\000\000\000\001\000\000\000@\000\000\000\273\243\060\000\060\000\000\000Sebastian Pufpaff_ Noch nicht Schicht!.ts\000\000\000\000\000\000\000\001\000\000\000\200\000\000\000"...
        from = 0x0
        fromfd = 0
        cookie = 0
        ev = 0x7fc7c31fa8a0
        from_prev = '\000' <repeats 255 times>
        fromfd_prev = 0
        cookie_prev = 0
#4  0x0000558632bf3f09 in thread_wrapper (p=0x558638d81fd0) at src/tvh_thread.c:91
        ts = 0x558638d81fd0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#5  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#6  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 23 (Thread 0x7fc7c33fc700 (LWP 21217) "tvh:epgdata"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558634b0c7e0 <epggrab_data_cond>, mutex=0x558634b0c760 <epggrab_data_mutex>) at src/tvh_thread.c:363
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632c16891 in _epggrab_data_thread (aux=0x0) at src/epggrab.c:177
        mod = 0x0
        eq = 0x0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x5586378f3c30) at src/tvh_thread.c:91
        ts = 0x5586378f3c30
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 22 (Thread 0x7fc7c35fd700 (LWP 21216) "tvh:epggrabi"):
#0  0x00007fc7db8ba35b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf498f in tvh_cond_timedwait_ts (cond=0x558634b0c720 <epggrab_cond>, mutex=0x558634c76280 <epggrab_mutex>, ts=0x7fc7c35fca00) at src/tvh_thread.c:425
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632c16646 in _epggrab_internal_thread (aux=0x0) at src/epggrab.c:125
        mod = 0x0
        err = 0
        confver = 2
        ts = {tv_sec = 1614899040, tv_nsec = 0}
        t = 1614899040
#3  0x0000558632bf3f09 in thread_wrapper (p=0x5586379c3800) at src/tvh_thread.c:91
        ts = 0x5586379c3800
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 21 (Thread 0x7fc7c3dfe700 (LWP 21123) "tvh:epggrabso"):
#0  0x00007fc7db8bd667 in accept () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632c9a645 in _epggrab_socket_thread (p=0x558637e93820) at src/epggrab/module.c:606
        s = 851393850
        s1 = 29
        mod = 0x558637e93820
#2  0x0000558632bf3f09 in thread_wrapper (p=0x558637890980) at src/tvh_thread.c:91
        ts = 0x558637890980
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#3  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#4  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 20 (Thread 0x7fc7c3fff700 (LWP 20648) "tvh:svcmap"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558634b0fe00 <service_mapper_cond>, mutex=0x558634c6cda0 <global_lock>) at src/tvh_thread.c:363
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632c5de05 in service_mapper_thread (aux=0x0) at src/service_mapper.c:382
        s = 0x0
        smi = 0x0
        prch = {prch_link = {le_next = 0x0, le_prev = 0x7fc75c0792a0}, prch_linked = 1, prch_sharer = 0x0, prch_sharer_link = {le_next = 0x0, le_prev = 0x0}, prch_pro = 0x0, prch_id = 0x0, prch_ts_delta = 0, prch_flags = 0, prch_stop = 1, prch_start_pending = 0, prch_sq_used = 1, prch_sq = {sq_st = {st_link = {le_next = 0x0, le_prev = 0x0}, st_pad = 0x0, st_ops = {st_cb = 0x558632c1cc84 <streaming_queue_deliver>, st_info = 0x558632c1cd9e <streaming_queue_info>}, st_opaque = 0x7fc7c3ffe848, st_reject_filter = 0}, sq_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}, sq_cond = {cond = {__data = {{__wseq = 0, __wseq32 = {__low = 0, __high = 0}}, {__g1_start = 0, __g1_start32 = {__low = 0, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 0, __wrefs = 2, __g_signals = {0, 0}}, __size = '\000' <repeats 36 times>, "\002\000\000\000\000\000\000\000\000\000\000", __align = 0}}, sq_maxsize = 0, sq_size = 0, sq_queue = {tqh_first = 0x0, tqh_last = 0x7fc7c3ffe930}}, prch_post_share = 0x0, prch_st = 0x7fc7c3ffe848, prch_muxer = 0x0, prch_gh = 0x0, prch_tsfix = 0x0, prch_timeshift = 0x0, prch_input = {st_link = {le_next = 0x0, le_prev = 0x0}, st_pad = 0x0, st_ops = {st_cb = 0x0, st_info = 0x0}, st_opaque = 0x0, st_reject_filter = 0}, prch_share = 0x0, prch_can_share = 0x0}
        sub = 0x0
        run = 32711
        working = 0
        r = 0
        sq = 0x7fc7c3ffe848
        sm = 0x0
        err = 0x0
        timeout = 0
        timeout_other = 0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x5586378903e0) at src/tvh_thread.c:91
        ts = 0x5586378903e0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 19 (Thread 0x7fc7d8395700 (LWP 20647) "tvh:upnp"):
#0  0x00007fc7db8bd29c in __lll_lock_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7db8b6714 in pthread_mutex_lock () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#2  0x0000558632d4d88a in satip_discovery_service_received (data=0x7fc7d83909b0 "HTTP/1.1 200 OK\r\nCACHE-CONTROL: max-age=1800\r\nEXT:\r\nLOCATION: http://192.168.178.55:9981/satip_server/desc.xml\r\nSERVER: unix/1.0 UPnP/1.1 TVHeadend/4.2.4-dmo1~bpo9+1~rpt1\r\nST: urn:ses-com:device:SatIP"..., len=345, conn=0x7fc7bc000dd0, storage=0x7fc7d8390930) at src/input/mpegts/satip/satip.c:1228
        buf = 0x7fc7d8390530 "HTTP/1.1" 
        ptr = 0x0
        saveptr = 0x7fc7d8390689 "" 
        argv = {0x7fc7d8390671 "CONFIGID.UPNP.ORG", 0x7fc7d8390684 "0", 0x7fc7d8390632 "urn", 0x7fc7d8390636 "ses-com", 0x7fc7d839063e "device", 0x7fc7d8390645 "SatIPServer", 0x7fc7d8390651 "1", 0x7fffd37b135e "", 0x7fffd37b135f "", 0x7fc7db48b803 <epoll_wait+99> "\213D$\fH\203\304\030[]A\\A]\303f\017\037D"}
        st = 0x7fc7d83905e0 "urn:ses-com:device:SatIPServer:1" 
        location = 0x7fc7d839056e "http://192.168.178.55:9981/satip_server/desc.xml" 
        server = 0x7fc7d83905a8 "unix/1.0 UPnP/1.1 TVHeadend/4.2.4-dmo1~bpo9+1~rpt1" 
        uuid = 0x7fc7d839060c "b209420f-8762-2564-2239-b0099212f483" 
        bootid = 0x7fc7d8390665 "1612332636" 
        configid = 0x7fc7d8390684 "0" 
        deviceid = 0x0
        sockbuf = "0.0.0.0\000 UPnP/1.1 tvheadend/4.3-1936~g0046c96d8\000\n\r\n\000\n\000\000\000=\273\315\062\206U\000\000\060\t9\330\307\177\000\000\320\r\000\274\307\177\000\000Y\001\000\000\000\000\000\000\260\t9\330\307\177\000\000^\023{\323\377\177\000\000\026ڋ\333\307\177\000\000\000W9\330\307\177\000\000V\001\000\000\000\000\000" 
        d = 0x7fc7bc001130
        n = 7
        i = 1
#3  0x0000558632cda23b in upnp_thread (aux=0x0) at src/upnp.c:168
        bindaddr = 0x0
        poll = 0x7fc7bc000b20
        ev = {{fd = -1, events = 1, ptr = 0x7fc7bc000dd0}, {fd = -1, events = 1, ptr = 0x7fc7bc000dd0}}
        data = 0x0
        multicast = 0x7fc7bc000bf0
        unicast = 0x7fc7bc000dd0
        conn = 0x7fc7bc000dd0
        buf = "HTTP/1.1 200 OK\r\nCACHE-CONTROL: max-age=1800\r\nEXT:\r\nLOCATION: http://192.168.178.55:9981/satip_server/desc.xml\r\nSERVER: unix/1.0 UPnP/1.1 TVHeadend/4.2.4-dmo1~bpo9+1~rpt1\r\nST: urn:ses-com:device:SatIP"...
        us = 0x7fc7b8001c60
        ip = {ss_family = 2, __ss_padding = "\al\300\250\262\067", '\000' <repeats 111 times>, __ss_align = 0}
        iplen = 16
        size = 345
        r = 0
        delay_ms = 0
#4  0x0000558632bf3f09 in thread_wrapper (p=0x5586378906b0) at src/tvh_thread.c:91
        ts = 0x5586378906b0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#5  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#6  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 18 (Thread 0x7fc7d8596700 (LWP 20646) "tvh:tcp-loop"):
#0  0x00007fc7db8bd29c in __lll_lock_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7db8b6714 in pthread_mutex_lock () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#2  0x0000558632bfc6c1 in tcp_server_loop (aux=0x0) at src/tcp.c:814
        r = 1
        ev = {fd = -1, events = 1, ptr = 0x558637815c20}
        ts = 0x558637815c20
        tsl = 0x7fc7ac0031b0
        slen = 16
        c = 74 'J'
#3  0x0000558632bf3f09 in thread_wrapper (p=0x5586378886c0) at src/tvh_thread.c:91
        ts = 0x5586378886c0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 17 (Thread 0x7fc7d8797700 (LWP 20645) "tvh:tshift-reap"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558634b117c0 <timeshift_reaper_cond>, mutex=0x558634b11740 <timeshift_reaper_lock>) at src/tvh_thread.c:363
        r = 0
        filename = 0x0
        lineno = -1
#2  0x0000558632d682e7 in timeshift_reaper_callback (p=0x0) at src/timeshift/timeshift_filemgr.c:55
        dpath = 0x558632bf3d3a <atomic_get+29> "\311\303UH\211\345H\203\354\020H\211}\370\211u\364\213U\364H\213E\370\211\326H\211\307\350\251\377\377\377\311\303UH\211\345H\211}\370H\213E\370Hi\300@B\017" 
        tsf = 0x0
        ti = 0x7fc7d8796a20
        tid = 0x558634c6cfd4 <tvhlog_level>
        sm = 0x0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863788c5f0) at src/tvh_thread.c:91
        ts = 0x55863788c5f0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7fc7d8998700 (LWP 20497) "tvh:hdhm-disc"):
#0  0x00007fc7db8ba3f9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf48f6 in tvh_cond_timedwait (cond=0x558634b11660 <tvhdhomerun_discovery_cond>, mutex=0x558634b115e0 <tvhdhomerun_discovery_lock>, monoclock=717857070124) at src/tvh_thread.c:404
        r = 0
        filename = 0x0
        lineno = -1
        ts = {tv_sec = 717857, tv_nsec = 70124000}
#2  0x0000558632d591df in tvhdhomerun_device_discovery_thread (aux=0x0) at src/input/mpegts/tvhdhomerun/tvhdhomerun.c:459
        result_list = {{ip_addr = 0, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 24 times>, base_url = '\000' <repeats 28 times>}, {ip_addr = 0, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 14 times>, "\301=\277\062\206U\000\000\000\000", base_url = "\024\000\000\000\000@\230<\333\307\177\000\000\004", '\000' <repeats 14 times>}, {ip_addr = 0, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 24 times>, base_url = '\000' <repeats 28 times>}, {ip_addr = 0, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 24 times>, base_url = "\000\000\000\000\000\000\037\202N}I\346\305\000\000\000\000\000\000\000\000\064\226<\333\307\177\000"}, {ip_addr = 851393985, device_type = 21894, device_id = 4, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 24 times>, base_url = '\000' <repeats 28 times>}, {ip_addr = 0, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 24 times>, base_url = '\000' <repeats 28 times>}, {ip_addr = 268435456, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = "\000\000\000\000\000\000\000\000\000\000\301=\277\062\206U\000\000\004\000\000\000\000\000", base_url = '\000' <repeats 28 times>}, {ip_addr = 0, device_type = 0, device_id = 0, tuner_count = 0 '\000', is_legacy = false, device_auth = '\000' <repeats 24 times>, base_url = '\000' <repeats 28 times>}}
        numDevices = 0
        brk = 0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x5586384ef920) at src/tvh_thread.c:91
        ts = 0x5586384ef920
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7fc7d9199700 (LWP 20496) "tvheadend"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558633d3f268 in thread_cond_wait ()
No symbol table info available.
#2  0x0000558633d3b8a0 in hdhomerun_debug_thread_execute ()
No symbol table info available.
#3  0x0000558633d3ef0a in thread_task_execute ()
No symbol table info available.
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7fc7d939a700 (LWP 20232) "tvh:iptv"):
#0  0x00007fc7db48b7ef in epoll_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
#1  0x0000558632c47618 in tvhpoll_wait (tp=0x55863787c9d0, evs=0x7fc7d93999e0, num=1, ms=-1) at src/tvhpoll.c:370
        nfds = 0
        i = 21894
#2  0x0000558632d5db5e in iptv_input_thread (aux=0x55863788b490) at src/input/mpegts/iptv/iptv.c:530
        pool = 0x55863788b490
        nfds = 32711
        r = 21894
        n = 0
        im = 0x558634c6cfd4 <tvhlog_level>
        mi = 0x0
        ev = {fd = 0, events = 0, ptr = 0x0}
#3  0x0000558632bf3f09 in thread_wrapper (p=0x558637d25850) at src/tvh_thread.c:91
        ts = 0x558637d25850
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7fc7d959b700 (LWP 20231) "tvh:iptv"):
#0  0x00007fc7db48b7ef in epoll_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
#1  0x0000558632c47618 in tvhpoll_wait (tp=0x558637884210, evs=0x7fc7d959a9e0, num=1, ms=-1) at src/tvhpoll.c:370
        nfds = 0
        i = 21894
#2  0x0000558632d5db5e in iptv_input_thread (aux=0x558637871af0) at src/input/mpegts/iptv/iptv.c:530
        pool = 0x558637871af0
        nfds = 32711
        r = 21894
        n = 0
        im = 0x558634c6cfd4 <tvhlog_level>
        mi = 0x0
        ev = {fd = 0, events = 0, ptr = 0x0}
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863787c8b0) at src/tvh_thread.c:91
        ts = 0x55863787c8b0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7fc7d979c700 (LWP 19666) "tvh:service"):
#0  0x00007fc7db8bd29c in __lll_lock_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7db8b6714 in pthread_mutex_lock () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#2  0x0000558632c2b690 in service_saver (aux=0x0) at src/service.c:1225
        t = 0x55863808ded0
        __PRETTY_FUNCTION__ = "service_saver" 
#3  0x0000558632bf3f09 in thread_wrapper (p=0x5586378942a0) at src/tvh_thread.c:91
        ts = 0x5586378942a0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7fc7d999d700 (LWP 19291) "tvh:httpc"):
#0  0x00007fc7db48b7ef in epoll_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
#1  0x0000558632c47618 in tvhpoll_wait (tp=0x558637867e20, evs=0x7fc7d999ca00, num=1, ms=-1) at src/tvhpoll.c:370
        nfds = 0
        i = 21894
#2  0x0000558632c63a4d in http_client_thread (p=0x0) at src/httpc.c:1426
        n = 1
        ev = {fd = -1, events = 1, ptr = 0x7fc7b8002b30}
        hc = 0x7fc7b8002b30
        c = 0 '\000'
#3  0x0000558632bf3f09 in thread_wrapper (p=0x558637861370) at src/tvh_thread.c:91
        ts = 0x558637861370
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7fc7d9b9e700 (LWP 19290) "tvh:imagecache"):
#0  0x00007fc7db8ba3f9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf48f6 in tvh_cond_timedwait (cond=0x558634b0fd80 <imagecache_cond>, mutex=0x558634a98920 <imagecache_lock>, monoclock=717925024048) at src/tvh_thread.c:404
        r = 0
        filename = 0x0
        lineno = -1
        ts = {tv_sec = 717925, tv_nsec = 24048000}
#2  0x0000558632c58979 in imagecache_thread (p=0x0) at src/imagecache.c:425
        img = 0x0
        timer_expire = 717925024048
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863786a0c0) at src/tvh_thread.c:91
        ts = 0x55863786a0c0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7fc7d9d9f700 (LWP 19164) "tvh:fsmonitor"):
#0  0x00007fc7db8bd29c in __lll_lock_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7db8b6714 in pthread_mutex_lock () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#2  0x0000558632c669f0 in fsmonitor_thread (p=0x0) at src/fsmonitor.c:68
        fd = 13
        c = 32
        i = 160
        buf = "\001\000\000\000\000\002\000\000\000\000\000\000\020\000\000\000random\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\002\000\000\000\000\000\000\020\000\000\000random\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\001\000\000\000\000\000\000\020\000\000\000random\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\002\000\000\000\000\000\000\020\000\000\000random\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\001\000\000\000\000\000\000\020\000\000\000random\000\000\000\000\000\000\000\000\000" 
        path = "/dev/random", '\000' <repeats 581 times>...
        ev = 0x7fc7d9d9e9c0
        fmp = 0x5586378722a0
        fml = 0x0
        fsm = 0x558634af76a0 <devmon>
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863785f1d0) at src/tvh_thread.c:91
        ts = 0x55863785f1d0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7fc7d9fa0700 (LWP 19144) "tvh:tasklet"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558634b0c020 <tasklet_cond>, mutex=0x558634c6ce20 <tasklet_lock>) at src/tvh_thread.c:363
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632bdc748 in tasklet_thread (aux=0x0) at src/main.c:521
        tsk = 0x0
        tsk_cb = 0x558632cbe325 <dvr_get_disk_space_tcb>
        opaque = 0x7fc7b8023a60
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863782e100) at src/tvh_thread.c:91
        ts = 0x55863782e100
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7fc7da1a1700 (LWP 19143) "tvh:mtimer"):
#0  0x00007fc7db8bd29c in __lll_lock_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7db8b6714 in pthread_mutex_lock () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#2  0x0000558632bdce7f in mtimer_thread (aux=0x0) at src/main.c:703
        mti = 0x558634b11560 <satip_discovery_msearch_timer>
        cb = 0x558632d4dbb4 <satip_discovery_send_msearch>
        now = 713455678171
        next = 717055678171
        id = 0x0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863782df90) at src/tvh_thread.c:91
        ts = 0x55863782df90
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7fc7da3a2700 (LWP 19142) "tvh:mtick"):
#0  0x00007fc7db499123 in clock_nanosleep () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
#1  0x0000558632bf34cd in tvh_usleep (us=100000) at src/wrappers.c:158
        ts = {tv_sec = 0, tv_nsec = 22630019}
        val = 140496336460272
        r = 32711
#2  0x0000558632bf33f1 in tvh_safe_usleep (us=100000) at src/wrappers.c:135
        r = 94034899422288
#3  0x0000558632bdccc2 in mtimer_tick_thread (aux=0x0) at src/main.c:655
No locals.
#4  0x0000558632bf3f09 in thread_wrapper (p=0x55863782de20) at src/tvh_thread.c:91
        ts = 0x55863782de20
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#5  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#6  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7fc7da5a3700 (LWP 19141) "tvh:save"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558634b0c320 <save_cond>, mutex=0x558634c6cda0 <global_lock>) at src/tvh_thread.c:363
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632becde2 in save_thread (aux=0x0) at src/idnode.c:1970
        ise = 0x0
        in = 0x0
        m = 0x7fc794001ca0
        u32 = 32711
        uuid = 0x558634c6cfd4 <tvhlog_level>
        filename = "dvr/log/ae4416da774aedf5f5445ef380256f98\000d734350296/muxes/1ed05dfb82779a6d5f9c3bc7609e3f8c", '\000' <repeats 3414 times>...
        set = {us_array = 0x0, us_count = 0, us_size = 0, us_alloc_chunk = 10}
        tset = {us_array = 0x0, us_count = 0, us_size = 0, us_alloc_chunk = 10}
        lnotify = 0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863782d9e0) at src/tvh_thread.c:91
        ts = 0x55863782d9e0
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7fc7dada4700 (LWP 19140) "tvheadend"):
#0  0x00007fc7db48b7ef in epoll_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
#1  0x0000558632c47618 in tvhpoll_wait (tp=0x7fc7cc000b20, evs=0x7fc7dada3ac0, num=2, ms=500) at src/tvhpoll.c:370
        nfds = 0
        i = 21894
#2  0x0000558632c17896 in spawn_pipe_thread (aux=0x0) at src/spawn.c:128
        ev = {{fd = -1, events = 1, ptr = 0x558634b0c848 <spawn_pipe_error>}, {fd = 10, events = 1, ptr = 0x558634b0c848 <spawn_pipe_error>}}
        efd = 0x7fc7cc000b20
        nfds = 0
#3  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#4  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7fc7dafa5700 (LWP 19139) "tvh:notify"):
#0  0x00007fc7db8bd29c in __lll_lock_wait () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00007fc7db8b6714 in pthread_mutex_lock () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#2  0x0000558632c08268 in notify_thread (p=0x0) at src/notify.c:103
        q = 0x7fc754001930
        f = 0x0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x55863782d730) at src/tvh_thread.c:91
        ts = 0x55863782d730
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7fc7db1a6700 (LWP 19084) "tvh:log"):
#0  0x00007fc7db8ba00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632bf47f7 in tvh_cond_wait (cond=0x558634c6d080 <tvhlog_cond>, mutex=0x558634c6cf20 <tvhlog_mutex>) at src/tvh_thread.c:363
        r = 21894
        filename = 0x0
        lineno = -1
#2  0x0000558632be291c in tvhlog_thread (p=0x0) at src/tvhlog.c:357
        options = 272
        path = 0x0
        buf = '\000' <repeats 41 times>, "\030\":\003\000\000\000\001", '\000' <repeats 47 times>, "n?\000%d\n\000/tmp\000fail", '\000' <repeats 16 times>, "(;069(=!TF\032\001\b\033\fF", '\000' <repeats 57 times>...
        fp = 0x0
        msg = 0x0
#3  0x0000558632bf3f09 in thread_wrapper (p=0x558637814530) at src/tvh_thread.c:91
        ts = 0x558637814530
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007fc7db8b3fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007fc7db48b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7fc7db378b40 (LWP 19080) "tvheadend"):
#0  0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x0000558632cb1c1c in dvr_rec_unsubscribe (de=0x7fc750b057c0) at src/dvr/dvr_rec.c:198
        prch = 0x7fc75c0792a0
        postproc = 0x0
        __PRETTY_FUNCTION__ = "dvr_rec_unsubscribe" 
#2  0x0000558632cacd44 in dvr_stop_recording (de=0x7fc750b057c0, stopcode=0, saveconf=1, clone=0) at src/dvr/dvr_db.c:2868
        rec_state = DVR_RS_RUNNING
        dae = 0x558638da4950
#3  0x0000558632cad06d in dvr_timer_stop_recording (aux=0x7fc750b057c0) at src/dvr/dvr_db.c:2924
        de = 0x7fc750b057c0
#4  0x0000558632bdd0d1 in mainloop () at src/main.c:767
        gti = 0x7fc750b05868
        cb = 0x558632cace9d <dvr_timer_stop_recording>
        now = 1614867026
        ts = {tv_sec = 1614870626, tv_nsec = 0}
        id = 0x0
#5  0x0000558632be05fa in main (argc=8, argv=0x7fffd37b2348) at src/main.c:1364
        i = 8
        set = {__val = {16386, 0 <repeats 15 times>}}
        adapter_mask = 4294967295
        log_level = 6
        log_options = 304
        log_debug = 0x0
        log_trace = 0x0
        gid = 0
        uid = 0
        buf = "/\000\377\000\000\000\377\000\000\000\000\000\000\000\000\000", '/' <repeats 16 times>, '\000' <repeats 64 times>, "nd/bin/tvheadend", '\000' <repeats 18 times>, "\377\000\000\000\377\000\000\000\000\000\000\000\000\000", '/' <repeats 16 times>, '\000' <repeats 48 times>...
        pidfile = 0x55863780da00
        randseed = {thread_id = 0x7fc7db378b40, tv = {tv_sec = 1614846893, tv_usec = 528987}, ru = "\023\376\306N\341\037\360C8f!]\355\372֤w\230O\246\213N\203qR\030\270\252\224\225", <incomplete sequence \345\256>}
        rl = {rlim_cur = 8388608, rlim_max = 18446744073709551615}
        opt_help = 0
        opt_version = 0
        opt_fork = 1
        opt_firstrun = 0
        opt_stderr = 0
        opt_nostderr = 0
        opt_syslog = 0
        opt_nosyslog = 0
        opt_uidebug = 0
        opt_abort = 0
        opt_noacl = 0
        opt_fileline = 0
        opt_threadid = 0
        opt_libav = 0
        opt_ipv6 = 0
        opt_nosatip = 0
        opt_satip_rtsp = 0
        opt_tsfile_tuner = 0
        opt_dump = 0
        opt_xspf = 0
        opt_dbus = 0
        opt_dbus_session = 0
        opt_nobackup = 0
        opt_nobat = 0
        opt_subsystems = 0
        opt_tprofile = 0
        opt_thread_debug = 0
        opt_config = 0x7fffd37b3a1a "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/config" 
        opt_user = 0x7fffd37b3a4e "admin" 
        opt_group = 0x7fffd37b3a5c "administrators" 
        opt_logpath = 0x0
        opt_log_debug = 0x0
        opt_log_trace = 0x0
        opt_pidpath = 0x558633d4333d "/run/tvheadend.pid" 
        opt_dvb_adapters = 0x0
        opt_bindaddr = 0x0
        opt_subscribe = 0x0
        opt_user_agent = 0x0
        opt_satip_bindaddr = 0x0
        __opt_satip_xml = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
        opt_satip_xml = {max = 10, num = 0, str = 0x558634b0c060 <__opt_satip_xml.39098>}
        __opt_satip_tsfile = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
        opt_tsfile = {max = 10, num = 0, str = 0x558634b0c0c0 <__opt_satip_tsfile.39100>}
        cmdline_opts = {{sopt = 0 '\000', lopt = 0x0, desc = 0x558633d43350 "Generic options", type = OPT_BOOL, param = 0x0}, {sopt = 104 'h', lopt = 0x558633d43360 "help", desc = 0x558633d43365 "Show this page", type = OPT_BOOL, param = 0x7fffd37b1efc}, {sopt = 118 'v', lopt = 0x558633d43374 "version", desc = 0x558633d4337c "Show version information", type = OPT_BOOL, param = 0x7fffd37b1ef8}, {sopt = 0 '\000', lopt = 0x0, desc = 0x558633d43395 "Service configuration", type = OPT_BOOL, param = 0x0}, {sopt = 99 'c', lopt = 0x558633d433ab "config", desc = 0x558633d433b2 "Alternate configuration path", type = OPT_STR, param = 0x7fffd37b1e90}, {sopt = 66 'B', lopt = 0x558633d433cf "nobackup", desc = 0x558633d433d8 "Don't backup configuration tree at upgrade", type = OPT_BOOL, param = 0x7fffd37b1eac}, {sopt = 102 'f', lopt = 0x558633d43403 "fork", desc = 0x558633d43408 "Fork and run as daemon", type = OPT_BOOL, param = 0x7fffd37b1ef4}, {sopt = 117 'u', lopt = 0x558633d4341f "user", desc = 0x558633d43424 "Run as user", type = OPT_STR, param = 0x7fffd37b1e88}, {sopt = 103 'g', lopt = 0x558633d43430 "group", desc = 0x558633d43436 "Run as group", type = OPT_STR, param = 0x7fffd37b1e80}, {sopt = 112 'p', lopt = 0x558633d43443 "pid", desc = 0x558633d43447 "Alternate PID path", type = OPT_STR, param = 0x7fffd37b1e60}, {sopt = 67 'C', lopt = 0x558633d4345a "firstrun", desc = 0x558633d43468 "If no user account exists then create one with\nno username and no password. Use with care as\nit will allow world-wide administrative access\nto your Tvheadend installation until you create or edit\nthe "..., type = OPT_BOOL, param = 0x7fffd37b1ef0}, {sopt = 97 'a', lopt = 0x558633d43568 "adapters", desc = 0x558633d43578 "Only use specified DVB adapters (comma-separated, -1 = none)", type = OPT_STR, param = 0x7fffd37b1e58}, {sopt = 0 '\000', lopt = 0x558633d435b5 "satip_bindaddr", desc = 0x558633d435c8 "Specify bind address for SAT>IP server", type = OPT_STR, param = 0x7fffd37b1e38}, {sopt = 0 '\000', lopt = 0x558633d435ef "satip_rtsp", desc = 0x558633d43600 "SAT>IP RTSP port number for server\n(default: -1 = disable, 0 = webconfig, standard port is 554)", type = OPT_INT, param = 0x7fffd37b1ebc}, {sopt = 0 '\000', lopt = 0x558633d43660 "nosatip", desc = 0x558633d43668 "Disable SAT>IP client", type = OPT_BOOL, param = 0x7fffd37b1ec0}, {sopt = 0 '\000', lopt = 0x558633d4367e "satip_xml", desc = 0x558633d43688 "URL with the SAT>IP server XML location", type = OPT_STR_LIST, param = 0x7fffd37b1e20}, {sopt = 0 '\000', lopt = 0x0, desc = 0x558633d436b0 "Server connectivity", type = OPT_BOOL, param = 0x0}, {sopt = 54 '6', lopt = 0x558633d436c4 "ipv6", desc = 0x558633d436c9 "Listen on IPv6", type = OPT_BOOL, param = 0x7fffd37b1ec4}, {sopt = 98 'b', lopt = 0x558633d436d8 "bindaddr", desc = 0x558633d436e1 "Specify bind address", type = OPT_STR, param = 0x7fffd37b1e50}, {sopt = 0 '\000', lopt = 0x558633d436f6 "http_port", desc = 0x558633d43700 "Specify alternative http port", type = OPT_INT, param = 0x558634c6ce10 <tvheadend_webui_port>}, {sopt = 0 '\000', lopt = 0x558633d4371e "http_root", desc = 0x558633d43728 "Specify alternative http webroot", type = OPT_STR, param = 0x558634c6ccd8 <tvheadend_webroot>}, {sopt = 0 '\000', lopt = 0x558633d43749 "htsp_port", desc = 0x558633d43753 "Specify alternative htsp port", type = OPT_INT, param = 0x558634c6ccf0 <tvheadend_htsp_port>}, {sopt = 0 '\000', lopt = 0x558633d43771 "htsp_port2", desc = 0x558633d4377c "Specify extra htsp port", type = OPT_INT, param = 0x558634c6cd90 <tvheadend_htsp_port_extra>}, {sopt = 0 '\000', lopt = 0x558633d43794 "useragent", desc = 0x558633d437a0 "Specify User-Agent header for the http client", type = OPT_STR, param = 0x7fffd37b1e40}, {sopt = 0 '\000', lopt = 0x558633d437ce "xspf", desc = 0x558633d437d8 "Use XSPF playlist instead of M3U", type = OPT_BOOL, param = 0x7fffd37b1eb0}, {sopt = 0 '\000', lopt = 0x0, desc = 0x558633d437f9 "Debug options", type = OPT_BOOL, param = 0x0}, {sopt = 100 'd', lopt = 0x558633d43807 "stderr", desc = 0x558633d4380e "Enable debug on stderr", type = OPT_BOOL, param = 0x7fffd37b1eec}, {sopt = 110 'n', lopt = 0x558633d43825 "nostderr", desc = 0x558633d4382e "Disable debug on stderr", type = OPT_BOOL, param = 0x7fffd37b1ee8}, {sopt = 115 's', lopt = 0x558633d43846 "syslog", desc = 0x558633d4384d "Enable debug to syslog", type = OPT_BOOL, param = 0x7fffd37b1ee4}, {sopt = 83 'S', lopt = 0x558633d43864 "nosyslog", desc = 0x558633d4386d "Disable syslog (all messages)", type = OPT_BOOL, param = 0x7fffd37b1ee0}, {sopt = 108 'l', lopt = 0x558633d4388b "logfile", desc = 0x558633d43893 "Enable debug to file", type = OPT_STR, param = 0x7fffd37b1e78}, {sopt = 0 '\000', lopt = 0x558633d438a8 "debug", desc = 0x558633d438ae "Enable debug subsystems", type = OPT_STR, param = 0x7fffd37b1e70}, {sopt = 0 '\000', lopt = 0x558633d431f0 "trace", desc = 0x558633d438c6 "Enable trace subsystems", type = OPT_STR, param = 0x7fffd37b1e68}, {sopt = 0 '\000', lopt = 0x558633d438de "subsystems", desc = 0x558633d438e9 "List subsystems", type = OPT_BOOL, param = 0x7fffd37b1ea4}, {sopt = 0 '\000', lopt = 0x558633d438f9 "fileline", desc = 0x558633d43908 "Add file and line numbers to debug", type = OPT_BOOL, param = 0x7fffd37b1ed0}, {sopt = 0 '\000', lopt = 0x558633d4392b "threadid", desc = 0x558633d43934 "Add the thread ID to debug", type = OPT_BOOL, param = 0x7fffd37b1ecc}, {sopt = 0 '\000', lopt = 0x558633d431f6 "libav", desc = 0x558633d4394f "More verbose libav log", type = OPT_BOOL, param = 0x7fffd37b1ec8}, {sopt = 0 '\000', lopt = 0x558633d43966 "uidebug", desc = 0x558633d43970 "Enable web UI debug (non-minified JS)", type = OPT_BOOL, param = 0x7fffd37b1edc}, {sopt = 65 'A', lopt = 0x558633d43996 "abort", desc = 0x558633d4399c "Immediately abort", type = OPT_BOOL, param = 0x7fffd37b1ed8}, {sopt = 68 'D', lopt = 0x558633d439ae "dump", desc = 0x558633d439b3 "Enable coredumps for daemon", type = OPT_BOOL, param = 0x7fffd37b1eb4}, {sopt = 0 '\000', lopt = 0x558633d439cf "noacl", desc = 0x558633d439d8 "Disable all access control checks", type = OPT_BOOL, param = 0x7fffd37b1ed4}, {sopt = 0 '\000', lopt = 0x558633d439fa "nobat", desc = 0x558633d43a00 "Disable DVB bouquets", type = OPT_BOOL, param = 0x7fffd37b1ea8}, {sopt = 106 'j', lopt = 0x558633d43a15 "join", desc = 0x558633d43a20 "Subscribe to a service permanently", type = OPT_STR, param = 0x7fffd37b1e48}, {sopt = 0 '\000', lopt = 0x0, desc = 0x558633d43a43 "Testing options", type = OPT_BOOL, param = 0x0}, {sopt = 0 '\000', lopt = 0x558633d43a53 "tsfile_tuners", desc = 0x558633d43a61 "Number of tsfile tuners", type = OPT_INT, param = 0x7fffd37b1eb8}, {sopt = 0 '\000', lopt = 0x558633d43a79 "tsfile", desc = 0x558633d43a80 "tsfile input (mux file)", type = OPT_STR_LIST, param = 0x7fffd37b1e10}, {sopt = 0 '\000', lopt = 0x558633d43a98 "tprofile", desc = 0x558633d43aa8 "Gather timing statistics for the code", type = OPT_BOOL, param = 0x7fffd37b1ea0}, {sopt = 0 '\000', lopt = 0x558633d43ace "thrdebug", desc = 0x558633d43ad7 "Thread debugging", type = OPT_INT, param = 0x7fffd37b1e9c}}
(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 19080, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\210J\000\000\003", '\000' <repeats 26 times>, __align = 2}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}
(gdb) thread find 19080
Thread 1 has target id 'Thread 0x7fc7db378b40 (LWP 19080)'
(gdb) info threads 1
  Id   Target Id                                     Frame 
* 1    Thread 0x7fc7db378b40 (LWP 19080) "tvheadend" 0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (Thread 0x7fc7db378b40 (LWP 19080))]
#0  0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x0000558632cb1c1c in dvr_rec_unsubscribe (de=0x7fc750b057c0) at src/dvr/dvr_rec.c:198
#2  0x0000558632cacd44 in dvr_stop_recording (de=0x7fc750b057c0, stopcode=0, saveconf=1, clone=0) at src/dvr/dvr_db.c:2868
#3  0x0000558632cad06d in dvr_timer_stop_recording (aux=0x7fc750b057c0) at src/dvr/dvr_db.c:2924
#4  0x0000558632bdd0d1 in mainloop () at src/main.c:767
#5  0x0000558632be05fa in main (argc=8, argv=0x7fffd37b2348) at src/main.c:1364
(gdb) 

@Flole Systems
Hi, please don't consider this message as a push, just an information request. I know that there are a lot of things to do and you're following this project in your spare time, so no problem here!

But I'm just becoming curious: do you think that with the log above (post 57) you can understand a little more about the cause of the OP issue? I see that there are no missing symbols this time.
Of course I'm not able to interpret this kind of log :-(

I have to give also some new information: Some days ago (02. March 2021) QNAP offers an update of its fw and I installed it 2 days 16 hours ago. Since then the RAM usage of TVH seems not to be to high. Of cause, this can be an random.

Some news:
After a few days with a RAM requirement of approx. max. 130 MB, the RAM requirement jumped 10-fold to approx. 1.4 GB within an hour during a recording yesterday. This recording also ran for exactly one hour. The last few days this did not occur with other recordings After that it remained constant again for hours, even during the OTA EPG.

The issue comes again and again. now it seems more often after the start of recordings. So my next action is to change this debugging-version to the last version and use without ota-epg but external-xmltv.

Harald K wrote:

So my next action is to change this debugging-version to the last version and use without ota-epg but external-xmltv.

Sorry but I'm not skilled in C enough to decode the stack trace above myself, otherwise I would have tried...

Doesn't matter. I'm happy about every try and you have done much effort.
I had done one step in between: Continue using the debugging-version but stop the ota-epg and try to use the external xmltv like you wrote in #37. If I get back my old single tuner repaired from sundtek I'll try the container-solution. But I'm waiting since now for it for some months. But that's another topic. Another idea was to get the epg from my raspberry pi with its TVH and save this like written in #37 to the QNAP. But first I have to be sure the ota-epg is really the problem.

Up to now the only solution seems to be to have always a look on the system and restart TVH the right time. Not very comfortable and not so save, of cause.

Sorry, #37 means post 37...

I'm not happy: Now this issue commes without activated ota-epg. I'll restart TVH again now.

Harald K wrote:

I'm not happy: Now this issue commes without activated ota-epg. I'll restart TVH again now.

Maby problem is elsewhere

tvheadend# cppcheck . --force > err.txt
[src/api/api_epg.c:559]: (error) Common realloc mistake: 'bcast_entries' nulled but not freed upon failure
[src/htsmsg.c:1481]: (error) Common realloc mistake: 'ret' nulled but not freed upon failure
[src/htsmsg.c:1484]: (error) Common realloc mistake: 'ret' nulled but not freed upon failure
[src/htsmsg.c:1487]: (error) Common realloc mistake: 'ret' nulled but not freed upon failure
[src/htsp_server.c:3443]: (error) Address of local auto-variable assigned to a function parameter.
[src/idnode.c:1486]: (error) Common realloc mistake: 'ret' nulled but not freed upon failure
[src/idnode.c:1492]: (error) Common realloc mistake: 'ret' nulled but not freed upon failure
[src/input/mpegts/dvb_psi.c:584]: (error) Return value of allocation function 'dvb_bat_find_service' is not stored.
[src/input/mpegts/dvb_psi.c:612]: (error) Return value of allocation function 'dvb_bat_find_service' is not stored.
[src/input/mpegts/dvb_psi.c:2071]: (error) Return value of allocation function 'dvb_bat_find_service' is not stored.
[src/input/mpegts/mpegts_input.c:1884]: (error) syntax error
[src/parsers/parser_avc.c:138]: (error) Common realloc mistake: 'sps_array' nulled but not freed upon failure
[src/parsers/parser_avc.c:139]: (error) Common realloc mistake: 'sps_size_array' nulled but not freed upon failure
[src/parsers/parser_avc.c:144]: (error) Common realloc mistake: 'pps_size_array' nulled but not freed upon failure
[src/parsers/parser_avc.c:145]: (error) Common realloc mistake: 'pps_array' nulled but not freed upon failure
[src/plumbing/tsfix.c:164]: (error) Memory leak: tfs
[src/prop.c:628]: (error) Common realloc mistake: 'r' nulled but not freed upon failure
[src/satip/rtsp.c:199]: (error) syntax error
[src/tvh_locale.c:284]: (error) Return value of allocation function 'lng_add' is not stored.
[src/webui/xmltv.c:189]: (error) syntax error

saen acro wrote:

Harald K wrote:

I'm not happy: Now this issue commes without activated ota-epg. I'll restart TVH again now.

Maby problem is elsewhere

[...]

Those are all false positives/intended behaviour.

virtual dj wrote:

@Flole Systems
Hi, please don't consider this message as a push, just an information request. I know that there are a lot of things to do and you're following this project in your spare time, so no problem here!

But I'm just becoming curious: do you think that with the log above (post 57) you can understand a little more about the cause of the OP issue? I see that there are no missing symbols this time.
Of course I'm not able to interpret this kind of log :-(

Based on the log it looks like Tvheadend got stuck during the unsubscribe. That would need further investigation now with gdb to see what the thread which should be joined is doing now. I don't think thats the reason for the memory increase though.

Sorry, but I don't know what this all means. It seems this problem occures only on my system and only since the last main step of the firmware of the QNAP. On the other side it touchs mainly TVH and can be stopped by restarting this. Unfortunately I'm the absolute idiot on this topic and don't absotutly understand what all the loggings will tell me.

virtual dj wrote:

Harald K wrote:

I'm afraid something is not yet going as intended. What am I doing wrong?

I've tried to reproduce loading a core file in gdb (even though I don't have your issue so all is going well in my case) and I get the ??? in the stack trace, too .

Definitely I'm stupid or was thinking to something else... I've found the reason why the core dump file didn't load with the symbols in gdb! It was the command line that was missing the TVHeadend binary!

1. Start TVHeadend
   

   [/share/Public/package] # tvheadend start
   [/share/Public/package] # tvheadend status
   TVHeadend status: RUNNING.
   

2. Wait for the issue, when you see the mpegts: too much queued table input data messages on the log, attach gdb to TVHeadend:
   

   [/share/Public/package] # echo "set auto-load safe-path /" > ~/.gdbinit
   [/share/Public/package] # echo "set pagination off" >> ~/.gdbinit
   
   [/share/Public/package] # cat ~/.gdbinit
   set auto-load safe-path /
   set pagination off
   
   [/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend $(pidof tvheadend)
   ... cut ...
   [New LWP 11118]
   [New LWP 11119]
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
   0x00007f9e6a6ce35b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
   
   (gdb) set logging on
   Copying output to gdb.txt.
   Copying debug output to gdb.txt.
   
   (gdb) thread apply all bt full
   ... cut ...
   er timing statistics for the code", type = OPT_BOOL, param = 0x7ffef562ce30}, {sopt = 0 '\000', lopt = 0x55a66b256ace "thrdebug", desc = 0x55a66b256ad7 "Thread debugging", type = OPT_INT, param = 0x7ffef562ce2c}}
   
   (gdb) generate-core-file
   warning: target file /proc/11058/cmdline contained unexpected null characters
   Saved corefile core.11058
   
   (gdb) q
   A debugging session is active.
   
           Inferior 1 [process 11058] will be detached.
   
   Quit anyway? (y or n) y
   Detaching from program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 11058
   [Inferior 1 (process 11058) detached]
   

Now TVHeadend is still running (because gdb has been detached) and you can either stop it or let it run, it doesn't matter when you have the core file.

But here was the error! The core file is called, in this example, core.11058; scrolling up in the ticket, I saw that you (and I of course) used this command line:

[/share/Public/package] # gdb --core=core.11058
... cut ...
Core was generated by `/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend'.
#0  0x00007f9e6a6ce35b in ?? ()
[Current thread is 1 (LWP 11058)]

(gdb) bt
#0  0x00007f9e6a6ce35b in ?? ()
#1  0x0000000000000000 in ?? ()

(gdb) q

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend core.11058
... cut ...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
Core was generated by `/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend'.
#0  0x00007f9e6a6ce35b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
[Current thread is 1 (Thread 0x7f9e6a18ff80 (LWP 11058))]

(gdb) bt
#0  0x00007f9e6a6ce35b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x000055a66a10798f in tvh_cond_timedwait_ts (cond=0x55a66c01ef80 <gtimer_cond>, mutex=0x55a66c17fbe0 <gtimer_lock>, ts=0x7ffef562c400) at src/tvh_thread.c:425
#2  0x000055a66a0f012e in mainloop () at src/main.c:775
#3  0x000055a66a0f35fa in main (argc=8, argv=0x7ffef562d2d8) at src/main.c:1364

(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 2, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 12 times>, "\002", '\000' <repeats 26 times>, __align = 0}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}

(gdb) q

[/share/Public/package] # tvheadend stop
[/share/Public/package] # tvheadend status
TVHeadend status: STOPPED.

[/share/Public/package] # gdb /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend core.11058
... cut ...
[New LWP 11119]

warning: Could not load shared library symbols for /tmp/tvh-loader.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
Core was generated by `/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend'.
#0  0x00007f9e6a6ce35b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
[Current thread is 1 (Thread 0x7f9e6a18ff80 (LWP 11058))]

[/share/Public/package] # /bin/ln -sf /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader /tmp/tvh-loader

Flole Systems wrote:

That would need further investigation now with gdb to see what the thread which should be joined is doing now.

Now that I think we've solved the issue with the core file, can you provide some instructions to do what you've asked with gdb?

Select thread 1 again and figure out which thread de->de_thread points to. Then select that one and see what that thread is doing (bt full). There should be a dvr_thread if I remember correctly but that doesn't seem to be there in your gdb.

Since my last restart of the QNAP and so also restart TVH thy system runs without problems (without using ota epg). Unfortunately this time I'm not so free in my time so pleas let me some time watching wether theproblem occurs again. But I can do the do the steps that virtual dj has written because I have the first core file saved - only I'm not sure wether the issue was the right one that time.

I see one main differency of my output against that one of virtual dj. But I don't find something like "de->de_thread"

virtual dj:

(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 2, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 12 times>, "\002", '\000' <repeats 26 times>, __align = 0}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}

me:

(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 18346, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\252G\000\000\003", '\000' <repeats 26 times>, __align = 2},  magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}

the difference is

__size = "\002\000\000\000\000\000\000\000\252G\000\000\003", '\000' <repeats 26 times>, __align = 2

Harald K wrote:

But I don't find something like "de->de_thread"

I'm just guessing, here, as I never used gdb previously (like you, I suppose).

Flole Systems wrote:

Select thread 1 again and figure out which thread de->de_thread points to.

So you should open the core file as I've written in post 70, then do the same that you did on post 57.
I mean, at the end of the log you had:

$1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 19080, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\210J\000\000\003", '\000' <repeats 26 times>, __align = 2}, magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}
(gdb) thread find 19080
Thread 1 has target id 'Thread 0x7fc7db378b40 (LWP 19080)'
(gdb) info threads 1
  Id   Target Id                                     Frame 
* 1    Thread 0x7fc7db378b40 (LWP 19080) "tvheadend" 0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (Thread 0x7fc7db378b40 (LWP 19080))]
#0  0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007fc7db8b5495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x0000558632cb1c1c in dvr_rec_unsubscribe (de=0x7fc750b057c0) at src/dvr/dvr_rec.c:198
#2  0x0000558632cacd44 in dvr_stop_recording (de=0x7fc750b057c0, stopcode=0, saveconf=1, clone=0) at src/dvr/dvr_db.c:2868
#3  0x0000558632cad06d in dvr_timer_stop_recording (aux=0x7fc750b057c0) at src/dvr/dvr_db.c:2924
#4  0x0000558632bdd0d1 in mainloop () at src/main.c:767
#5  0x0000558632be05fa in main (argc=8, argv=0x7fffd37b2348) at src/main.c:1364

#1  0x0000558632cb1c1c in dvr_rec_unsubscribe (de=0x7fc750b057c0) at src/dvr/dvr_rec.c:198

I think we should investigate the value of that de->de_thread parameter. Maybe with:

(gdb) print dvr_rec_unsubscribe::de->de_thread

Flole Systems wrote:

Then select that one and see what that thread is doing (bt full).

So after discovering the thread ID (e.g. "9") with the previous point, probably:

(gdb) thread 9
(gdb) bt full

Yes, I have done exactly the same like in post 57 and get (after the output of thread apply all bt full):

(gdb) print global_lock
$1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 18346, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\252G\000\000\003", '\000' <repeats 26 times>, __align = 2},
  magic1 = 3614061450, tid = 0, filename = 0x0, lineno = 0, tstamp = 0, waiters = {lh_first = 0x0}, link = {tqe_next = 0x0, tqe_prev = 0x0}, magic2 = 4181353505}
(gdb) thread find 18346
Thread 1 has target id 'LWP 18346'
(gdb) info threads 1
  Id   Target Id         Frame
* 1    LWP 18346         0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (LWP 18346)]
#0  0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007f33c6075495 in __pthread_timedjoin_ex () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x00005650e4e04c1c in dvr_rec_unsubscribe (de=0x5650ec2bf5a0) at src/dvr/dvr_rec.c:198
#2  0x00005650e4dffd44 in dvr_stop_recording (de=0x5650ec2bf5a0, stopcode=0, saveconf=1, clone=0) at src/dvr/dvr_db.c:2868
#3  0x00005650e4e0006d in dvr_timer_stop_recording (aux=0x5650ec2bf5a0) at src/dvr/dvr_db.c:2924
#4  0x00005650e4d300d1 in mainloop () at src/main.c:767
#5  0x00005650e4d335fa in main (argc=8, argv=0x7fffc74af498) at src/main.c:1364
(gdb) print dvr_rec_unsubscribe::de->de_thread
$2 = 139859662386944
(gdb) thread 2
[Switching to thread 2 (LWP 18347)]
#0  0x00007f33c607a00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt full
#0  0x00007f33c607a00c in pthread_cond_wait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#1  0x00005650e4d477f7 in tvh_cond_wait (cond=0x5650e6dc0080 <tvhlog_cond>, mutex=0x5650e6dbff20 <tvhlog_mutex>) at src/tvh_thread.c:363
        r = 22096
        filename = 0x0
        lineno = -1
#2  0x00005650e4d3591c in tvhlog_thread (p=0x0) at src/tvhlog.c:357
        options = 272
        path = 0x0
        buf = '\000' <repeats 41 times>, "\030\":\003", '\000' <repeats 51 times>, "n?\000%d\n\000/tmp\000fail", '\000' <repeats 16 times>, "(;069(=!TF\032\001\b\033\fF", '\000' <repeats 57 times>...
        fp = 0x0
        msg = 0x0
#3  0x00005650e4d46f09 in thread_wrapper (p=0x5650e9e6c530) at src/tvh_thread.c:91
        ts = 0x5650e9e6c530
        set = {__val = {16388, 0 <repeats 15 times>}}
        r = 0x0
#4  0x00007f33c6073fa3 in start_thread () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
No symbol table info available.
#5  0x00007f33c5c4b4cf in clone () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
No symbol table info available.
(gdb)

Yesterday I have the issue again while ota-epg don't run. But the problem seems to be not so bad this time, because the system continues running and TVH works without failures. What do I see? The warning: "mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new" in the log, unfortunately it vanishes later and I can't exactly say what time this happens. You know, my system has 64 GB RAM, so it seems this time the problem was "not big enough"??

12:25 the usage of RAM starts to invrease until 15:25 from 9,5 GB to 5.2 GB. This time 2 recordings are running: one 12:25 until 13:40, the second one 12:49 until 14:04, both on the same channel.
15:25 the increasing stops until 15:50, then it continues until 16:35 to about 6 GB. Since then there is no change, but 2 more recordings in the evening have no influence.

But last view days I see that this issue often happnes (if it happens) at this time when start recording this channel. Random?

Of cause I startet gdb with the steps above. But there is nothing to see. Here are the last steps:

(gdb) thread find 26474
Thread 1 has target id 'LWP 26474'
(gdb) info threads 1
  Id   Target Id             Frame
* 1    LWP 26474 "tvheadend" 0x00007ffad09d535b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (LWP 26474)]
#0  0x00007ffad09d535b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) q
A debugging session is active.

        Inferior 1 [process 26474] will be detached.

Quit anyway? (y or n) y

The error (too much queue...) simply means there's something wrong and it's generated here:
https://github.com/tvheadend/tvheadend/blob/736ac427b1934832aab23391f5ce35f687c999c6/src/input/mpegts/mpegts_input.c#L1548

So probably it could be possible to put a breakpoint there and stop TVHeadend in that moment and then capturing the core file. You'll lose the recordings, though (because the breakpoint stops/pauses TVHeadend itself).

(gdb) break mpegts_input.c:1548

Just for my curiosity:

(gdb) print dvr_rec_unsubscribe::de->de_thread
$2 = 139859662386944
(gdb) thread 2

Then I really misunderstood. I thought $2 meant thread 2.

For my understanding: You mean I have to wait for this warning, start exactly in this moment gdb with THV and the current pid and then direct without all the other steps type

(gdb) break mpegts_input.c:1548

You printed a pointer, so that's an address in memory which you have printed there. You should do something like

print *dvr_rec_unsubscribe::de->de_thread

There's no point in breaking at the warning, all you'll see is that the buffer is full, which we already know. To me it starts to sound like a problem with the recordings if that's when the RAM usage increases, have you tried a different caching method?

What do you mean by "different caching method"?
The issue not only occuers while recordings and not while all recordings. Somtime it occures direct after starting TVH somtimes not for several days.

And again: This runs for more than 4 years on this machine without problems. Only since the last update of the fw of the QNAP.

Harald K wrote:

Only since the last update of the fw of the QNAP.

Yes, but also the TVHeadend version has been updated, so you cannot be sure that the culprit is only within the QNAP firmware. If it was possible, you should have run the same version of TVHeadend with the old firmware, but you can't.

Flole Systems wrote:

There's no point in breaking at the warning, all you'll see is that the buffer is full, which we already know.

Sorry, my fault!
So Harald try to reopen the core file and see what:

print *dvr_rec_unsubscribe::de->de_thread

The result is

(gdb) print *dvr_rec_unsubscribe::de->de_thread
$2 = -1652640000
(gdb)

I think this fact is not really important to find the failure. But You are right, I had updated two times TVH since the (main) update of the fw of the QNAP. But the failure with the RAB occures first time before this updates of TVH when I'm running the same version of TVH since long time. Only I don't see this time that it occures in combination from QNAP-fw and TVH.

I have to say sorry to you both, Flole Systems and virtual dj. It must be hard to work this style with someone like me: Completely stupid in this topic and need always mach time for the answer ...

Flole Systems wrote:

Select thread 1 again and figure out which thread de->de_thread points to. Then select that one and see what that thread is doing (bt full). There should be a dvr_thread if I remember correctly but that doesn't seem to be there in your gdb.

@Flole Systems
How do we search that -1652640000 value on the threads list?

(gdb) thread find -1652640000

I have one more idea: The fact that not only ota epg but also recording is the source of the failure is not true. Perhaps my configuration is not correct for this. I have stopped the cron for the ota epg but I don't disable the ota epg grabber. So when a recording starts the ota epg for this channel also starts.

Now I'll really disable the ota epg. I'm not so happy about this, because the xmltv is not so precise like the opta epg. Some channels are completely not included.

Since my last post here and running really without OTA-EPG and 2-3 restarts of TVH (because other reasons) the problem don't occure.

virtual dj: I'm running an older version of TVH on the raspberry pi and use it as sat->ip sever for this TVH on the QNAP. Is it possible to use the OTA of the EPG as an external XMLTV-base?

Harald K wrote:

virtual dj: I'm running an older version of TVH on the raspberry pi and use it as sat->ip sever for this TVH on the QNAP. Is it possible to use the OTA of the EPG as an external XMLTV-base?

I don't know if there's a way to "export" the OTA EPG from TVHeadend as a XMLTV file (to use it to feed the xmltv.sock in the other TVHeadend instance running on the QNAP). Probably not?

Sorry for the stupid question from a total layman: Is anything happening regarding this bug?

I run tvheadend on a Liva X x86-64 mini PC with Debian Buster and have a similar issue. Just a few minutes ago, tvheadend was using all 2GB of my RAM so I tried saen acro's suggestion https://tvheadend.org/issues/6002#note-1 :

 sync && echo 3 > /proc/sys/vm/drop_caches

This recovered the memory but shortly thereafter a recording started and all RAM was was immediately consumed by tvheadend. I tried the above command again while the recording was ongoing, but this time there was no change in memory usage. So at least in my case it looks like the memory leak is associated with recording and not with the EPG.

Most likely you aren't seeing a memory leak but a simple "can't write to HDD fast enough". Data is then cached in RAM. That shouldn't be related to this issue though (and is NOT an issue but perfectly normal behaviour), as in this case the issue is not there when EPG grabbing is not happening, recordings don't seem to matter.

@Flole Systems Systems, write speed benchmark with a recoding in progress is 135 MB/s. I think that is plenty fast enough to keep up with a recording.

Jonathan S wrote:
@Flole Systems, write speed benchmark with a recording in progress is 135 MB/s. I think that is plenty fast enough to keep up with a recording.

Then your caching settings might be wrong. Feel free to investigate the memory leak though if you really think there is one, I don't think so though.

@Flole Systems System
Have you got some suggestions on how to proceed and detect the EPG possible "issue" using the core file?
I mean, you didn't reply to my post number 83.

You said:

Select thread 1 again and figure out which thread de->de_thread points to.

And we got:

(gdb) print *dvr_rec_unsubscribe::de->de_thread
$2 = -1652640000
(gdb)

Then select that one and see what that thread is doing (bt full). There should be a dvr_thread if I remember correctly but that doesn't seem to be there in your gdb.

So now, how do we select the thread identified by -1652640000 and see what it is doing?

@Harald K

As we cannot get out with this gdb situation, after reading other tickets more carefully (such as #5579, #5353, #5295 and #4851), I found that we (you) didn't try to enable what Jaroslav suggested:
https://tvheadend.org/issues/5579#note-5

So please edit the TVHeadend start script /share/CACHEDEV7_DATA/.qpkg/TVHeadend/TVHeadend.sh, search for the word administrators (there should be 2 lines, apply to both of them) and edit from this (original):

--user admin --group administrators \

--user admin --group administrators --thrdebug 10020 \

# tvheadend debug start

thread: mutex 0x5562f9f859e0 at src/api/api_idnode.c:134 took 43ms
thread: mutex 0x5562f9f859e0 at src/api/api_idnode.c:134 took 31ms

If I do do TVH runs in the terminal and not in the background. So the terminal can‘t be closed. I‘ll try to get a situation that result in the problem soon.

Now I fear something runs compltetely worng. I had to restart TVH again and very soon after start I get this on the terminal while debugging but I don't get the file mutex-deadlock.txt in /share/CACHEDEV7_DATA/.qpkg/TVHeadend/

REASON: deadlock (src/tvh_thread.c:513)
mutex 0x5556b2587da0 locked in: src/input/mpegts/linuxdvb/linuxdvb_adapter.c:606 (thread 31893)
mutex 0x5556b2587da0   waiting in: src/fsmonitor.c:68 (thread 32157)
mutex 0x5556b2587da0   waiting in: src/service_mapper.c:372 (thread 1079)
mutex 0x5556b2587da0   waiting in: src/notify.c:103 (thread 32097)
2021-04-20 17:13:10.009 [  ALERT] CRASH: Signal: 6 in PRG: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend (4.3-1936~g0046c96d8) [af5409d04097b9014895945c081c51c6f2c4dd66] CWD: /root
2021-04-20 17:13:10.009 [  ALERT] CRASH: Fault address 0x7c95 (N/A)
2021-04-20 17:13:10.009 [  ALERT] CRASH: Loaded libraries: linux-vdso.so.1 /share/CACHEDEV7_DATA/.qpkg/TVH_Dev_Sundtek/sundtek/opt/lib/libmediaclient.so /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libdvbcsa.so.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libssl.so.1.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libcrypto.so.1.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libz.so.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpcre2-8.so.0 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/liburiparser.so.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/no-codexpack/libva.so.2 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/no-codexpack/libva-drm.so.2 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libdl.so.2 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libm.so.6 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/librt.so.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libmvec.so.1 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libstdc++.so.6 /share/CACHEDEV7_DATA/.qpkg/
2021-04-20 17:13:10.009 [  ALERT] CRASH: Register dump [23]: 000000000000000000007fe4eb9e96c00000000000000008000000000000024600007ffd0635de0e00007ffd0635de0f00007fe4eb9ea7000000000000000000000000000000000200007fe4eb9e96c000007fe4eb9e996000000000000000060000000000000000000000000000000000007fe4ebbc77bb00007fe4eb9e96c000007fe4ebbc77bb0000000000000246002b0000000000330000000000000000000000000000000000000000000000000000000000000000
2021-04-20 17:13:10.009 [  ALERT] CRASH: STACKTRACE
2021-04-20 17:13:10.020 [  ALERT] CRASH:  0x5556b0560810 0x5556b02c9000
2021-04-20 17:13:10.023 [  ALERT] CRASH:  0x7fe4ec0bc730 0x7fe4ec0aa000
2021-04-20 17:13:10.023 [  ALERT] CRASH: gsignal+0x10b  (/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
2021-04-20 17:13:10.023 [  ALERT] CRASH: abort+0x121  (/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
2021-04-20 17:13:10.034 [  ALERT] CRASH:  0x5556b050fd08 0x5556b02c9000
2021-04-20 17:13:10.036 [  ALERT] CRASH:  0x5556b050fde5 0x5556b02c9000
2021-04-20 17:13:10.038 [  ALERT] CRASH:  0x5556b050ef09 0x5556b02c9000
/usr/bin/tvheadend: line 746: 31893 Aborted                 /usr/bin/env ${TVH_CMDLINE_PREFIX} LD_LIBRARY_PATH=${QPKG_DIR}/bin/libc:${QPKG_DIR}/bin/libc/no-codexpack ${QPKG_DIR}/bin/tvheadend --config${QPKG_DIR}/config --user admin --group administrators --thrdebug 10020 ${CMDLINE_OPTS} ${TVH_CMDLINE_POSTFIX}
[~] #

Anyway, I have to leave the office now and have to restart my NAS because of other reasons.

Harald K wrote:

If I do do TVH runs in the terminal and not in the background. So the terminal can‘t be closed.

Yep, of course. So edit TVHeadend.sh again, search for the line:

# Interactive mode: output to stderr too
CMDLINE_OPTS="${CMDLINE_OPTS} --stderr"

# Interactive mode: output to stderr too
CMDLINE_OPTS="${CMDLINE_OPTS} --fork"

Harald K wrote:

I had to restart TVH again and very soon after start I get this on the terminal while debugging but I don't get the file mutex-deadlock.txt in /share/CACHEDEV7_DATA/.qpkg/TVHeadend/

According to the sources, you the application has reached tvh_thread.c:513:
https://github.com/tvheadend/tvheadend/blob/1619f9e44678dba5467e4ac94b3e47ea92b72f3e/src/tvh_thread.c#L513
That line calls the function tvh_thread_mutex_failed above which should write the file:
https://github.com/tvheadend/tvheadend/blob/1619f9e44678dba5467e4ac94b3e47ea92b72f3e/src/tvh_thread.c#L491
In fact:
https://github.com/tvheadend/tvheadend/blob/1619f9e44678dba5467e4ac94b3e47ea92b72f3e/src/tvh_thread.c#L447

So the file should be there, I suppose...

Harald K wrote:

I add the logfile in the appendix.

This does not help, as it doesn't show anything about why it crashed. It's very strange that it crash only when debugging.

Post 96 instead was more useful and showed a deadlock at least exist. Why it exists is another matter, unfortunately.

You are absolutely right: Unfortunately.

Last days I searched for alternatives to the ota epg. But its not really successful. There are some ways like XMLTV, WebGrab+Plus and easyepg. The last one is not so bad and free for use. But it's not the same range like ota epg and it's very complicate for installation. I run the first part - the real grabbing - on a linux VM on the NAS and only the second part - connect to the TVH-socet - on the NAS.

Unfortunate that such a great project apparently can not be maintained. But I have this problem now nearly since half a year.

Please excuse my (parhaps stupid) question:
Will something happen here at some point?