Bug #5345
too much queued table input data (over 2MB)
0%
Description
tvh is crashing multiple times everyday with errors below:
Nov 23 10:55:15 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #16 : DVB-T #0, discarding new
Nov 23 10:55:25 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #16 : DVB-T #0, discarding new
Nov 23 10:55:26 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6280 DVB-T/T2/C #2 : DVB-T #0, discarding new
Nov 23 10:55:26 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6280 DVB-T/T2/C #1 : DVB-T #0, discarding new
Nov 23 10:55:27 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #17 : DVB-T #0, discarding new
Nov 23 10:55:29 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #15 : DVB-T #0, discarding new
Nov 23 10:55:33 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6909 DVB-S/S2 #13 : DVB-S #0, discarding new
Nov 23 10:55:35 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #16 : DVB-T #0, discarding new
Nov 23 10:55:36 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6280 DVB-T/T2/C #2 : DVB-T #0, discarding new
Nov 23 10:55:36 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6280 DVB-T/T2/C #1 : DVB-T #0, discarding new
Nov 23 10:55:37 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #17 : DVB-T #0, discarding new
Nov 23 10:55:39 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #15 : DVB-T #0, discarding new
Nov 23 10:55:43 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6909 DVB-S/S2 #13 : DVB-S #0, discarding new
Nov 23 10:55:45 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #16 : DVB-T #0, discarding new
Nov 23 10:55:46 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6280 DVB-T/T2/C #2 : DVB-T #0, discarding new
Nov 23 10:55:46 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6280 DVB-T/T2/C #1 : DVB-T #0, discarding new
Nov 23 10:55:47 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #17 : DVB-T #0, discarding new
Nov 23 10:55:49 TVheadend tvheadend6348: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6205 DVB-T/T2/C #15 : DVB-T #0, discarding new
I'm experiencing this behaviour since the day one I've started using the TVheadend, however, that wasn't that big deal (crashed once a month or so) until I started recording most of the terrestrial channels. Worth noticing that I always used bleeding edge versions, 4.1-4.3
Despite I don't do recordings on TurboSight TBS 6909 (36.0 east) card but it's included in the log as well.
This is the first bug I'm logging therefore not sure what additional information should I provide.
If anything additional is needed, I'd be more than happy to provide it.
I'm using Ubuntu 16.04 LTS with open source TBS drivers.
CPU: i7-4770T
RAM: 16GB DDR3
Cards in the same server:
DVB-S:
TBS 6902
TBS 6909
DVB-T:
TBS 6205
TBS 6280
TBS 6284
Files
History
Updated by Luis Alves almost 6 years ago
Updated by Usually that message happens when tvheadend is "overloaded" and can't handle all the incomming data.
Some questions:
- How many channels recording simultaneously?
- What streaming profiles are in use? Are they transcoding?
- What is the cpu usage when that happens? (upload a capture of top/htop of tvh).
By the way, logs are most useful when they include the transition from "good" to "bad" (the moment the issue occurs) although not sure if in this case will help.
Also, have you tried with a build from the latest github master?
Updated by Luis Alves almost 6 years ago
Jaroslav just added lock detection code:
Try latest, add '--thrdebug 1' to the tvh command line arguments. Show 'CFGDIR/mutex-deadlock.txt' contents after the crash.
Updated by Auri Cool almost 6 years ago
Updated by Luis Alves wrote:
Jaroslav just added lock detection code:
Try latest, add '--thrdebug 1' to the tvh command line arguments. Show 'CFGDIR/mutex-deadlock.txt' contents after the crash.
My version of tvh is installed via apt-get. Can I somehow add that argument to the init.d script? If positive - how?
Or can the same be achieved via GUI?
Updated by Jaroslav Kysela almost 6 years ago
Updated by Add this to '/etc/default/tvheadend' configuration file (OPTIONS). No GUI - sorry - this debug settings must be activated before the HTTP server is started.
Updated by Auri Cool almost 6 years ago
Jaroslav Kysela wrote:
Add this to '/etc/default/tvheadend' configuration file (OPTIONS). No GUI - sorry - this debug settings must be activated before the HTTP server is started.
Thanks for the hint! no probs, I've already ran it via CMD. After upgrading tvh to the most recent version it begun crashing very frequently (several times per hour), however, once I managed to run it using command line with '--thrdebug 1' arg it works for an hour or so so far.
Should mutex-deadlock.txt be created regardless of crashes or just after a crash? Currently can't locate such file in /home/hts/.hts/tvheadend folder
Updated by Luis Alves almost 6 years ago
Auri Cool wrote:
Should mutex-deadlock.txt be created regardless of crashes or just after a crash?
Only after a crash.
Updated by
Updated by Auri Cool almost 6 years ago
- File tvhHTOP.png tvhHTOP.png added
tvh just hangs and nothing is happening, just this time there's no "too much queued table input data (over 2MB)" message at log.
No mutex-deadlock.txt is being generated.
All CPU cores are hitting 100%
Updated by Pablo R. almost 6 years ago
Auri Cool wrote:
tvh just hangs and nothing is happening, just this time there's no "too much queued table input data (over 2MB)" message at log.
No mutex-deadlock.txt is being generated.
All CPU cores are hitting 100%
Could you show top -H to see what Thread is consumming this CPU?
Updated by Mark Clarkstone almost 6 years ago
Updated by Auri Cool wrote:
tvh just hangs and nothing is happening, just this time there's no "too much queued table input data (over 2MB)" message at log.
No mutex-deadlock.txt is being generated.
All CPU cores are hitting 100%
:o perhaps attaching gdb and getting a backtrace might show something?
Updated by Auri Cool almost 6 years ago
ok, so after upgrading to the latest version and installing dbg version the pattern has changed.
I don't have "too much queued..." crashes anymore, however, tvh still crashes but not that frequently as before.
Just had another crash few moments ago with these lines:
Dec 06 11:21:26 TVheadend tvheadend30226: CRASH: Signal: 6 in PRG: /usr/bin/tvheadend (4.3-1616~g39b74cb) [6ceb4d53ead84c7f11362434a6daf19e5798ab30] CWD: /
Dec 06 11:21:26 TVheadend tvheadend30226: CRASH: Fault address 0x6f00007612 (N/A)
Dec 06 11:21:26 TVheadend tvheadend30226: CRASH: Loaded libraries: /usr/lib/x86_64-linux-gnu/libdvbcsa.so.1 /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 /usr/lib/x86_64-linux-gnu/liburiparser.so.1 /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/librt.so.1 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libsystemd.so.0 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/libselinux.so.1 /lib/x86_64-linux-gnu/liblzma.so.5 /lib/x86_64-linux-gnu/libgcrypt.so.20 /lib/x86_64-linux-gnu/libpcre.so.3 /lib/x86_64-linux-gnu/libgpg-error.so.0 /lib/x86_64-linux-gnu/libnss_compat.so.2 /lib/x86_64-linux-gnu/libnsl.so.1 /lib/x86_64-linux-gnu/libnss_nis.so.2 /lib/x86_64-linux-g
Prior installing the dbg package and after one of the crashes i checked "top -H" as asked by Pablo R.
CPU usage was very high and it was related to descrambling. Sorry, don't have a proof (screenshot).
I guess there is really something with tvh and oscam relationship. I had issues with dvbapi since the day one of tvh usage.
If dvbapi is enabled - tvh crashes very frequently, if mgcamd is used crashes wasn't that noticable until I started recording multiple channels.
Updated by Pablo R. almost 6 years ago
Auri Cool wrote:
[...]
Prior installing the dbg package and after one of the crashes i checked "top -H" as asked by Pablo R.
CPU usage was very high and it was related to descrambling. Sorry, don't have a proof (screenshot).I guess there is really something with tvh and oscam relationship. I had issues with dvbapi since the day one of tvh usage.
If dvbapi is enabled - tvh crashes very frequently, if mgcamd is used crashes wasn't that noticable until I started recording multiple channels.
It is important to know that. The problem now would be to find the part of code...
Did you enable '--thrdebug 1' at command line? So we can see the file generated in the tvheadend dir in relation to the crash.
Updated by Luis Alves almost 6 years ago
Auri Cool wrote:
I guess there is really something with tvh and oscam relationship. I had issues with dvbapi since the day one of tvh usage.
If dvbapi is enabled - tvh crashes very frequently, if mgcamd is used crashes wasn't that noticable until I started recording multiple channels.
Try using the cccam client...
Updated by Ricardo Rocha almost 6 years ago
Updated by luis Parada Alves it's also with problems as you can see on #5099
Updated by Auri Cool almost 6 years ago
- File tvh_ConfDir_ls.png tvh_ConfDir_ls.png added
- File tvhHTOP_w_thrdebug1.png tvhHTOP_w_thrdebug1.png added
Mark Clarkstone, I'll read more about gdb and will try to provide the log in more detail.
Pablo R, yes, I have added '--thrdebug 1' to '/etc/default/tvheadend' as Jaroslav Kysela instructed, but there is no 'mutex-deadlock.txt' file in there (see attached). Maybe the file should be somewhere else?
htop shows that it's definitely running with '--thrdebug 1' arguments (attached).
Luis Alves, will try using cccam client and will report the outcome.
Updated by Ricardo Rocha almost 6 years ago
@Auri Cool use also --thrdebug 10020 as show on https://tvheadend.org/projects/tvheadend/wiki/Debugging#Mutex-profiling
Updated by Pablo R. almost 6 years ago
Ricardo Rocha wrote:
@Auri Cool use also --thrdebug 10020 as show on https://tvheadend.org/projects/tvheadend/wiki/Debugging#Mutex-profiling
Yes, try that!
Updated by Auri Cool almost 6 years ago
Apologies for such a late response!
So I think I managed to solve this particular issue.
I forgot to mention that I had installed additional quad DVB-S card and removed it. After that adapter's configuration got messed up and I reconfigured it, then big fun begun.
So what I did is I went ahead and deleted everything from adapters config folder and reconfigured it.
So far so good, two weeks without any crash related to the bug title.
Updated by Victor S over 4 years ago
Updated by Auri Cool wrote:
Apologies for such a late response!
So I think I managed to solve this particular issue.
I forgot to mention that I had installed additional quad DVB-S card and removed it. After that adapter's configuration got messed up and I reconfigured it, then big fun begun.
So what I did is I went ahead and deleted everything from adapters config folder and reconfigured it.So far so good, two weeks without any crash related to the bug title.
No, it is not a solution. Tvheadend gets jammed with same messages with multi tuners even in 4.3 beta version. Devs are simply don't care.
Updated by Flole Systems over 4 years ago
Updated by 
- Status changed from New to Fixed
The original issue has been solved. If the same solution doesn't work for you the issue you are facing is not the same.