Tvheadend

Actually, it enters the tight loop before reporting the disconnect and attempting to reconnect.
So if you set keep alive interval to some large value, disconnecting idle connections from the server side will starve the system.

--trace cccam

how does that work? I tried adding that to the command line, but see no difference.
anyway, here's what I do see (kee alive timer set to 30s):
oscam log:
2018/12/20 01:46:01 41D62F0A c (work) user DD1 reached 120 sec idle limit. <-- at this moment 2 cc threads use 100% cpu each on the tvheadend machine
2018/12/20 01:46:01 611A5704 c (work) user DD3 reached 120 sec idle limit. <-- at this moment 2 cc threads use 100% cpu each on the tvheadend machine
2018/12/20 01:46:35 1B48CA8D c (newcamd) client connected to 33001 port
2018/12/20 01:46:35 187CB094 c (newcamd) client connected to 33004 port

syslog on tvheadend cc client:
Dec 20 01:45:14 tvheadend1804 tvheadend³⁶⁷⁴: mpegts: 12603.75H in ASTRA 192 scan complete
Dec 20 01:45:17 tvheadend1804 tvheadend³⁶⁷⁴: mpegts: 11229V in ASTRA 192 scan complete
<-- no entries here, only after the 30s keep alive timeout do we see more entries:
Dec 20 01:46:32 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33001: Disconnected <-- cpu usage of the 2 threads goes down to 'normal'
Dec 20 01:46:32 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33001: Automatic connection attempt in 2 seconds
Dec 20 01:46:32 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33004: Disconnected
Dec 20 01:46:32 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33004: Automatic connection attempt in 2 seconds
Dec 20 01:46:35 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33004: Attemping to connect to server
Dec 20 01:46:35 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33001: Attemping to connect to server
Dec 20 01:46:35 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33001: Connected as user DD1 to a CryptoWorks ICE-card-00000000 [CAID:0d95] with 4 providers
Dec 20 01:46:35 tvheadend1804 tvheadend³⁶⁷⁴: cwc: 192.168.1.10:33004: Connected as user DD3 to a Viaccess-card-00000000 [CAID:0500] with 6 providers

Also, I'm not sure if it is related, I have a lot of glitches due to discontinuities in AC3 and recently the descrambler is starting to complain about wrong keys every few 100s
Note that I don't have that on the same machine using alternate SW (with the same oscam server)

Please provide full trace of caclient,cwc,descrambler and provide oscam logs. Your snippets and description are not enough.

and how do I do that? I tried --trace cccam on the commandline, that's what I got.

I though that you're using cccam protocol, use the Joe's traces: --trace caclient,cwc,descrambler and specify the log file. Traces are not printed to stdout/stderr: https://tvheadend.org/projects/tvheadend/wiki/Traces

OK, so after reading that piece of info I understand that I need to build tvheadend from source with --enable-trace option configured.

While I'm at it, is there an up to date documentation of all configure options that are actually available and what they do?
I find tvh has a lot of features I don't need/want (in my specific case: iptv, sat ip, transcoding, will only ever run on x86_64 with dvbs2 tuners, etc.).

Also, when did the caclient code change, IIRC it used to work (in the 4.1/4.2 release times?). I don't remember having so many glitches at that time.

./configure --help

I enabled the traces.
enabled only a single CA client.
The system is idle (not doing anything, not even epg scan). This is to show that this does not depend on descrambling, just the tcp socket client implementation is buggy.
oscam closes the idle connection at 02:47:12. cc-newcamd thread spikes to 100% at 02:47:19, stays there til 02:47:50, then reconnencts.

oscam log:
2018/12/22 02:47:19 6BDE7F5C c (work) user DD1 reached 120 sec idle limit.
2018/12/22 02:47:53 6D3B2F94 c (newcamd) client connected to 33002 port
2018/12/22 02:47:53 6D3B2F94 c (client) encrypted newcamd:33002-client 192.168.1.12 granted (DD1, au=on (1 reader))
2018/12/22 02:47:53 6D3B2F94 c (newcamd) user DD1 authenticated successfully (Tvheadend)
2018/12/22 02:47:53 6D3B2F94 c (newcamd) AU enabled for user DD1 on reader SKY

tvheadend log (trace enabled for cwc,caclient,descrambler):
2018-12-22 02:46:06.663 [ DEBUG]:mpegts: 10722.88H in HOT Bird - add raw service
2018-12-22 02:46:38.883 [ DEBUG]:mpegts: 10743.75H in ASTRA 192 - add raw service
2018-12-22 02:47:20.055 [ TRACE]:cwc: 192.168.1.10:33002: sending queued message len 26
2018-12-22 02:47:20.055 [ TRACE]:cwc: 00 18 FE F2 B2 4A E4 C3 8A A7 5E 3E 6F 23 3B 46 .....J....^>o#;F
2018-12-22 02:47:20.055 [ TRACE]:cwc: F4 25 5D 5F 3C B0 04 85 A1 D4 .%]_<.....
2018-12-22 02:47:50.031 [ TRACE]:cwc: 192.168.1.10:33002: sending queued message len 26
2018-12-22 02:47:50.031 [ TRACE]:cwc: 00 18 D9 C5 AC 62 29 5A CA 7D 89 6E 0E B1 53 C6 .....b)Z.}.n..S.
2018-12-22 02:47:50.031 [ TRACE]:cwc: 4F 79 F7 AF 04 26 A2 9D 59 30 Oy...&..Y0
2018-12-22 02:47:50.031 [ DEBUG]:cwc: 192.168.1.10:33002: session exiting
2018-12-22 02:47:50.031 [ INFO]:cwc: 192.168.1.10:33002: Disconnected
2018-12-22 02:47:50.031 [ INFO]:cwc: 192.168.1.10:33002: Automatic connection attempt in 2 seconds
2018-12-22 02:47:53.031 [ INFO]:cwc: 192.168.1.10:33002: Attemping to connect to server
2018-12-22 02:47:53.032 [ DEBUG]:cwc: 192.168.1.10:33002: Connected

Since the descrambling issues I have might depend on this, I'll till this gets resolved and then open a separate issue if required.

Joe User wrote:

Please provide full trace of caclient,cwc,descrambler and provide oscam logs. Your snippets and description are not enough.

Recheck your CA settings and your oscam settings? Maybe post them (blocking out passwords, etc).
What device is your oscam running on? Is it used for something else?
You can try running the following on your tvheadend machine and/or your oscam machine:

tcpdump -n port nnnn

While that fixes the hanging due to oscam closing the connection, we still do not know why the keepalives are not working...

Right. Thanks. The second commit should fix this.