Tvheadend

Thanks

running in place, afbaf935d6 is still affected with the heavy cpu load.
I get the feel;ing that the new config script is somehow responsible for this.
Going to check.

I can't seem to build 6d260fc8b7 or 841ff8eb49.
It ends in errors at build.linux/bundle.c.
make: *** [build.linux/bundle.o] Error 1

However, I can build afbaf935d6 and confirm that afbaf935d6 has the same problems with high load.

841ff8eb49 is the one with the new script.

commit 9e53683ca0 is not affected (just before the script)

a259025d31 is the first after the script that will build ok on my machine.
It suffers fropm the heavy cpuload.
So I think the new script and/or the makefile (which was tidied up) are responsible for the trouble

bas t wrote:

commit 9e53683ca0 is not affected (just before the script)

Ok, and since it is hard to see how the zlib thing can make a difference (making it optional), it is probably in 841ff8eb49.

bas t wrote:

a259025d31 is the first after the script that will build ok on my machine.
It suffers fropm the heavy cpuload.
So I think the new script and/or the makefile (which was tidied up) are responsible for the trouble

Yeah, indeed, the problem occurs after that.
I will look through the code, but i am affraid i might not be of any help there.....

Does ffdecsa use zlib?

If it does, a259025d31 or 6d260fc8b7 may be the ones we are after

Alex. _ wrote:

Ok, and since it is hard to see how the zlib thing can make a difference (making it optional), it is probably in 841ff8eb49.

Missed that, you're right

Zlib is only used for the UI and some config files (if you enable file bundling as well). It should have ZERO impact on the main code.

Also I think looking at the configure script is way off track same goes for Makefile. Unless there is a clear diff in the CC flags being passed that mean I've cocked up some optimisation etc...

But I'm open to being proven wrong!

Adam

Adam Sutton wrote:

Also I think looking at the configure script is way off track same goes for Makefile. Unless there is a clear diff in the CC flags being passed that mean I've cocked up some optimisation etc...

Some slight differences, but nothing that is apparently strange to me:

New: .config.ml

# Automatically generated by configure - DO NOT EDIT!
ROOTDIR ?= .
BUILDDIR ?= ./build.linux
OSENV ?= posix
ARCH ?= x86_64
ifeq ($(origin CC),default)
CC = cc
endif
PYTHON ?= python
CFLAGS +=
LDFLAGS +=
prefix = /opt/tvheadend
bindir = ${prefix}/bin
mandir = ${datadir}/man
datadir = ${prefix}/share
libdir = ${prefix}/lib
LDFLAGS += -lssl -lcrypto
CFLAGS +=
LDFLAGS += -lz
CFLAGS +=
LDFLAGS += -lavahi-common -lavahi-client
CFLAGS += -D_REENTRANT
CONFIG_CWC = yes
CONFIG_V4L = yes
CONFIG_LINUXDVB = yes
CONFIG_DVBSCAN = yes
CONFIG_AVAHI = yes
CONFIG_ZLIB = yes
CONFIG_BUNDLE = no
CONFIG_EXECINFO = yes
CONFIG_MMX = yes
CONFIG_SSE2 = yes
CONFIG_PYGZIP = yes
CONFIG_SSL = yes


Old .config.mk:

# Automatically generated by configure - DO NOT EDIT!
ROOTDIR ?= .
BUILDDIR ?= ./build.linux
OSENV ?= posix
CFLAGS +=
LDFLAGS +=
LDFLAGS += -lssl -lcrypto
CFLAGS +=
LDFLAGS += -lz
CFLAGS +=
LDFLAGS += -lavahi-common -lavahi-client
CFLAGS += -D_REENTRANT
CONFIG_CWC = yes
CONFIG_V4L = yes
CONFIG_LINUXDVB = yes
CONFIG_DVBSCAN = yes
CONFIG_AVAHI = yes
CONFIG_ZLIB = yes
CONFIG_EXECINFO = yes
CONFIG_MMX = yes
CONFIG_SSE2 = yes
CONFIG_SSL = yes


@addamsutton:

Ruling out the zlib stuff, only 5 files have changed between 9e53683ca0 and a259025d31

That would be: Makefile, configure, support/configure.inc, src/filebundle.c, src/main.c

So where do you suggest to look?

Just a wild thought:

Could it be that this got messed up?

(from https://github.com/tvheadend/tvheadend/pull/54)

"tvheadend is 'cheating' a bit (in a very clever way) by selecting optimal settings at runtime (which the original FFdeCSA did at build time)"

Hi,

I rechecked and I am now also not sure about 9e53683ca0 (Where "Updated so that we only need a single binary to cover 3 possible use cases" )
It does seem to give high cpu loads.
Also, I can select that branch, however, if I then do:
git reset --hard HEAD

i get:

cd src-9e53683ca0/
src-9e53683ca0 # git reset --hard HEAD
HEAD is now at 00fa2f4 Add some additional support to configure.inc added for new feature check in configure that has now been abandoned.

Which seems a different thing. Also , if I install that one it give the high cpu load.

So, the last working one I now seem to get for sure is b350d44c4d .

( but please verify on your system, since I am confused about the numbering right now).

Alex.

I am sure that commit 9e53683ca0 works without the heavy cpu load on my machine

You could check again with a freshly cloned repo.

git reset --hard HEAD should give you the right commit as head (9e53683ca0 + some info)

bas t wrote:

Just a wild thought:

Could it be that this got messed up?

(from https://github.com/tvheadend/tvheadend/pull/54)

"tvheadend is 'cheating' a bit (in a very clever way) by selecting optimal settings at runtime (which the original FFdeCSA did at build time)"

Mmm, I see what it says, but I am not sure if I can understand why this made the change at that specific point in the commits?

Are you suggesting that actually FFdecsa is no longer used after that specific point?

No, I am suggesting that the optimal settings are no longer made at runtime.

If not using optimal settings, it is very easy to understand that the cpu load goes way up...

bas t wrote:

No, I am suggesting that the optimal settings are no longer made at runtime.

If not using optimal settings, it is very easy to understand that the cpu load goes way up...

Ok, i understand.
I tried to find something specific by looking in the code but can't seem to find any specific lines that adres the optimalisation.
Every mentioning of FFdeCSA or CWC seem very general, for example in the make-file.

I take the wild thought back

While starting tvheadend, the log shows: CSA: Using SSE2 128bit parallel descrambling

which is indeed the optimal setting for my system

EDIT:

I'm not thinking straight, I'm running the unaffected commit.

Tomorrow I'll see what happens when I run an affected commit

One other thing surprises me :
The increase in CPU load is very high ( like 5 or even 6 times as high as it should be, from 10% or less to 60% or more on my system) , so, this really shouldn't escape one's attention.
However: no-one else is mentioning this, which makes me think we might just have some 'bad luck' combination of stuff running...

So (questions to everyone):

- Is anyone else experiencing this high cpu load? ( Or, might this be something very specific to our two systems that (by chance) causes this on our systems?)

- Does anyone run the newest version of tvheadend and experiences a low CPU load when watching encrypted channels? (useful for us to know, since then it might be very specific to our setup)

- What is your cpu load on a GIT tvheadend system from after mid August when watching encrypted channels?

Thanks,

Alex.

PS:
To give some details about my system:

Gentoo AMD 64
dvbhdhomerun tuners
OSCAM as a card server

EDIT:
I'm not thinking straight, I'm running the unaffected commit.
Tomorrow I'll see what happens when I run an affected commit

Lol, yeah, might be after all this time trying ;-)

However, I will not be able to try much tomorrow ( mondays, mondays....)

bas t wrote:

While starting tvheadend, the log shows: CSA: Using SSE2 128bit parallel descrambling

which is indeed the optimal setting for my system

The log still shows "CSA: Using SSE2 128bit parallel descrambling" while running a259025d31

Since I'm not a skilled coder, I'm out of options right now.

I gave libdvbcsa a chance. Patched master for it (see https://github.com/bas-t/tvheadend/commits/dvbcsa)
compiled libdvbcsa and tvheadend and fired it up.

As expected, the result wasn't as good as compiling from 9e53683ca0, but still remarkable.

Using the unpatched master branch, cpuload was still over 3 times up! So using libdvbcsa gives an excelent performance,
compared with unpatched master.

I'm not suggesting to drop ffdecsa in favour of libdvbcsa, I was just asking myself: what would happen if I use lidvbcsa?

Now I know.

Yeah the PR has mostly been ignored so far as we've had no input since John and I joined the project.

Plus its not something either of us are really very familiar with.

I'm still surprised there should be a significant change in basic performance for encrypted channels since 2.12/3.0 as nothing has been changed (that I know of) that would affect it.

Adam

Adam Sutton wrote:

nothing has been changed (that I know of) that would affect it.

Well, that's a big bummer. The build-in ffdecsa (plus not using mysql) is the major advantage of using tvheadend over mythtv + sasc-ng.

Sadly I only can tell in whitch commits it happened. Not which part of the commits.

As it is, tvheadend's master branch is pretty much useless to me, since allmost all channels are encrypted where I live.

All future releases would be useless too, untill someone drops in and fixes it.