Tvheadend

Provide output using '--trace satip,httpc' . See wiki for how to do traces.

No, the traces were not enabled - there should be TRACE lines in the log (and it will be much bigger). https://tvheadend.org/projects/tvheadend/wiki/Traces

Not really. The traces for 'httpc' and 'satip' subsystems are missing.

Please, add also 'mpegts' to traces. When streaming stops, what does the status panel show ? Is there an input bandwidth detected ?

I don't see any issue in the log. The data are received all time. Have you tried another client than VLC (mplayer, kodi etc.?).

Also, you may try to use matroska profile.

And lastly, you can use the 'play' link URL for wget and save the received data for further analysis.

I am also having the same issues with the 4sat with regards to tuning and dropping out after 60 seconds

I have had a quick skim over the sat > ip specification and interestingly the below stands out.

3.5.4 Maintaining a session (OPTIONS)
Once an RTSP session has been setup, the client needs to maintain the RTSP session by sending an RTSP
request before the end of each timeout period (60 seconds by default). A SAT>IP server may define a
different timeout period in the Session Header of the response to a SETUP message.
Through the receipt of OPTIONS messages from clients, the server knows about the presence or absence of
each RTSP client.
Clients need to send an OPTIONS message at the latest a few seconds before each expiration period. As
OPTIONS messages are delivered using reliable TCP transport, there is no need to send excess OPTIONS
messages

I did get a little further. It seems to get further when I tried to run the PLAY command asynchronously, not waiting for the response of the PLAY command.
I even got it that far that it sent the OPTIONS message once. Unfortunately the code is really hard to understand. I don't really know how to do this properly. Hopefully someone can do this for us as I would really like to get this box running.

I have dug into this a bit... it seems that there is a issue with the queing of http requests which prevents any more requests being sent,

It seems that when the play command is being issued it is breaking the request que to the 4sat. must be something like keepalive or whatever which the 4sat is returning, this is in turn preventing the periodic options request from being sent which causes the 4sat to stop streaming data after 60 seconds

Looking at this on branch release/4.0

Looking into this further I have found the following so far from stepping through

The rtsp_send function does not send the Connection: close header.
The 4sat will keep the http connection open and tvheadends http/rstp que thingy will not send any further requests to the 4sat.

rstp.c

//The Elgato Netstream 4sat expects there to be a Connection: close and this is also in all of the Sat > IP Documentation example requests
if (hdr == NULL) {
hdr = &h;
http_arg_init(&h);

http_arg_set(hdr, "Connection", "close");
}

if (hc->hc_rtsp_session) {
http_arg_set(hdr, "Session", hc->hc_rtsp_session);
}

I have added some additional logging here aswell to get an idea of whats going on

tvherror("rtsp", "%s : Sending command %s rtsp://%s%s%s", hc->hc_rtsp_session, http_cmd2str(cmd), hc->hc_host, buf2, path ? path : "/");

and it is now sending the options request but it is absolutely spanking it out and then it is still stopping streaming.

looks like when the session is being kept alive some event / property is not set.. I have not yet checked to see if options is going over the wire

It sounded really interesting, but unfortunately it didn't take me any further either. I did check via wireshark though and found out that the OPTIONS messages are not being send over the wire.

I have now ran a wire trace on the above and it seems to me that the options message is not being sent allthough it is getting hit in the code.. in tvfrontend.c

Again i think this has something to do with the implementation of the queing of http / rtsp requests.

You are right about this. I have seen that after the play command goes out there seem to be another http message being sent which is a HEAD command. I only saw this from this from further log output that I added myself to httpc.c. I have not seen this on the wire either. However it returns with HTTP_CON_RECEIVING and that does not get any special handling in satip_frontend.c. The function http_client_send only sends out anything if hc->hc_wqueue is empty. I have seen this preventing the options to go out during my tests.

From looking into this further I have found the following is happening

Out of the box the following is happening

The Setup command is sent and the connection is kept open
Response is sent ok

The Play command is sent and the connection is kept open
Response is sent ok

Something blocks the connection from sending further data

60 seconds happens

The Options Command is not sent.

I have locally added the connection close headers and this breaks the current stuff because the code expects the connection to be kept open so it just keeps sending the setup command. the examples in the sat ip spec v1.2 all state connection close in there examples but I have not read into it deep enougth allthough the following is of interest

RTSP and TCP
The RTSP control channel in SAT>IP must be implemented using TCP. It is important to note that RTSP
sessions are independent of the underlying TCP connections.
An RTSP session generally spans multiple TCP connections (e.g. one TCP connection per
request-response). It is however also possible for an RTSP session to consist of a single TCP connection
(persistent mode) including several RTSP request-response pairs.

So I think there are two ways of going about this

Change so that all of the RTSP commands are sent indivually with close, at the moment some of the variables are in the connection object and when the connection is closed the state is sent to shutdown.. changing the hc_shutdown state is not enougth

Fix so that the que stuff is working properly..

From timo's comment from what I have seen is that stuff will be sent immediatly when the hc->hc_wque is empty, when stuff is sent it should remove itself from the que and then send the next item.

Tracing this through further (default)

Setup command is sent
Play command is sent
Something causes the connection to fault.. in satip_frontend so the connection is closed
Connection enters closed state
Options never happens but playing continues for 60 seconds

Basically the connection is being closed in the input thread or by the 4sat..

I really think that the satip_frontend should allow connection close rather than its current I love connections mantra.
as code for this is shared between this and the iptv this will also be affected.

You can login as root to the 4sat via telnet and SSH and then run /sbin/logread -f There you can see that it seems to be happy and finally expires the connection after 60 seconds. There is also a status html page which shows the current clients.

Ok, I have it running now. It is not a nice solution, but it does work. Here is what I did:

diff --git a/src/httpc.c b/src/httpc.c
index f9dd15a..5ade9fc 100644
--- a/src/httpc.c
++ b/src/httpc.c
@ -525,6 +525,9 @ skip:
wcmd->wpos += r;
if (wcmd->wpos >= wcmd->wsize) {
res = HTTP_CON_SENT;
if (hc->hc_cmd 8 || hc->hc_cmd 5) {
+ http_client_cmd_destroy(hc, wcmd);
+ }
wcmd = NULL;
}
break;
@ -915,8 +918,13 @ retry:
if (r < 0) {
if (errno EIO && hc->hc_in_data && !hc->hc_keepalive)
return http_client_finish(hc);
- if (ERRNO_AGAIN(errno))
- return HTTP_CON_RECEIVING;
+ if (ERRNO_AGAIN(errno)) {
+ if (hc->hc_cmd 8 || hc->hc_cmd == 5) {
+ return HTTP_CON_DONE;
+ } else {
+ return HTTP_CON_RECEIVING;
+ }
+ }
return http_client_flush(hc, errno);
}
if (r > 0 && tvhtrace_enabled()) {
diff --git a/src/rtsp.c b/src/rtsp.c
index 4f706f9..21a7014 100644
-- a/src/rtsp.c
+++ b/src/rtsp.c
@ -81,7 +81,9 @ rtsp_options_decode( http_client_t *hc )

p = http_arg_get(&hc->hc_args, "Public");
   if (p == NULL)
-    return -EIO;
+    tvhtrace("rtsp", "rtsp_options_decode: Returning EIO");
+    //return -EIO;
+    return HTTP_CON_OK;
   n = http_tokenize(p, argv, 32, ',');
   for (i = 1; i < n; i++) {
     if (strcmp(argv[i], "DESCRIBE") == 0)

I made a fork on github and put my changes in branch feature/elgato_4sat

https://github.com/twendt/tvheadend/tree/feature/elgato_4sat

The changes in rtsp are really ugly though.

I have scanning on your branch which is somewhat flakey. I will try playback tomorow.

When scanning I basically get 2 setup commands sent and one play in between then the connection is halting

SETUP rtsp://192.168.1.5/?src=1&fe=1&freq=12460&sr=27500&msys=dvbs&mtype=qpsk&pol=h&fec=23&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=5000-5001
CSeq: 1

RTSP/1.0 200 OK
CSeq:1
Transport:RTP/AVP;unicast;destination=192.168.1.15;client_port=5000-5001
com.ses.streamID:11
Session:17B0AB2E;timeout=60
Content-Length:0

PLAY rtsp://192.168.1.5/stream=11?addpids=0,1,16,17,3002,3003 RTSP/1.0
Session: 17B0AB2E
CSeq: 2

RTSP/1.0 200 OK
CSeq:2
RTP-Info:url=rtsp://192.168.1.5/stream=11;
Session:17B0AB2E
Content-Length:0

SETUP rtsp://192.168.1.5/stream=11?src=1&fe=1&freq=10744&sr=22000&msys=dvbs&mtype=qpsk&pol=h&fec=56&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=5000-5001
Session: 17B0AB2E
CSeq: 3

RTSP/1.0 200 OK
CSeq:3
Transport:RTP/AVP;unicast;destination=192.168.1.15;client_port=5000-5001
com.ses.streamID:11
Session:17B0AB2E
Content-Length:0

NO MORE TRAFFIC BUT CONNECTION OPEN

I will see what happens when i play.

Note i was working of 4.0 branch rather than master.

The above works for me...

It now plays for more than 60 seconds.. which is a good start.. Changing channel's doesn't really work for me and there are still issues with tuning!

I think the issues relating to the tuning and the changing of channels relate the teardown command not being sent and then the thing brought back up with setup then play.

This means that the following will happen

Client sends setup then play
Plays happily

Wants to change or tune
No teardown etc

Client sends setup
4Sat returns 503 unavailable/

(tuning)
Tuning stuff moves onto next channel rather than waiting for the source to become available
4sat returns 503

I have a feeling that when the connection is kept in open state that the 4sat will tear down playback after 60 seconds but will report 503 until the connection is freed / teardown has happened for the appropriate stream.

I have looked into the scanning and made the following change to your stuff

httpc line 526

if (wcmd->wpos >= wcmd->wsize) {
res = HTTP_CON_SENT;
if (strcmp(hc->hc_scheme, "rtsp")) {
tvherror("httpc", "Command Destroyed");
http_client_cmd_destroy(hc, wcmd);
}
wcmd = NULL;
}
break;

I am now able to scan for channels and from the trace I get the following happening which is good

I still may have issues with changing channels.. but I should hopefully have a full channel list and bouquets when the scan finishes

Will keep you posted

Trace

SETUP rtsp://192.168.1.5/?src=1&fe=1&freq=12012&sr=27500&msys=dvbs&mtype=qpsk&pol=v&fec=23&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=6000-6001
CSeq: 1

RTSP/1.0 200 OK
CSeq:1
Transport:RTP/AVP;unicast;destination=192.168.1.106;client_port=6000-6001
com.ses.streamID:27
Session:12D3D816;timeout=60
Content-Length:0

PLAY rtsp://192.168.1.5/stream=27?addpids=0,1,16,17,3002,3003 RTSP/1.0
Session: 12D3D816
CSeq: 2

RTSP/1.0 200 OK
CSeq:2
RTP-Info:url=rtsp://192.168.1.5/stream=27;
Session:12D3D816
Content-Length:0

SETUP rtsp://192.168.1.5/stream=27?src=1&fe=1&freq=12246&sr=27500&msys=dvbs&mtype=qpsk&pol=v&fec=23&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=6000-6001
Session: 12D3D816
CSeq: 3

RTSP/1.0 200 OK
CSeq:3
Transport:RTP/AVP;unicast;destination=192.168.1.106;client_port=6000-6001
com.ses.streamID:27
Session:12D3D816
Content-Length:0

SETUP rtsp://192.168.1.5/stream=27?src=1&fe=1&freq=11954&sr=27500&msys=dvbs&mtype=qpsk&pol=h&fec=23&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=6000-6001
Session: 12D3D816
CSeq: 4

RTSP/1.0 200 OK
CSeq:4
Transport:RTP/AVP;unicast;destination=192.168.1.106;client_port=6000-6001
com.ses.streamID:27
Session:12D3D816
Content-Length:0

SETUP rtsp://192.168.1.5/stream=27?src=1&fe=1&freq=11527&sr=27500&msys=dvbs&mtype=qpsk&pol=v&fec=23&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=6000-6001
Session: 12D3D816
CSeq: 5

RTSP/1.0 200 OK
CSeq:5
Transport:RTP/AVP;unicast;destination=192.168.1.106;client_port=6000-6001
com.ses.streamID:27
Session:12D3D816
Content-Length:0

SETUP rtsp://192.168.1.5/stream=27?src=1&fe=1&freq=10803&sr=22000&msys=dvbs&mtype=qpsk&pol=h&fec=56&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=6000-6001
Session: 12D3D816
CSeq: 6

RTSP/1.0 200 OK
CSeq:6
Transport:RTP/AVP;unicast;destination=192.168.1.106;client_port=6000-6001
com.ses.streamID:27
Session:12D3D816
Content-Length:0

SETUP rtsp://192.168.1.5/stream=27?src=1&fe=1&freq=10818&sr=22000&msys=dvbs&mtype=qpsk&pol=v&fec=56&ro=0.35 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=6000-6001
Session: 12D3D816
CSeq: 7

Note that the above is only using one source... I will try this again later with 2 of the tuners! I currently only have 2 cables running from my LNB so will not be able to test tuning using all 4 sources at once!

So scanning has completed. Seems ok but has not picked up all of the channels.. has added the additional muxes but does not seem to of picked up the freesat bouquets.

I am going to run a second scan with 2 sources enabled.

Will keep posted.

Note that I got the expected teardown command 200 and then connection closed at the end of the scan

So scanning with 2 sources enabled seems to be working and this time through it has picked up the Freesat bouquets.. It is still running through.. the scanning in tvheadend is a bit slower than dvblink.

Hopefully when this works I can stop using dvblink as it continually crashes on the rpi2 and the distro support and support for it is diabolical.

Once scan is complete I am going to set up local freesat bouquet and connect to xbmc. and try channel hopping etc/.

scanned through... did not find freesat bouqets unfortunatly but sky ones where there have selected thames valley and channels have mapped

That sounds very good even though not everything us running yet.
I also has issues after I posted last night. I usually use tvheadend in a Docker container on a different server. I have build a deb package from the code and that was installed in a new container. That didn't work at all.
Switching channels was also an issue for me. I will try your change tonight.
I am still thinking about your first approach using the connection close header. The reason that didn't work was because the variable hc->hc_shutdown was set to 1 somewhere and that caused the request not being sent out and return with EIO instead. I have not seen though where the shutdown was set and what else was done there. Maybe it can be reset to 0 in certain situations and then it works.
I am also not sure why it was not able to larse the header if the reply to the PLAY message. That was the reason why I had to do that dirty hack. It tried to read the Public header and could not do that. Maybe this is because we are destroying the command to early.
We are making progress and that is good.

Ok so with my tweak I am able to change channels etc now and I no longer get the dreadded 503 service unavailable...

I will send you a pull or similar or add to my own branch..

Unfortunately I have not yet managed to get the correct bouquet for freesat.

I have not managed to get the freesat bouquets...

With the setting connection closed I found it was a bit of a pig.. in reality the sat > ip specification and rtsp spec are a bit loose on this..

I tried pulling out the code for the rtsp state so that it is seperate from the http_client stuff and was most of the way there..

Let me know how you get on... I have lost a week of my life obessed with getting this running.

It works with your change for me as well.
I even got it running in my Docker container now.
I wonder if someone with another SATIP box would be able to test with this code as well. Otherwise it would be hard to get this upstream. Maybe Jaroslav could have a look at our code.

This seems to be working but I am still having a few issues with Freesat tuning / bouquets..

When scanning a lot of services with blank ID's are picked up.

Does not allways find the freesat bouquets
When automapping channels from freesat bouquet nothing happens.

but most importantly.

When checking availability of channels in the mapper it seems like some teardown etc over the rtsp connection may still not be happening

Timo Wendt wrote:

It works with your change for me as well.
I even got it running in my Docker container now.
I wonder if someone with another SATIP box would be able to test with this code as well. Otherwise it would be hard to get this upstream. Maybe Jaroslav could have a look at our code.

I think that Jaroslav or someone else will need to realistically look at the httpc and rtsp implementation.

As you have said the code above is a little bit hacky and in reality the sending of data on a connection thats kept alive needs to be looked into ie can a connection be kept alive and multiple requests sent successfully. likewise the Sat > IP spec is a bit weak with regards to the connection keep alive but basically the rtsp connection can be closed after each request. The current implementation expects the connection to be kept open at all times.. a faulty implementation may not care if there is no Connection: close in the headers and close the connection after each command thus causing tuner x with this fault to not work in tvheadend.

Likewise I think it would be nice if the RTSP state was kept seperate from the http client implementation as they are seperate concerns.

Could you revert your changes and try this patch ?

diff --git a/src/httpc.c b/src/httpc.c
index f9dd15a..4d75742 100644
--- a/src/httpc.c
+++ b/src/httpc.c
@@ -1018,7 +1018,8 @@ header:
     goto next_header;
   }
   hc->hc_rpos = 0;
-  if (hc->hc_version == RTSP_VERSION_1_0 && !hc->hc_csize) {
+  if (hc->hc_version == RTSP_VERSION_1_0 &&
+      (!hc->hc_csize || hc->hc_csize == -1)) {
     hc->hc_csize = -1;
     hc->hc_in_data = 0;
   } else {

The reply with 'Content-Length:0' header line is weird and it's not handled properly in the current code.

Regarding RTSP/HTTP client code split - it's possible, but it's also possible to create a function to copy state from old http client session to a new http client session, if required. This issue looks like that something is broken in the HTTP/RTSP parser, not a general connection close problem.

Are there any news, when this will be fixed?
I use tvheandend on my Synology Diskstation.
I don't know, how to implement this change and cross compile it.

Could you try v4.1-749-g3241f7c ? I applied the fix from comment#40 .

How can I apply this fix, my Linux knowledge is limited.

Use recent 4.1 code (master branch)... Actually v4.1-786-g4c54e63 .

recent code 4.1?? is that the tvheadend version?? im using the latest release and still unable to scan any channels, upon a full reboot it will scan the first mux, find a handful of channels then timeout and start coming back with error on the rest of the muxes.

Recent code is git master tree - https://github.com/tvheadend/tvheadend , you may pull it using git or download the zip file. The number means "number of changes after 4.0".

yeah I just found a site on adding the repository, ill try it later and come back here if I have any luck. thanks.

i tested Build: 4.1.859.

works fine ! thx for support