Tvheadend

I can reproduce the memory increase by sending a large SD xmltv to the xmltv.sock multiple times, though the process size doesn't increase every time. I don't know why you get different results from different inputs since it should share most of the same processing. Does the memory leak if you run the file grab repeatedly (waiting for it to be processed in between runs)?

A valgrind on master shows a few places of interest, but nothing that was too large in my runs, and nothing that would majorly affect 4.2 (since changes occurred in 4.3 but if it's leaking in 4.2 then it narrows down the scope of interest).

We appear to leak uri per programme in _xmltv_parse_programme_tags (so a few MB per file), we leak from the xmltv calls to _xmltv_make_str_list_from_matching when replacing credits/keywords which looks to be caused from a missing "free(item)" in string_list_destroy (but that didn't exist in 4.2).

My box is too slow to run substantial checks, but I'll do a patch for those two tomorrow and see if I can get a bigger leak to show up.

Thanks for reproducing it. It seems that the memory leak is worse when there is more detail for the programs. My stock zap2xml script pulls very little info per program. I see a memory leak but it's pretty small (2.5% of my 12gig of memory). Also multiple runs with the same xmltv.xml file seems to show no increase in memory usage.

Using tv_grab_zz_sdjson --days 14 --output xmltv.xml to produce a much more detailed xmltv.xml file yields a much larger memory leak (8.9%). Also, though I can't confirm this, it seems that the memork lead doesn't get worse by running the same xml file but only when there is new data. So it seems that the internal EPG handler is first checking to see if the data has changed and, if not, skip handling it. This would avoid the memory leak. But any new data would go through the full processing showing the memory leak.

The memory leak exists just as bad in 4.2.4 as it does in 4.3

Can you recheck with the latest 4.3 master? I don't believe the fixes could account for a 10% memory leak, but for me, I can replay the same SD file and the memory size is stable.

I also don't get any memory leaks reported now with valgrind memory checker (other than a couple of kb at startup). I sent the SD file via the xmltv.sock, repeated the same file, then altered the file so description tags had an extra word so they would all be re-parsed; then edited the file so 2017 date-stamp was 2018. I then repeated the file/altered file with autorec rule that matched some programmes in case it is the rescheduling that causes problems. I tried with the tv_grab but got the same results.

I pulled the master repo, installed gcc and a bunch of other stuff, ran .configure and build. copied the binary into my /usr/bin and started the service. It prompted for my login but the webui didn't show.

is there a binary somewhat I can just grab? I'm guessing I built it wrong.

Reggie Burnett wrote:

I pulled the master repo, installed gcc and a bunch of other stuff, ran .configure and build. copied the binary into my /usr/bin and started the service. It prompted for my login but the webui didn't show.

is there a binary somewhat I can just grab? I'm guessing I built it wrong.

It should be in the repos by now so no need to build really :).

Mark Clarkstone wrote:

Reggie Burnett wrote:

I pulled the master repo, installed gcc and a bunch of other stuff, ran .configure and build. copied the binary into my /usr/bin and started the service. It prompted for my login but the webui didn't show.

is there a binary somewhat I can just grab? I'm guessing I built it wrong.

It should be in the repos by now so no need to build really :).

4.3~762 is the one I got from the unstable repo. No idea how to determine what is the last commit in that.

Reggie Burnett wrote:

Mark Clarkstone wrote:

Reggie Burnett wrote:

I pulled the master repo, installed gcc and a bunch of other stuff, ran .configure and build. copied the binary into my /usr/bin and started the service. It prompted for my login but the webui didn't show.

is there a binary somewhat I can just grab? I'm guessing I built it wrong.

It should be in the repos by now so no need to build really :).

4.3~762 is the one I got from the unstable repo. No idea how to determine what is the last commit in that.

ok. I found it. ~762 I think should contain you patch for the memory leaks. I tested it with tv_grab_zz_sdjson and still had significant leakage. > 11% after 1 run. :(

Reggie Burnett wrote:

Reggie Burnett wrote:

Mark Clarkstone wrote:

Reggie Burnett wrote:

I pulled the master repo, installed gcc and a bunch of other stuff, ran .configure and build. copied the binary into my /usr/bin and started the service. It prompted for my login but the webui didn't show.

is there a binary somewhat I can just grab? I'm guessing I built it wrong.

It should be in the repos by now so no need to build really :).

4.3~762 is the one I got from the unstable repo. No idea how to determine what is the last commit in that.

ok. I found it. ~762 I think should contain you patch for the memory leaks. I tested it with tv_grab_zz_sdjson and still had significant leakage. > 11% after 1 run. :(

Maybe you could you get a leak log from valgrind? See Traces & Debugging. That might help Em to see where the leak is :).

@Em/Jaroslav. Not sure this is much help, but I went through the codacy results & picked out the ones that involved EPG.

src/epg.c
The scope of the variable 'eb' can be reduced.
826 epg_brand_t *eb;

src/epggrab/module/xmltv.c
Redundant condition: If 'year > 1800', the comparison 'year' is always true.
383 if (year && year > 1800 && year < 2500) {

src/epggrab/module/opentv.c
The scope of the variable 'f' can be reduced.
771 htsmsg_field_t *f;

The scope of the variable 'e' can be reduced.
792 htsmsg_t *e;

src/epggrab/module/pyepg.c
The scope of the variable 'e' can be reduced.
51 htsmsg_t *e;

src/epggrab/module/xmltv.c
The scope of the variable 'rating' can be reduced.
451 htsmsg_t *rating, *tags;
The scope of the variable 'egl' can be reduced.
614 epg_genre_list_t *egl;

src/epggrab/module/xmltv.c
Assignment of function parameter has no effect outside the function. Did you forget dereferencing it?
195 s = xmltv_ns_get_parse_num(s, &(epnum->p_num), &(epnum->p_cnt));

src/epggrab/module/xmltv.c
The scope of the variable 'ch' can be reduced.
804 channel_t *ch;
The scope of the variable 'i' can be reduced.
1099 size_t i, p, n;

src/epggrab/module/pyepg.c
The scope of the variable 'save' can be reduced.
396 int gsave = 0, save;

src/epggrab/module/xmltv.c
The scope of the variable 'c' can be reduced.
235 htsmsg_t *c, *a;
The scope of the variable 'de' can be reduced.
1148 struct dirent *de;

src/epg.c
The scope of the variable 'id' can be reduced.
1896 char id[16];
src/epggrab/module/xmltv.c
The scope of the variable 'e' can be reduced.
511 htsmsg_t *e, *attrib;

src/spawn.c
The scope of the variable 'e0' can be reduced.
455 char **e, **e0, **e2, **e3, *p1, *p2;

src/epggrab/module/opentv.c
Variable 'i' is modified but its new value is never used.
764 i++;
src/epggrab/module/pyepg.c
The scope of the variable 'season' can be reduced.
219 epg_season_t *season;

src/epggrab/module/xmltv.c
The scope of the variable 'save' can be reduced.
937 int gsave = 0, save;

src/epg.c
The scope of the variable 't' can be reduced.
3159 uint64_t h = 0, t;
src/epggrab/module/xmltv.c
The scope of the variable 'e' can be reduced.
493 htsmsg_t *e;

src/epggrab/module/opentv.c
Variable 'r' is reassigned a value before the old one has been used.
582 r = dvb_table_end((mpegts_psi_table_t *)mt, st, sect);

src/epg.c
The scope of the variable 'f' can be reduced.
1392 htsmsg_field_t *f;
src/epggrab/module/xmltv.c
The scope of the variable 'sys' can be reduced.
236 const char *sys, *cdata;

src/epg.c
The scope of the variable 'tm1' can be reduced.
1615 char tm1[32];
src/epggrab/module/opentv.c
The scope of the variable 'mt' can be reduced.
703 mpegts_table_t *mt;
src/epggrab/otamux.c
The scope of the variable 'name' can be reduced.
315 char ubuf[UUID_HEX_SIZE], name[256];

src/epggrab/module/opentv.c
The scope of the variable 't' can be reduced.
636 int *t, r;

src/epggrab/module/pyepg.c
The scope of the variable 'brand' can be reduced.
160 epg_brand_t *brand;

src/epggrab/otamux.c
The scope of the variable 'svcl' can be reduced.
902 epggrab_ota_svc_link_t *svcl;

There's some really nasty red ones for some others too. O.O

Thanks, I took a quick look through but can't see anything that looked related to a memory leak.

Of the majors, the ones that seemed problems to me:
- src/descrambler/cccam.c:1114 - 32 on memcpy should be 3;
- src/htsp_server.c:3385 - probably make htsp_connection_t static like with some other recent changes, but needs more investigation to be sure.

The others didn't look to be problems, for example the qsort warning is due to the static analysis not noticing that there is a setter loop above it so it's not uninitialized.

Of the minors, the "The scope of the variable" is basically coding style (some projects prefer variables in one place or another); the "reassigned a value" seem ok since they are often used in debugging to see what functions did, ditto for the one "Assignment of function parameter".

- src/modules/epggrab/xmltv.c:383 - remove "year &&" to avoid warning.

Em Smith wrote:

Thanks, I took a quick look through but can't see anything that looked related to a memory leak.
.. snip ..

Sorry it wasn't much help !

It may as well be in binary!

I know very little about C, but I always find code analysers an interesting read, same goes for other peoples code, just to see if I can understand it. I've already learnt so much from tvheadend code.

Maybe you could you get a leak log from valgrind? See Traces & Debugging. That might help Em to see where the leak is :).

I can try but I've already tried to build it and I built a binary that didn't work. I'll try again though since I'm a sucker for punishment :)

I checked and yes v4.3-762-gdf43bf093 is the one with the patches. I'm not sure how you can get that information from the github page.

It's best to run the grab twice since the first memory increase could just be memory stabilizing out (though I doubt it for your case), for example SD for me often has filler-programmes of four hours+ for some minor channels for days 12+ that they fill in with proper info later, so you'd expect some memory fluctuations as the four hour fillers become half-hour shows; but nothing that would account for such a large memory loss.

(Sorry, my previous post took three hours to arrive!)

@Mark: It was useful since it highlighted some other problems.

@Reggie: I don't know how the binaries are built, but you may be able to put the valgrind line before the pre-built binary. The results will be harder to work with (since the binary is probably optimized), but if you're losing a gigabyte per-EPG run then there is something that's odd and even optimized results may be useful. How are you identifying the memory usage?

Em Smith wrote:

(Sorry, my previous post took three hours to arrive!)

@Mark: It was useful since it highlighted some other problems.

@Reggie: I don't know how the binaries are built, but you may be able to put the valgrind line before the pre-built binary. The results will be harder to work with (since the binary is probably optimized), but if you're losing a gigabyte per-EPG run then there is something that's odd and even optimized results may be useful. How are you identifying the memory usage?

I had to roll back to 4.2.4. The 4.3~762 was too unstable for me. I completely blew away my /home/hts/.hts folder and still it would freeze trying to setup my IPTV networks. latest 4.2.4 has the patches as well ( I think ) so we'll see how it behaves.

That's right, it has the patches.
If you do decide to run valgrind then install the tvheadend-dbg package first for debug symbols. Thanks.

@Mark
BTW: I hope my brevity in 9 didn't come across as being rude/dismissive. I wanted to quickly document what I saw in the static analysis for future reference and fixing at the w/e, but on forums sometimes brevity could be interpreted otherwise, and it occurred to me that when I said it wasn't leak-related, that was meant as just a quick overview rather than "why post that? it's not leak related". The best way to learn the code is to dive in with some thing that either annoys you or interests you.

Em Smith wrote:

@Mark
BTW: I hope my brevity in 9 didn't come across as being rude/dismissive.
I wanted to quickly document what I saw in the static analysis for future reference and fixing at the w/e, but on forums sometimes brevity could be interpreted otherwise, and it occurred to me that when I said it wasn't leak-related, that was meant as just a quick overview rather than "why post that? it's not leak related". The best way to learn the code is to dive in with some thing that either annoys you or interests you.

Not at all. My response probably didn't help, looking back it sounds like I took offence, I was actually apologising for wasting your time. :)

Guys

I really appreciate the attention that has been given this.  I really like tvheadend and would like to get back to using it.

I noticed last night that Comcast had rolled out their lineup changes in my area and SD hasn't picked them up yet so right now I'm trying to get my EPG data correct. The linuxserver docker image should get updated to 4.2.4-30 today so those patches will be included. I'll keep working at it and reporting back here and maybe have some time over the weekend to run valgrind on it.

Mmh,

i think the patch has destroyed epg import via tv_grab_file completely.

Using tv_grab_wg++, since 4.2.4-30 it doesn´t import any epg data or epg grab channels.

Dominik Wicke wrote:

Mmh,

i think the patch has destroyed epg import via tv_grab_file completely.

It's really unlikely. The whole 4.2 change for XMLTV import is:

diff --git a/src/epggrab/module/xmltv.c b/src/epggrab/module/xmltv.c
index 0c58f1f8e..f67123bef 100644
--- a/src/epggrab/module/xmltv.c
+++ b/src/epggrab/module/xmltv.c
@@ -533,6 +533,8 @@ static int _xmltv_parse_programme_tags
    */
   if (uri) {
     ee = epg_episode_find_by_uri(uri, mod, 1, &save3, &changes3);
+    free(uri);
+    uri = NULL;
   } else {
     ee = epg_episode_find_by_broadcast(ebc, mod, 1, &save3, &changes3);
   }

Using tv_grab_wg++, since 4.2.4-30 it doesn´t import any epg data or epg grab channels.

Could you verify, if your grabber sends correct data to tvh?

Jaroslav Kysela wrote:

Could you verify, if your grabber sends correct data to tvh?

Hey Jaroslav,

mmh.. seems to be work after delete all epg data, epg grab channels & epgdb.v2, also clean my guide.xml and did a fresh grab via webgrab.

maybe a corrupt xml file :)

thanks

Dominik Wicke wrote:

Jaroslav Kysela wrote:

Could you verify, if your grabber sends correct data to tvh?

Hey Jaroslav,

mmh.. seems to be work after delete all epg data, epg grab channels & epgdb.v2, also clean my guide.xml and did a fresh grab via webgrab.

maybe a corrupt xml file :)

thanks

Still seeing memory like on 4.3-1028~g7de759e83. Using SD with the zz epg grabber. After one run of importing guide data the tvh process is using 14% of my system memory. around 1.8-2gig.