Bug #4854
OTA EPG grabber memory usage
0%
Description
Hi,
I would like to ask about the expected memory usage of the tvheadend's OTA EPG grabber.
I am running tvheadend 4.3-912~g5fe3ebd48 on a set-top-box with 1 GiB of memory. I have ~4000 mapped TV channels. Normally, tvheadend uses ~500 MiB of virtual memory, while ~250 MiB of it is present in RAM (resident set size).
When I run the OTA EPG grabber, it eventually causes tvheadend to be killed by the kernel's OOM killer. It takes about 2-3 hours of EPG data scanning until it happens. It is always reproducible.
I would like to ask whether such a behavior is expected. I would assume that the EPG data that has been grabbed so far should regularly be flushed to the on-disk database. Then, I think, it should not be necessary to have a lot of memory in order to grab EPG data from many channels.
The OOM event happens when tvheadend starts to save the EPG data to disk. I am not familiar with how that works exactly, but I would assume it should not require significantly more memory than what is already used by the EPG data which is being flushed to disk.
Unless, of course, the entire EPG database would have to be rebuilt in memory at first. That would explain the rather high memory requirement. In such case, the obvious follow-up question is, whether it could be improved. I believe that there are smart ways to handle "large" databases on disk with little memory. Maybe they should at least be considered.
History
Updated by Jaroslav Kysela almost 7 years ago
Updated by TVH have all EPG data in the system memory and there's no plan to store them on disk or so... If you use a debug version of tvheadend, there is 'Memory information entries' tab in the configuration / debugging, so you can watch how many EPG data are allocated.
Updated by Peter Bašista almost 7 years ago
Updated by Jaroslav Kysela wrote:
TVH have all EPG data in the system memory and there's no plan to store them on disk or so...
I see. There is no plan because no one has thought about it yet or because it has been considered and declined?
In my particular use case, the current behavior is not very convenient. I understand the EPG needs to be readily available to clients and therefore storing it in memory is convenient. At the same time, the clients typically only need EPG for a handful of channels at once. I think that tvheadend can safely keep all the EPG data on disk and only load those records to memory, which are currently needed by clients. It would be a logical optimization, at least from my point of view.
Updated by Jaroslav Kysela almost 7 years ago
The EPG grid shows all channels, so your point is irrelevant, because all EPG data are used. I don't see another solution when we have fast 'search as you type' function.
Updated by Peter Bašista almost 7 years ago
Jaroslav Kysela wrote:
The EPG grid shows all channels, so your point is irrelevant, because all EPG data are used.
That is true.
I don't see another solution when we have fast 'search as you type' function.
Me neither. However, the mentioned search feature may not be that interesting for some people. I typically do not need to view or edit the EPG data via tvheadend's web interface in any way. I only configure the scrappers or grabbers and that is it. And then I use clients for viewing it.
I think it would be nice to have an option to disable the instant search feature for EPG in exchange for lower memory usage.
Updated by Jaroslav Kysela almost 7 years ago
One problem might be that glibc's memory allocator has trouble with the fragmentation after an amount of runtime. Could you try LD_PRELOAD another allocator like jemalloc and let us know the result? https://github.com/jemalloc/jemalloc/wiki/Getting-Started