Tvheadend

Adam Sutton wrote:

Are these encrypted or FTA streams?

No its FTA HD channels. Recording even 3 such streams as mkv works well. I usually do not use TS but now I want material that the TV can play directly (see bug I just openned) ;-) Just compiling git today to see if TS memory leak bug is still there. It took one about one hour with two HD recordings to eat the available memory.

OK, I've got my own system recording 2 simultaneous HD streams (same mux), something I do regularly. Also got a few others doing the same, they're not seeing anything.

There must be something specific to the streams and/or setup. Maybe you could grab a raw mux dump, that might help with debugging.

Adam

Adam Sutton wrote:

OK, I've got my own system recording 2 simultaneous HD streams (same mux), something I do regularly. Also got a few others doing the same, they're not seeing anything.

There must be something specific to the streams and/or setup. Maybe you could grab a raw mux dump, that might help with debugging.

Adam

How much RAM do you have on your machine? I have only 2 GB and 2 GB swap... Why would MKV work and not TS on same stream? You have the stream info in the bug report on MKV and hardware decoder. Started a config from scratch when moving to 3.3.xx. Can give you a raw mux dump if needed.

BTW I have a question how can I produce the mkv via tvheadend when I recorded the TS? (to hunt the MKV bug).

That machine has 2G (with ZERO swap, and a reasonable chunk dedicated to RAM storage), but that's kinda missing my point. What I was implying is that the memory usage was static, about 3% constantly, a memory leak of the proportions you're talking about would be bringing my server down all the time.

So there must be something specific about the streams or something strange in the way your system is setup. There has been a long standing report about possible memory leaks when streaming multiple encrypted streams (#1383), however no one has managed to replicate and the issue was confused by several other problems.

I don't doubt there could be issues in there, just that there not clear cut and almost certainly specific to a given environment/input/etc. A muxdump would definitely help, especially if you can first reproduce the problem using that stream as input (-r) to tvheadend.

Adam

Adam Sutton wrote:

That machine has 2G (with ZERO swap, and a reasonable chunk dedicated to RAM storage), but that's kinda missing my point. What I was implying is that the memory usage was static, about 3% constantly, a memory leak of the proportions you're talking about would be bringing my server down all the time.

Well that's something new for me also. I used to record in TS for various testing purpose fine even with 3.3 until last night test. I will rebuild the last git and see if valgrind point to something (if I manage to run it correctly).

How to produce the full mux dump (I know there is a checkbox in the GUI but do not remember how to trigger it).

tvheadend -r populates the input but how do you program the output (trigger the recording on this input). I searched last time and missed the info.

Got a new one when trying to record as TS yesterday. I'm currently in coverity to see if static analysis finds something. So far only minor leak and non related to TS only...

[ 9866.155871] tvheadend invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[ 9866.155881] tvheadend cpuset=/ mems_allowed=0
[ 9866.155887] Pid: 11129, comm: tvheadend Tainted: P O 3.7.6 #87
[ 9866.155891] Call Trace:
[ 9866.155905] [<ffffffff8144e657>] ? dump_header.isra.9+0x6a/0x1d6
[ 9866.155914] [<ffffffff8108265c>] ? shrink_slab+0x18b/0x19d
[ 9866.155921] [<ffffffff8107aadb>] ? oom_kill_process+0x69/0x30c
[ 9866.155927] [<ffffffff8108410b>] ? do_try_to_free_pages+0x28e/0x30c
[ 9866.155934] [<ffffffff8107b0c8>] ? out_of_memory+0x1e8/0x247
[ 9866.155940] [<ffffffff8107dc91>] ? __alloc_pages_nodemask+0x5bb/0x6d0
[ 9866.155948] [<ffffffff8107a159>] ? filemap_fault+0x24e/0x36d
[ 9866.155955] [<ffffffff8108cf8b>] ? __do_fault+0xa1/0x3e9
[ 9866.155962] [<ffffffff8108f497>] ? handle_pte_fault+0x28d/0x610
[ 9866.155969] [<ffffffff81090713>] ? handle_mm_fault+0x4e/0x191
[ 9866.155977] [<ffffffff8101f69b>] ? __do_page_fault+0x373/0x3cb
[ 9866.155986] [<ffffffff814549a2>] ? page_fault+0x22/0x30
[ 9866.155989] Mem-Info:
[ 9866.155993] DMA per-cpu:
[ 9866.155997] CPU 0: hi: 0, btch: 1 usd: 0
[ 9866.156001] CPU 1: hi: 0, btch: 1 usd: 0
[ 9866.156044] CPU 2: hi: 0, btch: 1 usd: 0
[ 9866.156049] CPU 3: hi: 0, btch: 1 usd: 0
[ 9866.156052] DMA32 per-cpu:
[ 9866.156057] CPU 0: hi: 186, btch: 31 usd: 30
[ 9866.156060] CPU 1: hi: 186, btch: 31 usd: 5
[ 9866.156064] CPU 2: hi: 186, btch: 31 usd: 90
[ 9866.156068] CPU 3: hi: 186, btch: 31 usd: 0
[ 9866.156080] active_anon:272012 inactive_anon:91474 isolated_anon:0
[ 9866.156080] active_file:91 inactive_file:154 isolated_file:0
[ 9866.156080] unevictable:0 dirty:0 writeback:0 unstable:0
[ 9866.156080] free:2725 slab_reclaimable:3274 slab_unreclaimable:3169
[ 9866.156080] mapped:567 shmem:554 pagetables:2663 bounce:0
[ 9866.156080] free_cma:0
[ 9866.156106] DMA free:6024kB min:48kB low:60kB high:72kB active_anon:4760kB inactive_anon:4768kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:156kB slab_unreclaimable:52kB kernel_stack:0kB pagetables:80kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 9866.156111] lowmem_reserve[]: 0 1495 1495 1495
[ 9866.156130] DMA32 free:4876kB min:4920kB low:6148kB high:7380kB active_anon:1083288kB inactive_anon:361128kB active_file:364kB inactive_file:616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1531780kB mlocked:0kB dirty:0kB writeback:0kB mapped:2276kB shmem:2216kB slab_reclaimable:12940kB slab_unreclaimable:12624kB kernel_stack:1656kB pagetables:10572kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:598 all_unreclaimable? yes
[ 9866.156136] lowmem_reserve[]: 0 0 0 0
[ 9866.156143] DMA: 7*4kB 6*8kB 10*16kB 3*32kB 3*64kB 1*128kB 1*256kB 2*512kB 2*1024kB 1*2048kB 0*4096kB = 6028kB
[ 9866.156161] DMA32: 207*4kB 0*8kB 1*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4876kB
[ 9866.156181] 37263 total pagecache pages
[ 9866.156185] 36432 pages in swap cache
[ 9866.156191] Swap cache stats: add 552909, delete 516477, find 19178/25376
[ 9866.156195] Free swap = 0kB
[ 9866.156198] Total swap = 1952756kB
[ 9866.167833] 393104 pages RAM
[ 9866.167840] 9195 pages reserved
[ 9866.167843] 263300 pages shared
[ 9866.167846] 379733 pages non-shared
[ 9866.167849] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 9866.167849] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 9866.167876] [ 1070] 0 1070 6029 1 17 206 -1000 udevd
[ 9866.167883] [ 1134] 0 1134 5995 1 16 194 -1000 udevd
[ 9866.167890] [ 1135] 0 1135 5995 1 16 194 -1000 udevd
[ 9866.167900] [ 2438] 0 2438 4779 10 15 51 0 rpcbind
[ 9866.167906] [ 2473] 104 2473 5872 1 17 113 0 rpc.statd
[ 9866.167913] [ 2489] 0 2489 6359 0 17 56 0 rpc.idmapd
[ 9866.167920] [ 2857] 118 2857 5484 0 16 150 0 privoxy
[ 9866.167927] [ 2878] 0 2878 4584 7 12 78 0 oscam
[ 9866.167934] [ 2880] 0 2880 105644 65 24 150 0 oscam
[ 9866.167940] [ 2899] 0 2899 65748 0 30 150 0 rsyslogd
[ 9866.167947] [ 2973] 123 2973 4350 21 11 33 0 dirmngr
[ 9866.167954] [ 2995] 114 2995 31188 0 62 1203 0 timidity
[ 9866.167961] [ 3055] 116 3055 1378052 323096 1693 472888 0 tvheadend
[ 9866.167968] [ 3089] 0 3089 4204 5 12 36 0 atd
[ 9866.167974] [ 3111] 0 3111 1097 0 8 80 0 acpid
[ 9866.167981] [ 3208] 0 3208 17399 0 37 197 0 cupsd
[ 9866.167988] [ 3214] 0 3214 5731 22 15 41 0 cron
[ 9866.167994] [ 3256] 105 3256 8135 55 20 68 0 dbus-daemon
[ 9866.168002] [ 3636] 0 3636 21651 0 13 93 0 pcscd
[ 9866.168031] [ 3664] 0 3664 6704 0 18 54 0 kdm
[ 9866.168040] [ 3697] 0 3697 35857 667 72 4640 0 Xorg
[ 9866.168049] [ 3974] 120 3974 12888 2135 29 1439 0 tor
[ 9866.168057] [ 4075] 0 4075 41104 405 49 1709 0 wicd
[ 9866.168066] [ 4121] 0 4121 19558 262 42 1318 0 wicd-monitor
[ 9866.168074] [ 4139] 0 4139 10391 1 26 107 0 kdm
[ 9866.168082] [ 4144] 117 4144 95383 732 145 1208 0 kdm_greet
[ 9866.168090] [ 4160] 0 4160 4118 1 13 40 0 getty
[ 9866.168098] [ 4161] 0 4161 4118 1 12 42 0 getty
[ 9866.168106] [ 4162] 0 4162 4118 1 13 41 0 getty
[ 9866.168114] [ 4163] 0 4163 4118 1 13 41 0 getty
[ 9866.168122] [ 4164] 0 4164 4118 1 13 41 0 getty
[ 9866.168130] [ 4165] 0 4165 4118 1 14 40 0 getty
[ 9866.168139] [ 4179] 0 4179 2528 52 7 522 -1000 dhclient
[ 9866.168147] [ 4246] 0 4246 13075 1 28 152 -1000 sshd
[ 9866.168156] [ 7555] 0 7555 523429 0 59 370 0 console-kit-dae
[ 9866.168165] [ 7625] 124 7625 94339 0 48 667 0 polkitd
[ 9866.168171] Out of memory: Kill process 3055 (tvheadend) score 914 or sacrifice child
[ 9866.168185] Killed process 3055 (tvheadend) total-vm:5512208kB, anon-rss:1292384kB, file-rss:0kB

Valgrind reports:

Eric Valette - wrote:
av_bitstream_filter_filter(lm->lm_h264_filter,
st->codec,
NULL,
&packet.data,
&packet.size,
pktbuf_ptr(pkt->pkt_payload),
pktbuf_len(pkt->pkt_payload),
pkt->pkt_frametype < PKT_P_FRAME);

But
1) packet.data is never freed
2) if as code suggest its a real realloc then the pktbuf_ptr(pkt->pkt_payload) location is already freed!!!

Sample code of how to use av_bitstream_filter_filter:

while (bsfc) {
AVPacket new_pkt = *pkt;
int a = av_bitstream_filter_filter(bsfc, avctx, NULL,
&new_pkt.data, &new_pkt.size,
pkt->data, pkt->size,
pkt->flags & AV_PKT_FLAG_KEY);
if(a == 0 && new_pkt.data != pkt->data && new_pkt.destruct) {
uint8_t *t = av_malloc(new_pkt.size + FF_INPUT_BUFFER_PADDING_SIZE); //the new should be a subset of the old so cannot overflow
if(t) {
memcpy(t, new_pkt.data, new_pkt.size);
memset(t + new_pkt.size, 0, FF_INPUT_BUFFER_PADDING_SIZE);
new_pkt.data = t;
a = 1;
} else
a = AVERROR;
}
if (a > 0) {
av_free_packet(pkt);
new_pkt.destruct = av_destruct_packet;
} else if (a < 0) {
av_log(NULL, AV_LOG_ERROR, "Failed to open bitstream filter %s for stream %d with codec %s",
bsfc->filter->name, pkt->stream_index,
avctx->codec ? avctx->codec->name : "copy");
print_error("", a);
if (exit_on_error)
exit_program(1);
}
*pkt = new_pkt;

bsfc = bsfc->next;
    }

Removing libav useage (defaulting to no in config) makes the problem vanish. So until it is fixed I suggest to revert the default to no instead of auto...

Adam Sutton wrote:

Eric,

I'm about to start thinking about branching for 3.4, I agree that it would make sense to make libav usage default=off for 3.4, though I think I'll leave it on in master.

But I'll try and discuss with John.

Adam,

You're the boss but the code has 0 chance to work as it is with recent libav (1.0.x, 1.1.x):
the return value >0 indicates if the memory has been reallocated but then you should not suppose original passed pointer shall be freed again. 0 indicates the memory is still valid but the new buffer value pointer may be different than the original passed one. Hunting internet I found a lot of messages about memory leaks using this function. If you want the memory to be freed by call to av_interleaved_write_frame, you should set new_pkt.destruct = av_destruct_packet; as the ffmpeg.c code does.

So I'm not sure you want people using TS crash their recording as default behavior ;-) As far as I'm concerned I fixed the two bugs that were really annoying me (see mkv bug). As time permits I will provide some fixes for code flagged by coverity but I'm rather busy at work and the tool is only available there.

Indeed I do not want that and that is why I have accepted your suggestion for the upcoming release. However master is by its definition development/experimental code (i.e. use at your own risk) and so I'd prefer we leave this in (bug and all) and John can fix it as time permits.

If we were not about to make a new release I might think differently. But in the coming weeks other experimental stuff will be added to master that will make it pretty unstable again for a while anyway, so I'm not too concerned at this stage.

But hopefully your investigations will help John to fix it more quickly, so thank you.

Adam

John Törnblom wrote:

fix commited to master, pls test

Since you do no test the result of av_bitstream_filter_filter, the code cannot be right (ptr may change without being reallocated). Again read ffmpeg.c code.

well, the memory leak is fixed for me at least. the reallocation you suggest prevent overflow from happening but looking at avconv.c from ffmpeg, the reallocation is not done at all. But perhaps the buffer is big enough on so it might be okey. In any case, the overflow issues might affect other libav calls and is not specific to the use of filters.

John Törnblom wrote:

well, the memory leak is fixed for me at least. the reallocation you suggest prevent overflow from happening but looking at avconv.c from ffmpeg, the reallocation is not done at all. But perhaps the buffer is big enough on so it might be okey. In any case, the overflow issues might affect other libav calls and is not specific to the use of filters.

1) the function may fail and can produce negative return. Unchecked.
2) > 0 or equal to 0 tells if reallocation happens not the pointer chnaging value.

What version of ffmpeg are you looking at?

http://www.ffmpeg.org/doxygen/trunk/avconv_8c-source.html

John Törnblom wrote:

http://www.ffmpeg.org/doxygen/trunk/avconv_8c-source.html

look at line 943 in this file.

Eric Valette wrote:

John Törnblom wrote:

http://www.ffmpeg.org/doxygen/trunk/avconv_8c-source.html

look at line 943 in this file.

or line 579 of http://www.ffmpeg.org/doxygen/trunk/ffmpeg_8c_source.html