fix: release large muxer write buffers to the OS

[JxnLexn]

Summary

Sustained fMP4 muxing with large, variably sized writes can create severe process-heap retention under real long-lived camera workloads. This pull request moves large transient IOContext::WritePacket() callback buffers out of the regular process heap while preserving their copy and lifetime semantics.

Fixes #328

Allocation path

IOContext::WritePacket() currently copies every FFmpeg write through Napi::Buffer<uint8_t>::Copy(). The affected production workload uses four permanent HKSV prebuffers, 4-second fragments, the default 2 MiB AVIO buffer, and high-resolution H.264 remux input. The resulting large, variably sized writes repeatedly churn the Node/libc heap.

Changes

• Keep the existing regular Node-buffer path for writes below 64 KiB.
• Allocate larger transient write buffers with page-backed storage (mmap on POSIX, VirtualAlloc on Windows).
• Preserve page alignment by passing the allocation length through finalize_hint instead of storing a header in the mapping.
• Report the mapping to V8 with Napi::MemoryManagement::AdjustExternalMemory() and subtract it in the finalizer.
• Release the mapping from the external-buffer finalizer.
• Preserve a safe copy fallback for embedders that disallow external buffers and for allocation failure.
• Handle external-buffer creation failures without leaks or double frees.
• Add direct and thread-safe callback regression tests proving that retained large buffers remain valid after the native write returns.

FFmpeg data is still copied before the callback returns, so the patch does not introduce a zero-copy lifetime hazard.

The previously included FMP4Stream.emitBoxGroup() view optimization has been removed from this PR. It is independent and changes observable backing-buffer/byteOffset semantics.

Production evidence

Affected deployment:

• Ubuntu 24.04.4 x86_64, glibc 2.39
• Node.js 24.18.0
• fragDuration: 4_000_000
• default bufferSize: 2 * 1024 * 1024
• no production MALLOC_* tunables

A run started on August 19, 2026 at 18:36 CEST. Proxmox RRD data shows the container below 1 GiB around 19:30, approximately 1.6 GiB at 20:00, 5.2 GiB at 20:30, 9.4 GiB at 21:00, and at its 10 GiB cgroup limit by 21:30. At 00:16 CEST the HomeKit process alone held 9,360,352 KiB of anonymous/private-dirty RSS. The cgroup had more than 53 million memory.high events and no OOM kill. A preceding run had reached the same limit before restart.

This does not show the 300 MiB warm-up plateau seen in the single-stream Debian/glibc-2.36 reproduction. Short isolated tests against the same real camera source, without HomeKit/HAP queues, also showed lower post-GC growth with MALLOC_MMAP_THRESHOLD_=65536 or a low trim threshold.

Verification

The revised head 76016bb has been verified with:

• npm run build:tests
• npm run build:tsc
• npm run lint
• npm run lint:gyp
• git diff --check
• a complete native Linux x64 addon build and link on Ubuntu 24.04 / glibc 2.39 / Node.js 24.19.0, using the project Jellyfin FFmpeg build inputs and prebuildify --napi --strip --arch=x64 --tag-libc
• focused tests against that freshly built addon proving that retained large direct-callback and thread-safe-callback buffers remain valid after the native write returns

The GitHub workflow for this fork PR still requires maintainer approval before its jobs can start.

[seydx]

Hey, thanks for the PR

The write path really does allocate a fresh copy per call, and at realistic fragment lengths those writes really are large and variably sized, so that part is accurate. I measured it here and got p50 17.7 KiB / p90 78 KiB with 4s fragments, which lines up with what you describe.

Where I get stuck is the conclusion. I tried to reproduce the unbounded growth before merging and I can't. Setup: Debian bookworm, glibc 2.36, node-av 6.2.0-beta.17, a single long-lived FMP4Stream over an endless lavfi source, 4s fragments like HKSV, fragments dropped immediately, 5 minutes per config, RSS sampled after a forced GC.

Default glibc: 284 MiB at t=60, peak 308, and completely flat from t=160 to t=300. With MALLOC_MMAP_THRESHOLD_=65536: peak 313. That one matters because it forces every allocation of 64 KiB and up through mmap, which is exactly what this PR does by hand, and it does not lower the peak at all. With bufferSize: 64 * 1024 I get 9168 write buffers instead of 1140 for the same amount of data, so 8x the number of allocations, and RSS ends up at 302 instead of 303. And after stop() RSS drops from 303 to 181, so the high-water mark does come back, which contradicts the "stopping the stream does not return it" point in the issue.

I also ran a control with no node-av at all, just Buffer.allocUnsafe in the same size distribution. That plateaus at 98 MiB within the first 20 seconds and stays there across 6.2 TiB of allocation churn. Same step-then-plateau shape on macOS. So what I see is an allocator warm-up curve rather than a leak, and I think a 10 minute window extrapolated linearly to "200 MiB/hour" is exactly what that curve looks like before it flattens.

Two things about the patch itself, independent of the above. napi_create_external_buffer does not tell V8 how much memory is behind the buffer, so the GC loses the pressure it currently gets from the V8-owned backing store of Buffer::Copy. We already have a convention for that, see Napi::MemoryManagement::AdjustExternalMemory in frame.cc and packet.cc with the SyncExternalMemory() helper. Without it the mappings can end up living longer than the copies do today, which would work against the goal. The other one is smaller: storing the length in an 8 byte header inside the mapping and returning allocation + sizeof(size_t) means the buffer data is never page aligned. napi_create_external_buffer takes a finalize_hint, so you can pass the length through that and drop the header.

The emitBoxGroup change I'd want to look at separately. Skipping a Buffer.concat per fragment is a real saving, but it makes the emitted fragment a view with a non-zero byteOffset into a larger backing buffer, which is visible to anyone doing new Uint8Array(data.buffer). That deserves its own PR and a note in the changelog rather than riding along with the native change.

I'm not closing this. What would move it forward for me is a longer run, a few hours rather than ten minutes, showing that the curve does not flatten out, plus your exact glibc version and platform, and the fragDuration and bufferSize you're running with. That last part matters: with the default fragDuration of 1 the writes here are p50 0.3 KiB, so the new 64 KiB path would never even trigger. It would also help to know whether MALLOC_ARENA_MAX=2 changes anything in your setup, since a lot of the RSS in my runs is x264's encoder threads and their per-thread arenas, not the write buffers.

[JxnLexn]

Thanks for the detailed reproduction and review. Your three patch-level points are correct, and I updated the branch in 76016bb:

• added symmetric Napi::MemoryManagement::AdjustExternalMemory(+length/-length) accounting;
• passed length through finalize_hint, so the mapped buffer stays page-aligned and no header is stored in the mapping;
• removed the emitBoxGroup() view optimization from this PR entirely. I agree that its observable byteOffset/backing-buffer semantics make it a separate change.

I also rebased onto current main, including the merged pipeline teardown work from #331.

The affected production environment and actual FMP4 settings are:

• Ubuntu 24.04.4 x86_64, glibc 2.39
• Node.js 24.18.0
• four permanent HKSV prebuffer streams
• H.264/AAC remux for supported input codecs
• fragDuration: 4_000_000 (4 seconds)
• default bufferSize: 2 * 1024 * 1024
• no MALLOC_* environment variables

I agree that the earlier 10-minute isolated run alone was not enough to extrapolate indefinitely. The production history now gives a longer window, though: after a restart on August 19 at 18:36 CEST, Proxmox RRD shows the container below 1 GiB around 19:30, ~1.6 GiB at 20:00, ~5.2 GiB at 20:30, ~9.4 GiB at 21:00, and at the 10 GiB cgroup limit by 21:30. It then stays pinned there by reclaim. At 00:16 CEST the HomeKit process itself had 9,360,356 KiB RSS, of which 9,360,352 KiB was anonymous/private dirty, with zero swap. The cgroup had >53 million memory.high events and no OOM kill. A preceding run had independently reached the same limit before restart.

So I agree with avoiding a blanket conclusion from the short synthetic curve, but the real four-stream remux workload does not plateau near 300 MiB. The platform difference may matter as well: production is glibc 2.39, not 2.36.

For allocation isolation, the earlier test was a single FMP4Stream against the same real camera stream, outside HomeKit/HAP/application queues. In short A/B runs, MALLOC_MMAP_THRESHOLD_=65536 and a low trim threshold reduced post-GC anonymous growth materially. I have not set MALLOC_ARENA_MAX=2 in production because that requires restarting the currently memory-saturated service; I will report that separately from a controlled run rather than changing the live process just to complete the matrix.

I also revised #328 and this PR description to separate the multi-hour production non-plateau evidence from the shorter isolated allocator A/B evidence, and removed the claim that the ten-minute run by itself proved an indefinite linear rate.

[JxnLexn]

Additional verification for the revised head 76016bb:

I reconstructed the repository's Linux x64 CI build locally in an isolated worktree using the project's Jellyfin FFmpeg build inputs and the ghcr.io/seydx/jellyfin-ffmpeg/linux64-gpl:latest image.

Environment:

• Ubuntu 24.04 / glibc 2.39
• x86_64
• Node.js 24.19.0
• N-API prebuild produced by prebuildify --napi --strip --arch=x64 --tag-libc

Results:

• the complete native addon compiled and linked successfully;
• src/bindings/io_context.cc compiled as part of that full build;
• npm run build:tests passed;
• npm run build:tsc passed;
• the two focused IOContext lifetime tests passed against the newly built native addon:
  • large direct-callback write buffer remains valid after the native write returns;
  • large thread-safe-callback write buffer remains valid after the native write returns.

This closes the previous verification gap caused by the incomplete restored FFmpeg build tree. The GitHub fork workflow still requires maintainer approval before it can run.