[NellowTCS]

Description

Adds a streaming source API to elf_loader, so ELF images bigger than available DRAM can be checked and relocated without loading the whole file into memory. One example is apps of several hundred MB streamed from an SD card filesystem.

New public API (esp_elf.h)

• elf_source_t: a byte source you can read at any offset, with size, read, an optional close, and an opaque ctx.
• esp_elf_relocate_source(esp_elf_t *elf, const elf_source_t *src): validates the image through the source, then loads and relocates it. Headers, sections and relocation tables are read only when needed.

Loader (esp_elf.c)

• esp_elf_relocate() keeps its signature and behavior. It now validates the buffer and then wraps it in an in-memory elf_source_t, so the buffer path and the streaming path share one relocation core (esp_elf_relocate_source_impl).
• Section headers and the section-name table are read once and shared by the loader and the relocation loop.
• .symtab / .strtab are cached by section index, so they aren't read again for each .rela.* table. Each relocation table is read into memory only while it's being processed.
• Executable memory (.text, .got.loc) is filled through a 512-byte DRAM staging buffer using esp_elf_copy_executable(). Instruction RAM still only receives 32-bit stores, and the source never has to be fully in memory.
• Segment mode (!CONFIG_ELF_LOADER_BUS_ADDRESS_MIRROR) reads the program header table and PT_LOAD contents through the source.
• Error paths that used to return directly now free everything through a shared cleanup path. This also fixes leaks in the old code, for example on a failed symbol lookup.

Validation (esp_elf_validate.c)

• The validator core now reads everything through an internal elf_reader_t. The in-memory and streaming paths run the same structural checks without copied logic, and every read is checked against the image size.
• esp_elf_validate() is unchanged for callers. New private esp_elf_validate_source() runs the same checks on an elf_source_t.
• The section-name table is loaded into memory once, so a streamed source doesn't pay for a read on every name comparison.

Requirements for the source

• read must support any offset and must return the same bytes on every call. The image is validated first and then read again during relocation.
• close is called once on every return path; the caller must not close the source itself.

Memory note: the whole image no longer has to be in memory. The loaded sections themselves, the section header table, .symtab/.strtab, and one relocation table at a time still are.

Related

• Motivated by PocketMage, which can possibly large apps from SD.
• Useful in any app that uses ELF binaries as apps, including Tactility

Testing

Tested on hardware with the new streaming path:

• Target: ESP32-S3, ESP-IDF 5.5.5, built through Arduino-ESP32 3.3.11 (PlatformIO espressif32@55.3.311)
• Streaming path: loaded and relocated an ELF from an SD card with esp_elf_relocate_source(), using a file-backed elf_source_t. The full image was never loaded into DRAM.
• Test image: callgraph.app.elf (1484 bytes). Relocation succeeded, and the app ran as expected on the device.

---

Checklist

• 🚨 This PR does not introduce breaking changes. (esp_elf_relocate() / esp_elf_validate() signatures and behavior unchanged.)
• All CI checks (GH Actions) pass.
• Documentation is updated as needed.
• Tests are updated or added as necessary.
• Code is well-commented, especially in complex areas.
• Git history is clean — commits are squashed to the minimum necessary.

[CLAassistant]

All committers have signed the CLA.

[wujiangang]

Thanks for updating the host tests. I reran the 34-test suite on this revision; it passes now (2 skipped).

There are still a few read-failure cleanup issues to fix, noted inline. I'd also like to see a test through the public streaming API rather than just the internal loaders. A short README example showing how to set up a file-backed source would be useful too.

[Copilot]

🟡 Changes recommended

Failure-path cleanup can leave an executable MMU mapping pointing to freed memory and leak source handles or heap-backed validation tables.

3 open issues

• Remove executable mapping before freeing its backing buffer · New
• Close valid source when ELF handle is NULL · New
• Free loaded_ranges on program-header read failures · New

What changed in this PR

Adds random-access streaming to elf_loader, allowing ELF images to be validated, loaded, and relocated without keeping the entire file in DRAM.

Changes:

• Introduces elf_source_t and esp_elf_relocate_source() with source ownership documentation.
• Shares validation and relocation logic between buffer and streaming paths, with cached tables and staged executable-memory writes.
• Adapts existing section and segment host tests to source-backed loaders.

File	Description
components/​elf_loader/​tests/​host/​test_segment_load.c	Adapts segment tests to a memory-backed source.
components/​elf_loader/​tests/​host/​test_section_load.c	Adapts section tests to a memory-backed source.
components/​elf_loader/​src/​esp_elf.c	Implements source-based loading, relocation, caching, and cleanup.
components/​elf_loader/​src/​esp_elf_validate.c	Shares structural validation through a bounds-checked reader.
components/​elf_loader/​include/​private/​elf_validate.h	Declares the private source-validation API.
components/​elf_loader/​include/​esp_elf.h	Defines and documents the public streaming API.

_{Review effort: Balanced}

[wujiangang]

One more documentation item I missed in the previous review.

[NellowTCS]

Additionally, I have an additional use case for the library, using https://github.com/TalismanDesign/elf_overlay
Combined with this, it can reduce streaming RAM usage to an extremely low amount!