Odoc compress payloads - #1486
Open
jonludlam wants to merge 7 commits into
Open
Conversation
Before this change, the magic string was read by reading the number of bytes we're expecting to read based on our magic string. If ours happens to be a prefix of the real one (e.g. ours is 'odoc-3.1.0' and the real one is 'odoc-3.1.0-2-g12345789') then the check was succeeding and then the subsequent unmarshalling was failing. This fixes that by writing the length too, so we always unmarshal the right length.
These were introduced in 2c2ff6c to replace polymorphic compare and hash. Recent measurements show that these don't have a useful effect any more, and they cost a lot when storing the odoc files on disk as there's no sharing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No need to keep the record now that ikey/hash are gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Structurally equal paths used to be shared, via a memo in Lang_of that d1c0bad stopped applying and d90903a then removed. Repeated paths have been written out in full since. Hash-cons the converted paths instead, which doesn't need the maps and so shares across a whole run. Path.Resolved.Module gets the equal/hash/Hashtbl that Identifier already has, with the same hashing parameters. aliaschain.t goes from 1.1MB to 33KB, and peak RSS while linking from 70MB to 43MB; a full `odoc_driver core` run is ~14% smaller. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Route the marshalling in Odoc_file through the compiler's own Compression module (compiler-libs.common, already a dependency), the same mechanism it uses for .cmt/.cmti. Payloads are compressed when the compiler was built with zstd support and fall back to plain Marshal otherwise. Add a marker in the magic that states whether compression is being used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jonludlam
force-pushed
the
odoc-compress-payloads
branch
from
September 3, 2026 21:18
ecc681e to
b64080f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: this branch includes the changes in both #1479 and #1375
The last commit in this PR enables compression on odoc/odocl files if the compiler supports it. This results in a big drop in file sizes (around 4-5 times smaller).