test(bulk): convert the fixture corpus in bounded parallel batches - #1940
Merged
Merged
Conversation
The bulk regression gate converted ~2,700 fixtures one at a time and spent 236.6s doing it, one of the longest steps in CI. It now converts with two workers (BULK_CONVERSION_WORKERS overrides) in batches of 32 that drain before the next starts: pdf.rs evicts Typst's memoization only while no compilation is active, so an always-busy pool would keep every document's cache alive. Results stay in corpus order, so the report and the baseline comparison are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Yonghye Kwon <developer.0hye@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
File submission policy
Before attaching or committing files, read the submission policy.
Summary
Last of three follow-ups from the CI review with Codex. After #1938 and #1939, a clean PR run takes 5.7 min wall. The Bulk Third-Party Fixture Test is the only job over 5 min (5.5 min); its gate test converted ~2,700 fixtures one at a time in 236.6s.
run_bulk_testnow converts through a newmap_in_batcheshelper:BULK_CONVERSION_WORKERS=Noverrides it, and the module doc header documents the variable.pdf.rsevicts Typst's memoization only when no compilation is active (TypstCacheState::begin_compilation). A pool that never went idle would therefore never evict and would keep every document's cache alive. Draining restores the idle point that the sequential loop had after every file.convert_file'scatch_unwind.TDD: two new non-ignored tests.
test_map_in_batches_preserves_input_order: results stay in input order across worker and batch sizes.test_map_in_batches_drains_each_batch_and_bounds_concurrency: no item starts before every earlier batch finished, and peak concurrency stays ≤ workers while still exceeding 1.Related issue
Related: #1939
Testing
cargo test --locked -p office2pdf --test bulk_conversion— 9 passed, including both new tests (first run red:map_in_batchesdid not exist)cargo clippy --locked --workspace --all-targets -- -D warningsandcargo fmt --all -- --checkVisual impact
🤖 Generated with Claude Code