Skip to content

test(bulk): convert the fixture corpus in bounded parallel batches - #1940

Merged
developer0hye merged 1 commit into
mainfrom
perf/parallel-bulk-gate
Sep 28, 2026
Merged

developer0hye merged 1 commit into
mainfrom
perf/parallel-bulk-gate

Conversation

@developer0hye

Copy link
Copy Markdown
Owner

File submission policy

Before attaching or committing files, read the submission policy.

  • Any submitted sample files or attachments satisfy the submission policy, or none are submitted.

Summary

Last of three follow-ups from the CI review with Codex. After #1938 and #1939, a clean PR run takes 5.7 min wall. The Bulk Third-Party Fixture Test is the only job over 5 min (5.5 min); its gate test converted ~2,700 fixtures one at a time in 236.6s.

run_bulk_test now converts through a new map_in_batches helper:

  • Workers: two by default; BULK_CONVERSION_WORKERS=N overrides it, and the module doc header documents the variable.
  • Bounded batches: 32 files per batch, and each batch drains completely before the next starts. pdf.rs evicts Typst's memoization only when no compilation is active (TypstCacheState::begin_compilation). A pool that never went idle would therefore never evict and would keep every document's cache alive. Draining restores the idle point that the sequential loop had after every file.
  • Order preserved: results come back in corpus order, and per-file progress lines print in that order after conversion. The report and the baseline comparison are unchanged.
  • Panic isolation: each file still goes through convert_file's catch_unwind.

TDD: two new non-ignored tests.

  • test_map_in_batches_preserves_input_order: results stay in input order across worker and batch sizes.
  • test_map_in_batches_drains_each_batch_and_bounds_concurrency: no item starts before every earlier batch finished, and peak concurrency stays ≤ workers while still exceeding 1.

Related issue

Related: #1939

Testing

  • cargo test --locked -p office2pdf --test bulk_conversion — 9 passed, including both new tests (first run red: map_in_batches did not exist)
  • cargo clippy --locked --workspace --all-targets -- -D warnings and cargo fmt --all -- --check
  • This PR's Bulk job runs the full gate against the baseline and records the new duration.

Visual impact

  • No rendered PDF change
  • Rendered PDF change or visual evidence added
  • Reason: Only the bulk test harness changes; converter code is untouched.

🤖 Generated with Claude Code

The bulk regression gate converted ~2,700 fixtures one at a time and
spent 236.6s doing it, one of the longest steps in CI. It now converts
with two workers (BULK_CONVERSION_WORKERS overrides) in batches of 32
that drain before the next starts: pdf.rs evicts Typst's memoization
only while no compilation is active, so an always-busy pool would keep
every document's cache alive. Results stay in corpus order, so the
report and the baseline comparison are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Yonghye Kwon <developer.0hye@gmail.com>
@developer0hye
developer0hye merged commit 95c4bff into main Sep 28, 2026
25 checks passed
@developer0hye
developer0hye deleted the perf/parallel-bulk-gate branch September 28, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant