[CI] Evaluate and reduce GitHub Actions test runtime

[mickaelbegon]

Note

This issue is based on a preliminary evaluation performed by Codex using recent public GitHub Actions logs and the current workflow definitions.

The duration ranges and expected improvements below are estimates, not benchmark results. They must be confirmed with representative CI runs.

Context

The current pull-request CI runs:

• 6 Linux test shards;
• 12 macOS/Windows test shards;
• one coverage merge job;
• one Black lint job.

Recent complete runs typically take 35–42 minutes, partly because of runner availability and queueing.

A first-pass modernization prepared by Codex is available on mickaelbegon/codex/update-github-actions, currently at commit 0a47fd95.

The prototype:

• updates the actions to Node 24-compatible versions;
• cancels obsolete runs for the same pull request;
• removes redundant Conda operations;
• fixes action-level warnings;
• improves dependency reproducibility;
• reports the slowest tests in every shard.

Codex estimates that these initial changes could improve an isolated run by approximately 5–15%. A larger improvement is unlikely without changing environment caching, shard distribution, or the cross-platform execution policy.

Main limitations identified by Codex

Repeated environment creation

Each of the 18 test shards creates an independent Conda environment.

On Windows, environment setup and cleanup can take approximately 5–7 minutes per job. Complete Conda environments and downloaded packages are not currently cached.

Unbalanced shards

Observed test durations vary significantly:

• Linux: approximately 6–14 minutes;
• macOS: approximately 5–10 minutes;
• Windows: approximately 7–10 minutes.

The slowest shard determines the effective critical path. The prototype adds pytest --durations reporting so that the distribution can be improved using test-level measurements.

Full cross-platform matrix on every pull request

Every pull request executes all six shards on Linux, macOS, and Windows.

This provides broad coverage, but consumes many runners and increases queueing when multiple pull requests are updated concurrently.

Limited reproducibility

Black is pinned in the prototype and Conda now uses strict channel priority. However, the environment still relies on:

• miniforge-version: latest;
• broad package version ranges;
• unpinned pytest and coverage tooling.

Package resolution and CI performance can therefore change between runs.

Remaining dependency and test warnings

The prototype addresses action-level warnings, but the logs still contain warnings originating from dependencies and the test suite, including:

• qhull-static, X11, and Qt file collisions;
• Matplotlib pending deprecations and unclosed figures;
• NumPy and SWIG deprecation warnings.

These are separate from GitHub Actions maintenance and may require dedicated issues.

Proposed follow-up

1. Run the prototype on representative pull requests.
2. Compare active runner time independently from queueing time.
3. Collect the new slow-test reports and rebalance the shards.
4. Evaluate caching Conda environments by OS, architecture, environment hash, and refresh date.
5. Consider a reduced pull-request matrix while retaining the complete matrix on master, scheduled runs, or an explicit trigger.
6. Evaluate a resolved environment or lockfile for more reliable caching.
7. Track dependency and application warnings separately.

Success criteria

• Establish a measured baseline and publish before/after results.
• Reduce the slowest test critical path without silently removing coverage.
• Demonstrate an improvement beyond the initial Codex estimate of 5–15%.
• Keep the complete test matrix available through a documented trigger.
• Document cache invalidation and the intended CI coverage policy.
• Avoid deprecated GitHub Actions runtimes and spurious Conda activation warnings.

[EveCharbie]

@mickaelbegon "cancels obsolete runs for the same pull request" is suspicious, did you check that nothing got deleted?

[mickaelbegon]

Explanation provided by Codex:

Good point — the wording is ambiguous. Nothing is deleted from the repository or from the Actions history.

The prototype uses GitHub Actions’ native concurrency mechanism:

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

For a new commit pushed to a PR, GitHub computes the same concurrency group for the new run. It then:

• cancels an older queued run from the same workflow and PR;
• stops the currently running one, if any;
• starts the workflow for the newest commit.

Because the group contains github.workflow, the Linux, macOS/Windows and Black workflows remain independent and cannot cancel each other.

A cancelled run remains visible in the Actions history with the cancelled status. Commits, branches, completed runs and their existing logs are not deleted. Steps that had not finished simply do not produce their results or artifacts.

I also checked that the workflows only have:

permissions:
  contents: read

They therefore have no permission or command for deleting repository content, workflow runs or artifacts. Cancellation is performed by GitHub’s scheduler, not by a deletion API.

The trade-off is that we deliberately stop testing an obsolete commit once a newer commit is available on the same PR. If retaining a complete CI result for every intermediate commit is required, we should remove cancel-in-progress: true.

[mickaelbegon]

Update from the Codex evaluation

Items 4 and 6 now have a concrete implementation on mickaelbegon:codex/update-github-actions, at commit 489e4217.

The prototype now includes:

• explicit Conda lockfiles for linux-64, osx-arm64, and win-64;
• CI test dependencies in the resolved environment;
• cache keys containing the runner OS, architecture, lockfile hash, and ISO week;
• a trusted weekly cache warmer on master;
• restore-only cache usage in pull-request workflows, including forked PRs;
• a fallback to an explicit lockfile installation, without querying channel indexes or running the Conda solver.

A real installation of the Apple Silicon lockfile was validated locally. Imports of bioptim, biorbd, NumPy, Pinocchio, and pytest succeeded. The installed environment is about 1.9 GB before cache compression.

Safety when a PR changes the environment

There is partial protection:

• If a PR changes a lockfile, its hash changes, so an environment cached for the previous lockfile cannot be restored under the new exact key. The PR installs the new explicit environment instead. This is safe but slower until the new cache is published from master.
• If the weekly key changes, the previous cache may be restored only as a fallback and is reconciled against the same explicit lockfile before use.

One important gap remains: if a PR changes only environment.yml but does not regenerate the committed lockfiles, the test workflows still use the old lockfiles. The cache key cannot protect against that because the resolved specification itself has not changed. The PR would therefore not actually test the requested environment change.

Recommended next step

The most effective next step is to add a fast consistency check that fails when environment.yml or the CI dependency specification changes without regenerating all three lockfiles. After that guard is in place, open a draft PR and compare:

1. the first run with a cold cache;
2. a rerun or subsequent commit with a warm cache;
3. Conda setup/restore duration and the total critical-path duration on each OS.

This will quantify the real gain—expected to be largest on Windows—before deciding whether the cache size and weekly refresh cost are worthwhile.

[mickaelbegon]

Codex update: stale-lock guard implemented

The recommended consistency guard is now implemented on mickaelbegon:codex/update-github-actions at commit 9424b95.

Before any Conda setup, the Linux, macOS/Windows, and cache-warming workflows now run a dependency-free checker that:

• fingerprints environment.yml and the CI dependency specification;
• rejects a source-only change until the lockfiles and recorded fingerprint are updated;
• verifies that all three platform lockfiles exist;
• validates their platform headers, conda-lock input hashes, explicit markers, package URLs, and package checksums.

The checker was validated in both directions: the current lockfiles pass, while a simulated environment.yml-only change and a malformed Windows platform header both fail before Conda installation.

This closes the accidental stale-environment gap identified above. It is intentionally not presented as a security boundary: a PR can also edit its workflow and fingerprint, so reviewers must still verify that source, fingerprint, and generated lockfile changes belong together. Shared-cache poisoning remains prevented because pull-request jobs only restore caches; only the trusted master workflow saves them.

The next efficient step is now a draft PR with one cold run and one warm-cache run to measure the actual critical-path gain per OS.

[mickaelbegon]

Codex assessment of PR #1061

PR #1061 still has a valid objective after the changes proposed here: the cache-warming workflow from #1085 does not run the Linux test workflow and therefore does not refresh the Linux CI badge on master.

However, #1061 should not be merged independently as-is without reconciling it with the #1085 branch:

• [HOLD] Trigger test workflows on master pushes #1061 starts the complete Linux and macOS/Windows suites on every push to master;
• [CI] Evaluate and reduce GitHub Actions test runtime #1085 adds a separate three-platform cache warmer on relevant master pushes and on a weekly schedule;
• when the environment or lockfiles change, both workflows could start at the same time, before the new cache exists, causing duplicated cold Conda installations;
• both changes edit the same workflow trigger sections, so the branches will also need a small conflict resolution.

Recommendation: preserve the intent of #1061 but fold it into the eventual #1085 PR instead of merging both implementations unchanged. Keep the Linux master-push trigger to refresh the badge. Keep the macOS/Windows master-push trigger only if full post-merge validation on every commit is worth the additional runner cost. The cache warmer and master test execution should be coordinated so that the warmer completes first, or the trusted master test run itself becomes the cache producer.

In short: #1061 still makes sense functionally, but its implementation should be integrated with #1085 rather than merged as a separate final design.

[mickaelbegon]

Codex update: PR #1061 intent integrated

The proposed integration is now implemented on mickaelbegon:codex/update-github-actions at commit ed70481.

Final behavior:

• the Linux test workflow runs for pull requests and pushes to master, preserving the default-branch result used by the README badge;
• on master, a trusted Linux cache-preparation job completes before the six Linux shards can start;
• that preparation job performs a lookup-only check when the current cache already exists, avoiding an unnecessary cache download;
• on a miss, it restores the previous compatible cache when possible, reconciles it with the explicit lockfile, validates the environment, and publishes the current cache;
• after an environment-related master push, the separate warmer handles only macOS and Windows, so Linux is not prepared twice;
• weekly and manual warming still cover Linux, macOS, and Windows;
• the twelve macOS/Windows test shards remain pull-request only, avoiding the additional post-merge runner cost from PR [HOLD] Trigger test workflows on master pushes #1061.

This keeps the functional reason for PR #1061 while removing the cold-install race and duplicated work that would result from merging both designs unchanged. PR #1061 is therefore functionally superseded by this branch and can remain on hold until the eventual #1085 PR is reviewed.

Validation: actionlint, YAML parsing, Black, lockfile consistency, and git diff checks all pass. No PR has been opened for the #1085 branch yet.

[ThomasVergeres]

This may be a useful fit for the bounded CI-cost experiment described here. I maintain a small GitHub-native workflow that produces an artifact summarizing recent Actions run frequency, duration, cancellation, and likely optimization candidates (no billing-grade dollar claims and no credentials leave the workflow): https://github.com/ThomasVergeres/actions-cost-sentinel

If useful, a maintainer could copy .github/workflows/actions-cost-sentinel.yml into a non-sensitive branch/repository and report one of: actionable finding, false positive/missing signal, or no value. It is specifically intended to test whether the signal is useful before building anything hosted or paid; no code changes are required beyond the optional workflow file. The issue's stated need for measured before/after CI evidence makes this a bounded trial candidate.