STALE-bench

STALE — Benchmarking Silent Semantic Merge Failures in Parallel LLM Agents.

Reproducible artifacts for the STALE benchmark: when a backlog of coupled tasks is split across parallel coding agents, each agent works from a snapshot that is already stale — and two changes that each pass their own tests can break once merged, with no git conflict. This bundle releases the validated instances, the agent-study outputs, and the corrected grading harness so every number is checkable and the benchmark is extensible.

The benchmark has two kinds of artifact:

1. Deterministic core — the validated coupled instances and their gold patches. Given the PR set, these are reproducible bit-for-bit, independent of any model.
2. Result data — the agent study outputs (interference, cross-harness, communication). These depend on a stochastic agent; we ship the raw outputs so every number in the paper is checkable, and the harness so they can be regenerated.

Layout

artifacts/
  instances/
    django.jsonl     417 validated coupled instances (the main study repo)
    sympy.jsonl      11   } construction reproduced on
    xarray.jsonl     12   } three further SWE-bench repos
    seaborn.jsonl    7    }
    requests.jsonl   3
    control.jsonl    23   disjoint-file control pairs (the zero point)
  patches/django/pr<n>/{code,test,meta}.{patch,json}   gold diffs for every involved PR
  results/
    real_delta_blind.jsonl   UNCORRECTED naive-harness interference on Django (918 coupled + 44 control runs); retained only to document the measurement artifact
    real_corrected.jsonl     CORRECTED harness, union grading + discarded test edits (834 runs over the 417 validated pairs): exactly 1 run with delta_blind>0 (a single test), 0 both-solved-and-break
    synth_comm.jsonl         synthetic tier: blind vs informed vs one-message comm
    synth_gpt55.json         synthetic tier reproduced with the stronger gpt-5.5 model (planted symbols; rules out memorization)
    realstale_phone2numeric.json  the original single real-grounded instance (gpt-5.5)
    realstale_batch.jsonl    real-derived TIER: 36 instances (12 real Django helpers x 3 mechanisms), each anchored to a real validated PR pair; per-instance blind-interference + comm-recovery rates
    realstale_batch_runs.jsonl    the 216 raw gpt-5.5 runs behind realstale_batch.jsonl
    openhands.jsonl          cross-harness reproduction on the UNCORRECTED metric (not used for any paper claim)
    comm_real.jsonl          naive communication on the UNCORRECTED metric (not used for any paper claim)
    readwrite.json           read--write / inheritance coupling per pair (UNCORRECTED metric; not used for any paper claim)

Instance schema (instances/*.jsonl)

{
  "id": "django__21197_21344",
  "repo": "django/django",
  "tier": "coupled",                 // or "control"
  "base_commit": "53645750...",       // validated shared base (F2P-checked)
  "prs": [21197, 21344],
  "per_pr": {                         // gold spec for each PR
    "21197": {"code_files": ["django/contrib/admin/options.py", ...],
              "fail_to_pass": ["test_..."]},   // the PR's own added tests
    "21344": {"code_files": [...], "fail_to_pass": [...]}
  }
}

A pair is a valid coupled instance iff, on base_commit: (i) both gold code patches apply and merge with no textual conflict, and (ii) for each PR its fail_to_pass tests fail on base+test-patch and pass on base+code+test (the base-validity check that rules out the false-negative pitfall in §3 of the paper).

Result schema (results/real_delta_blind.jsonl)

{"prs": [21197,21344], "trial": 0, "tier": "coupled", "base": "...",
 "delta_blind": 49,                          // tests passing under each diff alone, failing merged
 "solo_fail": {"21197": 0, "21344": 0}, "merge_ok": true}

Reproducing the paper numbers

All commands run from the repo root with PYTHONPATH=. and the cluster .env loaded (source scripts/load_env.sh). The deterministic stages need only a Python env + git + a GitHub token; the agent stages additionally need the Azure model in .env.

Paper claim	File / command
417 mined → validated coupled pairs (deterministic)	python -m pdbench.cluster.gen_candidates then pdbench.coupling.validate_pair; instances in instances/django.jsonl
Construction reproduces on SymPy/xarray/seaborn (11/12/7)	instances/{sympy,xarray,seaborn}.jsonl
Base-validity rejects ~1/3 of pairs	pdbench/coupling/validate_pair.py
Synthetic tier: interference≈staleness; precise message recovers 98%	results/synth_comm.jsonl; pdbench/synth/synthbench.py
Cross-model + no memorization: gpt-5.5 reproduces Δ=d on planted symbols	results/synth_gpt55.json
Real tier (corrected grading): mined pairs show no interference	results/real_corrected.jsonl (834 runs / 417 pairs: 1 run with Δ>0, 0 both-solved-and-break) via corrected pdbench/harness/run_pair.py; gpt-5.5: 72 valid runs, 0 interference (results/real_gpt55.jsonl); decisive control is gold-composition (gold patches solve both, 0 failures / 417)
It's reconciliation, not weak agents: gold composes cleanly	gold-composition check: 388 pairs scanned, 46 initially flagged were git-concurrency artifacts, serial re-verify = 0 real failures (pdbench/coupling/gold_compose_batch.py)
Real-derived tier: gpt-5.5 silently interferes 105/108 (97%), one message recovers 89/108 (82%)	results/realstale_batch.jsonl (+ realstale_batch_runs.jsonl); pdbench/synth/realstale_batch.py, anchors from pdbench/analysis/readwrite.py

⚠️ Methodological caution (see paper §Preliminary Study)

A naive harness over-reports real-tier interference for two reasons, both fixed in pdbench/harness/run_pair.py:

1. Agent test-edits + permissive git apply --3way of the gold test patch → a task's tests silently fail to apply in solo (vacuous pass) yet apply in the clean blind merge.
2. Solos graded on their own test labels, the merge on the union → a solo's code breaking a test outside its labels is invisible in solo and surfaces only in blind.

Fixed: discard agent test edits and grade every condition on the union test set, counting a pair as observable only when both agents solve their own task. results/real_delta_blind.jsonl is the uncorrected (artifact-inflated) data, retained only to document the artifact; results/real_corrected.jsonl is the corrected run. The earlier openhands.jsonl, comm_real.jsonl, and readwrite.json were computed on the uncorrected metric and are not used for any paper claim.

Regenerating from scratch

# 0. environment
source scripts/load_env.sh                       # Azure + GitHub token
python -m pdbench.multi.bootstrap <repo>         # clone + venv + editable install

# 1. deterministic: mine + validate coupled pairs  (-> instances/)
python -m pdbench.cluster.gen_candidates
python -m pdbench.multi.validate_batch <repo>

# 2. agent study: blind / sequential, graded into Delta_blind  (-> results/real_delta_blind.jsonl)
bash pdbench/cluster/agent_orchestrate.sh        # distributed across nodes

# 3. analyses
python -m pdbench.analysis.readwrite             # read-write / inheritance coupling
python -m pdbench.harness.comm_pair  <jobs> <out>   # naive communication, real tier
python -m pdbench.harness.oh_pair    <a> <b> <t> <base> <out>   # OpenHands cross-harness
python -m pdbench.analysis.plot_dist             # Figure: Delta_blind distribution

The deterministic core (step 1) is model-independent and reproduces bit-for-bit. Steps 2–3 are stochastic; the shipped results/ are one run, and multiple trials per pair are included so the reported tail and rates are checkable.