Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
120d876
tests: one table of live/published pairs, read by fixer, gate and sel…
OSideMedia Sep 26, 2026
f6ab766
publicise: assert both anchors, refuse unterminated blocks, normalise…
OSideMedia Sep 26, 2026
100867e
publicise: redact machine-state claims in the playbook and the seat w…
OSideMedia Sep 26, 2026
5ee19e1
check-upstream-sync: REFUSED is its own verdict; UNKNOWN never passes
OSideMedia Sep 26, 2026
4c6bda8
tests: check-release.sh — every tag's README badge must name that tag
OSideMedia Sep 26, 2026
a2c548b
README: Requirements and Install right after the pitch; install the s…
OSideMedia Sep 26, 2026
13896ec
ci: run every selftest that needs no live source, and assert the sync…
OSideMedia Sep 26, 2026
c3efdec
publicise: scan an unterminated last deny token; cover three untested…
OSideMedia Sep 26, 2026
b81b634
publicise: refusal classes drive the gate's advice; the rule table is…
OSideMedia Sep 26, 2026
dbe88ce
README: name the block rules' insertion behaviour
OSideMedia Sep 26, 2026
29909ee
publicise: guard the stage promotion; pin the gate's anchor advice
OSideMedia Sep 26, 2026
adfa8ed
tests: clear shellcheck warnings (SC1007, SC2154)
OSideMedia Sep 27, 2026
2dad57f
publicise: redact the availability-fallback section's owner and hook …
OSideMedia Sep 27, 2026
5d852d9
Refresh the published playbook and seat wrapper from live
OSideMedia Sep 27, 2026
a39860d
Refresh the seat wrapper from live: the no-cache check asserts its re…
OSideMedia Sep 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/workflows/selftests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Every check that can run without the author's live ~/.claude runs here. The sync gate
# cannot — it needs the live files — so the last step asserts it says UNKNOWN (exit 2)
# on a runner rather than passing on a subject set of nothing.
name: selftests

on:
push:
pull_request:

jobs:
selftests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # check-release.sh reads every tag; a shallow clone has none

- name: publicise selftest (every refusal goes red)
run: sh tests/selftest-publicise.sh

- name: codex-seat selftest (fallback proven both ways against a fake codex)
run: bash scripts/codex-seat.sh --selftest

- name: check-release selftest (throwaway repo, red and green)
run: sh tests/check-release.sh --selftest

- name: release hygiene (every tag's README badge names that tag)
run: sh tests/check-release.sh

- name: shellcheck
run: shellcheck -S warning tests/*.sh scripts/*.sh

- name: sync gate is UNKNOWN on a runner, never a pass
run: |
set +e
sh tests/check-upstream-sync.sh
st=$?
set -e
if [ "$st" -ne 2 ]; then
echo "expected exit 2 (SYNC UNKNOWN: no live source on this runner), got $st"
exit 1
fi
180 changes: 106 additions & 74 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[![Version](https://img.shields.io/badge/version-1.1.2-blue)](https://github.com/OSideMedia/model-council/releases/tag/v1.1.2) [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![Platform](https://img.shields.io/badge/platform-Claude%20Code-purple)](https://claude.com/claude-code) [![Seats](https://img.shields.io/badge/seats-Claude%20%2B%20GPT%20%2B%20Gemini-black)](docs/MODEL-PLAYBOOK.md) [![Consultants](https://img.shields.io/badge/consultants-read--only-orange)](commands/council.md)
[![Version](https://img.shields.io/badge/version-1.2.0-blue)](https://github.com/OSideMedia/model-council/releases/tag/v1.2.0) [![Selftests](https://github.com/OSideMedia/model-council/actions/workflows/selftests.yml/badge.svg)](https://github.com/OSideMedia/model-council/actions/workflows/selftests.yml) [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![Platform](https://img.shields.io/badge/platform-Claude%20Code-purple)](https://claude.com/claude-code) [![Seats](https://img.shields.io/badge/seats-Claude%20%2B%20GPT%20%2B%20Gemini-black)](docs/MODEL-PLAYBOOK.md) [![Consultants](https://img.shields.io/badge/consultants-read--only-orange)](commands/council.md)

# Model Council

Expand All @@ -16,6 +16,54 @@ disagree in useful ways — and a council is only as good as the judge, so every
**read-only consultant** and nothing becomes a "finding" until the overseer has verified
it in the code itself.

## Requirements

- [Claude Code](https://claude.com/claude-code) — runs the overseer and the Opus seat. Works out of the box.
- [Codex CLI](https://github.com/openai/codex) (`npm i -g @openai/codex`) — the OpenAI seat. Optional; an absent seat is recorded and skipped.
- [Antigravity CLI](https://antigravity.google) (`agy`) — the Gemini seat. Optional; same graceful degradation.
- `bash` and `python3` for the Codex seat wrapper (`--check` reads the models cache with
python3); a POSIX `sh`, `awk`, `diff` and `shasum` for the tests. All of these ship with
macOS and every mainstream Linux.

A council convenes with whatever answers. Missing seats are reported, never silently
papered over.

## Install

```sh
mkdir -p ~/.claude/commands
cp commands/*.md ~/.claude/commands/
cp docs/MODEL-PLAYBOOK.md ~/.claude/
cp scripts/codex-seat.sh ~/.claude/ && chmod +x ~/.claude/codex-seat.sh
```

The last line is not optional if you want a Codex seat: `/council` calls
`~/.claude/codex-seat.sh` for that seat and nothing else, so without the wrapper in
place the seat comes back empty every time — reported as empty, but empty.

## Use

In any Claude Code session:

```
/council Is the caching layer in src/cache/ actually safe under concurrent writes?
```

The session writes the brief, seats the council in parallel, verifies the load-bearing
claims, and reports per-seat attribution, consensus, disputes, and a verdict. It does
**not** act on the verdict — fixes are a separate instruction, so you stay in the loop
between judgment and change.

### When to convene (from the playbook)

Worth a council: architecture decisions that are expensive to reverse, pre-release
audits of money-touching code, a bug that survived two fix attempts, "is this design
actually good?" before a big build.

Not worth it: anything one consultant covers — a verification pass (Opus), a second
implementation (Codex), a mechanical sweep (a fast model fanned out). The playbook has
the full routing table.

## Why this shape

Most multi-model setups either let several agents edit the same code (merge chaos) or
Expand Down Expand Up @@ -46,6 +94,22 @@ absorb that failure mode.
| `commands/council.md` | The `/council` slash command — brief, seats, deliberation, report |
| `commands/audit-claude-md.md` | Bonus: `/audit-claude-md`, a rule-by-rule CLAUDE.md tuner for the judgement-era models (guardrails stay hard, preferences become guidance, procedures move to skills) |
| `docs/MODEL-PLAYBOOK.md` | The routing doc — which model gets which job, how to brief external CLIs, when a council is worth it vs. one consultant |
| `scripts/codex-seat.sh` | The Codex seat wrapper — the one file that holds the Codex model IDs, with the usage-limit fallback; `--check` and `--selftest` |
| `tests/` | The publishing gate: `publicise.sh` (the redacting transform), `check-upstream-sync.sh` (is the published copy still the live one?), `refresh-from-live.sh` (its fixer), `pairs.sh` (the one table of live ↔ published files), `check-release.sh` (every tag's badge names that tag), `selftest-publicise.sh` (proves every refusal goes red, no live source needed) |

## Model pins

The command pins today's frontier tiers (`gpt-6-astra` at high reasoning effort, with
`gpt-5.6-sol` as the fallback when the usage window is spent; `gemini-3.1-pro-high`)
and documents the recovery path for when vendors rotate slugs. The Codex seat does not
call `codex exec` directly: it goes through `scripts/codex-seat.sh`, the one file that
holds the Codex model IDs. It runs the primary slug and, only when Codex answers *"You've
hit your usage limit"*, re-runs the same brief on the fallback and says so on stderr.
`--check` compares both slugs against `~/.codex/models_cache.json` (UNKNOWN, not FAIL,
when that cache was fetched by a different client version than your CLI — the server
filters the list per client); `--selftest` proves every branch against a fake `codex`.
Councils never install or upgrade tooling themselves — a broken seat is reported with
its fix for *you* to run.

## Staying current

Expand All @@ -59,27 +123,47 @@ one that is visibly old.
So the pair is now checked rather than remembered:

```sh
sh tests/check-upstream-sync.sh # exits 1 on drift, prints the diff
sh tests/check-upstream-sync.sh # ok / STALE / REFUSED / UNKNOWN per file — see below
sh tests/refresh-from-live.sh # regenerates from live, then review git diff
sh tests/selftest-publicise.sh # proves the redaction refuses; needs no live source
sh tests/selftest-publicise.sh # proves every refusal goes red; needs no live source
sh tests/check-release.sh # every tag's README badge names that tag
```

The first two run the same `tests/publicise.sh` transform, so "the check is green" and
"running the fixer changes nothing" are the same statement and the two cannot drift
apart. That transform is the *only* sanctioned difference between live and published,
and it does two jobs. It **redacts** the handful of spots that are true on one machine
and misleading in public — a fold-in of local review-dimension files (seats cannot read
your disk, and a public command should not tell them to try), a routing line pointing at
a private source repo, and two parentheticals asserting which CLIs are installed. And it
**refuses**: every redaction asserts the live wording it is anchored to, and every output
is scanned against a deny-list of tokens that must never ship.
The first two run the same `tests/publicise.sh` transform over the same table of files
(`tests/pairs.sh`), so "the check is green" and "running the fixer changes nothing" are
the same statement and the two cannot drift apart. That transform is the *only*
sanctioned difference between live and published, and it does two jobs. It **redacts**
the handful of spots that are true on one machine and misleading in public — a fold-in
of local review-dimension files (seats cannot read your disk, and a public command
should not tell them to try), a routing line pointing at a private source repo, two
parentheticals asserting which CLIs are installed, and a few machine-state claims (a
hook that is wired in, a plugin that is installed, a dated private audit) rewritten as
what a stranger's machine can actually do. And it **refuses**: every redaction asserts
the live wording it is anchored to — both ends of a block, not only the line that opens
it — and every output is scanned against a deny-list of tokens that must never ship.

The refusal is the load-bearing half. An unguarded transform anchored on live wording
silently does nothing the moment that wording is reworded — it exits 0 and emits the
un-redacted text, the sync gate reports STALE, and the fix it prescribes then *publishes*
that text and goes green. A guard whose repair step launders the leak is worse than no
guard, so a failed assertion aborts the refresh with every published file untouched, and
`tests/selftest-publicise.sh` proves it by moving an anchor and watching the refusal.
`tests/selftest-publicise.sh` proves it by moving each anchor and watching the refusal.

The gate reports a refusal as its own verdict. `REFUSED <file>` carries the transform's
reason and the advice is to fix the anchor or the live wording — never to refresh, which
would publish exactly what was refused. `STALE` is the ordinary case, where the fixer is
the right answer. A file this repo does not publish is refused rather than passed through
ruleless (`--no-rules` is the explicit override), and a live source that is missing on
this machine is `UNKNOWN` with exit 2 whether one file is missing or all of them — never
a pass.

One property of the block rules to keep in mind when reading a refresh diff: a rule
replaces everything from its opening anchor through its terminating one. Live prose
inserted *between* the two — a sentence added inside a paragraph the rule redacts — is
dropped from the published copy along with the lines around it, with no refusal, and
shows up only as `STALE` at the next gate run. That is by design (the paragraph is being
replaced, not edited), but it means the refresh diff is where a live edit that should
have landed outside the block, or become a rule of its own, gets caught. Read it.

The deny-list itself is machine-local, and that is the point: a list of private strings
committed to a public repo publishes the exact strings it exists to withhold. The repo
Expand All @@ -89,70 +173,18 @@ ships only the format —
cp tests/publicise-deny.example ~/.claude/publicise-deny.txt # then fill in your own
```

— and the transform refuses to run without one, rather than deciding a file is safe to
publish while having no idea what private means here. An absent live source reports
`UNKNOWN` and exits 2, never a pass.

## Requirements

- [Claude Code](https://claude.com/claude-code) — runs the overseer and the Opus seat. Works out of the box.
- [Codex CLI](https://github.com/openai/codex) (`npm i -g @openai/codex`) — the OpenAI seat. Optional; an absent seat is recorded and skipped.
- [Antigravity CLI](https://antigravity.google) (`agy`) — the Gemini seat. Optional; same graceful degradation.

A council convenes with whatever answers. Missing seats are reported, never silently
papered over.

## Install

```sh
cp commands/*.md ~/.claude/commands/
cp docs/MODEL-PLAYBOOK.md ~/.claude/
```

## Use

In any Claude Code session:

```
/council Is the caching layer in src/cache/ actually safe under concurrent writes?
```

The session writes the brief, seats the council in parallel, verifies the load-bearing
claims, and reports per-seat attribution, consensus, disputes, and a verdict. It does
**not** act on the verdict — fixes are a separate instruction, so you stay in the loop
between judgment and change.

### When to convene (from the playbook)

Worth a council: architecture decisions that are expensive to reverse, pre-release
audits of money-touching code, a bug that survived two fix attempts, "is this design
actually good?" before a big build.

Not worth it: anything one consultant covers — a verification pass (Opus), a second
implementation (Codex), a mechanical sweep (a fast model fanned out). The playbook has
the full routing table.

## Model pins

The command pins today's frontier tiers (`gpt-6-astra` at high reasoning effort, with
`gpt-5.6-sol` as the fallback when the usage window is spent; `gemini-3.1-pro-high`)
and documents the recovery path for when vendors rotate slugs. The Codex seat does not
call `codex exec` directly: it goes through `scripts/codex-seat.sh`, the one file that
holds the Codex model IDs. Install it at `~/.claude/codex-seat.sh` (`chmod +x`); it runs
the primary slug and, only when Codex answers *"You've hit your usage limit"*, re-runs
the same brief on the fallback and says so on stderr. `--check` compares both slugs
against `~/.codex/models_cache.json` (UNKNOWN, not FAIL, when that cache was fetched by
a different client version than your CLI — the server filters the list per client);
`--selftest` proves every branch against a fake `codex`. Councils never install or
upgrade tooling themselves — a broken seat is reported with its fix for *you* to run.
— and the transform refuses to run without one, or with one that has no usable tokens
in it, rather than deciding a file is safe to publish while having no idea what private
means here.

## Contributing

Issues welcome. One structural thing to know before opening a PR: `commands/*.md` and
`docs/MODEL-PLAYBOOK.md` (and `scripts/codex-seat.sh`) are **generated**. They are a redacted copy of files that run
live at `~/.claude` on the author's machine, and the next `tests/refresh-from-live.sh`
overwrites them wholesale — so a PR editing those files cannot be merged as-is even when
it is right, and merging it would only mean losing your change at the next release.
Issues welcome. One structural thing to know before opening a PR: `commands/*.md`,
`docs/MODEL-PLAYBOOK.md` and `scripts/codex-seat.sh` are **generated**. They are a
redacted copy of files that run live at `~/.claude` on the author's machine, and the next
`tests/refresh-from-live.sh` overwrites them wholesale — so a PR editing those files
cannot be merged as-is even when it is right, and merging it would only mean losing your
change at the next release.

Open an issue instead: the fix goes in upstream and flows back down here. PRs against
`README.md`, `tests/`, or anything else in the tree are ordinary PRs.
Expand Down
31 changes: 21 additions & 10 deletions docs/MODEL-PLAYBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ than the headline rate suggests. The 2x is on what the chair writes and reads fr

### Opus 5 (Claude subagent, `model: opus`)
The default worker seat for anything touching code: verification audits, design review,
hard debugging, implementation. $5/$25 — half the chair's rate. Precedent: the 2026-07-15
Opus verification audit caught real findings a single pass missed. Use for "is this
hard debugging, implementation. $5/$25 — half the chair's rate. A verification pass at
this seat has caught real findings a single pass missed. Use for "is this
actually correct?" passes over shipped work, and adversarial review of a plan.

### A cheaper tier (`model: sonnet` / `model: haiku`)
Expand All @@ -49,31 +49,42 @@ needs to hold a lot at once. Reach for one Opus 5 subagent at a lower effort bef
Sonnet passes.

### Codex — GPT-6 / GPT-5.x (`codex exec`)
Independent second implementation, stubborn-bug rescue, cross-vendor code review. Already
wired into the harness via the codex plugin (`codex:rescue` for fix work); for
Independent second implementation, stubborn-bug rescue, cross-vendor code review. Fix
work can go through the codex plugin (`codex:rescue`) if you have it installed; for
opinion-only work call
`~/.claude/codex-seat.sh --sandbox read-only -c model_reasoning_effort="high" -C <repo> - < <brief-file>`
(brief via stdin — matches the /council seat exactly) so it can read the code but not
touch it. The wrapper is the ONE file that holds the Codex model IDs: gpt-6-astra first,
and when Codex reports the usage window spent it re-runs the same brief on gpt-5.6-sol
and says so on stderr — never pass `-m` to it. The second-opinion Stop hook goes through
the same wrapper. The codex plugin does not: it leaves the model unset and inherits
`~/.codex/config.toml` (also gpt-6-astra, set from the Codex app), so if `/codex:rescue`
dies on a usage limit the manual fallback is `/codex:rescue --model gpt-5.6-sol …`. Its
and says so on stderr — never pass `-m` to it. Point every Codex caller you wire up (a
Stop hook, a plugin, an ad-hoc `codex exec`) at the same wrapper. The codex plugin does
not go through it: it leaves the model unset and inherits `~/.codex/config.toml`, so if
`/codex:rescue` dies on a usage limit the manual fallback is
`/codex:rescue --model gpt-5.6-sol …`. Its
value is exactly that it is NOT Claude — different training, different blind spots.

### Gemini via Antigravity CLI (`agy`)
Cross-vendor tiebreaker and alternative design perspective. Google retired the old
`@google/gemini-cli` OAuth for individual accounts on 2026-06-18; the supported
terminal path is now the Antigravity CLI (`agy`, auths through the Antigravity
subscription, self-updates). Headless:
`agy -p "<brief>" --model gemini-3.1-pro-high --print-timeout 5m` — use
`agy -p "$(cat <brief>)" --model gemini-3.1-pro-high --disable-slash-commands --sandbox --print-timeout 10m` — use
`gemini-3.1-pro-high` for council seats, a `gemini-3.6-flash-*` tier for quick checks.
Headless runs auto-deny all tool permissions, so the seat answers from the brief alone —
inline the relevant code in the brief. NEVER pass `--dangerously-skip-permissions`.
(`agy` also exposes claude-* and gpt-oss models; ignore them — Claude seats run as real
subagents and Codex covers OpenAI.) The Antigravity IDE itself stays manual-only.

## Availability fallback

When Codex, a council seat, or any review tool is unavailable —
usage limit, not installed, auth dead — the STEP still happens, with the strongest
available Claude: a fresh Opus/Fable subagent reads the raw diff (never the lane that
wrote it), and the result is labelled "fallback read (<seat> unavailable)". Use the real
seat the moment it is back; never skip the step, never wait for the seat. A council with
fewer than two external seats is a "single-consultant review" by name and still runs.
A Stop hook that runs a Codex review can name this fallback in its message when Codex is down, instead of going quiet.

## The two dials

Choosing a seat is two decisions now, not one. **Which model** is a question about shape.
Expand All @@ -99,7 +110,7 @@ rewrite — stays at `high` unless `xhigh`/`max` measured a gain: above `high` t
may draft the whole deliverable in its thinking and then write it out again, doubling
the turn for no better result. The reason `max` is reserved for the last row
is measured, not stylistic: those are the passes where a miss is expensive AND invisible —
the fail-open sweep, the ledger that was 3.6× low with every row present, a map that
the fail-open sweep, a spend ledger that under-counted with every row present, a map that
repeats its own error and looks identical to a correct one from the inside.

Set the dial; do not paste in verification boilerplate written for older models. Telling a
Expand Down
Loading
Loading