Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 47 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,54 @@
# Maida Behavioral Regression Gate Action
# Add Maida's PR gate

Maida runs your traced AI agent in CI, compares its behavior with a baseline and policy, and posts a Markdown report on the PR. Reports show PASS, FAIL, or INCONCLUSIVE so reviewers can inspect behavioral changes before merging.
**Check an agent change before merge.** Tests may say the answer or result looks fine while the agent's execution behavior regressed. After you have proved that distinction locally, this Action brings the reviewed checks into the PR and shows **PASS, FAIL, or INCONCLUSIVE** to reviewers.

## Get started
## Start with a useful local report

Start with the released CLI and inspect one failed check before configuring CI:
If you have not checked your coding agent yet, use the [coding-agent getting started guide](https://maida.ai/docs/getting-started/) first. With Python 3.12–3.14 and Claude Code in your own Git repository:

```bash
uv tool install "maida-ai==0.6.1"
maida demo --regression

cd my-repo
maida init

# Run one normal Claude Code task and exit the session.

maida check
# Then run the exact "View:" command printed by Maida.
```

The offline demo shows a regression verdict and the PR-comment preview without an API key or repository clone. Continue with the [coding-agent getting started guide](https://maida.ai/docs/getting-started/) and its [practical walkthrough](https://github.com/maida-ai/maida-tutorials/blob/main/guides/coding-agent.md): capture one bounded task in your repository, inspect the observation, review a small policy, and reproduce a change locally before adding the gate to a PR. Each step has an observable result; you do not need to read the Action reference first.
Approve setup and start a new agent session. **Success looks like `3 active checks passed`**, your task's trace ID, and its viewer command. If Maida is already in the project's uv environment, prefix its commands with `uv run`; plain `claude` works afterward. No tutorial clone is needed.

For example: `maida view 83aa19e3`. Use the command from your own report.

**Runs on your machine or CI runner. No Maida cloud account required.** Task evidence is not uploaded to Maida. Enabling this Action publishes the selected report to your GitHub repository; your agent's normal provider calls remain separate.

## Protect the next agent change

The first local check covers completion, recorded loops, and guardrail events. Next, [review the observed behavior, keep a baseline and policy, and reproduce a safe failure](https://github.com/maida-ai/maida-tutorials/blob/main/guides/coding-agent.md#protect-the-next-agent-change). Keep ordinary correctness tests and evals; Maida checks observed execution behavior, not answer correctness.

Instructions, skills, tools, model configuration, harness code, and application code can all affect behavior. Maida gates the resulting change whether a human or the agent authored it.

For a Python tool-calling agent, follow the [Python setup path](https://github.com/maida-ai/maida/blob/main/docs/python-agent.md). Install Maida into the project environment before importing it; `uv tool install` only installs the standalone CLI.
For memorable proof, the [canonical storefront demo](https://github.com/maida-ai/maida-tutorials/tree/main/demos/pr-gate) shows **green application tests approving $15 VIP shipping while Maida fails the agent change** for rewriting the protected regression test. It uses released Maida v0.6.1 and runs deterministically offline after installation. That stronger reviewed policy is separate from the first check's three built-in requirements.

## Add CI after the local check works
## Add the merge boundary

Use the released `v0.6.0` Action tag. For production workflows, pin its reviewed full commit SHA. Before enabling a required gate, complete the [consumer verification](docs/acceptance.md#verify-the-actual-post-accept-loop) for the pinned revision.
**Bring a repeatable task and reviewed requirements to CI.** An interactive session saved on your laptop is not a repeatable CI task. Before adding the workflow:

The current checkout defaults to hash-locked Maida `v0.6.1`; this engine sync is unreleased and will be bundled into a future Action release. The published Action `v0.6.0` still defaults to Maida `v0.6.0`. To use Maida `v0.6.1` with that released Action, set `maida-version: v0.6.1` on both the gate and acceptance capture; this explicit override bypasses the released dependency lock and emits a warning.
- Make the task repeatable with pinned agent/model configuration and explicit budgets; [coding-agent scenarios](https://maida.ai/docs/cli/scenario-run/) describe isolated comparisons.
- Keep the reviewed `.maida/policy.yaml` and baseline on the base branch.
- Prepare a trusted runner that records or imports exactly one completed Maida run per invocation. Install its dependencies and configure its required credentials in CI. Do not feed PR-controlled text into the runner command.
- Use [Action inputs and trace-source requirements](docs/usage.md#inputs) to connect that runner. For an existing traced Python entrypoint, the secondary [Python setup guide](https://github.com/maida-ai/maida/blob/main/docs/python-agent.md) explains `agent-script`.

For the Python integration, instrument your agent with `@trace` or `traced_run()`, install its project dependencies in the workflow, and check in `.maida/policy.yaml` and any baseline on the base branch. See the [setup reference](docs/usage.md#usage) for inputs and the [local gate commands](docs/usage.md#running-the-maida-statistical-gate-locally) for creating and reviewing a baseline.
The template below assumes you have prepared `./scripts/capture-agent-task` as that trusted runner; replace it with your actual command. A scenario's own verdict is not automatically an Action trace source: the runner must supply the completed run the Action evaluates.

Add `.github/workflows/maida-check.yml`:

```yaml
name: Agent Regression Check
on: [pull_request]

# Read-only by default; the gate job grants only its required writes.
permissions:
contents: read

Expand All @@ -46,28 +65,28 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0
persist-credentials: false
- uses: maida-ai/maida-assert@v0.6.0
- uses: maida-ai/maida-assert@8ba54c181ef26df9eaecd38bb46a107756297b28 # released Action v0.6.0
with:
agent-script: my_agent.py
baseline: baselines/my_agent.json
maida-version: v0.6.1
trace-command: ./scripts/capture-agent-task
baseline: .maida/baselines/agent.json
policy: .maida/policy.yaml
```

The `v0.6.0` Action includes blocking and acceptance features. Pin a reviewed SHA and follow the [required repository settings](docs/usage.md#blocking-mode-and-required-repository-settings). Blocking mode refuses FAIL, INCONCLUSIVE, unaccepted configuration changes and setup/publication errors; report-only mode is observational.
The Action revision above is the published release SHA. Explicitly selecting the released **Maida v0.6.1 engine** keeps CI aligned with your local check. This selection bypasses that Action release's bundled dependency lock and emits a warning; review [dependency integrity](SECURITY.md#dependency-integrity). Use the same engine selection and reviewed Action SHA for acceptance capture if you enable it.

## More information
**Configure required checks and workflow-file review protection, then verify an actual PR.** Blocking mode evaluates the trusted base policy and refuses FAIL, INCONCLUSIVE, unaccepted configuration changes, and setup/publication errors. Report-only mode is observational. Exit zero alone is not approval.

- [Inputs and token permissions](docs/usage.md#inputs)
- [Workflow examples](docs/usage.md#example-workflows), including imported traces and report-only runs
- [Policy examples](docs/usage.md#policy-example) and [configuration acceptance](docs/usage.md#explicit-configuration-acceptance)
- [Accept intentional changes from a PR](docs/acceptance.md#accept-an-intentional-change-from-a-pr) and [baseline write-back](docs/acceptance.md#baseline-write-back-engine)
- [Dependency integrity](SECURITY.md#dependency-integrity) and [release provenance](CONTRIBUTING.md#tagged-release-provenance)
- [v0.6.0 release notes](CHANGELOG.md#v060)
- [Contributor testing instructions](CONTRIBUTING.md#testing-the-action)
Follow the [required repository settings](docs/usage.md#blocking-mode-and-required-repository-settings) and [consumer verification](docs/acceptance.md#verify-the-actual-post-accept-loop) for your pinned revision. Test a safe failure and repair, and require fresh results on the new PR head after intentional acceptance. Generating the workflow alone does not establish merge enforcement.

## Security
## Investigate and accept a change

See [SECURITY.md](SECURITY.md) for private vulnerability reporting and repository security controls.
The PR report names failed checks and changes against the baseline. Reproduce the task locally and run the exact `View:` command printed in your local report; a CI trace ID is not automatically on your machine. Repair an unintended regression. Accept an intentional baseline or configuration change only after reviewing its evidence and reason, then require a fresh result on the resulting commit.

## Versioning
## Reference

See the [versioning policy](CONTRIBUTING.md#versioning) for compatibility, release tags, and legacy references.
- [Inputs, token permissions, and workflow examples](docs/usage.md).
- [Policy examples](docs/usage.md#policy-example) and [configuration acceptance](docs/usage.md#explicit-configuration-acceptance).
- [Intentional changes and baseline write-back](docs/acceptance.md).
- [Security and dependency integrity](SECURITY.md) and [release provenance](CONTRIBUTING.md#tagged-release-provenance).
- [Action release notes](CHANGELOG.md), [versioning](CONTRIBUTING.md#versioning), and [contributor checks](CONTRIBUTING.md#testing-the-action).
28 changes: 26 additions & 2 deletions tests/test_action_yaml.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import json
from pathlib import Path
import re
import tomllib

import yaml
Expand Down Expand Up @@ -360,11 +361,34 @@ def test_readme_starts_with_released_coding_agent_route_before_workflow_referenc
readme = README_PATH.read_text()
first_command = readme.split("```bash\n", 1)[1].split("```", 1)[0]
selected = _load_action()["inputs"]["maida-version"]["default"].removeprefix("v")
assert first_command == f'uv tool install "maida-ai=={selected}"\nmaida demo --regression\n'
assert f'uv tool install "maida-ai=={selected}"' in first_command
assert "cd my-repo" in first_command
assert first_command.index("maida init") < first_command.index("maida check")
assert first_command.index("maida check") < first_command.index('exact "View:" command')
assert "maida view 83aa19e3" in readme
for block in re.findall(r"```bash\n(.*?)```", readme, re.S):
assert not re.search(r"<[A-Z][A-Z_]*>", block)
assert "Claude Code task" in first_command
assert "3 active checks passed" in readme
assert readme[:readme.index("```bash")].count("\n") < 40
assert "https://maida.ai/docs/getting-started/" in readme
assert "maida-tutorials/blob/main/guides/coding-agent.md" in readme
assert readme.index("coding-agent getting started guide") < readme.index("```yaml")
assert _release_ref() in readme
assert "## Protect the next agent change" in readme
assert "$15" in readme and "VIP" in readme
for obsolete in ("unreleased", "wheel from main", "install_capture", "MAIDA_DATA_DIR", "--expect-status", "--no-loops", "--no-guardrails"):
assert obsolete not in readme
assert set(re.findall(r"maida-ai==([\d.]+)", readme)) == {selected}
assert set(re.findall(r"Maida (?:v)?(\d+\.\d+\.\d+)", readme)) == {selected}
for reference in re.findall(r"maida-ai/maida-assert@[^\s`]+", readme):
assert re.fullmatch(r"maida-ai/maida-assert@[0-9a-f]{40}", reference)
for block in _documentation_yaml_blocks()[:1]:
# The README's executable example uses an immutable release revision
# and explicitly selects the same engine the developer used locally.
steps = yaml.safe_load(block)["jobs"]["agent-check"]["steps"]
action = next(step for step in steps if step.get("uses", "").startswith("maida-ai/maida-assert@"))
assert re.fullmatch(r"maida-ai/maida-assert@[0-9a-f]{40}", action["uses"])
assert action["with"]["maida-version"] == f"v{selected}"


def test_documentation_workflows_use_current_action_version():
Expand Down
Loading