Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ versions follow Semantic Versioning.
footprint, warning band, limit, emergency line, and the SIGTERM marker.
`scripts/render_limit_figure.py` draws it from the committed report, and a test fails when the
figure and the evidence drift apart.
- The `mlx-train-perf` integration proof, re-recorded on the published 0.2.0 wheel against
`mlx-train-perf` 0.8.0, the first consumer release that ships the integration
(`evidence/v0.2.0/mlx-train-perf/`): a checkpointed wall-limit intervention with the partial
artifact and matching request ids, the failure paths each consumer test pins, and the four
points of client friction the first consumer met. `docs/integrations/MLX_TRAIN_PERF.md` now
cites this bundle; the 0.1.0-era one stays as the record of the first adapter.

### Changed

Expand Down
17 changes: 11 additions & 6 deletions docs/integrations/MLX_TRAIN_PERF.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,13 @@ the `mlx-guard` policy engine remain application-neutral.
The runner is the supervisor's launching parent for the lifetime of a supervised condition. Under
`mlx-guard`'s default `on_parent_exit="terminate"`, a runner process that dies — a crash, an operator
`kill`, a scheduler eviction — now takes the supervised worker down with it instead of leaving it
running as an orphaned background process. The [evidence bundle](../../evidence/v0.1.0/mlx-train-perf/README.md)
predates this default; none of its recorded scenarios (checkpoint acceptance, a missing
acknowledgement, a callback error, client cancellation, or a report/persistence failure) involve the
runner process itself exiting, so its measured numbers are unaffected by this change. A runner that
deliberately wants a condition to outlive it needs `on_parent_exit="detach"`.
running as an orphaned background process. The [evidence bundle](../../evidence/v0.2.0/mlx-train-perf/README.md)
was recorded under this default, on the published 0.2.0 wheel, against the first `mlx-train-perf`
release that ships the integration. None of its recorded scenarios (checkpoint acceptance, a
callback error, client cancellation, or a report/persistence failure) involve the runner process
itself dying. The interrupted-runner runs it reports are client cancellations: the
runner stays alive, cancels the supervisor, and waits for it. A runner that deliberately wants a
condition to outlive it needs `on_parent_exit="detach"`.

The worker may connect `mlx_guard.CheckpointWorker` when the inherited checkpoint descriptor is
present. It polls only at safe step or repetition boundaries. On a request, the callback writes and
Expand Down Expand Up @@ -70,4 +72,7 @@ This integration adds no `mlx-train-perf` condition kind, status, or policy rule
The native report and the benchmark artifact keep separate schemas and responsibilities.

The bounded M1 Max acceptance run and failure-path results are published in the
[`mlx-train-perf` evidence bundle](../../evidence/v0.1.0/mlx-train-perf/README.md).
[`mlx-train-perf` evidence bundle](../../evidence/v0.2.0/mlx-train-perf/README.md), recorded on the
0.2.0 wheel against `mlx-train-perf` 0.8.0. The
[0.1.0-era bundle](../../evidence/v0.1.0/mlx-train-perf/README.md) stays as the record of the first
adapter.
80 changes: 80 additions & 0 deletions evidence/v0.2.0/mlx-train-perf/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# mlx-train-perf integration proof

This bundle records a bounded end-to-end run through the consumer's shipped runner boundary, on
the published mlx-guard 0.2.0 wheel. The consumer was mlx-train-perf 0.8.0 at commit
`349c2babd6e7093ce24fc20f1e73367c6bab409b`, the first release that ships the integration
(`run_conditions(..., guard=ExternalGuardConfig(...))`, installed with the `guard` extra, which
pins `mlx-guard==0.2.0`). The host was an Apple M1 Max with 32 GB of unified memory, macOS 27.0,
MLX 0.32.0, and mlx-lm 0.31.3. It replaces the [0.1.0-era bundle](../../v0.1.0/mlx-train-perf/README.md)
as the current proof; that one stays as the record of the first adapter.

The runner launched a 1,000-repetition real MLX loss-layer condition with a 512 MiB OS-accounted
footprint limit, a 500 ms wall limit, 10 ms sampling, and a 1 s checkpoint acknowledgement
timeout. At 501 ms the supervisor requested a checkpoint. The worker finished repetition 278,
atomically wrote and synced the artifact normalized as [`partial-result.json`](partial-result.json),
and returned path-free `file` metadata (636 bytes). The supervisor accepted the authenticated
response at 531 ms, sent TERM to the owned process group, and finalized at 554 ms with zero final
footprint. The complete [`report.json`](report.json) contains 43 samples, a 48,464,616-byte maximum
aggregate footprint, and no artifact errors. [`report-summary.json`](report-summary.json) preserves
the policy, checkpoint (with its `request_id`, `reason` and `artifact`), signals, transitions,
outcome (with `child_status`), capabilities, privacy declaration, and hashes of the source
artifacts. [`supervision-record.json`](supervision-record.json) is what the consumer's runner wrote
beside the report from the same run: the outcome kind, how the worker itself ended, and the
checkpoint status, request id, reason and artifact. The request id in the partial artifact matches
the one in the report.

Unlike the 0.1.0 run, this one was recorded under `on_parent_exit=terminate`, the default since
0.2.0, and with keyword-only configurations.

## Failure-path verification

Each path is pinned by a consumer test. Those marked real run the published supervisor and real
workers; the rest use a stand-in for the package.

- Normal completion (real): a supervised condition finishes `ok` and the report says
`child_exited`, code 0.
- Policy intervention (real): the run above, as a test, asserting the acknowledged checkpoint, the
`wall_time` reason, and matching request ids.
- Retry after an intervention (real): a second launch of the same condition is not blocked by the
retained journal, because every launch gets its own report path.
- In-process protection under supervision (real): with no supervisor limit able to trip, the
consumer's own wall backstop aborts the worker; the report records `child_exited` with code 70
and the consumer classifies it as the worker's status, by outcome kind rather than exit code.
- Callback error (real): with the artifact directory read-only, the callback's write fails; the
report shows `requested_unverified` with no artifact, no partial file exists, and the worker
stays up to be stopped by TERM (`policy_intervention`, child signaled 15).
- Missing package, broken install, failed discovery at `start()`: one recorded direct-launch
fallback, announced on stderr before the worker starts.
- Discovery error after the run, report or persistence failure, client failure: never a second
launch; the failure is recorded beside whatever artifact the worker left.
- Runner interrupted while waiting (four hand-driven runs, not included in this bundle): the
consumer cancels the supervisor and waits a bounded moment; all four ended `child_signaled`,
signal 2, with nothing left running.

## Integration friction

Four things the first consumer had to learn from the 0.2.0 client, recorded here because the next
consumer will meet them too.

The direct-launch fallback has to be decided by call site, not by exception type. The client
re-verifies the binary while loading the final report, so `SupervisorDiscoveryError` can also
surface from `wait()`, after the command has run. Only a discovery error from `start()` is safe to
fall back on.

A report path cannot be reused. The supervisor retains `.<name>.journal` beside each report and
refuses a path whose journal exists, so a consumer that retries work needs a fresh report path per
attempt.

A broken install raises `PackageNotFoundError` at import, whose `name` is the distribution name
`mlx-guard`, not the import name. A consumer that checks only for `mlx_guard` re-raises it.

A worker should survive a failed checkpoint. `poll()` raises after it has sent the failed
acknowledgement; a slow-exiting MLX worker that died there was sometimes gone before the TERM
arrived, and that run was reported as `supervisor_failure`. Catching `CheckpointError` around
`poll()` and continuing until the signal avoids the race.

No consumer-specific exception, status, condition kind, or policy rule was added to mlx-guard core.
The consumer's wired limit, active-memory watchdog and wall backstop remain the direct-launch
fallback and stay active inside supervised workers. Native reports need a private owner-controlled
directory; the runner creates `_mlx_guard/` with mode 0700 and refuses one that is a symlink or
owned by another user.
28 changes: 28 additions & 0 deletions evidence/v0.2.0/mlx-train-perf/partial-result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"identity": {
"schema_version": 1,
"mlx_version": "0.32.0",
"mlx_lm_version": "0.31.3",
"machine": "arm64",
"macos": "27.0",
"code_sha": "03ccb1e2a477a813",
"package_version": "0.8.0",
"kind": "loss_layer",
"session_id": "806f1ca290b341bf95b4f8deed74604d",
"n": 1024,
"d": 256,
"v": 2048,
"dtype": "float32",
"impl": "naive",
"reps": 1000,
"seed": 11
},
"status": "checkpointed_partial",
"checkpoint_request_id": 4860816705003653979,
"supervisor_deadline_ns": 1501892833,
"progress": {
"stage": "loss_layer",
"completed": 278,
"total": 1000
}
}
108 changes: 108 additions & 0 deletions evidence/v0.2.0/mlx-train-perf/report-summary.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
{
"source": {
"report_sha256": "03bcb9e6d3634f10c51019e97225bb29d952454c8fc541b4740a2ee240853272",
"partial_result_sha256": "4f19bcd113497bbca576d4f81bd45999fe3b1ca049a307dc5634c837c1cfe4b3",
"supervision_record_sha256": "2be3a444212c194c5d51a023a8cd19e5d15151fc1987b2d3a97b1c120a8f6a4e",
"sample_count": 43,
"max_aggregate_footprint_bytes": 48464616
},
"schema_version": 1,
"package_version": "0.2.0",
"configuration": {
"mode": "enforce",
"max_footprint_bytes": 536870912,
"warning_footprint_bytes": 483183821,
"recovery_footprint_bytes": 429496730,
"emergency_footprint_bytes": 590558003,
"required_breach_samples": 2,
"max_missing_samples": 3,
"wall_time_ms": 500,
"sample_interval_ms": 10,
"max_sample_age_ms": 500,
"max_sample_window_ms": 250,
"checkpoint_timeout_ms": 1000,
"term_grace_ms": 1000,
"on_parent_exit": "terminate",
"parent_watch": "active"
},
"checkpoint": {
"status": "acknowledged_unverified_durability",
"at_ms": 531,
"request_id": 4860816705003653979,
"reason": "wall_time",
"artifact": {
"kind": "file",
"size_bytes": 636
}
},
"outcome": {
"at_ms": 554,
"kind": "policy_intervention",
"final_footprint_bytes": {
"status": "available",
"value": 0
},
"child_status": {
"status": "signaled",
"signal": 15
}
},
"signals": [
{
"at_ms": 501,
"signal": 30,
"target": "cooperative_endpoint",
"result": "delivered",
"reason": "wall_time"
},
{
"at_ms": 531,
"signal": 15,
"target": "owned_process_group",
"result": "delivered",
"reason": "wall_time"
}
],
"transitions": [
{
"at_ms": 501,
"from": "normal",
"to": "checkpoint_requested",
"aggregate_footprint_bytes": 48464616
},
{
"at_ms": 531,
"from": "checkpoint_requested",
"to": "terminating",
"aggregate_footprint_bytes": 48464616
},
{
"at_ms": 554,
"from": "terminating",
"to": "exited",
"aggregate_footprint_bytes": 0
}
],
"artifact_errors": [],
"capabilities": {
"darwin_footprint": {
"status": "available",
"value": true
},
"owned_process_group": {
"status": "available",
"value": true
},
"checkpoint_channel": {
"status": "available",
"value": true
}
},
"privacy": {
"redacted_before_persistence": true,
"file_mode": "0600",
"upload": "disabled",
"retention": "user_managed",
"capture": "redacted_metadata_only"
}
}
Loading