CHANGELOG.md

Changelog

All notable changes to madengine will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[2.4.0] - 2026-10-18

Added

• Spur (Crusoe) scheduler support (#157): Spur ships SLURM-compatible CLI shims, but srun cannot fan out across nodes (SLURM_PROCID is empty), scontrol show hostname[s] is unsupported, and the Raft-based control plane makes squeue/sacct eventually consistent. A new SpurDeployment backend reuses the SLURM template and presets but drives multi-node runs with a job array of single-node tasks that self-form the cluster through a shared-filesystem rendezvous, and detects completion from per-rank marker files instead of sacct. Select it with "slurm": {"scheduler": "spur"} — the slurm block is otherwise unchanged, so spur is reachable from both madengine build and madengine run --additional-context. Peers that do not see rank 0's MASTER_ADDR within slurm.rendezvous_timeout (default 900s) fail with a diagnostic instead of starting with an empty address. The srun-based node health preflight is skipped on spur (it cannot target a node there, and the --nodelist it pins conflicts with the job array), an explicit "deploy": "spur" selects the backend at run time as well as at build time, and monitoring gives up with a diagnostic if the array never appears in squeue instead of polling forever. Stock SLURM behaviour is unchanged.

• Containers no longer outlive a cancelled run: A cancelled CI job (or any SIGINT/SIGTERM/SIGHUP) left madengine's detached benchmark container running, holding the GPU until the host was rebooted — the GitHub Actions runner signals only the step's top-level shell and then kills that process tree, which a container started with docker run -d was never part of. Three layers now cover this, in decreasing order of what they can survive:

  • In-container dead man's switch: every container madengine starts runs a POSIX-shell watchdog as PID 1 that stops the container as soon as madengine stops refreshing that container's own heartbeat file in the bind-mounted workspace. Each container has its own file, so closing one does not stop another launched from the same directory. This holds even when madengine is SIGKILLed, since it needs nothing from the host. Images without a shell fall back to the previous cat PID 1, with a warning that the container is unwatched. --keep-alive opts out, since leaving the container behind is its purpose. Tunable via MADENGINE_HEARTBEAT_INTERVAL (default 15s) and MADENGINE_HEARTBEAT_STALE_AFTER (default 120s). A non-integer or non-positive value falls back to that default, and the interval is kept below the staleness threshold.
  • Signal handlers (core/lifecycle.py): drop the heartbeats, then tear down registered containers within a 6s budget — inside the ~7.5s the runner allows between SIGINT and SIGTERM — and exit 130. Every docker call during teardown has a hard deadline and runs in its own session, because a wedged GPU blocks the docker daemon too and a cleanup path that can hang is worse than none.
  • madengine cleanup: sweeps containers labelled madengine.session=* whose owning process is gone. Safe to run from if: always(), cron, or by hand on a shared runner — containers with a live owner are skipped. --all ignores ownership, --dry-run reports only.

  Teardown escalates docker stop → docker kill → docker rm -f → direct SIGKILL of the container's host PIDs (read from the cgroup, so it works when dockerd itself is unresponsive). Processes stuck in uninterruptible sleep inside the GPU driver cannot be recovered from userspace: those are reported rather than retried — a gpu_wedged.json marker is written to the state dir and madengine cleanup exits 5 (ExitCode.GPU_WEDGED) so CI can pull the runner out of rotation. A wedge is only claimed on evidence (D-state PIDs, or a container the daemon still reports after being told twice to remove it); an unresponsive daemon alone is not enough.

• llm-d deployment target (#181): Benchmarks an llm-d stack — vLLM/SGLang model servers behind a Gateway API Inference Extension InferencePool — from an ordinary single-pod Kubernetes Job, so the existing ConfigMap bundling, log streaming, result harvesting and perf.csv path are reused unchanged. Two modes: attach (llm_d.endpoint_url set: madengine benchmarks a stack it did not create and never installs or uninstalls anything) and managed (madengine installs three pinned helm releases, waits for model-server readiness, reads the endpoint off the live Gateway, benchmarks, then tears the releases down on the way out — on success as well as on failure). Target selection follows the existing Convention over Configuration rule: an llm_d key selects it, checked before k8s because an llm-d config carries both. The benchmark client itself is CPU-only — k8s.gpu_count defaults to 0 under llm-d so the load generator does not compete with the stack it measures. See docs/llm-d.md.

• --wait-for-gpu-memory (#214): A local run can wait until every GPU reports at most 512 MB in use. 0 checks once and fails if any GPU is over that threshold; a positive value is a wall-clock timeout in seconds. A run that cannot query GPU memory fails closed. SLURM and Kubernetes warn and ignore the flag. Documented in docs/cli-reference.md.

• madengine setup-page (#162): Writes a self-contained HTML picker from the current repo's models.json. The page lists models and context fields and shows the madengine run command to copy. It has no external assets, so it can be published as GitHub Pages. Options are --output / -o (default index.html), --title, --repo-url, and --install-cmd. The external publish workflow checks the model repo out with persist-credentials: false.

• Layered model configuration (docs/distributed-config.md): Run, site, model, and measurement settings are alternatives, not a stack — any one of them is enough, and a missing optional input is skipped. A site: block in mad-config.yaml is refused, because nothing would apply it.

Changed

• llm-d reporting and model identity: Result rows stamp deployment_type=llm-d, launcher=vllm, and serving_mode=disaggregated|aggregated (also the CLI Mode column). The selected MAD card's --model_repo fills llm_d.model.hf_repo / model.name when unset; an explicit value that differs from the card is rejected. When the card sets multiple_results, that CSV is preferred over log performance: lines.

• SLURM multi-node runs inside an existing allocation (#213): When SLURM_JOB_ID is set, a multi-node templated launcher runs in place with srun instead of nesting another sbatch. Single-node runs still submit sbatch. Each in-place run gets its own workspace and DeepSpeed hostfile (/tmp/deepspeed_hostfile_<job tag>.txt). Workspaces are madengine_job_<job id>_node_<rank> under SLURM_TMPDIR or /tmp, removed after a successful artifact copy, and reclaimed after 7 days by a later job from the same user.

• Stale multiple_results files: A normal run deletes a leftover results CSV before the model script so an appending script cannot report the previous run. --skip-model-run leaves the file. The Docker path keeps perf.csv, perf_super.csv, perf_super.json, and the --output basename. The slurm_multi wrapper keeps those three report names and perf.csv; it does not see a custom --output. MAD_CONTAINER_IMAGE records the build as skipped and the summary prints which pre-built image was used.

Fixed

• Layered config and SLURM preflight follow the file they were given: A model.serve block in mad-config.yaml is refused, because no launcher applies those flags. BENCHMARK_SCRIPT is taken from the way-4 file, then the card, then the submit-time override, and that choice selects the benchmark block. Manifest docker_env_vars reach SLURM and the Kubernetes pod environment, with a runtime value winning per key; a local run does not copy a saved manifest value over the environment Context just read. auto_cleanup_nodes stops only containers labelled madengine.session. A health check stands down only when every examined node was unreachable or unknown; measured dirty nodes stay excluded. A bracketed slurm.nodelist is expanded before the GPU-architecture probe. skip_gpu_arch submits from a unique per-run copy and leaves the build manifest in place; a copy that cannot be saved aborts before submission. SKIPPED rows use the card name. Session CSV offsets count records, a cwd offset is applied only to the cwd perf.csv, and --output that names that same file through another path is not appended to itself.

• A cancel tears down cluster work as well as local containers: The signal handler reaps local containers and then runs registered teardown before os._exit(130), because that exit does not unwind finally. An in-place SLURM step is signalled, a submitted SLURM or Kubernetes job is cleaned up, and a managed llm-d stack is uninstalled. --keep-alive containers are left running. madengine cleanup exits 1 when docker ps times out or fails, instead of reporting that no containers were found.

• Cancellation cleanup no longer misreports Docker state: A failed docker inspect is an unknown state, not a missing container, so a daemon or permission error is not recorded as a successful removal. madengine cleanup exits with a failure when a container is still present and was not wedged. An exited container that docker rm cannot delete is not reported as a wedged GPU. Signal-handler teardown stops escalating once its deadline has passed, including inside a single container's stop/kill/rm calls. Heartbeat files are replaced atomically and are not recreated after they have been stopped.

• Git safe.directory no longer aborts a worktree checkout: git config --global resolves the workspace .git before writing the setting. A worktree is a .git file whose gitdir sits outside the container bind mount, so the command exited 128 on images that ship git and failed the run. The setting is now written with git -C /, which does not open that repository. A missing git is still skipped, and a real git config failure still fails the command.

• Kubernetes Job names longer than 63 characters failed admission: Batch Jobs require the pod job-name label to equal metadata.name, and label values are capped at 63. Long card paths (for example llm-d MoE names) are now built with sanitize_k8s_object_name(..., max_total_len=63) so the Job name and label stay identical.

• Copying a model script directory stops when the destination already has that file (#171): The container copies the model script directory into the model directory. A name that already exists there fails the run instead of being overwritten.

• A card builds only its own Dockerfile variants: Prefix matching no longer builds a sibling card whose Dockerfile name merely starts with this card's. Context-specific Dockerfiles that belong to the same card are still built.

Docs

• Documented llm-d card-driven model identity, serving_mode / multiple_results reporting, Job naming (≤63), and MI250 path constraints (1 GPU/pod, no RDMA/MoRI/WideEP) in docs/llm-d.md, docs/deployment.md, docs/configuration.md, deploy/llm-d/mi250/README.md, and examples/llm-d-configs/. Example model.name values use full Hugging Face ids.

• Corrected llm-d docs that disagreed with the code: managed model.name must be the full Hugging Face id, hf:// URIs need model.size, helm releases use the model-qualified stem (MAD_LLM_D_RELEASE_PREFIX), and a failed model-cache Job is left for kubectl logs. Documented in-place SLURM workspaces, stale multiple_results deletion, and the MAD_CONTAINER_IMAGE skip summary.

• CLI reference documents madengine cleanup (--all, --dry-run, exit 5), madengine setup-page, and MADENGINE_HEARTBEAT_INTERVAL / MADENGINE_HEARTBEAT_STALE_AFTER. The README version badge is 2.4.0. Command lists include cleanup and setup-page, and the architecture diagrams include the llm-d target. The DeepSpeed hostfile example shows the in-place suffix.

[2.3.0] - 2026-10-09

Added

• --config YAML frontend for run and build (#121): A Hydra compose path fills the same additional_context dict as --additional-context. Config groups (scheduler, launcher, hardware, append-only +profile / +env / +tools / +data / +build) live under src/madengine/configs/ and ship in the wheel. --config cannot be combined with --additional-context or --additional-context-file. Inline key=value overrides win over a user YAML file. Empty lists and dicts are dropped so they do not shadow defaults. When a user YAML sets a distributed launcher, distributed.enabled: true must be set explicitly, because the default launcher: none group sets enabled: false. launcher=megatron is a Hydra group alias of launcher=megatron-lm; the launcher value is always megatron-lm. Documented in docs/configuration.md, docs/cli-reference.md, and docs/usage.md. Annotated templates are in examples/configs/.

• TraceLens reports and on-demand PyTorch traces (#167): madengine report tracelens analyzes traces a run already wrote (Kineto, rocprofv3 JSON, Perfetto) into operator, kernel, roofline, and collective reports. madengine report tracelens-compare diffs those workbooks. Host analysis is an optional extra, pip install 'madengine[tracelens]', so TraceLens' pinned protobuf and xprof stay out of workload images. The tracelens tool does the same analysis inside an isolated virtualenv. torch_profiler_dynolog captures a Kineto trace from an unmodified PyTorch workload. SLURM and Kubernetes collection now gather torch_profiler_output/ and tracelens_output/ per node. Documented in docs/profiling.md and docs/cli-reference.md.

• docker_run_cmd on build_manifest.json (#189): After a local madengine run, the exact docker run command is written onto the manifest next to the build command, base image, and digest. The write is best-effort and never fails the run. Downstream delivery tooling can reproduce the container invocation from that one file. An earlier draft wrote a separate commands.json; that file is not produced.

• Multi-node Kubernetes without ReadWriteMany storage (#219): results_layout and data_layout default to auto. A shared volume is used when the configured StorageClass can serve ReadWriteMany (or an existing madengine-shared-data claim already is). Otherwise each pod gets a ReadWriteOnce disk on the local-path class, pods wait until every indexed completion finishes, and performance CSVs are recovered from pod logs when the local disk disappears with the pod. A shared-volume copy is still preferred when it is present. Image pulls use the dockerhub-rocm secret and fall back to an existing Docker Hub login. Documented in docs/deployment.md, docs/configuration.md, and examples/k8s-configs/README.md.

Changed

• Unknown launcher names fail instead of running as one process (#174, behavior change): validate_launcher() accepts one spelling per launcher (torchrun, deepspeed, megatron-lm, torchtitan, primus, vllm, sglang, sglang-disagg, slurm_multi) plus the aliases slurm-multi and sglang_disagg. Anything else, including the old megatron string and the reporting sentinels docker and native, raises ConfigurationError with a suggestion. The check runs for CLI context, model cards, and manifests, on SLURM and Kubernetes. slurm_multi is rejected when a Kubernetes target is also set. The Hydra group launcher=native still means "no launcher": the translator clears distributed.launcher before validation. megatron-lm dispatch on both backends compares against that canonical name. sglang-disagg is included in the Ray GPU-visibility guard.

• SLURM reads distributed.master_port when distributed.port is omitted: Templated SLURM launchers already used distributed.port (default 29500). Kubernetes already accepted launcher.master_port, distributed.master_port, and distributed.port. A YAML group that set only master_port therefore changed the Kubernetes port and left SLURM on 29500. SLURM now uses port when it is set, and master_port otherwise.

Fixed

• Kubernetes runs honor runtime distributed (#221): The job template read the launcher only from additional_context.launcher and from deployment_config saved at build time. A run that passed distributed.launcher (for example vLLM, 4 nodes, 8 GPUs) was submitted as a single pod, and a model card n_gpus of -1 was used as the process count. Runtime distributed now wins over the manifest, n_gpus: -1 is not a process count, and multi-node vLLM and SGLang pods join the headless Service subdomain so pod-0 DNS resolves.

• Dynolog writes a Kineto trace for the training worker (#221): The trigger accepted the first process that imported torch and then exited, and the worker stopped calling optimizer.step() before its Kineto thread received the config. Short-lived tools no longer register. The trigger waits until the registered set settles and requests only the worker. On TheRock, the start script imports torch.profiler (iteration capture counts optimizer.step() only after that import), the request skips the torchrun launcher, and the trace is written on the container's own disk and copied out, because Kineto does not finalize a file on the workspace bind mount. A request that includes the launcher is reported as installed and then never finalizes the worker's trace. dyno gputrace has no --fail-on-no-process in the pinned v0.5.0 and exits 0 either way (#172); the trigger now reads processesMatched and stops when dyno rejected the request. Stopping the daemon no longer waits out a 20s grace period on a zombie reparented to the container's PID 1 (#170).

• slurm_multi honors the model card (#176): distributed.nnodes sizes #SBATCH --nodes when slurm.nodes was not set; an explicit slurm.nodes wins and warns. Launcher selection and the emitted env block use one resolver (deployment config, then the model card). multiple_results is read from the card and searched next to the model script. Angle-bracket placeholder DOCKER_IMAGE_NAME values are rejected at submit time. Non-numeric or non-positive distributed.nnodes is rejected. A self-managed launcher implies a SLURM deployment. Per-job perf aggregation no longer appends perf.csv onto itself. A stale multiple_results CSV from an earlier run is ignored.

• rocm-trace-lite keeps the requested mode, and a leftover database cannot pass an empty run (#220): rtl_trace_wrapper.sh decided --mode support by piping rtl trace --help into grep -q under pipefail, so RTL_MODE=lite and RTL_MODE=default were dropped. The probe now reads the help text after the CLI exits and matches the --mode flag, not the substring inside --model. Existing trace*.db files in the output directory are removed before the workload. An unreadable database is a read error. A missing merged trace.db still keeps per-process trace_*.db files.

• rocm-trace-lite records dispatches on TheRock and on torchrun workers: rtl trace sets HSA_TOOLS_LIB only on the process it execs, so a launcher that then starts torchrun ran with zero GPU ops. The wrapper exports HSA_TOOLS_LIB and RTL_OUTPUT=trace_%p.db, and sets HSA_TOOLS_DISABLE_REGISTER=1 so TheRock's rocprofiler-register does not swallow the tool library (RTL_WRAPPER_KEEP_REGISTER=1 opts out). On images without /opt/rocm, the pre-script rebuilds librtl.so from the installed package's release tag (ROCM_TRACE_LITE_GIT_REF, otherwise v0.3.3) and copies it onto the wheel path rtl trace actually loads. The rebuild also replaces hsa_amd_queue_create, which is how HIP 7.15 creates compute queues; replacing only hsa_queue_create records no dispatches. RTL_WRAPPER_ALLOW_EMPTY=1 keeps exit code 0 when the trace has no kernel dispatches. dummy_profiling's Kineto window uses a wait/warmup/active schedule and leaves tensor shapes off unless DUMMY_TORCH_PROFILE_SHAPES=1.

• Profiler output survives a directory change, a failed model, and a long TraceLens pass (#212): rocprof_wrapper.sh makes a relative rocprofv3 output directory absolute against the wrapper's cwd, so a launcher that cds (Primus run.sh) no longer writes the trace outside run_directory. rocprofv3_perfetto no longer passes --perfetto-trace-filename, which rocprofv3 1.3.5 rejects. Post-scripts run best-effort after a non-zero model exit; the run stays failed, and one collector failure does not skip the rest. run_pre_post_script honors a per-script timeout (0 means none; the default is still 600). TraceLens presets set 7200 on tracelens.sh. The TraceLens pre-script installs curl when Perfetto traceconv needs it, and a package-manager failure there is non-fatal.

• TraceLens reports real GPU traces (#170, #172, #173): rocprofv3 JSON is not always valid UTF-8; the analyzer sanitizes a copy and leaves the original trace. The analyzer's module fallback used to record every failed report as SUCCESS. --python no longer falls back to a TraceLens on PATH, which defeated the isolated venv. A trace with no GPU activity, and a torchrun launcher trace, are SKIPPED rather than a failed run. The collective report is built only from rank-labelled traces. When the filename has no rank (Megatron-Bridge <host>_<pid>.<ts>.pt.trace.json), the rank is read from Kineto's distributedInfo.rank in the head of the file, plain or gzip. rocprof reports still request the CSV sheets, including the short-kernel study; that writer does not also emit the .xlsx workbook. Credential-bearing dynolog and TraceLens URLs are kept out of xtrace. madengine report tracelens fails when the analyzer crashes without writing a summary.

• TheRock images: git, MIOpen logs, and TraceLens setup: Container runs no longer require git. The unconditional git config --global --add safe.directory aborted every run on images that do not ship it (#215); it now runs only when git is installed. dummy_prof uses python3 when python is absent. A gzip trace missing its trailer raises EOFError, which is recorded as unreadable while the summary is still written. The library-trace parser accepts both MIOpen(HIP): and TheRock's MIOpen: prefix. The TraceLens pre-script installs git and recreates a partial venv when the image has no ensurepip.

• rpd builds with upstream CMake (#200): rocmProfileData replaced make rlog rpd with make install. The pre-script now runs make install with CMAKE_ARGS disabling the remote helper and web viewer, and no longer applies the old Makefile sed workaround. ldconfig for librlog.so is unchanged from 2.2.1.

• apt-get update retries when Ubuntu archive signatures fail to verify (#218): apt 2.8 on some TheRock images reports a correctly signed archive as unsigned. The rpd and rocm-trace-lite pre-scripts mark the configured archives trusted and fetch the package lists once.

Docs

• Release version badge is 2.3.0. Removed the fabricated timeout_multiplier and service_account keys from the examples that still showed them. Kubernetes storage docs now describe the auto / shared / per_pod layout instead of requiring ReadWriteMany for every multi-node job. The launcher troubleshooting text matches Unknown launcher, and the master-port note matches SLURM port / master_port and the Kubernetes resolution order. The Hydra launcher=torchtitan group selects torchtitan, not torchrun.

[2.2.1] - 2026-09-24

Added

• madengine discover --full (renamed from --all): Outputs complete model cards (all tags and metadata) for every discovered model without needing to pass --tags — for programmatic/CI consumption of the full model catalog.

• madengine discover --json: Emits plain JSON only, with no status panel or formatting, for piping directly into other tools or CI/CD pipelines.

Fixed

• SLURM multi-node jobs hung when a node's own hostname resolved to loopback (#179): The generated job script resolved peer node IPs with getent hosts, which on Ubuntu returns a node's /etc/hosts self-mapping (127.0.1.1) before its real address, so every node published itself as loopback in the peer list every other node was waiting to connect to — collapsing multi-node rendezvous (e.g. the sglang-disaggregated socket barrier) until the job hit the SLURM time limit. Node IPs are now resolved via getent ahostsv4, ignoring 127.* entries, with the local node's address taken from the configured cluster interface (NCCL_SOCKET_IFNAME) or the kernel's outbound-routing source address instead of hostname -I (which can return a Docker bridge or management address). The resolved SGLANG_* environment variables are now also forwarded into the container, which previously only had an allowlist that didn't include them.

• madengine run misreported scheduler and multi-image/multiple-results failures (#187): A SLURM/Kubernetes deployment that failed before any model reported metrics (e.g. the job itself died) returned empty result lists, so the CLI's exit code read it as a clean run; a failed deployment with no per-model results now synthesizes a failure from the scheduler's own error. Failure attribution also resolved manifest keys to the underlying Docker image tag instead of the model's logical name, blamed a model once per build image instead of once overall (multi-arch/multi-Dockerfile models), and did not recognize a multiple_results row (reported as <model>_<row>) as that model succeeding — each is now corrected so failures are attributed to the right model exactly once, and models that already reported success are excluded from the failure list.

• rpd profiling tool failed with librlog.so: cannot open shared object file: The rpd pre-script builds rlog and cmake --installs librlog.so into /usr/local/lib, but the container's ld.so cache was never refreshed afterward, so runTracer.sh's LD_PRELOAD of librpd_tracer.so (which links -lrlog) aborted before the profiled command could start. trace.sh now runs ldconfig /usr/local/lib after building rlog/rpd, and the rpd tool's LD_LIBRARY_PATH in tools.json includes /usr/local/lib as a fallback in case ldconfig is unavailable.

[2.2.0] - 2026-08-12

Added

• Pinned image digest enforcement for madengine run (#180): A run could pull a different image than the one the build pushed, because registry tags are mutable and a concurrent push can move a tag between the build and the run. DockerBuilder.push_image() now always captures the sha256: digest from the docker push output (falling back to docker image inspect --format '{{index .RepoDigests 0}}' when the registry prints no digest line) and records it as image_digest in each build_manifest.json entry — purely additive and best-effort, a missing digest never fails the build. Enforcement is opt-in via madengine run --require-pinned-image (equivalently the require_pinned_image additional-context key, which is persisted into manifest["context"] so the nested madengine run on SLURM compute nodes inherits it). When enabled, the local Docker path, the K8s pod spec image, and the slurm_multi DOCKER_IMAGE_NAME (covering both the parallel srun docker pull and the docker run inside the model script) all resolve through resolve_pinned_image(), which rewrites the reference to repo@sha256:… or raises ConfigurationError when no digest is available. A moved tag now surfaces as a pull failure / ImagePullBackOff instead of a silent wrong-image run. New pure-helper module core/image_digest.py (parse_push_digest, parse_repo_digest, build_pinned_reference, resolve_pinned_image) with unit tests; documented in docs/cli-reference.md and docs/configuration.md.

• Per-model docker_build_arg honored from models.json (#175): Build args could only come from --additional-context, which is global to the invocation, so a model had no way to declare its own build-time pins (e.g. VLLM_REPO/VLLM_REF) in-repo — every caller had to repeat them on the command line, and two models needing different values could not be built together. build_image() now merges a model card's docker_build_arg into that model's own build only. Context, credential, and multi-arch args win on conflict, matching the precedence _pick_context_over_model() already applies to run-time keys, so an operator can still redirect a build without editing models.json. A malformed docker_build_arg (string or list instead of a JSON object) now fails early with a message naming the model and the offending type, rather than surfacing as an AttributeError deep in the build path. Documented in docs/configuration.md.

• slurm.skip_gpus_directive opts out of #SBATCH --gpus-per-node (#163): Clusters that don't advertise GPU GRES reject any job script carrying --gpus-per-node, failing submission before launch. Setting slurm.skip_gpus_directive: true omits the directive from the generated job.sh.j2, relying on exclusive/nproc_per_node instead. Defaults to false (directive still emitted).

• Reuse existing docker login (OAT) instead of requiring credential.json (#168): madengine previously required credentials to be duplicated into credential.json/env vars even when the machine already had a working docker login (e.g. an organization access token). has_ambient_docker_auth() (core/auth.py) now reads ${DOCKER_CONFIG:-~/.docker}/config.json the same way the Docker CLI does — covering auths, credHelpers, and credsStore — so an existing login is reused and blank placeholder credentials ({"username": "", "password": ""}) never override or break it. docker build --pull now only logs in to the base image's registry when there's no existing login to reuse. Base-image pull failures also distinguish insufficient_scope (authorization — token needs wider repo scope) from unauthorized/authentication required (login problem), and explain_registry_denial() now names the actual failing registry (e.g. ghcr.io) in its docker login/credential.json suggestions instead of always assuming Docker Hub. Documented in docs/configuration.md under a new "Registry Authentication" section.

Changed

• --timeout precedence restored to the v1 rule; default unified at 7200s (behavior change): The v2 rewrite spread timeout handling across five layers, and the sentinel was materialized into a concrete value at the CLI before the resolver ever saw it. Because the resolver then detected "the user did not pass --timeout" by comparing the value against the default, an explicit --timeout 7200 was indistinguishable from no flag at all and silently lost to a model card timeout — in v1 it won. The CLI now forwards the user's value verbatim and precedence is applied in one place: default < model card < explicit --timeout, where -1 means "not specified" and falls through, 0 means "no timeout" and beats the levels below it. Separately, the distributed path defaulted to 3600s while local execution used 7200s and docs/cli-reference.md documented 7200 as the default; SLURM runs now get the same 2h default as local runs. Resolution moved from execution/container_runner_helpers.py to core/timeout.py (re-exported from its old home) so the deployment layer shares one rule instead of reimplementing it, and build_orchestrator/run_orchestrator now agree that a manifest timeout of null means "the model card specified none".

  Upgrade note: a build_manifest.json written by 2.1.x or earlier stored -1 as filler for every model without a timeout key (model.get("timeout", -1)). Since a model card's -1 is now a real value meaning "no timeout", replaying such a manifest runs those models unbounded rather than at the 7200s default. Rebuild the manifest, or pass an explicit --timeout, when reusing one written before this change.

Fixed

• docker push no longer subject to the 60s command timeout (#180): push_image() called Console.sh() with the default 60s timeout, so pushing a multi-GB image failed an otherwise successful build with "Console script timeout". It now passes timeout=None like build_image() already did. Console.sh's timeout annotation was widened to Optional[int] to match the five call sites already passing None.

• Docker-legal container name derived from pinned image references (#180): Container names were built straight from the run image reference, mapping only / and : to _. Under --require-pinned-image the run image is repo@sha256:…, so the @ survived into --name and the daemon rejected it with Invalid container name. New container_name_from_image_ref() strips the digest, applies the same / and : mapping as before, and sanitizes any remaining out-of-charset byte; output was verified byte-identical to the previous expression across 60 non-digest references (bare tags, registry hosts, localhost:5000 ports, CI tags, dotted tags).

• Pinned pulls fail instead of falling back to a local tag (#180): Under --require-pinned-image, a failed registry pull fell back to the local image_name. Local tags are mutable too, so the fallback broke the pinning guarantee in exactly the failure mode it matters most — a digest/tag mismatch or auth error would silently run whatever the local tag pointed at. The pull failure is now fatal for that model; the surrounding handler still records it as a failed run and continues with the rest of the manifest.

• get_build_arg() KeyError on missing context key (#175): A non-empty run_build_arg combined with a context lacking docker_build_arg raised KeyError from the direct ctx["docker_build_arg"] index.

• Generated SLURM job script portable across clusters (#163): The rendered job.sh.j2 made several assumptions that don't hold on every cluster. It now inherits the submitter's PATH explicitly — a batch job isn't guaranteed to keep it (a site can default sbatch to --export=NONE, and module load can rewrite it) — by re-exporting $HOME/.local/bin and the directory madengine was resolved from at submission time (SlurmDeployment._submission_bin_dir()), so a job that passed the pre-submission availability check no longer fails on the compute node with "madengine not found in PATH". The pre-submission madengine --version probe's timeout was also raised from 5s to 600s, since a cold interpreter start off shared/NFS storage routinely exceeded 5s on a healthy environment. The single-node shared-workspace probe now reads the filesystem type alone via df --output=fstype (with an awk-based fallback for pre-8.21 coreutils) instead of grepping the whole df -T line, which previously classified a local disk mounted at a path like /mnt/nfs-scratch as shared storage; the type pattern also now matches nfs4 (not just nfs) and adds beegfs/panfs. Separately, RunOrchestrator's informational rocm-libs package-manager query (apt/yum/zypper/tdnf) is now wrapped in timeout 10 so a node prompting interactively (e.g. yum asking to import a repo GPG key) can no longer hang an entire multi-node run. Covered by a new tests/unit/test_slurm_job_template.py.

• multiple_results preserved in local-image manifest (#166): MAD_CONTAINER_IMAGE (local-image) mode built a synthetic manifest that omitted the model's multiple_results field from models.json. Without it, ContainerRunner never set MAD_OUTPUT_CSV or copied the perf CSV out of the container, instead falling back to scraping the run log for a "performance: NUMBER METRIC" line — reporting FAILURE even when the model produced valid perf-CSV results. RunOrchestrator now carries multiple_results through to the manifest it synthesizes for local-image runs.

• GPU tool detection honors non-default ROCm install paths and PATH (#168): amd_smi_utils.py, rocm_smi_utils.py, and gpu_info_profiler.py hardcoded /opt/rocm; they now honor $ROCM_PATH (falling back to /opt/rocm). gpu_info_pre.sh detects nvidia-smi/rocm-smi/amd-smi via command -v instead of a fixed binary path, and calls rocminfo only when it's available instead of failing the pre-script when it's absent. The rpd tracer's LD_LIBRARY_PATH is now ROCm-path-aware, and rpd2tracing.py failures fall back to saving the raw trace.rpd instead of losing the trace.

• --timeout 0 crashed instead of disabling the timeout: The CLI mapped 0 to None, and three consumers were unprepared for it. execution/container_runner.py and deployment/slurm.py guarded with a bare timeout > 0, raising TypeError: '>' not supported between instances of 'NoneType' and 'int' on the self-managed-launcher and in-allocation paths. The SLURM job template used {{ timeout | default(3600) }}, but Jinja's default filter only substitutes for undefined values, so None rendered the literal string --timeout None into the generated script, which Typer then rejected. Only an int crosses layer boundaries now, and the sentinel is mapped to subprocess/communicate semantics by a single named guard, core.timeout.subprocess_timeout() — needed because subprocess reads timeout=0 as "expire immediately", not "no timeout". A fourth site with the same latent bug (the model script invocation in container_runner.py, which reaches communicate()) was fixed at the same time. Regression tests added for each.

• SLURM runs lost their wall-clock timeout cap by default: _execute_distributed forwarded the CLI's -1 sentinel straight into DeploymentConfig.timeout instead of resolving it first, unlike the local path (which resolves in container_runner.py). subprocess_timeout(-1) maps to None, so the SLURM in-allocation path (slurm.py's _run_inside_existing_allocation) ran with no timeout at all on any run that didn't pass an explicit --timeout. _execute_distributed now calls resolve_run_timeout() before building DeploymentConfig, restoring the 7200s default cap. The generated SLURM job script's --timeout argument (job.sh.j2) no longer uses {{ timeout | default(3600) }}, which only ever caught undefined, not None.

• A model card's timeout was ignored on SLURM: SLURM resolves the timeout twice — once in _execute_distributed, and again inside the job, where the generated script re-invokes madengine run. Rendering the resolved value into that script made the inner run see an explicit --timeout, which correctly outranks the model card, so a model declaring "timeout": 3600 silently ran under the 7200s default instead. (Before the precedence fix above this was masked: the rendered 7200 happened to equal the constant the old resolver compared against, so it read as "no flag passed".) DeploymentConfig now carries the two values separately — timeout, the resolved cap on madengine's own wait, and cli_timeout, the sentinel forwarded verbatim for the in-job run to resolve against the model card itself. Local runs were never affected.

• A model card's timeout was never enforced on Kubernetes: job.yaml.j2 never referenced the timeout at all, so K8s jobs ran the model script unbounded regardless of what the model card or --timeout said. Unlike SLURM, K8s has no inner madengine run inside the pod to re-resolve against the card, so k8s_template_context.py now calls resolve_run_timeout() at render time and the template wraps the model script in timeout {{ timeout }}, treating a non-positive resolved value as "run unbounded" per v1 precedence. A script killed by the timeout surfaces as exit code 124 with an explicit model script timed out after Ns line in the pod log. Both the launcher and direct-script paths are covered.

• A failing or timed-out K8s model discarded its own results: The pod script runs under set -e and copies artifacts to the results PVC only after the model returns, so a non-zero model exit aborted the container before its post-scripts, perf.csv, and logs were published — the runs most worth diagnosing were the ones that left nothing behind. Both invocation branches in job.yaml.j2 now capture the exit code and defer to the single exit ${MODEL_EXIT_CODE:-0} at the end of the script. Pre-existing on the unbounded path; the timeout wrapper added above would otherwise have extended it to timeouts.

• Programmatically omitting timeout overrode model cards: RunOrchestrator.execute(), ContainerRunner.run_container(), and ContainerRunner.run_models_from_manifest() each defaulted the parameter to DEFAULT_RUN_TIMEOUT, but resolve_run_timeout() reads any non-negative value as an explicit --timeout. A caller that omitted the argument therefore forced 7200s over every model card, contradicting the documented precedence. All three now default to the -1 sentinel, which still resolves to 7200s when no card specifies one. The CLI was unaffected — it always passes a value.

Docs

• README rewritten as a concise landing page (#161): Trimmed the root README from 707 to 258 lines by moving deep reference material (profiling tables, extended config/usage recipes, tips) into docs/ and linking out. Replaced the stale ASCII architecture block and unreferenced docs/img PNGs with accurate inline Mermaid diagrams for the layered architecture, build→run→report pipeline, and deployment-target inference; added matching diagrams to docs/deployment.md and docs/README.md. Also corrects numerous stale references across docs: --csv-file → --csv-file-path/--file, missing database command flags (--unique-key/-k, --batch-size, --no-upsert, --no-index, --dry-run, MONGO_AUTH_SOURCE/MONGO_TIMEOUT_MS), wrong run --output/--tools-config defaults, megatron → megatron-lm launcher name, fabricated timeout_multiplier/service_account config keys, missing Kubernetes/SLURM additional_context keys, DOCKER_CONFIG/MAD_SKIP_DOCKER_LOGIN documentation, and corrected SGLang Disaggregated minimum node counts/split formula for SLURM vs. Kubernetes.

[2.1.3] - 2026-07-15

Added

• Unscoped tags match dir-prefixed model short names (#159): Per-directory models.json prefixes model names with their directory (e.g. pyt_foo becomes dir/pyt_foo), which silently broke --tags pyt_foo for users referencing models by their original flat name. DiscoverModels now falls back to matching the short name (the part after the last /) for unscoped tags, so existing --tags invocations keep working after the migration.

Fixed

• amd-smi path resolution in get_gpu_renderD_nodes (#156, ROCM-27727): amd-smi was invoked as a bare command, relying on PATH, while every other call site in core/context.py resolves it via self._rocm_path. On hosts where /opt/rocm*/bin is not on PATH, this caused GPU detection to fail with exit code 127. The call now resolves through self._rocm_path like the rest of the file.

Docs

• Stale directory-scoped model tag examples corrected (#158): Colon-separated tags (e.g. dummy2:dummy_2) are parsed as model name plus extra script args, not directory scope. Directory-scoped selection uses scope/tag syntax (e.g. dummy2/model1) since #109. README and docs/{cli-reference,usage}.md examples updated accordingly.

[2.1.2] - 2026-07-13

Changed

• Log error-pattern match no longer overrides valid performance metrics (#154, ROCM-27774): The post-run log scanner greps the entire run log, including a model's own generated stdout, so a generative benchmark (e.g. an LLM emitting "ValueError:" in a code sample) could flip an otherwise-successful run to FAILURE. Since the scan cannot distinguish framework/harness diagnostics from model output, valid extracted performance metrics now take priority: a pattern match no longer fails a run that produced valid perf data — the match is still surfaced (in yellow) for triage instead of silently failing. Status logic was extracted into a new resolve_run_status() helper (execution/container_runner_helpers.py) with unit tests. When no valid metrics exist, a pattern match still fails the run as before.

• sglang-disagg SLURM launcher supports a co-located proxy (#149): The launcher previously hard-required a dedicated proxy node and a minimum of 3 nodes (1 proxy + 1 prefill + 1 decode), unlike vllm-disagg which leaves topology to the model script. It now also accepts a co-located proxy/router on the first prefill node, lowering the minimum cluster size to 2 nodes. Custom-split validation accepts both topologies — dedicated proxy (1 + xP + yD == nnodes) or co-located (xP + yD == nnodes); the proxy is started by the model run.sh, so both layouts are valid. Docstrings, error messages, and the generated cluster-config header were updated accordingly.

Fixed

• sglang-disagg default split invalid for 2-node co-located topology (#149): The default (no custom prefill/decode) split assumed a dedicated proxy and computed yD = nnodes - 1 - xP, yielding yD=0 for nnodes=2 — an invalid topology that contradicted the new 2-node co-located minimum. nnodes == 2 is now special-cased to a co-located 1 prefill + 1 decode split. Unit tests added in tests/unit/test_slurm_multi.py cover the default-split path across supported node counts.

Security

• MAD_SECRETS_* values redacted in printed/raised commands (#150): docker run/docker build commands and the "Docker options:" line were printed with MAD_SECRETS_HFTOKEN (and similar) in plaintext, leaking the HF token into SLURM and run logs. A central redact_secrets() helper (core/console.py) now masks secret values at every command print/exception site — Console.sh stdout, the raised RuntimeError message, and container_runner Docker options. Only the logged representation is scrubbed; the executed command is unchanged. The MAD_SECRETS*=value matcher handles unquoted, single-/double-quoted (including spaces), and empty values, with optional whitespace around =; a fallback also masks known token shapes (hf_, sk-, ghp_/gho_/…, xoxb-/…). Covered by a new TestRedactSecrets integration suite.

[2.1.1] - 2026-06-02

Changed

• All dependencies are now included by default: Kubernetes support (kubernetes>=28.0.0) and development tools (pytest>=7.0, black, mypy, isort, pre-commit, etc.) are bundled into the base dependencies list. The [kubernetes], [dev], and [all] extras have been removed — a plain pip install madengine or pip install -e . installs everything. All documentation and in-package install guidance has been updated accordingly.

• pytest lower bound pinned to >=7.0: Aligns the dependency pin with minversion = "7.0" already declared in [tool.pytest.ini_options], preventing accidental resolution of older pytest versions that cannot run this project's tests.

Changed

• --skip-model-run now matches v1 semantics: The flag previously short-circuited the entire run before any container started, and only took effect when a build ran in the same invocation (otherwise it was ignored with a warning). It now starts the container and runs pre_scripts and post_scripts as normal, skipping only the model script invocation — regardless of whether a build ran. The skip decision was moved out of RunOrchestrator and into ContainerRunner, so it applies uniformly to build+run and manifest-only (--manifest-file) invocations.

• --skip-model-run runs report SKIPPED, not FAILURE: A skipped model is now aggregated as a successful run with status SKIPPED, and the overall workflow exits 0. Previously a skipped run could surface as a failure.

Added

• --skip-model-run --keep-alive for live container debugging: Combining the two flags leaves a fully-set-up container alive after the skipped run, ready for manual exec (docker exec -it <container> bash). When --keep-alive is set, the run prints the exact cd <model_dir> && <script> <args> command to invoke the model by hand; otherwise it hints to re-run with --keep-alive.

• Warning for local-only flags on distributed targets: Passing --skip-model-run, --keep-alive, or --keep-model-dir with a SLURM or Kubernetes target now prints a yellow warning that these local Docker-only flags are ignored.

Fixed

• tools/ build context path corrected: docker build now resolves the shared tools directory as ./docker/common (project root) instead of ./scripts/common/tools. The previous path was stale — scripts/common/tools is a temporary directory populated at runtime by madengine run, so it was absent during standalone madengine build invocations, silently omitting the --build-context tools=… flag and breaking Dockerfiles that rely on it via COPY --from=tools.

• Hatch package artifacts include scripts/: pyproject.toml now uses [tool.hatch.build.artifacts] to include the scripts/ directory in the built wheel. The previous force-include directive caused duplicate file errors with newer hatchling versions (which are stricter about files already covered by the default source inclusion). Switching to artifacts bypasses .gitignore exclusion without risk of duplication. The deployment/templates force-include was also removed as it is already captured by the default wheel source scan.

[2.1.0] - 2026-05-28

Added

• slurm_multi SLURM escape-hatch launcher: New self-managed multi-node launcher for workloads that orchestrate their own per-node Docker containers via srun (e.g. SGLang Disaggregated proxy + prefill + decode topologies). Selected via distributed.launcher: "slurm_multi" (or "slurm-multi" alias). Generates a wrapper SBATCH script that runs the model's .slurm script directly on baremetal so srun/scontrol work inside it; performs parallel srun docker pull of the registry image on all allocated nodes when the model card sets env_vars.DOCKER_IMAGE_NAME. Honors model-card and --additional-context slurm fields (partition, nodes, gpus_per_node, time, exclusive, reservation, nodelist). This launcher coexists with the standard templated launchers (torchrun, vllm, sglang, deepspeed, megatron, torchtitan, primus) — those continue to flow through the standard sbatch template unchanged; only slurm_multi/slurm-multi takes the self-managed bypass path.

• madengine build --use-image [IMAGE | auto]: Skip the local Docker build and use a pre-built image instead. With no value, resolves to the model card's env_vars.DOCKER_IMAGE_NAME automatically. Mutually exclusive with --registry and --build-on-compute. Manifest entries are keyed by model name with local_image: True so ContainerRunner.run_models_from_manifest() resolves run_image correctly and pulls on demand.

• madengine build --build-on-compute: Build Docker images on a SLURM compute node and push to a registry, then have madengine run pull the image in parallel on all allocated nodes. Requires --registry. The resulting manifest carries built_on_compute: true.

• slurm_multi build registry gate: When madengine build discovers a slurm_multi model and no --registry/--use-image/--build-on-compute is given, the orchestrator either auto-uses env_vars.DOCKER_IMAGE_NAME from the model card (implicit --use-image fallback) or raises a structured ConfigurationError with the four supported options listed.

• bash-in-salloc execution path for slurm_multi: when madengine run detects SLURM_JOB_ID (i.e. running inside an existing salloc), the slurm_multi launcher runs the generated wrapper synchronously with bash instead of nesting another sbatch job. Other launchers continue to use sbatch even inside salloc (no behavior change for non-slurm_multi).

• Local self-managed launcher execution (container_runner.py): ContainerRunner._run_self_managed() runs the model script directly on the host for self-managed launchers, bypassing madengine's Docker wrapper. Used when madengine run detects a slurm_multi launcher in local/non-SLURM contexts. Environment variables from the model card and --additional-context are injected; keys are logged without values to avoid leaking credentials.

• Model card config merge into manifest deployment_config: _execute_with_prebuilt_image now merges the model card's distributed and slurm sections into the manifest's deployment_config, so the run phase auto-detects SLURM deployment and launcher settings without requiring --additional-context. User-supplied CLI values take precedence over model card defaults.

• DockerBuilder registry image injection for parallel pull: After a successful registry push, DockerBuilder.generate_manifest() now sets DOCKER_IMAGE_NAME in each built_models entry's env_vars to the registry image, enabling slurm_multi parallel srun docker pull on all nodes without requiring manual image specification.

• DeploymentResult.skip_monitoring (deployment/base.py): new dataclass field so synchronous deploy paths (e.g. slurm_multi's bash-in-salloc) can skip the monitor poll.

• SlurmNodeSelector reservation parameter: optional reservation name forwarded to srun health/cleanup commands so node-prep srun calls run inside the reservation.

• tests/unit/test_slurm_multi.py: contract tests for slurm_multi registry membership, hyphen alias normalization, end-to-end env_vars-export contract against MAD-private PR #186's pyt_sglang_disagg_qwen3-32b_short model card, and _execute_with_prebuilt_image manifest key-set contract (built_images.keys() == built_models.keys()).

• examples/slurm-configs/minimal/slurm-multi-minimal.json: minimal reference config for the new launcher.

• Docker build context — shared tools/ API access: docker build now passes --build-context tools=./tools, making the ./tools directory available as a named build context inside every Dockerfile. This allows Dockerfiles to COPY --from=tools shared helper scripts and APIs without duplicating them into each model's build context.

Changed

• Early model discovery reuse in BuildOrchestrator: The DiscoverModels result from the slurm_multi registry-gate check is now cached and reused for the actual build step, avoiding duplicate get_models_json.py execution and duplicate console output.

• E2E test cleanup defaults expanded: DEFAULT_CLEAN_FILES in tests/fixtures/utils.py now includes build_manifest.json and related perf artefacts (perf_super.json, perf_entry.csv, etc.) so stale manifests from prior e2e tests cannot silently cause the wrong image to be executed.

Fixed

• slurm_multi: cwd perf.csv aggregation: After a successful slurm_multi run, madengine run previously printed a cosmetic Performance CSV not found: perf.csv warning even though _collect_slurm_multi_results had ingested the per-job CSV from /shared_inference/$USER/$JOBID/perf.csv. The reporter (display_performance_table) reads cwd perf.csv by default. Now _collect_slurm_multi_results also writes the per-job rows into cwd perf.csv (copy if absent, append-data-rows if present) so reporting and HTML generation work without extra args. Local + classic-SLURM flows are unchanged.

Security

• Shell injection hardening in slurm_multi wrapper scripts: shlex.quote() is applied to env_var values, the model script name, and model args in the generated SBATCH wrapper script (slurm.py::_prepare_slurm_multi_script) and the local self-managed runner (container_runner.py::_run_self_managed), preventing shell metacharacters ($(), backticks, ;, ", etc.) in user-supplied inputs from triggering host-shell expansion.

[2.0.3] - 2026-05-26

Added

• rocEnvTool full mode (rocenv_mode in --additional-context, default "lite"): set "rocenv_mode": "full" to also collect hardware_information (lshw), bios_settings (dmidecode), dmsg_gpu_drm_atom_logs (dmesg), and amdgpu_modinfo (modinfo). Missing diagnostic tools are auto-installed best-effort using the guest_os-native package manager — apt-get on UBUNTU, microdnf/dnf/yum (first one found) on CENTOS. Install failures (no network, unprivileged container, unsupported guest) are non-fatal: the affected sections are simply omitted. Wired through both local Docker runs (container_runner.py) and Kubernetes deployments (k8s_scripts.py, k8s_template_context.py). See System environment collection.

• MAD_GUEST_OS in container env: container_runner now exports the run's guest_os as MAD_GUEST_OS so in-container pre-scripts (notably run_rocenv_tool.sh) can select the correct package manager without re-detecting from /etc/os-release.

• K8s storage_class field: New generic storage_class key in the K8s preset defaults (src/madengine/deployment/presets/k8s/defaults.json). It is the broadest fallback for both the data PVC and the single-node results PVC, behind the more specific data_storage_class / nfs_storage_class and single_node_results_storage_class / local_path_storage_class keys. The legacy local_path_storage_class key continues to be honoured for backward compatibility. Default change: the bundled preset now sets storage_class: "nfs-banff" in place of local_path_storage_class: "local-path", so out-of-the-box single-node results PVCs land on the NFS class instead of local-path. Clusters that still want local-path should set "local_path_storage_class": "local-path" (or "single_node_results_storage_class": "local-path") in --additional-context. See K8s storage classes.

Changed

• Profiling: rocm_trace_lite now sets RTL_MODE=lite explicitly; added tool rocm_trace_lite_default with RTL_MODE=default for A/B overhead comparison. rtl_trace_wrapper.sh passes rtl trace --mode … when RTL_MODE is set.

• Kubernetes deployment refactor: Decomposed the monolithic kubernetes.py (~2800 lines) into focused mixin modules — k8s_pvc.py (PVC lifecycle), k8s_results.py (log/artifact collection and performance aggregation), k8s_scripts.py (script extraction and ConfigMap building), and k8s_template_context.py (Jinja2 template context assembly). KubernetesDeployment now composes these mixins; no functional changes.

• run_rocenv_tool.sh argument signature: now accepts <output_basename> <rocenv_mode> <guest_os> (was just <output_basename>). rocenv_mode and guest_os default to lite and UBUNTU respectively when omitted, so existing direct callers remain functional. container_runner and the K8s scripts mixin pass all three.

• Pytest configuration consolidation: pytest settings now live solely in [tool.pytest.ini_options] in pyproject.toml; the redundant pytest.ini was removed. tests/conftest.py lost its sys.path hack and duplicate marker registration (markers are declared in pyproject.toml). The minversion value was also corrected from "3.8" (which was the Python version) to "7.0" — the actual pytest floor required for the pythonpath option used by the config.

Fixed

• Multi-arch image names broken for slashed model names: _create_base_image_name in execution/docker_builder.py interpolated model_info["name"] directly into the image tag, so a model named dummy/dummy built with --target-archs gfx950 produced ci-dummy/dummy_dummy/dummy.ubuntu.amd_gfx950. Docker image tags cannot contain / (it is a repository separator), so the subsequent docker tag … rocm/mad-private:<image> failed with invalid reference format and the registry push was skipped, even though the local build succeeded. Single-arch builds were unaffected because build_image() already sanitised the name. The helper now mirrors the single-arch convention — lowercase the model name, replace / with _, and append the dockerfile basename — yielding ci-dummy_dummy_dummy.ubuntu.amd_gfx950. Regression tests added in tests/integration/test_platform_integration.py cover both the helper and the end-to-end multi-arch path with a slashed model name (existing tests used "name": "dummy" and never tripped the bug).

• K8s collector pod name mismatch: The cleanup code in kubernetes.py used the full job name (collector-{job_name}) while the creation code in k8s_results.py truncated it (collector-{deployment_id[:15]}). For any job name longer than 15 characters (i.e. virtually all real jobs), cleanup would fail to delete the collector pod, leaving it running and potentially blocking PVC deletion on the next deploy. Extracted a shared collector_pod_name() helper so both sites use the same truncated name.

• rocEnvTool full-mode dumps crashed on empty tool output: dump_hardware_information_in_csv, dump_bios_settings_in_csv, dump_dmsg_gpu_drm_atom_logs_in_csv, and dump_amdgpu_modinfo_in_csv indexed lines[0] unconditionally. In unprivileged containers, dmesg (no CAP_SYSLOG) and dmidecode (no /dev/mem) commonly emit empty output, which raised IndexError and aborted the entire CSV dump — losing the sections that had succeeded. Each handler now returns [] early when the source file is empty.

• RPD pre-script: xxd missing in rocm/pytorch base image: upstream rocmProfileData/rpd_tracer/Makefile uses xxd -i to embed tableSchema.cmd/utilitySchema.cmd as C arrays, so make rpd exited 127 and the e2e suite saw no trace.rpd. trace.sh now installs xxd on Ubuntu and vim-common (provides xxd) on CentOS.

• RPD pre-script: failed as root with no sudo: the install path used sudo apt/sudo yum unconditionally, which is missing in many CI containers running as root. trace.sh now branches on id -u — direct apt-get/yum when root, sudo otherwise — and adds the build deps the upstream Makefile expects (git, build-essential, pkg-config on Ubuntu; gcc, gcc-c++, make, git on CentOS).

• TypeError on restricted ROCm < 6.4.1 systems: Context assumed every /dev/dri/renderD* entry exposed a non-None kfd_renderDs value. On restricted hosts (ROCm < 6.4.1, certain VFIO/passthrough setups) this returned None and crashed downstream consumers. core/context.py now guards the iteration so missing/None entries are skipped instead of raising.

• Deployment monitor infinite loop on cancelled jobs: BaseDeployment._monitor_job treated only COMPLETED/FAILED as terminal, so a CANCELLED job (manual scancel, K8s job deletion, etc.) would loop forever waiting for a state that never arrived. CANCELLED is now in the terminal-state set in deployment/base.py.

• Docker local: missing MAD_MULTI_NODE_RUNNER: SLURM (job.sh.j2) and Kubernetes (kubernetes_launcher_mixin.py) already export MAD_MULTI_NODE_RUNNER with the appropriate distributed launcher command, but local Docker runs had no equivalent. Models that delegate process spawning to $MAD_MULTI_NODE_RUNNER (e.g. Megatron-LM train_7b.sh) failed on madengine run with MULTI_NODE_RUNNER is not defined. ContainerRunner now resolves the launcher from --additional-context → model distributed.launcher → MAD_LAUNCHER (same priority chain as elsewhere), treats deployment-level values (docker, native) as torchrun, and sets MAD_MULTI_NODE_RUNNER via _generate_local_launcher_command() after GPU resolution (MAD_RUNTIME_NGPUS). Supports torchrun, megatron-lm, torchtitan, deepspeed, vllm, sglang, and primus; models that hardcode their own launcher (e.g. HuggingFace scripts) simply ignore the variable. Skipped when MAD_MULTI_NODE_RUNNER is already set in docker_env_vars.

Security

• Shell injection hardening (extended): shlex.quote() is now applied to every shell interpolation of a user-controlled value across core/docker.py, execution/container_runner.py, execution/docker_builder.py, and orchestration/run_orchestrator.py (image names, paths, container names, build-args). A follow-up pass closed the last remaining sites in docker_builder.py (grep, docker manifest inspect, docker tag, docker push, head). This is a defence-in-depth extension of the v2.0.2 build-arg quoting work — values that flow through --additional-context, model configs, or registry credentials can no longer break out of the shell command they are embedded in.

Tests

• Dummy dummy_rocenv_full fixture: new Dockerfile installs lshw, dmidecode, kmod, and util-linux so e2e tests can exercise rocenv full mode end-to-end inside the container.

• RCCL profiling e2e stabilization: tests/fixtures/dummy/scripts/dummy/run_nccl_trace.sh now pins HIP_VISIBLE_DEVICES/NCCL_IB_DISABLE/NCCL_SOCKET_IFNAME defaults to avoid topology-detection hangs in CI. The rccl_trace log assertion in tests/e2e/test_profiling_workflows.py was relaxed for minor NCCL log-format drift.

• New test_shell_quoting.py: 11-test suite covering the shell-quoting behaviour described above end-to-end across docker.py, container_runner.py, docker_builder.py, and run_orchestrator.py. Includes regression coverage for spaces, $, backticks, command-substitution, and quote characters in interpolated values.

• Test isolation fix: tests/unit/test_error_handling.py was leaking the global error-handler state across tests, so test order could mask or fabricate failures. The handler is now reset around the affected tests.

Known Issues

• K8s multi-node: node reported as FAILED due to log collection error: In multi-node Kubernetes jobs, a node may be reported as FAILED in the results table even though the pod completed successfully (Status: Succeeded). This happens when the kubelet on the node becomes unreachable (502 Bad Gateway) between job completion and log collection — madengine cannot retrieve stdout logs and therefore cannot parse performance metrics for that node. The PVC artifacts are still collected. Check kubectl describe pod <pod> to confirm the pod actually succeeded; the issue is infrastructure-level (kubelet/API server), not a workload failure.

[2.0.2] - 2026-04-28

Fixed

• credential.json type validation: load_credentials() now raises ConfigurationError if credential.json contains a non-object value (e.g. a JSON array or string). Previously, json.load() could return a non-dict and assign it to credentials before the broad except handler fired, causing AttributeError: 'list' object has no attribute 'keys' or silent downstream failures. The loaded value is now checked with isinstance(..., dict) before being used.

[2.0.1] - 2026-04-27

Added

• ROCm path auto-detection (madengine.utils.rocm_path_resolver): Host ROCm root is now resolved automatically via a priority chain — top-level MAD_ROCM_PATH in --additional-context → auto-detect (traditional /opt/rocm, versioned /opt/rocm-*, TheRock rocm-sdk + markers, rocminfo/amd-smi/rocm-smi on PATH) → ROCM_PATH env var → /opt/rocm fallback. Set MAD_AUTO_ROCM_PATH=0 to skip scanning and use the legacy env-var / default behaviour only.

• In-container ROCM_PATH resolution: For AMD Docker runs, the container ROCM_PATH is now resolved independently of the host: docker_env_vars.MAD_ROCM_PATH (consumed and not forwarded as-is) → ROCM_PATH/ROCM_HOME from the image OCI config (docker image inspect) → in-image shell probe (docker run --rm) → /opt/rocm with a warning. The host-resolved path is no longer mirrored into the container by default, preventing mismatches when host and image ROCm layouts differ.

• TheRock layout support (madengine.utils.therock_markers): Shared file-marker constants for detecting TheRock (rocm-sdk) installs used by both host path resolution and container compatibility checks.

• Run phase environment table: container_runner now prints a side-by-side table at run time showing host vs. container installation type (apt/therock/unknown), ROCm/CUDA root, and version, making it easier to diagnose path mismatches without inspecting logs manually.

• --timeout 0 crashing with signal.alarm(None): Timeout.__enter__ called signal.alarm(None) when --timeout 0 was passed because the CLI correctly maps 0 → None but Timeout had no guard for a falsy value. Added early-return in __enter__/__exit__ when seconds is None or 0. Also fixed the run command panels printing Nones for timeout when --timeout 0 was used; they now display disabled.

• Docker container name regex false positives: The docker ps --filter name=^/<name>$ exact-match filter embedded the container name directly into the regex without escaping, so names containing metacharacters (e.g. ., [) could match unintended containers. Applied re.escape() to the name before building the filter pattern.

• login_to_registry type annotation: The registry parameter was typed as str but the implementation handled None and callers (including tests) passed None to mean DockerHub. Corrected to Optional[str].

• Registry password process-list exposure: docker login was invoked with the password in the argument list (visible via /proc or ps). Changed to pass it via a MAD_REGISTRY_PASSWORD environment variable consumed through printf %s "$MAD_REGISTRY_PASSWORD" | docker login --password-stdin.

• login_to_registry — raise_on_failure not fully honoured: Missing-key and invalid-format errors in login_to_registry always raised RuntimeError regardless of raise_on_failure. All three failure paths (missing registry key, invalid credential format, docker login error) are now gated on raise_on_failure, allowing ContainerRunner to fall through to public image pulls.

• Kubernetes missing-package warning invisible: DeploymentFactory raised ImportWarning when the kubernetes package was absent, which Python silences by default. Changed to UserWarning so the install hint is always visible.

Changed

• GPU arch auto-detection for full-run mode: madengine run --tags now automatically detects and injects MAD_SYSTEM_GPU_ARCHITECTURE into the Docker build args during the build phase. Previously, Dockerfiles declaring ARG MAD_SYSTEM_GPU_ARCHITECTURE without a default were built with an empty value unless the user manually passed --additional-context. The detection reuses the existing detect_gpu_vendor() + get_gpu_tool_manager() + normalize_architecture_name() pipeline; a user-provided value is never overridden. Standalone madengine build is unaffected (detection is off by default). Added detect_local_gpu_arch parameter to Context, BuildOrchestrator, and threaded it through RunOrchestrator._build_phase().

• Model discovery — scope-based tag selection: Replaced the strict mode flag on DiscoverModels with a cleaner scope-based rule that applies uniformly to both madengine run and madengine build:

  • Unscoped tag (e.g. --tags inference, --tags pyt_foo): matches any model with that value in its tags field (scope-agnostic), or a model whose full name equals the tag exactly (root-only).
  • Scoped tag (e.g. --tags MAD/inference, --tags MAD/pyt_foo): restricts candidates to models prefixed with MAD/, then matches by tag field or exact full name within that scope.
  • --tags all and --tags scope/all continue to select all models globally or within a scope respectively.
  • Removed strict_discovery parameter from BuildOrchestrator.execute() and the corresponding call in RunOrchestrator._build_phase() as they are no longer needed.

• Shared login_to_registry utility: Extracted duplicated Docker registry login logic (~120 lines) from DockerBuilder and ContainerRunner into core/auth.py::login_to_registry(). Both classes now delegate to it. DockerBuilder uses raise_on_failure=True; ContainerRunner uses raise_on_failure=False to allow fallback to public images.

• Centralised credential loading: Extracted _load_credentials from BuildOrchestrator and RunOrchestrator into core/auth.py::load_credentials(). Environment variables (MAD_DOCKERHUB_USER, MAD_DOCKERHUB_PASSWORD, MAD_DOCKERHUB_REPO) take precedence over credential.json.

• Dead code removal: Removed unused functions find_and_replace_pattern and substring_found (utils/ops.py), highlight_log_section (utils/log_formatting.py), SessionTracker.get_session_start and SessionTracker.load_marker (utils/session_tracker.py), and the unused _filter_images_by_dockerfile_context method from RunOrchestrator.

• ConfigurationError instead of SystemExit in orchestrator config loading: BuildOrchestrator now raises a structured ConfigurationError (with suggestions) instead of calling sys.exit() directly when configuration loading fails.

• Removed --rocm-path CLI flag: The flag was an alias for MAD_ROCM_PATH but its help text implied it could set both host and container paths, causing confusion. Use --additional-context instead: {"MAD_ROCM_PATH": "/host/rocm"} for the host root and {"docker_env_vars": {"MAD_ROCM_PATH": "/container/rocm"}} for the in-container root.

Fixed

• MAD_OUTPUT_CSV env var — empty value guard: container_runner now uses model_info.get('multiple_results') instead of 'multiple_results' in model_info when deciding whether to inject MAD_OUTPUT_CSV into the Docker container. The previous check passed MAD_OUTPUT_CSV='' whenever multiple_results was present but empty (e.g. via CustomModel.to_dict() which always serialises the field with its default value of "").

• Performance log parsing: Unified and extended the performance: log regex across all execution paths (base.py, container_runner.py) to correctly parse values with unit suffixes (e.g. /s), comma separators between the value and metric name, explicit sign prefixes (+/-), uppercase scientific notation (E), and leading-dot decimals (e.g. .5). Previously the narrow [\d.]+ pattern silently dropped records from training scripts that emitted performance: 14164/s, samples_per_second-style lines. The pattern is now defined as a single module-level constant (PERFORMANCE_LOG_PATTERN in deployment/base.py) shared by both parsers.

• TheRock container compatibility — rocEnvTool: csv_parser.py now resolves rocm-smi via shutil.which() so images where tools live in a Python venv (not /opt/rocm/bin/) are detected correctly. Accepts a path_resolver argument to read the ROCm version from RocmPathResolver.get_version() rather than hardcoding /opt/rocm/.info/version. Added bounds check in the NVIDIA GPU info parser. rocenv_tool.py passes the resolver to CSVParser so version resolution works for both TheRock and traditional installs.

• TheRock container compatibility — GPU checks: Container exec commands for amd-smi/rocm-smi in container_runner.py now use PATH-based resolution instead of host-resolved absolute paths, so they work in TheRock images where the tools are not under /opt/rocm/bin/.

• In-container installation type detection: The shell command used to distinguish TheRock from apt installs was broken by quoting issues when passed through docker exec bash -c "...", causing the check to always fall through to unknown. Replaced with a quoting-safe two-step check: test if rocm-sdk exists and returns a root path (TheRock), otherwise check /opt/rocm/.info/version (apt).

• Model discovery — tag selection with extra args: In the unscoped --tags path, tag-list matching and the all check incorrectly used the raw tag string (e.g. inference:batch-size=32) instead of the pre-colon model name (inference). This caused tag-based selection to silently fail whenever extra args were appended via the colon syntax. Fixed for both models and custom_models loops.

• Model discovery — cross-scope name leakage: Unscoped tags (e.g. --tags pyt_foo) previously matched models in any scope via a short-name split (model["name"].split("/")[-1]), so pyt_foo would silently select MAD/pyt_foo. Removed the short-name backward-compat matching; an unscoped name now only matches a model whose full name equals the tag exactly.

• datetime.utcnow() deprecation in mongodb.py: Replaced all datetime.utcnow() calls with datetime.now(timezone.utc) to silence Python 3.12+ deprecation warnings.

• E2E tests — hardware-agnostic GPU arch skip: test_commandline_argument_skip_gpu_arch and its companion test now detect the current GPU architecture at runtime and inject it into the fixture's skip_gpu_arch list, so both tests pass on any GPU (gfx942, gfx950, etc.) without hardcoding arch names. Added get_gpu_arch() utility to tests/fixtures/utils.py.

• E2E tests — test_docker_gpus pre-script OOM on MI350X: The run_rocenv_tool.sh system-env pre-script was being OOM-killed (exit 137) inside Docker on gfx950 nodes with 6 GPUs bound, failing a test whose purpose is only GPU binding verification. Fixed by correcting the gen_sys_env_details condition in container_runner.py — the old or made the context key a no-op since generate_sys_env_details defaults to True — and passing gen_sys_env_details: False in the test's additional_context.

Security

• Registry password no longer in process argument list: Docker login commands previously passed the password as a CLI argument visible to other users via /proc or ps. All registry logins now inject the password through a dedicated MAD_REGISTRY_PASSWORD environment variable and use --password-stdin.

• build-arg values shell-quoted: All Docker --build-arg key/value pairs are now wrapped with str() before shlex.quote() to prevent shell injection from non-string config values.

Tests

• New TestTimeout suite: Covers None, 0, and positive-second cases for Timeout.__enter__/__exit__, plus a resolve_run_timeout passthrough regression test.

• New TestLoginToRegistry suite: Covers all success and failure paths of login_to_registry, including raise_on_failure=True/False behaviour, missing registry key, invalid credential format, and docker.io normalisation.

• Test suite cleanup: Removed dead imports across 14 test files; replaced try/assert False/except antipattern with pytest.raises() (with match=); narrowed 5 bare except: clauses to except Exception:; deleted a pass-only dead test; removed duplicate tests; reclassified test_profiling_tools_config.py from unit to integration (reads real disk files) and test_errors.py from integration to unit (pure mocks).

[2.0.0] - 2026-04-09

Overview

madengine v2.0 is a complete rewrite with a unified CLI architecture, replacing the legacy v1.x codebase. This release introduces a 5-layer architecture (CLI → Orchestration → Deployment → Execution → Core), comprehensive error handling, and production-grade quality standards.

🚨 Breaking Changes: See Migration Guide below.

---

🎯 Major Features

Unified CLI Architecture

• Single entry point: madengine command with subcommands (discover, build, run, report, database)
• Removed legacy v1.x CLI: All legacy commands (mad.py, mad-* tools) removed
• Rich console integration: Beautiful terminal output with progress bars, panels, and formatted text
• Consistent error handling: Structured exceptions with ErrorCategory enum and detailed context

Multi-Target Deployment

• Local execution: Direct Docker container execution for single-GPU workloads
• Kubernetes Jobs: Template-based K8s job generation with launcher support
• SLURM integration: Batch job submission with intelligent presets and nodelist pinning
• Factory pattern: Automatic deployment target selection based on configuration

Distributed Framework Support

• Training launchers: torchrun, DeepSpeed, Megatron-LM, TorchTitan, Primus
• Inference launchers: vLLM, SGLang, SGLang Disaggregated
• Launcher mixin: Unified launcher configuration via kubernetes_launcher_mixin.py
• Template-driven: Jinja2 templates for each launcher type
• Full documentation: Comprehensive launcher guide in docs/launchers.md

GPU Vendor Support

• AMD ROCm: Full support with amd-smi/rocm-smi detection
• NVIDIA CUDA: Complete CUDA toolkit integration
• Build defaults: Automatically defaults to AMD + UBUNTU if not specified
• Explicit configuration: Override via --additional-context '{"gpu_vendor": "NVIDIA", "guest_os": "CENTOS"}'

---

✨ New Features

Log Error Pattern Scanning (#92, #93)

• Automatic failure detection: Scans container logs for common error patterns (RuntimeError, OOM, Traceback)
• Configurable patterns: Override default patterns via log_error_patterns in additional_context
• Benign exclusion: Exclude false positives with log_error_benign_patterns (e.g., ROCProf logs)
• Disable option: Set "log_error_pattern_scan": false when pytest/JUnit is authoritative
• Implementation: src/madengine/execution/container_runner_helpers.py
• Test coverage: TestErrorPatternMatching class validates ROCProf exclusion

Skip Model Run Flag (#91)

• Build-only workflow: madengine run --tags model --skip-model-run
• Use case: CI/CD pipelines that only need image validation
• Full workflow: Discover → Build → Skip execution, but validate configuration
• Exit code preservation: Returns appropriate exit codes for build failures

ROCprofv3 Profiling Suite (ROCm 7.0+)

• 8 pre-configured profiles: compute, memory, communication, full, lightweight, perfetto, api_overhead, pc_sampling
• Hardware counter definitions: 4 counter files for targeted profiling scenarios
• Configuration examples: Ready-to-use JSON configs in examples/profiling-configs/ (including rocm_trace_lite.json for rocm-trace-lite)
• Custom command support: Fixed argument parsing with -- separator requirement
• Auto-detection: Seamlessly switches between rocprof (legacy) and rocprofv3

SLURM Nodelist Pinning

• Node specification: Pin jobs to specific nodes via slurm.nodelist (comma-separated)
• Health check bypass: Automatic node health preflight skipped when nodelist set
• Configuration: See examples/slurm-configs/basic/03-multi-node-basic-nodelist.json
• Documentation: Enhanced SLURM deployment guide

Kubernetes Secrets Management

• Automatic conversion: secrets dict in additional_context → K8s Secret objects
• Environment variables: Secrets mounted as env vars in containers
• Template: templates/kubernetes/secret.yaml.j2
• Security: Follows K8s best practices for secret handling

Batch Build Support

• Selective builds: Manifest-driven builds for CI/CD efficiency
• Format: [{"model_name": "...", "build_new": true/false, ...}]
• Optimization: Only build images marked with "build_new": true
• Output manifest: All models included regardless of build status

Data Provider Abstraction

• Multiple backends: Local filesystem, NAS, S3, MinIO
• Unified interface: core/dataprovider.py::Data class
• Model discovery: Support for models.json and get_models_json.py scripts
• Configuration flexibility: Per-model data source configuration

---

🔧 Improvements

Code Quality (#94)

• Rating: 4.5/5 (up from estimated 4.0/5 in v1.x)
• Type coverage: 71% type hints (industry standard: 50-80%)
• Documentation: 82% Google-style docstrings
• Zero technical debt: No TODO/FIXME/HACK markers
• Production-ready: Comprehensive test coverage and error handling

Error Handling System

• Structured exceptions: Base MADEngineError with category classification
• 10 error types: ValidationError, ConnectionError, AuthenticationError, ExecutionError, BuildError, DiscoveryError, OrchestrationError, RunnerError, ConfigurationError, TimeoutError
• Rich console output: Formatted error panels with context, suggestions, and recovery indicators
• Exit codes: Fixed enum values for CI/CD integration (SUCCESS=0, BUILD_FAILURE=2, RUN_FAILURE=3, etc.)
• Backward compatibility: RuntimeError alias preserved for ExecutionError

Console Output

• Rich library: All output via Console class (removed all direct print() calls)
• Live/non-live modes: --live-output flag for streaming vs buffered output
• Formatted panels: Color-coded panels for errors, warnings, and info messages
• Progress tracking: Rich progress bars for long-running operations
• Database module: Replaced 15+ print() calls with console.print() in mongodb.py

Testing Infrastructure

• Test reduction: Streamlined from 503 to 278 lines (-45%) by removing edge cases
• Focus on behavior: Test core functionality, not implementation details
• 39 unit tests: All passing with 100% backward compatibility
• Parametrized tests: Efficient testing of multiple error types and scenarios
• Pattern validation: ROCProf exclusion tests ensure no false positives

Pre-commit Hooks

• Automated quality: black, isort, flake8, mypy, bandit
• File safety: check-yaml, check-json, check-toml, check-merge-conflict
• Security: bandit scans for common vulnerabilities
• Configuration: .pre-commit-config.yaml with madengine-specific rules
• Easy setup: pip install pre-commit && pre-commit install

Documentation

• Code Quality Report: Detailed metrics and industry comparisons
• Inline docstrings: 82% coverage with Google-style format
• Examples: Configuration examples in examples/{k8s,slurm,profiling}-configs/
• README overhaul: Merged all documentation into single comprehensive source
• Launcher guide: Centralized documentation for all distributed frameworks

---

🏗️ Architecture Changes

5-Layer Design

1. CLI Layer (cli/): Typer-based commands with Rich output
2. Orchestration Layer (orchestration/): BuildOrchestrator, RunOrchestrator
3. Deployment Layer (deployment/): K8s, SLURM, factory pattern, presets, templates
4. Execution Layer (execution/): container_runner, docker_builder, log scanning
5. Core Layer (core/): context, dataprovider, console, errors, constants

Design Patterns

• Template Method: Deployment base class with subclass customization
• Factory: DeploymentFactory for target selection
• Strategy: Launcher strategies (torchrun, DeepSpeed, etc.)
• Mixin: Launcher-specific template selection (kubernetes_launcher_mixin.py)
• Builder: Progressive Docker image configuration

Configuration Flow

1. CLI args → merge with --additional-context JSON/file
2. Context object created with merged config
3. Orchestrator determines target (local vs distributed)
4. Deployment layer applies presets + renders Jinja2 templates
5. Execution layer runs containers or submits jobs

---

🐛 Bug Fixes

ROCprofv3 Argument Parsing

• Fixed custom command parsing: rocprof_wrapper.sh now requires -- separator
• Error prevented: ValueError: invalid truth value bash (type=str)
• Compatibility: Works with both rocprof (legacy) and rocprofv3 (ROCm >= 7.0)
• Documentation: Enhanced usage guide with examples

Error Pattern False Positives

• ROCProf exclusion: Benign patterns for ROCProf logs (E20251230, W20251230, rocpd_op, etc.)
• Pattern specificity: Changed from Error: to RuntimeError: to reduce false positives
• HuggingFace models: GPT2/BERT no longer fail due to profiling tool output
• Test coverage: TestErrorPatternMatching validates benign pattern exclusion

ROCm Path Resolution

• Fallback chain: --rocm-path flag → ROCM_PATH env var → /opt/rocm default
• GPU detection: Tries amd-smi first, falls back to rocm-smi for older ROCm versions
• Run-only: GPU detection only during run command, not build (avoids failures on build-only nodes)

Import Path Consistency

• Standardized imports: All imports use from madengine.core.errors import ...
• No circular dependencies: Clean layer separation prevents import cycles
• Type annotations: Proper use of Optional, List, Dict from typing module
• Cleanup: Removed unused typing_extensions import in core/console.py

---

🔒 Security Fixes

SQL Injection Vulnerability (CRITICAL)

• Fixed: SQL injection in src/madengine/db/database_functions.py
• Solution: Replaced string formatting with parameterized queries using SQLAlchemy text()
• Impact: Prevents potential SQL injection attacks in get_matching_db_entries() function

Exception Handling

• Fixed: 4 instances of bare except: blocks that could mask critical exceptions
• kubernetes.py: Replaced with specific exception types (ConfigException, FileNotFoundError, ApiException)
• console.py: Replaced with specific exception types (OSError, ValueError) for resource cleanup

---

🗑️ Removed (Breaking Changes)

Legacy v1.x CLI

• Removed files:
  • src/madengine/mad.py - Legacy CLI entry point (v1.x)
  • src/madengine/tools/run_models.py - Legacy model runner
  • docs/legacy-cli.md - Legacy CLI documentation
• Replaced by: Unified madengine CLI with subcommands
• Migration required: See Migration Guide

Legacy Documentation

• Removed: docs/distributed-execution-solution.md, docs/madengine-cli-guide.md
• Removed: docs/TORCHTITAN_LAUNCHER.md (consolidated into docs/launchers.md)
• Justification: Consolidated into comprehensive single-source documentation

Direct Print Calls

• Removed: All direct print() calls in production code
• Replaced by: console.print() from Rich library
• Exception: Test files may still use print for debugging

RuntimeError Class (Renamed)

• Renamed: RuntimeError → ExecutionError (avoids shadowing Python built-in)
• Backward compatibility: RuntimeError = ExecutionError alias preserved
• Impact: Minimal - existing code using RuntimeError continues to work

Stale Artifacts

• Removed: Compiled Python files (__init__.pyc) from source tree
• Removed: Python cache files and build artifacts
• Removed: Unnecessary debug print statements

---

📊 Metrics & Quality

Code Quality Improvements

• Overall rating: 4.5/5 (industry-leading)
• Type hints: 71% coverage (target: 50-80%)
• Docstrings: 82% coverage (target: 70-90%)
• Technical debt: 0 TODO/FIXME/HACK markers
• Test reduction: -45% lines while maintaining coverage
• Net change: -185 lines across 7 files (cleaner codebase)

Test Coverage

• 39 unit tests: All passing
• Test types: Unit, integration, end-to-end
• Focus: Behavior over implementation
• Backward compatibility: 100% preserved

Security & Standards

• Pre-commit hooks: 10+ automated checks
• Bandit scans: Security vulnerability detection
• Type checking: mypy static analysis
• Linting: flake8 + black + isort

---

🔄 Migration Guide

Command Structure Changes

v1.x (Legacy):

# Old commands
mad-discover --tags dummy
mad-build --tags dummy
mad-run --tags dummy

v2.0 (Current):

# New unified CLI
madengine discover --tags dummy
madengine build --tags dummy
madengine run --tags dummy

# Or full workflow with single command
madengine run --tags dummy

Configuration Changes

v1.x: Configuration scattered across multiple files and environment variables

v2.0: Unified --additional-context flag

# File-based config
madengine run --tags model --additional-context config.json

# Inline JSON config
madengine run --tags model --additional-context '{"gpu_vendor": "NVIDIA", "guest_os": "CENTOS"}'

# Build defaults (NEW in v2.0)
madengine build --tags model
# Automatically uses: gpu_vendor=AMD, guest_os=UBUNTU

Error Handling Changes

v1.x: Generic exceptions with minimal context

v2.0: Structured error classes with Rich formatting

# Import structured errors
from madengine.core.errors import (
    ValidationError,
    ExecutionError,  # Previously RuntimeError
    BuildError,
    ConfigurationError,
    create_error_context
)

# Create error with context
context = create_error_context(
    operation="model_training",
    component="GPTRunner",
    model_name="gpt2"
)
raise ExecutionError("Training failed", context=context, suggestions=["Check GPU memory"])

Deployment Target Changes

v1.x: Limited deployment options

v2.0: Multi-target deployment

# Local execution (default)
madengine run --tags model

# Kubernetes deployment
madengine run --tags model --additional-context '{
  "deployment_target": "kubernetes",
  "distributed": {
    "launcher": "torchrun",
    "num_nodes": 2,
    "gpus_per_node": 8
  }
}'

# SLURM deployment with nodelist
madengine run --tags model --additional-context '{
  "deployment_target": "slurm",
  "slurm": {
    "partition": "gpu",
    "nodes": 2,
    "gpus_per_node": 8,
    "nodelist": "node01,node02"
  }
}'

Log Error Detection (NEW)

v2.0: Automatic log error pattern scanning

# Default behavior: scan enabled
madengine run --tags model

# Disable scanning (when pytest/JUnit is authoritative)
madengine run --tags model --additional-context '{"log_error_pattern_scan": false}'

# Custom error patterns
madengine run --tags model --additional-context '{
  "log_error_patterns": ["CustomError:", "FATAL"],
  "log_error_benign_patterns": ["ExpectedWarning", "ROCProf"]
}'

Breaking Changes Summary

Feature	v1.x	v2.0	Action Required
CLI entry point	mad-* commands	madengine unified CLI	Update all scripts/workflows
Configuration	Multiple files	--additional-context	Consolidate config into JSON
Error classes	Generic exceptions	Structured MADEngineError types	Update error handling code
Console output	Direct print()	Rich console.print()	Use Console API in extensions
GPU defaults	No defaults	AMD + UBUNTU defaults	Explicit config for other vendors
RuntimeError	N/A	Renamed to ExecutionError	Use alias or update imports
ROCprofv3	N/A	Requires -- separator	Update profiling configs

---

📝 Installation & Setup

Requirements

• Python: 3.8+ (use typing_extensions for 3.8 compatibility)
• Docker: Required for all execution (local and distributed)
• MAD Package: Separate repo (git clone https://github.com/ROCm/MAD.git) for model definitions
• Pre-commit (dev): pip install pre-commit && pre-commit install

Installation

# Development installation
python3 -m venv venv && source venv/bin/activate
pip install -e ".[dev]"

# With Kubernetes support
pip install -e ".[dev,kubernetes]"

# Production installation
pip install madengine

Useful Commands

# Run all tests
pytest

# Format code (mandatory before commits)
black src/ tests/ && isort src/ tests/

# Run pre-commit hooks manually
pre-commit run --all-files

# Skip model script (container starts, pre_scripts run); leave live container for debugging
madengine run --tags model --skip-model-run --keep-alive

# Debug with verbose output
madengine run --tags model --verbose --live-output

# Disable log error scan
madengine run --tags model --additional-context '{"log_error_pattern_scan": false}'

---

🙏 Acknowledgments

This release represents a complete architectural overhaul focused on:

• Developer experience: Clear architecture, comprehensive docs, helpful error messages
• Production readiness: Automated quality checks, comprehensive testing, security scanning
• Extensibility: Plugin-friendly design, template-driven deployment, launcher abstraction
• Performance: Optimized builds with selective rebuilds, efficient log scanning

---

[1.x] - Legacy (Deprecated)

Legacy v1.x releases are deprecated and no longer supported. All users should migrate to v2.0.

For v1.x documentation and changelogs, see the git history or the legacy-v1 branch (if available).

---

Guidelines for Changelog Updates

Categories

• Added for new features
• Changed for changes in existing functionality
• Deprecated for soon-to-be removed features
• Removed for now removed features
• Fixed for any bug fixes
• Security for vulnerability fixes

Format

• Keep entries brief but descriptive
• Include ticket/issue numbers when applicable
• Group related changes together
• Use present tense ("Add feature" not "Added feature")
• Target audience: users and developers of the project