feat(config): add --config YAML configuration system with Hydra compose API

[coketaste]

Summary

Adds --config as a composable, Hydra-based YAML alternative to --additional-context JSON strings, available on both run and build.

• New src/madengine/config/ module — HydraConfigLoader (Compose API), ConfigTranslator (YAML → internal context key mapping), ConfigValidator (cross-field checks)
• Hydra config groups under src/madengine/configs/ — default groups (platform, scheduler, hardware, launcher) and append-only groups (+profile, +env, +tools, +data, +build)
• Mutual exclusion — --config cannot be combined with --additional-context / --additional-context-file
• Empty-value filtering — empty lists and dicts are excluded from the translated context so they don't shadow existing defaults
• examples/configs/ — annotated templates (local.yaml, slurm.yaml, k8s.yaml) plus ready-to-run demos under demo/local/, demo/slurm/, demo/k8s/
• Fixture fix — dummy_torchrun / dummy_torchrun_multi n_gpus corrected from "1" to "4" so local multi-GPU demos show 1N×4G topology

Rebase onto develop

Merged current develop into this branch and resolved conflicts (pinned image digests, slurm_multi, timeout 0, launcher aliases, K8s mixin split, hatch artifacts packaging). Hydra configs/ are included in the wheel via hatch artifacts alongside scripts/.

Post-merge --config updates:

• Inline --config key=value overrides now win over a user YAML file (previously YAML was merged last and silently overwrote them)
• CLI applies YAML metadata for live_output, output, summary_output, data_config, model.skip_run, and model.timeout (including 0)
• Validator accepts require_pinned_image and treats kubernetes as an alias of k8s for slurm conflict checks
• Launcher aliases: megatron → megatron-lm, slurm_multi group, SLURM reservation / skip_gpus_directive in the slurm scheduler preset

Behaviour

Config group overrides:

madengine run --tags model --config scheduler=slurm --config launcher=torchrun

Append-only groups:

madengine run --tags model --config +profile=mi300x_8gpu --config +env=nccl_debug

User YAML file, then inline overrides (inline wins):

madengine run --config my_job.yaml
madengine run --config my_job.yaml --config distributed.nnodes=8

When a user YAML sets a distributed launcher, distributed.enabled: true must also be set explicitly — the default launcher: none group sets enabled: false and setting a launcher key alone does not override it.

Test plan

• pytest tests/unit (789 passed after merge)
• pytest tests/unit/test_config_schema.py tests/unit/test_config_translator.py tests/unit/test_config_integration.py tests/unit/test_hydra_config_loader.py tests/unit/test_cli.py
• madengine run --config examples/configs/demo/local/single-gpu.yaml completes and writes perf.csv
• madengine run --config examples/configs/demo/local/multi-gpu-torchrun.yaml shows 1N×4G topology and torchrun in the launcher column
• madengine run --config my_job.yaml --additional-context '{}' exits with INVALID_ARGS (exit code 4)
• pre-commit run --all-files passes

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[coketaste]

Merged current develop into this branch (2f50ac9) and pushed — the PR should no longer be conflicted.

What landed in the merge besides conflict resolution:

• --config key=value now overrides fields already set in a user YAML file
• YAML live_output / output / summary_output / data_config / model.skip_run / model.timeout (including 0) are applied to the CLI
• Schema knows about require_pinned_image and slurm+kubernetes conflicts
• megatron canonicalizes to megatron-lm; added launcher=slurm_multi group

pytest tests/unit: 789 passed. Manual demo runs and pre-commit still need a pass on a GPU box.

[coketaste]

Pushed 1ba44c0 onto this branch (not merged).

--config stays a frontend for the existing additional-context dict:

• Docs: job YAML uses JSON-shaped keys (slurm, distributed, …), not a Hydra defaults: list; only platform=docker is supported; hardware sets gpu_vendor/guest_os (engine Docker flags still come from vendor, not a runtime context key).
• +build=ci / +build=multi_arch are real again (docker_clean_cache / build.target_archs), matching --clean-docker-cache and --target-archs.
• YAML docker.keep_alive / docker.clean_cache apply to the CLI flags when true.
• Golden tests: tests/unit/test_config_context_parity.py.

pytest tests/unit: 795 passed.