fix(checkpoint): validate FTW v1 indexes eagerly

[Avicennasis]

Problem. A malformed or truncated FTW v1 index (missing field, wrong type, out-of-range offset, duplicate key, shard shorter than declared) surfaces late, as an mmap or O_DIRECT failure inside the load, with no indication of which index field was wrong.

Change. Parse and validate the index once at FTWReader open: required fields, JSON integers within the signed 64-bit range, duplicate keys rejected, entry names checked, tensors and shards checked for coverage and overlap, each shard stat'd and size-checked. Errors raise FTWFormatError (a ValueError) naming the field. The writer validates its own metadata.

Shard symlinks are followed — Hugging Face hub snapshots are symlink farms — and the regular-file and size checks apply to the link target. A shard name that is not a filesystem name (a JSON-escaped lone surrogate) is refused under the same error type rather than escaping as UnicodeEncodeError.

Nothing the writer produces is rejected; existing valid checkpoints load unchanged.

Not in this PR. Runtime integrity re-checks, admission, generations, and bank-schema validation are separate changes.

Tested. Ubuntu 24.04 x86_64, kernel 7.0.0, 16 cores / 125 GiB, no GPU — this is a pure-Python change and the added tests are CPU-only. Python 3.12, torch 2.11.0+cu130, transformers 5.17.0, pytest 9.1.1, safetensors 0.8.0. Run with PYTHONPATH=python (an editable install needs CUDA_HOME); the C++ extension is not built in this environment, which accounts for one of the pre-existing failures below.

Focused:

PYTHONPATH=python python -m pytest -q tests/checkpoint/test_ftw_validation.py
85 passed

Also under python -O (85 passed), since several of the checks would be lost if they were assertions.

Full suite from the repo root, this branch:

PYTHONPATH=python python -m pytest -q tests
4 failed, 1346 passed, 427 skipped

and main at 9535656 in the same venv, same command:

4 failed, 1261 passed, 427 skipped

Identical failure sets — tests/engine/test_cache_budget.py::test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend, tests/engine/test_cache_budget.py::test_adjust_config_resolves_num_tokens_generic, tests/moe/test_offload.py::test_adjust_config_converts_moe_cache_rate_to_cache_size, and tests/kernels/test_swiglu_clamp.py::test_cpu_extension_supports_swiglu_clamp (the CPU extension is not built here). The +85 is this PR's test file.

No checkpoint was converted; the tests build their own FTW files with FTWWriter.

git diff --check clean; compileall ok; ruff 0.15.0 reports nothing on the four touched files (and nothing on their main counterparts).

Fixes #458.

[Avicennasis]

Rebased onto current main (cac247a, 0.1.3). #486 touched checkpoint/ftw.py underneath this — it added ftw_tensor_names(), a keep(name) predicate on iter_ftw_weights, and the Callable import — and all of that merged cleanly around the eager _validate_ftw_v1_index; the one conflict was the __all__ list, where #486 exports ftw_tensor_names and this PR exports FTWFormatError, so both are listed now. A git range-diff against the previous head shows nothing else moving.

#486's tests/checkpoint/test_ftw_hotfix_vision.py and test_ftw_weights.py now sit next to test_ftw_validation.py. iter_ftw_weights still opens the FTWReader (and so validates the index) before keep is consulted, so a malformed index is refused before any name is filtered.

Re-ran on a CPU-only Ubuntu 24.04 x86_64 box (AMD Ryzen 7 5800X, 16 threads, 125 GiB, no GPU; Python 3.12, torch 2.11.0+cu130, transformers 5.17.0, pytest 9.1.1, run with PYTHONPATH=python since an editable install needs CUDA_HOME):

• tests/checkpoint (all three files): 102 passed, 6 skipped, normally and under python -O; main at the same commit gives 17 passed, 6 skipped, so the +85 is this PR's tests. The 6 skips are fix(checkpoint): serve and repair FTW checkpoints around the vision encoder #486's FREETOKEN_*_MODEL not set cases.
• Full pytest -q tests from the repo root: this branch 8 failed / 1447 passed / 488 skipped, against main at cac247a in the same venv: 8 failed / 1362 passed / 488 skipped. Identical failure sets — tests/engine/test_cache_budget.py ×2, tests/moe/test_offload.py ×1, tests/kernels/test_swiglu_clamp.py::test_cpu_extension_supports_swiglu_clamp (the C++ extension is not built in this venv), three tests/mm/test_processor.py cases (No module named 'PIL'; Pillow is not in this venv), and tests/models/test_quant_config.py::…[bert-base-uncased] (the only snapshot in this box's HF cache; it yields zero probe modules). A first full run of the branch also tripped tests/daemon/test_daemon_serve_manager.py::test_crash_is_reaped_and_reported_without_stop — the log-ring line landing after the status flip, on a box under heavy load; it passes 10/10 in isolation on both trees and did not recur on a second full run.

Nothing in the change touches a kernel or a model family; the validator runs once at FTWReader open, on the index alone, which is why a CPU-only box is an honest environment for it.