I make vLLM start faster. I built thaw to snapshot and fork live LLM sessions.
vLLM contributor and Inferact open-source fellow, working with Simon Mo on cold start and model hot-swap. Based in San Jose.
I maintain the cold-start roadmap. My merged work includes pluggable sleep-mode backends, compile-cache invalidation fixes, and memory-profile reuse across boots.
Current work, in review:
- Engine snapshots: restore an initialized engine instead of rebuilding it. 23.9 s → 9.4 s from command start to first correct token versus ordinary warm startup (A10, Qwen3-0.6B, TP1, warm model and compile caches; median, n=5 per arm).
- Fast CLI help:
vllm --helpin 40 ms instead of 3.5 s by keeping it out of the runtime import graph (Apple M5, Python 3.12; median, n=5 per arm). - Precompiled Python bytecode: move compilation into the image build so fresh containers do less work before serving.
Snapshot a running LLM session, fork independent continuations from its KV state, and inspect or diff saved sessions on a laptop without a GPU. Rust + CUDA, with vLLM and SGLang integrations.
This is the project that led me into upstream vLLM.
Re-feeding Is Not Replaying · sole-author preprint.
Replaying a transcript can change which tokens a credit-estimation method identifies as important. I measured that against exact KV-state resume and a replica noise floor. Batch-invariant kernels eliminated the discrepancy in the tested configurations.
- Recall: a Rust/Tauri League of Legends overlay that recommends your next purchase from live game state.
- ProjectGorgon: speculative decoding with custom Triton/CUDA kernels. Where I learned GPU programming.
- sentinel: a Go log-streaming engine with LSM storage, a write-ahead log, and replication.
- Madison Bus ETA: arrival predictions with XGBoost and conformal uncertainty intervals. Retired; source available.
M.S. CS at Northeastern, Silicon Valley. B.S. Data Science, UW–Madison.