LLM inference · Serving reliability · ML systems
I find the numbers LLM serving systems get quietly wrong (routing on stale latency, metrics that hide a stall, benchmarks that disagree about what "latency" means), prove it with a test or a measurement, and fix it upstream.
2 merged PRs in the vLLM project's AIBrix gateway · open work in vLLM, Ray, GuideLLM · case studies
| Project | What was wrong | Status |
|---|---|---|
| AIBrix routing | Least-latency scored pods by a lifetime average, so a pod that had slowed down kept winning traffic for about 10 minutes after 100,000 requests. A one-minute window, with fallbacks for quiet pods and engine restarts, cuts that to 15 seconds; the change covers all three schedulers that used the lifetime mean. The maintainer reproduced the analysis before merging. Case study → | merged · #3327 (from my issue #3297) |
| AIBrix metrics | An empty histogram was exported as a real p50/p90/p99, so dashboards showed latencies that never happened. | merged · #3290 |
| vLLM scheduler | With speculative decoding, a request that could not fit its draft tokens preempted another request. Dropping the drafts first cut preemptions from 72.5 to 54.0 in a small local benchmark, with token-identical outputs. | open · #63079 |
| vLLM streaming | Reproduced a streaming bug and validated another author's fix on a GPU: 18 of 640 requests broken before, 0 of 640 after. The author cited the numbers to request review. | validation on #62726 |
| AIBrix batch | One NaN latency in a response leaves the adaptive concurrency controller at full load while the engine slows 50x. |
issue #3339 accepted · fix #3350 open |
| Ray Data LLM | The documented "-1 means unlimited" setting crashed the vLLM engine stage, and error rows lost their prompt; mocked tests hid both. | open · #67460 |
| GuideLLM | Non-streaming requests were counted as 0 ms inter-token latency samples. | open · #1406 |
| CodeCarbon | Country carbon intensities were 2 to 4 years old. A script now rebuilds them from the maintained dataset. | open · #1468 |
llm-bench-wire-audit: a logging proxy that checks what vllm bench serve, GuideLLM and AIPerf report against what actually crossed the wire. Where definitions match they agree with the wire within a few percent. Under an injected 300 ms stall, one tool's p99 inter-token latency reads 311 ms and another's reads 16.6 ms, because one averages per request and one does not.
Statuses checked 2026-10-31.
| System | Engineering problem | Evidence |
|---|---|---|
| Furnace · demo | Pull-request gate for LLM apps on their own vLLM server: replays both revisions' real prompts, gates on time to first token, catches prompt edits that break prefix caching | 234 → 43 ms p50 TTFT · 94% cache reuse restored · 720 requests re-checkable from raw records |
| OpsVerse · demo | Post-train Qwen3-4B, retrieve with BGE + BM25 + RRF, rerank, ground citations, gate releases with evals, serve with vLLM | MRR@10 0.705 · 1,914 graded judgments · 245 regression tests · ~6× lower cost/response |
| AxonFDE · demo | Fuse telemetry, orders, and shipping docs; predict breaches; reconcile conflicts; require human approval; audit every action | 49 min median lead time · 23/23 planted conflicts · 0 unauthorized actions / 50 tests · 4.92 s decision cycle |
| Pro2Pro · demo | Checkpointed LangGraph + MCP orchestration with resumable state, human approval, guardrails, tracing, and model fallback | 78 evaluation tests |
| FIFA2026MLOps · demo | Elo + gradient boosting + Monte Carlo forecasting with automated training, validation, deployment, lineage, and drift monitoring | Airflow · MLflow · Evidently · OpenTelemetry |
|
Research with Prof. Kenneth W. Regan Compare a Regan-inspired statistical reconstruction, Qwen3-8B, and Maia-3 on the same chess decisions using Stockfish MultiPV and controlled engine-assisted substitutions. Measured result
|
Research with Prof. Haipeng Cai Systematizing failure modes of LLM/VLM-driven physical agents across prompt/tool attacks, adversarial perception, leakage, and unsafe actions. Current direction LLM-assisted vulnerability localization → patch proposal → sandbox regression test. |
LANGUAGES Python · Go · SQL
SERVING vLLM · AIBrix · Ray Data LLM · FastAPI · asyncio / aiohttp · Docker · Kubernetes · AWS
MEASUREMENT TTFT / ITL / TPOT definitions · benchmark design · fault injection · Prometheus histograms
MODELS PyTorch · Transformers · Qwen · TRL · PEFT · QLoRA · SFT · DPO
RETRIEVAL RAG · BGE · BM25 · RRF · reranking · Qdrant
AGENTS LangGraph · LangChain · MCP · tool use · human-in-the-loop
EVALUATION benchmarks · regression tests · paired bootstrap · guardrails · red teaming
OBSERVABILITY Langfuse · OpenTelemetry · MLflow · Evidently · CI/CD
- GWO-Based Fed-UNet-CNN Model for Leukocytes Classification Across Developmental Stages
- The Future of Smart Factories: Real-Time Automation and Big Data Analytics
