Repository navigation
|
If I scale GPU serving to zero when idle, how bad is the cold start for a 7B model? |
Answered by
iacker
Jul 10, 2026
Replies: 1 comment
|
Dominated by two costs: pod scheduling + image pull, then model weight load into VRAM. With the image pre-cached on the node, a 7B in FP16 loads in roughly 15-40s depending on disk/PCIe; add pod startup and you are often 30-60s cold. Mitigations: keep the container image warm on the node, load weights from local NVMe not object storage, and use a 1-replica floor during business hours so only off-peak hits the cold path. |
0 replies
Answer selected by
iacker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Dominated by two costs: pod scheduling + image pull, then model weight load into VRAM. With the image pre-cached on the node, a 7B in FP16 loads in roughly 15-40s depending on disk/PCIe; add pod startup and you are often 30-60s cold. Mitigations: keep the container image warm on the node, load weights from local NVMe not object storage, and use a 1-replica floor during business hours so only off-peak hits the cold path.