Memory snapshots: restoring CUDA processes in two seconds

The cold start problem

Loading a 16 GB model into GPU memory takes 20 to 40 seconds: pull the image, start Python, import the framework, read the weights, compile kernels. Users waiting on a first request see all of it.

Snapshotting the process

We checkpoint the whole process after warm-up, including CUDA context and device memory, and store it next to the GPU host. A restore maps the snapshot back in and resumes at the first instruction after warm-up.

On an H100 with a 16 GB model, restore takes 1.9 seconds on average and 2.6 seconds at P95.

Limits

Snapshots are tied to a driver version and card family. When either changes we rebuild the snapshot on the next deploy, which adds one normal cold start.