Speculative decoding on per-second GPUs

Summary

This guide covers speculative decoding on per-second GPUs. It lists the settings that matter, the numbers we measured and the mistakes we see most often.

Setup

Start from a working endpoint with tracing turned on. Measure P50 and P95 latency for a week before changing anything, so you can compare.

What to watch

Cost per 1,000 requests, cold starts per hour and queue time at peak. If one of them moves more than 20%, roll back and check the change.