If you mostly use chat LLMs, you can ignore most inference papers. This one is worth filing away. Speculative decoding's biggest selling point is not raw speed. It is the promise of speed without changing the answer [C002].
That flips the usual intuition. Most "faster LLM inference" pitches sound like a trade: less waiting, more risk. Lower quality, a different model, extra retraining. This paper is interesting because it tries to remove that fear instead of dodging it.
The concrete claim is narrower, but more useful. Leviathan et al. report 2x-3x speedup on T5-XXL without retraining or architecture changes, while preserving the target model's output distribution. A follow-up reports 2-2.5x on Chinchilla 70B with the same distribution-preserving angle.
An update matters less for how many features it lists and more for whether it changes your next decision. That is why this reads like a rollout lever, not a lab flex. DSpark: Speculative decoding accelerates LLM inference [pdf] [C001]
Boundary: "same distribution" does not mean every real-world implementation will match token for token in production. These are paper-reported results on T5-XXL and Chinchilla 70B, with no production environment provided. Safe takeaway: this looks like a practical switch for rollout, not proof that every inference stack gets free speed with zero surprises. Share if that distinction is useful.