Concept Briefing LLM Inference GPU Memory Bandwidth 2026-09-14

Why Speculative
Decoding Is Fast

Plain answer

A normal decode step loads all model weights to make one token. A verify step loads the same weights once and checks many draft tokens at the same time. The extra checks use compute time that was empty before. This is the real source of the speedup.

12%

Compute used per normal decode step

58%

Compute used per 5-token verify step

57–85%

Faster per user — DSpark in DeepSeek-V4 production

15%+

Token efficiency gain — GPT-5.6 draft model retune

The Bottleneck

A decode step needs two things: the model weights, and compute time to multiply numbers. Normal decode makes one token per step.

The step must load every weight first. Compute then runs for a short time. The chip waits idle until the next weight load starts.

This step is memory-bound. Weight movement sets the total time. Compute does not set the total time.

Normal decode1 token out

idle

compute busy ~12% of the step

Verify pass5 tokens scored

idle

compute busy ~58% — same weight load

In numbers One matrix-vector multiply (1 token) vs. one matrix-matrix multiply (K tokens). Same weight read. More useful output per read.

The Verify Pass

A small draft model works first. It makes K tokens one at a time, cheap and fast. This is the Scout step.

The big model then checks all K draft tokens together. It does not reread the prompt. The prompt already sits in a cache from earlier steps.

It reads its own weights once. It scores every draft position at the same time. This one pass costs about the same wall-clock time as one normal decode step.

Draft model proposes K tokens, one at a time

Big model reads weights once, against cached prompt

Big model scores all K positions in one pass

Each position accepted or rejected, in order

First rejection stops the chain; a new token is drawn

Accept rule Accept a draft token with probability min(1, p / q).
p = the big model's probability for that token.
q = the small model's probability for the same token.
A rejection stops the check. Every token drafted after it is dropped, with no exception.

The big model does not stop at a rejection. It draws one new token from its own leftover probability at that position. This token is called the bonus token. It comes from the big model itself, not from the draft.

What a Draft Token Is

A draft is a flat list of token IDs in this system, not a branching tree. Each token ID carries the small model's own probability for that pick.

Two designs build this list:

Design	How it builds K tokens	Cost as K grows
Autoregressive drafter	One token at a time. Each guess depends on the last one.	Rises with K — K small passes
Parallel drafter (e.g. DFlash)	All K tokens in one pass, using mask-token inputs and hidden states copied from the big model.	Stays near flat — 1 pass

DSpark, a 2026 paper from DeepSeek-AI, mixes both designs. A parallel step makes rough guesses for all K positions first. A small serial step then fixes weak spots between neighboring tokens, since independent parallel guesses drift apart the further ahead they look.

The Real Cost Trade-Off

This method does not lower total compute. It raises it.

The verify step scores K positions. Most systems accept only some of them.
The small model also runs its own pass, on top of that.
Total FLOPs go up. Wait time per token goes down. These are two different measures.

The saving has a limit, too. It depends on the chip sitting idle during normal decode.

Large-batch serving already fills that idle compute, because many requests share one weight load at once. At that point, a wide verify step no longer rides on free cycles. It competes for the same busy compute other requests need.

From the source DSpark's own paper states that checking wide draft blocks without limits "wastes critical batch capacity." Its fix is a load-aware verify length: shorter blocks when the server is busy, longer blocks when it is not.