Speculative Decoding Without Quality Loss

Speculative decoding can make an LLM feel faster by proposing several tokens cheaply and verifying them with the target model in parallel. The useful question is not whether the technique is fast in a demo, but when it preserves the target distribution and improves the latency experienced by your actual workload.

This guide turns that question into a paired benchmark, a tiny verifier simulation, and a workload-fit table. It is for serving engineers who need evidence about acceptance, batching pressure, rollback work, and the exact boundary behind the word lossless.

The evaluation separates draft model choice, token acceptance rate, inter-token latency, and lossless decoding so a throughput claim cannot hide a changed distribution.

Speculative decoding changes the schedule, not the model

Speculative decoding lets a cheaper process guess a short continuation while the authoritative model evaluates those positions in one pass. Accepted guesses save sequential target steps; the first unsuitable guess is replaced according to the verifier rule. The target remains the source of the final distribution.

The Leviathan, Kalman, and Matias paper derives a sampling method that preserves the target distribution under its stated assumptions. That guarantee belongs to the algorithm and sampler combination, not to every serving feature that happens to use a draft network.

Separate algorithmic equality from deterministic equality. Two valid stochastic runs need not produce identical text, even when both sample from the same distribution. A sound evaluation compares distribution-sensitive outcomes or matched random streams rather than declaring any token difference a quality regression.

My position is that the technique should be marketed as a scheduling optimization, not a smarter model. The boundary matters because quality claims invite the wrong tests, while scheduling claims direct attention to verification, accepted work, queue effects, and time between visible tokens.

Implement the draft and verify loop explicitly

Speculative decoding starts from a shared prefix state for target and drafter. The drafter proposes gamma tokens and records its probability for each. The target evaluates the same continuation positions in parallel, producing authoritative probabilities used by the acceptance rule.

With greedy output, keep the longest run whose draft token equals the target's choice. With sampling, use the correction rule defined by the selected algorithm; naive equality or a homemade probability threshold can bias results. Name the algorithm and sampler in every experiment.

After a rejection, discard later proposals because they depend on an invalid prefix. Sample or select the authoritative token at that position, update both states, and start the next cycle. Count rejected suffix work so an attractive acceptance percentage cannot conceal expensive over-proposal.

Keep target-only decoding available in the same code path. It provides a control, a rollback, and a safe route for unsupported samplers or request features. A serving feature that cannot bypass speculation cleanly is much harder to diagnose when output or latency changes.

Illustrative TypeScript — a reviewable design sketch, not a compiled production implementation.

type Verification = {
  proposed: number[];
  acceptedPrefix: number[];
  authoritativeToken: number;
  targetPasses: number;
  rejectedSuffix: number[];
};

function verifyCycle(input: VerificationInput): Verification {
  // Apply the named paper's acceptance and correction rule here.
  return targetDistributionPreservingVerification(input);
}

Signal	Decision	Proof
High agreement	Use a longer proposal	Paired output distribution holds
Batch pressure	Shorten or disable	Queue and verifier cost improve
Low agreement	Target-only decode	Rollback waste removed

Build a paired verifier simulation

Speculative decoding is easiest to understand with a tiny vocabulary and fixed probability tables. Create one target distribution and one approximate proposer for four positions. Feed both target-only and speculative paths the same controlled random values, then inspect accepted runs and correction samples.

The worked fixture proposes A, B, C, D. The target accepts A and B, rejects C, samples X from the corrected distribution, and discards D because its prefix no longer exists. The committed sequence is A, B, X; the next cycle starts after X with synchronized state.

Run the fixture thousands of times over enumerated seeds and compare token frequencies with the target-only control within a predeclared statistical tolerance. This does not prove a production implementation, but it catches the common biased shortcut where a rejected proposal is replaced without the required correction.

Preserve a human-readable trace for a few seeds: proposal probabilities, target probabilities, acceptance draw, accepted prefix, discarded suffix, correction distribution, and committed token. That artifact makes a subtle probability contract reviewable without requiring a reviewer to infer behavior from throughput charts.

Measure acceptance as saved target steps

Speculative decoding needs more than an average accepted-token count. Report accepted tokens per target pass, the distribution of accepted run lengths, rejected suffix tokens, proposer compute, verifier compute, and end-to-end response timing. Long accepted runs matter because they replace sequential authoritative passes.

Segment by prompt type, output length, temperature, sampling features, language, and request priority. Code completion may align closely with a related drafter, while creative prose at higher temperature may diverge. One blended number can lead to a policy that helps a narrow cohort and taxes everyone else.

Compare time to first token and the gaps between later tokens separately. The proposer can add setup work before the first visible token even when later output arrives faster. Product experience depends on both phases, especially for short responses that end before the steady-state advantage appears.

Use a saved-target-passes metric rather than raw acceptance alone: accepted positions minus verification and rollback overhead expressed in equivalent target work. It is imperfect but closer to the mechanism, and it exposes configurations where a high agreement ratio still loses after scheduling and memory costs.

Account for batching and memory pressure

Speculative decoding competes with continuous serving for accelerator slots. A verifier pass covers several positions, which can alter batch shapes, key-value memory growth, and fairness between speculative and ordinary requests. A single-request benchmark cannot reveal those interactions.

Replay mixed traffic at increasing concurrency. Include short target-only requests, long high-agreement requests, and low-agreement requests. Observe queue age and completion time for every class. The feature fails the service if speculative work delays higher-priority ordinary traffic beyond its agreed budget.

Bound proposal length dynamically using recent agreement and current scheduler pressure. Longer runs are attractive when the system is lightly loaded and the drafter is aligned; shorter runs reduce waste when queues are deep. Keep the rule deterministic enough to explain in a trace.

The vLLM feature documentation is the versioned reference for supported configurations in that engine. Validate the precise release you deploy; serving implementations can add constraints that are outside the papers' theoretical model.

1. ProposePropose

   The smaller model emits a bounded token run.

2. ScoreScore

   The target evaluates all proposed positions together.

3. AcceptAccept

   The verifier keeps the valid prefix under the declared sampler.

4. ResumeResume

   The target samples at the rejection boundary and continues.

Name the workloads that are a bad fit

Speculative decoding is a poor fit when responses are extremely short, drafter agreement is low, verification is memory-bound, or the serving engine cannot batch the mixed work efficiently. It may also be unsupported for a sampler, adapter combination, structured constraint, or model architecture.

A particularly deceptive failure mode is speedup on generated tokens alongside slower completed requests. Queue interference or setup cost can raise user-perceived delay even while the decoding loop looks efficient. Mitigate it by optimizing for task completion and measuring cohorts under shared load.

Do not force one proposal length across all requests. Classify workloads using stable inputs such as model pair, decoding settings, expected output band, and current pressure. If the classification is uncertain, choose the target-only path rather than turning an optimization into mandatory overhead.

Quality-sensitive structured generation deserves an explicit compatibility test. Validate schema adherence, stop sequences, log-probability options, and constrained samplers under the speculative path. Distribution preservation in the core algorithm does not guarantee that every surrounding feature has been integrated correctly.

Roll out with a decoding receipt

Speculative decoding should emit a receipt containing target revision, proposer revision, algorithm, sampler settings, proposed count, accepted count, discarded count, target passes, policy decision, and timing phases. Keep the receipt small and sample detailed probability traces only in protected diagnostic environments.

Start with offline distribution fixtures, then shadow scheduling estimates without changing output. Enable a low-consequence cohort and compare against a simultaneous target-only control. Expand by workload class only when both correctness checks and completion-time guardrails remain healthy.

Define automatic fallback for unsupported options, low recent agreement, excessive queue pressure, or verifier errors. Fallback must preserve the request state and continue authoritatively, not restart the response or expose duplicated text. Exercise that transition in the benchmark before release.

Version the eligibility policy separately from engine configuration. A later engine may change memory cost or supported algorithms while the product's workload rule remains stable. The two identifiers let an incident review distinguish policy choice from runtime behavior.

Prove the release with three views

Speculative decoding is ready when the verifier simulation matches the target distribution, paired offline tasks show no material quality or constraint regression, and shared-load tests improve completed-request timing for the eligible cohort without harming protected traffic.

Keep three views in the release packet: the probability-level fixture, the workload decision matrix, and the service-level latency chart. They answer correctness, eligibility, and operational value respectively. Omitting any one produces an argument that is easy to overgeneralize.

Review bad-fit routes as carefully as successful routes. The system should explain why it chose target-only decoding and demonstrate that the bypass is cheap. A high enablement rate is not the goal; selecting the technique only where it pays is the goal.

Schedule a revisit when the target, proposer, engine, or traffic mix changes. Agreement is a relationship between particular models on particular work, not a permanent model property. Re-running the paired corpus protects the claim from quiet drift.

Put the method into practice

Speculative decoding is worthwhile when a named verification algorithm preserves the target behavior and accepted proposals remove more sequential work than they add in drafting, verification, memory, and queue pressure. That conclusion must be demonstrated per workload class.

Start with the four-token verifier fixture and a target-only control. Once the distributions agree, replay mixed traffic and keep the feature only for cohorts whose completed responses become faster without moving delay onto other users.

The narrow method connects to four existing Journal notes: LLM routing by cost, risk, and latency, AI streaming UX without jitter, AI evaluation measurement contracts, architecture decisions need baselines. They cover adjacent implementation boundaries without changing this article's single search intent. Preserve the worked fixture, its visual evidence, and the release receipt so a later update can compare behavior instead of relying on memory.