Test-Time Compute With Stop Rules

Test-time compute is valuable when extra samples or search steps buy verifiable correctness, and wasteful when they merely generate more confident-looking variations. This guide defines a stop rule before inference so difficult problems can receive more work without turning cost into an unbounded hope strategy.

The proposed decision frame is a per-problem budget with observable evidence. Its receipt includes difficulty buckets, candidate trajectories, verifier behavior, marginal accuracy, latency, token cost, and the exact condition that stopped each request.

The operating vocabulary connects inference-time scaling, best-of-N, verifier search, and reasoning budget once each while preserving one intent: deciding when additional work should stop. That vocabulary supports the article's test-time compute decision without creating a second intent.

1. Prompt
2. Candidates
3. Verifier
4. Stop

Test-time compute starts with a verifiable task

Extra generation is easiest to justify when correctness can be checked independently of the model's own confidence. The Scaling LLM Test-Time Compute paper studies how inference strategy and problem difficulty affect the value of additional computation. Adaptive inference starts with tasks whose correctness can be checked independently; otherwise extra sampling can amplify persuasive error. A proposed review of “Test-time compute starts with a verifiable task” has four inspectable moves.

• Define the answer unit
• Write deterministic checks where possible
• Separate partial from final correctness
• Keep uncheckable cases in their own cohort

The proposed evidence for test-time compute starts with a verifiable task is agreement between the proposed verifier and trusted outcomes. The test-time compute starts with a verifiable task receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is using fluency or self-consistency as universal proof. If using fluency or self-consistency as universal proof occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is adaptive allocation starts only where the verifier clears its calibration gate. Promotion under “adaptive allocation starts only where the verifier clears its calibration gate” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Bucket difficulty without leaking the answer

A routing signal should predict expected value from permitted request features rather than from post-hoc knowledge of failure. The s1 paper investigates explicit test-time budgets and simple mechanisms for controlling reasoning length. Difficulty routing must use signals available before the answer, or the evaluation quietly gains information the production policy will not have. A proposed review of “Bucket difficulty without leaking the answer” has four inspectable moves.

• Choose pre-answer features
• Freeze easy, medium, and hard thresholds
• Validate on held-out problems
• Audit language and topic cohorts

The proposed evidence for bucket difficulty without leaking the answer is accuracy and cost inside each predicted bucket. The bucket difficulty without leaking the answer receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is calling items hard only after the first answer is known to be wrong. If calling items hard only after the first answer is known to be wrong occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is a bucket is useful only when its held-out marginal-gain curve differs materially. Promotion under “a bucket is useful only when its held-out marginal-gain curve differs materially” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Plot marginal accuracy against cost

Cumulative accuracy hides the price of the next candidate, branch, or verifier pass. The Rewarding Progress paper examines process signals for directing search toward more promising reasoning states. Marginal correct answers per added token reveal where another candidate stops earning its cost more clearly than cumulative accuracy. A proposed review of “Plot marginal accuracy against cost” has four inspectable moves.

• Measure N=1 as the visible baseline
• Add samples on a declared grid
• Count all generated and verifier tokens
• Plot p95 latency beside dollar cost

The proposed evidence for plot marginal accuracy against cost is incremental correct answers per additional unit of cost. The plot marginal accuracy against cost receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is selecting the highest accuracy point without a resource denominator. If selecting the highest accuracy point without a resource denominator occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is stop expanding when the next step falls below the declared value threshold. Promotion under “stop expanding when the next step falls below the declared value threshold” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Signal	Decision	Evidence
N=1	Easy default	$1.00 · 71%
N=4	Ship hard	$3.58 · 82%
N=16	Reject	$13.6 · 83%

Encode the stop rule before sampling

An executable policy prevents a promising trajectory from negotiating for endless exceptions. A named stop reason makes proof, diminishing value, and hard budget visible outcomes instead of hidden implementation details. A proposed review of “Encode the stop rule before sampling” has four inspectable moves.

• Stop immediately on deterministic proof
• Set a hard candidate cap
• Set token and wall-clock ceilings
• Record a machine-readable stop reason

The proposed evidence for encode the stop rule before sampling is a fixture covering proof, diminishing return, and exhausted budget. The encode the stop rule before sampling receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is adding candidates until one answer feels persuasive. If adding candidates until one answer feels persuasive occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is every request exits on proof, low expected gain, or the hard budget. Promotion under “every request exits on proof, low expected gain, or the hard budget” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Runnable artifact. Save this bounded specimen as test-time-compute-stop-rules.test.mjs and run node --test test-time-compute-stop-rules.test.mjs. Expected result: PASS: proof and budget stop search.

import assert from "node:assert/strict";
import test from "node:test";
const stop = ({ verified, samples, max }) => verified ? "proof" : samples >= max ? "budget" : "continue";
test("names every exit", () => {
  assert.equal(stop({ verified: true, samples: 2, max: 4 }), "proof");
  assert.equal(stop({ verified: false, samples: 4, max: 4 }), "budget");
  console.log("PASS: proof and budget stop search");
});

Calibrate the verifier

A learned critic can turn additional search into reward hacking unless its false approvals and false rejections are measured. Verifier precision at the actual stop threshold matters more than a broad classification score that includes unused regions. A proposed review of “Calibrate the verifier” has four inspectable moves.

• Blind candidate identity and order
• Use adversarial near-miss answers
• Plot reliability by problem type
• Route uncertain decisions to a stronger check

The proposed evidence for calibrate the verifier is precision at the threshold used to terminate search. The calibrate the verifier receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is treating an uncalibrated score as a probability of correctness. If treating an uncalibrated score as a probability of correctness occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is automatic proof requires the verifier precision declared in the release contract. Promotion under “automatic proof requires the verifier precision declared in the release contract” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Preserve candidate diversity

Repeated samples only help when they explore meaningfully different solution paths rather than paraphrasing the same mistake. Candidate diversity is an economic constraint: repeated paraphrases of one failed approach should not consume the remaining budget. A proposed review of “Preserve candidate diversity” has four inspectable moves.

• Hash normalized final answers
• Cluster intermediate approaches
• Penalize duplicate branches
• Retain one representative failure per cluster

The proposed evidence for preserve candidate diversity is new verified outcomes per distinct trajectory cluster. The preserve candidate diversity receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is paying for sixteen cosmetic variations of one reasoning path. If paying for sixteen cosmetic variations of one reasoning path occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is allocate the next sample only when unresolved approach space remains. Promotion under “allocate the next sample only when unresolved approach space remains” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

1. ClassifyClassify

   Estimate difficulty from allowed pre-answer signals.

2. GenerateGenerate

   Create bounded candidates with saved trajectories.

3. VerifyVerify

   Apply task-grounded checks and calibrated confidence.

4. StopStop

   Exit on proof, low marginal gain, or hard budget.

Bound latency for interactive use

A policy that improves offline accuracy may still be wrong for a person waiting in a product flow. Interactive deadlines turn theoretically useful search into product failure when hard prompts monopolize latency and shared capacity. A proposed review of “Bound latency for interactive use” has four inspectable moves.

• Set product-specific wall clocks
• Stream honest progress without fake certainty
• Permit cancellation
• Choose a lower-cost fallback

The proposed evidence for bound latency for interactive use is task success and abandonment across latency bands. The bound latency for interactive use receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is letting difficult prompts monopolize shared serving capacity. If letting difficult prompts monopolize shared serving capacity occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is interactive requests stop or degrade before their experience-level deadline. Promotion under “interactive requests stop or degrade before their experience-level deadline” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Publish the allocation policy

A result is reproducible only when each request carries its budget, branch count, verifier revision, and stop reason. The compute distribution by difficulty bucket is part of the published result because it explains what the accuracy number cost. A proposed review of “Publish the allocation policy” has four inspectable moves.

• Version difficulty and verifier models
• Report cost by bucket
• Show rejected frontier points
• Keep examples that exhausted the cap

The proposed evidence for publish the allocation policy is replaying the policy on the fixed evaluation set. The publish the allocation policy receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this test-time compute decision is reporting a final score without the compute distribution that bought it. If reporting a final score without the compute distribution that bought it occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is another reviewer can derive the same stop decisions from saved trajectories. Promotion under “another reviewer can derive the same stop decisions from saved trajectories” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Keep the boundary visible

Allocate extra work where a held-out difficulty bucket shows useful marginal gain and an independent verifier can recognize success. Candidate diversity, resource cost, and interactive latency belong beside accuracy.

The stop reason is part of the answer's provenance. Proof, diminishing expected value, and hard budget are all legitimate endings; invisible overspending is not.

Continue through four related field notes: LLM judge calibration, AI evaluation contracts, LLM routing by cost and risk, human review escalation. Each extends the test-time compute method without changing this article's single search intent.