Attention Sinks for Stable Streaming LLMs

Attention sinks keep a small set of initial key-value positions while a streaming cache evicts older conversational history. Paired with a recent window, those anchors can stabilize attention behavior without pretending that bounded memory preserves every fact from an unbounded conversation.

The implementation problem is a cache policy with explicit positions, resets, model compatibility, and quality decay. This guide uses a tiny eviction simulator as a worked fixture and treats every performance or quality claim as something to measure on the actual checkpoint.

Attention sinks define two kinds of retained time

Let a cache budget hold (S) initial positions and (W) recent positions. After the sequence exceeds (S+W), eviction removes the middle: positions ([0,S)) remain, while the tail advances with each token. The initial positions are architectural anchors; the tail is local conversational evidence. Neither is a summary of everything between them.

The StreamingLLM paper reports the importance of keeping initial tokens for stable streaming attention and frames the method for efficient long-sequence generation. Use its checkpoint and experiment context precisely. Attention sinks are not a universal switch: tokenization, positional encoding, cache layout, fine-tuning, and backend implementation determine whether the policy is compatible and useful.

Write the index policy before writing a ring buffer. Declare whether the beginning-of-sequence token counts toward (S), how prompt prefixes and chat templates occupy the anchor region, what happens before the cache fills, and which logical positions each KV block represents. A sliding-window KV cache that merely retains the last (S+W) tokens is the control, not the same algorithm.

Preserve logical positions through physical eviction

KV pages may move or reuse storage, but rotary or absolute position semantics still refer to logical token positions. Do not renumber the recent window to follow directly after the anchors unless the model and method explicitly require a remapping. Store logical positions beside block tables and test the exact kernel path used in production. An off-by-one can look fluent while steadily corrupting attention geometry.

The arXiv version of the streaming study is useful for method details and evaluation context. A serving implementation should add its own tensor-shape receipt: layers, KV heads, head dimension, dtype, block size, anchor positions, recent capacity, and bytes. Sink tokens refer to retained attention anchors; they should not be confused with user-visible memory tokens or a hidden store of old conversation.

Handle multi-token appends and chunked prefill. If one chunk crosses the budget, the resulting logical set must match token-at-a-time updates. Beam search, speculative decoding, prefix caching, and sequence forks need their own ownership rules. Attention sinks are safe only when every derived sequence can identify which cache pages and logical positions belong to it without mutating a sibling.

Runnable artifact: The cache simulator contrasts last-window eviction with two fixed anchors, covers warm-up and full-budget states, rejects an impossible budget, removes duplicates, and proves deterministic logical indices.

Save this worked fixture as attention-sink-cache.test.mjs and run node attention-sink-cache.test.mjs. Expected final line: PASS: 9 cache assertions.

import assert from "node:assert/strict";
const windowOnly=(tokens,budget)=>tokens.slice(-budget);
function withSinks(tokens,budget,sinks=2){
  if (budget < sinks) throw new Error("budget");
  return [...tokens.slice(0,sinks),...tokens.slice(-(budget-sinks))].filter((x,i,a)=>a.indexOf(x)===i);
}
const stream=[0,1,2,3,4,5,6,7]; let n=0; const check=fn=>{fn();n++};
check(()=>assert.deepEqual(windowOnly(stream,4),[4,5,6,7]));
check(()=>assert.deepEqual(withSinks(stream,4),[0,1,6,7]));
check(()=>assert.equal(withSinks(stream,4).includes(0),true));
check(()=>assert.equal(withSinks(stream,4).includes(5),false));
check(()=>assert.deepEqual(withSinks([0,1],4),[0,1]));
check(()=>assert.deepEqual(withSinks(stream,8),stream));
check(()=>assert.throws(()=>withSinks(stream,1),/budget/));
check(()=>assert.equal(new Set(withSinks(stream,4)).size,4));
check(()=>assert.deepEqual(withSinks(stream,4),withSinks(stream,4)));
assert.equal(n,9); console.log("PASS: 9 cache assertions");

Compare eviction policies on the same token trace

Build a trace that names each logical position and a small observation task. Run full cache where feasible, recent-only eviction, anchors plus recent window, and a deliberately broken renumbering control. Compare retained indices before model quality. The artifact's eight-token example produces ([4,5,6,7]) for recent-only and ([0,1,6,7]) for two anchors under budget four.

The MIT HAN Lab reference repository provides code and evaluation material connected to the research. Pin a revision rather than treating the default branch as permanent evidence. Port one small cache-index fixture before integrating the optimized implementation. If the backend already exposes a streaming policy, inspect its anchor count, position treatment, model support, and batching behavior instead of assuming the label implies identical semantics.

Attention sinks also need an ordinary full-context control at short lengths. Outputs should match the unmodified model before eviction begins, within the backend's numerical tolerance. When the cache first overflows, trace which indices disappear. A quality curve without an eviction trace cannot distinguish architectural degradation from an implementation that retained the wrong blocks.

Evaluate local fluency and missing-history behavior separately

Perplexity or next-token quality over long streams can show stability, but a product also needs tasks that reveal what eviction means. Include local continuation, recent instruction following, early system constraint adherence, facts placed just inside and just outside the recent window, and explicit retrieval from an external memory layer. Report quality over stream position rather than one final average.

Long dialogue inference is particularly easy to overclaim. Fixed anchors do not make an old preference or promise available after its tokens leave the cache. Product designs should summarize, retrieve, or ask again, with provenance and user control. Attention sinks address attention behavior under bounded state; they are not a complete conversational memory architecture.

Compare language, code, structured tool calls, repeated templates, and adversarial long filler. Inspect whether system prompts fit inside the retained anchors and whether chat-template changes shift meaningful instructions out of that region. RoPE scaling recall probes offer related long-context controls, but scaled full context and bounded streaming answer different workload questions. Keep both baselines where the product may choose between them.

Treat reset as a first-class state transition

Every new conversation, tenant change, model revision, tokenizer revision, system-prompt change, and explicit user reset should create a fresh streaming generation. Cancellation must release pages. A resumed handle should verify generation and configuration hashes before accepting another token. Otherwise bounded memory becomes a cross-session contamination channel.

Mamba streaming state has a different recurrent representation but the same operational lesson: chunking, reset, checkpoint, and version ownership belong in the interface. For attention sinks, record anchor indices and recent range on every debug receipt. Use synthetic token IDs in broad telemetry and reserve content-bearing traces for controlled investigations.

Test empty input, one token, exactly full budget, first overflow, a chunk larger than the budget, repeated reset, concurrent streams, cancellation during append, and forked decoding. A reset test should confirm both logical emptiness and physical page reclamation. The next stream must not inherit initial KVs merely because an allocator reused the same device memory. Stable output is irrelevant if state belongs to the wrong principal or conversation.

Budget memory from tensor shapes, then measure churn

Calculate payload bytes as retained tokens × layers × KV heads × head dimension × two tensors × bytes per element, then add block tables, allocator slack, alignment, graph workspaces, and prefix-cache ownership. Attention sinks bound the token term, but the system can still fragment under many short-lived streams. Measure admitted concurrency and page churn on the real request distribution.

KV cache optimization provides the wider serving ledger. Report time to first token, inter-token latency, tokens per second, memory per active sequence, allocation failures, and reset latency for full, recent-only, and anchor-window policies. Separate prefill from decode and include mixed stream lengths. A smaller cache that triggers inefficient copies can lose the promised latency benefit.

Context parallelism is a training or full-context distribution strategy rather than a replacement for bounded streaming; context parallelism for long context helps compare that alternative. Choose from workload needs: exact access to a bounded long document, stable indefinite local continuation, or remembered product facts. Those are different contracts and should not share one “supports long context” badge.

Deploy behind compatibility and drift gates

Create an allowlist by model, tokenizer, positional scheme, backend, dtype, quantization, and cache kernel. Unknown combinations use the supported default or fail configuration validation. Run the logical-index simulator and short full-cache equivalence in CI, then a longer quality suite before promotion. Attention sinks should be a versioned cache policy, not an environment variable applied to every checkpoint.

Monitor retained-count invariants, duplicate logical positions, missing anchors, page ownership, generation mismatch, full-budget utilization, and quality canaries. Trigger rollback on state corruption or compatibility errors; quality drift can use bounded review thresholds. Do not log prompt contents to explain every eviction. Index receipts and synthetic sentinel sequences can reveal most cache defects safely.

Publish model hash, template, anchor count, recent capacity, block size, logical-position policy, eviction trace, reset semantics, tensor bytes, allocator overhead, quality-by-position curves, throughput profile, and fallback. That receipt states what attention sinks actually accomplish: a stable, finite attention substrate for a named workload. It also states what they discard, which is the essential truth a streaming product must design around.