Prompt Caching Metrics for AI Products

Prompt caching metrics are useful only when eligible tokens, reused tokens, latency, and price assumptions share one receipt. Here is a byte-first way to find a fractured prefix before a global hit rate hides it.

Prompt caching metrics begin with ordered bytes

Prompt caching metrics answer a narrower question than “did the request look familiar?”: how many eligible input tokens reused an exact leading sequence under a declared model, cache key, and retention window. Serialize the system message, developer instructions, tool schemas, examples, and stable conversation history in their real request order, then assign that byte sequence a prefix version before comparing anything.

The OpenAI prompt caching guide describes automatic reuse of exact prefixes and exposes cached input tokens in usage details. Its operational lesson is structural: put material that changes least near the beginning, move request-specific content later, and preserve the same order instead of assuming a logically equivalent tool list has the same cache identity.

This article’s fixture is synthetic and the lab makes no provider call. It hashes ordered JSON bytes so one changed character, reversed tool array, model switch, cache-key switch, or simulated expiry has a visible and specifically named consequence rather than becoming an unexplained miss.

Treat the prefix as an ordered byte contract, not a bag of repeated ideas. A moved tool definition changes every byte after the move, while a revised user tail can leave the expensive shared beginning intact; that distinction decides which engineering fix is worth making.

Separate eligibility, reuse, and savings

A cache hit rate can hide three different denominators. Request hit rate counts requests with any reuse, cached-token share divides cached input tokens by eligible input tokens, and prefix coverage measures how far the shared leading bytes extend before the first mismatch; keep all three because a workload can score highly on one while wasting most of its input on another.

Prompt caching metrics should also preserve the ineligible region. A short request below a provider’s eligibility threshold is not a prefix failure, and a long user tail after a stable prefix is neither a miss nor reusable evidence; label those tokens separately so the metric does not punish expected request-specific work.

The synthetic receipt therefore records eligible tokens, cached tokens, shared leading bytes, and a reason code for every request. Aggregate only after the row-level ledger balances, because cohort averages cannot reveal a serialization change that affects one prefix version or tenant.

The first receipt should name the exact serializer, model route, tenant boundary, and prefix version. Without those dimensions, two requests that look identical in a dashboard can have different wire representations, retention rules, or eligibility thresholds and therefore should not share an operational conclusion.

Version the prefix as a release artifact

Treat the stable prompt prefix like a small deployable artifact. Give its system instructions, tool definitions, examples, and response contract a version; compute a digest from the bytes actually sent; and put that identifier beside application and model versions in every usage receipt.

The OpenAI Cookbook example demonstrates why ordering matters across repeated requests. A reviewable version prevents two releases from sharing a friendly label while producing different serialized prefixes, and it makes a deliberate invalidation distinguishable from accidental churn.

When a tool schema changes, create a new prefix version even if the tool name stays fixed. That discipline also helps teams isolate shared prompt prefixes by tenant, because a cache key can describe an authorized cohort without pretending the provider’s undisclosed routing is known.

LLM prompt caching belongs beside, but never inside, semantic answer caching. The former can reuse an exact leading computation while still generating a new answer; the latter may return stored application content and carries separate freshness, authorization, and invalidation obligations.

Runnable artifact — This is a deterministic teaching simulator. It does not call a provider, predict a provider's undisclosed cache routing, or claim production latency.

import assert from "node:assert/strict";
import { createHash } from "node:crypto";

const hash = (value) => createHash("sha256").update(value).digest("hex");
const encode = new TextEncoder();
const commonBytes = (a, b) => {
  const x = encode.encode(a), y = encode.encode(b);
  let index = 0;
  while (index < x.length && index < y.length && x[index] === y[index]) index++;
  return index;
};
const visibleTokens = Object.freeze(Array.from({ length: 1024 }, (_, index) => "fixture_token_" + String(index).padStart(4, "0")));
const stable = Object.freeze({
  model: "gpt-teaching-1",
  cacheKey: "tenant-7",
  prefixTokens: visibleTokens,
  tools: Object.freeze([
    Object.freeze({ name: "lookup", schema: Object.freeze({ type: "object", required: Object.freeze(["id"]) }) }),
    Object.freeze({ name: "cite", schema: Object.freeze({ type: "object", required: Object.freeze(["url"]) }) }),
  ]),
});
const serialize = (input) => JSON.stringify({ model: input.model, cacheKey: input.cacheKey, prefixTokens: input.prefixTokens, tools: input.tools });
const changeToken = (index, value) => Object.freeze(visibleTokens.map((token, tokenIndex) => tokenIndex === index ? value : token));
const variants = [
  { id: "stable-repeat", input: stable, reason: "hit", reusableDomain: true },
  { id: "changed-token", input: { ...stable, prefixTokens: changeToken(768, "fixture_token_CHANGED") }, reason: "prefix_mismatch", reusableDomain: true },
  { id: "tool-order", input: { ...stable, tools: Object.freeze([...stable.tools].reverse()) }, reason: "serialized_tool_order_changed", reusableDomain: true },
  { id: "model-domain", input: { ...stable, model: "gpt-teaching-2" }, reason: "model_cache_domain_changed", reusableDomain: false },
  { id: "retention-expired", input: stable, reason: "retention_expired", reusableDomain: false },
  { id: "cache-key-domain", input: { ...stable, cacheKey: "tenant-8" }, reason: "cache_key_changed", reusableDomain: false },
];
const stableSerialized = serialize(stable);
const eligibleInputTokens = stable.prefixTokens.length;
assert.ok(eligibleInputTokens >= 1024, "fixture must contain at least 1,024 committed visible tokens");
const receipts = variants.map((variant, index) => {
  const serialized = serialize(variant.input);
  const exactPrefix = serialized === stableSerialized;
  const hit = variant.id === "stable-repeat" && exactPrefix && variant.reusableDomain;
  const cachedInputTokens = hit ? eligibleInputTokens : 0;
  const sharedLeadingBytes = commonBytes(stableSerialized, serialized);
  const deterministicWorkUnits = encode.encode(serialized).length + (index + 1) * 97;
  const uncachedInputTokens = eligibleInputTokens - cachedInputTokens;
  return {
    id: variant.id,
    fixtureEligibility: { tokenizer: "committed visible-token fixture v1", visibleInputTokens: eligibleInputTokens, cacheable: eligibleInputTokens >= 1024 },
    prefixSha256: hash(serialized),
    sharedLeadingBytes,
    providerUsageModel: { source: "deterministic teaching model; no provider call", eligibleInputTokens, cachedInputTokens, uncachedInputTokens },
    missReason: hit ? "hit" : variant.reason,
    deterministicWorkUnits,
    estimatedInputUsd: Number((cachedInputTokens * 0.0000005 + uncachedInputTokens * 0.000002).toFixed(6)),
  };
});
const expectedCached = new Map([["stable-repeat", 1024], ["changed-token", 0], ["tool-order", 0], ["model-domain", 0], ["retention-expired", 0], ["cache-key-domain", 0]]);
for (const row of receipts) {
  assert.equal(row.providerUsageModel.cachedInputTokens, expectedCached.get(row.id));
  assert.equal(row.providerUsageModel.cachedInputTokens + row.providerUsageModel.uncachedInputTokens, row.providerUsageModel.eligibleInputTokens);
  assert.ok(row.sharedLeadingBytes >= 0 && row.sharedLeadingBytes <= encode.encode(stableSerialized).length);
}
const csvHeader = "id,eligible_input_tokens,cached_input_tokens,uncached_input_tokens,shared_leading_bytes,miss_reason,work_units,estimated_input_usd";
const csv = [csvHeader, ...receipts.map((row) => [row.id, row.providerUsageModel.eligibleInputTokens, row.providerUsageModel.cachedInputTokens, row.providerUsageModel.uncachedInputTokens, row.sharedLeadingBytes, row.missReason, row.deterministicWorkUnits, row.estimatedInputUsd].join(","))].join("\n");
const exportPayload = { fixtureVersion: "visible-token-cache-fixture-v1", providerCalled: false, stableSerializedBytes: encode.encode(stableSerialized).length, receipts, csv, csvSha256: hash(csv) };
const json = JSON.stringify(exportPayload, null, 2);
const result = { ...exportPayload, jsonSha256: hash(json) };
console.log(JSON.stringify(result, null, 2));
console.log("PASS: eligible-token fixture, zero-domain misses, CSV, and every cache row reconcile");

Measure prompt prefix stability before latency

Prompt caching metrics become diagnostic when they include a stability score over actual request bytes. For each cohort, compare the candidate prefix with its frozen reference, retain the common-leading-byte length, and identify the first field whose serialization crosses that boundary; do not normalize arrays or whitespace after the request has already been formed.

Stability belongs upstream of latency because a cache cannot plausibly influence a request that never presented the same eligible prefix. Segment by prefix version, model, region if disclosed, retention mode, cache key, and request class before comparing p50 or p95, otherwise deployment mix can manufacture an apparent speedup.

The teaching simulator uses fixed latency samples only to demonstrate receipt arithmetic. Those numbers are not a forecast for a hosted model; a real experiment needs contemporaneous cached and uncached cohorts, identical output requirements, enough observations for uncertainty, and explicit exclusion of retries or rate-limit delay.

Cached input tokens are a provider receipt, not proof that an entire prompt was reused. Preserve eligible tokens, matched-prefix tokens, uncached tail tokens, and the provider's reported cached count on the same row so a partial match remains visible instead of becoming a binary hit.

A final privacy boundary belongs in the cache contract: hashes can still become stable identifiers, and prefix versions can reveal deployment or tenant structure. Restrict access, rotate or salt identifiers when cross-system correlation is unnecessary, and retain the minimum row evidence needed to diagnose reuse. Redaction must happen before export, not after a prompt fragment has entered a broadly searchable log.

Read cached input tokens from usage receipts

Provider usage is the authority for cached token accounting when that field exists. Capture total input tokens and the nested cached-token value without reconstructing the number from latency, because network conditions, scheduling, generation length, safety processing, and server load can change duration independently of prefix reuse.

The OpenAI documentation names cached token reporting, while Anthropic’s prompt caching documentation exposes its own cache creation, read, boundary, and time-to-live semantics. These are separate contracts, so use provider-specific adapters that emit one normalized internal ledger without erasing which source field produced each value.

Prompt caching metrics should fail closed when a receipt lacks the expected accounting field. “Unavailable” is more truthful than zero: zero means the provider reported no cached tokens, while unavailable means the application cannot establish either outcome from the response it retained.

A prompt cache hit rate becomes meaningful only after cohorting by stable prefix version. One high-volume, healthy version can overwhelm a new broken cohort in the global average, so publish both request-weighted totals and a small multiple for every version still receiving traffic.

Turn token receipts into cost deltas

Cost savings require a dated price table and the provider’s distinction between cached and uncached input. Multiply each token class by its applicable unit price, preserve currency and price effective date, and keep output charges outside the cache delta unless a separate analysis deliberately models changed output behavior.

The arithmetic should reconcile to the request invoice dimension before aggregation. Agent cost attribution can then allocate cached input tokens to a run, feature, or tenant while retaining the original usage receipt rather than distributing a guessed percentage from an account-level bill.

Do not call a lower estimated input charge realized savings until an invoice or billing export supports it. In the lab, every dollar amount is an explicit synthetic price assumption used to test calculation shape; replacing the table is part of running the artifact for a real product.

Latency attribution needs a paired view because request shape and queue conditions are confounders. Compare repeats within the frozen synthetic cohort, retain p50 and p95, and label the result as teaching latency rather than extrapolating its milliseconds to a provider or production region.

Compare latency inside controlled cohorts

A credible latency view includes request start, first response byte or first token when available, completion, model, prefix version, cached-token share, output tokens, and error status. Compare distributions rather than single averages, and report sample counts beside p50 and p95 so a tiny new cohort cannot masquerade as a stable improvement.

Prompt caching metrics can be exposed to a web boundary through Server-Timing for AI latency, but only fields safe for the client should leave the server. Detailed hashes, tenant keys, prompt fragments, and provider receipts belong in protected telemetry with access controls and retention rules.

A useful evaluation pairs successive prefix versions during a bounded rollout. If cached share rises but first-token latency does not move, inspect output mix and queueing before concluding the cache failed; if duration falls without reported cached tokens, investigate other release changes instead of awarding the cache credit.

Cost deltas should be recomputable from token classes and a dated price table. Keep currency units explicit, retain full-precision intermediate values, and round only the presentation layer; otherwise tiny per-request rounding errors become surprisingly large when a finance ledger aggregates millions of calls.

Explain misses with mutually useful reasons

Prefix mismatch, model-domain change, cache-key change, retention expiry, eligibility shortfall, and missing usage evidence require different repairs. Choose one primary reason at the earliest verified checkpoint, then retain secondary context fields so dashboards do not double-count one request across several failure buckets.

One-byte mutations deserve special treatment because they often reveal timestamp insertion, nondeterministic schema emission, reordered JSON properties, or a deployment identifier embedded too early. The simulator’s exact-byte diff intentionally refuses semantic comparison, which mirrors the engineering lesson even though it does not reproduce any provider’s private cache implementation.

Keep semantic answer reuse outside this taxonomy. A semantic cache invalidation strategy handles whether a prior answer remains acceptable for a related meaning; prompt caching concerns computation reused for a repeated leading input, so mixing them corrupts both safety and economics.

Use prompt caching metrics to decide whether to stabilize bytes, raise eligible volume, or change nothing. In this guide, prompt caching metrics stay dispersed through the receipt rather than collapsing into one vanity percentage; related terms include prompt prefix stability and cached input tokens.

Build a cache release dashboard

A release dashboard should start with prefix version and request class, then show request count, eligible input tokens, cached input tokens, cached share, exact-prefix stability, miss reasons, first-token latency, completion latency, and estimated input cost. Link each aggregate back to a small redacted sample of row receipts so investigators can verify the denominator.

Display deployment markers where prefix bytes, tools, model, cache key, or retention settings changed. That timeline helps reviewers distinguish expected cold-start behavior from persistent churn, while a comparison table prevents a healthy high-volume cohort from hiding a broken low-volume workflow.

Prompt caching metrics are most actionable when one owner can answer a miss. Route serialization drift to the prompt release owner, absent receipts to telemetry, model changes to platform engineering, tenant-key problems to authorization, and price-table staleness to FinOps rather than treating “cache” as an indivisible service.

A schema release deserves its own prefix version even when its human meaning barely changes. Pin canonical JSON rules, reject accidental key reordering in tests, and keep the old cohort visible during rollout so the graph shows a controlled migration instead of an unexplained cache fracture.

Ship a reproducible cache measurement contract

Write the contract before declaring success: one workload, one prefix version, an eligibility rule copied from the current provider contract, exact usage fields, stable cohort dimensions, latency boundaries, price table, exclusions, and a revisit trigger. Store sample receipts and the aggregation query together so a later release can reproduce the same question.

Run the included Node lab, inspect the stable repeat, and trace each mutation to its named miss reason. Then replace only the fixture and price assumptions with rights-cleared application data, keeping secrets and prompt content out of exported evidence; the simulator teaches instrumentation and cannot validate a hosted cache route.

The practical threshold is not a universal hit percentage. Set a product-specific target for cached share and latency or cost delta, require enough traffic to interpret it, and review again when provider accounting, cache retention, cache-key behavior, or pricing changes.

The revisit trigger is contractual, not ceremonial: rerun the generated corpus when pricing, usage fields, retention, or cache-key guidance changes. Prompt caching metrics must be recalculated before the visible date changes, then the prefix-fracture graphic can be republished with the same synthetic-data caveat.