Structured Output Validation Beyond Schema

Structured output validation begins after a model returns parseable JSON, because a well-shaped object can still encode an impossible or unauthorized action. This tutorial builds a syntax, schema, and domain-invariant pipeline, then keeps authorization as a separate final gate before any side effect.

Structured output validation has four boundaries

Treat generated arguments as untrusted input even when constrained decoding produced them. Boundary one parses bytes into a value; boundary two checks the supported LLM JSON schema; boundary three proves business invariants across fields and current state; boundary four asks whether this actor may perform this action on this resource. Structured output validation should return a typed rejection at each boundary instead of one vague “invalid tool call.”

The distinction prevents authority laundering. A payload such as {"amount": 5000, "currency": "USD", "account": "A"} may be valid JSON and schema-conformant while exceeding the account's remaining limit or targeting an account outside the user's tenant. Semantic validation explains whether the requested state transition is coherent. Authorization remains separate because coherent actions are not automatically permitted actions.

Draw the gates as separate trace spans with one sanitized outcome each. Reviewers can then prove that a denied object stopped before execution and that a provider-side schema success never skipped local policy.

Test a read-only dry run that returns the normalized proposal and every gate receipt. It gives reviewers a safe way to understand what would happen without granting the model a shadow execution path.

Freeze one contract and reject unknown meaning

Start with a narrow operation, such as scheduling a refund, and define every accepted field, unit, range, enum, and null rule. Set additionalProperties to false when the operation has a closed vocabulary, because ignored fields create room for conflicting meanings. Version the schema and place its identifier in traces so structured output validation can be reproduced after the contract evolves.

Use the JSON Schema 2020-12 core specification for vocabulary semantics, and compare code-boundary tradeoffs in JSON Schema vs Zod. Schema validation can require an ISO-looking date string, positive amount, and known currency. It cannot know whether that date falls inside the refund window or whether the currency matches the original charge unless those facts enter a domain check.

Add positive and negative examples beside every important constraint. Examples are not substitutes for validation, but they expose unit mistakes and nullable-field ambiguities early enough for prompt, API, and executor owners to resolve them together.

Document whether defaults are inserted by the producer, validator, or executor. Hidden defaulting can make the signed or reviewed object differ from the action that finally reaches the transactional boundary.

Use constrained decoding for shape, not truth

Current OpenAI Structured Outputs guidance describes enforcing a supplied supported JSON Schema. That capability can reduce malformed objects and repair loops, but the bounded claim matters: schema adherence does not establish fresh business state, ownership, authorization, or factual truth. Pin the model and API behavior used by a fixture, because supported schema features and platform behavior can change.

Constrained decoding should simplify the first two gates, never erase the last two. Keep the same structured output validation behind model calls, cached outputs, human-edited retries, and direct API clients so no alternate path bypasses invariants. If a provider refuses an unsupported schema feature, fail contract installation visibly rather than silently weakening the schema and teaching operators to trust a contract the runtime never enforced.

Keep a provider-independent corpus at the application boundary. When a model or SDK version changes, replay the same accepted and rejected objects so a successful integration test proves your contract, not merely the vendor's happy path.

Treat refusals as a distinct, valid model outcome rather than malformed output. A refusal should stop cleanly and preserve its reason without being repaired into a tool call by generic retry logic.

Write invariants as named pure decisions

Domain checks deserve names that a reviewer can challenge: originalChargeExists, currencyMatches, amountWithinRemainingBalance, refundWindowOpen, and destinationIsOriginalMethod. Pass a snapshot of required state into a pure validator and return all relevant violations without performing the action. This shape makes semantic validation deterministic in tests and prevents database reads from being hidden inside error formatting.

Distinguish validation races from validator design. After a payload passes against snapshot version 42, a concurrent refund may consume the remaining balance before execution. Enforce the decisive invariant again inside the transaction or compare-and-swap boundary. Structured output validation is evidence for attempting a transition, not a lock on future state. Log the snapshot version and final conflict without storing unnecessary customer data.

Return machine-readable violations with stable codes and bounded field paths. Human prose can change for clarity, while tests and retry policy continue to recognize currency_mismatch, limit_exceeded, window_closed, and stale_snapshot without parsing sentences.

Keep domain state inputs minimal and versioned. Passing an entire customer record into a validator increases exposure and makes a later receipt harder to reproduce than a purpose-built immutable snapshot.

Make mutation tests attack plausible objects

Happy-path examples prove very little. Begin with one authorized payload, then mutate one property at a time: delete currency, add an ignored override field, turn an integer into a numeric string, use non-finite-number text, cross the maximum by one, swap tenant, expire the window, duplicate an idempotency key, and change state between validation and execution. Ajv's getting-started guide shows a common compiled-schema workflow; the fixture below exposes equivalent gates without adding a dependency.

Every mutation should name the gate that owns it and assert that no side effect occurred. This corpus turns tool argument validation into executable documentation for model prompts, API handlers, and security review. Keep representative rejected payloads redacted in telemetry so operators can distinguish a schema regression from a policy denial without logging secrets or full user content.

Generate mutations from the contract, then hand-author adversarial combinations the generator will miss. A foreign tenant plus an expired window, for example, verifies that the system reports safely without leaking whether a protected resource exists.

Include boundary values on both sides of every limit. Zero, maximum, maximum plus one, empty collections, and Unicode identifiers often expose coercion or serialization behavior that ordinary examples never touch.

The mutation corpus proves that parseable, schema-shaped arguments can still fail a domain invariant or authorization, and that only one fixture reaches the effect ledger.

Runnable artifact — semantic-validator-corpus.test.mjs

import assert from "node:assert/strict";
const parse=x=>{try{return{ok:true,value:JSON.parse(x)}}catch{return{ok:false,gate:"syntax"}}};
const validate=x=>Number.isInteger(x.amount)&&x.amount>0&&typeof x.account==="string"&&!Object.keys(x).some(k=>!["amount","account","tenant"].includes(k));
const check=(text,state,actor)=>{const p=parse(text);if(!p.ok)return p;const x=p.value;if(!validate(x))return{ok:false,gate:"schema"};if(x.amount>state.remaining)return{ok:false,gate:"domain"};if(x.tenant!==actor.tenant)return{ok:false,gate:"policy"};return{ok:true,value:x}};
const state={remaining:90},actor={tenant:"t1"},cases=[['{"amount":',"syntax"],['{"amount":"20","account":"a","tenant":"t1"}',"schema"],['{"amount":100,"account":"a","tenant":"t1"}',"domain"],['{"amount":20,"account":"a","tenant":"t2"}',"policy"]];
for(const [text,gate] of cases)assert.equal(check(text,state,actor).gate,gate);assert.equal(check('{"amount":20,"account":"a","tenant":"t1"}',state,actor).ok,true);console.log("PASS: only authorized semantic output reaches the effect");

Run node semantic-validator-corpus.test.mjs. Expected receipt: PASS: only authorized semantic output reaches the effect.

Keep authorization and idempotency outside the model

The model may propose an account, scope, or approval reference, but trusted server context must supply the actor and tenant. Resolve resource ownership from authoritative storage, evaluate policy, and bind the approved arguments to an idempotency key. The bounded programmatic tool-calling guide expands that execution envelope beyond structured output validation.

Return stable, non-sensitive problem categories rather than raw stack traces. HTTP Problem Details in TypeScript provides a useful error boundary for syntax_error, schema_error, invariant_error, forbidden, conflict, and effect_failed. A model can use those categories to repair a request, but cap attempts and never let a repair broaden scope, swap resource ownership, or convert a denial into a different operation.

Bind the idempotency key to the normalized operation and authorized resource. Reusing a key with changed arguments must return a conflict, not silently replay an earlier result or create a second effect under newly proposed meaning.

Require a fresh policy decision when repaired arguments change any protected field. A prior allow result belongs to one normalized proposal and must not become transferable approval for a nearby object.

Observe rejections without exposing payloads

Count outcomes by schema version, model version, operation, gate, and invariant name. A sudden increase in schema failures may mean prompt drift or provider behavior changed; a rise in domain failures may mean the model lacks fresh state; policy denials may reveal a dangerous request pattern. Structured output validation telemetry should describe boundary behavior without retaining account numbers, free text, access tokens, or full generated objects.

Sample a small redacted set for human review and preserve correlation IDs through generation, validation, authorization, and effect. Compare examples against executable API contracts so documentation payloads pass the same code. An alert should identify which gate changed and which contract version is affected, not merely announce that “the AI tool failed.”

Measure repair success separately from first-pass validity. A high eventual success rate can hide wasteful loops or repeated policy pressure, while a sudden zero-repair rate can expose an error contract the model no longer understands.

Create separate service-level objectives for validator latency and effect latency. Operators can then see when safety checks degrade without being pressured to bypass them merely to restore overall request time.

Release the validator before the prompt

Deploy new schema and semantic checks in observe-only mode against captured, redacted fixtures, then enforce them before asking a model to emit the new shape. This order avoids a window in which novel fields reach an older executor. Preserve backward compatibility only when both versions have explicit invariants; a catch-all adapter that guesses missing meaning weakens structured output validation at exactly the moment a contract is changing.

The release receipt should include the supported schema, compiled-validator version, invariant list, mutation corpus, policy reference, transactional recheck, and rollback path. Revisit the boundary whenever an operation gains a field or side effect. Parseable JSON is a useful transport milestone, but the safe product claim is smaller: one authorized transition survived every named gate and produced one reviewable effect.

Roll back prompts and validators as a compatible pair when possible. If emergency rollback leaves a new producer talking to an old executor, default denial and an explicit compatibility error are safer than coercion or unknown-field deletion.

Retire old contract versions deliberately after observing usage and replaying stored fixtures. An indefinite compatibility branch becomes an undocumented second executor whose invariants drift away from the current product.