LLM Metamorphic Testing Without Gold Labels

LLM metamorphic testing asks a sharper question than “is this answer perfect?” It asks whether a declared transformation—reordering facts, paraphrasing a request, adding irrelevant text, or flipping one premise—produces the relationship the product promises.

That paired view is useful when ideal free-form outputs are expensive or ambiguous. The method still needs human judgment: relations come from product semantics, and a passing consistency test cannot prove truth, safety, or usefulness on its own.

LLM metamorphic testing starts with a product promise

Write the observable contract first. A support classifier should keep its label when non-causal details move. A grounded answer should preserve cited claims when sentence order changes.

A tool planner may change arguments when a quantity changes, while retaining tool choice. LLM metamorphic testing turns those promises into paired assertions; random text mutation merely creates more inputs.

The worked fixture uses a refund-routing feature. Its source case contains charge duplication, order identifier, amount, and an unrelated color sentence. Reordering or paraphrasing the facts should preserve the “refund” route.

Deleting the duplicate-charge premise should predict a different or abstaining route. This is semantic invariance testing at a narrow decision boundary, not a claim that two essays should share every word.

For each relation, name preconditions, transformation, expected invariant or directional change, fields compared, tolerance, and an explanation a failure can expose. AI evaluations need measurement contracts offers the broader discipline: construct, unit, cohort, and decision must agree. A relation without a product decision becomes an impressive-looking consistency number nobody knows how to act on.

Separate invariants from equivariants and anti-relations

An invariant expects the same normalized outcome: reorder a list but keep the classification. An equivariant expects a predictable transformation: translate the requested locale and expect the answer locale to follow. A monotonic relation expects a score or risk decision not to move in the wrong direction when evidence strengthens.

An anti-relation changes a causal premise and expects the output to change. These families prevent “consistent” from becoming an unquestioned virtue.

LLM metamorphic testing should compare structured observations whenever possible: selected action, cited source IDs, extracted fields, refusal category, calibrated band, or constraint violations. Free-form similarity can assist review but should not replace the actual product contract. Oracle-free LLM evaluation is best understood as reducing reliance on exact answers; it does not eliminate oracles because the relation itself encodes expert expectations.

The 2025 metamorphic-testing preprint provides a current research anchor for applying the method to language-model systems. Record exactly which relation definitions and systems you implement rather than borrowing the paper's validity wholesale. A relation may be appropriate for classification and harmful for creative generation, where controlled variation is part of the intended behavior.

Generate transformations with bounded provenance

Store the source fixture as the authority. A transformation function should emit the new prompt, relation identifier, changed spans, preserved facts, expected comparison, generator version, and seed. Prefer deterministic edits for reorder and distractor cases. If another model paraphrases text, retain its prompt and validate that required facts survived before the case can score the system under test.

Metamorphic relations can compose, but composition increases ambiguity. A paraphrase plus locale change plus distractor may fail without revealing which property broke. Start with one axis, then add pairwise interactions only where production prompts genuinely combine them. Keep an identity transform as a repeatability control and a deliberately invalid transform to confirm that the relation validator rejects corrupted cases.

Tests should challenge generated intent applies directly: generated fixtures need assertions against the intent they claim to test. For a refund case, compare order ID, amount, charge count, customer request, and irrelevant material before admitting a paraphrase. LLM metamorphic testing becomes safer when the fixture builder can say “transformation invalid” rather than forcing every generated prompt into a model-quality score.

Runnable artifact: The small harness exercises reorder, paraphrase-shaped wording, distractor placement, case normalization, an identity control, and one causal contrast. It intentionally grades a route label rather than pretending lexical equality is a semantic oracle.

Save this worked fixture as metamorphic-relations.test.mjs and run node metamorphic-relations.test.mjs. Expected final line: PASS: 9 relation assertions.

import assert from "node:assert/strict";
const normalize = value => value.toLowerCase().replace(/[^a-z0-9 ]/g,"").split(/\s+/).filter(Boolean).sort().join(" ");
const decide = prompt => ({ label: /refund/.test(prompt.toLowerCase()) ? "refund" : "other", evidence: normalize(prompt) });
const sameLabel = (a,b) => decide(a).label === decide(b).label;
const fixtures = [
  ["Refund order 17 after duplicate charge", "After a duplicate charge, refund order 17"],
  ["Please refund the card purchase", "The card purchase should be refunded, please"],
  ["Classify: refund requested. Color is blue.", "Color is blue. Classify: refund requested."],
];
let n=0; const check=fn=>{fn();n++};
for (const [a,b] of fixtures) check(()=>assert.equal(sameLabel(a,b),true));
check(()=>assert.equal(sameLabel("refund invoice 2","REFUND INVOICE 2"),true));
check(()=>assert.equal(sameLabel("refund invoice 2","hello"),false));
check(()=>assert.equal(decide("hello").label,"other"));
check(()=>assert.equal(decide("refund").label,"refund"));
check(()=>assert.equal(normalize("b, A!"),"a b"));
check(()=>assert.deepEqual(fixtures, structuredClone(fixtures)));
assert.equal(n,9); console.log("PASS: 9 relation assertions");

Grade pairs before aggregating cases

Run source and transformed inputs with pinned model, system prompt, tool schema, decoding settings, and external state. Parse both through the same observation adapter. Grade the relation, then preserve the pair side by side.

A source failure and a relation failure are distinct: if both answers are consistently wrong, invariance passes while task quality fails. LLM metamorphic testing must report those axes separately.

For probabilistic endpoints, repeat a bounded number of seeds and estimate relation pass rate with uncertainty. Deterministic decoding can reduce noise but cannot guarantee deterministic infrastructure. Classify failures as source invalid, transform invalid, parser failure, base-task failure, relation violation, or infrastructure error. Only the relation violation belongs in the headline metric.

The IEEE Access article supplies another research perspective on metamorphic evaluation for language models. Production evidence should add product-shaped slices: languages, input lengths, tool availability, policy categories, and known fragile prompt forms. The useful unit is not “10,000 mutations.” It is a relation failure with enough provenance to reproduce the source, transform, outputs, and grader decision.

Calibrate judges on disagreements, not convenience

Some relations require semantic comparison: two rationales support the same action, or a paraphrase preserves all grounded claims. Create a compact rubric with allowed variations and disallowed changes. Calibrate a model judge against blinded human labels, stratified by relation and failure severity. LLM judge calibration shows why agreement, bias slices, and threshold selection belong in the receipt.

Use deterministic checks first. Exact action names, JSON fields, citation sets, numerical tolerances, and forbidden claims need no generative judge. Send only unresolved semantic pairs to the judge, and preserve its rationale as review material rather than truth. LLM metamorphic testing should fail safely when judge output is malformed or confidence lies near a decision threshold.

Audit directional bias. A judge may prefer the longer answer, the source order, or wording closer to its own training style. Swap pair order and paraphrase the rubric.

Include obvious pass and fail anchors. Human reviewers should see both outputs without being told which came from the candidate release. The goal is prompt robustness tests with an evidence chain, not an automated taste score that reproduces the same sensitivity being measured.

Read failures as a map of hidden dependencies

A reorder failure suggests positional reliance, truncation, or prompt-template coupling. A distractor failure points to salience control. A paraphrase failure can expose tokenizer, language, or instruction-form sensitivity.

A premise-flip non-response suggests the system repeats a default regardless of evidence. Group failures by relation and product slice before reading examples; the cluster often names the hidden dependency more clearly than an average.

LLM metamorphic testing benefits from counterfactual debugging. Remove half the transform, shorten the prompt, substitute one entity, or replace the model judge with a deterministic observation. Find the smallest paired change that retains the violation.

Preserve that minimized pair as a regression fixture. Do not automatically add every generated failure; unstable or invalid cases make the suite noisy and teach teams to ignore it.

Track relation coverage beside ordinary task coverage. Which actions, languages, policy classes, context lengths, tools, and user cohorts have at least one meaningful invariant and anti-relation? A hundred paraphrases of one refund sentence are not broad coverage. A smaller portfolio spanning independent risks is a stronger release instrument because each failure class leads to a different engineering or product response.

Gate releases with slices and explicit exceptions

Define minimum pass rates and zero-tolerance relations before running a candidate. A forbidden tool call under an irrelevant distractor may be a hard stop; a mild rationale-style drift may create a review queue. Compare against the current production model on identical pairs, report wins and regressions, and require an owner plus expiry for accepted exceptions. LLM metamorphic testing is most useful as a release comparison, not an abstract leaderboard.

Canary evals for safer AI releases connects offline fixtures to shadow and bounded production cohorts. Promote the highest-signal metamorphic pairs into those stages without sending sensitive prompts to unauthorized systems. Monitor whether relation failures correlate with user corrections, escalations, or tool reversals. Retire relations that no longer represent product semantics.

The newer metamorphic evaluation preprint is a refresh trigger rather than a substitute for local validity. Publish the source schema, transformation code, relation catalog, observation adapter, judge calibration, per-slice results, minimized failures, and release decision. Then a reader can inspect not only whether a model was consistent, but whether consistency was expected, meaningful, and sufficient for the decision that followed.

Maintain the relation catalog like an API

Give every relation an identifier, owner, semantic version, applicability predicate, examples, counterexamples, comparison fields, tolerance, severity, and retirement history. When the product contract changes, update the relation and rerun both production and candidate models. Do not rewrite an old relation in place and erase what a previous release actually passed.

LLM metamorphic testing should include a budget. Bound generated cases, model calls, judge calls, retries, and retained payloads. Cache only when the complete model and prompt identity match.

Redact or synthesize sensitive source fixtures while preserving the property under test. The worked refund examples are fixtures, not claimed customer transcripts or production outcomes.

The durable advantage is diagnostic leverage. A gold label says one output missed one target. A well-designed relation says the system depends on word order it was supposed to ignore, fails to react to evidence it was supposed to use, or changes citations when only tone changed.

Those statements are closer to engineering causes and product promises. Keep the catalog small enough to understand, broad enough to challenge hidden dependencies, and strict enough that every passing relation has a reason to matter.