AI Evaluation Measurement Contract

# AI Evaluation Measurement Contract

Use this before an AI evaluation is designed, run, or cited in a release decision. It connects a product claim to a defined population, measurement method, uncertainty statement, failure policy, and decision owner.

This is not a substitute for statistical or domain expertise. It is a working agreement that makes assumptions visible early enough to change the evaluation.

## Quick Start

1. Name the decision the evaluation is allowed to inform.
2. Write one falsifiable product claim.
3. Define the users, scenarios, environments, and time window covered by the claim.
4. Choose the observation unit and quality dimensions.
5. Define graders, disagreement handling, and critical failures.
6. Set release rules before seeing the result.
7. Record uncertainty, missing coverage, and follow-up monitoring with the score.

If a field cannot be completed, mark it `unknown`, assign an owner, and decide whether the gap blocks the run or only limits the conclusion.

---

## 1. Decision Header

```md
Evaluation name:
Product or feature:
Decision to support:
Decision options: [ship / hold / route to review / limited release / investigate]
Decision owner:
Evaluation owner:
Domain reviewer:
Run deadline:
Evidence freshness window:
Contract version:
Last changed:
```

### Decision Boundary

```md
This evaluation may authorize:

This evaluation may not authorize:

Additional evidence required before release:

Conditions that require a new evaluation:
```

Do not let a model-level benchmark authorize a system-level release when retrieval, tools, permissions, latency, recovery, or user behavior are outside the run.

## 2. Product Claim

Write the claim so a failed case can contradict it.

```md
For [population]
using [workflow and environment],
the system will [observable behavior]
within [quality / latency / cost boundary],
without [unacceptable failure],
so that [user or business outcome].
```

### Claim Review

- [ ] The population is specific.
- [ ] The behavior is observable.
- [ ] The claim names a product outcome, not only model capability.
- [ ] The environment includes relevant tools, data, permissions, and UI.
- [ ] Failure can be detected from collected evidence.
- [ ] The claim does not imply coverage the evaluation cannot provide.

### Non-Claims

```md
This evaluation does not establish:
-
-
-
```

## 3. Evaluation Objective

Select the primary objective. Add another contract when the objectives require different evidence.

- [ ] **Capability** — can the model perform a bounded task under controlled conditions?
- [ ] **Workflow quality** — does the complete product system support the user job?
- [ ] **Regression** — did a model, prompt, tool, policy, or interface change make known behavior worse?
- [ ] **Comparative selection** — which candidate configuration is preferable for a defined use?
- [ ] **Risk discovery** — where and how does the system fail?
- [ ] **Field impact** — what happens when intended users use the product in realistic conditions?

```md
Primary objective:
Why this objective matches the decision:
Why other objectives are out of scope:
```

## 4. Target Population And Evaluation Sample

Separate the fixed cases being measured from the broader population the result is meant to represent.

```md
Target user population:
Target scenario population:
Languages and locales:
Devices or channels:
Permission roles:
Data shapes:
Time or policy window:
Excluded populations:
Known coverage gaps:
```

### Sampling Plan

| Slice | Why it matters | Source | Cases | Minimum evidence | Highest failure severity |
| --- | --- | --- | ---: | --- | --- |
| Typical path |  |  |  |  |  |
| Ambiguous input |  |  |  |  |  |
| Long-tail valid use |  |  |  |  |  |
| Permission boundary |  |  |  |  |  |
| Tool or retrieval failure |  |  |  |  |  |
| High-consequence case |  |  |  |  |  |
| Adversarial use |  |  |  |  |  |
| Accessibility or locale |  |  |  |  |  |

### Representativeness Note

```md
Cases were selected by:
Cases are representative of:
Cases are not representative of:
Synthetic cases are used for:
Production-derived cases were de-identified by:
Potential selection bias:
```

## 5. Unit Of Analysis

Choose the unit that matches the product claim.

- [ ] Single model response
- [ ] Multi-turn session
- [ ] Complete workflow
- [ ] Tool call
- [ ] Claim or citation
- [ ] User task completion
- [ ] Human review decision
- [ ] Production incident or intervention
- [ ] Other: ___

```md
Observation unit:
Why it matches the claim:
Repeated observations per case:
How repeated observations are grouped:
What counts as independent evidence:
```

## 6. System Under Evaluation

Pin the complete configuration. A score without a reproducible system identity cannot support a later comparison.

```md
Model and version:
Provider:
Prompt or policy version:
Retrieval corpus and snapshot:
Embedding or reranker version:
Tools and versions:
Tool permissions:
Temperature / seed / sampling settings:
Maximum turns or tool calls:
Interface or workflow version:
Feature flags:
Runtime region:
Evaluation harness commit:
Dataset version:
```

## 7. Measurement Specification

| Dimension | Operational definition | Method | Scale | Direction | Aggregation |
| --- | --- | --- | --- | --- | --- |
| Task success |  |  |  | Higher / lower |  |
| Grounding |  |  |  | Higher / lower |  |
| Constraint adherence |  |  |  | Higher / lower |  |
| Tool behavior |  |  |  | Higher / lower |  |
| Recovery |  |  |  | Higher / lower |  |
| Safety / privacy |  |  |  | Higher / lower |  |
| Latency |  |  |  | Higher / lower |  |
| Cost |  |  |  | Higher / lower |  |

### Metric Guardrails

```md
Primary metric:
Why it represents the claim:
Secondary metrics:
Metrics reported by slice:
Metrics that must not be averaged together:
Known metric blind spots:
```

Never let a good average erase a critical failure or an unevaluated population.

## 8. Grader Contract

Complete one block for every programmatic, model-based, or human grader.

```md
Grader name:
Grader type: [deterministic / model / human / domain expert]
Dimension graded:
Inputs visible to grader:
Inputs hidden from grader:
Rubric version:
Output scale:
Calibration set:
Known grader bias:
Conflict-of-interest risk:
Fallback when grader fails:
Owner:
```

### Human Review Plan

```md
Reviewer qualifications:
Number of reviewers per case:
Blind or identified conditions:
Calibration cases:
Agreement measure:
Disagreement threshold:
Adjudication process:
Cases requiring a domain expert:
```

### Model-Grader Controls

- [ ] The grader does not evaluate its own hidden reasoning.
- [ ] The rubric contains behavior-specific anchors.
- [ ] Position and formatting sensitivity were checked.
- [ ] A human-reviewed calibration set exists.
- [ ] Critical failures are not decided by one model grader alone.
- [ ] Grader model, prompt, and settings are versioned.

## 9. Failure Taxonomy

| Failure label | Observable definition | Severity | Automatic gate? | Owner |
| --- | --- | --- | --- | --- |
| Wrong task | Solves a nearby job instead of the requested job |  |  |  |
| Unsupported claim | States a fact not supported by allowed evidence |  |  |  |
| Missed constraint | Ignores policy, permission, format, or non-goal |  |  |  |
| Unsafe action | Attempts a consequential action outside its boundary |  |  |  |
| Poor recovery | Fails without preserving work or a useful next step |  |  |  |
| False completion | Claims an action succeeded when it did not |  |  |  |
| Privacy leak | Exposes sensitive data or hidden context |  |  |  |
| Other |  |  |  |  |

### Severity Anchors

```md
Critical — unacceptable because:
High — release-limiting because:
Medium — correctable but material because:
Low — limited friction because:
```

## 10. Statistical And Uncertainty Plan

Define this before viewing the final scores.

```md
Estimand — the quantity being estimated:
Fixed benchmark or generalized population claim:
Point estimate:
Uncertainty method:
Confidence or credible interval:
Sources of variation:
Minimum sample by slice:
Repeated runs per case:
Missing-result handling:
Outlier handling:
Multiple-comparison handling:
Practical significance threshold:
Statistical assumptions:
Assumptions not tested:
Analysis owner:
```

If the team lacks enough evidence for a generalized claim, report performance on the fixed evaluation set and state that boundary plainly.

## 11. Release Rules

Write gates before the run to reduce threshold shopping.

```md
Ship when:
-
-

Ship only behind review or limited exposure when:
-
-

Hold when:
-
-

Automatic failure conditions:
-
-

Required sign-offs:
```

### Slice Gates

| Slice | Metric | Threshold | Critical-failure allowance | Decision if missed |
| --- | --- | ---: | ---: | --- |
|  |  |  |  |  |

## 12. Run Manifest

```md
Run ID:
Started / completed:
Operator:
System configuration hash:
Dataset version:
Cases attempted / completed:
Retries:
Failed tool calls:
Grader versions:
Human review sample:
Raw evidence location:
Analysis notebook or commit:
Deviations from contract:
```

## 13. Results And Decision Record

```md
Primary result:
Uncertainty statement:
Slice results:
Critical and high failures:
Reviewer disagreement:
Coverage gaps:
Unexpected findings:
Result interpretation:
Result must not be interpreted as:
Decision:
Decision owner:
Conditions attached to decision:
Follow-up owner and date:
```

## 14. Production Link

The evaluation should connect to field evidence after release.

```md
Production signals:
User correction or override signal:
Human escalation signal:
Incident trigger:
Sampling and privacy rules:
Monitoring owner:
Refresh cadence:
Drift trigger:
Rollback trigger:
```

Add production failures to the evaluation set with provenance. Do not silently rewrite old expected behavior to make a regression disappear.

## 15. Change Control

A new version or rerun is required when any checked item changes materially.

- [ ] Product claim or user population
- [ ] Model or provider
- [ ] Prompt, policy, or system instructions
- [ ] Retrieval source or freshness rule
- [ ] Tool, permission, or side effect
- [ ] User interface or review boundary
- [ ] Dataset composition
- [ ] Grader or rubric
- [ ] Release threshold
- [ ] Relevant policy or domain rule
- [ ] Production failure pattern

```md
Change:
Reason:
Expected measurement impact:
Contract sections updated:
Approved by:
```

---

## Worked Example: Support Drafting Assistant

```md
Decision: Limited release to billing-support agents with review required.

Claim: For English and Spanish refund-policy questions submitted by trained
support agents, the complete drafting workflow will produce a policy-grounded
draft with the correct escalation path, without inventing eligibility or
claiming an action occurred, in at least 95% of evaluated sessions.

Population: Refund-policy questions from the current policy window. Excludes
chargebacks, legal threats, and unsupported locales.

Unit: Complete session, including retrieval, draft, source display, and review.

Critical failures: Invented refund eligibility, disclosure of another account,
or a claim that money moved when no tool action occurred.

Graders: Deterministic policy checks, two calibrated support reviewers on every
high-severity case, and one model grader for style only.

Uncertainty: Report the session pass rate with an interval, plus separate slice
results. Do not generalize beyond the sampled policy topics and languages.

Release gate: Zero critical failures; lower confidence bound above the agreed
minimum for policy adherence; every draft remains editable and review-required.

Field link: Monitor edits, escalations, source opens, and policy corrections.
Refresh after policy, retrieval, prompt, or model changes.
```

## Prompt For An AI Agent

```md
Use this AI Evaluation Measurement Contract to critique the evaluation below.

1. Restate the product claim and the decision the evaluation can support.
2. Separate the target population from the fixed evaluation sample.
3. Identify mismatches between the claim, observation unit, metrics, and graders.
4. Find averages that could hide severe or slice-specific failures.
5. Review statistical assumptions and missing uncertainty.
6. Propose pre-committed release gates and automatic failures.
7. List missing evidence and label each as blocking or limiting.
8. Do not invent results or claim statistical validity without the run data.

Evaluation context:
[PASTE CONTEXT]
```

## Research Basis

- [NIST AI 800-2 draft practices for automated benchmark evaluations](https://www.nist.gov/news-events/news/2026/01/towards-best-practices-automated-benchmark-evaluations)
- [NIST AI 800-3 on statistical models and uncertainty in AI evaluation](https://www.nist.gov/publications/expanding-ai-evaluation-toolbox-statistical-models)
- [NIST AI Risk Management Framework — Measure](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/)

Adapt this contract to the consequence of the system. A low-risk writing aid and an agent that can change money, access, identity, or production data should not share the same evidence threshold.