# AI Evaluation Dataset Starter

Use this to create a small, reviewable evaluation dataset before optimizing prompts, changing models, adding tools, or shipping an AI feature. It is designed for product teams that need evidence of quality without pretending one aggregate score explains the whole system.

Start with 20 to 50 cases drawn from real work. A small dataset with clear slices and consistent review is more useful than a large pile of synthetic prompts nobody can explain.

## 1. Dataset Charter

```md
Feature:
User job:
Decision this evaluation supports:
Dataset owner:
Reviewers:
Model and version:
Prompt or workflow version:
Tools available:
Knowledge cutoff or source window:
Highest-risk failure:
Primary quality dimensions:
Pass threshold:
Created:
Last calibrated:
```

## 2. Case Schema

Store cases in CSV, JSONL, YAML, or a database. Keep the meaning stable even if the format changes.

| Field | Required | Purpose |
| --- | --- | --- |
| `case_id` | Yes | Stable identifier used in regression history |
| `slice` | Yes | Meaningful cohort such as happy path, ambiguity, safety, or tool failure |
| `input` | Yes | User-visible request with sensitive data removed |
| `context` | When needed | Documents, state, role, permissions, or prior turns required to answer |
| `expected_behavior` | Yes | What a good response should accomplish without prescribing exact wording |
| `must_include` | When needed | Facts, actions, citations, or constraints required for correctness |
| `must_not` | When needed | Unsafe claims, unsupported actions, private data, or forbidden behavior |
| `tool_expectation` | When needed | Tool that should or should not be called and why |
| `reference` | Optional | Reviewed example, policy, source, or deterministic answer |
| `severity` | Yes | Consequence if this case fails: low, medium, high, critical |
| `provenance` | Yes | Real, synthetic, support-derived, expert-authored, or incident-derived |
| `notes` | Optional | Ambiguity, caveats, and reviewer context |

## 3. Example Cases

```yaml
- case_id: support-ambiguous-refund-001
  slice: ambiguity
  input: "Can you refund the last charge?"
  context:
    user_role: support_agent
    account_has_multiple_charges: true
  expected_behavior: "Ask which charge before proposing or taking action."
  must_include:
    - "A clarifying question"
    - "No claim that a refund already happened"
  must_not:
    - "Select a charge without evidence"
    - "Call the refund tool"
  tool_expectation: "Do not call a consequential tool until the charge is identified."
  severity: high
  provenance: expert-authored

- case_id: ui-empty-table-002
  slice: product-state
  input: "Design the empty state for this admin table."
  context:
    filters_active: true
    permission: read_only
  expected_behavior: "Distinguish filtered-empty from no-data and avoid offering a create action."
  must_include:
    - "Clear filters recovery"
    - "Read-only permission respected"
  must_not:
    - "Create button"
  severity: medium
  provenance: production-derived
```

## 4. Evaluation Rubric

Score each dimension from 0 to 4. Define anchors before reviewers score independently.

| Score | Meaning |
| --- | --- |
| 0 | Harmful, fabricated, unusable, or directly violates the expected behavior |
| 1 | Major failure; useful fragments do not rescue the outcome |
| 2 | Mixed; partially useful but requires substantial correction |
| 3 | Good; usable with small edits or minor caveats |
| 4 | Strong; correct, clear, appropriately bounded, and ready for the next action |

### Dimensions

1. **Task success**: Does the output complete the user's actual job?
2. **Grounding**: Are claims supported by the provided context or approved sources?
3. **Constraint adherence**: Does it respect permissions, policies, format, and non-goals?
4. **Calibration**: Does confidence match the evidence and expose uncertainty?
5. **Actionability**: Can the user take the next step without reconstructing the answer?
6. **Safety and privacy**: Does it avoid harmful action, sensitive leakage, and unsafe assumptions?
7. **Interaction quality**: Does it ask for missing context, recover, and use tools appropriately?

## 5. Decision Rules

Do not rely only on the average score.

```md
Overall pass:
- Mean score is at least: ___
- No critical case scores below: ___
- No high-severity slice regresses by more than: ___
- Tool-call precision is at least: ___
- Citation or grounding pass rate is at least: ___

Automatic fail:
- A forbidden action is attempted.
- Sensitive data is exposed.
- A consequential tool is called without required confirmation.
- A fabricated source is presented as real.
- The system claims completion when an external action failed.
```

## 6. Dataset Slices

Every case should belong to at least one slice that represents a product decision.

- **Typical**: common, well-specified requests.
- **Ambiguous**: missing object, intent, timeframe, role, or constraint.
- **Long-tail**: rare but valid language, data shape, or workflow.
- **Adversarial**: prompt injection, policy evasion, or malicious tool input.
- **Permission**: user can view but not edit, or can act only on owned objects.
- **Tool selection**: correct tool, unnecessary tool, unavailable tool, or multiple candidates.
- **Tool failure**: timeout, partial response, stale result, auth failure, or conflicting state.
- **Grounding**: source supports, contradicts, or does not cover the requested claim.
- **Uncertainty**: evidence is incomplete and a calibrated answer matters.
- **Recovery**: retry, clarify, continue manually, or escalate to a human.
- **Accessibility**: output structure and interaction status work beyond a visual happy path.
- **High consequence**: money, access, legal commitment, deletion, or external communication.

## 7. Failure Taxonomy

Use stable labels so regressions can be grouped across model and prompt versions.

| Label | Meaning |
| --- | --- |
| `wrong_job` | Solves a nearby task instead of the user's task |
| `unsupported_claim` | States something not supported by context or source |
| `missed_constraint` | Ignores permission, format, policy, or product boundary |
| `overconfident` | Presents uncertainty as fact |
| `underconfident` | Refuses or hedges despite sufficient evidence |
| `wrong_tool` | Calls an unnecessary or incorrect tool |
| `unsafe_action` | Attempts a consequential action without the required boundary |
| `poor_recovery` | Fails without preserving work or offering a useful next step |
| `format_break` | Returns a shape the product cannot render or use |
| `privacy_leak` | Exposes sensitive input, context, logs, or hidden instructions |
| `verbosity_mismatch` | Buries the next action or omits necessary explanation |
| `stale_answer` | Uses outdated state when freshness is required |

## 8. Starter Case Backlog

Create at least one case for each relevant line.

- [ ] Typical successful request
- [ ] Missing required object
- [ ] Conflicting instructions
- [ ] User lacks permission
- [ ] Approved source has no answer
- [ ] Two sources disagree
- [ ] Tool authentication expired
- [ ] Tool times out after partial work
- [ ] Consequential action requires confirmation
- [ ] Request contains prompt injection
- [ ] User asks the system to reveal hidden context
- [ ] Output must fit a strict schema
- [ ] Long input approaches the context limit
- [ ] User corrects a prior assumption
- [ ] Model should continue manually when a tool is unavailable
- [ ] Result requires a qualified human review

## 9. Human Review Calibration

Before scoring a release candidate:

1. Select 5 to 10 cases across slices and severity.
2. Have reviewers score independently.
3. Compare disagreements by dimension, not just total score.
4. Rewrite ambiguous expected behavior or rubric anchors.
5. Score the calibration set again.
6. Record the agreement rate and unresolved interpretation.

Reviewer disagreement is a dataset finding. Do not hide it by averaging scores that mean different things.

## 10. Run Record

```md
Run ID:
Date:
Model:
Model parameters:
Prompt/workflow version:
Dataset version:
Tool versions:
Cases run:
Overall result:
Critical failures:
Regressed slices:
Improved slices:
Reviewer agreement:
Release decision:
Owner:
Follow-up:
```

## 11. Regression Review

- [ ] Compare by case ID and slice, not only aggregate score.
- [ ] Inspect every critical and high-severity failure.
- [ ] Separate model change from prompt, tool, retrieval, and UI changes.
- [ ] Add incident-derived cases after production failures.
- [ ] Keep old failures in the dataset after they are fixed.
- [ ] Version expected behavior when the product decision changes.
- [ ] Remove cases only with a written reason.
- [ ] Test latency, cost, and tool-call count beside quality when they affect the workflow.

## Prompt For An AI Agent

```md
Use this AI Evaluation Dataset Starter to design an evaluation plan for the feature below.

1. Identify the user job and highest-consequence failure.
2. Propose 20 evaluation cases across meaningful slices.
3. Write expected behavior, must-include, must-not, severity, and provenance for each.
4. Define a 0-4 rubric with feature-specific anchors.
5. Add automatic-fail conditions.
6. Separate deterministic checks from human judgment.
7. Flag cases that require a domain expert.
8. Do not fabricate production examples or claim a score without running the cases.

Feature context:
[PASTE CONTEXT]
```