RAG Chunking Evaluation for Answerable Context

RAG chunking evaluation should begin with a simple failure question: did segmentation destroy or separate the evidence needed to answer? Average chunk length cannot reveal that boundary failure.

This guide labels answer spans in a small document corpus, then compares fixed, sentence, and heading-aware chunkers on answer span coverage, retrieval recall, context overlap, redundancy, and token cost. The result is a workload-specific frontier, not a universal chunk size.

Define RAG chunking evaluation around evidence

A useful example is a policy manual where the phrase naming a refund window sits beside the condition that starts the clock. The label stores question, exact evidence spans, acceptable supporting spans, document version, and answerability. If the answer requires two distant sections, record that explicitly rather than pretending one chunk should contain everything.

Chunking is evaluated before generation: first ask whether a chunker preserved a retrievable evidence unit, then whether the retriever surfaced it, and only then whether a model used it. This decomposition makes a weak answer diagnosable instead of attributing every failure to the language model.

Runnable artifact: The reference fixture proves that a fixed character boundary can split an answer phrase while sentence and overlapping chunks preserve it. Its chunk-span-survival.test.mjs receipt keeps the article's simplified boundary executable and reviewable.

Save the inspectable proof as chunk-span-survival.test.mjs and run node chunk-span-survival.test.mjs. Expected final line: PASS: answer spans measured.

import assert from "node:assert/strict";
const text="Alpha keeps context. The refund window is thirty days after delivery. Omega closes context.";const answer="refund window is thirty days";
const fixed=(s,n)=>Array.from({length:Math.ceil(s.length/n)},(_,i)=>s.slice(i*n,(i+1)*n));const sentence=s=>s.split(/(?<=\.)\s+/);const coverage=chunks=>chunks.some(x=>x.includes(answer));
assert.equal(coverage(fixed(text,30)),false);assert.equal(coverage(sentence(text)),true);const overlap=[text.slice(0,55),text.slice(35,90)];assert.equal(coverage(overlap),true);console.log("PASS: answer spans measured");

Build a corpus that stresses chunk boundaries

Random queries underrepresent the cases chunking is most likely to break. Include headings followed by short definitions, tables whose row labels matter, list introductions, code plus explanation, cross-page sentences, footnotes, and facts near the beginning and end of long material. Add adversarial placements where an answer phrase crosses a fixed boundary by a few tokens.

Split corpus development from evaluation so parameter tuning cannot move the targets. Preserve the raw document and extraction output, because a PDF parser or HTML cleaner can destroy structure before the chunker receives it.

Compare fixed, sentence, and heading-aware chunkers

Implement each chunker against the same normalized document. Fixed token windows provide a simple baseline and predictable budget. Sentence grouping respects punctuation but can still detach a heading or table label.

Heading-aware chunks preserve editorial hierarchy while risking very large sections that need secondary splitting. Record chunk start and end coordinates, inherited heading path, and parent document ID for every output. The goal is not to crown the most sophisticated parser; it is to learn which structural assumptions preserve evidence in this corpus without creating unusable retrieval units.

Measure answer span coverage before retrieval

Answer span coverage asks whether at least one emitted chunk contains every span required by the label, or which spans survive when the question is multi-hop. Report full coverage, partial coverage, and destroyed-by-boundary counts. Also measure how much irrelevant context surrounds preserved evidence, because a giant chunk trivially contains more spans while diluting retrieval and consuming context.

The boundary SVG makes this distinction visible: overlap can rescue one phrase, but repeated context increases index size and may surface near-duplicates. This stage isolates the segmentation ceiling that no embedding or reranker can exceed.

Add retrieval recall and rank-sensitive measures

Index each chunk set with the same embedding model and search configuration, then query the same held-out questions. Retrieval recall at k counts whether a fully covering chunk—or the declared required set—appears in the candidate list. Add reciprocal rank or another rank-sensitive view so a hit at position one differs from a hit at twenty.

If hybrid search or reranking is used, stage results separately to show whether the chunker, retriever, or reranker rescued the example. Freeze nondeterministic services or repeat runs enough to expose instability rather than mixing it into a single score.

Price overlap as redundancy and context cost

Context overlap is not free insurance. Measure emitted tokens divided by source tokens, duplicate n-grams across neighboring chunks, index bytes, candidate near-duplication, and final prompt tokens. Plot those costs against evidence coverage and retrieval recall.

A moderate overlap may repair fragile sentence boundaries; more overlap can flatten the gains while increasing indexing and inference cost. Heading inheritance may add a small repeated context prefix that improves retrieval more efficiently than copying entire trailing windows. The right frontier is the smallest cost that reaches the workload's agreed evidence floor.

Inspect positional and citation behavior

Retrieval success still does not guarantee the model uses evidence. Build a secondary generation evaluation where identical supporting text appears at different positions in the assembled context, informed by research on lost-in-the-middle behavior. Require citations to preserve document and span identity through chunk merges, reranking, and answer rendering.

A heading-aware chunk can improve readability yet cite too broad a section if coordinates are discarded. Keep this stage downstream of the segmentation metrics, so a generator failure does not cause arbitrary chunk-size changes that hide the actual boundary.

Choose and revisit a workload-specific frontier

Summarize each candidate with answer-span survival, retrieval recall at agreed k, rank, redundancy, index cost, prompt cost, and citation precision. Select the simplest configuration that meets the release floor and inspect its remaining failures. Document content type matters: legal clauses, API references, support threads, and narrative essays may need different policies.

Monitor extraction drift, empty chunks, chunk-length distributions, retrieval misses, and user-reported citation failures after launch. Re-run the labeled boundary corpus whenever the parser, embedding model, reranker, or document population changes materially.

Read the evidence in retrieval order

Lost in the Middle motivates position-sensitive context tests, the RAGAS paper supplies evaluation vocabulary, and Anthropic's contextual retrieval note offers a production-oriented technique to test rather than assume. Continue with this journal's experiments on hybrid search with RRF, retrieval reranking, citation durability, and late-interaction retrieval after the boundary corpus has exposed which failures belong to segmentation and which occur later.

Build the boundary corpus before tuning knobs

Copy a dozen real document fragments in which the answer crosses a heading, list item, table row, sentence boundary, or boilerplate seam, then label the minimal evidence span and its parent document. Run every candidate chunker on that frozen corpus before embedding anything; segmentation failures should be visible without retrieval noise. RAG chunking evaluation becomes actionable when the team can point to specific spans lost, duplicated, or made unaffordable by a configuration, rather than debating whether 512 or 800 tokens sounds generally sensible.

Retain the chunk texts and stable identifiers for every miss, not merely aggregate recall. A RAG chunking evaluation review should let an engineer inspect whether the answer span was severed, surrounded by misleading boilerplate, pushed beyond a rank cutoff, or retrieved intact but ignored later by generation; each diagnosis leads to a different change.

Run the chosen configuration once more on documents held out by template and source, then compare the frontier rather than a single headline score. This second RAG chunking evaluation prevents a handful of familiar boundary fixtures from becoming an accidental tuning set and exposes whether overlap cost travels to unseen layouts.

RAG chunking evaluation chooses a workload frontier rather than a universal window. Repeat the boundary study when documents, extraction, embeddings, retrieval, or citation behavior changes.