OpenAI Batch API With Replay-Safe Results
A replay-safe OpenAI Batch API pipeline validates JSONL, reconciles every output by custom_id, preserves partial results, and retries only unresolved items.
OpenAI Batch API reliability begins after the upload: output order is not a reconciliation key, partial results are valuable, and a timeout does not justify replaying every item. Build the pipeline around stable custom IDs and an append-only local ledger.
This tutorial follows twenty illustrative JSONL requests through validation, submission, cancellation, expiry, duplicate ingestion, and selective retry. It is a control-plane pattern for batch inference and asynchronous AI jobs, not a claim about one production run's speed or savings.
Design the OpenAI Batch API ledger first
Treat each requested item as a durable record before creating a file. The record needs a workload ID, stable custom_id, payload digest, endpoint, model configuration, consent or retention classification, attempt number, and business destination. A batch ID is transport identity, not item identity; one workload can span several batches after validation errors, cancellation, or expiry.
The official batch object exposes lifecycle timestamps, request counts, output file, and error file, but the application ledger decides which business items are still unresolved. That separation prevents a completed batch from being confused with a completely reconciled workload.
Validate JSONL requests before upload
Generate JSONL requests from normalized records, one JSON object per line, and fail locally on duplicate custom_id values. Validate that every line uses the intended endpoint and method, contains the pinned model configuration, and stays within documented file and request constraints. Secrets never belong in the file body, and user data should follow the endpoint's current data-control policy.
Persist the exact input-file digest after generation so an operator can prove which payload set created a provider batch. A regenerated file with different whitespace may be semantically equivalent, but it is still a different operational artifact.
- 1Validate
Freeze JSONL + digest
- 2Submit
Bind provider IDs
- 3Reconcile
Join by custom_id
- 4Retry
Only unresolved items
Submit once and persist identity atomically
Upload the file, create the batch, and store provider identifiers with the local workload in a transaction or compensating sequence. If the client loses the response, search the application ledger and provider inventory before creating another job; do not assume the absence of a local ID proves submission failed. Metadata can carry bounded correlation hints, while the database remains the authoritative mapping.
Polling should use backoff and record status transitions rather than overwrite one status field without history. The OpenAI Batch API reports validating, in progress, finalizing, completed, failed, expired, cancelling, and cancelled states that deserve distinct operator language.
Runnable artifact: The local reconciler accepts deliberately unordered rows, catches a duplicate custom ID, and derives retries from unresolved identity rather than position. Its batch-reconciler.test.mjs receipt keeps the article's simplified boundary executable and reviewable.
Save the inspectable proof as batch-reconciler.test.mjs and run node batch-reconciler.test.mjs. Expected final line: PASS: unordered batch reconciled.
import assert from "node:assert/strict";
const requested=new Map(Array.from({length:20},(_,i)=>["item-"+i,{attempt:1}]));
const output=[{custom_id:"item-4",ok:true},{custom_id:"item-0",ok:true},{custom_id:"item-9",error:"request_timeout"},{custom_id:"item-4",ok:true},{custom_id:"item-7",error:"batch_expired"}];
const reconcile=(wanted,rows)=>{const settled=new Map(),duplicates=[];for(const row of rows){if(!wanted.has(row.custom_id))throw Error("unknown custom_id");if(settled.has(row.custom_id)){duplicates.push(row.custom_id);continue}settled.set(row.custom_id,row)}return {settled,duplicates,retry:[...wanted.keys()].filter(id=>!settled.get(id)?.ok)}};
const receipt=reconcile(requested,output);assert.deepEqual(receipt.duplicates,["item-4"]);assert.ok(receipt.retry.includes("item-9")&&receipt.retry.includes("item-19"));assert.equal(receipt.settled.get("item-0").ok,true);console.log("PASS: unordered batch reconciled");
Reconcile by custom_id, never output order
Download both output and error files when they become available, parse every line independently, and join it to the requested item by custom_id. The API reference defines custom_id as the developer-provided identifier used to match outputs to inputs and requires uniqueness within a batch. A parser should reject unknown IDs, quarantine duplicate result lines, and preserve request IDs and status codes beside decoded output.
It must not zip the result file with the input file, because asynchronous execution does not promise source order. custom_id reconciliation is the core replay boundary: once a result is accepted for an item attempt, ingesting the same line again changes nothing.
Preserve partial results from cancellation and expiry
Cancellation and expiry are workload states, not reasons to discard successful items. The batch reference notes that a cancelled job can expose partial results; an expired job can likewise leave per-request outcomes to reconcile. First ingest every available success and error, then compare the settled ID set with the original ledger.
Mark unresolved items separately from explicit failures so policy can distinguish never executed, provider timeout, invalid request, and application parse failure. This protects already-paid and already-reviewed work from a blanket replay and gives users a truthful progress count while the next attempt is prepared.
Retry failed items through a new immutable attempt
A retry file contains only items whose policy says retry, with fresh per-batch custom IDs or an attempt-qualified mapping that remains unique. Keep the stable business item ID behind that transport identifier. Permanent validation errors should stop for correction; request timeouts and expiry may be eligible after checking the current contract; safety or authorization failures should not be transformed into transient infrastructure errors.
The retry never overwrites the first attempt. Instead, a resolution rule selects the accepted terminal result and retains every rejected or superseded outcome for audit.
Make ingestion idempotent and restartable
The reconciler should be able to crash after any accepted line and resume from its ledger. Use a uniqueness constraint on workload item plus attempt plus provider result identity, and write the decoded result and ingestion receipt together. Store raw files immutably before parsing so parser upgrades can be replayed without another model call.
A checkpoint records file digest and last confirmed offset only as an optimization; correctness still comes from item identity. This architecture makes local restarts boring and lets an operator compare provider request_counts with application-settled counts without forcing them to match before downloads are processed.
| Observed outcome | Item disposition | Next action | Preserve |
|---|---|---|---|
| Success | Settled | Deliver once | Request ID |
| Timeout | Reviewable | Policy retry | Error line |
| Expired | Unresolved | New attempt | Original ID |
| Cancelled | Mixed | Ingest first | Partial files |
Close asynchronous AI jobs with a workload receipt
A workload is complete when every original item has an accepted result or a documented terminal disposition, not merely when the provider batch says completed. The closing report lists counts by outcome and attempt, unresolved IDs, model and prompt versions, token usage where the current API provides it, input and output digests, and downstream delivery status. Reconcile invoice and usage separately; cost metadata does not replace item correctness.
Retain the evidence according to the data policy cited at submission. This final OpenAI Batch API receipt turns an opaque asynchronous file exchange into a reviewable, restartable product operation.
Pin the API contract before replaying work
The current Batch resource reference defines the object surface, and the Batch guide explains JSONL submission and result retrieval; data handling belongs in the separate endpoint controls reference. The operational patterns continue in continuous batching, completion receipts, idempotency lifecycles, and replayable backfill evidence, which cover the queue mechanics that the API itself cannot own for an application.
Rehearse an ugly output file
Before connecting production data, manufacture a JSONL result with success rows out of order, one duplicate custom ID, one explicit item error, one malformed line, and three requests that never appear. Feed it to the same reconciler that will close the real job, then prove that a second ingestion leaves terminal records unchanged. An OpenAI Batch API pipeline is replay-safe only when this hostile fixture yields an exact accounted set, a quarantined set, and a retry set without relying on row position or a human scanning logs.
Include the input-file hash, submitted batch identifier, output-file hash, reconciliation version, and retry-manifest hash in one closure record. The OpenAI Batch API then remains only the asynchronous execution surface; the application can prove which workload items reached a terminal result, which evidence was quarantined, and why a later attempt contains exactly its selected subset.
| Decision | Evidence retained | Stop condition |
|---|---|---|
| Design the OpenAI Batch API ledger first | twenty unique business items, their payload hashes, one workload ID, and no provider batch ID before submission | an array index or upload line number is the only durable join key |
| Validate JSONL requests before upload | line count, unique custom IDs, endpoint allowlist, input digest, data class, and upload purpose | validation happens only after the provider rejects a large asynchronous job |
| Submit once and persist identity atomically | input file ID, batch ID, observed status transition, request counts, and the local write that bound them | a network retry creates a second batch for the same unchanged workload without reconciliation |
| Reconcile by custom_id, never output order | accepted ID, attempt, provider request ID, response or error digest, ingestion time, and duplicate disposition | a result's meaning depends on its physical line position |
| Preserve partial results from cancellation and expiry | settled, failed, and unresolved ID sets whose union equals the original twenty-item workload | the pipeline deletes output because the parent batch did not reach completed |
| Retry failed items through a new immutable attempt | failure category, eligibility decision, source attempt, new custom ID, unchanged payload digest or reviewed change | all non-success lines are blindly replayed with expanded permissions or a modified prompt |
| Make ingestion idempotent and restartable | raw file digest, parser version, accepted identity constraint, decoded output hash, and restart cursor | restarting the worker duplicates downstream writes or changes which result wins |
| Close asynchronous AI jobs with a workload receipt | coverage equation, terminal disposition for all twenty items, artifact hashes, usage fields, delivery state, and deletion schedule | the dashboard shows green while even one original item has no accepted outcome or explicit terminal reason |
OpenAI Batch API operations close at item coverage, not at a green parent status. Preserve every input, unordered output, retry decision, and workload receipt by custom ID.