Background jobs need completion receipts

Accepted is not completed, and a queue identifier is not proof that the requested product consequence occurred.

A report can be enqueued and never rendered. An import can finish with rejected rows. A media job can write the asset and crash before updating status. A worker can lose its lease while another repeats an external call. If the interface collapses these states into submitted and done, support inherits an investigation.

HTTP makes the boundary explicit: a 202 Accepted response says processing has not been completed, is intentionally noncommittal, and ought to describe current status while pointing to a status monitor. The product still has to define what completion means after that response.

The OpenTelemetry messaging conventions distinguish message creation, delivery, processing, and settlement. Transport activity and domain completion are different evidence, even when one trace connects them.

I would design the terminal receipt before choosing the worker library. Completion means every requested unit has an explainable outcome, committed effects reconcile with recorded totals, and the person has a safe next action for anything unresolved.

Define completion before enqueueing

The contract should state the domain condition that closes the job, including required effects and per-item accounting.

For an export, completion might require a readable object whose checksum matches the manifest, while email delivery remains a separate best-effort effect. For an import, completion may instead mean every accepted row is classified as applied, rejected, or unchanged. Those definitions lead to different terminal states and different retry buttons. I would write them before selecting a queue because the broker can report delivery and acknowledgement, but it cannot decide whether the requested business result exists.

A completion definition should be readable by the person waiting, the worker author, and the operator repairing a failure. I would compare an export whose asset exists but checksum fails, an import with valid and rejected rows, and a notification that is optional after the primary commit. Each example should land in a different terminal class for a stated reason. The review artifact is a table that names required effects, best-effort effects, balanced item classes, and the evidence needed to close them. Before any queue code ships, reviewers should challenge the product promise with these decisions:

• What must become true?
• Which effects are best effort?
• Can partial work be retained?
• What evidence permits closure?

Release proof is a terminal receipt whose required effects and item classes balance against the accepted denominator, including partial and cancelled examples reviewed by product.

Create durable operation identity

Acceptance should atomically create an operation scoped to tenant, initiator, fingerprint, status authority, and retention before delivery is trusted.

The operation ID belongs to the product domain, not to a particular queue delivery. One accepted export can produce several broker messages, worker attempts, storage uploads, and notifications while remaining one user-visible operation. The status resource should survive all of those substitutions. It also needs tenant-scoped authorization: an unguessable identifier is not an access-control rule, and a copied status URL must not expose another customer's filenames, errors, or row samples.

The operation record must be created in the same acceptance boundary that authorizes and fingerprints the request. I would test a client timeout followed by an identical retry, a reused idempotency key with different normalized options, and two queue deliveries for the same accepted operation. The first retry should join the original status resource; the conflicting request should fail visibly; delivery duplication should remain an execution detail. A copied URL also needs a tenant and role check rather than security by entropy. The identity contract is ready only when the team can answer these questions without referring to a broker message ID:

• Which identity survives worker loss?
• Can duplicates join one operation?
• Who may read or cancel it?
• When can it expire?

I would accept the identity design after concurrent retry tests yield one authorized status resource, one fingerprint decision, and no cross-tenant access through copied identifiers.

Operation example — an export accepted twice. A browser times out after acceptance and repeats the same request with the same idempotency key. The API should return the existing operation rather than enqueue a second export. If the normalized options or snapshot differ, the key collision becomes an explicit conflict. This example tests product identity, not only broker deduplication: two queue deliveries may still serve one operation, while two genuinely different export scopes must never be merged merely because their filenames match.

Snapshot consequential input

A worker needs a stable scope, normalized options, policy version, and authority so execution cannot apply a changed query or default.

A live query is dangerous input for delayed work. If a user requests every invoice matching a filter, the operation must say whether it exports the membership observed at acceptance or whatever matches when a worker eventually starts. I prefer an immutable manifest or snapshot reference for consequential batches. Authorization can still be rechecked before effects, but a later role change should produce a named denial or review state rather than silently shrinking or widening the original request.

Input freezing needs a product decision about membership, not just serialization. For a filtered export, I would show the review both an immutable manifest captured at acceptance and a live query resolved by the worker, then ask which one matches what the interface promised. Policy and authority may still need execution-time checks, but those checks should create named denial or review outcomes rather than silently alter the accepted denominator. The envelope should contain only consequential normalized values and resolvable version references. A delayed replay from the same envelope should produce the same eligible scope under the documented rules, so the design review must settle:

• Which values are immutable?
• Can scope use a snapshot?
• What is rechecked at execution?
• How is input minimized?

The snapshot contract earns approval when delayed and replayed workers select the promised membership and record any execution-time authority change as a visible outcome.

Make worker ownership explicit

Claims, lease epochs, heartbeats, and fenced transitions should stop stale workers after ownership moves and distinguish slow from abandoned attempts.

A visibility timeout alone does not fence a stale worker. Imagine worker A writing a large object after its lease expires while worker B has already resumed from the same checkpoint. A monotonically increasing lease epoch can travel with checkpoint and commit writes so the storage or database boundary rejects A's late result. Heartbeats then describe liveness; the epoch decides authority. That distinction matters during pauses, network partitions, long garbage-collection stops, and manual requeues.

Lease expiry proves that a worker may continue running after it loses authority. I would diagram worker A pausing during an upload, worker B acquiring the next epoch, and A attempting a late checkpoint or terminal write. Every mutable boundary that matters must reject the stale epoch; a heartbeat alone cannot provide that fence. Takeover should begin from the last committed checkpoint and reconcile any provider state left by the abandoned attempt. The claim state machine also needs a deliberate manual-requeue path that cannot resurrect an old writer. Reviewers should trace authority through the complete sequence and decide:

• What grants worker authority?
• When may another take over?
• Can a stale lease write?
• Which version decides ownership?

Ownership proof comes from a lease-theft test in which the late worker cannot checkpoint, publish an asset, close the receipt, or overwrite the new epoch.

Ownership example — a lease changes during upload. Worker A starts a multipart upload, loses its lease, and finishes a part after worker B takes ownership. The current epoch must prevent A from publishing the manifest or closing the operation. B can inspect the provider's upload state, reuse safe parts, or abort according to policy. The receipt records both attempts and the ownership transition. A user sees delayed progress, then one reconciled asset, rather than two competing workers both claiming they completed the same request.

Report progress as bounded evidence

Progress should name its denominator, durable checkpoint, counters, and freshness rather than extrapolate from worker memory.

A useful progress view says 9,000 of 12,480 classified at checkpoint 9,000, observed forty seconds ago. It does not convert that fraction into a promise that the remaining upload, manifest verification, and closure will take the same proportion of time. When scope is discovered during traversal, I would show a processed count and phase instead of a false percentage. After reconciliation, counters may legitimately move as ambiguous items acquire their final class.

Progress copy should describe durable knowledge, not estimate optimism. I would make the interface distinguish a fixed denominator from discovered work, processed items from verified effects, and a fresh checkpoint from a worker that has stopped reporting. A status such as 9,000 of 12,480 classified is useful; ninety-nine percent complete is misleading if upload and reconciliation remain unbounded. Counters may move during final classification, so the receipt should preserve their phase and observation time. Product and support should approve what each state means before dashboards depend on it, using these questions:

• Is scope fixed or discovered?
• Which checkpoint is durable?
• Can counters change?
• When should status say delayed?

Progress is trustworthy when a restart preserves the last durable checkpoint, stale activity becomes delayed, and final reconciliation explains every counter adjustment to the waiting person.

Join product effects to the receipt

Domain transition, item outcome, outbox, and receipt should share a transaction or an explicit external-effect reconciliation protocol.

The cleanest boundary is one transaction that writes the domain change, per-item outcome, outbox entry, and job checkpoint together. Object storage, payment providers, and email systems prevent that in many jobs. Then the design needs a stable external idempotency key, an effect ledger written before or after the call according to the provider's contract, and a queryable reconciliation state. Successful closure waits for that protocol; it does not assume that one HTTP 200 closed the gap.

When an effect cannot share the job transaction, the ledger has to describe the uncertainty explicitly. I would review a storage upload that succeeded before a timeout, an email outbox written with the domain change, and a payment provider that supports idempotency but not lookup. Each case requires a different order of intent record, external call, provider reference, and reconciliation. The job cannot close merely because one worker received a successful response; it closes when every required consequence is discoverable and balanced. The commit sequence should be tested at each crash point, and reviewers should resolve:

• Where is the commit boundary?
• Can an effect precede its ledger?
• How is delivery deduplicated?
• What query finds gaps?

I would close this review only after commit-window crashes produce one effect ledger, provider ambiguity is queried before repetition, and required consequences remain discoverable.

Effect example — the provider committed before timeout. A payment, email, or object call can succeed remotely while the worker receives no response. Immediate repetition is unsafe. The ledger should retain the provider idempotency key and unknown state, then query or reconcile using a stable external reference. If the provider offers no lookup or idempotency, the product must budget a manual review path. This is the honest boundary where availability, duplicate cost, and operator effort shape the protocol more than a generic retry library does.

Classify every terminal outcome

Success, partial completion, rejection, dependency failure, cancellation, supersession, and unknown effect need distinct totals and actions.

Partial is not a softer word for failed. An import with valid rows committed and invalid rows rejected can be a useful terminal product outcome if totals balance and the rejection file is available. Cancellation is different again: it stops unclaimed work but cannot pretend committed rows or delivered emails vanished. Unknown effect deserves its own state when a dependency timed out after a possible commit. Each class needs a narrower action than rerun everything.

Terminal states should narrow the next action. A partially successful import can retain applied rows and offer a rejection file, while an unknown external effect must block blind repetition until reconciliation. Cancellation should name the committed boundary rather than suggest that completed effects disappeared. Supersession should link the replacing operation and explain which work was abandoned. I would ask support to walk each state without reading worker logs and ask engineering to balance its totals against accepted scope. If either group reaches for a generic failed label, the outcome model is incomplete. The terminal-action review needs answers to:

• Can successful units remain?
• Which failures are retryable?
• What does cancellation leave?
• How is unknown effect escalated?

The outcome matrix passes when support can select the safe action for success, partial work, rejection, cancellation, supersession, dependency failure, and unknown effect.

Make retry and reconciliation converge

Retries should resume checkpoints, enforce item preconditions, fence stale attempts, and reconcile ambiguous calls before repeating them.

Retry budgets should follow failure classes. A transient database disconnect before commit can resume automatically. A storage timeout after upload needs a HEAD or provider lookup before another PUT. A deterministic validation rejection should not consume ten worker attempts. When automated reconciliation exhausts its evidence, the receipt should move one operation to an owned review queue with the last safe checkpoint and suspected effect, not bounce forever between workers.

Retry logic should branch on what is known about the last attempt. A disconnect before a database commit can repeat a guarded chunk; a timeout after an object upload requires a provider query; a validation rejection should leave the retry budget untouched. Checkpoint resumption must preserve prior success, and a stale lease must be unable to publish later work. Once automated evidence is exhausted, the operation moves to an owned review state with the suspected effect and safest next action. I would inject crashes around each boundary and require the playbook to explain:

• Can a chunk run twice?
• How is unknown result queried?
• Which errors consume budget?
• When does an operator intervene?

Convergence is demonstrated by replaying each failure class until the same item ledger emerges, without duplicate external effects or a job trapped in automatic retry.

Recovery example — cancellation arrives between chunks. Cancellation can fence future claims and stop work at the next durable boundary, but it cannot undo rows already imported or a notification already sent. The interface should say what stopped, what committed, and whether a compensating action exists. A cancellation receipt balances completed, skipped, and unresolved units under the original denominator. This avoids the familiar failure where the button changes the status label to cancelled while workers continue producing effects nobody can later explain.

Publish a privacy-bounded receipt

Status should expose useful detail to authorized people while minimizing raw input, secrets, personal data, and retained payloads.

Status pages often become accidental data warehouses because raw input and stack traces feel useful during development. I would keep stable identifiers, bounded reason codes, counters, version references, and redacted samples; deeper diagnostics remain in protected operational tooling with shorter retention. Deleting source data should not leave a copy embedded in a receipt. Support still needs enough evidence to say what happened, but that need rarely requires the complete uploaded file or exception object.

A receipt is a support surface, not permission to retain the submitted payload forever. I would classify fields by audience and purpose: customers see phase, bounded reasons, totals, and recovery; support sees redacted samples and source references; engineering retrieves deeper diagnostics from shorter-lived protected systems. Tenant checks, deletion propagation, and retention expiry deserve executable tests. Stable reason codes should remain useful even after raw evidence is removed. The access map is acceptable when support can resolve representative failures without opening an unrestricted debug archive, and reviewers can answer:

• Who sees item failures?
• Can identifiers be redacted?
• How long is detail useful?
• What survives source deletion?

Privacy proof combines tenant-isolation tests, redacted support fixtures, deletion propagation, and retention expiry while preserving enough structured evidence to explain a representative dispute.

Test every failure window

QA should delay and duplicate delivery, steal leases, crash around commits, interrupt effects, race cancellation, exhaust retries, and repeat reconciliation.

The highest-value tests crash the worker immediately before and after every durable boundary: operation claim, domain commit, outbox write, checkpoint, external call, and terminal closure. Another set races lease theft and cancellation against those points. Assertions inspect product state and receipt balances rather than log messages. A passing suite shows either one reconciled outcome or one deliberately unresolved item with an owner; it never accepts a permanently running status as harmless test debris.

Fault injection should assert product invariants after the system settles, not merely confirm that an exception was logged. The matrix should delay initial delivery, duplicate messages, steal leases, crash before and after commits, interrupt external calls, race cancellation, exhaust retries, and rerun reconciliation. Each scenario must end with balanced terminal evidence or one explicitly owned unresolved effect. Repeating reconciliation should not create another consequence or change a settled classification without new evidence. I would make the matrix part of release approval and ask the final review to prove:

• Can acceptance lack delivery?
• Can effect commit without closure?
• Can cancellation race a claim?
• Is reconciliation repeatable?

The suite is complete when every injected window produces either reconciled terminal truth or one bounded unresolved state with an owner, deadline, and non-destructive repair.

Support example — a partial import needs a narrow retry. Suppose valid rows committed, validation rejected another cohort, and one dependency lookup remained unknown. The receipt should offer the rejection artifact and a retry selector for unresolved items, preserving successful results and the original input version. Support can explain why rerunning the whole file would be risky. Engineering can trace the unknown lookup to an attempt and reconciliation query. Product language, repair scope, and worker evidence all point to the same operation state. I would also preview which units the repair will touch, require renewed authority when the original grant expired, and issue a child receipt that links back to the accepted scope. That keeps remediation auditable without pretending the first operation never reached a terminal partial result, and preserves one coherent customer history.

The completion receipt I would publish

The public status contract would expose operation identity, accepted scope, current phase, freshness, cancellation availability, and a terminal summary. The internal projection would add lease epochs, attempt history, checkpoints, effect references, reconciliation queries, and redacted failure samples. Those are two views over one state machine, not separate stories maintained by support and engineering. I would version reason codes and terminal fields because clients may automate downloads or repair flows from them. A receipt is closed only when its denominator balances: requested units equal applied, unchanged, rejected, cancelled, superseded, and unresolved units. Optional notifications have their own totals so a successful export is not relabeled failed because an email provider was unavailable.

# operation
job exp_731 / fp:91… / contract v3
Accepted 14:02Z; snapshot s_44; 12,480 rows requested; status restricted to the initiating role.
# execution
attempts 2 / lease 5 / checkpoint 9,000
A worker lost its lease; another resumed at the cursor; 9,000 processed and one storage write reconciled.
# completion
12,360 exported / 120 rejected / 0 unresolved
Asset 7af… verified, notification sent once, receipt closed 14:11Z, rejection CSV retained by policy.

Support should not have to read queue logs

A support view should begin with the user's operation and requested consequence. It can show that the request was accepted, which phase is stale, which effects committed, and which action is safe now. Queue name, delivery count, and worker host are secondary diagnostic facts. The illustrative receipt in Figure 4 uses invented identifiers, counts, and times to demonstrate the shape; none describes a production customer or measured rate. For a partial import, support should be able to download the rejection summary or retry only unresolved rows. For an unknown storage write, the action should be reconcile, not generate another export and hope the duplicate is harmless.

A rollout that protects in-flight work

I would ship the operation table and status read path first, then write new jobs into both the old observability path and the new receipt without changing execution. Next comes worker claims with fencing, durable checkpoints, and per-effect records behind a limited cohort. Only after reconciliation shows equivalent outcomes would the interface use the receipt as its source of truth. Rollback keeps the new records readable even if execution returns to the old worker, because deleting evidence during rollback would make in-flight work harder to recover. The migration plan also states how old jobs acquire a terminal class rather than leaving two permanent definitions of done.

Operational signals that predict stuck work

I would alert on age by phase, expired leases, repeated owner changes, checkpoint staleness, attempt-budget exhaustion, effect-ledger gaps, and operations whose item totals do not balance. Queue depth remains useful capacity evidence, but it is not the main customer signal. A small queue can still contain one high-value job stuck after an external commit. Dashboards should let an operator move from the aggregate to the operation receipt and then to the exact reconciliation query. The service objective I care about is time to reconciled terminal truth by operation class, paired with the percentage that require human review.

How I would package this case study

The case study would open with one ambiguous failure window: an asset existed, the worker response was lost, and a retry threatened to create another charge or object. I would show the original state machine, the revised claim-and-effect sequence, one rejected design that relied on visibility timeout, and fault-injection evidence. The centerpiece would be a redacted receipt that balances scope and effects after a crash. I would avoid presenting queue technology as the achievement. The stronger story is how the system changed what the product could honestly tell a waiting person and how an operator could repair the exceptional path without guessing.

The atomicity boundary I would debate

The useful interview discussion is where atomicity ends. If the domain database and external provider cannot share a transaction, I would explain why I chose an outbox, idempotency key, provider lookup, or manual review for each effect. I would also discuss the cost of durable per-item evidence: storage, write amplification, retention, and privacy. The answer should not claim exactly-once execution. It should show how at-least-once delivery converges on one explainable product result, where uncertainty remains, and what evidence prevents a retry from making that uncertainty worse.

What completion receipts demonstrate

Designing this well demonstrates more than background-job familiarity. It shows I can move between interface language, HTTP acceptance, queue delivery, concurrency control, transaction boundaries, external side effects, privacy, and support operations without collapsing their meanings. The artifact is credible when another engineer can inject a crash, an operator can locate the affected consequence, and the status view remains honest throughout recovery. That combination is the hiring signal: I can ship an asynchronous feature and also design the evidence that lets customers and teammates trust what it eventually did.