Product experiments need assignment receipts

An experiment result is hard to trust when the team cannot say who was assigned, who actually saw the treatment, and which rule version made that true.

Eligibility can change after signup. A person can cross devices or merge accounts. A cookie can disappear. A flag can be edited in place. An exposure event can fire during prefetch. Outcome events can arrive without assignment context. The dashboard may still render a lift, but the causal population has become an undocumented query.

OpenFeature's evaluation-context specification gives the targeting key special meaning as an identifier for the subject of evaluation and defines how contexts merge. Its evaluation details include reason and variant alongside the resolved value. Those primitives help make assignment inputs and outputs explicit.

HTTP cookies can help preserve a browser experience, but RFC 6265 notes that user agents may evict cookies before their stated expiration. A cookie therefore should not become the only authoritative record of a high-consequence or long-running assignment.

I would record assignment as a versioned product decision, record exposure as a separate observed event, and join outcomes to the experiment revision and unit. The receipt makes analysis replayable when flags, identities, and targeting rules move on.

Start with the decision

Name the product decision, hypothesis, outcome, guardrails, meaningful threshold, owner, duration, and action for each result.

An experiment should begin with a pending product choice, not the availability of a flag. I would name the behavior the treatment is expected to change, the primary outcome and observation window, guardrails, the smallest worthwhile effect, and what the team will do after positive, neutral, or harmful evidence. That brief prevents a statistically tidy dashboard from becoming the only definition of success and makes an inconclusive result a planned decision state rather than a reason to keep testing forever.

The decision brief should make every plausible result actionable before anyone sees data. I would ask the owner to specify what positive, neutral, harmful, or inconclusive evidence changes in the product, using the primary outcome, smallest worthwhile effect, guardrails, duration, and implementation cost. A neutral result might preserve the current design, justify a qualitative launch, or trigger a new hypothesis, but it should not automatically extend the test. This keeps dashboard movement subordinate to a real choice. The experiment is ready for design and instrumentation only after reviewers can answer:

• Which decision will change?
• What behavior should treatment cause?
• Which harm is guarded?
• What follows neutral evidence?

Decision readiness is a signed brief that maps positive, neutral, harmful, and inconclusive evidence to a product action, owner, guardrail response, and explicit stopping or follow-up rule.

Choose one randomization unit

Person, account, tenant, session, device, or request should match the interference boundary, with canonical identity and missing-identity behavior.

The randomization unit follows the interference boundary. If one teammate's treatment changes a shared workspace, randomizing people can contaminate both groups; an account or tenant may be the honest unit. A checkout-copy experiment might use an account for known users and a documented anonymous-device cohort before login. I would never switch among cookie, user, and account identifiers by surface without recording that policy because the same person could otherwise receive both treatments and influence several outcomes.

The randomization unit should match who can influence whose outcome. I would map shared workspaces, household or account effects, cross-device behavior, anonymous traffic, and pre-login use before choosing person, account, tenant, device, session, or request. If treatment changes a collaborative workspace, person-level allocation can contaminate both groups. Every surface must resolve the same canonical unit or enter a documented anonymous cohort; it cannot select whichever identifier is easiest locally. The identity contract should include missing and conflicting cases, and the design review needs to decide:

• Who influences whose outcome?
• Which identity is stable?
• Can one account hold several units?
• What happens before login?

Unit selection is proven when cross-surface fixtures assign the same logical participant, collaboration or account interference is represented at the chosen boundary, and missing identity follows a declared path.

Decision example — neutral evidence has a destination. Before launch, the owner should say whether a neutral estimate means keep the current product, ship for qualitative value, redesign the treatment, or collect more information under a new plan. That decision may depend on the confidence interval, smallest worthwhile effect, guardrails, and implementation cost. Recording it prevents endless duration extensions after a dashboard disappoints. It also helps recruiters and engineering reviewers see that the experiment exists to improve a product choice, not to manufacture a statistically significant artifact.

Version eligibility and exclusions

Inclusion, geography, consent, account, staff, prior exposure, experiment conflicts, and capability should be immutable per revision.

Eligibility is part of the experiment revision. Geography, consent, account state, prior exposure, staff exclusion, conflicting tests, and product capability should each yield a stable included or excluded reason. The receipt records when the rule was evaluated and whether late entry is allowed. A material targeting edit creates another revision or cohort boundary. Editing it in place makes early and late participants look comparable while they actually entered under different product and compliance contracts.

Eligibility belongs to an immutable experiment revision. I would list inclusion, geography, consent, account state, staff, prior exposure, conflicting tests, capability, and late-entry rules, each with a stable included or excluded reason. The receipt should record when the rule ran and which revision it used. A material targeting edit creates another cohort boundary instead of silently changing who enters the same experiment. Reviewers should compare early and late cases around every effective-time edge and verify that excluded units never acquire an assignment through another surface. The eligibility review should resolve:

• When is eligibility evaluated?
• Can units enter after launch?
• Which conflicts exclude?
• How are rule changes versioned?

Eligibility passes when every included or excluded unit cites one immutable rule revision, effective-time edges behave as published, and material targeting changes open a new cohort.

Make assignment deterministic or durable

Allocation, algorithm, salt, weights, revision, and persistence should prevent retries, deploys, or cache loss from moving a unit.

Deterministic bucketing needs a canonical byte representation of unit, experiment, revision, algorithm version, and a protected salt reference. Cross-language test vectors prove that web, mobile, and backend implementations choose the same bucket. Allocation ramps should add ranges without reshuffling existing units, or persist assignment authoritatively when that guarantee cannot hold. Randomness during each evaluation is not experimentation; it is a changing experience whose observed outcome cannot be attributed to one stable treatment.

Assignment requires either reproducible bucketing or a durable authoritative row. For deterministic allocation, I would specify canonical unit bytes, experiment and revision, algorithm version, salt reference, weights, and range map, then publish cross-language vectors. Ramps should add ranges without moving existing units unless the design explicitly allows crossover. For stored assignment, retries and concurrent creation must converge on one variant. Cache loss, deploy, and device change should not reroll the unit. The implementation is credible when repeat evaluation reproduces the same receipt under planned lifecycle changes, so reviewers should determine:

• What enters the bucket?
• Can weight changes reshuffle units?
• Where is assignment authoritative?
• How is missing state handled?

Assignment evidence includes stable cross-language vectors, concurrency-safe creation, preserved variants across retries and deploys, and ramp behavior that never reshuffles an existing unit without a declared intervention.

Assignment example — a ramp preserves existing buckets. A launch can allocate ten percent of units, then expand to twenty-five percent by adding deterministic ranges while leaving previously assigned control and treatment units unchanged. The receipt records the revision, algorithm, range map, and first assignment. If weights must rebalance in a way that moves existing units, that is a new intervention or explicit crossover design, not a configuration tweak. Cross-language vectors and sampled production comparisons catch subtle encoding differences that would otherwise create surface-specific treatment switching.

Treat client storage as a mirror

Client storage can aid anonymous stickiness, but authority must handle eviction, consent, device change, tampering, expiry, and login.

A browser cookie can mirror anonymous assignment and improve continuity, but deletion, eviction, consent withdrawal, expiry, device change, and tampering are normal states. The server or experiment store should be authoritative for logged-in or high-consequence units. Login invokes a written transition policy: preserve anonymous treatment, prefer existing account assignment, exclude the conflict, or start a new analysis unit. Silent last-write wins creates treatment switching exactly where the product has the richest identity signal.

Client storage can improve continuity without becoming the sole source of truth. I would walk cookie eviction, consent withdrawal, expiry, tampering, a second device, login, logout, and an existing account assignment. The transition policy may preserve the anonymous experience, prefer account authority, or exclude a conflict, but it must not use silent last-write wins. High-consequence and logged-in assignments should resolve through an authoritative service or store. The client value should carry only the minimum signed or validated reference needed for presentation. Before launch, product, privacy, and engineering should agree on:

• What if storage disappears?
• Can clients alter variant?
• How does anonymous become known?
• What may remain device-local?

Persistence is credible when eviction, tampering, consent changes, login, logout, second devices, and account conflicts follow the authority map without silently switching a participant's treatment.

Separate assignment from exposure

Assignment is a variant decision; exposure is a qualifying opportunity to experience it, with surface, trigger, build, and deduplication.

Exposure should fire only when the treatment has a real opportunity to influence behavior. Fetching configuration, rendering offscreen content, preloading a route, or evaluating a flag in a worker may not qualify. The contract names the surface, trigger, implementation build, first-exposure identity, and meaningful repeat rules. Assignment remains useful for intent-to-treat analysis; exposure is a separate implementation and reach signal. Conflating them can condition the analysis on behavior that treatment itself changes.

Exposure should represent an opportunity for treatment to affect behavior. I would define the qualifying state per surface: necessary data resolved, treatment visible or interactive, correct implementation build, and a stable deduplication identity. Configuration fetch, prefetch, offscreen render, or a flag evaluation inside unrelated work should not count automatically. Assignment remains the basis for intent-to-treat reasoning; exposure diagnoses reach and implementation separately. Tests should trigger rerenders, navigation, retries, and hidden content to verify first and repeat rules. The instrumentation review needs answers to:

• When can treatment affect behavior?
• Can hidden render count?
• Is repeat exposure meaningful?
• Who owns the event?

Exposure instrumentation passes when only qualifying opportunities emit the governed event, rerenders obey deduplication rules, implementation build is captured, and prefetch or hidden content stays diagnostic.

Exposure example — prefetch is not experience. A mobile client may request configuration while warming the next screen, and a web application may render treatment markup outside the viewport. Neither necessarily gave the variant an opportunity to change behavior. The surface contract can require visible content, resolved dependencies, and an interactive state before the first exposure event. It should also state which repeat exposures matter. This discipline turns missing exposure into an implementation signal and keeps assignment-based analysis from being quietly replaced by a convenience event emitted by the flag wrapper.

Carry context into outcomes

Outcomes should resolve revision, unit, assignment, event time, and causal window while avoiding duplicates and unrelated personal detail.

Outcome events need event time, canonical unit, experiment revision, assignment reference, deduplication identity, and the causal window declared in the brief. Joining a purchase to current flag state is wrong after configuration changes or experiment closure. I would specify how late arrival, refunds, repeated conversion, and events spanning devices behave. The receipt does not require copying unrelated profile attributes; a stable unit and revision link should be enough to reproduce the population under governed warehouse logic.

Outcome attribution should join to the immutable assignment and experiment revision rather than current flag state. I would specify the canonical unit, primary event, event-time window, deduplication key, repeat conversion rule, refund or reversal behavior, and treatment of late arrival. Events crossing devices should follow the unit identity contract without copying unrelated profile fields into the receipt. Warehouse replay must return the same assigned and converted counts after configuration changes. Before analysts publish a result, the outcome contract should make these choices explicit:

• Which event is primary?
• What window follows assignment?
• Can one event count twice?
• How are late events handled?

Outcome replay should reproduce assignment-linked counts after flag deletion while honoring event-time windows, deduplication, reversals, late arrival, and the chosen unit across devices or services.

Handle identity transitions explicitly

Anonymous linking, merges, tenant switches, shared devices, deletion, and reassignment need precedence that preserves causal meaning.

Identity merges can create two valid but conflicting prior assignments. The system should not pick whichever profile wrote last. It can preserve treatment for the active causal window, choose a precedence based on the declared unit, exclude the merged unit from a primary analysis, or expose a sensitivity cohort. Tenant switches and shared devices need similarly explicit handling. Deletion may remove direct identifiers while retaining appropriately aggregated experiment evidence according to consent and retention policy.

Identity transitions can create genuine causal ambiguity. Two accounts assigned opposite variants may later merge; an anonymous device may log into an account with prior treatment; a person may switch tenants whose workspace is the actual unit. I would preserve original assignment history, apply a declared rule to future experience, and label exclusion or sensitivity cohorts rather than rewrite the past. Deletion should detach direct identity according to policy while retaining only permitted aggregate evidence. The transition state machine should be tested with conflicting receipts, and reviewers need to decide:

• Which assignment survives login?
• Can merged accounts conflict?
• Does tenant outrank person?
• How does deletion affect evidence?

Transition proof is a fixture set where login, merge, tenant change, shared device, and deletion preserve original evidence and produce the published future-treatment or analysis caveat.

Identity example — account merges produce a caveat, not magic. Two accounts may enter opposite variants and later merge during the outcome window. Preserving both is impossible for future shared surfaces, yet choosing one retroactively corrupts history. I would retain original assignments, apply the declared precedence for subsequent experience, and label the merged unit for primary or sensitivity analysis according to the brief. The receipt links the transition without copying unnecessary profile detail. Analysts can then explain the denominator change instead of blaming an unexplained join anomaly.

Freeze revisions and stop states

Material treatment, allocation, eligibility, instrumentation, or analysis changes need a revision, while pause, stop, ramp, and close govern existing units.

A treatment-code change, primary instrumentation change, material allocation edit, eligibility change, or analysis-contract change should produce a new immutable revision. Pause stops new exposure or entry according to the protocol; stop records the decision and treatment behavior for already assigned units; close ends the outcome window. A ramp can preserve buckets while widening allocation if the algorithm planned for it. Lifecycle semantics prevent a dashboard range from combining several undocumented interventions into one apparent experiment.

Treatment code, allocation, eligibility, instrumentation, and analysis assumptions each need a revision rule. I would define which edits create a new immutable intervention and what pause, ramp, stop, and close do to existing assignments and future exposures. A stop decision should record why it happened and whether already assigned units keep treatment through their outcome window. Ramps may preserve buckets only when the allocation scheme was designed for that. Dashboards must not combine revisions by default. The lifecycle policy should let operators handle an incident without improvising, so approval depends on:

• Which edit needs a revision?
• Do units keep treatment after stop?
• Can ramps preserve buckets?
• When does outcome close?

Lifecycle operations are safe when pause, ramp, stop, close, rollback, and material edits produce unambiguous revision boundaries, treatment behavior, exposure rules, and outcome windows for existing units.

QA the causal plumbing

Test allocation, deterministic vectors, stickiness, context, exclusion, identity, flag failure, storage loss, exposure, deduplication, late events, and replay.

QA should reproduce allocation weights over synthetic populations, verify cross-language vectors, and test restarts, cache loss, cookie eviction, login, merge, deletion, targeting failure, and policy revision. Instrumentation fixtures prove exposure follows the qualifying render and outcomes link once inside the correct window. Replay rebuilds assigned, exposed, excluded, censored, and outcome populations from receipts. Visual review of two variants is still necessary product QA, but it says nothing about the causal plumbing that makes a measured difference credible.

Causal-plumbing QA should begin with synthetic populations and end with replay. I would test allocation weights, cross-language vectors, retries, cache loss, cookie eviction, login, merges, deletion, eligibility edges, flag failure, qualifying exposure, deduplication, late outcomes, pause, and rollback. Each case asserts unit, revision, assignment, reason, exposure state, and analysis class, not only the rendered variant. A warehouse fixture should rebuild assigned, exposed, excluded, censored, and outcome populations from receipts after configuration is gone. Visual product QA remains separate. Before trusting the experiment, the final review should prove:

• Do buckets match weights?
• Can units switch unexpectedly?
• Does treatment precede exposure?
• Can evidence reproduce population?

The causal pipeline earns trust when independent replay reproduces every analysis population and injected identity, storage, instrumentation, and event faults remain visible rather than being filtered into a cleaner chart. The final receipt should also reconcile assignment, exposure, censoring, and outcome totals by revision, preserve declared exclusions, and surface any warehouse-model drift before the product owner reads an effect estimate. That evidence keeps analytical caveats attached to the decision instead of burying them in a separate validation notebook.

Replay example — configuration has already moved on. Months later, the flag may be deleted, targeting rules changed, and cookies expired. A reproducible analysis should still recover immutable eligibility revision, stable assignment, qualifying exposure, lifecycle state, and linked outcomes from governed records. Synthetic replay fixtures verify the query after warehouse model changes. A cohort checksum and class totals can reveal drift even after direct identifiers follow their deletion schedule. If any component cannot be reconstructed, the limitation is part of the result and receives an explicit caveat in the decision record before the product decision is revisited. The assignment receipt protects the causal story from operational cleanup without pretending that more retained data can rescue a poorly defined experiment.

The assignment receipt I would publish

The receipt identifies experiment and immutable revision, randomization unit, eligibility result, allocation and algorithm references, assigned variant, decision reason, assignment time, and persistence authority. It links later identity transitions and qualifying exposure without rewriting the original decision. Analysts can derive intent-to-treat populations from assignment, implementation teams can diagnose reach through exposure, and support can answer why an account saw a stable experience. A downloadable specification would include field definitions, lifecycle states, synthetic examples, cross-language bucket vectors, and queries that reconstruct each analysis population without consulting mutable flag configuration.

# entry
experiment checkout-copy / revision 7 / unit account:42
Eligibility rules e12 passed; staff and region exclusions false; targeting context hash 7a…; assigned 2026-07-24T14:00Z.
# assignment
variant treatment_b / allocation 25% / reason deterministic
Algorithm version a3 and salt reference s9; authoritative assignment store row ar_731; browser cookie mirrors revision and variant.
# observation
exposure ex_91 / surface checkout.summary / outcome 24h
First qualifying render 14:03Z, implementation build 211, purchase event linked once, identity merge policy not invoked.

Exposure instrumentation needs a product definition

The illustrative experiment IDs, revision numbers, account identifiers, percentages, dates, times, build numbers, and event IDs in Figure 4 are invented examples, not production results or measured rates. For a checkout summary, exposure might require the treatment text to enter the viewport with the price available; a background configuration fetch is insufficient. I would review that trigger with design, engineering, data, and the decision owner, then test deduplication across rerenders and navigation. A separate diagnostic can count evaluation calls. Keeping evaluation, assignment, render, and exposure distinct prevents convenient telemetry from defining the causal population after launch.

Identity policy belongs in the launch review

Before launch I would walk through anonymous browsing, login, logout, a second device, account merge, tenant switch, shared hardware, consent withdrawal, and deletion. Each transition should preserve one assignment, deliberately choose precedence, or label a caveat or exclusion. The policy depends on the randomization unit and product consequence; there is no universally correct merge rule. Receipts make those cases visible so analysts can run sensitivity checks instead of discovering unexplained treatment switching in aggregate. They also let product teams judge whether identity complexity makes a proposed experiment too ambiguous to run.

Publish analysis populations before results

The decision brief should name the primary population, typically assigned units for an intent-to-treat question, plus diagnostic exposed, eligible, excluded, censored, and protocol-deviation populations. It defines windows, deduplication, late events, missing outcomes, guardrails, and revision boundaries before anyone sees a lift. Exposure-conditioned analysis can be informative about implementation, but it should not silently replace assignment when treatment affects whether exposure occurs. I would version the warehouse query or model and reconcile its counts to receipt classes, leaving deviations visible instead of filtering them away until the chart looks clean.

How I would present this experiment system

The case study would start with a plausible but untrustworthy lift caused by treatment switching or premature exposure events. I would show the identity contract, deterministic vectors, assignment store, surface-specific exposure trigger, one merge conflict, and a replay that reconstructs the published populations after configuration changes. The downloadable artifact would use synthetic receipts and analysis fixtures. I would not claim that infrastructure guarantees causal truth; the stronger argument is that it preserves the assignment and observation evidence needed for statisticians and product leaders to assess that truth.

The identity collision I would debate

I would ask whether an anonymous assignment should survive login when the account already has the opposite treatment. Preserving the browser experience reduces visible switching but violates an account-level unit; preferring the account protects that unit but changes what the person sees; excluding the conflict protects interpretation but reduces power and may bias the remaining population. The answer depends on the declared decision and interference boundary. I would also discuss deterministic assignment versus a durable store, including cross-language risk, privacy, latency, audit needs, and how allocation ramps affect existing buckets.

What assignment receipts demonstrate

This work demonstrates an ability to connect product decisions, identity architecture, deterministic behavior, client constraints, event design, privacy, lifecycle operations, and causal analysis. The evidence is not a dashboard screenshot. It is a versioned decision brief, a stable unit contract, reviewed assignment vectors, durable receipts, meaningful exposure definitions, explicit transition policy, governed outcome joins, and replayable populations. Recruiters can see product judgment in the action rules; engineers can inspect failure and persistence boundaries; data partners can reproduce the denominator. That is a stronger hiring signal than simply wiring variants into a flag provider.