ReadableStream BYOB for Binary Protocols

ReadableStream BYOB makes buffer ownership and valid fill boundaries explicit for binary consumers. This tutorial builds a framed decoder that carries partial reads, observes returned views, cancels cleanly, and rejects truncation or oversize input.

ReadableStream BYOB begins with ownership

ReadableStream BYOB lets a consumer offer a typed-array view to a readable byte stream. Its durable advantage is explicit buffer ownership and fill boundaries, not a blanket “zero-copy” promise. A framed decoder must know who may write each view, which bytes are valid, and what state survives when a read returns fewer bytes than requested.

This tutorial uses a one-byte length prefix followed by that many payload bytes. The decoder limits frame size, accumulates an incomplete header or payload, emits only complete frames, and rejects a truncated end. Each call reads through a real BYOB reader in the executing Node Web Streams implementation. The deterministic fixture also records requested and returned view lengths, so ownership claims remain tied to observable byte ranges rather than to an assumed allocation story.

The WHATWG Streams Standard defines readable byte streams, BYOB readers and requests, response algorithms, cancellation, close, and view behavior. Those algorithms are the source of truth; this fixture is a focused decoder test. Begin with explicit producer and consumer pressure so ReadableStream BYOB joins an existing flow-control design instead of becoming an isolated allocation trick.

Construct a readable byte stream and BYOB reader

Create the source with type set to bytes, then acquire a reader in BYOB mode. In pull(), inspect controller.byobRequest and fill only the number of bytes allowed by the frozen adversarial schedule. Call respond() with the count actually written. Close only after all source bytes have been delivered and the outstanding request has been answered according to the runtime contract.

The consumer supplies a fresh view for each read. After await reader.read(view), use the returned value rather than assuming the original view remains attached or retains its length. The Node Web Streams API documents BYOB support and warns about pooled Buffer ownership; this fixture uses standalone Uint8Array allocations to avoid pretending a pooled buffer is safe.

ReadableStream BYOB does not remove copies everywhere. The stream implementation may transfer or replace views, and the decoder may need a carry buffer for fragmented frames. What it offers is a precise place to observe how many bytes became valid. The receipt records requested capacity, returned capacity, byte length, and detachment state on every read.

Carry partial headers and payloads

Network-shaped data rarely aligns with protocol frames. A single ReadableStream BYOB fill may contain half a header, one complete frame, or the tail of one frame plus the beginning of another. The parser therefore owns a carry array independent of the view used for the current read.

Append only the returned bytes, then loop while the carry contains enough data. In this deterministic fixture, read the length prefix, reject it if it exceeds the configured maximum, and wait when the payload is incomplete. When a complete frame exists, copy or slice its payload into the output record and retain the remaining bytes for the next loop. The artifact freezes schedules such as one byte at a time and 2–1–4–3 to exercise every boundary.

This differs from newline-delimited text parsing because byte framing cannot rely on character decoding or delimiter search. It also differs from resuming transport bytes, where range offsets must be validated before the frame parser resumes. The second figure makes the carry length visible after each fill so partial state cannot masquerade as a complete message.

Treat view replacement and detachment honestly

BYOB algorithms can transfer the backing buffer and return a different view. Code that reads from the originally supplied Uint8Array after await may encounter a detached buffer or stale assumptions. Always consume result.value, check result.done, and record what happened in the executing runtime.

The artifact includes a detachment probe: it captures the supplied view's buffer and byte length before the read, then observes them afterward without requiring one universal result. The correctness oracle is that decoded payloads match and every byte consumed came from the returned view. The receipt labels whether the original buffer was detached, replaced, or retained.

That distinction keeps ReadableStream BYOB portable. A conformance statement belongs to the Web Platform Tests suite, not this article. The WPT streams tests provide primary interoperable cases for readable byte streams and transfer behavior. Our fixture reports Node's behavior for its pinned runtime and refuses to generalize it to every browser or version.

Reject truncation and oversize frames

End-of-stream is clean only when the carry buffer is empty. If one length byte remains or the declared payload is incomplete, the decoder throws a named truncation error that includes expected and received counts. ReadableStream BYOB must not quietly emit a partial record because the transport closed.

Reject oversize lengths before allocating a payload buffer. The fixture's maximum is 32 bytes; a prefix of 255 fails immediately. A production protocol may use a multi-byte length, but the rule is the same: validate arithmetic, enforce a finite ceiling, and avoid allocation based on untrusted length alone. Zero-length frames are either explicitly legal or explicitly rejected; this teaching protocol accepts them and tests that case.

Zero progress also needs a bound. An underlying source that repeatedly responds with no useful bytes can create a busy loop or runtime error. The fixture treats unexpected zero-progress observations as failure and caps read count. These hostile states appear as separate rows in the third figure, because “decoder failed” is not enough information to fix ownership, framing, or source behavior.

Cancel without leaking parser state

Cancellation is a protocol outcome, not merely a rejected promise. When the consumer cancels, release the reader lock in a finally block, invoke the source cancellation hook, clear the carry buffer, and prevent any partial frame from entering the result. The artifact creates a separate stream for cancellation so that the success receipt cannot inherit hidden state.

Record the cancellation reason and whether the source observed it. Do not retry automatically unless the transport and application have a resume contract. If resumption is allowed, start from a verified byte offset and a fresh parser state; otherwise a duplicated prefix or missing tail can corrupt the next frame. Give the framed binary decoder an explicit terminal state, and reject any later pull or emit attempt. That makes cancellation testable at the parser boundary as well as at the stream boundary.

ReadableStream BYOB composes with transforms, including standards-based browser compression streams, only when each stage documents close, flush, error, and cancel propagation. A decoder that catches every error and returns the frames accumulated so far makes downstream success ambiguous. Preserve the named failure and let the caller decide whether partial results are useful.

Replay adversarial chunk schedules

The runnable script encodes three generated frames and replays them through several frozen fill schedules. One schedule supplies a byte at a time; another splits every header from its payload; a third crosses multiple frame boundaries. Every ReadableStream BYOB run must decode the exact same payload hex values and report its read trace.

Hostile runs cover truncation, oversize length, cancellation, and the executing runtime's view behavior. Repeating the canonical schedule must produce an identical normalized receipt. The script hashes the semantic result after excluding no fields, because it contains no clock or random runtime data. Tests parse the JSON, recompute the decoded frames, and assert each named failure.

This is correctness evidence, not a benchmark. No throughput, allocation reduction, or browser-wide compatibility number appears. Real performance work would require warmed and cold runs, payload distributions, memory profiles, competing decoders, and confidence intervals. Low-copy JavaScript streams deserve that separate experiment because buffer reuse, transfer, parser carry, and application materialization may each move the allocation boundary. ReadableStream BYOB should first earn trust by surviving hostile boundaries; optimization claims can be a separate experiment with a separate intent.

Runnable artifact — The fixture proves decoder correctness on the tested Node Web Streams runtime. It is not a browser-conformance suite, allocation profile, throughput benchmark, or universal zero-copy claim.

import assert from "node:assert/strict";
import { createHash } from "node:crypto";
const canonical=value=>JSON.stringify(value,(_,item)=>item&&typeof item==="object"&&!Array.isArray(item)?Object.fromEntries(Object.entries(item).sort(([a],[b])=>a.localeCompare(b))):item);
const sha=value=>createHash("sha256").update(ArrayBuffer.isView(value)?Buffer.from(value.buffer,value.byteOffset,value.byteLength):typeof value==="string"?value:canonical(value)).digest("hex");
const payloads=[new Uint8Array([0x41,0x42,0x43]),new Uint8Array([]),new Uint8Array([0xde,0xad,0xbe,0xef,0x01])];
function encode(frames){const size=frames.reduce((total,frame)=>total+1+frame.length,0),out=new Uint8Array(size);let offset=0;for(const frame of frames){out[offset++]=frame.length;out.set(frame,offset);offset+=frame.length}return out}
const sourceBytes=encode(payloads);
function byteStream(bytes,schedule,cancelLog=[]){let offset=0,turn=0;return new ReadableStream({type:"bytes",pull(controller){if(offset>=bytes.length){controller.close();return}const request=controller.byobRequest;if(!request)throw new Error("missing-byob-request");const allowance=schedule[turn%schedule.length],turnIndex=turn++;if(!Number.isInteger(allowance)||allowance<=0){controller.error(new Error("zero-progress-schedule:turn="+turnIndex+":allowance="+allowance));return}const view=request.view,count=Math.min(view.byteLength,allowance,bytes.length-offset);view.set(bytes.subarray(offset,offset+count));offset+=count;request.respond(count);if(offset>=bytes.length)controller.close()},cancel(reason){cancelLog.push(String(reason))}})}
async function decode(bytes,schedule,{maximum=32,bufferSize=5,cancelAfterReads=0,cancelReason="fixture-cancel"}={}){const cancelLog=[],reader=byteStream(bytes,schedule,cancelLog).getReader({mode:"byob"});let carry=new Uint8Array(),frames=[],trace=[],terminal={kind:"active"};try{for(let reads=0;reads<100;reads++){const supplied=new Uint8Array(bufferSize),originalBuffer=supplied.buffer,result=await reader.read(supplied),value=result.value||new Uint8Array();trace.push({read:reads,requested:bufferSize,returned:value.byteLength,done:result.done,originalDetached:originalBuffer.byteLength===0,replaced:value.buffer!==originalBuffer,carryBefore:carry.length});if(value.byteLength){const joined=new Uint8Array(carry.length+value.length);joined.set(carry);joined.set(value,carry.length);carry=joined}while(carry.length){const length=carry[0];if(length>maximum)throw new Error("oversize-frame:length="+length+":maximum="+maximum);if(carry.length<1+length)break;frames.push(carry.slice(1,1+length));carry=carry.slice(1+length)}trace[trace.length-1].carryAfter=carry.length;trace[trace.length-1].framesAfter=frames.length;if(cancelAfterReads===reads+1){const carryBeforeClear=carry.length,framesBeforeCancel=frames.length;await reader.cancel(cancelReason);carry=new Uint8Array();const post=await reader.read(new Uint8Array(1));terminal={kind:"cancelled",reason:cancelReason,sourceObserved:cancelLog[0]||null,carryBeforeClear,carryAfterClear:carry.length,framesBeforeCancel,framesAfterCancel:frames.length,postCancelDone:post.done,postCancelReturned:(post.value||new Uint8Array()).byteLength};return{frames,trace,terminal}}if(result.done){if(carry.length){const expected=carry[0],received=Math.max(0,carry.length-1);throw new Error("truncated-frame:expected="+expected+":received="+received)}terminal={kind:"closed",carryBytes:0,frames:frames.length};return{frames,trace,terminal}}}throw new Error("read-limit-exceeded")}catch(error){terminal={kind:"errored",message:error.message,carryBytes:carry.length,frames:frames.length};error.decoderTerminal=terminal;throw error}finally{reader.releaseLock()}}
const toHex=frame=>Buffer.from(frame).toString("hex"),normalize=result=>({frames:result.frames.map(toHex),trace:result.trace,terminal:result.terminal});
const schedules=[[1],[2,1,4,3],[5,2,1]],runs=[];for(const schedule of schedules)runs.push({schedule,result:normalize(await decode(sourceBytes,schedule))});
const expected=payloads.map(toHex);for(const run of runs)assert.deepEqual(run.result.frames,expected);
const replayA=normalize(await decode(sourceBytes,[2,1,4,3])),replayB=normalize(await decode(sourceBytes,[2,1,4,3]));assert.deepEqual(replayA,replayB);
let truncation;try{await decode(sourceBytes.slice(0,-1),[2])}catch(error){const match=/expected=(\d+):received=(\d+)/.exec(error.message);truncation={message:error.message,expected:Number(match?.[1]),received:Number(match?.[2]),terminal:error.decoderTerminal}}
let oversize;try{await decode(new Uint8Array([255]),[1])}catch(error){oversize={message:error.message,terminal:error.decoderTerminal}}
let zeroProgress;try{await decode(sourceBytes,[0,2])}catch(error){zeroProgress={message:error.message,terminal:error.decoderTerminal}}
const cancellation=normalize(await decode(sourceBytes,[2],{bufferSize:2,cancelAfterReads:1}));
assert.deepEqual(cancellation.terminal,{kind:"cancelled",reason:"fixture-cancel",sourceObserved:"fixture-cancel",carryBeforeClear:2,carryAfterClear:0,framesBeforeCancel:0,framesAfterCancel:0,postCancelDone:true,postCancelReturned:0});assert.deepEqual([truncation.expected,truncation.received],[5,4]);assert.match(zeroProgress.message,/zero-progress/);
const semanticReceipt={schema:"byob-frame-decoder-v2",runtime:process.version,protocol:{lengthBytes:1,maximumFrameBytes:32,zeroLengthLegal:true},provenance:"Generated framed packets and frozen fill schedules; no network capture or throughput sample.",claimBoundary:"Correctness on this Node Web Streams runtime; not conformance, allocation, throughput, or zero-copy evidence.",sourceHex:Buffer.from(sourceBytes).toString("hex"),sourceSha256:sha(sourceBytes),runs,cancellation,hostile:{truncation,oversize,zeroProgress},replay:{first:replayA,second:replayB,equal:canonical(replayA)===canonical(replayB),normalizedSha256:sha(replayA)}};
console.log(JSON.stringify({...semanticReceipt,semanticReceiptHash:sha(semanticReceipt)},null,2));console.log("PASS: BYOB decoder proves frames, terminal cancellation, hostile schedules, and canonical replay");

Ship correctness receipts without performance theater

A release receipt should name the runtime, protocol version, maximum frame length, source-byte digest, fill schedule, read traces, decoded payloads, cancellation result, and hostile failures. Preserve it with the decoder tests. When the runtime or protocol changes, replay the same corpus before adding new cases.

The claim boundary matters: this fixture proves exact decoding for generated packets under tested Node Web Streams behavior. It is not a browser conformance suite, throughput benchmark, allocation profile, or universal zero-copy result. If a stakeholder asks whether BYOB is faster, answer with a planned benchmark rather than pointing at a correctness PASS. Include representative payload sizes, carry frequency, consumer work, garbage-collection observations, and a non-BYOB baseline in that plan. A smaller view count alone does not establish lower end-to-end cost.

ReadableStream BYOB is ready for a transport when incomplete state is visible, every end condition is explicit, and the artifact can reproduce its receipt. That is a stronger engineering story than an attractive low-copy slogan. Add the smallest real protocol packet only after the generated boundary corpus is stable, and keep its sensitive payload out of the archived trace. Replay the decoder against every partial-read schedule before connecting a real transport.