Build a streaming NDJSON parser

A network chunk is not a line, a character, or a JSON record.

A single UTF-8 character can span byte chunks. One NDJSON line can span a thousand chunks, while one chunk can contain a thousand lines. Splitting each decoded chunk on a newline works in demos and fails at exactly the boundaries streaming was supposed to handle.

The NDJSON specification requires UTF-8 JSON texts followed by newline, accepts LF and CRLF during parsing, and requires the parser's empty-line behavior to be documented. The WHATWG Streams Standard supplies the readable-stream, transform, backpressure, cancellation, and error model a web implementation needs around those records.

I would implement the parser as an async generator over validated records. Bytes are decoded incrementally, the unconsumed tail stays bounded, each complete line gets a stable number and byte context, and cancellation reaches the reader immediately.

The useful API is not parse everything. It is yield valid records as capacity allows, then fail or report according to one explicit policy.

Specify the input contract

Define accepted stream types, UTF-8 strictness, media type, LF and CRLF handling, empty lines, final unterminated line, maximum record size, and whether JSON primitives are allowed.

I would pressure-test that decision with four questions:

• Which bytes are accepted?
• Are empty lines ignored or errors?
• Must the file end in LF?
• Can a record be an array or scalar?

The failure mode here is coding undocumented conveniences that different consumers interpret differently. In browser and server clients that consume long-running exports, logs, search results, model events, telemetry, and data pipelines as newline-delimited JSON over a byte stream where UTF-8 characters, lines, records, and network chunks do not share boundaries, that can hide the exact boundary a reviewer or teammate needs to understand. My working artifact would be an NDJSON parser behavior table. I want it close enough to the implementation that it can change the work, not created afterward to decorate the story.

The result I would look for is conformance fixtures producing the published result on every runtime. That is a narrower claim than saying the whole system improved, but it is also one I can verify and defend.

In practice, I would put an NDJSON parser behavior table beside the question “Which bytes are accepted?” before the first implementation review. The next pass would use “Are empty lines ignored or errors?” to test the boundary, then “Must the file end in LF?” to expose the state most likely to be missed. I would keep “Can a record be an array or scalar?” for the release check because it asks whether the decision still holds outside the ideal path. The work is ready to move when the artifact can explain the choice and the observed result supports conformance fixtures producing the published result on every runtime.

Decode incrementally

Use TextDecoderStream where appropriate or TextDecoder with stream true so multibyte UTF-8 sequences survive chunk boundaries and invalid encodings follow one deliberate fatal or replacement policy.

The practical review starts here:

• Can a character span chunks?
• What happens to invalid UTF-8?
• Is the decoder flushed at EOF?
• Which runtimes need a fallback?

Those questions keep calling new TextDecoder().decode on each independent chunk from becoming the default. I would capture the decision in a streaming decoder adapter, then use it while the work is still cheap to change. For NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input, the artifact should make ownership, constraint, and next action visible without requiring a private explanation.

Success would look like split-code-point fixtures reproducing the original Unicode exactly. If I cannot point to that evidence, I have a direction, not a finished decision.

The implementation move is to make a streaming decoder adapter part of the working surface. I would use it to answer “Can a character span chunks?” while scope is still flexible, and “What happens to invalid UTF-8?” before code or content becomes expensive to unwind. During QA, “Is the decoder flushed at EOF?” and “Which runtimes need a fallback?” become concrete checks rather than discussion prompts. That sequence turns NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input into something the team can operate and gives me a specific outcome to report: split-code-point fixtures reproducing the original Unicode exactly.

Frame without quadratic copying

Carry one incomplete tail, scan complete delimiters, slice consumed content, and choose data structures that avoid repeatedly concatenating or rescanning an ever-growing buffer.

Before implementation, I would answer:

• How often is the tail copied?
• Can one line grow forever?
• Are delimiters scanned twice?
• What does the benchmark measure?

The artifact is a bounded line-framing transform. Its job is to expose the tradeoff early enough that design, engineering, support, or product can disagree with something concrete. The common trap is appending an unbounded response string and splitting it after download; it moves uncertainty downstream and makes the final interface carry a problem the system never resolved.

For me, the useful receipt is memory tracking the largest active line rather than the whole stream. That connects an incremental parser that decodes bytes safely, carries an incomplete line across chunks, bounds memory, validates and classifies each record, propagates backpressure and cancellation, reports line-aware errors, and accounts for the final unterminated line to an observable result instead of a process claim.

I would test this with one typical case and one boundary case. The typical case should make “How often is the tail copied?” easy to answer. The boundary should force a decision about “Can one line grow forever?” and “Are delimiters scanned twice?.” I would record both in a bounded line-framing transform, including the part that stayed unresolved after the first pass. The final check, “What does the benchmark measure?,” is where the artifact earns its place: it either supports memory tracking the largest active line rather than the whole stream, or it shows exactly why another iteration is needed.

Normalize CRLF precisely

Treat LF as the delimiter and remove exactly one preceding CR so Windows-style lines parse while carriage returns inside escaped JSON strings remain ordinary encoded content.

I would use these prompts during the working review:

• Where is CR allowed?
• Can a bare CR delimit?
• Is whitespace otherwise preserved?
• Do empty CRLF lines follow policy?

If the team slips into trimming every line and silently changing JSON whitespace or diagnostics, the product can still look complete while its operating rule stays ambiguous. I would make a delimiter normalization fixture set the shared reference and keep it small enough to update as evidence changes.

The standard is LF and CRLF sources yielding identical records and stable line numbers. That tells me whether the decision helped the product, not merely whether the document was completed.

The working sequence is small: draft a delimiter normalization fixture set, review it against “Where is CR allowed?,” implement the narrowest useful path, and then return with evidence for “Can a bare CR delimit?.” I would use “Is whitespace otherwise preserved?” to inspect product consequence and “Do empty CRLF lines follow policy?” to decide whether the result is stable enough to ship. This keeps trimming every line and silently changing JSON whitespace or diagnostics visible as a known risk and makes LF and CRLF sources yielding identical records and stable line numbers the release receipt rather than a hopeful conclusion.

Bound record size

Enforce a maximum while the partial line is accumulating, cancel the source when exceeded, and report measured size without echoing an entire attacker-controlled record.

I would pressure-test that decision with four questions:

• What maximum matches the product?
• Is it measured in bytes or code units?
• When is the check applied?
• How much content enters logs?

The failure mode here is checking size only after a newline finally arrives. In browser and server clients that consume long-running exports, logs, search results, model events, telemetry, and data pipelines as newline-delimited JSON over a byte stream where UTF-8 characters, lines, records, and network chunks do not share boundaries, that can hide the exact boundary a reviewer or teammate needs to understand. My working artifact would be a maximum-line guard. I want it close enough to the implementation that it can change the work, not created afterward to decorate the story.

The result I would look for is oversized-line fixtures stopping with bounded memory and safe diagnostics. That is a narrower claim than saying the whole system improved, but it is also one I can verify and defend.

In practice, I would put a maximum-line guard beside the question “What maximum matches the product?” before the first implementation review. The next pass would use “Is it measured in bytes or code units?” to test the boundary, then “When is the check applied?” to expose the state most likely to be missed. I would keep “How much content enters logs?” for the release check because it asks whether the decision still holds outside the ideal path. The work is ready to move when the artifact can explain the choice and the observed result supports oversized-line fixtures stopping with bounded memory and safe diagnostics.

Parse and validate separately

JSON syntax parsing and domain-schema validation should produce distinct errors with line number, record index, safe field path, schema version, and configurable continuation policy.

The practical review starts here:

• Was the line valid JSON?
• Does it match the expected event?
• Which version applies?
• Can processing continue safely?

Those questions keep catching every exception and returning null from becoming the default. I would capture the decision in a typed record-result union, then use it while the work is still cheap to change. For NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input, the artifact should make ownership, constraint, and next action visible without requiring a private explanation.

Success would look like callers handling syntax, schema, and transport failures without guesswork. If I cannot point to that evidence, I have a direction, not a finished decision.

The implementation move is to make a typed record-result union part of the working surface. I would use it to answer “Was the line valid JSON?” while scope is still flexible, and “Does it match the expected event?” before code or content becomes expensive to unwind. During QA, “Which version applies?” and “Can processing continue safely?” become concrete checks rather than discussion prompts. That sequence turns NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input into something the team can operate and gives me a specific outcome to report: callers handling syntax, schema, and transport failures without guesswork.

Preserve backpressure

The parser should read and yield on consumer demand rather than draining the source into an internal queue, with any concurrency bounded and order behavior documented.

Before implementation, I would answer:

• Who pulls the next chunk?
• Can validation outrun rendering?
• What queue limit applies?
• Must records remain ordered?

The artifact is a backpressure propagation diagram. Its job is to expose the tradeoff early enough that design, engineering, support, or product can disagree with something concrete. The common trap is starting a background read loop that buffers the stream regardless of consumer speed; it moves uncertainty downstream and makes the final interface carry a problem the system never resolved.

For me, the useful receipt is slow-consumer tests keeping queue and memory inside configured bounds. That connects an incremental parser that decodes bytes safely, carries an incomplete line across chunks, bounds memory, validates and classifies each record, propagates backpressure and cancellation, reports line-aware errors, and accounts for the final unterminated line to an observable result instead of a process claim.

I would test this with one typical case and one boundary case. The typical case should make “Who pulls the next chunk?” easy to answer. The boundary should force a decision about “Can validation outrun rendering?” and “What queue limit applies?.” I would record both in a backpressure propagation diagram, including the part that stayed unresolved after the first pass. The final check, “Must records remain ordered?,” is where the artifact earns its place: it either supports slow-consumer tests keeping queue and memory inside configured bounds, or it shows exactly why another iteration is needed.

Propagate cancellation

An AbortSignal, early iterator return, route change, consumer error, and timeout should cancel the reader, stop transforms, release locks in finally, and preserve the original failure cause.

I would use these prompts during the working review:

• What aborts the network request?
• Does return call cancel?
• Are reader locks released?
• Which error reaches the caller?

If the team slips into breaking out of iteration while the response continues downloading, the product can still look complete while its operating rule stays ambiguous. I would make an abort-and-cleanup wrapper the shared reference and keep it small enough to update as evidence changes.

The standard is cancellation fixtures ending reads and releasing resources promptly. That tells me whether the decision helped the product, not merely whether the document was completed.

The working sequence is small: draft an abort-and-cleanup wrapper, review it against “What aborts the network request?,” implement the narrowest useful path, and then return with evidence for “Does return call cancel?.” I would use “Are reader locks released?” to inspect product consequence and “Which error reaches the caller?” to decide whether the result is stable enough to ship. This keeps breaking out of iteration while the response continues downloading visible as a known risk and makes cancellation fixtures ending reads and releasing resources promptly the release receipt rather than a hopeful conclusion.

Handle end of stream

Flush the decoder, inspect the remaining tail, apply the final-line policy, distinguish clean EOF from truncation signals where possible, and emit a terminal summary only after all accepted records yield.

I would pressure-test that decision with four questions:

• Is an unterminated final JSON accepted?
• Can transport truncation be detected?
• Was the decoder flushed?
• When is completion announced?

The failure mode here is dropping the last record because no newline followed it or accepting corrupt truncation silently. In browser and server clients that consume long-running exports, logs, search results, model events, telemetry, and data pipelines as newline-delimited JSON over a byte stream where UTF-8 characters, lines, records, and network chunks do not share boundaries, that can hide the exact boundary a reviewer or teammate needs to understand. My working artifact would be an EOF decision table. I want it close enough to the implementation that it can change the work, not created afterward to decorate the story.

The result I would look for is final-line fixtures producing explicit accepted, rejected, or incomplete outcomes. That is a narrower claim than saying the whole system improved, but it is also one I can verify and defend.

In practice, I would put an EOF decision table beside the question “Is an unterminated final JSON accepted?” before the first implementation review. The next pass would use “Can transport truncation be detected?” to test the boundary, then “Was the decoder flushed?” to expose the state most likely to be missed. I would keep “When is completion announced?” for the release check because it asks whether the decision still holds outside the ideal path. The work is ready to move when the artifact can explain the choice and the observed result supports final-line fixtures producing explicit accepted, rejected, or incomplete outcomes.

Test adversarial boundaries

Tests should split every byte position around multibyte characters, backslashes, CRLF, long lines, blank lines, JSON tokens, EOF, abort, invalid encoding, and consumer exceptions.

The practical review starts here:

• Does every possible split preserve records?
• Can memory exceed the line limit?
• Does abort release the source?
• Are diagnostics safe?

Those questions keep testing only one server's convenient chunk sizes from becoming the default. I would capture the decision in a chunk-boundary property test suite, then use it while the work is still cheap to change. For NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input, the artifact should make ownership, constraint, and next action visible without requiring a private explanation.

Success would look like identical logical input yielding identical records across randomized chunking. If I cannot point to that evidence, I have a direction, not a finished decision.

The implementation move is to make a chunk-boundary property test suite part of the working surface. I would use it to answer “Does every possible split preserve records?” while scope is still flexible, and “Can memory exceed the line limit?” before code or content becomes expensive to unwind. During QA, “Does abort release the source?” and “Are diagnostics safe?” become concrete checks rather than discussion prompts. That sequence turns NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input into something the team can operate and gives me a specific outcome to report: identical logical input yielding identical records across randomized chunking.

What I would show in the work

The public version needs evidence from the work itself. For this topic, the first five artifacts I would reach for are:

• an NDJSON parser behavior table
• a streaming decoder adapter
• a bounded line-framing transform
• a delimiter normalization fixture set
• a maximum-line guard

I would not publish all five at equal weight. One should orient the reader, one should reveal the hardest tradeoff, and one should prove the result. The others can live in a downloadable note or appear as supporting frames. That edit matters because an incremental parser that decodes bytes safely, carries an incomplete line across chunks, bounds memory, validates and classifies each record, propagates backpressure and cancellation, reports line-aware errors, and accounts for the final unterminated line becomes harder to understand when every process detail is treated as equally important.

I would also show one rejected direction. The useful version is specific: which option looked attractive, which constraint made it wrong, and what evidence supported the narrower choice. That gives an engineering manager something real to question and keeps the case study from reading like the final answer was obvious from the beginning.

# input
response 200 / application/x-ndjson / gzip
Abort signal active, fatal UTF-8 true, empty lines reject, max line 1 MiB, schema event-v3.
# progress
8.2 MiB / 44,812 lines / 44,811 records
One CRLF normalized, maximum buffered tail 18 KiB, consumer wait 62 percent, network reads naturally backpressured.
# terminal
complete / final LF present / reader released
0 malformed, 0 schema errors, digest recorded, duration 18.4s; cancellation fixture releases source within 20ms.

Resource path

The practical follow-up I would build is a streaming NDJSON parser kit with TypeScript async generator, TextDecoderStream and fallback paths, CRLF normalization, empty-line policy, maximum-line enforcement, schema validation hook, abort propagation, reader cleanup, backpressure demo, byte and line counters, error modes, final-line policy, fixtures, and benchmark harness. I am treating that as a resource backlog item, not pretending the adjacent downloads below are the same artifact. The related cards cover useful pieces of the workflow today; this specific file should only be published when its examples, fields, and instructions are complete.

The first version should stay concise: context, constraint, decision, evidence, owner, and follow-up. Its value would come from helping someone repeat this exact review, not from adding another generic PDF to the site.

Review checklist

The article-specific review questions are:

• Which bytes are accepted?
• Can a character span chunks?
• How often is the tail copied?
• Where is CR allowed?
• What maximum matches the product?
• Was the line valid JSON?
• Who pulls the next chunk?
• What aborts the network request?
• Is an unterminated final JSON accepted?
• Does every possible split preserve records?

I would add two editorial checks before publishing: can a recruiter find the point in the first minute, and can an engineer trace at least one claim to an implementation or production receipt? If either answer is no, the article needs another edit.

Implementation notes

For NDJSON consumers that deliver useful records early without corrupting boundaries or hiding malformed input, I would write the implementation note before polish. It would name the changed surface, source of truth, owner, failure boundary, and verification path. Those details prevent the principle from floating above the actual code or operational workflow.

The proof signals I care about are specific to this article:

• callers handling syntax, schema, and transport failures without guesswork
• slow-consumer tests keeping queue and memory inside configured bounds
• cancellation fixtures ending reads and releasing resources promptly
• final-line fixtures producing explicit accepted, rejected, or incomplete outcomes
• identical logical input yielding identical records across randomized chunking

I would choose two or three of those signals for the first release rather than instrumenting everything. The strongest pair usually combines one direct behavior check with one operating check: a route and a data query, a keyboard path and a support state, a handler replay and a reconciliation result, or a migration count and a rendered screen.

The follow-up belongs in the note before shipping. It should say what remains temporary, what evidence would trigger another pass, and who owns that decision. That is how the first version stays intentionally narrow without making the boundary invisible.

Case-study packaging

I would structure the case-study version around the four visual lessons already established:

• Boundaries are reconstructed in stages.
• One line can cross every chunk boundary.
• Error modes are product decisions.
• The parser receipt makes completion measurable.

The opening frame explains the product pressure. The middle two show the decision moving through the system. The last frame is the receipt: what was checked, what held, and what remained unresolved. That order lets the reader move from product judgment into implementation detail without reconstructing the whole project first.

I would include one caveat tied to browser and server clients that consume long-running exports, logs, search results, model events, telemetry, and data pipelines as newline-delimited JSON over a byte stream where UTF-8 characters, lines, records, and network chunks do not share boundaries: a data limit, rollout boundary, unsupported state, external dependency, or result that is still directional. A precise caveat makes the evidence easier to trust because it shows where the claim stops.

The final test is whether the page creates a better conversation. If the artifact helps someone ask a sharper question about product judgment, implementation detail, or release proof in a live interview, it belongs in the story.

Interview angle

In an interview, I would explain this through an incremental parser that decodes bytes safely, carries an incomplete line across chunks, bounds memory, validates and classifies each record, propagates backpressure and cancellation, reports line-aware errors, and accounts for the final unterminated line. The story should start with the product pressure, then move into the system constraint, the artifact, and the proof. That order keeps the answer grounded. It also gives the interviewer several places to go deeper: data, frontend architecture, design systems, support, migration, accessibility, or release process.

The strongest version of the answer includes a tradeoff. I want to be able to say what I chose, what I left alone, and how I knew the work helped. That is more credible than presenting every project as a clean win.

The hiring signal

A production-grade NDJSON parser is a hiring signal because it shows I can connect byte encodings, Web Streams, async iteration, memory bounds, failure policy, cancellation, observability, and end-user progress.

That is the level I want this site to communicate. The work should show taste, but it should also show operating judgment. It should make me look like someone who can enter a real product system, understand the messy middle, ship the useful version, and leave enough proof for the next person to trust it.