Giant JSON Files Hit a Wall Long Before RAM Runs Out

Every team that exports data as JSON eventually produces a file that JSON.parse(fs.readFileSync(...)) can't swallow. The folk explanation is "it ran out of memory." Usually that's wrong. On 64-bit Node you hit two hard-coded ceilings well before the heap gives out, and understanding them changes which fix you reach for.

The wall is a constant, not your RAM

V8 caps a single string at 536,870,888 UTF-16 code units (0x1fffffe8, roughly 512 MiB). You can see it yourself with require('buffer').constants.MAX_STRING_LENGTH. Read a 600 MB file with encoding: 'utf8' and Node throws ERR_STRING_TOO_LONG — "Cannot create a string longer than 0x1fffffe8 characters" — before JSON.parse ever runs. Adding --max-old-space-size=16000 does nothing, because the limit isn't the heap.

The second ceiling sits in the filesystem layer. Node's error reference documents ERR_FS_FILE_TOO_LARGE as a 2 GiB limit on fs.readFile(), explicitly "not a limitation of Buffer, but an internal I/O constraint," and points you at fs.createReadStream(). (Buffer's own cap was raised to 2^53 − 1 bytes in Node 22; on Node 20 it's still 4 GiB.)

So the honest budget for the whole-file approach is: files under ~500 MB of text, and a peak working set of roughly two to three times the file size once you count the source string plus the object tree. Past that, no flag saves you. The design has to change.

The container is the problem, not the parser

A top-level JSON array is a terrible envelope for bulk data. Nothing about [ tells a parser how many elements follow, and nothing before the final ] confirms the document is complete. That's why a naive parser has to hold everything, and it's why the streaming libraries exist at all — they're compensating for a format decision made upstream.

Newline-delimited JSON (NDJSON, also JSON Lines) removes the problem instead of working around it. Each line is a self-contained document, so the "parser" is readline plus the built-in JSON.parse you already trust:

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const rl = createInterface({ input: createReadStream("events.ndjson") });
for await (const line of rl) {
  if (!line) continue;
  const event = JSON.parse(line);
  // handle event, then let it go
}

Memory stays flat regardless of file size, corrupt records fail one line at a time instead of poisoning the whole document, and you can split the file with split -l and fan it out across worker threads or machines without any parser cooperation. No streaming JSON tokenizer matches that on speed or simplicity, because per-line JSON.parse runs in V8's native fast path rather than in a JavaScript state machine.

If you control the producer — an export job, an ETL step, an internal API — emit NDJSON and gzip it. This is the single highest-leverage change in this whole space, and it's the one the tutorials tend to bury under library recommendations.

When someone hands you a 4 GB array anyway

You don't always control the format. Then you have three good options, and they're not interchangeable.

Convert once, with jq. jq's --stream mode emits [path, leaf] events instead of building a value tree. The idiom to turn a top-level array into NDJSON in constant memory is:

jq -cn --stream 'fromstream(1|truncate_stream(inputs))' huge.json > huge.ndjson

truncate_stream(1) strips the array index off each path, and fromstream reassembles each element as it completes. It's slower than a normal jq '.[]', which would have to load the whole file, but it's the difference between a job that finishes and one that doesn't. Do this once at the boundary and never touch the array form again.

Stream in-process with stream-json. If the transformation has to live in your Node service, stream-json is the mature choice. Version 3 is ESM-only and ships Web Streams builds, so the same pipeline works in Node, Bun, Deno, and the browser. The architecture is a token stream — parser() tokenizes, Pick selects a subtree by path, StreamArray or StreamValues assembles objects one at a time — and the important property is that bytes you skip are never materialized. For a document shaped like {"meta": {...}, "records": [...]}, you pick({ filter: 'records' }) and then streamArray(); the meta object costs you nothing.

The trade-off: you're now tokenizing in JavaScript, so throughput is well below per-line JSON.parse, and you inherit backpressure as a correctness concern. If your database writes 2,000 rows/s and your parser emits 50,000 objects/s, on('data') handlers will happily buffer the difference in memory. Use for await over the stream or stream.pipeline with an async consumer so the read pauses when downstream stalls, and batch inserts (a few hundred rows per call) so the write side isn't the bottleneck by construction.

Python has the same shape. ijson.items(f, 'item') does for Python what streamArray does for Node, and it's worth checking ijson.backend — the yajl2_c C extension is dramatically faster than the pure-Python fallback, and it's what you get by default on most platforms. Pass use_float=True unless you actually need Decimal.

Maybe you don't need a parser at all

A lot of "process this giant JSON file" tasks are really "answer questions about this giant JSON file." For those, loading records into your application language is the slow path. DuckDB's read_json and read_ndjson infer a schema from a sample and scan the file as a table, handling both newline-delimited and format = 'array' inputs. The one knob to know is maximum_object_size, which defaults to 16 MiB per object — fine for records, not for a document that's one enormous nested blob.

SELECT country, count(*) FROM read_ndjson('events.ndjson') GROUP BY 1;

That's an aggregation over gigabytes without writing a stream handler, a batch loop, or a backpressure fix. If the end state of your pipeline is a Parquet file or a warehouse table, this is usually where you should have started.

Where the ceiling really is

The C++ world shows what's left on the table. simdjson parses at multiple gigabytes per second with an On-Demand front-end that treats a document as an iterator over the JSON text — it parses values as you touch them and skips the rest. Its iterate_many handles NDJSON streams with a sliding window (1 MB by default, capped at 4 GB), and the docs are candid that it's tuned for many small documents, not one big one.

The lesson transfers even if you never write C++: the expensive part of JSON isn't scanning bytes, it's materializing objects you don't need. Every technique above — NDJSON, Pick, --stream, columnar scans — is a way of not building the tree. Choose the one that fits where you sit in the pipeline: fix the format if you're the producer, convert once at the boundary if you're not, stream in-process only when the logic genuinely has to live there, and skip the application layer entirely when the job is analytical.