2.3Streams & Buffers: handling large volumes
The essentials in 30 seconds
- In chunks, not all at once: the stream processes each chunk as it flows — memory stays flat whatever the size.
- .pipe() connects: reading flows into writing automatically, with flow control — the 80/20 of streams in one line.
- Large volumes = streams: big files, uploads, network flows — small files = readFile is enough.
Ongoing mini-case — Salim copies archives without crashing the server
Salim must archive the year's reports: a single 2 GB export. His first attempt with readFile blows up the small server's memory. He switches to streams: chunk-by-chunk reading, copy to the archive via .pipe(), then a read → compress → write pipeline to shrink the size. Memory barely moves, and processing starts from the very first chunk. In 2.4, his server will send large files to clients as streams.
Sequel to the 2.2 thread: Salim's reports become a large volume to process — and continue in 2.4.
1. The large-volume problem
Imagine processing a 2 GB file. The naive approach loads it entirely into memory (fs.readFile puts all the content in one variable) — but 2 GB in memory is huge, even impossible: RAM fills up. Streams solve this: you process data in chunks, as it comes, like a flowing stream. You read one chunk, process it, move to the next — memory never holds more than a small chunk. Efficient in memory (volumes bigger than RAM) and in time (you start from the first chunk, without waiting for the full load). Salim measured it: his 2 GB export goes through with a few MB of RAM.
2. Stream types and .pipe()
Four types: Readable (reading from a source: file, network request), Writable (writing to a destination), Duplex (both, like a socket), Transform (transforms on the way: compression, encryption). Streams are EventEmitters (section 2-2): you read them via 'data' (chunk received), 'end' (finished), 'error' (problem). But the most elegant tool is .pipe(): it CONNECTS a read to a write, data flows automatically chunk by chunk with flow control. Copying a big file: fs.createReadStream('source').pipe(fs.createWriteStream('destination')) — whatever the size. You even chain them: lecture.pipe(compression).pipe(ecriture).
3. Buffers and when to use streams
Related concept: Buffers — Node handles binary data (files, network) through raw byte memory areas, since JavaScript has no native binary type; streams often carry Buffers (the chunks). When to use streams: for LARGE volumes — big files, uploads/downloads (a big upload held in memory would saturate the server: a performance and security flaw), continuous network flows, bulk processing, transformation pipelines, serving big files from a server. For small files, plain readFile is enough: simple and readable. Streams take some learning, but they are a mark of professional Node code.
🔎 Concrete example — 🔎 Concrete example — Salim's compressed archive
The 80/20 pipeline: read, compress, write — without loading everything:
const fs = require('node:fs');
const zlib = require('node:zlib');
fs.createReadStream('export-2026.csv')
.pipe(zlib.createGzip())
.pipe(fs.createWriteStream('export-2026.csv.gz'))
.on('finish', () => console.log('Archive ready'));Reading: 2 GB processed with a few MB of RAM, immediate processing.
Lesson: chaining streams = transformation pipeline.
Section vocabulary
- Stream
- Processing data in chunks, as it comes, rather than loading everything into memory; efficient for large volumes.
Simple example: Salim archives his 2 GB export chunk by chunk.
What it is not: a readFile that loads the whole file into a variable.
Reflex: large volume = think flow, not full load.
- Stream types
- Readable (reading), Writable (writing), Duplex (both, e.g. socket), Transform (transforms on the way: compression, encryption).
Simple example: Salim's gzip is a Transform between reading and writing.
What it is not: four interchangeable syntaxes with no distinction.
Reflex: transformation on the way = Transform.
- .pipe()
- Connects a readable stream to a writable stream: data flows automatically, chunk by chunk, with flow control.
Simple example: createReadStream('source').pipe(createWriteStream('archive')).
What it is not: a manual loop that accumulates everything before writing.
Reflex: source to destination = .pipe() first.
- Buffer
- Raw byte memory area for binary data (files, network), which JavaScript has no native type for; streams carry them.
Simple example: Each chunk of Salim's export arrives as a Buffer.
What it is not: an ordinary character string.
Reflex: binary data = think Buffer.
- When to use streams
- Whenever you process large volumes (big files, uploads/downloads, network flows); for small files, readFile is enough.
Simple example: Salim streams the 2 GB export but reads the inventory (a few KB) with readFile.
What it is not: streaming systematically, even a config file.
Reflex: size and flow continuity = choice criterion.
In practice — In practice — Processing large volumes with streams
- Copy a file with a stream: fs.createReadStream('source').pipe(fs.createWriteStream('dest')) — without loading everything into memory.
- Read a stream “by hand” with events: .on('data', chunk => ...), .on('end', ...), .on('error', ...).
- Chain several streams with pipe (e.g. read → gzip compress → write) for a processing pipeline.
- Compare: measure the memory used to process a big file with readFile (all in memory) vs with a stream (in chunks).
Ready-to-say script — justifying streams to your team (30 s)
“My first attempt loaded the whole export into memory and saturated the server. As a stream, we process chunk by chunk: memory barely moves whatever the size, and we start from the first chunk. One pipe line is enough to copy, and we chain compression without storing anything between steps.”
Say it with the memory measurement in front of you (readFile vs stream); the number convinces better than the principle.
Copy-paste template — robust copy and gzip pipeline
const fs = require('node:fs');
const zlib = require('node:zlib');
const { pipeline } = require('node:stream/promises');
async function copier(source, destination) {
await pipeline(
fs.createReadStream(source),
fs.createWriteStream(destination)
);
console.log('Copied: ' + destination);
}
async function compresser(source, destination) {
await pipeline(
fs.createReadStream(source),
zlib.createGzip(),
fs.createWriteStream(destination)
);
console.log('Compressed: ' + destination);
}
copier('export-2026.csv', 'archive-2026.csv').catch(console.error);Small file
A few-KB inventory: readFile is enough — a stream would be needless complexity.
Team's big export
Salim compresses the yearly export as a pipeline: read → gzip → write, constant memory.
User upload
Receiving a client-sent file as a stream: otherwise a big upload saturates the server's memory.
Manual reading
Need to react to each chunk (counting lines): .on('data'/'end'/'error') on the Readable.
✓ Best practices
- Start with .pipe() (or pipeline): the 80/20 of streams in one line.
- Always listen for error on a manually read stream.
- Reserve streams for large volumes; keep readFile for small files.
✗ Common pitfalls
- Loading a big file with readFile (saturated RAM, crash).
- Forgetting the error listener on a manual stream.
- Streaming systematically, even tiny files.
Key takeaways
- STREAMS process data in CHUNKS, as it flows, instead of loading everything into memory — efficient for LARGE volumes (memory AND time).
- Types: Readable (reading), Writable (writing), Duplex, Transform (transforms on the way: compression, encryption). Streams are EventEmitters (.on('data'/'end'/'error')).
- .pipe() CONNECTS a readable stream to a writable one (data flows automatically, chunk by chunk): readStream.pipe(writeStream). Chain them (read→compress→write).
- Use streams whenever you process LARGE volumes (big files, uploads, network flows); for small files, readFile is enough. A mark of performant Node code.
48h challenge — apply it tomorrow
Take a real big file (export, log, video) and copy it with .pipe(), then compress it in a read → gzip → write pipeline. Measure memory in both cases and compare with a readFile.
- I can explain the chunk principle and constant memory.
- I can copy and compress with pipe/pipeline.
- I measured the stream vs readFile memory gap on a real file.
Frequently asked questions
When should you really use streams rather than plain readFile/writeFile?
Whenever size or continuity demands it: readFile loads the file entirely into memory — perfect for a few KB or MB, but catastrophic for hundreds of MB or GB (saturated RAM, even crashes). Use streams for big files (logs, exports, media), uploads/downloads (otherwise a big upload saturates the server), continuous flows (voluminous response, live data), bulk processing (huge CSV line by line) and pipelines (read → compress → write with no intermediate storage). Salim's rule: small and one-off = readFile; big or continuous = stream.
Streams look complicated: where do I start learning them gradually?
In 4 steps, from simple to complete. 1) The principle: processing in flowing chunks (the pipe, not the bucket) — that's the core. 2) .pipe() for simple cases: createReadStream('source').pipe(createWriteStream('dest')) — the 80/20, with flow control included. 3) Chaining: lecture.pipe(gzip).pipe(ecriture), then pipeline() from the stream module for robust pipelines with good error handling. 4) Reading by hand: .on('data'/'end'/'error') when you must react to each chunk. Like Salim: pipe first, the rest when needed.
Salim must copy a 2 GB export without saturating the server's RAM. What does he do?