CreateReadStream. Sometimes a file may be too big to read into a single string. In Node.js, we can use a stream returned by createReadStream to process the file instead.
With the useful on() method, we can process a file in chunks—each chunk has a maximum number of bytes. And we must deal with a Buffer that contains the data.
Example. This program opens a large text file from the disk and counts its newlines in 2 ways. To run it, make sure the path points to a large, local text file.
Version 1 This loop within the on() function acts upon the Buffer. It uses indexOf on the buffer to count newline characters.
Result When we run this program, the target file is opened, and each version of the code returns the same newline count for each chunk.
const fs = require("node:fs");
// Create a read stream for this file.
const stream = fs.createReadStream("programs/words.txt");
// Call on to read the file until its end in chunks.
stream.on("data", (chunk) => {
// Version 1: count newlines in stream buffer with indexOf loop.
var newlines = 0;
while (true) {
var index = chunk.indexOf("\n", i);
if (index == -1) {
break;
}
newlines += 1;
i = index + 1;
}
// Version 2: count newlines in stream buffer with for-loop.
let newlines2 = 0;
const newlineInt = "\n".charCodeAt(0);
for (var i = 0; i < chunk.length; i++) {
if (chunk[i] == newlineInt) {
newlines2 += 1;
}
}
// Print results.
console.log("Newlines:", newlines, "; newlines2:", newlines2);
}); Newlines: 6390 ; newlines2: 6390
Newlines: 6643 ; newlines2: 6643
Newlines: 7051 ; newlines2: 7051
Newlines: 6582 ; newlines2: 6582
Newlines: 5961 ; newlines2: 5961
Newlines: 6408 ; newlines2: 6408
Newlines: 6199 ; newlines2: 6199
Newlines: 6465 ; newlines2: 6465
Newlines: 6640 ; newlines2: 6640
Newlines: 7022 ; newlines2: 7022
Newlines: 6528 ; newlines2: 6528
Newlines: 5478 ; newlines2: 5478
Newlines: 7078 ; newlines2: 7078
Newlines: 6677 ; newlines2: 6677
Newlines: 6420 ; newlines2: 6420
Newlines: 6201 ; newlines2: 6201
Newlines: 6239 ; newlines2: 6239
Newlines: 6207 ; newlines2: 6207
Newlines: 6099 ; newlines2: 6099
Newlines: 6284 ; newlines2: 6284
Newlines: 6884 ; newlines2: 6884
Newlines: 6947 ; newlines2: 6947
Newlines: 6487 ; newlines2: 6487
Newlines: 6449 ; newlines2: 6449
Newlines: 6430 ; newlines2: 6430
Newlines: 6571 ; newlines2: 6571
Newlines: 4480 ; newlines2: 4480
Summary. For efficiency, processing a file in chunks with createReadStream and on() is a clear improvement. The entire file does not need to be read into memory or processed at once.
Dot Net Perls is a collection of pages with code examples, which are updated to stay current. Programming is an art, and it can be learned from examples.
Donate to this site to help offset the costs of running the server. Sites like this will cease to exist if there is no financial support for them.
Sam Allen is passionate about computer languages, and he maintains 100% of the material available on this website. He hopes it makes the world a nicer place.