`CreateReadStream`

Sometimes a file may be too big to read into a single string. In Node.js, we can use a stream returned by createReadStream to process the file instead.

With the useful on() method, we can process a file in chunks—each chunk has a maximum number of bytes. And we must deal with a Buffer that contains the data.

Example

This program opens a large text file from the disk and counts its newlines in 2 ways. To run it, make sure the path points to a large, local text file.

Version 1 This loop within the on() function acts upon the Buffer. It uses indexOf on the buffer to count newline characters.

Version 2 Here we also count newline chars, but we access individual bytes from the Buffer in a for-loop.

Result When we run this program, the target file is opened, and each version of the code returns the same newline count for each chunk.

const fs = require("node:fs");

// Create a read stream for this file.
const stream = fs.createReadStream("programs/words.txt");

// Call on to read the file until its end in chunks.
stream.on("data", (chunk) => {

    // Version 1: count newlines in stream buffer with indexOf loop.
    var newlines = 0;
    while (true) {
        var index = chunk.indexOf("\n", i);
        if (index == -1) {
            break;
        }
        newlines += 1;
        i = index + 1;
    }

    // Version 2: count newlines in stream buffer with for-loop.
    let newlines2 = 0;
    const newlineInt = "\n".charCodeAt(0);
    for (var i = 0; i < chunk.length; i++) {
        if (chunk[i] == newlineInt) {
            newlines2 += 1;
        }
    }

    // Print results.
    console.log("Newlines:", newlines, "; newlines2:", newlines2);
}); Newlines: 6390 ; newlines2: 6390
Newlines: 6643 ; newlines2: 6643
Newlines: 7051 ; newlines2: 7051
Newlines: 6582 ; newlines2: 6582
Newlines: 5961 ; newlines2: 5961
Newlines: 6408 ; newlines2: 6408
Newlines: 6199 ; newlines2: 6199
Newlines: 6465 ; newlines2: 6465
Newlines: 6640 ; newlines2: 6640
Newlines: 7022 ; newlines2: 7022
Newlines: 6528 ; newlines2: 6528
Newlines: 5478 ; newlines2: 5478
Newlines: 7078 ; newlines2: 7078
Newlines: 6677 ; newlines2: 6677
Newlines: 6420 ; newlines2: 6420
Newlines: 6201 ; newlines2: 6201
Newlines: 6239 ; newlines2: 6239
Newlines: 6207 ; newlines2: 6207
Newlines: 6099 ; newlines2: 6099
Newlines: 6284 ; newlines2: 6284
Newlines: 6884 ; newlines2: 6884
Newlines: 6947 ; newlines2: 6947
Newlines: 6487 ; newlines2: 6487
Newlines: 6449 ; newlines2: 6449
Newlines: 6430 ; newlines2: 6430
Newlines: 6571 ; newlines2: 6571
Newlines: 4480 ; newlines2: 4480

For efficiency, processing a file in chunks with createReadStream and on() is a clear improvement. The entire file does not need to be read into memory or processed at once.