Two use cases of writable streams

Writable streams are instances of the Writable class and they represent chunks of data that come out of our application. For example, we can display data in the terminal or in the browser or we can write the data in a file. Data sort of “leave” our application this time and get written in the terminal, file or browser.

Writable streams also inherit from EventEmitter, which means that they also emit various events we can listen to, that is we can write listeners to various phases of the streams. Said differently, listening to an event means that we can run some custom code at given stages of the data flow.

The write() method

Writable streams come with a method called write. We need to pass what we want to write to the stream as the first parameter of the method.

Consider the following pseudo-code:

writable.write('Streams are great!');

In this case, we sort of “place” this message (the data) on the writable stream and want to put it out somewhere. This “somewhere” can be the terminal or the browser, when the data is visible, or even a file (where data is visible as well but we need to click on the file to see the content).

write can also accept a second (optional) argument, which is the encoding parameter, for example utf8 (not shown in the code snippet above). We can also pass it a third parameter, which is a callback function called once the data has been handled.

About pipe() again

Although I covered pipe in the previous post, I briefly mention it here, simply because it’s a really cool feature and makes our life much easier when working with streams. pipe automatically manages the phenomenon called backpressure, which means that data is arriving faster from the readable part than the writable one can consume. pipe is easy to use, since all we have to remember is the formula:

readable.pipe(writable)

However, it’s not compulsory to use pipe. Streams can be managed using events as well if we want more control over the whole process. By writing listener callbacks to the events, we can touch each phase and customize the behaviour of the stream.

Writing to files

Let’s have a look first how to create a file that has a 100 000 lines. We will create this file with the help of a writable stream. (Homework for experiment-oriented readers: Compare the time spent creating a hundred thousand lines by hand vs using streams. Good luck.)

Consider the following:

const fs = require('fs');

const writable = fs.createWriteStream('./bigfile.txt');

function writeToFile(toWrite) {
  writable.on('open', () => {
    console.log('File is being created.');
  });

  for (let i = 1; i <= 1e5; i++) {
    writable.write(`Line ${i}: ${toWrite}\n`);
  }

  writable.end('End of data');
}

writeToFile('This is a line of the big file.');

The createWriteStream method of the fs module returns a writable stream, so we can use the methods and events available on writable streams here, too.

The writeToFile function runs a for loop that generates 100 000 lines of the input (toWrite) and we also display the line number to make them easier to count. We call the write method on each iteration so that the expression is copied in the file at the required number of times.

We can also listen to the open event the writable stream emits. Doing it is not compulsory and it’s nothing wrong to omit this step. This is just to show how we can listen to various events writable streams emit. There are more events to listen to and feel free to play around with them.

Here we are simply logging a message that informs us about the start of the write process.

After the loop has run and all lines have been created, we call the end method on the writable stream. This method indicates that no more data left to be written on the stream.

We can optionally pass end a chunk of data (in this case the “End of data” string) and this short message will be written last to the stream. If end receives data, the finish event will be emitted and the stream gets closed. No more data can be written to the stream (really).

By invoking writeToFile, the “File is being created” message will be written to the console first and bigfile.txt will get created in the same folder as the file containing the code above is located. After a second or so, 100 000 lines will be found in bigfile.txt. Really cool!

Writing to file from the terminal

The content of bigfile.txt is hardcoded. Can we create this file by typing the input in the terminal?

The answer is yes, we can.

process.stdin

process.stdin or standard input is a readable stream. You put data on this readable stream by typing in the terminal.

The following easy example will help:

process.stdin.pipe(process.stdout);

Yes, this is the good old stream-formula. The readable stream is process.stdin and the writable stream is process.stdout, i.e. the standard output.

Create a file called write-to-terminal.js, copy the line above in the file and run the node write-to-terminal.js command from the folder where the file is located. Nothing will happen but as soon as you type something and press Enter, the words you typed will be displayed again in the terminal. You can continue typing until you get bored and press Ctrl + D, which terminates the process.

Writing data to file

Now that we have seen how the terminal input can be displayed in the terminal itself, the next step can be to write that data in a file.

All we have to do is just replace process.stdout with fs.createWriteStream and everything we type in the terminal will be written into the file:

const fs = require('fs');

const writable = fs.createWriteStream('./from-terminal.txt');

process.stdin.pipe(writable);

If you run the code above and then open from-terminal.txt, you will have everything you typed in the file.

Multiplying data

Writing data one line after the other is not much fun, so why not create, say, 100 000 lines of the typed input?

In this case we won’t use pipe but will manage the stream with events.

The following snippet shows how we can achieve this. Create a file called data-to-file.js and ensure your file has the code below:

const fs = require('fs');

const writable = fs.createWriteStream('./bigfile2.txt');

let dataToWrite = '';

const writeToFile = (data) => {
  for (let i = 1; i <= 1e5; i++) {
    writable.write(`Section ${i}: ${data}`);
  }
}

process.stdin.on('data', (chunk) => {
  dataToWrite += chunk;
});

process.stdin.on('end', () => {
  writeToFile(dataToWrite);
  writable.end('End of data');
});

Quite a few concepts are repeated from the example above. We again use a writable stream to write into the file as the number of lines (i.e. the quantity of data) is quite large, but this time the name of the file is bigfile2.txt.

I also copied the writeToFile function, which runs the loop with our input data.

But we can also see some differences as well. First, a dataToWrite variable is created and is initially equal to an empty string. This variable will “collect” the input from the terminal. The listener to the data event saves everything we type including the line break entered by hitting Enter.

Listening to the data event emitted by the standard input (process.stdin) basically means that whenever we type a word or sentence in the terminal and then press Enter, we will get notified that a new chunk of data is available for use.

Now when you feel that you typed enough in the terminal, press ctrl + D. By doing this we indicate that we don’t want to enter more data, and the end event on our readable stream (process.stdin, i.e. the terminal) will be released.

If end is emitted, we can surely listen to it. dataToWrite contains everything we have entered, so this is the right time to call the writeToFile function. writeFile will run the for loop and write 100 000 of everything we typed in the terminal. Once the loop terminates, we call the end method on the writable stream and pass it a short message, which will also be written at the end of the file.

Conclusion

Streams are cool and very powerful features of Node.js! By mixing the code from the last article with those above, we can easily copy content from one file to another, create data files and the display data in the browser and so on. Although these two posts only serve as a brief introduction to Node.js streams, I hope that you can see the benefits of using them.