Building a Multi-Response Streaming API with Node.js, Express, and React

Introduction

As web applications become increasingly complex and data-driven, efficient and effective data transfer methods become critically important. A streaming API that can send multiple responses to a single request can be a powerful tool for handling large amounts of data or for delivering real-time updates. In this article, we will guide you through the process of creating such an API.

We will use video streaming as an illustrative example. With their large file sizes and the need for flexible, on-demand delivery, videos present a fitting scenario for showcasing the power of multi-response streaming APIs. The backend will be built with Node.js and Express, utilizing HTTP range requests to facilitate efficient data delivery in chunks.

Next, we'll build a React front-end to interact with our streaming API. This front-end will handle both the display of the streamed video content and its download, offering users real-time progress updates.

By the end of this walkthrough, you will have a working example of a multi-response streaming API, and you will be able to apply the principles learned to a wide array of use cases beyond video streaming.

Let's jump right into it!

Hands-On

Implementing the Streaming API in Express

In this section, we will dive into the server-side implementation, specifically our Node.js and Express application. We'll be implementing an API endpoint to deliver video content in a streaming fashion.

Assuming you have already set up your Express server with TypeScript, we first need to define our video-serving route. We'll create a GET endpoint that, when hit, will stream a video file back to the client.

Please make sure to install cors for handling cross-origin requests, dotenv for loading environment variables, and throttle for controlling the rate of data transfer. You can install these with the following command:

yarn add cors dotenv throttle @types/cors @types/dotenv @types/throttle

import cors from 'cors';
import 'dotenv/config';
import express, { Request, Response } from 'express';
import fs from 'fs';
import Throttle from 'throttle';

const app = express();
const port = 8000;

app.use(cors());

app.get('/video', (req: Request, res: Response) => {
  // Video by Zlatin Georgiev from Pexels: https://www.pexels.com/video/15708449/
  // For testing purposes - add the video in you `static` folder
  const path = 'src/static/pexels-zlatin-georgiev-15708449 (2160p).mp4';
  const stat = fs.statSync(path);
  const fileSize = stat.size;
  const range = req.headers.range;
  if (range) {
    const parts = range.replace(/bytes=/, '').split('-');
    const start = parseInt(parts[0], 10);
    const end = parts[1] ? parseInt(parts[1], 10) : fileSize - 1;

    const chunksize = end - start + 1;
    const file = fs.createReadStream(path, { start, end });
    const head = {
      'Content-Range': `bytes ${start}-${end}/${fileSize}`,
      'Accept-Ranges': 'bytes',
      'Content-Length': chunksize,
      'Content-Type': 'video/mp4',
    };

    res.writeHead(206, head);
    file.pipe(res);
  } else {
    const head = {
      'Content-Length': fileSize,
      'Content-Type': 'video/mp4',
    };

    res.writeHead(200, head);
    fs.createReadStream(path).pipe(res);
  }
});

app.listen(port, () => {
  console.log(`Server listening at ${process.env.SERVER_URL}:${port}`);
});

In the code snippet above, we are implementing a basic video streaming server that responds to HTTP range requests. Here's a brief overview of the key parts:

1. File and Range Setup: We start by determining the path to the video file and getting the file size. We also grab the range header from the request, which contains the range of bytes the client is requesting.
2. Range Requests Handling: If a range is provided, we extract the start and end bytes from the range header, then create a read stream for that specific range. This allows us to stream a portion of the file rather than the entire thing.
3. Response Headers: We then set up our response headers. In the case of a range request, we send back a '206 Partial Content' status along with information about the byte range and total file size. For non-range requests, we simply send back the total file size and the file type.
4. Data Streaming: Finally, we pipe the read stream directly to the response. This step is where the video data actually gets sent back to the client. The use of pipe() here automatically handles backpressure, ensuring that data isn't read faster than it can be sent to the client.

With this setup in place, our streaming server is capable of efficiently delivering large video files to the client in small chunks, providing a smoother user experience.

Implementing the Download API in Express

Now, let's add another endpoint to our Express application, which will provide more granular control over the data transfer process. We'll set up a GET endpoint for '/download', and within this endpoint, we'll handle streaming the video file to the client for download.

app.get('/download', (req: Request, res: Response) => {
  // Again, for testing purposes - add the video in you `static` folder
  const path = 'src/static/pexels-zlatin-georgiev-15708449 (2160p).mp4';
  const stat = fs.statSync(path);
  const fileSize = stat.size;

  res.writeHead(200, {
    'Content-Type': 'video/mp4',
    'Content-Disposition': 'attachment; filename=video.mp4',
    'Content-Length': fileSize,
  });

  const readStream = fs.createReadStream(path);
  const throttle = new Throttle(1024 * 1024 * 5); // throttle to 5MB/sec - simulate lower speed

  readStream.pipe(throttle);

  throttle.on('data', (chunk) => {
    Console.log(`Sent ${chunk.length} bytes to client.`);
    res.write(chunk);
  });

  throttle.on('end', () => {
    console.log('File fully sent to client.');
    res.end();
  });
});

This endpoint has a similar setup to the video streaming endpoint, but it comes with a few key differences:

1. Response Headers: Here, we include a 'Content-Disposition' header with an 'attachment' directive. This header tells the browser to present the file as a downloadable file named 'video.mp4'.
2. Throttling: We use the 'throttle' package to limit the data transfer rate. Throttling can be useful for simulating lower-speed connections during testing, or for preventing your server from getting overwhelmed by data transfer operations.
3. Data Writing: Instead of directly piping the read stream to the response, we attach 'data' and 'end' event listeners to the throttled stream. On the 'data' event, we manually write each chunk of data to the response, and on the 'end' event, we close the response.

This implementation provides a more hands-on way to control the data transfer process. It allows for the addition of custom logic to handle events like pausing and resuming the data transfer, adding custom transformations to the data stream, or handling errors during transfer.

Utilizing the APIs: A React Application

Now that we have a server-side setup for video streaming and downloading, let's put these APIs into action within a client-side React application. Note that we'll be using Tailwind CSS for quick, utility-based styling in our components.

Our React application will consist of a video player that uses the video streaming API, a download button to trigger the download API, and a progress bar to show the real-time download progress.

First, let's define the Video Player component that will play the streamed video:

import React from 'react';

const VideoPlayer: React.FC = () => {
  return (
    <div className="mb-3">
      <video controls width="860">
        <source src="http://localhost:8000/video" type="video/mp4" />
        Your browser does not support the video tag.
      </video>
    </div>
  );
};

export default VideoPlayer;

In the above VideoPlayer component, we're using an HTML5 video tag to handle video playback. The src attribute of the source tag is set to the video endpoint of our Express server. When this component is rendered, it sends a request to our video API and starts streaming the video in response to the range requests that the browser automatically makes.

Next, let's create the DownloadButton component that will handle the video download and display the download progress:

import React, { useState } from 'react';

const DownloadButton: React.FC = () => {
  const [downloadProgress, setDownloadProgress] = useState(0);

  const handleDownload = async () => {
    try {
      const response = await fetch('http://localhost:8000/download');
      const reader = response.body?.getReader();

      if (!reader) {
        return;
      }

      const contentLength = +(response.headers?.get('Content-Length') || 0);
      let receivedLength = 0;
      let chunks = [];

      while (true) {
        const { done, value } = await reader.read();

        if (done) {
          console.log('Download complete.');

          const blob = new Blob(chunks, { type: 'video/mp4' });
          const url = window.URL.createObjectURL(blob);
          const a = document.createElement('a');
          a.style.display = 'none';
          a.href = url;
          a.download = 'video.mp4';
          document.body.appendChild(a);
          a.click();
          window.URL.revokeObjectURL(url);

          setDownloadProgress(100);
          break;
        }
        chunks.push(value);
        receivedLength += value.length;
        const progress = (receivedLength / contentLength) * 100;
        setDownloadProgress(progress);
      }
    } catch (err) {
      console.error(err);
    }
  };

  return (
    <div className="flex gap-3 items-center">
      <button
        onClick={handleDownload}
        className="py-2 px-3 bg-indigo-500 text-white text-sm font-semibold rounded-md shadow focus:outline-none"
      >
        Download Video
      </button>
      {downloadProgress > 0 && downloadProgress < 100 && (
      <p className="flex gap-3 items-center">
        Download progress: <progress value={downloadProgress} max={100} />
      </p>
      )}
      {downloadProgress === 100 && <p>Download complete!</p>}
    </div>
  );
};

export default DownloadButton;

In this DownloadButton component, when the download button is clicked, it sends a fetch request to our download API. It then uses a while loop to continually read chunks of data from the response as they arrive, updating the download progress until the download is complete. This is an example of more controlled handling of multi-response APIs where we are not just directly piping the data, but instead, processing it and manually sending it as a downloadable file.

Bringing It All Together

Let's now integrate these components into our main application component.

import React from 'react';
import VideoPlayer from './components/VideoPlayer';
import DownloadButton from './components/DownloadButton';

function App() {
  return (
    <div className="flex flex-col items-center justify-center min-h-screen bg-gray-100 py-2">
      <h1 className="text-3xl font-bold mb-5">My Video Player</h1>
      <VideoPlayer />
      <DownloadButton />
    </div>
  );
}

export default App;

In this simple App component, we've included our VideoPlayer and DownloadButton components. It places the video player and download button on the screen in a neat, centered layout thanks to Tailwind CSS.

Here is a summary of how our system operates:

• The video player makes a request to our Express server as soon as it is rendered in the React application. Our server handles this request, reading the video file and sending back the appropriate chunks as per the range requested by the browser. This results in the video being streamed in our player.
• When the download button is clicked, a fetch request is sent to our server's download API. This time, the server reads the file, but instead of just piping the data to the response, it controls the data sending process. It sends chunks of data and also logs the sent chunks for monitoring purposes. The React application collects these chunks and concatenates them, displaying the download progress in real-time. When all chunks are received, it compiles them into a Blob and triggers a download in the browser.

This setup allows us to build a full-featured video streaming and downloading application with fine control over the data transmission process. To see this system in action, you can check out this video demo.

Conclusion

While the focus of this article was on video streaming and downloading, the principles we discussed here extend beyond just media files. The pattern of responding to HTTP range requests is common in various data-heavy applications, and understanding it can be a useful tool in your web development arsenal.

Finally, remember that the code shown in this article is just a simple example to demonstrate the concepts. In a real-world application, you would want to add proper error handling, validation, and possibly some form of access control depending on your use case.

I hope this article helps you in your journey as a developer. Building something yourself is the best way to learn, so don't hesitate to get your hands dirty and start coding!