ReactJS

Building a Multi-Response Streaming API with Node.js, Express, and React

Aug 9, 2023

6 min read

Introduction

As web applications become increasingly complex and data-driven, efficient and effective data transfer methods become critically important. A streaming API that can send multiple responses to a single request can be a powerful tool for handling large amounts of data or for delivering real-time updates. In this article, we will guide you through the process of creating such an API.

We will use video streaming as an illustrative example. With their large file sizes and the need for flexible, on-demand delivery, videos present a fitting scenario for showcasing the power of multi-response streaming APIs. The backend will be built with Node.js and Express, utilizing HTTP range requests to facilitate efficient data delivery in chunks.

Next, we'll build a React front-end to interact with our streaming API. This front-end will handle both the display of the streamed video content and its download, offering users real-time progress updates.

By the end of this walkthrough, you will have a working example of a multi-response streaming API, and you will be able to apply the principles learned to a wide array of use cases beyond video streaming.

Let's jump right into it!

Hands-On

Implementing the Streaming API in Express

In this section, we will dive into the server-side implementation, specifically our Node.js and Express application. We'll be implementing an API endpoint to deliver video content in a streaming fashion.

Assuming you have already set up your Express server with TypeScript, we first need to define our video-serving route. We'll create a GET endpoint that, when hit, will stream a video file back to the client.

Please make sure to install cors for handling cross-origin requests, dotenv for loading environment variables, and throttle for controlling the rate of data transfer. You can install these with the following command:

yarn add cors dotenv throttle @types/cors @types/dotenv @types/throttle

import cors from 'cors';
import 'dotenv/config';
import express, { Request, Response } from 'express';
import fs from 'fs';
import Throttle from 'throttle';

const app = express();
const port = 8000;

app.use(cors());

app.get('/video', (req: Request, res: Response) => {
  // Video by Zlatin Georgiev from Pexels: https://www.pexels.com/video/15708449/
  // For testing purposes - add the video in you `static` folder
  const path = 'src/static/pexels-zlatin-georgiev-15708449 (2160p).mp4';
  const stat = fs.statSync(path);
  const fileSize = stat.size;
  const range = req.headers.range;
  if (range) {
    const parts = range.replace(/bytes=/, '').split('-');
    const start = parseInt(parts[0], 10);
    const end = parts[1] ? parseInt(parts[1], 10) : fileSize - 1;

    const chunksize = end - start + 1;
    const file = fs.createReadStream(path, { start, end });
    const head = {
      'Content-Range': `bytes ${start}-${end}/${fileSize}`,
      'Accept-Ranges': 'bytes',
      'Content-Length': chunksize,
      'Content-Type': 'video/mp4',
    };

    res.writeHead(206, head);
    file.pipe(res);
  } else {
    const head = {
      'Content-Length': fileSize,
      'Content-Type': 'video/mp4',
    };

    res.writeHead(200, head);
    fs.createReadStream(path).pipe(res);
  }
});

app.listen(port, () => {
  console.log(`Server listening at ${process.env.SERVER_URL}:${port}`);
});

In the code snippet above, we are implementing a basic video streaming server that responds to HTTP range requests. Here's a brief overview of the key parts:

File and Range Setup: We start by determining the path to the video file and getting the file size. We also grab the range header from the request, which contains the range of bytes the client is requesting.
Range Requests Handling: If a range is provided, we extract the start and end bytes from the range header, then create a read stream for that specific range. This allows us to stream a portion of the file rather than the entire thing.
Response Headers: We then set up our response headers. In the case of a range request, we send back a '206 Partial Content' status along with information about the byte range and total file size. For non-range requests, we simply send back the total file size and the file type.
Data Streaming: Finally, we pipe the read stream directly to the response. This step is where the video data actually gets sent back to the client. The use of pipe() here automatically handles backpressure, ensuring that data isn't read faster than it can be sent to the client.

With this setup in place, our streaming server is capable of efficiently delivering large video files to the client in small chunks, providing a smoother user experience.

Implementing the Download API in Express

Now, let's add another endpoint to our Express application, which will provide more granular control over the data transfer process. We'll set up a GET endpoint for '/download', and within this endpoint, we'll handle streaming the video file to the client for download.

app.get('/download', (req: Request, res: Response) => {
  // Again, for testing purposes - add the video in you `static` folder
  const path = 'src/static/pexels-zlatin-georgiev-15708449 (2160p).mp4';
  const stat = fs.statSync(path);
  const fileSize = stat.size;

  res.writeHead(200, {
    'Content-Type': 'video/mp4',
    'Content-Disposition': 'attachment; filename=video.mp4',
    'Content-Length': fileSize,
  });

  const readStream = fs.createReadStream(path);
  const throttle = new Throttle(1024 * 1024 * 5); // throttle to 5MB/sec - simulate lower speed

  readStream.pipe(throttle);

  throttle.on('data', (chunk) => {
    Console.log(`Sent ${chunk.length} bytes to client.`);
    res.write(chunk);
  });

  throttle.on('end', () => {
    console.log('File fully sent to client.');
    res.end();
  });
});

This endpoint has a similar setup to the video streaming endpoint, but it comes with a few key differences:

Response Headers: Here, we include a 'Content-Disposition' header with an 'attachment' directive. This header tells the browser to present the file as a downloadable file named 'video.mp4'.
Throttling: We use the 'throttle' package to limit the data transfer rate. Throttling can be useful for simulating lower-speed connections during testing, or for preventing your server from getting overwhelmed by data transfer operations.
Data Writing: Instead of directly piping the read stream to the response, we attach 'data' and 'end' event listeners to the throttled stream. On the 'data' event, we manually write each chunk of data to the response, and on the 'end' event, we close the response.

This implementation provides a more hands-on way to control the data transfer process. It allows for the addition of custom logic to handle events like pausing and resuming the data transfer, adding custom transformations to the data stream, or handling errors during transfer.

Utilizing the APIs: A React Application

Now that we have a server-side setup for video streaming and downloading, let's put these APIs into action within a client-side React application. Note that we'll be using Tailwind CSS for quick, utility-based styling in our components.

Our React application will consist of a video player that uses the video streaming API, a download button to trigger the download API, and a progress bar to show the real-time download progress.

First, let's define the Video Player component that will play the streamed video:

import React from 'react';

const VideoPlayer: React.FC = () => {
  return (
    <div className="mb-3">
      <video controls width="860">
        <source src="http://localhost:8000/video" type="video/mp4" />
        Your browser does not support the video tag.
      </video>
    </div>
  );
};

export default VideoPlayer;

In the above VideoPlayer component, we're using an HTML5 video tag to handle video playback. The src attribute of the source tag is set to the video endpoint of our Express server. When this component is rendered, it sends a request to our video API and starts streaming the video in response to the range requests that the browser automatically makes.

Next, let's create the DownloadButton component that will handle the video download and display the download progress:

import React, { useState } from 'react';

const DownloadButton: React.FC = () => {
  const [downloadProgress, setDownloadProgress] = useState(0);

  const handleDownload = async () => {
    try {
      const response = await fetch('http://localhost:8000/download');
      const reader = response.body?.getReader();

      if (!reader) {
        return;
      }

      const contentLength = +(response.headers?.get('Content-Length') || 0);
      let receivedLength = 0;
      let chunks = [];

      while (true) {
        const { done, value } = await reader.read();

        if (done) {
          console.log('Download complete.');

          const blob = new Blob(chunks, { type: 'video/mp4' });
          const url = window.URL.createObjectURL(blob);
          const a = document.createElement('a');
          a.style.display = 'none';
          a.href = url;
          a.download = 'video.mp4';
          document.body.appendChild(a);
          a.click();
          window.URL.revokeObjectURL(url);

          setDownloadProgress(100);
          break;
        }
        chunks.push(value);
        receivedLength += value.length;
        const progress = (receivedLength / contentLength) * 100;
        setDownloadProgress(progress);
      }
    } catch (err) {
      console.error(err);
    }
  };

  return (
    <div className="flex gap-3 items-center">
      <button
        onClick={handleDownload}
        className="py-2 px-3 bg-indigo-500 text-white text-sm font-semibold rounded-md shadow focus:outline-none"
      >
        Download Video
      </button>
      {downloadProgress > 0 && downloadProgress < 100 && (
      <p className="flex gap-3 items-center">
        Download progress: <progress value={downloadProgress} max={100} />
      </p>
      )}
      {downloadProgress === 100 && <p>Download complete!</p>}
    </div>
  );
};

export default DownloadButton;

In this DownloadButton component, when the download button is clicked, it sends a fetch request to our download API. It then uses a while loop to continually read chunks of data from the response as they arrive, updating the download progress until the download is complete. This is an example of more controlled handling of multi-response APIs where we are not just directly piping the data, but instead, processing it and manually sending it as a downloadable file.

Bringing It All Together

Let's now integrate these components into our main application component.

import React from 'react';
import VideoPlayer from './components/VideoPlayer';
import DownloadButton from './components/DownloadButton';

function App() {
  return (
    <div className="flex flex-col items-center justify-center min-h-screen bg-gray-100 py-2">
      <h1 className="text-3xl font-bold mb-5">My Video Player</h1>
      <VideoPlayer />
      <DownloadButton />
    </div>
  );
}

export default App;

In this simple App component, we've included our VideoPlayer and DownloadButton components. It places the video player and download button on the screen in a neat, centered layout thanks to Tailwind CSS.

Here is a summary of how our system operates:

The video player makes a request to our Express server as soon as it is rendered in the React application. Our server handles this request, reading the video file and sending back the appropriate chunks as per the range requested by the browser. This results in the video being streamed in our player.
When the download button is clicked, a fetch request is sent to our server's download API. This time, the server reads the file, but instead of just piping the data to the response, it controls the data sending process. It sends chunks of data and also logs the sent chunks for monitoring purposes. The React application collects these chunks and concatenates them, displaying the download progress in real-time. When all chunks are received, it compiles them into a Blob and triggers a download in the browser.

This setup allows us to build a full-featured video streaming and downloading application with fine control over the data transmission process. To see this system in action, you can check out this video demo.

Conclusion

While the focus of this article was on video streaming and downloading, the principles we discussed here extend beyond just media files. The pattern of responding to HTTP range requests is common in various data-heavy applications, and understanding it can be a useful tool in your web development arsenal.

Finally, remember that the code shown in this article is just a simple example to demonstrate the concepts. In a real-world application, you would want to add proper error handling, validation, and possibly some form of access control depending on your use case.

I hope this article helps you in your journey as a developer. Building something yourself is the best way to learn, so don't hesitate to get your hands dirty and start coding!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

William Mimura

@mimurawil

How to test React custom hooks and components with Vitest

Introduction In this guide, we'll navigate through the process of testing React hooks and components using Vitest—a powerful JavaScript unit testing framework. Discover how Vitest simplifies testing setups, providing an optimal solution for both Vite-powered projects and beyond. Vitest is a javascript unit testing framework that aims to position itself as the Test Runner of choice for Vite projects and as a solid alternative even for projects not using Vite. Vitest was built primarily for Vite-powered projects, to help reduce the complexity of setting up testing with other testing frameworks like Jest. Vitest uses the same configuration of your App (through vite.config.js), sharing a common transformation pipeline during dev, build, and test time. Prerequisites This article assumes a solid understanding of React and frontend unit testing. Familiarity with tools like React Testing Library and JSDOM will enhance your grasp of the testing process with Vitest. Installation and configuration Let’s see how we can use Vitest for testing React custom hooks and components. But first, we will need to create a new project with Vite! If you already have an existing project, you can skip this step. ` Follow the prompts to create a new React project successfully. For testing, we need the following dependencies installed: Vitest as the unit testing framework JSDOM as the DOM environment for running our tests React Testing Library as the React testing utilities. To do so, we run the following command: ` Once we have those packages installed, we need to configure the vite.config.js file to run tests. By default, some of the extra configs we need to set up Vitest are not available in the Vite config types, so we will need the vite.config.ts file to reference Vitest types by adding /// reference types=”vitest” /> at the top of the file. Add the following code to the vite.config.ts ` We set globals to true because, by default, Vitest does not provide global APIs for explicitness. So with this set to true, we can use keywords like describe, test and it without needing to import them. To get TypeScript working with the global APIs, add vitest/globals to the types field in your tsconfig.json. ` The environment property tells Vitest which environment to run the test. We are using jsdom as the environment. The root property tells Vitest the root folder from where it should start looking for test files. We should add a script for running the test in package.json ` With all that configured, we can now start writing unit tests for customs hooks and React components. Writing test for custom hooks Let’s write a test for a simple useCounter hook that takes an initial value and returns the value, an increment function and a decrement function. ` We can write a test to check the default return values of the hook for value as below: ` To test if the hook works when we increment the value, we can use the act() method from @testing-library/react to simulate the increment function, as shown in the below test case: ` Kindly Note that you can't destructure the reactive properties of the result.current instance, or they will lose their reactivity. Testing hooks with asynchronous logic Now let’s test a more complex logic that contains asynchronous logic. Let’s write a useProducts hook that fetches data from an external api and return that value ` Now, let’s see what the test looks like: ` In the above example, we had to spy on the global fetch API, so that we can mock its return value. We wrapped that inside a beforeAll so that this runs before any test in this file. Then we added an afterAll method and called the mockRestore() to run after all test cases have been completed and return all mock implementations to their original function. We can also use the mockClear() method to clear all the mock's information, such as the number of calls and the mock's results. This method is handy when mocking the same function with different return values for different tests. We usually use mockClear() in beforeEach() or afterEach() methods to ensure our test is isolated completely. Then in our test case, we used a waitFor(), to wait for the return value to be resolved. Writing test for components Like Jest, Vitest provides assertion methods (matchers) to use with the expect methods for asserting values, but to test DOM elements easily, we will need to make use of custom matchers such as toBeInTheDocument() or toHaveTextContent(). Luckily the Vitest API is mostly compatible with the Jest API, making it possible to reuse many tools originally built for Jest. For such methods, we can install the @testing-library/jest-dom package and extend the expect method from Vitest to include the assertion methods in matchers from this package. ` After installing the jest-dom testing library package, create a file named vitest-setup.ts on the root of the project and import the following into the project to extend js-dom custom matchers: ` Since we are using typescript, we also need to include our setup file in our tsconfig.json: ` In vite.config.ts, we need to add the vitest-setup.ts file to the test.setupFiles field: ` Now let’s test the Products.tsx component: ` We start by spying and mocking the useProducts hook with vi.spyOn() method from Vitest: ` Now, we render the Products component using the render method from @testing-library/react and assert that the component renders the list of products as expected and also the product has the title as follows: ` In the above code, we use the render method from @testing-library/react to render the component and this returns some useful methods we can use to extract information from the component like getByTestId and getByText. The getByTestId method will retrieve the element whose data-testid attribute value equals product-list, and we can then assert its children to equal the length of our mocked items array. Using data-testid attribute values is a good practice for identifying a DOM element for testing purposes and avoiding affecting the component's implementation in production and tests. We also used the getByText method to find a text in the rendered component. We were able to call the toBeInTheDocument() because we extended the matchers to work with Vitest earlier. Here is what the full test looks like: ` Conclusion In this article, we delved into the world of testing React hooks and components using Vitest, a versatile JavaScript unit testing framework. We walked through the installation and configuration process, ensuring compatibility with React, JSDOM, and React Testing Library. The comprehensive guide covered writing tests for custom hooks, including handling asynchronous logic, and testing React components, leveraging custom matchers for DOM assertions. By adopting Vitest, developers can streamline the testing process for their React applications, whether powered by Vite or not. The framework's seamless integration with Vite projects simplifies the setup, reducing the complexities associated with other testing tools like Jest....

Mar 15, 2024

5 mins

ViteReactTesting

CSS Hooks: A new way to style your React apps

In the world of web development, the management of styles has always been a crucial aspect of building modern and responsive user interfaces. With the rise of CSS in JS libraries like Material UI and Chakra, developers have started creating dynamic and reusable styles using JavaScript; however, the performance implications of these libraries have led to the exploration of alternative solutions. One such solution is CSS Hooks, a library that offers a different approach to managing styles in web applications. The Problem with CSS in React and CSS in JS Libraries In React applications, you typically write styles directly with the style prop by applying classes with className, or if you’re using a framework like Material UI or Chakra then you may use the sx attribute, which is more powerful than the baseline attributes. There are some performance concerns in the case of the sx attribute specifically. The framework has to dynamically generate classes with your styles and apply them to the document whenever it is used. This can become an issue with large and complex applications as your styles will be regularly regenerated on the fly as the user navigates through the application. This has led developers to seek alternative solutions that offer similar flexibility but with better performance. Introduction to CSS Hooks CSS Hooks improve upon the aforementioned issues by pre-generating your CSS using configuration hooks. The styles that these hooks generate remain static during the lifetime of the application and are reused wherever possible. CSS variables are utilized under the hood whenever dynamic behavior is needed. By dynamic behavior, I’m referring to advanced CSS features such as pseudo-class selectors and media queries, which are both traditionally unavailable for use with inline styles. CSS Hooks utilize an interesting trick under the hood to accomplish this, which we will now explore. The Power of CSS Variables and the Fallback Trick As mentioned before, CSS variables are utilized in order to support using advanced dynamic behavior such as pseudo-selectors in our inline styles. CSS variables are core to something known as the “fallback trick”. The gist of how the fallback trick works is to have the pseudo-selector or media query toggle the state of a variable between initial and an empty invalid value, and then we put the value that we actually want to toggle as the “fallback” value in the reference to the CSS variable inside of the inline style itself. Changing this fallback value inside of an inline style essentially allows us to control the values used by the pseudo-selector without having to regenerate linked stylesheets as we only have to change the inline style attribute. This is best explained with an example: ` This plain HTML and CSS code effectively allows us to utilize the focus pseudo-selector on any element that we want using inline styles only. This is what CSS Hooks effectively does in the background when you use dynamic styles. No nasty re-renders and mucking with stylesheets needed! How to use CSS Hooks Now that we know how CSS Hooks fundamentally work, let’s try using them. The library itself is much easier to use than the aforementioned example. Installation and usage of CSS Hooks is easy. Firstly, we must install the @css-hooks/react package into a React project with the package manager of your choice (npm, pnpm yarn, etc). Once that’s done, all that’s left is a little bit of configuration. You have to first use a utility function called createHooks that is exported from the package we just installed. The result of that function returns a couple of variables in an array that can be deconstructed, the first being a stylesheet in the form of a string, and the second being a function. ` I recommend putting this in a separate file that can be imported from multiple other places in the project as this is where we will be configuring hooks across the project. We’re exporting both hooks and css, and both are important. We need to include hooks into the DOM as it contains the styles of the hooks that we’ll be referencing in our components. In your root component you need to add the following tag so that the generated CSS can be used. ` How this is done may vary based on the libraries that you are using. After that’s done, you can integrate CSS Hooks into your components while writing inline styles mostly just like you would do normally, with the only change being the addition of an additional call to the css function that we exported earlier: ` Now that the core boilerplate is set up, your components can now utilize CSS Hooks in theory; however we need to actually add some hooks to start with before we can use them, or else we’ll just be limited to basic styles in the same way that React inline styles are. Make your own hooks We can define our hooks inside of the css-hooks file that we created earlier, the same one that exports the css helper function that gets CSS Hooks working with our components. Hooks are defined in the options parameters that are currently empty. Defining a simple hook that allows us to define inline hover styles for a component is very easy and can be done as follows: ` Although both the key and the value in this example are the same, they both serve different purposes. The key is the name of the hook that we reference when we want to use it, while the value is the spec. The spec is the actual CSS selector that we’re writing. The name can be anything that you want it to be so long as it doesn’t clash with any existing style properties or hooks. Recommended hooks Although making your own hooks may be necessary at times, there is another repository provided by the CSS Hooks authors that adds a way to generate many useful hooks called @css-hooks/recommended. I highly recommend using this library to generate your hooks if at all possible, and the usage is pretty simple. Here’s an example showing how you can use it to generate media queries, color scheme specific styles and pseudo-selectors: ` Then with that done, these can be utilized as follows: ` We can also modify the hooks configuration to add our own hooks in addition to the recommended hooks by using the spread operator: ` Then that custom hook can be used by simply using its key name: ` Summary CSS Hooks offers a performant alternative to traditional CSS in JS libraries by leveraging CSS variables and providing a unique approach to managing styles in web applications. By pre-generating the stylesheet based on configured hooks and utilizing CSS Variables for dynamic behavior, developers can create reusable styles without the performance overhead associated with dynamic stylesheet generation. If you're interested in exploring CSS Hooks further, be sure to check out @CSSHooks for more information. You can also find some of the examples demonstrated here in a working app in our demos repository. Thank you for reading!...

Feb 28, 2024

5 mins

CSSReact

TypeScript Integration with .env Variables

Introduction In TypeScript projects, effectively managing environment variables can significantly enhance the development experience. However, a common hurdle is that TypeScript, by default, doesn't recognize variables defined in .env files. This oversight can lead to type safety issues and, potentially, hard-to-trace bugs. In this blog post, we'll walk you through the process of setting up an environment.d.ts file. This simple yet powerful addition enables TypeScript to seamlessly integrate with and accurately interpret your environment variables. Let's dive into the details! Creating and Configuring environment.d.ts Install @types/node Before creating your environment.d.ts file, make sure you have the @types/node package installed as it provides TypeScript definitions for Node.js, including the process.env object. Install it as a development dependency: ` Setting Up environment.d.ts To ensure TypeScript correctly recognizes and types your .env variables, start by setting up an environment.d.ts file in your project. This TypeScript declaration file will explicitly define the types of your environment variables. 1. Create the File: In the root of your TypeScript project, create a file named environment.d.ts 2. Declare Environment Variables: In environment.d.ts, declare your environment variables as part of the NodeJS.ProcessEnv interface. For example, for API_KEY and DATABASE_URL, the file would look like this: ` 3. Typescript config: In you tsconfig.json file, ensure that Typescript will recognize our the new file: ` 4. Usage in Your Project: With these declarations, TypeScript will provide type-checking and intellisense for process.env.API_KEY and process.env.DATABASE_URL, enhancing the development experience and code safety. Checking on Your IDE By following the steps above, you can now verify on your IDE how your environment variables recognizes and auto completes the variables added: Conclusion Integrating .env environment variables into TypeScript projects enhances not only the security and flexibility of your application but also the overall developer experience. By setting up an environment.d.ts file and ensuring the presence of @types/node, you bridge the gap between TypeScript’s static type system and the dynamic nature of environment variables. This approach leads to clearer, more maintainable code, where the risks of runtime errors are significantly reduced. It's a testament to TypeScript's versatility and its capability to adapt to various development needs. As you continue to build and scale your TypeScript applications, remember that small enhancements like these can have a profound impact on your project's robustness and the efficiency of your development processes. Embrace these nuanced techniques, and watch as they bring a new level of precision and reliability to your TypeScript projects....

Mar 13, 2024

2 mins

TypeScript

Intro to EdgeDB - The 10x ORM

Intro to EdgeDB - The 10x ORM I’ve written a couple of posts recently covering different TypeScript ORMs. One about Prisma, and another about Drizzle. ORM’s are a controversial topic in their own right - some people think they are evil, and others think they are great. I enjoy them quite a bit. They make it easy to interact with your databases. What is more important and magical for an application than data? SQL without an ORM is amazing as well, but there are some pain points with that approach. Today I’m excited to write about EdgeDB, which isn’t _exactly_ an ORM or a database from my perspective (although they call themselves one). It is, however an incredibly impressive piece of technology that solves these common pain points in a pretty novel way. So if it’s not an ORM or a database, what exactly is it? I don’t think I can answer that in one or two sentences, so we will explore the various pieces that make up EdgeDB in this article. From a high-level standpoint, though, an interface/query language that sits in front of PostgreSQL. This may seem like some less important implementation detail, but in my eyes, it’s a feature and one of the most compelling selling points. Data Modeling EdgeDB advertises itself as a “graph-relational database”. The core components of EdgeDB data modeling are the schema, type system, and relationship definitions. A schema will consist of objects that contain various typed attributes and links that connect the objects. In SQL, a table is analogous to an Object, and a foreign-key is associated with a link. Here’s what a simple schema in EdgeDB looks like ` There’s a few things to highlight here * We defined two different objects (tables) User and Post * Each object contains properties with their types defined * str is one of several scalar types (bool, int, float, json, datetime, etc) * the author property is a required link to the User object Defining relations / associations In our example above we defined a one-to-many relationship between a user and posts. All the relation types that you can define in traditional SQL are available. One interesting feature though is called backward links. These can be defined in your schema and it allows you to access related data from both sides of a relationship. ` likes are a many-to-many relationship between Tweet and User. With a backlink defined multi link likers := Tweet.<likes[is User]; - we can access likes from a User and likers from a Tweet. ` That's how we can access these relations in our queries. You might be looking at these queries and thinking it looks a lot like GraphQL. This is why they call it a ‘Graph-relational’ database. We’ve only scratched the surface of EdgeDB schema’s. Hopefully, I’ve at least managed to pique your interest. Computed properties Computed properties are a super powerful feature that can be added to your schema or queries. This example user schema creates a computed discriminator and username property. The discriminator uses an EdgeDB standard library function to generate a discriminator number and the username property is a combination between the name and discriminator properties. ` Globals and Access Policies EdgeDB allows you to define global variables as part of your schema. The most common use case I’ve seen for this is to power the access policy feature. You can define a global variable as part of your schema: global current_user: uuid; With a global variable defined, you can provide the value as a sort of context from your application by providing it into your EdgeDB driver/client. ` You can then add access policies directly to your schema, for example, to provide fine-grained access control to blog posts in your blogging application. ` Aside from access policies, you can use your global variables in your queries as well. For example, if you wanted a query to select the current user. ` Types and Sets EdgeDB is very type-centric. All of your data is strongly typed. We’ve touched on some of the types already. * Scalars - There are a lot of scalar types available out of the box * Custom scalars - Custom scalar types are user-defined extensions of existing types * Enums - Are supported out of the box - enum<Admin, Moderator, Member> * Arrays - Defined by passing the singular value type - array<str>; * Tuples - In EdgeD, tuples can contain more than 2 elements and come in named and unnamed varieties - <str, number>; tuple<name: str, jersey_number: float64, active: bool>; All queries return a Set which is a collection of values of a given type. In the query language, all values are Sets. Sets are a collection of values of a given type. A comma-separated list of values inside a set of {curly braces}. A query with no results will return an empty or singleton set. If we have no User values stored yet - select User returns {}. Paired with the query language types and set provides an incredibly powerful and expressive system for interacting with your data. If you thought TypeScript was cool, wait until you start writing EdgeQL! 🙂 EdgeQL Now to the fun stuff: the query language. We’ll use a schema from the docs and start by looking at some of those queries and build on those. The example schema has an abstract type Person with two sub-types based on it Hero and Villian. This is known as a polymorphic type in EdgeDB. The Movie type includes a 1:m association with Person ` Selecting properties / data Before we dig into some real queries we should just touch on how we select actual data from a query. It’s pretty obvious and GraphQL-like but worth mentioning. To specify properties to select, you attach a shape. This works for getting nested link / association data as well. Based on our schema, here’s how we could select fields from Movie, including data from the collection of related characters. ` There is also a feature called a splats that allows you to select all fields and/or all linked fields without specifying them individually. ` If you don’t specify any properties or splats, only id’s get returned select Movie; . Adding some objects with insert To get started, we can use insert to add objects to our database. We’ll start big by looking at the nested insert example. This example is interesting because it shows the creation of two objects in a single query. You’ll notice the simplicity of the syntax. Even though this is the first EdgeQL query we’re looking at, in my experience, it’s like this across the board. I’ve found EdgeQL queries to be simple and intuitive to the point where I’ve been able to intuit how to accomplish things in my head without having to reference the docs or ask the AI. This example adds a new Villian and a new Hero which gets assigned as a link or association to the nemesis field on our Villian. To accomplish this we see that we can nest queries by wrapping them in (). ` The next example is pretty similar, but instead of creating the linked property, we are select ing and adding several potential objects to the characters list of Movie since it is a multi link. This is a pretty complex query that is doing a lot of different things. It’s deceivingly succinct. To accomplish the same thing with SQL this would probably be about 3 different queries. This query finds the objects to add to the characters multi link by filtering on a collection of different strings to match against the name property. ` The last thing we’ll cover for insert is bulk inserts. This is particularly useful for things like seed scripts. In this example, you can just imagine that you have a JSON array of objects with hero names that gets passed in as an argument to your query ` Querying data with select We’ve already seen subqueries and a select in the last section where we found a collection of Person records with a filter. We’ll build on that and see what tools are available to us when it comes to querying data. This one covers a lot of ground. Very similar to SQL we have order and limit, and offset operators to support sorting and pagination. Also there is a whole standard library of functions and operators like count that can be used in our queries. This example returns a collection of villian names, excluding the first and last result. ` Most commonly, you will want to filter by an id ` Here’s another common example filtering by datetime. Since we’re using a string value here we need to cast it to the EdgeDB datetime type. ` You get a pretty similar toolbox to SQL when it comes to filtering with your common operators and things. Combined with all the tools in the standard library, you can get pretty creative with it. Changing values and links with update The update..filter..set statement is how we can update existing data with EdgeQL. set is followed by a shape with assignments of properties to be updated. ` You can replace links for an object ` or add additional ones ` An even more interesting example is removing links matched on a type. Since Villian is a sub-type of Person , this query will remove all characters linked of the Villian type. ` Deleting objects with delete Deleting is pretty straight forward. Using the delete command you can just filter for the objects that you would like to remove. ` When the EdgeQL pieces fall into place As you become more familiar with the EdgeQL query language chances are you’ll start writing very complex queries fluently because everything just makes sense once you’ve learned the building blocks. Domain and business concerns I don’t think they explicitly mention this as a goal anywhere but it’s something that I picked up on pretty quickly. EdgeDB nudges you to move more of what might have traditionally been application logic into your database layer. This is a topic that can bring a lot of division since even things like foreign keys and constraints in SQL are frowned upon in some circles. EdgeDB goes as far as providing constraints, global variables, contexts, and authorization support built into the database. I think that the ability to bake some of these concerns into your EdgeDB Schema is great. The way you model your schema and database in EdgeDB map to your domain in a much more intuitive way where domain concerns don’t really feel out of place there. Database Clients and Query Builders and Generators We’ve covered a lot so far to highlight what EdgeDB is and how to handle common use cases with the query language. To use it in your project though, you will need a client/driver library. There are clients available in several different languages. The one that they clearly have put the most investment into is the TypeScript query builder. We’ll briefly look at both options: simple driver/client and query builder. Whichever you end up choosing you will need to instantiate a driver and make sure you have a connection to your database instance configured. Basic client Although the TS query builder is very popular and pretty amazing, I couldn’t get away from just writing EdgeQL queries. In my application, I composed queries using template strings, and it worked great. The clients all have a set of methods available for passing in EdgeQL queries and parameters. querySingle is a method for queries where you are only expecting a single result. If your query will have multiple results you would use query instead. There is also a queryRequiredSingle which will throw an error if no results are found. There are some other methods available as well including one for running queries in a transaction ` The first argument is the query, and the second is a map of parameters. In this example we include the title parameter and it is accessed in our query via $title. TypeScript query builder If you have a TypeScript app and type-safety is important, you might prefer using the query builder. It is a pretty incredible feat of TypeScript magic initially developed by the same developer behind the popular library Zod. We can’t cover it in very much depth here but we’ll look at an example just to have an idea of what the query builder looks like in an application. ` The query builder is able to infer the result type automatically. It knows which fields you’ve selected, it knows that the result will be a single item. Query generator There are generators for queries and types. So even if you opt out of using the query builder you can still have queries that are strongly typed. It’s nice to have this option if you want to just write your queries as EdgeQL in .edgeql files. ` We end up with an exported function named getUser that is strongly typed. ` Tools and Utilities The team at EdgeDB puts a big emphasis on developer experience. It shows up all over the place. We’ve already seen some utilities with the generators that are available. There are some other tools available as well that help complete the entire experience. EdgeDB CLI The first and most important tool to mention is the CLI. If you’ve started using EdgeDB then you’ve most likely already installed and used it. The CLI is pretty extensive. It includes commands for things like migrations, managing EdgeDB versions and installations, managing projects and local/cloud database instances, dumps and restores, a repl, and more. The CLI makes managing EdgeDB a breeze. Admin UI The CLI includes a command to launch an admin UI for any project or database. The Admin UI includes a awesome interactive diagram of your database schema, a repl for running queries, and a table to inspect and make changes to the data stored in your database. Summary Adopting newer database technology is a tough sales pitch. Replacing your application’s database technology at any point in its lifecycle is not a problem that anyone wants to have. This is one of the reasons why EdgeDB being a superset of PostgreSQL is a huge feature in my opinion. The underlying database technology is tried and true, and EdgeDB is open-source. Based on this, I would feel confident using EdgeDB if it aligned well from a technical and business perspective. We’ve covered a lot of ground in this post. EdgeDB is feature-packed and powerful. Databases is a tough nut to crack, and I commend the team for all their hard work to help continue pushing forward one of the most important components of almost any application. I’m typically pretty conservative when it comes to databases, but EdgeDB took a great approach, in my opinion. I recommend at least giving it a try. You might catch the EdgeDB bug like I did!...

Apr 26, 2024

11 mins

JavaScriptEdgeDB

Building a Multi-Response Streaming API with Node.js, Express, and React

Introduction

Hands-On

Implementing the Streaming API in Express

Implementing the Download API in Express

Utilizing the APIs: A React Application

Bringing It All Together

Conclusion

William Mimura

You might also like

How to test React custom hooks and components with Vitest

CSS Hooks: A new way to style your React apps

TypeScript Integration with .env Variables

Intro to EdgeDB - The 10x ORM

You might also like

How to test React custom hooks and components with Vitest

CSS Hooks: A new way to style your React apps

TypeScript Integration with .env Variables

Intro to EdgeDB - The 10x ORM