Skip to content

JavaScript Marathon: Intro to GraphQL DataLoaders

As a part of JavaScript Marathon week, Star Richardson, Software Engineer at This Dot Labs, presented a session on GraphQL DataLoaders. She discussed the following topics:

What are GraphQL DataLoaders?

A dataloader is a generic utility used as part of your application's data fetching layer to provide a simplified and consistent API over various remote data sources, such as databases or web services, via batching and caching.

In sum, it is a very small library that provides batching and caching around your APIs.

DataLoaders can be used on any JavaScript project.

Why use DataLoaders?

DataLoaders can be used on any JavaScript project, but they work especially well with GraphQL because GraphQL has an “N + 1” problem, and DataLoaders are here to solve it!

What is N + 1 problem?

The problem is that in practice, you wouldn't just have one join. Your review data model might include associations with any number of other models, each one requiring its own join clause. Additionally, those models might have relationships to other models themselves. Trying to craft a single SQL query that will account for all possible GraphQL queries becomes not only difficult but also prohibitively expensive. A client might request only the reviews with none of their associated models, but the query to fetch those reviews now include 30 additional, unnecessary views. That query might have taken less than a second, but now takes 10.

You can read more about this problem here.

Let’s start with

First, install DataLoaders:

npm install --save dataloader

Second, Create DataLoader instance.

Finally, create a batching function.

Create your first DataLoader

First, create a new file and import the DataLoader library.

const DataLoader = require(‘dataloader’);

Then, import the model you want. For example:

const { Characters } = require(../../models’)

Next, create the DataLoader instance for this model.

const characterLoader = new DataLoader(getBatchCharacters);

You may be wondering: what is the getBatchCharacters? Well it’s the batching function that we discussed before. Let’s create it!

Batching Functions

Batching functions for GraphQL Dataloaders need to follow the next specific structure in order to work with DataLoaders!

DataLoaders expect your batching function to accept an array of keys and return a promise that resolves to an array of values or errors. It’s very important that the array or the promise that it resolves to an array that you're returning is the same length and in the same order of the keys that you are receiving.

So our batching function will be for example like this:

const getBatchCharacters = async (keys) => {
  const characters = await Character.findAll({
    // Add your query here
  });
};

And then export the dataloader:

module.exports = characterLoader;

If you missed the JavaScript marathon session on DataLoaders, you can check out the video here!

This Dot Labs is a development consultancy that is trusted by top industry companies, including Stripe, Xero, Wikimedia, Docusign, and Twilio. This Dot takes a hands-on approach by providing tailored development strategies to help you approach your most pressing challenges with clarity and confidence. Whether it's bridging the gap between business and technology or modernizing legacy systems, you’ll find a breadth of experience and knowledge you need. Check out how This Dot Labs can empower your tech journey.

You might also like

How to Resolve Nested Queries in Apollo Server cover image

How to Resolve Nested Queries in Apollo Server

When working with relational data, there will be times when you will need to access information within nested queries. But how would this work within the context of Apollo Server? In this article, we will take a look at a few code examples that explore different solutions on how to resolve nested queries in Apollo Server. I have included all code examples in CodeSandbox if you are interested in trying them out on your own. Prerequisites** This article assumes that you have a basic knowledge of GraphQL terminology. Table of Contents - How to resolve nested queries: An approach using resolvers and the filter method - A refactored approach using Data Loaders and Data Sources - What are Data Loaders - How to setup a Data Source - Setting up our schemas and resolvers - Resolving nested queries when microservices are involved - Conclusion How to resolve nested queries: An approach using resolvers and the filter method In this first example, we are going to be working with two data structures called musicBrands` and `musicAccessories`. `musicBrands` is a collection of entities consisting of id and name. `musicAccessories` is a collection of entities consisting of the product name, price, id and an associated `brandId`. You can think of the `brandId` as a foreign key that connects the two database tables. We also need to set up the schemas for the brands and accessories. `graphql const typeDefs = gql scalar USCurrency type MusicBrand { id: ID! brandName: String } type MusicAccessories { id: ID! product: String price: USCurrency brandId: Int brand: MusicBrand } type Query { accessories: [MusicAccessories] } ; ` The next step is to set up a resolver for our Query` to return all of the music accessories. `js const resolvers = { Query: { accessories: () => musicAccessories, }, }; ` When we run the following query and start the server, we will see this JSON output: `graphql query Query { accessories { product brand { brandName } } } ` `json { "data": { "accessories": [ { "product": "NS Micro Violin Tuner Standard", "brands": null }, { "product": "Standard Gong Stand", "brands": null }, { "product": "Black Cymbal Mallets", "brands": null }, { "product": "Classic Series XLR Microphone Cable", "brands": null }, { "product": "Folding 5-Guitar Stand Standard", "brands": null }, { "product": "Black Deluxe Drum Rug", "brands": null } ] } } ` As you can see, we are getting back the value of null` for the `brands` field. This is because we haven't set up that relationship yet in the resolvers. Inside our resolver, we are going to create another query for the MusicAccessories` and have the value for the `brands` key be a filtered array of results for each brand. `js const resolvers = { Query: { accessories: () => musicAccessories, }, MusicAccessories: { // parent represents each music accessory brand: (parent) => { const isBrandInAccessory = (brand) => brand.id === parent.brandId; return musicBrands.find(isBrandInAccessory); }, }, }; ` When we run the query, this will be the final result: `graphql query Query { accessories { product brand { brandName } } } ` `json { "data": { "accessories": [ { "product": "NS Micro Violin Tuner Standard", "brands": [ { "brandName": "D'Addario" } ] }, { "product": "Standard Gong Stand", "brands": [ { "brandName": "Zildjian" } ] }, { "product": "Black Cymbal Mallets", "brands": [ { "brandName": "Zildjian" } ] }, { "product": "Classic Series XLR Microphone Cable", "brands": [ { "brandName": "D'Addario" } ] }, { "product": "Folding 5-Guitar Stand Standard", "brands": [ { "brandName": "Fender" } ] }, { "product": "Black Deluxe Drum Rug", "brands": [ { "brandName": "Zildjian" } ] } ] } } ` This single query makes it easy to access the data we need on the client side as compared to the REST API approach. If this were a REST API, then we would be dealing with multiple API calls and a Promise.all` which could get a little messy. You can find the entire code in this CodeSandbox example. A refactored approach using Data Loaders and Data Sources Even though our first approach does solve the issue of resolving nested queries, we still have an issue fetching the same data repeatedly. Let’s look at this example query: `graphql query MyAccessories { accessories { id brand { id brandName } } } ` If we take a look at the results, we are making additional queries for the brand each time we request the information. This leads to the N+1 problem in our current implementation. We can solve this issue by using Data Loaders and Data Sources. What are Data Loaders Data Loaders are used to batch and cache fetch requests. This allows us to fetch the same data and work with cached results, and reduce the number of API calls we have to make. To learn more about Data Loaders in GraphQL, please read this helpful article. How to setup a Data Source In this example, we will be using the following packages: - apollo-datasource - apollo-server-caching - dataloader We first need to create a BrandAccessoryDataSource` class which will simulate the fetching of our data. `js class BrandAccessoryDataSource extends DataSource { ... } ` We will then set up a constructor with a custom Dataloader. `js constructor() { super(); this.loader = new DataLoader((ids) => { if (!ids.length) { return musicAccessories; } return musicAccessories.filter((accessory) => ids.includes(accessory.id)); }); } ` Right below our constructor, we will set up the context and cache. `js initialize({ context, cache } = {}) { this.context = context; this.cache = cache || new InMemoryLRUCache(); } ` We then want to set up the error handling and cache keys for both the accessories and brands. To learn more about how caching works with GraphQL, please read through this article. `js didEncounterError(error) { throw new Error(There was an error loading data: ${error}`); } cacheKey(id) { return music-acc-${id}`; } cacheBrandKey(id) { return brand-acc-${id}`; } ` Next, we are going to set up an asynchronous function called get` which takes in an `id`. The goal of this function is to first check if there is anything in the cached results and if so return those cached results. Otherwise, we will set that data to the cache and return it. We will set the `ttl`(Time to Live in cache) value to 15 seconds. `js async get(id) { const cacheDoc = await this.cache.get(this.cacheKey(id)); if (cacheDoc) { return JSON.parse(cacheDoc); } const doc = await this.loader.load(id); this.cache.set(this.cacheKey(id), JSON.stringify(doc), { ttl: 15 }); return doc; } ` Below the get` function, we will create another asynchronous function called `getByBrand` which takes in a `brand`. This function will have a similar setup to the `get` function but will filter out the data by brand. `js async getByBrand(brand) { const cacheDoc = await this.cache.get(this.cacheBrandKey(brand.id)); if (cacheDoc) { return JSON.parse(cacheDoc); } const musicBrandAccessories = musicAccessories.filter( (accessory) => accessory.brandId === brand.id ); this.cache.set( this.cacheBrandKey(brand.id), JSON.stringify(musicBrandAccessories), { ttl: 15 } ); return musicBrandAccessories; } ` Setting up our schemas and resolvers The last part of this refactored example includes modifying the resolvers. We first need to add an accessory` key to our `Query` schema. `graphql type Query { brands: [Brand] accessory(id: Int): Accessory } ` Inside the resolver`, we will add the `accessories` key with a value for the function that returns the data source we created earlier. `js // this is the custom scalar type we added to the Accessory schema USCurrency, Query: { brands: () => musicBrands, accessory: (, { id }, context) => context.dataSources.brandAccessories.get(id), }, ` We also need to refactor our Brand` resolver to include the data source we set up earlier. `js Brand: { accessories: (brand, , context) => context.dataSources.brandAccessories.getByBrand(brand), }, ` Lastly, we need to modify our ApolloServer object to include the BrandAccessoryDataSource`. `js const server = new ApolloServer({ typeDefs, resolvers, dataSources: () => ({ brandAccessories: new BrandAccessoryDataSource() }), }); ` Here is the entire CodeSandbox example. When the server starts up, click on the Query your server` button and run the following query: `graphql query Query { brands { id brandName accessories { id product price } } } ` Resolving nested queries when microservices are involved Microservices is a type of architecture that will split up your software into smaller independent services. All of these smaller services can interact with a single API data layer. In this case, this data layer would be GraphQL. The client will interact directly with this data layer, and will consume API data from a single entry point. You would similarly resolve your nested queries as before because, at the end of the day, there are just functions. But now, this single API layer will reduce the number of requests made by the client because only the data layer will be called. This simplifies the data fetching experience on the client side. Conclusion In this article, we looked at a few code examples that explored different solutions on how to resolve nested queries in Apollo Server. The first approach involved creating custom resolvers and then using the filter` method to filter out music accessories by brand. We then refactored that example to use a custom DataLoader and Data Source to fix the "N+1 problem". Lastly, we briefly touched on how to approach this solution if microservices were involved. If you want to get started with Apollo Server and build your own nested queries and resolvers using these patterns, check out our serverless-apollo-contentful starter kit!...

Leveraging GraphQL Scalars to Enhance Your Schema cover image

Leveraging GraphQL Scalars to Enhance Your Schema

Introduction GraphQL has revolutionized the way developers approach application data and API layers, gaining well-deserved momentum in the tech world. Yet, for all its prowess, there's room for enhancement, especially when it comes to its scalar types. By default, GraphQL offers a limited set of these primitives — Int, Float, String, Boolean, and ID — that underpin every schema. While these types serve most use cases, there are scenarios where they fall short, leading developers to yearn for more specificity in their schemas. Enter graphql-scalars, a library designed to bridge this gap. By supplementing GraphQL with a richer set of scalar types, this tool allows for greater precision and flexibility in data representation. In this post, we'll unpack the potential of enhanced Scalars, delve into the extended capabilities provided by graphql-scalars, and demonstrate its transformative power using an existing starter project. Prepare to redefine the boundaries of what your GraphQL schema can achieve. Benefits of Using Scalars GraphQL hinges on the concept of "types." Scalars, being the foundational units of GraphQL's type system, play a pivotal role. While the default Scalars — Int, Float, String, Boolean, and ID — serve many use cases, there's an evident need for more specialized types in intricate web development scenarios. 1. Precision**: Using default Scalars can sometimes lack specificity. Consider representing a date or time in your application with a String; this might lead to ambiguities in format interpretation and potential inconsistencies. 2. Validation**: Specialized scalar types introduce inherent validation. Instead of using a String for an email or a URL, for example, distinct types ensure the data meets expected formats at the query level itself. 3. Expressiveness**: Advanced Scalars provide clearer intentions. They eliminate ambiguity inherent in generic types, making the schema more transparent and self-explanatory. Acknowledging the limitations of the default Scalars, tools like graphql-scalars have emerged. By broadening the range of available data types, graphql-scalars allows developers to describe their data with greater precision and nuance. Demonstrating Scalars in Action with Our Starter Project To truly grasp the transformative power of enhanced Scalars, seeing them in action is pivotal. For this, we'll leverage a popular starter kit: the Serverless framework with Apollo and Contentful. This kit elegantly blends the efficiency of serverless functions with the power of Apollo's GraphQL and Contentful's content management capabilities. Setting Up the Starter: 1. Initialize the Project: `shell npm create @this-dot/starter -- --kit serverless-framework-apollo-contentful ` 2. When prompted, name your project enhance-with-graphql-scalars`. `shell Welcome to starter.dev! (create-starter) ✔ What is the name of your project? … enhance-with-graphql-scalars > Downloading starter kit... ✔ Done! Next steps: cd enhance-with-graphql-scalars npm install (or pnpm install, yarn, etc) ` 3. For a detailed setup, including integrating with Contentful and deploying your serverless functions, please follow the comprehensive guide provided in the starter kit here. 4. And we add the graphql-scalars` package `shell npm install graphql-scalars ` Enhancing with graphql-scalars: Dive into the technology.typedefs.ts` file, which is the beating heart of our GraphQL type definitions for the project. Initially, these are the definitions we encounter: `javascript export const technologyTypeDefs = gql type Technology { id: ID! displayName: String! description: String url: URL } type Query { "Technology: GET" technology(id: ID!): Technology technologies(offset: Int, limit: Int): [Technology!] } type Mutation { "Technology: create, read and delete operations" createTechnology(displayName: String!, description: String, url: String): Technology updateTechnology(id: ID!, fields: TechnologyUpdateFields): Technology deleteTechnology(id: ID!): ID } input TechnologyUpdateFields { "Mutable fields of a technology entity" displayName: String description: String url: String } ; ` Our enhancement strategy is straightforward: Convert the `url` field from a String to the `URL` scalar type, bolstering field validation to adhere strictly to the URL format. Post-integration of graphql-scalars`, and with our adjustments, the revised type definition emerges as: `javascript export const technologyTypeDefs = gql type Technology { id: ID! displayName: String! description: String url: URL } type Query { "Technology: GET" technology(id: ID!): Technology technologies(offset: Int, limit: Int): [Technology!] } type Mutation { "Technology: create, read and delete operations" createTechnology(displayName: String!, description: String, url: URL): Technology updateTechnology(id: ID!, fields: TechnologyUpdateFields): Technology deleteTechnology(id: ID!): ID } input TechnologyUpdateFields { "Mutable fields of a technology entity" displayName: String description: String url: URL } ; ` To cap it off, we integrate the URL` type definition along with its resolvers (sourced from `graphql-scalars`) in the `schema/index.ts` file: `javascript import { mergeResolvers, mergeTypeDefs } from '@graphql-tools/merge'; import { technologyResolvers, technologyTypeDefs } from './technology'; import { URLResolver, URLTypeDefinition } from 'graphql-scalars'; const graphqlScalars = [URLTypeDefinition]; export const typeDefs = mergeTypeDefs([...graphqlScalars, technologyTypeDefs]); export const resolvers = mergeResolvers([{ URL: URLResolver }, technologyResolvers]); ` This facelift doesn't just refine our GraphQL schema but infuses it with innate validation, acting as a beacon for consistent and accurate data. Testing in the GraphQL Sandbox Time to witness our changes in action within the GraphQL sandbox. Ensure your local server is humming along nicely. Kick off with verifying the list query: ` query { technologies { id displayName url }, } ` Output: `json { "data": { "technologies": [ { "id": "4UXuIqJt75kcaB6idLMz3f", "displayName": "GraphQL", "url": "https://graphql.framework.dev/" }, { "id": "5nOshyir74EmqY4Jtuqk2L", "displayName": "Node.js", "url": "https://nodejs.framework.dev/" }, { "id": "5obCOaxbJql6YBeXmnlb5n", "displayName": "Express", "url": "https://www.npmjs.com/package/express" } ] } } ` Success! Each url` in our dataset adheres to the pristine URL format. Any deviation would've slapped us with a format error. Now, let's court danger. Attempt to update the url` field with a _wonky_ format: ` mutation { updateTechnology(id: "4UXuIqJt75kcaB6idLMz3f", fields: { url: "aFakeURLThatShouldThrowError" }) { id displayName url } } ` As anticipated, the API throws up a validation roadblock: `json { "data": {}, "errors": [ { "message": "Expected value of type \"URL\", found \"aFakeURLThatShouldThrowError\"; Invalid URL", "locations": [ { "line": 18, "column": 65 } ], "extensions": { "code": "GRAPHQLVALIDATION_FAILED", "stacktrace": [ "TypeError [ERRINVALID_URL]: Invalid URL", " at new NodeError (node:internal/errors:399:5)", " at new URL (node:internal/url:560:13)", ... ] } } ] } ` For the final act, re-run the initial query to reassure ourselves that the original dataset remains untarnished. Conclusion Enhancing your GraphQL schemas with custom scalars not only amplifies the robustness of your data structures but also streamlines validation and transformation processes. By setting foundational standards at the schema level, we ensure error-free, consistent, and meaningful data exchanges right from the start. The graphql-scalars` library offers an array of scalars that address common challenges developers face. Beyond the `URL` scalar we explored, consider diving into other commonly used scalars such as: **DateTime**: Represents date and time in the ISO 8601 format. **Email**: Validates strings as email addresses. **PositiveInt**: Ensures integer values are positive. **NonNegativeFloat**: Guarantees float values are non-negative. As a potential next step, consider crafting your own custom scalars tailored to your project's specific requirements. Building a custom scalar not only offers unparalleled flexibility but also provides deeper insights into GraphQL's inner workings and its extensibility. Remember, while GraphQL is inherently powerful, the granular enhancements like scalars truly elevate the data-fetching experience for developers and users alike. Always evaluate your project's needs and lean into enhancements that bring the most value. To richer and more intuitive GraphQL schemas!...

Next.js 13 Server Actions cover image

Next.js 13 Server Actions

Introduction May 2023, Vercel announced that the App Router is recommended for production in the new version of Next.js v13.4, and it came with a lot of good features. In this article, we will focus on one of the best new features in the App Router: the Server Actions. Server Actions vs Server Components At first, you may think both Server Components and Server Actions are the same, but they are not. Server Components are featured in React 18 to render some components on the server side. On the other hand, Server Actions is a Next.js 13 feature that allows you to execute functions on the backend from the frontend. Back to Web 1.0 Before Single Page Applications frameworks, we were using HTML forms for most things. All you had to do was add a directory to their action attribute, and it would push the data to the server with each input name. No state was required, no async and await, etc. When Remix became open source last year, it introduced this idea again to the market. Its form actions would work even without JavaScript enabled in the browsers. It was mind-blowing. Next.js Server Actions are similar, with a wider range of use cases. They added a lot to it so you don’t only use them on HTML forms, but also on buttons and client components. How useful are Server Actions As I said in the section above, the Server Actions are like Form Actions in Remix and also they work on both Server and Client in a way similar to tRPC or Telefunc. Think about it. Instead of creating an API endpoint and making a fetch request to it from the client side, you execute a function that’s already on the server from the client like any other JavaScript function. Server Actions on the Server Side To make this post more effective, I’ll build a simple counter component with Server Actions. It runs even if JavaScript is turned off. First, create a new Next.js 13 App router project: ` npx create-next-app@latest ` Then, in the app/page.tsx, add a variable outside of the page component. `js let count = 0; export default function Home() { // ... } ` Then, add the Server Action, which is an async function with the “use server”; tag. `js let count = 0; export default function Home() { async function increment() { "use server"; count++; } return ( Count {count} Increment ) } ` After implementing the above, when you click on the increment button, you shouldn’t see the change until you refresh the page. You need to use revalidatePath to get a reactive state. `js import { revalidatePath } from "next/cache"; let count = 0; export default function Home() { async function increment() { "use server"; count++; revalidatePath("/"); } return ( Count {count} Increment ) } ` Now you have a Counter that can run with zero JavaScript. Server Actions on the Client Side You can also use the Server Action on the client component (it will be more like tRPC and Telefunc) to trigger a function on the server from the client and get data. But it will not work in the exact same way because in Next.js, each file has to be run on the server or the client. You can’t run the same file on both. So we need to move the function to its own new file, and add the ”use server”; tag on the top of the file. `js "use server"; export async function increment() { "use server"; count++; revalidatePath("/"); } ` We can import it from the client component, and use this server action there! Conclusion Server Actions are a great addition to Next.js, and it will make it easier to build full-stack applications that require a lot of work to be done in traditional frontend/backend ways like creating the API route, and then fetching it. I hope you enjoyed this article, and if you have any questions or feedback, feel free to reach out to us....

Being a CTO at Any Level: A Discussion with Kathy Keating, Co-Founder of CTO Levels cover image

Being a CTO at Any Level: A Discussion with Kathy Keating, Co-Founder of CTO Levels

In this episode of the engineering leadership series, Kathy Keating, co-founder of CTO Levels and CTO Advisor, shares her insights on the role of a CTO and the challenges they face. She begins by discussing her own journey as a technologist and her experience in technology leadership roles, including founding companies and having a recent exit. According to Kathy, the primary responsibility of a CTO is to deliver the technology that aligns with the company's business needs. However, she highlights a concerning statistic that 50% of CTOs have a tenure of less than two years, often due to a lack of understanding and mismatched expectations. She emphasizes the importance of building trust quickly in order to succeed in this role. One of the main challenges CTOs face is transitioning from being a technologist to a leader. Kathy stresses the significance of developing effective communication habits to bridge this gap. She suggests that CTOs create a playbook of best practices to enhance their communication skills and join communities of other CTOs to learn from their experiences. Matching the right CTO to the stage of a company is another crucial aspect discussed in the episode. Kathy explains that different stages of a company require different types of CTOs, and it is essential to find the right fit. To navigate these challenges, Kathy advises CTOs to build a support system of advisors and coaches who can provide guidance and help them overcome obstacles. Additionally, she encourages CTOs to be aware of their own preferences and strengths, as self-awareness can greatly contribute to their success. In conclusion, this podcast episode sheds light on the technical aspects of being a CTO and the challenges they face. Kathy Keating's insights provide valuable guidance for CTOs to build trust, develop effective communication habits, match their skills to the company's stage, and create a support system for their professional growth. By understanding these key technical aspects, CTOs can enhance their leadership skills and contribute to the success of their organizations....