Javascript

Seeding Initial Data in Amplify

Sep 8, 2021

6 min read

Challenges With Amplify

Amplify is a powerful framework and toolchain that enables developers to build their frontend applications with a serverless backend using all major languages and platforms. It relieves the developers from the effort of writing a backend from scratch - the entire backend stack is provided by AWS.

However, one of the things that does not come with Amplify out of the box, at least at this moment, is a way to perform initial database seeding right after setting up an environment. Database seeding is the process of populating an empty database with data. The data can be some fake placeholder data to use in development, or some necessary data such as a set of initial configuration properties required for the proper functioning of the application.

In this blog post, we'll walk you through our attempt to write our own database seeding script, while having some guiding principles in mind. First of all, the script should be generic - it should not be specific to a particular project. Anyone should be able to copy it and place it in their project with no or minimal adjustments required. Secondly, the database seeding process should be automatic and ideally not require any manual steps by the user - the user should just execute the script and the database should be populated after the script is finished running. The script is responsible for reading the Amplify environment, and putting the data in proper tables.

Prerequisites

Before proceeding further, note that some of the concepts presented here assume at least basic Amplify experience. Likewise, there should be an existing Amplify project. For this purpose, we scaffolded a simple Angular project using Amplify starter project instructions and the entire project is available on GitHub. The tutorial also assumes that a GraphQL API is generated as part of the project, just like in the starter project.

Looking at the official AWS guide for writing items to a DynamoDB database in batch, we know that we need several pieces of information before making a connection to the database:

AWS credentials
The profile under which AWS credentials are stored
Region
Amplify environment name
Database ID (Tables created by Amplify have the format of tableName-databaseId-environmentName)

A recommended way of reading AWS credentials is through a shared credentials file. The credentials file is located in the .aws folder of your user folder and contains your access key ID and secret access key which are used to sign programmatic requests to AWS. It should have been created after executing amplify configure, and before initializing the Amplify project. The file has the following format:

[your-profile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

There is one additional file that contains your region preferences. It is named config, and it is also located in the .aws folder:

[profile your-profile]
region = eu-central-1

Extracting Information From Amplify Environment

Now that we have our credentials, we need to find the remaining information. Going through the amplify folder (generated by the Amplify CLI) we see that all information that we need is already there, but scattered across multiple files.

For example:

The environment name is located in amplify/.config/local-env-info.json
The profile name is located in amplify/.config/local-aws-info.json, assuming that the Amplify project was initialized using a local AWS profile name and not IAM keys
Database table ID and region is located in amplify/backend/amplify-meta.json

The risk in this approach is that this logic could change in the future if AWS decides to change the format of any of the above files, which aren't really meant for this kind of use. However, the benefit is ease and simplicity of use, and we feel that the benefits outweigh the risks. Should the files change in the future, we'll simply update the seeding script.

Writing the Script

Reading the environment and the profile name from local-env-info.json and local-aws-info.json is easy. Reading the database table ID and region is a bit tricky since we need to know the GraphQL API name first. Each GraphQL API will generate its own set of database tables on DynamoDB, so knowing GraphQL API name will easily get us the database table ID as well. Fortunately, amplify-meta.json has the list of APIs under the api field, and we'll iterate over this object to get API names.

Within the API object, there are also two fields that we are interested in. The first one is the name of the provider, found in the providerPlugin field. We'll use this to fetch the region from the providers root field in amplify-meta.json. The other one is output.GraphQLAPIIdOutput, which is the GraphQL API ID, and the same ID is used as the database ID for the tables.

Having everything we said so far in mind, the first version of the script is:

const AWS = require('aws-sdk');
const localEnvInfo = require('../../amplify/.config/local-env-info.json');
const localAwsInfo = require('../../amplify/.config/local-aws-info.json');
const amplifyMeta = require('../../amplify/backend/amplify-meta.json');

const environmentName = localEnvInfo.envName;
const profileName = localAwsInfo[environmentName]?.profileName;

if (!profileName) {
  throw Error('Please reinitialize your Amplify project using your AWS profile');
}

for (const [apiName] of Object.entries(amplifyMeta.api)) {
  const providerName = amplifyMeta.api[apiName].providerPlugin;
  const databaseId = amplifyMeta.api[apiName].output.GraphQLAPIIdOutput;
  const region = amplifyMeta.providers[providerName].Region;

  AWS.config.credentials = new AWS.SharedIniFileCredentials({ profile: profileName });
  AWS.config.update({ region: region })
  const documentClient = new AWS.DynamoDB.DocumentClient();

  // ToDo: read seed data and write to DynamoDB
}

The script reads the local Amplify environment from local-env-info.json, local-aws-info.json, amplify-meta.json, performs some safety checks, then iterates over each API found in amplify-meta.json. In the end, it creates an instance of the DocumentClient class which is used to write to DynamoDB.

Organizing Seed Data

The missing step is reading and writing seed data. Again, we don't want any manual configuration here - the script should just run, read the seed data and use the Amplify information we extracted earlier to insert the data to the DynamoDB database. For this to work, we'll organize our directories and files in a way that they are easily mapped to objects on AWS. The directory structure we're aiming for is this:

tools/
├─ seeder/
│  ├─ index.js (this is our seeding script)
│  ├─ fixtures/
│  │  ├─ [API 1 name]/
│  │  │  ├─ [Table 1 name].json
│  │  │  ├─ [Table 2 name].json
│  │  │  ├─ ...
│  │  ├─ [API 2 name]/
│  │  ├─ .../

For each API found in amplify-meta.json, the script will find the respective directory under fixtures, and iterate over JSON files found in that directory. Each file's name corresponds to the table name we're populating, and the content of the file is a JSON array with table items. For example, the JSON file to populate the Restaurant's table from our Angular starter project would be named Restaurants.json and would have the following contents:

[
  {
    "city": "Menton",
    "description": "",
    "name": "Mirazur"
  },
  {
    "city": "Copenhagen",
    "description": "",
    "name": "Noma"
  },
  {
    "city": "Axpe",
    "description": "",
    "name": "Asador Etxebarri"
  }
]

DynamoDB's DocumentClient

The next step is implementing the logic that reads JSON files, and writes them to DynamoDB. DocumentClient's batchWrite method allows providing multiple items for multiple tables, all in the same request. This means that the method will be called only once, and all items will be pushed in batch, which is very performant.

The structure of this write request looks as follows:

const writeParams = {
  RequestItems: {
    table1Name: [
      {
        PutRequest: {
          Item: {
            id: 1,
            field1:'field 1 value',
            field2: 'field 2 value',
            //...
          },
        }
      },
      {
        PutRequest: {
          Item: {
            id: 2,
            field1:'field 1 value',
            field2: 'field 2 value',
            //...
          },
        }
      },
      //...
    ]
  }
};

The table name is the complete table name, which is in the form of baseTableName-databaseId-environmentName when using Amplify. baseTableName normally comes from the GraphQL schema type, databaseId is equal to GraphQL API ID, while environmentName is the Amplify environment name (such as dev).

In the PutRequest, we should specify all fields that would normally be provisioned when using the GraphQL API. This includes some fields that Amplify uses internally, like:

id (primary key, which must be unique and can be generated using uuidv4() for example)
__typename (equal to type in the GraphQL schema, which is normally equal to baseTableName)
_lastChangedAt (can be the current date, written as Unix timestamp)
_version (can be 1)
createdAt (can be the current date, written as ISO-8601 string)
updatedAt (can be the current date, written as ISO-8601 string)

Other, external fields, can be provided from seed data.

Written in code, this piece of logic looks like this:

const writeParams = {
  RequestItems: {}
};
const baseTableNames = readdirSync(join(__dirname, 'fixtures', apiName)).map((filename) => parse(filename).name);
for (const baseTableName of baseTableNames) {
  const fullTableName = `${baseTableName}-${databaseId}-${environmentName}`;
  const tableItems = require(`./fixtures/${apiName}/${baseTableName}.json`);
  writeParams.RequestItems[fullTableName] = tableItems.map((tableItem) => ({
    PutRequest: {
      Item: {
        id: uuidv4(),
        __typename: baseTableName,
        _lastChangedAt: new Date().getTime(),
        _version: 1,
        createdAt: new Date().toISOString(),
        updatedAt: new Date().toISOString(),
        ...tableItem
      }
    }
  }));
}

documentClient.batchWrite(writeParams, (error, data) => {
  if (error) {
    console.error('Error in batch write', error);
  } else {
    console.error('Successfully executed batch write', data);
  }
});

Conclusion

And that's it - the script is complete. In just over 50 lines of code, we now have an efficient way of populating multiple DynamoDB tables with initial data. The initial data itself is stored in JSON files, which itself is quite readable and anyone can customize it to their project needs.

We hope you enjoyed this tutorial. Should you have any questions, do not hesitate to drop us a line.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

Dario Djuric

Dario is a full-stack engineer who has spent most of his career doing enterprise Java projects. He has always had a hidden passion for frontend, though -- and he is now able to pursue that passion at This Dot. He spends most of his free time with his two sons, but occasionally, he manages to squeeze in some casual sports activities such as jogging, cycling, and soccer.

@dario_djuric @dariodjuric

Why Every Developer Should Try Elm, + Are We Abandoning JavaScript? with Lindsay Wardell

Lindsay Wardell, Senior Software Engineer at NoRedInk joins Rob Ocel by covering many of the functional programming advantages afforded by Elm. By immersing themselves in the intricacies of languages like Elm, developers can harness its functional approach to bolster application resilience and minimize errors. Lindsay draws parallels with TypeScript's emphasis on strong typing, and how it empowers developers to detect and rectify errors early on, fostering the creation of more maintainable codebases. A pivotal theme emerging from Lindsay's discussions is the importance of adaptability in choosing which languages a developer focuses their energy on. As technology continues to evolve, developers must embrace versatility to remain competitive. Lindsay and Rob Ocel talk about the potential shifts in front-end development paradigms, beyond the dominance of component-based frameworks like React. This underscores the necessity of keeping an open mind and exploring emerging technologies to stay at the forefront of the industry's evolution. Moreover, Lindsay and Rob accentuate the significance of tools such as Nuxt for facilitating seamless backend integration within frameworks like Vue.js. These integrations streamline the development process, enabling developers to concentrate on crafting robust and scalable applications. Their conversations serve as a testament to the ongoing evolution of programming tools, marked by a trend towards simplification and enhanced development efficiency. Download this episode....

Apr 3, 2024

1 min

JavaScriptTypeScript

How to create and use custom GraphQL Scalars

How to create and use custom GraphQL Scalars In the realm of GraphQL, scalars form the bedrock of the type system, representing the most fundamental data types like strings, numbers, and booleans. As explored in our previous post, "Leveraging GraphQL Scalars to Enhance Your Schema," scalars play a pivotal role in defining how data is structured and validated. But what happens when the default scalars aren't quite enough? What happens when your application demands a data type as unique as its requirements? Enter the world of custom GraphQL scalars. These data types go beyond the conventional, offering the power and flexibility to tailor your schema to precisely match your application's unique needs. Whether handling complex data structures, enforcing specific data formats, or simply bringing clarity to your API, custom scalars open up a new realm of possibilities. In this post, we'll explore how to understand, create, and effectively utilize custom scalars in GraphQL. From conceptualization to implementation, we'll cover the essentials of extending your GraphQL toolkit, empowering you to transform abstract ideas into concrete, practical solutions. So, let's embark together on the journey of understanding and utilizing custom GraphQL scalars, enhancing and expanding the capabilities of your GraphQL schema. Understanding Custom Scalars Custom scalars in GraphQL extend beyond basic types like String or Int, allowing data to be defined, validated, and processed more precisely. They're instrumental when default types don't quite capture the complexity or specificity of the data, such as with specialized date formats or unique identifiers. The use of custom scalars brings several benefits: * Enhanced Clarity: They offer a clearer representation of what data looks like and how it behaves. * Built-in Validation: Data integrity is bolstered at the schema level. * Flexibility: They can be tailored to specific data handling needs, making your schema more adaptable and robust. With this understanding, we'll explore creating and integrating custom scalars into a GraphQL schema, turning theory into practice. Creating a Custom Scalar Defining a Custom Scalar in TypeScript: Creating a custom scalar in GraphQL with TypeScript involves defining its behavior through parsing, serialization, and validation functions. * Parsing: Transforms input data from the client into a server-understandable format. * Serializing: Converts server data back to a client-friendly format. * Validation: Ensures data adheres to the defined format or criteria. Example: A 'Color' Scalar in TypeScript The Color scalar will ensure that every color value adheres to a valid hexadecimal format, like #FFFFFF for white or #000000 for black: ` In this TypeScript implementation: * validateColors: a function that checks if the provided string matches the hexadecimal color format. * parseValue: a method function that converts the scalar’s value from the client into the server’s representation format - this method is called when a client provides the scalar as a variable. See parseValue docs for more information * serialize: a method function that converts the scalar’s server representation format to the client format, see serialize docs for more information * parseLiteral: similar to parseValue, this method function converts the scalar’s value from the client to the server’s representation format. Still, this method is called when the scalar is provided as a hard-coded argument (inline). See parseLiteral docs for more information In the upcoming section, we'll explore how to incorporate and validate these custom scalars within your schema, ensuring they function seamlessly in real-world scenarios. Integrating Custom Scalars into a Schema Incorporating the 'Color' Scalar After defining your custom Color scalar, the next crucial step is effectively integrating it into your GraphQL schema. This integration ensures that your GraphQL server recognizes and correctly utilizes the scalar. Step-by-Step Integration 1. Add the scalar to Type Definitions: Include the Color scalar in your GraphQL type definitions. This inclusion informs GraphQL about this new scalar type. 2. Resolver Mapping: Map your custom scalar type to its resolver. This connection is key for GraphQL to understand how to process this type during queries and mutations. ` 1. Use the scalar: Update your type to use the new custom scalar ` Testing the Integration With your custom Color scalar integrated, conducting thorough testing is vital. Ensure that your GraphQL server correctly handles the Color scalar, particularly in terms of accepting valid color formats and rejecting invalid ones. For demonstration purposes, I've adapted a creation mutation to include the primaryColor field. To keep this post focused and concise, I won't detail all the code changes here, but the following screenshots illustrate the successful implementation and error handling. Calling the mutation (createTechnology) successfully: Calling the mutation with forced fail (bad color hex): Conclusion The journey into the realm of custom GraphQL scalars reveals a world where data types are no longer confined to the basics. By creating and integrating scalars like the Color type, we unlock precision and specificity in our GraphQL schemas, which significantly enhance our applications' data handling capabilities. Custom scalars are more than just a technical addition; they testify to GraphQL's flexibility and power. They allow developers to express data meaningfully, ensuring that APIs are functional, intuitive, and robust. As we've seen, defining, integrating, and testing these scalars requires a blend of creativity and technical acumen. It encourages a deeper understanding of how data flows through your application and offers a chance to tailor that experience to your project's unique needs. So, as you embark on your GraphQL journey, consider the potential of custom scalars. Whether you're ensuring data integrity, enhancing API clarity, or simply making your schema a perfect fit for your application, the possibilities are as vast as they are exciting. Embrace the power of customization, and let your GraphQL schemas shine!...

Apr 10, 2024

5 mins

GraphQLJavaScript

Angular 17: Continuing the Renaissance

Angular 17: A New Era November 8th marked a significant milestone in the world of Angular with the release of Angular 17. This wasn't just any ordinary update; it was a leap forward, signifying a new chapter for the popular framework. But what made this release truly stand out was the unveiling of Angular's revamped website, complete with a fresh brand identity and a new logo. This significant transformation represents the evolving nature of Angular, aligning with the modern demands of web development. To commemorate this launch, we also hosted a release afterparty, where we went deep into its new features with Minko Gechev from the Angular core team, and Google Developer Experts (GDEs) Brandon Roberts, Deborah Kurata, and Enea Jahollari. But what exactly are these notable new features in the latest version? Let's dive in and explore. The Angular Renaissance Angular has been undergoing a significant revival, often referred to as Angular's renaissance, a term coined by Sarah Drasner, the Director of Engineering at Google, earlier this year. This revival has been particularly evident in its recent versions. The Angular team has worked hard to introduce many new improvements, focusing on signal-based reactivity, hydration, server-side rendering, standalone components, and migrating to esbuild and Vite for a better and faster developer experience. This latest release, in particular, marks many of these features as production-ready. Standalone Components About a year ago, Angular began a journey toward modernity with the introduction of standalone components. This move significantly enhanced the developer experience, making Angular more contemporary and user-friendly. In Angular's context, a standalone component is a self-sufficient, reusable code unit that combines logic, data, and user interface elements. What sets these components apart is their independence from Angular's NgModule system, meaning they do not rely on it for configuration or dependencies. By setting a standalone: true flag, you no longer need to embed your component in an NgModule and you can bootstrap directly off that component: ` Compared to the NgModules way of adding components, as shown below, you can immediately see how standalone components make things much simpler. ` In this latest release, the Angular CLI now defaults to generating standalone components, directives, and pipes. This default setting underscores the shift towards a standalone-centric development approach in Angular. New Syntax for Enhanced Control Flow Angular 17 introduces a new syntax for control flow, replacing traditional structural directives like ngIf or ngFor, which have been part of Angular since version 2. This new syntax is designed for fine-grained change detection and eventual zone-less operation when Angular completely migrates to signals. It's more streamlined and performance-efficient, making handling conditional or list content in templates easier. The @if block replaces *ngIf for expressing conditional parts of the UI. ` The @switch block replaces ngSwitch, offering benefits such as not requiring a container element to hold the condition expression or each conditional template. It also supports template type-checking, including type narrowing within each branch. ``` The @for block replaces *ngFor for iteration and presents several differences compared to its structural directive predecessor, ngFor. For example, the tracking expression (calculating keys corresponding to object identities) is mandatory but offers better ergonomics. Additionally, it supports @empty blocks. ` Defer Block for Lazy Loading Angular 17 introduces the @defer block, a dramatically improving lazy loading of content within Angular applications. Within the @defer block framework, several sub-blocks are designed to elegantly manage different phases of the deferred loading process. The main content within the @defer block is the segment designated for lazy loading. Initially, this content is not rendered, becoming visible only when specific triggers are activated or conditions are met, and after the required dependencies have been loaded. By default, the trigger for a @defer block is the browser reaching an idle state. For instance, take the following block: it delays the loading of the calendar-imp component until it comes into the viewport. Until that happens, a placeholder is shown. This placeholder displays a loading message when the calendar-imp component begins to load, and an error message if, for some reason, the component fails to load. ` The on keyword supports a wide a variety of other conditions, such as: - idle (when the browser has reached an idle state) - interaction (when the user interacts with a specified element) - hover (when the mouse has hovered over a trigger area) - timer(x) (triggers after a specified duration) - immediate (triggers the deferred load immediately) The second option of configuring when deferring happens is by using the when keyword. For example: ` Server-Side Rendering (SSR) Angular 17 has made server-side rendering (SSR) much more straightforward. Now, a --ssr option is included in the ng new command, removing the need for additional setup or configurations. When creating a new project with the ng new command, the CLI inquires if SSR should be enabled. As of version 17, the default response is set to 'No'. However, for version 18 and beyond, the plan is to enable SSR by default in newly generated applications. If you prefer to start with SSR right away, you can do so by initializing your project with the --ssr flag: ` For adding SSR to an already existing project, utilize the ng add command of the Angular CLI: ` Hydration In Angular 17, the process of hydration, which is essential for reviving a server-side rendered application on the client-side, has reached a stable, production-ready status. Hydration involves reusing the DOM structures rendered on the server, preserving the application's state, and transferring data retrieved from the server, among other crucial tasks. This functionality is automatically activated when server-side rendering (SSR) is used. It offers a more efficient approach than the previous method, where the server-rendered tree was completely replaced, often causing visible UI flickers. Such re-rendering can adversely affect Core Web Vitals, including Largest Contentful Paint (LCP), leading to layout shifts. By enabling hydration, Angular 17 allows for the reuse of the existing DOM, effectively preventing these flickers. Support for View Transitions The new View Transitions API, supported by some browsers, is now integrated into the Angular router. This feature, which must be activated using the withViewTransitions function, allows for CSS-based animations during route transitions, adding a layer of visual appeal to applications. To use it, first you need to import withViewTransitions: ` Then, you need to add it to the provideRouter configuration: ` Other Notable Changes - Angular 17 has stabilized signals, initially introduced in Angular 16, providing a new method for state management in Angular apps. - Angular 17 no longer supports Node 16. The minimal Node version required is now 18.13. - TypeScript version 5.2 is the least supported version starting from this release of Angular. - The @Component decorator now supports a styleUrl attribute. This allows for specifying a single stylesheet path as a string, simplifying the process of linking a component to a specific style sheet. Previously, even for a single stylesheet, an array was required under styleUrls. Conclusion With the launch of Angular 17, the Angular Renaissance is now in full swing. This release has garnered such positive feedback that developers are showing renewed interest in the framework and are looking forward to leveraging it in upcoming projects. However, it's important to note that it might take some time for IDEs to adapt to the new templating syntax fully. While this transition is underway, rest assured that you can still write perfectly valid code using the old templating syntax, as all the changes in Angular 17 are backward compatible. Looking ahead, the future of Angular appears brighter than ever, and we can't wait to see what the next release has in store!...

Nov 20, 2023

6 mins

AngularTypeScriptJavaScript

Transforming Auth in an AI World with Rod Boothby

In today's digital landscape, the need for secure identity verification has become paramount. With the increasing risks of personal data exposure and the rise of sophisticated cyber threats aided by Artificial Intelligence, it is crucial to adopt robust verification processes to protect individuals and organizations alike. In a recent discussion with Rod Boothby, CEO of ID Partner Systems, the significance of trusted institutions for identity verification was emphasized, particularly the efficiency of bank-based ID verification over traditional methods. One of the key takeaways from the conversation was the importance of continuous authentication. As technology advances, so do the methods employed by cybercriminals. Deepfake technology, for instance, poses a significant threat to identity verification systems. To combat this, tighter security measures and continuous authentication are essential. By constantly verifying and validating user identities, organizations can stay one step ahead of potential fraudsters. Rod Boothby also highlighted the need for a developer-focused approach to identity verification. By providing developers with the tools and resources they need, companies like ID Partner Systems aim to streamline the verification process and enhance security. This approach not only ensures a more efficient experience for users but also allows for the integration of behavioral biometrics, which can further strengthen the verification process. Download this episode here....

Apr 22, 2024

1 min

Engineering Leadership

Seeding Initial Data in Amplify

Challenges With Amplify

Prerequisites

Extracting Information From Amplify Environment

Writing the Script

Organizing Seed Data

DynamoDB's DocumentClient

Conclusion

Dario Djuric

You might also like

Why Every Developer Should Try Elm, + Are We Abandoning JavaScript? with Lindsay Wardell

How to create and use custom GraphQL Scalars

Angular 17: Continuing the Renaissance

Transforming Auth in an AI World with Rod Boothby

You might also like

Why Every Developer Should Try Elm, + Are We Abandoning JavaScript? with Lindsay Wardell

How to create and use custom GraphQL Scalars

Angular 17: Continuing the Renaissance

Transforming Auth in an AI World with Rod Boothby