This Dot Blog
This Dot provides teams with technical leaders who bring deep knowledge of the web platform. We help teams set new standards, and deliver results predictably.
GitHub Actions for Serverless Framework Deployments
Background Our team was building a Serverless Framework API for a client that wanted to use the Serverless Dashboard) for deployment and monitoring. Based on some challenges from last year, we agreed with the client that using a monorepo tool like Nx) would be beneficial moving forward as we were potentially shipping multiple Serverless APIs and frontend applications. Unfortunately, we discovered several challenges integrating with the Serverless Dashboard, and eventually opted into custom CI/CD with GitHub Actions. We’ll cover the challenges we faced, and the solution we created to mitigate our problems and generate a solution. Serverless Configuration Restrictions By default, the Serverless Framework does all its configuration via a serverless.yml file. However, the framework officially supports alternative formats) including .json, .js, and .ts. Our team opted into the TypeScript format as we wanted to setup some validation for our engineers that were newer to the framework through type checks. When we eventually went to configure our CI/CD via the Serverless Dashboard UI, the dashboard itself restricted the file format to just the YAML format. This was unfortunate, but we were able to quickly revert back to YAML as configuration was relatively simple, and we were able to bypass this hurdle. Prohibitive Project Structures With our configuration now working, we were able to select the project, and launch our first attempt at deploying the app through the dashboard. Immediately, we ran into a build issue: ` What we found was having our package.json in a parent directory of our serverless app prevented the dashboard CI/CD from being able to appropriately detect and resolve dependencies prior to deployment. We had been deploying using an Nx command: npx nx run api:deploy --stage=dev which was able to resolve our dependency tree which looked like: To resolve, we thought maybe we could customize the build commands utilized by the dashboard. Unfortunately, the only way to customize these commands is via the package.json of our project. Nx allows for package.json per app in their structure, but it defeated the purpose of us opting into Nx and made leveraging the tool nearly obsolete. Moving to GitHub Actions with the Serverless Dashboard We thought to move all of our CI/CD to GitHub Actions while still proxying the dashboard for deployment credentials and monitoring. In the dashboard docs), we found that you could set a SERVERLESS_ACCESS_KEY and still deploy through the dashboard. It took us a few attempts to understand exactly how to specify this key in our action code, but eventually, we discovered that it had to be set explicitly in the .env file due to the usage of the Nx build system to deploy. Thus the following actions were born: api-ci.yml ` api-clean.yml ` These actions ran smoothly and allowed us to leverage the dashboard appropriately. All in all this seemed like a success. Local Development Problems The above is a great solution if your team is willing to pay for everyone to have a seat on the dashboard. Unfortunately, our client wanted to avoid the cost of additional seats because the pricing was too high. Why is this a problem? Our configuration looks similar to this (I’ve highlighted the important lines with a comment): serverless.ts ` The app and org variables make it so it is required to have a valid dashboard login. This meant our developers working on the API problems couldn’t do local development because the client was not paying for the dashboard logins. They would get the following error: Resulting Configuration At this point, we had to opt to bypass the dashboard entirely via CI/CD. We had to make the following changes to our actions and configuration to get everything 100% working: serverless.ts - Remove app and org fields - Remove accessing environment secrets via the param option ` api-ci.yml - Add all our secrets to GitHub and include them in the scripts - Add serverless confg ` api-cleanup.yml - Add serverless config - Remove secrets ` Conclusions The Serverless Dashboard is a great product for monitoring and seamless deployment in simple applications, but it still has a ways to go to support different architectures and setups while being scalable for teams. I hope to see them make the following changes: - Add support for different configuration file types - Add better support custom deployment commands - Update the framework to not fail on login so local development works regardless of dashboard credentials The Nx + GitHub actions setup was a bit unnatural as well with the reliance on the .env file existing, so we hope the above action code will help someone in the future. That being said, we’ve been working with this on the team and it’s been a very seamless and positive change as our developers can quickly reference their deploys and know how to interact with Lambda directly for debugging issues already....
Sep 26, 2022
Migrating an Amplify Backend on Serverless Framework - Part 3
Jun 21, 2022
Migrating an Amplify Backend on Serverless Framework - Part 2
This is Part Two of a three part series on Migrating an Amplify Backend on Serverless Framework. You can find Part One here and Part Three here. Welcome to the second part of our "Migrating an Amplify Backend to Serverless Framework", where I will give you a step-by-step guide on how to migrate Amplify-based services so they can be deployable using the Serverless Framework. In the first part, we scaffolded our example application and explained how to deploy Cognito user pools. In this blog post, we'll focus on the core of the application, which is the GraphQL API powered by AppSync. How AppSync Works Before we go deep into the code, it's worth understanding how AppSync actually works. AppSync is a powerful service by AWS that allows you to deploy GraphQL APIs with ease, as well as connect those APIs with data sources like AWS DynamoDB, lambda, and more. It can work without Amplify, but when you pair Amplify with it, you get some extra benefits. Deployment, for example, is way easier. All you need is a GraphQL schema, and Amplify will generate all the required AppSync components for it to work end-to-end. Some of these components are: * Data sources, which are interfaces to other AWS resources such as lambdas and DynamoDB tables. * Resolvers, which connect your GraphQL types and fields to data sources. A resolver specifies how data is mapped between your GraphQL API and your data sources. This mapping is done using mapping templates, and mapping templates are written using Apache Velocity Template Language (VTL). A mapping template contains one template for request, and one for response. * DynamoDB tables, usually one for each GraphQL type. Below, you can see the flow of data when a GraphQL request is made that interacts with a DynamoDB table: A standard GraphQL schema is not enough to generate everything that AppSync needs, though. Amplify needs some additional metadata in the form of Amplify-specific directives which you can place in your GraphQL schema, and Amplify will use the information from the directives to set up everything correctly. Then, during deployment, Amplify will *transform* your schema so that all Amplify-specific directives are stripped, and additional GraphQL types are generated if needed. This is especially important for the @connection directive, which allows you to define 1:1, 1:M, or N:M relationships between models (which are basically GraphQL types annotated with @model directive). Having additional GraphQL types is necessary in order for GraphQL API clients to properly read those relationships. For example, if the Amplify GraphQL schema contains the following type definition: ` Then, this will be transformed to the following in the standard GraphQL schema: ` It's not quite the same, isn't it? Yet, only the latter form can be recognized by AppSync. Hence, a simplified sequence of steps executed during deployment is: 1. Transform your GraphQL schema enriched with Amplify directives to a standard GraphQL schema, which is then uploaded to AppSync. 2. Create/update/delete DynamoDB tables based on GraphQL types annotated with @model 3. Create/update/delete data sources based on directives such as @model and @function 4. Create/update/delete resolvers and connect them to GraphQL schema's fields on one side as well as to data sources on the other side. Resolvers are the "glue" that connects GraphQL fields to data sources. In our Serverless deployment, we'll need to replicate the above deployment steps. Let's get to it. Transforming GraphQL Schema As part of our Amplified ToDo application, we provided a GraphQL schema that will be used for our API. Now, if we tried to deploy the schema to AppSync, with or without Serverless Framework, it would fail because the schema contains Amplify-specific directives. We need to run it through a *transformer* first - the same one that Amplify runs when the application is deployed using Amplify. Fortunately, the Amplify CLI is open-source, and the transformer is part of it, meaning that we can freely use it. Guided by an excellent article by Ronny Roeller, we wrote a script that takes an Amplify GraphQL schema and produces an AppSync-compatible schema. Not only that, it also generates VTL templates along with it, just like Amplify! The script assumes that the Amplify GraphQL schema is located in the amplify-schema.graphql file (we copied it from amplify/backend/api/amplifiedtodo/schema.graphql). In the script, we first create an instance of GraphQLTransform and pass it different transformer instances, each covering one specific Amplify directive. For example, the ModelConnectionTransformer instance will process @connection directives, and the KeyTransformer will process @key directives. Then, we call the GraphQLTransform.transform() method which will output the transformed schema to appsync-schema.graphql. Finally, we go through the generated types and fields to generate the mapping templates for the resolvers. Mapping templates are stored to serverless.mapping-templates.yml file. There are two types of mapping templates in the file: "UNIT" (which is the default) and "PIPELINE". They correspond to unit and pipeline resolvers, respectively. Unit resolvers are used for mapping to a single data source, while pipeline resolvers allow you to invoke a series of functions in a serial manner. In our case, pipeline resolvers are used for invoking lambda-based data sources. Invoking the script is a manual step, and it's best to invoke it before deployment. We made an npm task called deploy that runs the transformer, and then runs sls deploy. This task should be used for deploying from now on as it makes sure that no changes to the GraphQL are left out by accident. Generating GraphQL TypeScript Types Another thing that Amplify does is that it can generate TypeScript files that correspond to GraphQL types. Unfortunately, this is hard to do without the Amplify CLI, as the code generation part is deep in the Amplify CLI and cannot be extracted like the transformer. If you use TypeScript, you'll need to keep the Amplify CLI to perform code generation using the amplify codegen command. You can add this command to the deploy command as well. ` Configuring the AppSync Plugin The final step is configuring the serverless-appsync-plugin plugin so we can deploy our schema to AppSync. Before we do that, we should create the lambdas and DynamoDB tables that our GraphQL API will use. If you recall from the previous blog post, we've defined one Lambda called processQueue under the Mutation type: ` Let's create a placeholder for that Lambda by copying handlers/insert-user-preference to handlers/process-queue and adding it to our Serverless config: ` We will also need to create one DynamoDB table for each GraphQL type, namely: * List * Item * UserPreference * NotificationQueue Let's add the table names to the environment so that they can be passed into our Lambda functions: ` Now, let's create the actual tables under the Resources section: ` Note how we did not need to define all fields that are defined in the schema. DynamoDB is schemaless, meaning that you can provision arbitrary columns to table rows. We only need to define the columns that are primary key columns as well as those covered by indexes. Indexes are not created automatically by the serverless-appsync-plugin - you need to define manually in the Resources section. Next, we need to create data sources. Create a file called serverless.data-sources.yml with the following contents: ` The above file defines data sources for our lambdas and DynamoDB tables. The names of data sources are not arbitrary. They need to match the names that were generated by the transformer script in the serverless.mapping-templates.yml file. Also, each data source references a table by using the Ref intrinsic function to find the table name by resource name (defined under Resources section of the main Serverless config). The final piece are the function configs, which is a configuration part of the serverless-appsync-plugin that is required only for pipeline resolvers. In the serverless.mapping-templates.yml file, each mapping template for a pipeline resolver refers to a function config. For example, the pipeline resolver for the processQueue lambda references a function config named InvokeProcessQueueLambdaDataSource. The function config provides a request/response mapping, also written in VTL templates, for the lambda in question. Create a file named serverless.appsync-function-configs.yml, with the following contents: ` Create default-invoke-lambda.req.vtl inside the mapping-templates directory, with the following contents: ` Likewise, create default-invoke-lambda.res.vtl with the following contents: ` As you can see, the VTL request/response templates for pipeline resolvers are not complicated, and they are same for every pipeline resolver if there are more of them. For that reason, all pipeline resolvers can reference the same files: default-invoke-lambda.req.vtl for request, and default-invoke-lambda.res.vtl for response. Finally, to bring all the pieces together, this is the final configuration of the AppSync plugin: ` Looking above, that's quite a bit of configuration work. It's a smooth ride once you set it up, though. To help you understand which configuration references which, here's a diagram with all the pieces: AppSync Configuration Is Complete With the above configuration in place, the AppSync configuration is complete. You can view the entire code on GitHub. This was by far the most complex part of the migration. In the next and final blog post, we'll be covering some easier topics, like setting up DynamoDB triggers and configuring S3 buckets....
May 20, 2022
How to Use Custom Domain with Serverless: The Perfbuddy API Use Case
Recently, Perfbuddy has transitioned its backend stack to utilize Serverless when managing the AWS Amplify backend, in favor of using pure Amplify. I was brought on near the end of that process to lend a hand, after previously wrapping up migrating another project's backend over to the Serverless framework. Out of the tasks I helped with, one of the bigger challenges came from setting up custom domains for our API endpoints that the frontend would point to. That is, instead of a difficult-to-keep-track-of URL generated by AWS, we can use a custom domain such as api.perfbuddy.com that points to an API Gateway domain where the API routes are set up. Much more developer friendly to remember, _especially_ if you have multiple environments set up for various stages. Fortunately, Serverless has a plugin called Domain Manager that can handle all of that for us. The potential problem? The site had already deployed in a production state inside of another AWS account, and its domain was registered to that account -- separately from where the Serverless backend was being set up. There is a process to migrate that, but having never done it before and the site needing to stay live, I decided to see if I could make it work as it was. Fortunately, I was able to. Prerequisite Before we take a deeper look, if you're following along looking to solve a similar problem and you haven't yet started your Serverless migration, we've discussed that process previously in detail, and you'll definitely want to have the basics already set up for your Serverless deployment instead of doing this all in one go. If you're already deploying with Serverless, you probably also have an IAM role created and set up with the permissions you'll be needing. Just to be sure, you'll be needing these permissions: ` The Problem As I stated previously, the Perfbuddy domain was registered in a separate AWS account than the one our new Serverless deployment was in under our organization account. Fortunately, I had access to both accounts so either a) I could make this work or b) I would have to migrate the domain over and be a nervous wreck the whole time that I'd break something. My first step was just to naively follow the excellent Serverless Domain Manager plugin guide, which got me most of the way there. I added the plugin to the project: ` Then, added the plugin to the the serverless.yml: ` Note: You'll want to make sure you pay attention to the order of your plugins, since for some plugins, the order matters greatly. If you've been paying attention to the order of your plugins already, you probably know whether or not if it's safe to list it first, last, etc. For my purposes, I added it to the end of the list without issue. Next, I needed to add the plugin configuration under the custom field: ` The documentation specifies multiple ways to set up multiple domains in one configuration setup, but for my first attempt, I decided to try to get just one domain setup to see if it worked (more on setting up multiple domains later). I left out a few of the parameters that were listed in the guide, and only added a few that I knew I needed to specify. Most parameters have a default value if not specified, which you can find on the plugin's Github. After running the create command: ` I hit my first real snag: there was no hosted zone set up for the custom domain. The Solution At this step, I feared I would actually potentially have to migrate the domain over. However, when reading the migration documentation, I noticed a specific line in the second optional step that would clue me in on exactly what I needed to know. It reads as follows: > If you're using Route 53 as the DNS service for the domain, Route 53 doesn't transfer the hosted zone when you transfer a domain to a different AWS account. If domain registration is associated with one account and the corresponding hosted zone is associated with another account, neither domain registration nor DNS functionality is affected. The only effect is that you'll need to sign into the Route 53 console using one account to see the domain, and sign in using the other account to see the hosted zone. So, since I had access to both accounts under the same organization, it read to me that AWS would handle the DNS automatically without transferring anything. A hosted zone can live on one account while the main domain registration lived on the other. Perfect. Navigating to the Route 53 console where the Serverless backend was being deployed, I created a hosted zone under the exact domain name I wanted to deploy to (I also created hosted zones for the other specific domains I would need): At this point, the Serverless CLI may have worked with the configuration we specified as is, but I decided to add an additional parameter using the hosted zone ID from the one I just created: ` Now, running the create_domain worked and navigating to the API Gateway console showed the custom domain we just created: Additionally, navigating back to our hosted zone in the Route 53 console, we created shows two additional records the plugin created for us. Finally, running a deploy command: ` Successfully deployed the backend, now with the domain manager plugin info appearing in the output: ` Deploying Multiple Domains Of course, it wasn't just a single API backend I needed to setup; I needed to setup multiple. The problem I saw with using the plugin's built-in capability for multiple domains was that it would try to deploy to all of them at the same time. We needed to deploy to different stages, at different points of development and release independently between them. So, utilizing the Serverless stage parameters I was able to use the same single custom domain structure for the domain manager plugin configuration. You can even specify whole arrays as a parameter, so I did exactly that (with the other stages and parameters added as needed, of course): ` Note: Be sure to take note that you are using the correct YAML formatting and structure for you use case -- at one point, I was using the multiple domains YAML structure, but incorrectly so my deployments weren't actually going through as a result, I believe. I spent far more time resolving that specifically than I needed to. Then, back under the actual plugin configuration, I changed it as follows: ` Now, running any of the commands: ` Deployed to all needed environments separately, and pointing my locally running frontend to api.perfbuddy.com worked like a charm: Conclusion Working in Serverless truly has been a smooth process, and we're constantly blown away with its power that can only succinctly be described as magic. And, of course, if you haven't yet checked out Perfbuddy, be sure to do so! It's completely free and was made to help all developers and their teams improve their products....
May 11, 2022
Migrating a classic Express.js to Serverless Framework
Jan 17, 2022