This Dot Blog
This Dot provides teams with technical leaders who bring deep knowledge of the web platform. We help teams set new standards, and deliver results predictably.
GitHub Actions for Serverless Framework Deployments
Background Our team was building a Serverless Framework API for a client that wanted to use the Serverless Dashboard) for deployment and monitoring. Based on some challenges from last year, we agreed with the client that using a monorepo tool like Nx) would be beneficial moving forward as we were potentially shipping multiple Serverless APIs and frontend applications. Unfortunately, we discovered several challenges integrating with the Serverless Dashboard, and eventually opted into custom CI/CD with GitHub Actions. We’ll cover the challenges we faced, and the solution we created to mitigate our problems and generate a solution. Serverless Configuration Restrictions By default, the Serverless Framework does all its configuration via a serverless.yml file. However, the framework officially supports alternative formats) including .json, .js, and .ts. Our team opted into the TypeScript format as we wanted to setup some validation for our engineers that were newer to the framework through type checks. When we eventually went to configure our CI/CD via the Serverless Dashboard UI, the dashboard itself restricted the file format to just the YAML format. This was unfortunate, but we were able to quickly revert back to YAML as configuration was relatively simple, and we were able to bypass this hurdle. Prohibitive Project Structures With our configuration now working, we were able to select the project, and launch our first attempt at deploying the app through the dashboard. Immediately, we ran into a build issue: ` What we found was having our package.json in a parent directory of our serverless app prevented the dashboard CI/CD from being able to appropriately detect and resolve dependencies prior to deployment. We had been deploying using an Nx command: npx nx run api:deploy --stage=dev which was able to resolve our dependency tree which looked like: To resolve, we thought maybe we could customize the build commands utilized by the dashboard. Unfortunately, the only way to customize these commands is via the package.json of our project. Nx allows for package.json per app in their structure, but it defeated the purpose of us opting into Nx and made leveraging the tool nearly obsolete. Moving to GitHub Actions with the Serverless Dashboard We thought to move all of our CI/CD to GitHub Actions while still proxying the dashboard for deployment credentials and monitoring. In the dashboard docs), we found that you could set a SERVERLESS_ACCESS_KEY and still deploy through the dashboard. It took us a few attempts to understand exactly how to specify this key in our action code, but eventually, we discovered that it had to be set explicitly in the .env file due to the usage of the Nx build system to deploy. Thus the following actions were born: api-ci.yml ` api-clean.yml ` These actions ran smoothly and allowed us to leverage the dashboard appropriately. All in all this seemed like a success. Local Development Problems The above is a great solution if your team is willing to pay for everyone to have a seat on the dashboard. Unfortunately, our client wanted to avoid the cost of additional seats because the pricing was too high. Why is this a problem? Our configuration looks similar to this (I’ve highlighted the important lines with a comment): serverless.ts ` The app and org variables make it so it is required to have a valid dashboard login. This meant our developers working on the API problems couldn’t do local development because the client was not paying for the dashboard logins. They would get the following error: Resulting Configuration At this point, we had to opt to bypass the dashboard entirely via CI/CD. We had to make the following changes to our actions and configuration to get everything 100% working: serverless.ts - Remove app and org fields - Remove accessing environment secrets via the param option ` api-ci.yml - Add all our secrets to GitHub and include them in the scripts - Add serverless confg ` api-cleanup.yml - Add serverless config - Remove secrets ` Conclusions The Serverless Dashboard is a great product for monitoring and seamless deployment in simple applications, but it still has a ways to go to support different architectures and setups while being scalable for teams. I hope to see them make the following changes: - Add support for different configuration file types - Add better support custom deployment commands - Update the framework to not fail on login so local development works regardless of dashboard credentials The Nx + GitHub actions setup was a bit unnatural as well with the reliance on the .env file existing, so we hope the above action code will help someone in the future. That being said, we’ve been working with this on the team and it’s been a very seamless and positive change as our developers can quickly reference their deploys and know how to interact with Lambda directly for debugging issues already....
Sep 26, 2022
5 mins
Migrating an Amplify Backend on Serverless Framework - Part 3
This is Part Three of a three part series on Migrating an Amplify Backend on Serverless Framework. You can find Part One here and Part Two here. This is the third and final part of our series where we're showing the steps needed to migrate an Amplify backend to Serverless Framework. After scaffolding the project in the first part, and setting up the GraphQL API in the second part, what now remains is setting up final touches like DynamoDB triggers and S3 buckets. Let's get to it. DynamoDB Triggers A DynamoDB trigger allows you to invoke a lambda every time a DynamoDB table is updated, and the lambda will receive the updated row in the input event. In our application, we will be using this to add a new notification to the NotificationQueue table every time an Item row is created that has remindAt set. For this purpose, let's create that lambda, which will be just a placeholder since we're focusing mainly on the Serverless configuration. Copy the contents of handlers/process-queue/index.js to handlers/add-to-queue/index.js. This lambda has the following content: ` Now we need to make a slight modification to our Item table resource, to add stream configuration. Having stream configured on the DynamoDB table is a prerequisite for a trigger to be invoked on row modification. The stream is configured by adding a StreamViewType property to the table configuration. The table configuration for the Item resource now becomes: ` The only remaining part is to connect the lambda and stream configuration together. This is done in the functions property of the Serverless configuration: ` We have the standard lambda function definition as well as an events property that hooks up the lambda to the stream of the Item table. Again, we use an intrinsic function, in this case !GetAtt to fetch the ARN (Amazon Resource Name) of the stream. With this in place, lambda is now hooked to the Item data stream and will begin listening to modification events. One such event might look like this: ` Setting up S3 Buckets In case a user of our app would like to upload an image as part of the todo item's note, we could upload that image to an S3 bucket and then serve it from there when displaying the note in the UI. For this to work, we would need to provision an S3 bucket through our Serverless configuration. An S3 bucket is a resource, just like any other DynamoDB table in the configuration. We need to give it a name, so let's configure that in the environment first: ` The S3 bucket name is composed of service name and stage, suffixed by the string "-images". In our case, for dev environment, the bucket would be named amplified-todo-api-dev-images. Now we need to configure the resources for this S3 bucket. We can append the following configuration to the end of the Resources section: ` In the above configuration, we create a resource for the bucket, and a policy specifying public read permissions for that resource. Note how ImageBucketPolicy is referencing ImageBucketResource. We're using intrinsic functions again to avoid hardcoding the image bucket resource name. If we wanted to have a lambda that would upload to this S3 bucket, then we would need to add the permissions for it: ` Our S3 bucket is now set up. Bonus: Lambda Bundling The project in this state is relatively simple and should take less than a couple of minutes to deploy. With time, however, it will probably grow larger, and the lambdas may start to require some external dependencies. The deployment will become slower, and lambda deployments will contain more files than are really necessary. At that point, it will be a good idea to introduce lambda bundling. serverless-esbuild is a plugin that utilizes esbuild to bundle and minify your lambda code. It's almost zero-config, and works out of the box without the need to install any additional plugins. With it, you can have both TypeScript and JavaScript code. To start using it, install it first: ` Now add it to the plugins array: ` Finally, configure it to both bundle and minify your lambdas: ` That's it. Your lambdas will now be bundled and minified on every deploy. Conclusion This is the end of our three-part series on migrating an Amplify backend to Serverless Framework. We hope you enjoyed the journey! Even if you're not migrating from Amplify, these guides should help you configure various services such as AppSync and DynamoDB in Serverless Framework. Don't forget that the entire source code for this project is up on GitHub. Should you need any help, though, with either Amplify or Serverless Framework, please do not hesitate to drop us a line!...
Jun 21, 2022
4 mins
Migrating an Amplify Backend on Serverless Framework - Part 2
This is Part Two of a three part series on Migrating an Amplify Backend on Serverless Framework. You can find Part One here and Part Three here. Welcome to the second part of our "Migrating an Amplify Backend to Serverless Framework", where I will give you a step-by-step guide on how to migrate Amplify-based services so they can be deployable using the Serverless Framework. In the first part, we scaffolded our example application and explained how to deploy Cognito user pools. In this blog post, we'll focus on the core of the application, which is the GraphQL API powered by AppSync. How AppSync Works Before we go deep into the code, it's worth understanding how AppSync actually works. AppSync is a powerful service by AWS that allows you to deploy GraphQL APIs with ease, as well as connect those APIs with data sources like AWS DynamoDB, lambda, and more. It can work without Amplify, but when you pair Amplify with it, you get some extra benefits. Deployment, for example, is way easier. All you need is a GraphQL schema, and Amplify will generate all the required AppSync components for it to work end-to-end. Some of these components are: * Data sources, which are interfaces to other AWS resources such as lambdas and DynamoDB tables. * Resolvers, which connect your GraphQL types and fields to data sources. A resolver specifies how data is mapped between your GraphQL API and your data sources. This mapping is done using mapping templates, and mapping templates are written using Apache Velocity Template Language (VTL). A mapping template contains one template for request, and one for response. * DynamoDB tables, usually one for each GraphQL type. Below, you can see the flow of data when a GraphQL request is made that interacts with a DynamoDB table: A standard GraphQL schema is not enough to generate everything that AppSync needs, though. Amplify needs some additional metadata in the form of Amplify-specific directives which you can place in your GraphQL schema, and Amplify will use the information from the directives to set up everything correctly. Then, during deployment, Amplify will *transform* your schema so that all Amplify-specific directives are stripped, and additional GraphQL types are generated if needed. This is especially important for the @connection directive, which allows you to define 1:1, 1:M, or N:M relationships between models (which are basically GraphQL types annotated with @model directive). Having additional GraphQL types is necessary in order for GraphQL API clients to properly read those relationships. For example, if the Amplify GraphQL schema contains the following type definition: ` Then, this will be transformed to the following in the standard GraphQL schema: ` It's not quite the same, isn't it? Yet, only the latter form can be recognized by AppSync. Hence, a simplified sequence of steps executed during deployment is: 1. Transform your GraphQL schema enriched with Amplify directives to a standard GraphQL schema, which is then uploaded to AppSync. 2. Create/update/delete DynamoDB tables based on GraphQL types annotated with @model 3. Create/update/delete data sources based on directives such as @model and @function 4. Create/update/delete resolvers and connect them to GraphQL schema's fields on one side as well as to data sources on the other side. Resolvers are the "glue" that connects GraphQL fields to data sources. In our Serverless deployment, we'll need to replicate the above deployment steps. Let's get to it. Transforming GraphQL Schema As part of our Amplified ToDo application, we provided a GraphQL schema that will be used for our API. Now, if we tried to deploy the schema to AppSync, with or without Serverless Framework, it would fail because the schema contains Amplify-specific directives. We need to run it through a *transformer* first - the same one that Amplify runs when the application is deployed using Amplify. Fortunately, the Amplify CLI is open-source, and the transformer is part of it, meaning that we can freely use it. Guided by an excellent article by Ronny Roeller, we wrote a script that takes an Amplify GraphQL schema and produces an AppSync-compatible schema. Not only that, it also generates VTL templates along with it, just like Amplify! The script assumes that the Amplify GraphQL schema is located in the amplify-schema.graphql file (we copied it from amplify/backend/api/amplifiedtodo/schema.graphql). In the script, we first create an instance of GraphQLTransform and pass it different transformer instances, each covering one specific Amplify directive. For example, the ModelConnectionTransformer instance will process @connection directives, and the KeyTransformer will process @key directives. Then, we call the GraphQLTransform.transform() method which will output the transformed schema to appsync-schema.graphql. Finally, we go through the generated types and fields to generate the mapping templates for the resolvers. Mapping templates are stored to serverless.mapping-templates.yml file. There are two types of mapping templates in the file: "UNIT" (which is the default) and "PIPELINE". They correspond to unit and pipeline resolvers, respectively. Unit resolvers are used for mapping to a single data source, while pipeline resolvers allow you to invoke a series of functions in a serial manner. In our case, pipeline resolvers are used for invoking lambda-based data sources. Invoking the script is a manual step, and it's best to invoke it before deployment. We made an npm task called deploy that runs the transformer, and then runs sls deploy. This task should be used for deploying from now on as it makes sure that no changes to the GraphQL are left out by accident. Generating GraphQL TypeScript Types Another thing that Amplify does is that it can generate TypeScript files that correspond to GraphQL types. Unfortunately, this is hard to do without the Amplify CLI, as the code generation part is deep in the Amplify CLI and cannot be extracted like the transformer. If you use TypeScript, you'll need to keep the Amplify CLI to perform code generation using the amplify codegen command. You can add this command to the deploy command as well. ` Configuring the AppSync Plugin The final step is configuring the serverless-appsync-plugin plugin so we can deploy our schema to AppSync. Before we do that, we should create the lambdas and DynamoDB tables that our GraphQL API will use. If you recall from the previous blog post, we've defined one Lambda called processQueue under the Mutation type: ` Let's create a placeholder for that Lambda by copying handlers/insert-user-preference to handlers/process-queue and adding it to our Serverless config: ` We will also need to create one DynamoDB table for each GraphQL type, namely: * List * Item * UserPreference * NotificationQueue Let's add the table names to the environment so that they can be passed into our Lambda functions: ` Now, let's create the actual tables under the Resources section: ` Note how we did not need to define all fields that are defined in the schema. DynamoDB is schemaless, meaning that you can provision arbitrary columns to table rows. We only need to define the columns that are primary key columns as well as those covered by indexes. Indexes are not created automatically by the serverless-appsync-plugin - you need to define manually in the Resources section. Next, we need to create data sources. Create a file called serverless.data-sources.yml with the following contents: ` The above file defines data sources for our lambdas and DynamoDB tables. The names of data sources are not arbitrary. They need to match the names that were generated by the transformer script in the serverless.mapping-templates.yml file. Also, each data source references a table by using the Ref intrinsic function to find the table name by resource name (defined under Resources section of the main Serverless config). The final piece are the function configs, which is a configuration part of the serverless-appsync-plugin that is required only for pipeline resolvers. In the serverless.mapping-templates.yml file, each mapping template for a pipeline resolver refers to a function config. For example, the pipeline resolver for the processQueue lambda references a function config named InvokeProcessQueueLambdaDataSource. The function config provides a request/response mapping, also written in VTL templates, for the lambda in question. Create a file named serverless.appsync-function-configs.yml, with the following contents: ` Create default-invoke-lambda.req.vtl inside the mapping-templates directory, with the following contents: ` Likewise, create default-invoke-lambda.res.vtl with the following contents: ` As you can see, the VTL request/response templates for pipeline resolvers are not complicated, and they are same for every pipeline resolver if there are more of them. For that reason, all pipeline resolvers can reference the same files: default-invoke-lambda.req.vtl for request, and default-invoke-lambda.res.vtl for response. Finally, to bring all the pieces together, this is the final configuration of the AppSync plugin: ` Looking above, that's quite a bit of configuration work. It's a smooth ride once you set it up, though. To help you understand which configuration references which, here's a diagram with all the pieces: AppSync Configuration Is Complete With the above configuration in place, the AppSync configuration is complete. You can view the entire code on GitHub. This was by far the most complex part of the migration. In the next and final blog post, we'll be covering some easier topics, like setting up DynamoDB triggers and configuring S3 buckets....
May 20, 2022
8 mins
How to Use Custom Domain with Serverless: The Perfbuddy API Use Case
Recently, Perfbuddy has transitioned its backend stack to utilize Serverless when managing the AWS Amplify backend, in favor of using pure Amplify. I was brought on near the end of that process to lend a hand, after previously wrapping up migrating another project's backend over to the Serverless framework. Out of the tasks I helped with, one of the bigger challenges came from setting up custom domains for our API endpoints that the frontend would point to. That is, instead of a difficult-to-keep-track-of URL generated by AWS, we can use a custom domain such as api.perfbuddy.com that points to an API Gateway domain where the API routes are set up. Much more developer friendly to remember, _especially_ if you have multiple environments set up for various stages. Fortunately, Serverless has a plugin called Domain Manager that can handle all of that for us. The potential problem? The site had already deployed in a production state inside of another AWS account, and its domain was registered to that account -- separately from where the Serverless backend was being set up. There is a process to migrate that, but having never done it before and the site needing to stay live, I decided to see if I could make it work as it was. Fortunately, I was able to. Prerequisite Before we take a deeper look, if you're following along looking to solve a similar problem and you haven't yet started your Serverless migration, we've discussed that process previously in detail, and you'll definitely want to have the basics already set up for your Serverless deployment instead of doing this all in one go. If you're already deploying with Serverless, you probably also have an IAM role created and set up with the permissions you'll be needing. Just to be sure, you'll be needing these permissions: ` The Problem As I stated previously, the Perfbuddy domain was registered in a separate AWS account than the one our new Serverless deployment was in under our organization account. Fortunately, I had access to both accounts so either a) I could make this work or b) I would have to migrate the domain over and be a nervous wreck the whole time that I'd break something. My first step was just to naively follow the excellent Serverless Domain Manager plugin guide, which got me most of the way there. I added the plugin to the project: ` Then, added the plugin to the the serverless.yml: ` Note: You'll want to make sure you pay attention to the order of your plugins, since for some plugins, the order matters greatly. If you've been paying attention to the order of your plugins already, you probably know whether or not if it's safe to list it first, last, etc. For my purposes, I added it to the end of the list without issue. Next, I needed to add the plugin configuration under the custom field: ` The documentation specifies multiple ways to set up multiple domains in one configuration setup, but for my first attempt, I decided to try to get just one domain setup to see if it worked (more on setting up multiple domains later). I left out a few of the parameters that were listed in the guide, and only added a few that I knew I needed to specify. Most parameters have a default value if not specified, which you can find on the plugin's Github. After running the create command: ` I hit my first real snag: there was no hosted zone set up for the custom domain. The Solution At this step, I feared I would actually potentially have to migrate the domain over. However, when reading the migration documentation, I noticed a specific line in the second optional step that would clue me in on exactly what I needed to know. It reads as follows: > If you're using Route 53 as the DNS service for the domain, Route 53 doesn't transfer the hosted zone when you transfer a domain to a different AWS account. If domain registration is associated with one account and the corresponding hosted zone is associated with another account, neither domain registration nor DNS functionality is affected. The only effect is that you'll need to sign into the Route 53 console using one account to see the domain, and sign in using the other account to see the hosted zone. So, since I had access to both accounts under the same organization, it read to me that AWS would handle the DNS automatically without transferring anything. A hosted zone can live on one account while the main domain registration lived on the other. Perfect. Navigating to the Route 53 console where the Serverless backend was being deployed, I created a hosted zone under the exact domain name I wanted to deploy to (I also created hosted zones for the other specific domains I would need): At this point, the Serverless CLI may have worked with the configuration we specified as is, but I decided to add an additional parameter using the hosted zone ID from the one I just created: ` Now, running the create_domain worked and navigating to the API Gateway console showed the custom domain we just created: Additionally, navigating back to our hosted zone in the Route 53 console, we created shows two additional records the plugin created for us. Finally, running a deploy command: ` Successfully deployed the backend, now with the domain manager plugin info appearing in the output: ` Deploying Multiple Domains Of course, it wasn't just a single API backend I needed to setup; I needed to setup multiple. The problem I saw with using the plugin's built-in capability for multiple domains was that it would try to deploy to all of them at the same time. We needed to deploy to different stages, at different points of development and release independently between them. So, utilizing the Serverless stage parameters I was able to use the same single custom domain structure for the domain manager plugin configuration. You can even specify whole arrays as a parameter, so I did exactly that (with the other stages and parameters added as needed, of course): ` Note: Be sure to take note that you are using the correct YAML formatting and structure for you use case -- at one point, I was using the multiple domains YAML structure, but incorrectly so my deployments weren't actually going through as a result, I believe. I spent far more time resolving that specifically than I needed to. Then, back under the actual plugin configuration, I changed it as follows: ` Now, running any of the commands: ` Deployed to all needed environments separately, and pointing my locally running frontend to api.perfbuddy.com worked like a charm: Conclusion Working in Serverless truly has been a smooth process, and we're constantly blown away with its power that can only succinctly be described as magic. And, of course, if you haven't yet checked out Perfbuddy, be sure to do so! It's completely free and was made to help all developers and their teams improve their products....
May 11, 2022
6 mins
Migrating a classic Express.js to Serverless Framework
Problem Classic Express.js applications are great for building backends. However, their deployment can be a bit tricky. There are several solutions on the market for making deployment "easier" like Heroku, AWS Elastic Beanstalk, Qovery, and Vercel. However, "easier" means special configurations or higher service costs. In our case, we were trying to deploy an Angular frontend served through Cloudfront, and needed a separately deployed backend to manage an OAuth flow. We needed an easy to deploy solution that supported HTTPS, and could be automated via CI. Serverless Framework The Serverless Framework is a framework for building and deploying applications onto AWS Lambda, and it allowed us to easily migrate and deploy our Express.js server at a low cost with long-term maintainability. This was so simple that it only took us an hour to migrate our existing API, and get it deployed so we could start using it in our production environment. Serverless Init Script To start this process, we used the Serverless CLI to initialize a new Serverless Express.js project. This is an example of the settings we chose for our application: ` Here's a quick explanation of our choices: What do you want to make? This prompt offers several possible scaffolding options. In our case, the Express API was the perfect solution since that's what we were migrating. What do you want to call this project? You should put whatever you want here. It'll name the directory and define the naming schema for the resources you deploy to AWS. What org do you want to add this service to? This question assumes you are using the serverless.com dashboard for managing your deployments. We're choosing to use Github Actions and AWS tooling directly though, so we've opted out of this option. Do you want to deploy your project? This will attempt to deploy your application immediately after scaffolding. If you don't have your AWS credentials configured correctly, this will use your default profile. We needed a custom profile configuration since we have several projects on different AWS accounts so we opted out of the default deploy. Serverless Init Output The init script from above outputs the following: - .gitignore - handler.js - package.json - README.md - serverless.yml The key here is the serverless.yml and handler.js files that are outputted. serverless.yml ` handler.js ` As you can see, this gives a standard Express server ready to just work out of the box. However, we needed to make some quality of life changes to help us migrate with confidence, and allow us to use our API locally for development. Quality of Life Improvements There are several things that Serverless Framework doesn't provide out of the box that we needed to help our development process. Fortunately, there are great plugins we were able to install and configure quickly. Environment Variables We need per-environment variables as our OAuth providers are specific per host domain. Serverless Framework supports .env files out of the box but it does require you to install the dotenv package and to turn on the useDotenv flag in the serverless.yml. Babel/TypeScript Support As you can see in the above handler.js file, we're getting CommonJS instead of modern JavaScript or TypeScript. To get these, you need webpack or some other bundler. serverless-webpack exists if you want full control over your ecosystem, but there is also serverless-bundle that gives you a set of reasonable defaults on webpack 4 out of the box. We opted into this option to get us started quickly. Offline Mode With classic Express servers, you can use a simple node script to get the server up and running to test locally. Serverless wants to be run in the AWS ecosystem making it. Lucky for us, David Hérault has built and continues to maintain serverless-offline allowing us to emulate our functions locally before we deploy. Final Configuration Given these changes, our serverless.yml file now looks as follows: ` Some important things to note: - The order of serverless-bundle and serverless-offline in the plugins is critically important. - The custom port for serverless-offline can be any unused port. Keep in mind what port your frontend server is using when setting this value for local development. - We set the profile and stage in our provider configuration. This allowed us to use specify the environment settings and AWS profile credentials to use for our deployment. With all this set, we're now ready to deploy the basic API. Deploying the new API Serverless deployment is very simple. We can run the following command in the project directory: ` This command will deploy the API to AWS, and create the necessary resources including the API Gateway and related Lambdas. The first deploy will take roughly 5 minutes, and each subsequent deply will only take a minute or two! In its output, you'll receive a bunch of information about the deployment, including the deployed URL that will look like: ` You can now point your app at this API and start using it. Next Steps A few issues we still have to resolve but are easily fixed: - New Lambdas are not deploying with their Environment Variables, and have to be set via the AWS console. We're just missing some minor configuration in our serverless.yml. - Our deploys don't deploy on merges to main. For this though, we can just use the official Serverless Github Action. Alternatively, we could purchase a license to the Serverless Dashboard, but this option is a bit more expensive, and we're not using all of its features on this project. However, we've used this on other client projects, and it really helped us manage and monitor our deployments. Conclusion Given all the above steps, we were able to get our API up and running in a few minutes. And due to it being a 1-to-1 replacement for an existing Express server, we were able to port our existing implementations into this new Serverless implementation, deploy it to AWS, and start using it in just a couple of hours. This particular architecture is a great means for bootstraping a new project, but it does come with some scaling issues for larger projects. As such, we do not recommend a pure Serverless, and Express for monolithic projects, and instead suggest utilizing some of the amazing capabilities of Serverless Framework with AWS to horizontally scale your application into smaller Lambda functions....
Jan 17, 2022
5 mins