How to run multiple API Gateway instances locally using SAM

When trying to run multiple API Gateway instances locally with SAM, you may encounter numerous issues, especially when invoking API Gateways using the warm containers parameter.

sam build  --paralle && sam local start-api --warm-containers EAGER  -p 8090 --docker-network blowstack-private -n env.json

The exact issues depend on the project itself, the resources and parameters you declare, and whether you use Layers, etc. Personally, I encountered the following issues when I tried to start multiple independent API Gateways locally with warm containers:

1. Source code has been changed; terminate its warm container. The new container will be created in lazy mode
2. Lambda function container initialization failed due to an error (UnrecognizedClientException) when calling the GetLayerVersion operation: The security token included in the request is invalid
3. Lambda function container initialization failed because of 'default_config_resolver'

However, all the issues listed above in my case were related to the way SAM builds and runs templates with warm containers.

Prerequisites

If you have already decoupled API Gateway into separate templates (can be nested templates) or separate projects, skip to the Build and Run paragraph. We need decoupled APIs because each of them will be built and run separately, so we can exclude any interference.

However, if you have multiple API Gateways in one template, you need to separate each API Gateway into a nested templates to run them in parallel locally without interference.

Nested Templates Structure

Remove any API Gateways from the root template so that the root template will serve only the purpose of bonding resources together in the cloud. Locally, you will omit the root template altogether and run each API Gateway in a nested template separately. Take a look at the example below.

Root Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  BlowStack Multi API Gateway
Resources:
  Service1ApiGateway:
    Type: AWS::Serverless::Application
    Properties:
      Location: template_service1.yaml
  Service2ApiGateway:
    Type: AWS::Serverless::Application
    Properties:
      Location: template_service2.yaml

Service 1 Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  BlowStack Service 1 example
Resources:
  Service1API:
    Type: AWS::Serverless::HttpApi
    Properties:
      Name: Service 1 API
      StageName: 'dev'
      CorsConfiguration:
        AllowMethods:
          - GET
          - OPTIONS
          - POST
          - PUT
          - HEAD
          - PATCH
          - DELETE
        AllowHeaders:
          - "*"
        AllowOrigins:
          - "*"
      FailOnWarnings: true
  Example1Function:
    Type: AWS::Serverless::Function
    Properties:
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /example1/{id}
            Method: post
            ApiId:
              Ref: Service1API
      Runtime: nodejs18.x
      Handler: handler.example1Function
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        EntryPoints:
          - "./lambdas/service1/handler.js"

Service 2 template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  BlowStack Service 2 example
Resources:
  Service1API:
    Type: AWS::Serverless::HttpApi
    Properties:
      Name: Service 2 API
      StageName: 'dev'
      CorsConfiguration:
        AllowMethods:
          - GET
          - OPTIONS
          - POST
          - PUT
          - HEAD
          - PATCH
          - DELETE
        AllowHeaders:
          - "*"
        AllowOrigins:
          - "*"
      FailOnWarnings: true
  Example2Function:
    Type: AWS::Serverless::Function
    Properties:
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /example2/{id}
            Method: post
            ApiId:
              Ref: Service2API
      Runtime: nodejs18.x
      Handler: handler.example1Function
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        EntryPoints:
          - "./lambdas/service2/handler.js"

Build and Run

In the example, we have two API Gateways in two different templates. Let's build each of them separately, providing the exact location of templates and, importantly, different locations for build outputs. Notice that the parallel parameter is not required but greatly increases build speed for larger APIs.

// service 1
sam build --template_service1.yaml --parallel --build-dir .aws-sam-service1/build

// service 2
sam build --template_service2.yaml --parallel --build-dir .aws-sam-service2/build

Thus, we have 100% assurance that none of the services will update the build of the other.

Next, run the API Gateways, each in a separate terminal or in the background (which IMHO hinders debugging). What is important this time is to pass the template parameter as the output target (notice that each path has to refer to template.yaml) and provide different ports where APIs will be available on your localhost.

// service 1
sam local start-api --warm-containers EAGER --docker-network blowstack-private  -n env.json -p 8090 --template .aws-sam-service1/build/template.yaml --skip-pull-image --debug

// service 2
sam local start-api --warm-containers EAGER --docker-network blowstack-private  -n env.json -p 8091 --template .aws-sam-service2/build/template.yaml --skip-pull-image --debug

Summary

By having separate build folders and different ports where APIs run, you will have full assurance that running APIs won't interfere with each other. However, if you drop warm containers (which greatly negatively impacts local performance), you can just build the whole project from the root template if it's not dispersed into smaller microservices.