How to use AWS Lambda Layers with SAM

There are many nuances related to AWS Lambda layers that are declared in SAM templates. However, there are also some fixed must-haves that always need to be done in order for AWS Layer to work in both local and cloud environments alike. For those in a hurry or coming back to peek at what is most important, just follow the Golden Rules.

I advise reading the whole article at least once, as there are plenty of possibilities and nuances. At the end there is a link to GitHub repository with comprehensive solution.

The presented solution, including a step-by-step guide as well as a GitHub repository project, exaplains the process of using Lambda Layers with NPM libraries and custom code in both local and cloud environments as well in nested templates. 

To ensure the explanation is as clear and solution-focused as possible, I selected the simple Slugify library, which converts any text into a slug suitable for URLs. Additionally, custom code within the Layers replicates Slugify functionality without requiring an NPM library, offering insight into how custom code should be integrated within a Lambda Layer.

GOLDEN RULES

There are four rules you must follow in order to effectively use lambda layers in both local and cloud development, as well as in nested templates:

1. Create layer resources and reference them in lambda functions.
2. Mark layer dependencies as external for Lambdas.
3. In your Lambda function folder, use tsconfig.json to declare paths linking the layer with its build version.
4. When using nested templates, employ a combination of parameters and conditions, and redeclare layers for local development inside the child template.

Project Structure with Lambda Layers

Instead of the default structure of one Lambda per folder at the project root, it's easier to maintain a project with layers when there is a clear separation between Lambdas and Layers in the root of the project.

Thus, each Lambda should be placed in a Lambdas folder, and within that, a specific folder named after the Lambda for separation from other Lambdas.

Similarly, Layers should be placed in a Layers folder, and within that, a specific folder named after the Layer for separation from other Layers.

Throughout this article, as well as in the GitHub repository containing the entire code from this tutorial, this project structure will be emphasized.

Create a Custom Lambda Layer

Begin by focusing on the Lambda Layer since it serves as a dependency for the Lambda function. This approach simplifies the declaration of the full Lambda code later on at once.

Create a Text Processor Layer

Directly create a text-processor-layer folder within the layers folder. Contrary to some misconceptions, an intermediate nodejs folder is not required. In my opinion, a simpler structure is better if it works effectively.

Install the Slugify Library

npm install slugify

Create Custom Code to Mimic Slugify Functionality

To understand how to declare custom code from a Lambda Layer within a Lambda, let's create a custom code example that mimics the functionality of slugify. Create a file named customCode.ts and insert the following code, which replicates the mentioned functional

export const customSlugify = (text: string): string => {
    return text.toString().toLowerCase()
        .replace(/\s+/g, '-')
        .replace(/[^\w\-]+/g, '')
        .replace(/\-\-+/g, '-')
        .replace(/^-+/, '')
        .replace(/-+$/, '');
};

Setting Up Lambda Functions with a Custom Layer

Create a Slugify Lambda Function

Inside the lambdas folder, create a slugify directory. Within this directory, create an app.ts file and paste the following code:

import Slugify from "/opt/nodejs/node_modules/slugify";

export const slugifyFunction = async (): Promise<{ body: string; statusCode: number }> => {
    try {
        return {
            statusCode: 200,
            body: JSON.stringify({
                message: Slugify('hello world'),
            }),
        };
    } catch (err) {
        console.log(err);
        return {
            statusCode: 500,
            body: JSON.stringify({
                message: 'some error happened',
            }),
        };
    }
};

Notice the import path for Slugify. It's intended to mirror the execution environment's file structure, not the local development structure. The Slugify library is assumed to be available in the /opt/nodejs directory when the Lambda function is executed both in local and cloud environments.

Create a Custom Slugify Function

For the second Lambda function, which uses the customSlugify function from the Layer instead of the Slugify library, follow a similar setup.

Name this Lambda function custom-slugify and place it in a correspondingly named folder. Paste the following code:

import {customSlugify} from "/opt/nodejs/customCode";

export const customSlugifyFunction = async (): Promise<{ body: string; statusCode: number }> => {
    try {
        return {
            statusCode: 200,
            body: JSON.stringify({
                message: customSlugify('hello world'),
            }),
        };
    } catch (err) {
        console.log(err);
        return {
            statusCode: 500,
            body: JSON.stringify({
                message: 'some error happened',
            }),
        };
    }
};

This demonstrates how to import custom implementations differently, directly from /opt/nodejs/customCode, without navigating through node_modules.

Address TypeScript Configuration for Layers

To preserve typing support in TypeScript, which is crucial for development, adjust the TypeScript configuration as follows. In each Lambda function's directory, create a tsconfig.json file (not tsconfig.ts) with the content below. Adjust the paths as necessary for different Lambda functions:

{
    "compilerOptions": {
      "target": "es2020",
      "strict": true,
      "preserveConstEnums": true,
      "noEmit": true,
      "sourceMap": false,
      "module":"es2015",
      "moduleResolution":"node",
      "esModuleInterop": true,
      "skipLibCheck": true,
      "forceConsistentCasingInFileNames": true,
      "paths": {
        "/opt/nodejs/node_modules/slugify": [
          "../../layers/text-processor-layer/node_modules/slugify"
        ],
      }
    },
    "exclude": ["node_modules", "**/*.test.ts"]
  }

Infrastructure as Code (Iac) with SAM Templates

Finally, declare your infrastructure as code using AWS SAM (Serverless Application Model) templates (AWS CloudFormation templates on steroids). Proceed with the following steps to implement your infrastructure using SAM.

Create an AWS SAM Template for Custom Lambda Layers

To begin, create a template.yaml file in the root folder of your project and include the following content:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  lambda-layers-with-sam
Parameters:
  EnvironmentType:
    Type: String
    Default: 'Development'
Resources:
  TextProcessorLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: TextProcessorLayer
      Description: Text processor dependencies
      ContentUri: 'layers/text-processor-layer/'
      CompatibleRuntimes:
        - nodejs18.x
    Metadata:
      BuildMethod: nodejs18.x
  MainAPI:
    Type: AWS::Serverless::HttpApi
    Properties:
      Name: API
      StageName: !Ref EnvironmentType
      DefaultRouteSettings:
        DetailedMetricsEnabled: true
        ThrottlingBurstLimit: 5
        ThrottlingRateLimit: 5
      CorsConfiguration:
        AllowMethods:
          - GET
          - OPTIONS
          - POST
          - PUT
          - HEAD
          - PATCH
          - DELETE
        AllowHeaders:
          - "*"
        AllowOrigins:
          - "*"
      FailOnWarnings: true
  SlugifyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/slugify
      Handler: app.slugifyFunction
      Layers:
        - !Ref TextProcessorLayer
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /slugify/{slug}
            Method: get
            ApiId:
              Ref: MainAPI
            Auth:
              Authorizer: NONE
      Runtime: nodejs18.x
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        Sourcemap: true
        EntryPoints:
          - app.ts
        External:
          - "/opt/nodejs/node_modules/slugify"
  CustomSlugifyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/custom-slugify
      Handler: app.customSlugifyFunction
      Layers:
        - !Ref TextProcessorLayer
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /custom-slugify/{slug}
            Method: get
            ApiId:
              Ref: MainAPI
            Auth:
              Authorizer: NONE
      Runtime: nodejs18.x
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        Sourcemap: true
        EntryPoints:
          - app.ts
        External:
          - "/opt/nodejs/customCode"

For the purpose of this tutorial, I have additionally declared an API Gateway to simplify testing. It's more convenient to test Lambda functions when a local web development server uses localhost to serve the entire API, allowing tools like Postman to call any endpoint, as opposed to invoking selected Lambdas through the SAM API. 

To enhance productivity, I've prepared a custom script to automate the SAM build and sam local start-api command, as detailed in subsequent paragraphs.

Let's briefly discuss the resources declared in the SAM template, focusing on Lambdas and custom layers.

For the SlugifyFunction (which utilizes the Slugify library) and CustomSlugifyFunction (which uses custom slugify functions), there are Layers properties that reference the TextProcessorLayer declared in the same template (hence the !Ref prefix). However, it's not enough; you also need to declare the imported library and customCode as external dependencies in the Metadata section:

Metadata:
  BuildMethod: esbuild
  BuildProperties:
    Minify: true
    Target: "es2020"
    Sourcemap: true
    EntryPoints:
      - app.ts
    External:
      - "/opt/nodejs/node_modules/slugify"

Failure to specify these external dependencies correctly will result in the following build error

Build Failed
Error: NodejsNpmEsbuildBuilder:EsbuildBundle - Esbuild Failed: ✘ [ERROR] Could not resolve "/opt/nodejs/node_modules/*"

After using the sam build command for building your project, you might encounter an issue when starting the API locally with sam local start-api and calling the custom-slugify endpoint. The error displayed could look something like this:

{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '/opt/nodejs/customCode'\nRequire stack:\n- /var/task/app.js\n- /var/runtime/index.mjs",
    "trace": [
        "Runtime.ImportModuleError: Error: Cannot find module '/opt/nodejs/customCode'",
        "Require stack:",
        "- /var/task/app.js",
        "- /var/runtime/index.mjs",
        "    at _loadUserApp (file:///var/runtime/index.mjs:1087:17)",
        "    at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)",
        "    at async start (file:///var/runtime/index.mjs:1282:23)",
        "    at async file:///var/runtime/index.mjs:1288:1"
    ]
}

This issue arises because the custom code, unlike the NPM slugify module, has not been compiled. In your build folder within the TextProcessorLayer, you will find the customCode.ts file, which should be compiled to a .js file.

To resolve this, it's necessary to compile the custom functions. It is debatable whether it's better to compile before running sam build or after sam build but before executing sam local start-api or sam deploy. In my opinion, compiling after the build is more efficient in terms of code structure and project management, as you don't need the JavaScript file for development or in your Git repository.

You can use either the tsc (TypeScript compiler) or esbuild tools for compilation. For example, while operating from the root folder, you can compile the TypeScript files in the TextProcessorLayer in the following manner using tsc:

Install dependencies (it's a good practice to do this at the root of the folder)

npm install typescript --save-dev
npm install @types/node --save-dev

Compile the Custom Lambda Layer

Next, compile the TextProcessorLayer. This time, I am building the development version for simplicity. In the next paragraph, you will find a script ready to compile any layer encountered in the build folder:

npx tsc ./layers/text-processor-layer/*.ts

This approach ensures that your custom TypeScript code is properly compiled and ready for deployment or local testing with AWS SAM

Build an AWS SAM Template with Custom Layers

To build your SAM template effectively, especially when it includes custom layers with TypeScript files, follow this workflow:

Build the SAM Application:

sam build

This command compiles your application, preparing it for local testing or deployment.

Compile TypeScript in Custom Layers

node ./scripts/ts_compiler.mjs

Note: This step is necessary only if you have custom code in your layers. Otherwise, TypeScript modules should already be compiled to JavaScript.

Start the API Locally

sam local start-api

This allows you to test your API locally before deploying it to AWS.

To simplify and automate the compilation of any layers containing TypeScript, I've created a script (used in the step two in the above flow), which you can find in the scripts folder. This script, ts_compiler.mjs, automates the compilation process for layers within the build folder:

import { exec } from "child_process";
import fs from 'fs/promises';
import * as path from "path";
async function compileTS(dir) {
   const dirents = await fs.readdir(dir, { withFileTypes: true });
   for (const dirent of dirents) {
       const res = path.resolve(dir, dirent.name);
       if (dirent.isDirectory() && !res.includes('node_modules')) {
           await compileTS(res);
       } else if (dirent.name.includes('.ts') && res.includes('Layer')) {
           await execShellCommand(`npx tsc ${res}`);
           console.log(dirent.name);
       }
   }
}
function execShellCommand(cmd) {
   return new Promise((resolve, reject) => {
       exec(cmd, (error, stdout, stderr) => {
           if (error) {
               console.warn(error);
           }
           resolve(stdout ? stdout : stderr);
       });
   });
}
(async () => {
   const buildDirectory = process.argv[2] || '.aws-sam/build';
   await compileTS(buildDirectory);
})();

This script traverses the custom build folder (.aws-sam/build) or any directory provided as an argument, compiling TypeScript files in folders named with a "Layer" suffix (e.g., TextProcessorLayer).
 

Additionally, to streamline the entire build and local start-up process, I've written another script in Bash, also located in the scripts folder:

#!/bin/bash
sam build && node ./scripts/ts_compiler.mjs && sam local start-api -n env.json --debug

This script includes additional arguments to assist in development and is prepared for the next challenge related to nested templates, as indicated by the -n env.json flag.

How to use custom layers with nested templates in SAM that work in local development

In AWS SAM, passing custom layer ARNs directly and using them in child templates isn't natively supported for local development as it is for cloud environments. However, there's a workaround to utilize custom layers in child templates without duplicating layers in the cloud. This requires introducing an environment type to differentiate layer handling between cloud and local environments.

Environment Configuration

Create an env.json file at the root of your project with the following content. This file passes the Development value as the environment to the root SAM template:

{
  "Parameters": {
    "EnvironmentType": "Development"
  }
}

Child Template Configuration

Introduce a child template that specifies a local custom layer, which will not be created in production or cloud environments:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  lambda-layers-with-samsligidy/12
Parameters:
  EnvironmentType:
    Type: String
  GlobalTextProcessorLayer:
    Type: String
Conditions:
  IsProd: !Equals [!Ref EnvironmentType, Production]
  IsLocal: !Equals [!Ref EnvironmentType, Development]
Resources:
  # this layer will only be generated in local Environment
  LocalTextProcessorLayer:
    Condition: IsLocal
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: LocalTextProcessorLayer
      Description: Text processor dependencies
      ContentUri: './layers/text-processor-layer/'
      CompatibleRuntimes:
        - nodejs18.x
    Metadata:
      BuildMethod: nodejs18.x
  ChildAPI:
    Type: AWS::Serverless::HttpApi
    Properties:
      Name: API
      StageName: !Ref EnvironmentType
      DefaultRouteSettings:
        DetailedMetricsEnabled: true
        ThrottlingBurstLimit: 20
        ThrottlingRateLimit: 20
      CorsConfiguration:
        AllowMethods:
          - GET
          - OPTIONS
          - POST
          - PUT
          - HEAD
          - PATCH
          - DELETE
        AllowHeaders:
          - "*"
        AllowOrigins:
          - "*"
      FailOnWarnings: true
  ChildSlugifyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/slugify
      Handler: app.slugifyFunction
      Layers: !If
        - IsProd
        - - !Ref GlobalTextProcessorLayer
        - - !Ref LocalTextProcessorLayer
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /child/slugify/{slug}
            Method: get
            ApiId:
              Ref: ChildAPI
            Auth:
              Authorizer: NONE
      Runtime: nodejs18.x
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        Sourcemap: true
        EntryPoints:
          - app.ts
        External:
          - "/opt/nodejs/node_modules/slugify"
  ChildCustomSlugifyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/custom-slugify
      Handler: app.customSlugifyFunction
      Layers: !If
        - IsProd
        - - !Ref GlobalTextProcessorLayer
        - - !Ref LocalTextProcessorLayer
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /child/custom-slugify/{slug}
            Method: get
            ApiId:
              Ref: ChildAPI
            Auth:
              Authorizer: NONE
      Runtime: nodejs18.x
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        Sourcemap: true
        EntryPoints:
          - app.ts
        External:
          - "/opt/nodejs/customCode"

This setup uses the IsLocal condition, which depends on the EnvironmentType parameter being Development, to ensure the layer is only used locally.

Local API Testing Command

To start the API locally with custom configurations, use the following command:

sam local start-api --warm-containers EAGER -p 8095 -n env.json --debug

Root Template Integration

Ensure the root template integrates the child template as an application to achieve true nesting:

ChildApiGateway:
  Type: AWS::Serverless::Application
  DependsOn: [ MainAPI ]
  Properties:
    Location: child_template.yaml
    Parameters:
      EnvironmentType: !Ref EnvironmentType
      GlobalTextProcessorLayer: !Ref TextProcessorLayer

When you utilize the auto_start.sh script, the project will build custom layers both in the root and in nested templates locally, ensuring there is no duplication in the cloud (production) environment.

However, attempting to pass a custom layer ARN directly to a nested template will result in the following error:

2024-02-14 16:57:45,250 | Exception raised during the execution                                                                                                         
2024-02-14 16:57:45,251 | Lambda functions containers initialization failed because of TextProcessorLayer is an Invalid Layer Arn.

This error indicates the need for a proper configuration to ensure that custom layers are correctly referenced and utilized across your SAM application, especially when dealing with nested templates

Do you really need lamba layers?

Lambda layers can be a double-edged sword, offering significant benefits but also presenting challenges. Let's delve into what Lambda layers can accomplish and why they might be essential or advantageous for your project.

Capabilities and Benefits of Lambda Layers:

Share Libraries Across Lambda Functions

• Lambda layers enable the sharing of libraries and dependencies across multiple Lambda functions, facilitating reuse across different projects within the same AWS account.

Reduce Lambda Function Size

• By externalizing libraries and dependencies to layers, the size of Lambda functions can be reduced. This not only streamlines deployment but also potentially improves cold start times.

Ensure Environment Consistency

• Lambda layers can help maintain consistency across environments by providing a centralized way to manage global version control of libraries and dependencies.

Accelerate Development

• Developers can iterate and deploy changes faster, as layers allow for the easy management and update of shared libraries without modifying each Lambda function individually.

Override Default Packages

• Layers offer the flexibility to overlap or replace default packages available in the Lambda execution environment, such as the AWS SDK. This makes it easy to upgrade or downgrade libraries globally across all functions that utilize the layer.

Simplify the Use of Large Packages

• Some larger packages, such as Sharp for image processing, are more conveniently managed and deployed through layers, avoiding the complexities of bundling them directly with your Lambda function's deployment package.

Serve as a Custom Runtime, Data, or Configuration Store

• Beyond libraries, layers can be used to provide a custom runtime or to store data and configuration files, further extending the versatility of your Lambda functions.

Summary and BONUS TIP

You've learned the process of creating Lambda layers in Node.js and TypeScript, which can be seamlessly integrated as dependencies in other Lambda functions and tested locally without build issues.

Further Insights

Lambda layers offer additional benefits, such as the ability to overlap default installed packages like aws-sdk and simplify the installation of complex packages like sharp. For instance, while aws-sdk is pre-installed in the Lambda environment, using a specific version through a layer can circumvent the limitations of the managed version. Conversely, attempting to install sharp directly might lead to errors, showcasing the practicality of layers for managing dependencies.

Using aws-sdk and sharp as layers not only demonstrates these advantages but also highlights considerations like bundle size limits and the importance of managing dependencies efficiently within the constraints of Lambda environments.

Project Code Repository

This article is based on a specific micro-project that demonstrates the implementation of custom Lambda layers using npm packages and custom code modules. Additionally, it incorporates the use of nested templates for enhanced project structure and deployment strategies. To provide complete transparency and to aid in understanding, the entire codebase for this project has been made available in a public repository named aws-sam-custom-layers.