BlowStack logo

How to use AWS Lambda Layers with SAM

How to create a project structure and template with layers that work in local TypeScript and NodeJS environment.

GOLDEN RULES

 

There are 4 rules you have to follow in order to use lambda layers:

  • in local development
  • in the same template as the rest of resources

 

RULE #1

Nodejs folder have to be created as a parent folder for layer dependencies (node_modules).

RULE #2

Create layer resources and reference them in lambda functions.

RULE #3

Mark layer dependencies as external during build

RULE #4

Build your project using containers: sam build --use--container

 

Read further if you need more explanation.

 

Do you really need lamba layers?

 

Using lambda layers can be troublesome but at the same time beneficial or even necessary in a project.

So what can lambda layers do in the first place?

  • share libraries between different lambda functions (in different projects, the same account)
    - lower lambdas size
    - environment consistency (global version control)
    - faster development
     
  • can overlap default installed packages (in lambda environment) like aws-sdk
    - easy to upgrade or downgrade any library/package globally
     
  • some big packages like Sharp are easier to install that way
  • can be utilized as a custom runtime, data or configuration file


This post is focused on layers as a library.

 

Project structure with lambda layers

 

Basically there are two options that you can choose from.

Keep or not keep layers in the same SAM template where you define other resources.

The main reason for not keeping layers in the same template is when you are going to use them in different projects.
Then even keeping them in different repositories makes a lot of sense.
Unfortunately there are many people pointing out technical difficulties as the reason to always choose a multi template option.

For example one of the articles about nested SAM stacks states that:


“When working with a nested SAM stack, you cannot import a Lambda Layer exported from another stack.”

 

But it's not true.
You can nest layers in SAM templates and test them locally as dependencies as long as you stick to some rules.
So what are the rules? Proper structure is one of them. So take a look at the example project.

 

AWS layers one SAM template structure

 

  • the initial structure is based on SAM init “Hello World template”
  • there is only one SAM template in the root of the project
  • both layers sit in layers directories and have separate node_modules folders
  • there is only one lambda function `thumbnail-post-cover-creator` which uses dependencies from both layers
  • node_modules are in nodejs folders

 

RULE #1

Nodejs folders have to be created as a parent folder for layer dependencies (node_modules).

 

Node modules

 

I also post contents of all three packages.json files to make it clear where specific dependencies are installed.

layers/sharp-layer/nodejs/package.json
{
  "name": "sharp-layer",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "BlowStack,
  "license": "MIT-0",
  "dependencies": {
    "sharp": "^0.30.7"
  }
}

layers/aws-layer/nodejs/package.json
{
  "name": "aws-layer",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "BlowStack,
  "license": "MIT-0",
  "dependencies": {
    "sharp": "^0.30.7"
  }
}

package.json
{
  "name": "blowstack_website_sam",
  "version": "1.0.0",,
  "main": "handler.js",
  "scripts": {
    "unit": "jest",
    "lint": "eslint '*.ts' --quiet --fix",
    "compile": "tsc",
    "test": "npm run compile && npm run unit"
  },
  "homepage": "https://github.com/blowstack",
  "devDependencies": {
    "@types/aws-lambda": "^8.10.92",
    "@types/jest": "^27.4.0",
    "@types/node": "^17.0.13",
    "@typescript-eslint/eslint-plugin": "^5.10.2",
    "@typescript-eslint/parser": "^5.10.2",
    "esbuild": "^0.14.14",
    "esbuild-jest": "^0.5.0",
    "eslint": "^8.8.0",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-prettier": "^4.0.0",
    "jest": "^27.5.0",
    "prettier": "^2.5.1",
    "ts-node": "^10.4.0",
    "typescript": "^4.5.5"
  }
}

 

Template with nested layers

 

Take a look at the template. There are three important things to focus on.

  • layer resource creation and its proper content uri path
  • layer referencing in lambda function
  • marking layers dependencies as external

 

template.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Layer test SAM

Resources:
  SharpLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: SharpLayer
      Description: Sharp NPM package.
      ContentUri: 'layers/sharp-layer/'
      CompatibleRuntimes:
        - nodejs12.x
  AwsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: AwsLayer
      Description: AWS NPM package.
      ContentUri: 'layers/aws-layer/'
      CompatibleRuntimes:
        - nodejs12.x
  ThumbnailPostCoverCreator:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handler.thumbnailPostCoverCreator
      Runtime: nodejs14.x
      Architectures:
        - x86_64
      Layers:
        - !Ref SharpLayer
        - !Ref AwsLayer
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        Sourcemap: true
        EntryPoints:
        - handler.js
        External:
          - 'aws-sdk'
          - 'sharp'
 

 

Now take a look at the lambda code which uses both layers.
Notice that aws-sdk and sharp are required as usual (as if they were in root node_modules but they are not).


thumbnail-post-cover-crator.ts
import { S3Event } from 'aws-lambda';
const AWS = require('aws-sdk');
const sharp = require('sharp');
const path = require('path');
const util = require('util');
const s3 = new AWS.S3();

export const thumbnailPostCoverCreator = async (event: S3Event): Promise<void> => {};

 

In fact there is no sharp package in package.json in the root folder of the project.
But because the rules were applied it's possible to use layers dependencies automatically.

 

RULE #2

Create layer resources and reference them in lambda functions.

 SharpLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: SharpLayer
      Description: Sharp NPM package.
      ContentUri: 'layers/sharp-layer/'
      CompatibleRuntimes:
        - nodejs12.x

Layers:
        - !Ref SharpLayer
        - !Ref AwsLayer

 

RULE #3

Mark layer dependencies as external during build

External:
    - 'aws-sdk'
    - 'sharp'

 

POSSIBLE ERROR

If you don't mark layer dependencies as external the following error will pop up during building.

Could not resolve "sharp"
   thumbnail-post-cover-creator/app.ts:6:22:
     6 │ const sharp = require('sharp');
       ╵                       ~~~~~~~
 You can mark the path "sharp" as external to exclude it from the bundle, which will remove this error. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

 

 

Handler.js

Just to make everything as clear as possible I show contents of handler.js too.

import { thumbnailPostCoverCreator } from './thumbnail-post-cover-creator/app.ts';

module.exports.thumbnailPostCoverCreator = thumbnailPostCoverCreator;


Building

 

To make the whole thing work together properly you need to build your project with docker containers.
 

RULE #4

Build your project using containers: sam build --use--container

After building the project is ready for local testing and deployment.

 

Summary and BONUS TIP

 

In this article you learned how to create lambda layers in NodeJS and TypeScript that can be

  • used in other lambdas as dependencies
  • tested locally with other lambdas without build problems

 

But there is more you can learn.

 

If you read “Do you really need lambda layers?” part you know that layers can also:

  •  overlap default installed packages
  •  make some of them easier to install

 

I show how to use aws-sdk and sharp packages as layers in this article. They are both great choices to explain the above features.


For example aws-sdk is already installed in the lambda environment by default. You can refer to this package in your code immediately without installing anything. The downside is the current managed aws-sdk version, it's not the latest. If you want to use the specific aws-sdk, wrapping it as a layer is a good choice. Of course you can try to add aws-sdk without a layer but then does it make sense to install it multiple times? The bundle size limits are also not too high!

 

More interesting things happen when you try to install a sharp package not as a layer. The following error will likely occur.

2022-07-07T20:04:57.198Z undefined ERROR Uncaught Exception {"errorType":"Error","errorMessage":"\nSomething went wrong installing the \"sharp\" module\n\nCannot find module '../build/Release/sharp-linux-x64.node'\nRequire stack:\n- /var/task/handler.js\n- /var/runtime/UserFunction.js\n- /var/runtime/Runtime.js\n- /var/runtime/index.js\n\nPossible solutions:\n- Install with verbose logging and look for errors: \"npm install --ignore-scripts=false --foreground-scripts --verbose sharp\"\n- Install for the current linux-x64 runtime: \"npm install --platform=linux --arch=x64 sharp\"\n- Consult the installation documentation: https://sharp.pixelplumbing.com/install","stack":["Error: ","Something went wrong installing the \"sharp\" module","","Cannot find module '../build/Release/sharp-linux-x64.node'","Require stack:","- /var/task/handler.js","- /var/runtime/UserFunction.js","- /var/runtime/Runtime.js","- /var/runtime/index.js","","Possible solutions:","- Install with verbose logging and look for errors: \"npm install --ignore-scripts=false --foreground-scripts --verbose sharp\"","- Install for the current linux-x64 runtime: \"npm install --platform=linux --arch=x64 sharp\"","- Consult the installation documentation: https://sharp.pixelplumbing.com/install"," at /var/task/handler.js:32:1753110"," at /var/task/handler.js:1:35"," at /var/task/handler.js:33:121"," at /var/task/handler.js:1:35"," at /var/task/handler.js:33:70531"," at /var/task/handler.js:1:35"," at Object.<anonymous> (/var/task/handler.js:33:70661)"," at Module._compile (internal/modules/cjs/loader.js:1085:14)"," at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)"," at Module.load (internal/modules/cjs/loader.js:950:32)"]}

 

You can workaround this error but then ask yourself is it worth it? Using layers in this case can save you a lot of time and effort.

 

BlowStack 2023