BlowStack
->
Blog
->
How to use AWS Lambda Layers with SAM

How to use AWS Lambda Layers with SAM
Last update 10.2022

TABLE OF CONTENTS

GOLDEN RULES
Do you really need lamba layers?
Project structure with lambda layers
Template with nested layers
Building
Summary and BONUS TIP

GOLDEN RULES

There are 4 rules you have to follow in order to use lambda layers:

in local development
in the same template as the rest of resources

RULE #1

Nodejs folder have to be created as a parent folder for layer dependencies (node_modules).

RULE #2

Create layer resources and reference them in lambda functions.

RULE #3

Mark layer dependencies as external during build

RULE #4

Build your project using containers: sam build --use--container

Read further if you need more explanation.

Do you really need lamba layers?

Using lambda layers can be troublesome but at the same time beneficial or even necessary in a project.

So what can lambda layers do in the first place?

share libraries between different lambda functions (in different projects, the same account)
- lower lambdas size
- environment consistency (global version control)
- faster development
can overlap default installed packages (in lambda environment) like aws-sdk
- easy to upgrade or downgrade any library/package globally
some big packages like Sharp are easier to install that way
can be utilized as a custom runtime, data or configuration file

This post is focused on layers as a library.

Project structure with lambda layers

Basically there are two options that you can choose from.

Keep or not keep layers in the same SAM template where you define other resources.

The main reason for not keeping layers in the same template is when you are going to use them in different projects.
Then even keeping them in different repositories makes a lot of sense.
Unfortunately there are many people pointing out technical difficulties as the reason to always choose a multi template option.

For example one of the articles about nested SAM stacks states that:

“When working with a nested SAM stack, you cannot import a Lambda Layer exported from another stack.”

But it's not true.
You can nest layers in SAM templates and test them locally as dependencies as long as you stick to some rules.
So what are the rules? Proper structure is one of them. So take a look at the example project.

the initial structure is based on SAM init “Hello World template”
there is only one SAM template in the root of the project
both layers sit in layers directories and have separate node_modules folders
there is only one lambda function `thumbnail-post-cover-creator` which uses dependencies from both layers
node_modules are in nodejs folders

RULE #1

Nodejs folders have to be created as a parent folder for layer dependencies (node_modules).

Node modules

I also post contents of all three packages.json files to make it clear where specific dependencies are installed.

layers/sharp-layer/nodejs/package.json
{
"name": "sharp-layer",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "BlowStack,
"license": "MIT-0",
"dependencies": {
"sharp": "^0.30.7"
}
}

layers/aws-layer/nodejs/package.json
{
"name": "aws-layer",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "BlowStack,
"license": "MIT-0",
"dependencies": {
"sharp": "^0.30.7"
}
}

package.json
{
"name": "blowstack_website_sam",
"version": "1.0.0",,
"main": "handler.js",
"scripts": {
"unit": "jest",
"lint": "eslint '*.ts' --quiet --fix",
"compile": "tsc",
"test": "npm run compile && npm run unit"
},
"homepage": "https://github.com/blowstack",
"devDependencies": {
"@types/aws-lambda": "^8.10.92",
"@types/jest": "^27.4.0",
"@types/node": "^17.0.13",
"@typescript-eslint/eslint-plugin": "^5.10.2",
"@typescript-eslint/parser": "^5.10.2",
"esbuild": "^0.14.14",
"esbuild-jest": "^0.5.0",
"eslint": "^8.8.0",
"eslint-config-prettier": "^8.3.0",
"eslint-plugin-prettier": "^4.0.0",
"jest": "^27.5.0",
"prettier": "^2.5.1",
"ts-node": "^10.4.0",
"typescript": "^4.5.5"
}
}

Template with nested layers

Take a look at the template. There are three important things to focus on.

layer resource creation and its proper content uri path
layer referencing in lambda function
marking layers dependencies as external

`template.yml`
`AWSTemplateFormatVersion: '2010-09-09'`
`Transform: AWS::Serverless-2016-10-31`
`Description: Layer test SAM`

`Resources:`
`SharpLayer:`
`Type: AWS::Serverless::LayerVersion`
`Properties:`
`LayerName: SharpLayer`
`Description: Sharp NPM package.`
`ContentUri: 'layers/sharp-layer/'`
`CompatibleRuntimes:`
`- nodejs12.x`
`AwsLayer:`
`Type: AWS::Serverless::LayerVersion`
`Properties:`
`LayerName: AwsLayer`
`Description: AWS NPM package.`
`ContentUri: 'layers/aws-layer/'`
`CompatibleRuntimes:`
`- nodejs12.x`
`ThumbnailPostCoverCreator:`
`Type: AWS::Serverless::Function`
`Properties:`
`Handler: handler.thumbnailPostCoverCreator`
`Runtime: nodejs14.x`
`Architectures:`
`- x86_64`
`Layers:`
`- !Ref SharpLayer`
`- !Ref AwsLayer`
`Metadata:`
`BuildMethod: esbuild`
`BuildProperties:`
`Minify: true`
`Target: "es2020"`
`Sourcemap: true`
`EntryPoints:`
`- handler.js`
`External:`
`- 'aws-sdk'`
`- 'sharp'`

Now take a look at the lambda code which uses both layers.
Notice that aws-sdk and sharp are required as usual (as if they were in root node_modules but they are not).


thumbnail-post-cover-crator.ts
import { S3Event } from 'aws-lambda';
const AWS = require('aws-sdk');
const sharp = require('sharp');
const path = require('path');
const util = require('util');
const s3 = new AWS.S3();

export const thumbnailPostCoverCreator = async (event: S3Event): Promise<void> => {};

In fact there is no sharp package in package.json in the root folder of the project.
But because the rules were applied it's possible to use layers dependencies automatically.

RULE #2

Create layer resources and reference them in lambda functions.

SharpLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: SharpLayer
Description: Sharp NPM package.
ContentUri: 'layers/sharp-layer/'
CompatibleRuntimes:
- nodejs12.x

Layers:
- !Ref SharpLayer
- !Ref AwsLayer

RULE #3

Mark layer dependencies as external during build

External:
- 'aws-sdk'
- 'sharp'

POSSIBLE ERROR

If you don't mark layer dependencies as external the following error will pop up during building.

Could not resolve "sharp"
   thumbnail-post-cover-creator/app.ts:6:22:
     6 │ const sharp = require('sharp');
       ╵                       ~~~~~~~
 You can mark the path "sharp" as external to exclude it from the bundle, which will remove this error. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

Handler.js

Just to make everything as clear as possible I show contents of handler.js too.

import { thumbnailPostCoverCreator } from './thumbnail-post-cover-creator/app.ts';

module.exports.thumbnailPostCoverCreator = thumbnailPostCoverCreator;

Building

To make the whole thing work together properly you need to build your project with docker containers.

RULE #4

Build your project using containers: sam build --use--container

After building the project is ready for local testing and deployment.

Summary and BONUS TIP

In this article you learned how to create lambda layers in NodeJS and TypeScript that can be

used in other lambdas as dependencies
tested locally with other lambdas without build problems

But there is more you can learn.

If you read “Do you really need lambda layers?” part you know that layers can also:

overlap default installed packages
make some of them easier to install

I show how to use aws-sdk and sharp packages as layers in this article. They are both great choices to explain the above features.

For example aws-sdk is already installed in the lambda environment by default. You can refer to this package in your code immediately without installing anything. The downside is the current managed aws-sdk version, it's not the latest. If you want to use the specific aws-sdk, wrapping it as a layer is a good choice. Of course you can try to add aws-sdk without a layer but then does it make sense to install it multiple times? The bundle size limits are also not too high!

More interesting things happen when you try to install a sharp package not as a layer. The following error will likely occur.

2022-07-07T20:04:57.198Z undefined ERROR Uncaught Exception {"errorType":"Error","errorMessage":"\nSomething went wrong installing the \"sharp\" module\n\nCannot find module '../build/Release/sharp-linux-x64.node'\nRequire stack:\n- /var/task/handler.js\n- /var/runtime/UserFunction.js\n- /var/runtime/Runtime.js\n- /var/runtime/index.js\n\nPossible solutions:\n- Install with verbose logging and look for errors: \"npm install --ignore-scripts=false --foreground-scripts --verbose sharp\"\n- Install for the current linux-x64 runtime: \"npm install --platform=linux --arch=x64 sharp\"\n- Consult the installation documentation: https://sharp.pixelplumbing.com/install","stack":["Error: ","Something went wrong installing the \"sharp\" module","","Cannot find module '../build/Release/sharp-linux-x64.node'","Require stack:","- /var/task/handler.js","- /var/runtime/UserFunction.js","- /var/runtime/Runtime.js","- /var/runtime/index.js","","Possible solutions:","- Install with verbose logging and look for errors: \"npm install --ignore-scripts=false --foreground-scripts --verbose sharp\"","- Install for the current linux-x64 runtime: \"npm install --platform=linux --arch=x64 sharp\"","- Consult the installation documentation: https://sharp.pixelplumbing.com/install"," at /var/task/handler.js:32:1753110"," at /var/task/handler.js:1:35"," at /var/task/handler.js:33:121"," at /var/task/handler.js:1:35"," at /var/task/handler.js:33:70531"," at /var/task/handler.js:1:35"," at Object.<anonymous> (/var/task/handler.js:33:70661)"," at Module._compile (internal/modules/cjs/loader.js:1085:14)"," at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)"," at Module.load (internal/modules/cjs/loader.js:950:32)"]}

You can workaround this error but then ask yourself is it worth it? Using layers in this case can save you a lot of time and effort.