Trigger AWS Lambda on S3 Events Using SAM with Separate Templates

Single Template

Triggering a Lambda function on S3 events is quite straightforward when both the Lambda and S3 resources are declared in the same SAM template. There is a ready-to-use configuration that makes things really easy. Let's take a look at how simple it is with the following example:

Root Template (template.yaml)

Resources:  
  ExampleBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: "SomeBucketName"
  
  LambdaExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "S3_Event_Lambda_Role"
      Path: /
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: "S3_Event_Lambda_Policy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "logs:CreateLogStream"
                  - "logs:CreateLogGroup"
                  - "logs:PutLogEvents"
                Resource: "*"
              - Effect: "Allow"
                Action:
                  - "s3:*"
                Resource: "*"
  
  S3EventLambda:
    Type: AWS::Serverless::Function
    Properties:
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: handler.s3EventLambda
      Runtime: nodejs18.x
      Events:
        PostCoverUploaded:
          Type: S3
          Properties:
            Bucket: !Ref ExampleBucket
            Events: s3:ObjectCreated:*
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        EntryPoints:
          - "/lambdas/s3Event/handler.js"

// lambdas/s3EventLambda/app.ts

import { S3Event } from 'aws-lambda';
import util from 'util';
import path from 'path';
import sharp from '/opt/nodejs/node_modules/sharp';

export const s3EventLambda = async (event: S3Event): Promise<void> => {
    const srcBucket = event.Records[0].s3.bucket.name;
    const srcKey = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
    const srcDirectory = path.dirname(srcKey);
    const srcFilename = path.parse(srcKey).name;

    console.log('Successfully uploaded ' + srcBucket + '/' + srcKey);
};

// lambdas/s3EventLambda/handler.js

import { s3EventLambda } from './app.ts';

module.exports.s3EventLambda = s3EventLambda;

The code above is a ready solution that will trigger a Lambda function whenever a file is uploaded to the S3 bucket. However, things get more complicated when you keep your infrastructure (like S3 buckets) and application logic (like Lambdas) properly separated. This separation, though following best practices, may result in errors if you attempt to declare them in different templates and pass ARNs/names between them (via parameters, for instance). You might encounter the following error:

Error: Failed to create changeset for the stack: sam-app, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED" Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [S3EventLambda] is invalid. Event with id [S3UploadEvent] is invalid. S3 events must reference an S3 bucket in the same template.

Multi and Nested Templates

To achieve the same effect as shown in the single-template example, we need to create a custom resource, specifically a custom event. Let’s start by separating the S3 bucket and Lambda. This solution can also be integrated as nested templates. However, for clarity and best practices, I'll cover the templates as separate repositories (projects) altogether. This approach is particularly beneficial when infrastructure and application logic are managed by separate teams or roles in a project. Additionally, the solution can be extended to work across multiple AWS accounts with slight adjustments.

Infrastructure Root Template (template.yaml)

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Infra template

Resources:
  Roles:
    Type: AWS::Serverless::Application
    Properties:
      Location: roles/template.yaml
  Storages:
    Type: AWS::Serverless::Application
    Properties:
      Location: storages/template.yaml
Outputs:
  ExampleBucket:
    Description: "Example Bucket Name"
    Value: !GetAtt Storages.Outputs.ExampleBucket
    Export:
      Name: ExampleBucketName
  LambdaEventsExecutionRoleArn:
    Description: "Lambda Events Execution Role Arn"
    Value: !GetAtt Roles.Outputs.LambdaEventsExecutionRoleArn
    Export:
      Name: LambdaEventsExecutionRoleArn

S3 Bucket Template (storages/template.yaml)

Resources:  
  ExampleBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: 'SomeBucketName'
Outputs:
  ExampleBucket:
    Description: ExampleBucket
    Value: !Ref ExampleBucket

IAM Role Template (roles/template.yaml)

Resources:
  LambdaEventsExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: "S3_Event_Lambda_Role"
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: "S3_Event_Lambda_Policy"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - lambda:InvokeFunction
                Resource: "*
              - Effect: Allow
                Action:
                  - s3:*
                Resource: "*"
              - Effect: "Allow"
                Action:
                  - "sns:*"
                Resource: "*"
              - Effect: "Allow"
                Action:
                  - "ec2:*"
                Resource: "*"
Outputs:
  LambdaEventsExecutionRoleArn:
    Value: !GetAtt LambdaEventsExecutionRole.Arn

Application Root Template for Lambdas (template.yaml)

This template defines the Lambda functions and separates the logic for handling different events:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: App template

Resources:
  EventsLambdas:
    Type: AWS::Serverless::Application
    Properties:
      Location: lambdas/template.yaml
      Parameters:
        LambdaEventsExecutionRoleArn: !ImportValue LambdaEventsExecutionRoleArn
  Events:
    DependsOn: [EventsLambdas]
    Type: AWS::Serverless::Application
    Properties:
      Location: events/template.yaml
      Parameters:
        S3CustomEventTriggerFunctionArn: !GetAtt EventsLambdas.Outputs.S3CustomEventTriggerFunctionArn
        ExampleLambdaArn: !GetAtt EventsLambdas.Outputs.S3EventLambdaArn
        ExampleBucket: !ImportValue ExampleBucket
  Permissions:
    Type: AWS::Serverless::Application
    Properties:
      Location: permissions/template.yaml
      Parameters:
        ExampleLambdaArn: !GetAtt EventsLambdas.Outputs.S3EventLambdaArn

By separating the events and Lambdas, this structure makes it easy to attach different Lambda functions to various events.

Lambda Functions Template (lambdas/template.yaml)

// lambdas/template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Events lambdas.

Parameters:
  ExampleBucket:
    Type: String
  LambdaEventsExecutionRoleArn:
    Type: String

Resources:
  S3CustomEventTriggerFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handler.s3CustomEventTriggerLambda
      Role: !Ref LambdaEventsExecutionRoleArn
      Runtime: nodejs18.x
      Policies:
        - S3CrudPolicy:
            BucketName: !Ref ExampleBucket
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        EntryPoints:
          - "/lambdas/s3EventCustomEventTrigger/handler.js"
      
  S3EventLambda:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handler.s3EventLambda
      Role: !Ref LambdaEventsExecutionRoleArn
      Runtime: nodejs18.x
      Policies:
        - S3CrudPolicy:
            BucketName: !Ref ExampleBucket
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        EntryPoints:
          - "/lambdas/s3Event/handler.js"

Outputs:
  S3CustomEventTriggerFunctionArn:
    Description: S3 custom event lambda ARN
    Value: !GetAtt S3CustomEventTriggerFunction.Arn
  S3EventLambdaArn:
    Description: S3 event lambda Arn
    Value: !GetAtt S3EventLambda.Arn

Since S3EventLambda has the same content as the Lambda function from the single template solution presented at the beginning, I have omitted it here. If needed, you can refer back to the earlier part of the page for its details.

The more interesting functionality lies in the S3CustomEventTriggerFunction. This function is responsible for creating a custom event when deploying a CloudFormation (SAM) template and removing it when the template is destroyed. These types of Lambda functions are typically used to create custom resources. In this case, we want to create a custom resource—an event equivalent to what is created when S3 and Lambda are included in a single template.

Here’s the implementation of the custom event Lambda function:

// lambdas/s3EventCustomEventTrigger/app.ts

import { S3 } from 'aws-sdk';
import { Context, CloudFormationCustomResourceEvent } from 'aws-lambda';
import https from 'https';

const s3 = new S3();

interface ResponseBody {
    Status: 'SUCCESS' | 'FAILED';
    Reason: string;
    PhysicalResourceId: string;
    StackId: string;
    RequestId: string;
    LogicalResourceId: string;
    Data: Record<string, unknown>;
}

export const s3CustomEventTriggerLambda = async (
    event: CloudFormationCustomResourceEvent,
    context: Context
): Promise<void> => {
    try {
        if (event.RequestType === 'Delete') {
            await s3.putBucketNotificationConfiguration({
                Bucket: event.ResourceProperties.BucketName,
                NotificationConfiguration: {}
            }).promise();
        } else {
            await s3.putBucketNotificationConfiguration({
                Bucket: event.ResourceProperties.BucketName,
                NotificationConfiguration: event.ResourceProperties.NotificationConfiguration as S3.NotificationConfiguration
            }).promise();
        }
        
        await sendResponse(event, context, "SUCCESS");
    } catch (error) {
        console.error('Error:', error instanceof Error ? error.message : String(error));
        await sendResponse(event, context, "FAILED");
    }
};

async function sendResponse(
    event: CloudFormationCustomResourceEvent,
    context: Context,
    status: 'SUCCESS' | 'FAILED'
): Promise<void> {
    const responseBody: ResponseBody = {
        Status: status,
        Reason: `See the details in CloudWatch Log Stream: ${context.logStreamName}`,
        PhysicalResourceId: context.logStreamName,
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
        Data: {}
    };

    const jsonResponseBody = JSON.stringify(responseBody);

    const options: https.RequestOptions = {
        method: 'PUT',
        headers: {
            'Content-Type': '',
            'Content-Length': Buffer.byteLength(jsonResponseBody)
        }
    };

    return new Promise((resolve, reject) => {
        const request = https.request(event.ResponseURL, options, (response) => {
            response.on('end', () => {
                resolve();
            });
        });

        request.on('error', (error) => {
            reject(error);
        });

        request.write(jsonResponseBody);
        request.end();
    });
}

Event Definition Template (events/template.yaml)

This template defines the S3 event triggers for Lambda functions:

// events/template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Events events.

Parameters:
  S3CustomEventTriggerFunctionArn:
    Type: String
  S3EventLambdaArn:
    Type: String
  ExampleBucket:
    Type: String

Resources:
  S3EventTrigger:
    Type: Custom::S3BucketNotification
    Properties:
      ServiceToken: !Ref S3CustomEventTriggerFunctionArn
      BucketName: !Ref ExampleBucket
      NotificationConfiguration:
        LambdaFunctionConfigurations:
          - Events:
              - 's3:ObjectCreated:*'
            LambdaFunctionArn: !Ref S3EventLambdaArn

Permissions Template (permissions/template.yaml)

// permissions/template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Events permissions.

Parameters:
  S3EventLambdaArn:
    Type: String

Resources:
  LambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref S3EventLambdaArn
      Action: lambda:InvokeFunction
      Principal: s3.amazonaws.com

Final Thoughts

This multi-template and modular approach separates the application logic and infrastructure components. It allows the infrastructure to evolve independently from the application code, promoting better team collaboration and scalability. You can even extend this solution to operate across multiple AWS accounts, though that would require some adjustments to resource sharing and permissions.