Overview on Deploying Serverless Services

It’s hard to believe that Lambda function have already been around only for the last 5 years (at the time of writing this), and it became so popular so quickly.

I’ve been working as a DevOps engineer for the last 8 years or so (working on AWS for the last 7 years), and I wanted to review some of the tools I came across for deploying serverless solutions on AWS.

I. First wave: bash script

It works! And it still does, although it’s not very scalable, it’s mainly (as I still does today) for installing dependencies, testing and packaging them into zip.

One example of bash script code I keep re-using is this one

#!/usr/bin/env bash
# Create python packages for lambda
#title          :python_packages.sh
#description    :This script will install pip packages, run pylint and pytest and finally zip python scripts
#author         :Oli
#date           :07/01/2020
#version        :0.1
#bash_version   :3.2.57(1)-release
#===================================================================================

set -o errexit
set -o pipefail
set -o nounset

function install_packages() {
  pip install -r requirements_test.txt
}

function zip_python() {
  pushd sns-guardduty-slack/
  zip -r ../guardduty-sns-slack-payload.zip sns_guardduty_slack.py
  popd
  mv guardduty-sns-slack-payload.zip ../
}

function test_python() {
  pushd sns-guardduty-slack/
  pylint sns_guardduty_slack.py
  popd
}

install_packages
test_python
zip_pythonCode language: PHP (php)

This is working quite well for the first part (Continuous Integration) but doesn’t fit the second part (Continuous Delivery). Back in these days, I started using CloudFormation for CD requirements, let’s see an example.

II. Cloudformation Era

Around 2016, a Cloudformation allowed the use of Yaml format for declaring AWS resources, that was a massive change specially for the sake of my eyes

{
  "Type" : "AWS::Lambda::Function",
  "Properties" : {
      "Code" : Code,
      "DeadLetterConfig" : DeadLetterConfig,
      "Description" : String,
      "Environment" : Environment,
      "FunctionName" : String,
      "Handler" : String,
      "KmsKeyArn" : String,
      "Layers" : [ String, ... ],
      "MemorySize" : Integer,
      "ReservedConcurrentExecutions" : Integer,
      "Role" : String,
      "Runtime" : String,
      "Tags" : [ Tag, ... ],
      "Timeout" : Integer,
      "TracingConfig" : TracingConfig,
      "VpcConfig" : VpcConfig
    }
}Code language: JSON / JSON with Comments (json)

From this to this:

Type: AWS::Lambda::Function
Properties: 
  Code: 
    Code
  DeadLetterConfig: 
    DeadLetterConfig
  Description: String
  Environment: 
    Environment
  FunctionName: String
  Handler: String
  KmsKeyArn: String
  Layers: 
    - String
  MemorySize: Integer
  ReservedConcurrentExecutions: Integer
  Role: String
  Runtime: String
  Tags: 
    - Tag
  Timeout: Integer
  TracingConfig: 
    TracingConfig
  VpcConfig: 
    VpcConfigCode language: JavaScript (javascript)

Declaring AWS Lambda functions for Operational purposes, Cloudformation was indeed doing the job. Unfortunately, it can become rather complexe if you start adding API Gateway / SNS Topics and so on to the equations:

GatewayRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - apigateway.amazonaws.com
        Action:
        - sts:AssumeRole
    Path: "/"
    Policies:
      - PolicyName: GatewayRolePolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
              - sns:Publish
            Resource: "*" 
          - Effect: Allow 
            Action:
              - logs:PutLogEvents
              - logs:CreateLogGroup
              - logs:CreateLogStream
            Resource: "*"
ApiGatewayRestApi:
  Type: AWS::ApiGateway::RestApi
  Properties:
    Name: API_GW
      Fn::Join:
        - ""
        - - Ref: AWS::StackName
          - "-api"
ApiGatewayGETMethod:
  Type: AWS::ApiGateway::Method
  Properties:
    AuthorizationType: NONE
    HttpMethod: GET
    RequestParameters: 
      method.request.querystring.message: false
      method.request.querystring.subject: false
      method.request.querystring.topic: false
    Integration:
      Type: AWS
      Credentials:
        Fn::GetAtt: [ GatewayRole, Arn ] #use the already defined role
      Uri:  #required URI for using SNS service
        Fn::Join:
          - ""
          - - "arn:aws:apigateway:"
            - Ref: AWS::Region
            - ":sns:action/Publish"
...................Code language: PHP (php)

This is a truncated example of an API Gateway declaration in Cloudformation format. Let’s see Terraform way

III. Does Terraform make it better?

Depending on who you ask, I believe it slightly does but not that much:

AWS: aws_api_gateway_integration – Terraform by HashiCorpProvides an HTTP Method Integration for an API Gateway

Based on my experience, I had trouble deploying AWS Lambda functions with Terraform, especially around Stages and Testings which would see later. Terraform seems to have issues picking up the latest zip file containing the source code (even when build locally), I had to ran it

resource "aws_lambda_function" "sns_slack_lambda" {
  filename         = var.filename
  function_name    = var.lambda_function_name
  role             = aws_iam_role.sns_slack_lambda_role.arn
  handler          = "sns_guardduty_slack.lambda_handler"
  source_code_hash = base64sha256(var.filename)
  runtime          = "python3.7"

  environment {
    variables = {
      SLACK_CHANNEL = data.aws_ssm_parameter.slack_channel.value
      HOOK_URL      = data.aws_ssm_parameter.slack_incoming_webhook.value
    }
  }
}Code language: JavaScript (javascript)

IV. AWS SAM

Around 2018, AWS introduced AWS Serverless Application Model to answer the needs for managing Serverless deployments and the management of them. SAM is simply an extension of CloudFormation, is also using Yaml format and the installation process is as simple as AWS CLI one:

pip install aws-sam-cli

SAM CLI is then available as sam. Once completed, let’s crack on the first step: Package

sam package \
  --template-file template.yml \
  --output-template-file package.yml \
  --s3-bucket my-bucket

This command is to upload any artifacts to an AWS S3 bucket, generate a package AWS SAM template file that’s ready to be used.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-appSample SAM Template for sam-appGlobals:
  Function:
    Timeout: 3Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.7
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: getOutputs:
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Here is a template example of an hello world application deploy with AWS SAM, the syntax is a mixed between Cloudformation template and addons AWS SAM, I personally found this confusing. Once your package is ready with the package command, you can deploy by using the following command:

sam deploy \
  --template-file ./packaged.yaml 
  --stack-name mystack 
  --capabilities CAPABILITY_IAM

AWS SAM is helping solving the previous mentionned issues with Cloudformation and Terraform, this tool helps for building scaffolding for Lambda / API Gateway / SNS and so on, it also helps for performing integration testings locally before pushing your code to production. After using AWS SAM, I also discovered that this tool would inherited problems from Cloudformation such as modifying records in Route53. If you compare AWS SAM with Serverless (which we will see next), you will quickly switch to Serverless as I did.

V. Serverless Framework

Serverless framework managed to solve most of these issues:

• Readable
• Cloud agnostic
• Speed of deployment
• And so on…

Installing this tool require Node.js installed, simply install that tool that way:

npm install -g serverless

Packing and deploying by only using a single command line:

sls deploy

After using this tool extensively, I’m no longer having issue managing API Gateway / SNS / Lambda and link them automagically together, deploying is really easy and fast.

VI. My take on deploying Serverless services

My main recommendation would be to jump straight to Serverless framework and avoid investing too much of your time on Cloudformation / Terraform for deploying a whole set of serverless services together.