Run scheduled Lambda functions using CloudWatch Events

A great use case of Lambda functions is when a task needs to run on schedule, like every five minutes or once a day.

This job, especially if the frequency is low (like once a day or once a month) shouldn’t require a server to run. Why maintain it if its full potentials are not used?

It’s one of the advantages of Lambda that you only pay what you use, and there’s no need to provision, maintain (and pay for) a server - hence the name, serverless. You pay by function invocation, and if it happens only once a month, then the edge of Lambda over having the code run on a paid server is clear.

1. Pre-requisites

If you decide to follow what’s described below, you will need a free AWS account, user credentials for programmatic access to AWS services and the AWS CLI installed on your computer.

2. About the scheduler

The easiest way to set up a scheduled workflow is to set up a CloudWatch scheduled event. This event will trigger a Lambda function, which will run on a regular basis.

CloudWatch Events is part of Amazon EventBridge now, and will be eventually named as such, but now it can be found in the CloudWatch part of the Console. Currently, both CloudWatch Events and EventBridge use the same API.

The biggest advantage of using CloudWatch Events is that you won’t need to mess around with schedulers in your code. CloudWatch takes care of everything, and you can focus on creating the logic.

One drawback of CloudWatch Events is that they don’t allow configuration for second’s accuracy, and the smallest unit is minutes. If you need to configure schedules to the second, you’ll need to use cron configuration.

3. The architecture

This post won’t contain any difficult architecture. A CloudWatch event will trigger a Lambda function and the function will log and return something.

In future posts, a database and Step Functions will also be added, but not now, otherwise the post would be very long, and no one would read it.

4. The Lambda function

I wrote a post on creating a Lambda function using CLI, so I won’t go into detail here, and I mention only the main steps.

4.1. Write the function code

The Lambda function will list secret agents, who live in full secrecy, but we can, of course, know about them.

Create a file called secret-mission.js in your project folder, and write the following lines there:

exports.handler = async (event) => {
  const { agent } = event
  console.log(`Secret agent ${ agent } is now scheduled.`)

  return `Agent ${ agent } has been returned.`
}

Nothing fancy here; the event object will have a property called agent, and the function simply logs the name of the agent, and then returns it.

4.2. Create a function execution role

Next, we’ll need a role that allows Lambda function to do some basic stuff like logging to CloudWatch.

In the referred post, you will find how to create the role if you don’t have one.

4.3. Upload the function

We need to compress the file containing the handler code first:

zip secret-mission secret-mission.js

This command will generate a compressed file called secret-mission.zip.

Let’s upload the zip file to AWS:

aws lambda create-function --function-name scheduled-secret-mission --runtime nodejs10.x \\
--role arn:aws:iam::<ACCOUNT NUMBER>:role/LambdaBasicExecutionRole
--handler secret-mission.handler \\
--zip-file fileb://secret-mission.zip --region us-west-2

We can check if the upload was successful with the following command:

aws lambda list-functions --region us-west-2

The response should include scheduled-secret-mission:

{
  "Functions": [
    {
      "FunctionName": "scheduled-secret-mission",
      "FunctionArn": "arn:aws:lambda:us-west-2:<ACCOUNT NUMBER>:function:scheduled-secret-mission",
      "Runtime": "nodejs10.x",
      "Role": "arn:aws:iam::<ACCOUNT NUMBER>:role/LambdaBasicExecutionRole",
      "Handler": "secret-mission.handler",
      "CodeSize": 309,
      "Description": "",
      "Timeout": 3,
      "MemorySize": 128,
      "LastModified": "2019-09-01T20:37:18.283+0000",
      "CodeSha256": "efVNaoORN8K+//wjPbXqTCxxaUqJm3Ddq2OBYNGTf/g=",
      "Version": "$LATEST",
      "TracingConfig": {
          "Mode": "PassThrough"
      },
      "RevisionId": "ce44efff-ea64-4494-9736-3553ac5dd09c"
    }
  ]
}

5. Create the scheduled event

Now that we have the function ready, we’ll need something to trigger the invocation. We don’t want to sit here, and keep pressing the invoke button, so let’s quickly set up a scheduled event in CloudWatch.

5.1. Create the rule

First we need a rule, which determines when the above Lambda function will run.

Of course, there is a CLI command for this case, and it looks like this:

aws events put-rule --name schedule-secret-mission --schedule-expression 'rate(6 hours)' \\
--description 'A scheduled event for secret agents'

The name of the service is events, and we use the put-rule command, which creates a rule. The rule is called schedule-secret-mission, and it runs every 6 hours. You can choose a smaller frequency, like 5 minutes (rate(5 minutes)) if you like. The reason why I chose 6 hours is that I wanted it to run for a longer time, and I didn’t want it to cost me much (we pay after execution). With just four invocations a day I’m still within a free tier after a few days of having the function run.

Let’s check if the rule has arrived:

aws events list-rules

The response should be something like this:

{
  "Rules": [
    {
      "Name": "schedule-secret-mission",
      "Arn": "arn:aws:events:us-west-2:<ACCOUNT NUMBER>:rule/schedule-secret-mission",
      "State": "ENABLED",
      "ScheduleExpression": "rate(6 hours)"
    }
  ]
}

Now that we have the scheduler in CloudWatch and the function that will run on schedule, we should bring them together to make the scheduled function invocation work.

5.2. Add permission to Lambda to accept CloudWatch

First, we need to give permission to Lambda to accept a trigger from CloudWatch Events, or, looking at it from a different perspective, we’ll give permission to CloudWatch to invoke our Lambda function.

We could go to the Console, and add a trigger to Lambda, but instead, let’s do it from the CLI now.

The permission needs to be configured on the Lambda side, and the command looks similar to this:

aws lambda add-permission --function-name scheduled-secret-mission \\
--statement-id scheduled-secret-mission-event --action 'lambda:InvokeFunction' \\
--principal events.amazonaws.com \\
--source-arn arn:aws:events:us-west-2:<ACCOUNT ID>:rule/schedule-secret-mission

With this, we attach a policy to the function. We need to specify the name of the Lambda function (scheduled-secret-mission), the action (which is function invocation) and the principal (the other AWS service which the permission is given to; this is going to be CloudWatch Events in this case).

source-arn specifies the unique identifier (ARN) of the rule, and this way we apply the principle of granting least privilege, which is a best practice. No other CloudWatch Events rule can trigger this function.

The statement-id is simply just a marker that differentiates this entry from other permissions inside the policy. There won’t be other entries in the policy for now.

5.3. Add target to the CloudWatch Events rule

The Lambda function is now aware that a CloudWatch Events rule will invoke it in the future, but we have to tell CloudWatch Events as well which function to invoke. We can do this by adding a target to the rule.

One way of doing it is to create a file called secret-mission-target.json in the same folder, and have at least the following properties:

[
  {
    "Id": "1",
    "Arn": "arn:aws:lambda:us-west-2:678185312562:function:scheduled-secret-mission",
    "Input": "{\"agent\": \"James Bond\"}"
  }
]

The Id and Arn have to be there, and we can feed the Lambda function with some data (later this will be stored in a database) as Input. It’s a JSON-like syntax where the curly braces in the key have to be wrapped in double-quotes. Not nice, but that’s life. Input will be converted to the event object when the Lambda function runs, so it will have the agent property.

One rule can have multiple targets (i.e. one CloudWatch Events rule can trigger multiple Lambda functions or even multiple services), hence the Id needs to be specified. The Arn is the identifier of the Lambda function, which needs to be invoked.

Let’s add the target then:

aws events put-targets --rule schedule-secret-mission --targets file://secret-mission-target.json

By specifying the rule (i.e. the schedule) we can add the targets. Alternatively, the properties of the targets can be added to the command line in "key1"="value1","key2"="value2" format.

5.4. Wait

It’s all set up now; all we have to do is wait.

After the period you set in CloudWatch Events, scheduled-secret-mission will be invoked, and then again and again.

If you head over to the Monitoring part of Lambda, you’ll see the invocations there.

CloudWatch Logs should show the Secret agent James Bond is now scheduled. message, which is what we wanted to log from the Lambda function.

6. Conclusion

CloudWatch Events provides a great way to schedule function invocations at regular time intervals.

To do this, a rule needs to be set up in CloudWatch Events, and the Lambda function has to be added as a target.

On the other side, Lambda has to have permissions to accept requests from CloudWatch Events and to execute the function.

Thanks for reading, and see you next time.