How to generate presigned URLs using a Lambda function

Generating presigned URLs programmatically is not hard; but how can we make it automatic?

1. When to use presigned URLs

When we don’t want to give the users of an application permission to access S3 buckets, we can use presigned URLs through which users can download or upload objects. Presigned URLs provide users with temporary permissions to access a specific object in a specific S3 bucket.

A typical use case is when you develop an application where the user needs to upload images or some documents to support their claim with a government agency.

Presigned URLs can provide secure access to S3 from on-premises systems, too. If these systems need to upload objects to AWS, we don’t want to create individual username and password for each client. In this case, we have little control how clients secure their systems, so exposing access keys and secret access keys is a security risk.

The good thing about presigned URLs is that we can block all public access to the S3 bucket (recommended setting by AWS) and the user of the URL will still be able to upload objects to the bucket.

Let’s see how a minimalist presigned URL system can be created for uploading objects to S3.

2. Architecture

One potential architecture design to generate presigned URLs is the following.

The client can make a POST request to API Gateway, which has a Lambda integration, i.e. when the endpoint exposed by the API Gateway is hit, the Lambda function will be automatically triggered.

The Lambda function will generate the presigned URL and will return it to the API Gateway, which forwards it to the client.

When a presigned URL is generated, it’s important that the generating entity (in this case, the Lambda function) has the relevant permissions. For presigned URLs allowing uploads to S3, the Lambda function must have - at least - PutObject permission:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

If the Lambda function did not have the relevant permissions, the presigned URL would be invalid and the client wouldn’t be able to upload objects through the URL.

The API Gateway might need to be protected so that only authorized users (for example, those who signed in to our app) can query the endpoint. Otherwise we would need to pay for storing everyone’s stuff in S3, and we (at least I) don’t want that.

3. Lambda code

The following snippet will create a presigned URL that allows anyone to upload to a bucket whose name is stored in the BUCKET_NAME environment variable in the Lambda function. The uploaded object will be saved to the <USERNAME>/<DATE IN MILLISECONDS> key, which creates a folder in S3 with the username of the client (it’s actually part of the object key).

For example, if the authenticated user’s username is johndoe, the uploaded object will be saved with a key like johndoe/1595114575104. Any subsequent uploads from the same user will have the johndoe prefix in the key.

Let’s now see the code that generates and returns the presigned URL:

const AWS = require('aws-sdk');

const s3 = new AWS.S3({ apiVersion: '2006-03-01' });

exports.handler = async (event) => {
  if (!event && !event.body) {
    throw new Error('No payload has been received');
  }

  const body = JSON.parse(event.body);
  const now = Date.now();

  if (!body.username) {
    throw new Error('Invalid payload');
  }

  const params = {
    Bucket: process.env.BUCKET_NAME,
    Key: `${body.username}/${now}`,
    Expires: 300,
    ContentType: 'application/json',
  };

  try {
    const presignedUrl = await s3.getSignedUrlPromise('putObject', params);
    console.log('The presigned URL is:', presignedUrl);

    return presignedUrl;
  } catch (error) {
    console.error('Error while generating presigned URL:', error.message);
    throw new Error(error);
  }
};

The presigned URL is generated by the getSignedUrl method. We need to specify the method that represents the purpose of the URL: It’s putObject for uploads and getObject for downloads.

getSignedUrl comes in both synchronous and asynchronous versions. Because in this case, the generating entity (the Lambda function) works with temporary IAM permissions (it has a role attached and not a permanent user with an access key and secret access key), it’s better to use the asynchronous method.

The Expires key in the params object refers to the expiration of the URL, which is 5 minutes in this case. This value can be modified as needed.

4. Client code

The client does not have AWS permissions: It can be, for example, a front-end or a Node.js application on premise.

The file that needs to be uploaded can also come from different sources. It can be something from the disk or can be downloaded from another place beforehand.

I have created a dummy file called banana.json and saved it to the same folder from which the code is run:

{
  "fruit": "banana",
  "size": "medium",
  "calories": 88,
  "macros": {
    "carbs": "94%",
    "protein": "5%",
    "fat": "1%"
  }
}

In this case, I have a small Node.js code that will upload banana.json to my bucket:

const path = require('path');
const fs = require('fs');
const axios = require('axios');

const uploadToS3 = async (fileName) => {
  if (!fileName) {
    console.log('No file to upload');
    return;
  }

  let presignedUrl;
  try {
    const response = await axios({
      method: 'POST',
      url: '<API_GATEWAY URL OR CUSTOM DOMAIN>/generate-presigned-url',
      data: {
        username: 'johndoe',
      },
      headers: {
        'Content-Type': 'application/json',
      },
    });

    presignedUrl = response.data;
  } catch (error) {
    console.error('Error while generating presigned url', error.message);
    throw new Error(error);
  }

  const fileToUpload = fs.readFileSync(path.join(__dirname, fileName));
  try {
    const uploadResponse = await axios({
      method: 'PUT',
      url: presignedUrl,
      data: fileToUpload.toString(),
      headers: {
        'Content-Type': 'application/json',
      },
    });

    return uploadResponse.data;
  } catch (error) {
    console.error('Error while uploading object to S3:', error.message);
    throw new Error(error);
  }
};

uploadToS3('banana.json').then(console.log).catch(console.error);

The code has two axios calls.

The first one is to fetch the generated presigned URL by sending a POST request with the user name in the body to the API endpoint exposed by the API Gateway. The <API_GATEWAY URL OR CUSTOM DOMAIN> part needs to be replaced with either a raw API Gateway deployment URL or a custom domain (the latter more frequently occurs in production). The Lambda handler is behind the /generate-presigned-url endpoint.

The second API call is a PUT request, which is the required method for uploading objects through presigned URLs.

After the above code is run, the nutrition data of the banana should be in S3 under a key similar to johndoe/1595114575104.

A quick CLI command (or AWS Console check) can verify this:

aws s3api list-objects-v2 --bucket NAME_OF_THE_BUCKET_HERE

# response
{
  "Contents": [
    {
      "Key": "johndoe/1595114575104",
      "LastModified": "2020-07-18T23:22:57.000Z",
      "ETag": "\"e198f1f1ed27a8df91b18c07cddd1ef6\"",
      "Size": 119,
      "StorageClass": "STANDARD"
    }
  ]
}

5. Summary

Giving permanent S3 access to users of an application might not always be the best solution. In these cases, presigned URLs can be used.

One popular way to generate a presigned URL is to provision a Lambda function behind an API Gateway, create the URL in the body of the function and return the URL in the response.

The client can then use this URL to upload objects to S3.

Thanks for reading and see you next time.

Related posts:

Generating pre-signed URLs in NodeJS