A Simple Serverless Computing Implementation Using AWS Lambda And S3

In this tutorial we are going to build a simple automated application using lambda which will copy the image files from primary S3 bucket to a secondary S3 bucket when a user uploads an image to the primary S3 bucket. We will also append a timestamp to the copied filename and delete the original file. We will use Java for writing our code.

AWS Technologies used :

AWS Lambda : Is a serverless computing (backend services) offering by Amazon AWS, which means you don't have to worry anymore about setting up or configuring a sever, but just concentrate on your code and Lambda will run it for you by managing the resources required by it. Lambda will handle scaling by increasing or decreasing the number of instances to run your code based on the demand. The service is charged only for the processing time and memory used unlike EC2 which is charged per hour basis. Lambda supports following languages as of this day - Ruby, Python, NodeJS, Java, .NET(C#), Go. Lamda is driven by events (Triggers) via various sources such as S3, SNS (Simple Notification Service), SQS (Simple Queue Service), Kinesis, Code commit. Alexa and many more. Events include some examples like - An event triggered when a file uploaded to S3, A message notification is sent to SNS, A feed received on Kinesis stream, etc.

AWS S3 (Simple storage service) : Is a cloud storage service offered by Amazon. It is charged based on the storage capacity and the bandwidth used and is easily scalable.

Let's get started. 

We will first download and install the Eclipse IDE for writing our Java code.

https://www.eclipse.org/downloads/

When installing you can select Eclipse IDE for Java EE to get most of the plugins.

After opening eclipse IDE select a directory for eclipse to save all your project settings. Next go to help and select Install New Software. Enter https://aws.amazon.com/eclipse and hit enter. Eclipse will download the list of plugins available. Once ready, search for lambda in the next search box. Select AWS Lambda plugin and install the same.

When it is successfully installed restart eclipse, and this time you will see AWS icon in the toolbar. 

If you have aws toolkit already installed and configured (using aws configure from command prompt) you will not be asked for access key details. Other wise you need to generate an access key from IAM console (or use existing one) and use it in the preferences window of AWS icon drop down.

Next from the drop down option select New AWS Lambda Java project. Enter your project name and Artifact Id.  We will use S3 event, but if you select the drop down, you will see other options like SNS, Kinesis too. Click finish. The plugin will automatically create a basic skeleton for you.

We will be performing some modifications to this file. We see that there are some lambda and S3 libraries already imported. We will add 2 more - CopyObjectRequest (To copy our images) and DeleteObjectRequest (To delete the original image)

package com.amazonaws.lambda.test;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.CopyObjectRequest;
import com.amazonaws.services.s3.model.DeleteObjectRequest;

Below you will see 4 highlighted lines of code which does our tasks. When a file is uploaded, an event is triggered and the function - handleRequest() from LambdaFunctionHandler class is called. This functions receives an S3Event as an argument, from which we can get the details of our source S3 bucket which raised the event. The CopyObjectRequest() requires 4 arguments - the source bucket name, the source filename, the target bucket name and the target filename. I have given the target bucket name as "output-testing" which will be bucket I am going to create for target. The timestamp is also appended to the target filename using System.CurrentTimeMillis(). DeleteObjectRequest() requires 2 arguments - Bucket name and the filename. But this code will not work unless we setup the source/target S3 buckets and Lambda on AWS. 

public class LambdaFunctionHandler implements RequestHandler {

    private AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();

    public LambdaFunctionHandler() {}

    // Test purpose only.
    LambdaFunctionHandler(AmazonS3 s3) {
        this.s3 = s3;
    }

    @Override
    public String handleRequest(S3Event event, Context context) {
        context.getLogger().log("Received event: " + event);

        // Get the object from the event and show its content type
        String bucket = event.getRecords().get(0).getS3().getBucket().getName();
        String key = event.getRecords().get(0).getS3().getObject().getKey();

        try {
            S3Object response = s3.getObject(new GetObjectRequest(bucket, key));
            String contentType = response.getObjectMetadata().getContentType();
            context.getLogger().log("CONTENT TYPE: " + contentType);
            CopyObjectRequest copyObjRequest = new CopyObjectRequest(bucket, key, "output-testing", key + "_"  + System.currentTimeMillis());
            s3.copyObject(copyObjRequest);
            DeleteObjectRequest delObjRequest = new DeleteObjectRequest(bucket, key);
            s3.deleteObject(delObjRequest);

            return contentType;
        } catch (Exception e) {
            e.printStackTrace();
            context.getLogger().log(String.format(
                "Error getting object %s from bucket %s. Make sure they exist and"
                + " your bucket is in the same region as this function.", key, bucket));
            throw e;
        }
    }
}

Now lets go to our AWS console to create our S3 buckets and Lambda service.

To create S3 buckets search for S3 in the main AWS console homepage and select S3.

On the next page click on create bucket and enter the bucket details like name. Click create. I have created 2 buckets with name input-testing for source and output-testing for target.

Switch to Eclipse again. Right click on your code and select AWS Lambda -> Upload function to AWS lambda. Enter a name for your Lambda function and click next. In the next window click on Create next to IAM Role to create a new role to use for execution. Enter a name and Click Ok.

Now go back to AWS console home and search for Lambda.  You will see the newly created Lambda function here.

Let us add one more privilege to the lambda IAM role - to have read/write access to S3 bucket.

Search for IAM in the AWS console and find the new role that was created during the Lambda function upload process. IAM -> Roles

Click on the role to open it. Click on Attach policies under permissions. Search for S3, Select Amazon S3 full access and click on attach policy.

Next Open Lambda again from AWS console and click on your Lambda function. Click on Add trigger. And from the drop down select S3.

Select the input bucket name from the drop down. Select event type as All object create events. And in the suffix we will use .png to apply trigger only to png images. You can keep it blank or or add another filter like .jpg as per your preference. Our code is uploaded to the same bucket so we have added the filter to avoid deletion of our code. Accept the disclaimer and click add.

You should see the trigger attached to the Lambda function.

Now go to your input bucket from AWS Console -> S3. Upload a .png/.jpg image file to test.

Now open the output bucket. You should see your newly copied file with timestamp here.