Using AWS Lambda with Transposit

AWS Lambda functions are a powerful integration tool. They allow you to run (almost) arbitrary codebases (more on Lambda here), and are great for a variety of use cases like processing log or moving files in a data pipeline. In this post, I'm going to cover basics of using a Lambda in your Transposit application, show an example of file munging, and why you might choose to do so.

Transposit makes it easy to use Lambda functions. It's a great pairing, like wine and cheese. All you need is to make sure:

• the AWS account with which you are accessing the Lambda has invoke permissions (as well as any permissions needed for other resources).
• the Lambda returns a JSON object as a result.
• the Lambda finishes in less than five minutes (if longer, use the Event invocation type to invoke the Lambda asynchronously).

Why not do everything in Lambda?#

First, if Lambda is so great, why not do everything in Lambda? There are a few reasons why developing in Transposit is better than developing in Lambda.

Deployment#

Deployment is far easier with Transposit than with Lambda. Especially if you are receiving an HTTP request, where Transposit endpoints are trivial to set up. You can add endpoint authentication with a dropdown selection and a check of a box (compared to 15+ steps with Lambda).

Development#

Development is easier with Transposit. To be fair this is in part because Lambda has so many configuration options due to its power. But that complexity is imposing and you often don't need all that power. In addition, Transposit applications are all version controlled with git. You can of course use external version control for Lambda (and you should! please!) but it's not enforced from the get-go.

Access to APIs#

With Lambda, you have access to AWS APIs through the SDK. Other APIs are available too, but you have to manage authentication yourself, embedded in the Lambda code. You also have to learn each SDK and the peculiarities of how to use it. Transposit provides a uniform interface and prebuilt connections to over seventy APIs.

For another take on Transposit's strengths, read this post, which contrasts a Lambda that cleans up old ECS images with a Transposit application which does the same.

Combining Lambda and Transposit#

Here's an example of calling a Lambda function from a Transposit operation. This code calls a pre-existing Lambda named removeNewLinesFromTheFullOrdersFile and passes two parameters to the Lambda, fileS3Bucket and fileS3Key.

SELECT * FROM aws_lambda.invoke
  WHERE FunctionName='removeNewLinesFromTheFullOrdersFile'
  AND X-Amz-Invocation-Type='RequestResponse'
  and $body=(SELECT {
    'fileS3Bucket' : @fileS3BucketName,
    'fileS3Key' : @fileS3KeyName
  })

Here's the JavaScript Lambda that is being called. It reads from an S3 location, and writes back to that same object after removing any embedded newlines in a CSV file.

var AWS = require('aws-sdk');

var s3 = new AWS.S3();

exports.handler = async (event) => {
    const bucket = event.fileS3Bucket;
    const key = event.fileS3Key;

    const params = {
        Bucket: bucket,
        Key: key,
    };

    const fullorderscsv_raw = await s3.getObject(params).promise();

    const fullorderscsv_removed_embedded_newlines = fullorderscsv_raw.Body.toString('utf-8').replace(/"[^"]*(?:""[^"]*)*"/g, function(m) { return m.replace(/\n/g, ' '); });

    params["Body"] = fullorderscsv_removed_embedded_newlines;
    const result = await s3.putObject(params).promise();

    const response = {
        statusCode: 200,
        body: JSON.stringify(result)
    };
    return response;
};

Why reach for Lambda?#

Lambda is a great way of encapsulating functionality. In some ways, it's like a private, low maintenance API. If you build a private API for a Transposit application, you need a data connector, and to plan to maintain the API and its infrastructure. If you have multiple client applications or a need for a robust API, this investment makes sense, but being able to craft an easy solution on top of Lambda is sometimes a better choice.

There are three reasons you might reach for a Lambda function to power a portion of your Transposit application.

Libraries#

There may be certain libraries that you want to use in your application. For example, there are great Java libraries for planar geometry and constraint solving. If you want to use these in a Transposit application, you create a Lambda function which provides glue code to take requests, call the libraries and returns the response. If you have more data than can be passed in a call, you can use S3 or DynamoDB as data stores; the Lambda and Transposit application can use the data store as a 'bus'.

Or perhaps you already have business logic in a Lambda function that you want to re-use. Re-use will accelerate development and minimize logic duplication.

Language#

You may want to work in another language because it is the appropriate level of abstraction for the problem; Lambda supports far more languages than Transposit. Depending on the domain or problem difficulty, you can write in a type safe language, which will avoid certain classes of bugs.

Limits#

There are things you cannot do in Transposit because of limitations. For example, if you wanted to process a large amount of data in-memory, Lambda may be a better fit than Transposit because you can scale to 3GB of memory. There may occasionally be a bug in Transposit that you need to work around (but please tell us about it so we can fix it). You may need to access a service that only talks SOAP or FTP, or some other format that is not OpenAPI compatible. Finally, Lambda can reach into areas of your infrastructure not accessible to Transposit, such as a VPC or an on premise network.

Conclusion#

Transposit and AWS Lambda are a powerful pairing, especially when you build an application that leverages the strengths of each. Using Lambda allows you to create a simple, focused JSON endpoint with access existing systems and logic, but without building a full API. You can leverage that from the friendly deployment and developer environment of Transposit.