Serving static web site from S3

Performance of the SDK for uploading to S3

S3 SDK does not implement the sync command. Therefore we must upload each generated file one by one (using the upload function). This is not very performant. By way of comparison, creating a zip file of all the generated contents, copying it to a remote server (at Dreamhost) via scp, and then expanding the zip on the remote host (again via scp) took about 32 seconds. Uploading the same set of generated files one by one via S3.upload took about 1 minute 28 seconds. Not cool.

Using the AWS CLI for the sync to S3

By leveraging the AWS CLI we can make use of the s3 sync command, which is optimized for syncing a local directory and S3 bucket (or vice-versa) and has many options. For our use case we just want to ensure that whatever we have just generated is uploaded to S3 to ensure our web site has the latest content. The first time s3 sync runs on a directory it will have to upload every single file. The next time it runs it will only have to upload things that are new or changed, thus reducing the execution time. When uploading my full site (from my local host) it took about 20 seconds the first time, and about 2 seconds the next time (with no changes).

But the AWS CLI is not available in Lambda

The AWS CLI is not part of the Lambda runtime by default. But all is not lost. We can create a Lambda layer to get it. I won’t detail the steps since others have done it already (see Call aws-cli from AWS Lambda and How to use AWS CLI within a Lambda function for a few examples).