S3 Pre-Signed URLs

In the course “Security Engineering on AWS” there is a section on S3. On one of the slides there is a bullet point:

Use pre-signed URLs for applications that refer to S3 objects

The course notes say:

“Make use of pre-signed URLs for applications that refer to S3 objects with anonymous access, e.g. downloading of restricted content. Authentication and authorization must be done in the application.”

One use case is as follows. You have an application which requires the customer to register with the application in order to access content, for example some videos in S3. You do not want to make the videos public as then anyone who knows the URL would be able to access them without paying.

The application would typically use the SDK to request a pre-signed URL. This is a URL with a temporary security token appended to it. The token expires in a configurable time. The token is validated by S3 when the object access is requested.

To test it:

Create a role and attach the policy S3FullAccess.

Launch an AMZ Linux EC2 Instance with the role attached.

SSH in to the instance, use aws configure as usual to choose  the a default region. You may need to research this bit if you don’t already have such an instance)

Issue the following commands (you will have to replace the bucket name, and you can use any file. I copied a jpg to the instance for this test)

aws s3 mb s3://bucket-presigned-urls
aws s3 cp file.jpg s3://bucket-presigned-urls

Now switch to the console, navigate to the object and click on the Object URL which will look something like this:

Object URL


You will get Access Denied. This is because the object does not have public permissions.

Back in the SSH terminal window, create a pre-signed URL:

aws s3 presign s3://bucket-presigned-urls/file.jpg --expires-in 60 

The –expires-in is optional. The default is 1 hour.

The output of the command will be similar to:

https://bucket-presigned-urls.s3.amazonaws.com/file.jpg?AWSAccessKeyId=ASIA26NEMI3H7GPEO2D5&Expires=1562947328&x-amz-security-token=XXXX <several lines of text ommitted>

Copy and paste the URL into a browser. It should work.

Wait 1 minute and try again. You should get “Access Denied. Request has expired”


Amazon Macie

In the “Security Engineering on AWS” course there is an overview of Macie.

Amazon Macie is a security service that uses machine learning to automatically discover, classify, and protect sensitive data in AWS.

As usual for these blogs, I assume you have seen a basic introduction, possibly having attended a course, and want to spend a little time getting hands on with the service.

At the time of writing, Macie is only available in selected regions, including N. Virginia.

To get started with Macie, create a bucket in one of the supported regions.

Create some content with some dummy PII data. Here are some ideas:

Create a spreadsheet with  a name, address, phone, email, credit card number. Use only fictitious data.

Create an EC2 keypair. Do not use it to launch an instance. Download the pem file to your local machine.

Create a dummy IAM user with CLI credentials. Do not give the user any permissions. Download the credentials.csv file to your local machine.

Select the Macie service and enable it. It shows you the service role it will use. Enabling it takes a few seconds.

Choose Integrations, Add and select your bucket. You will also see a bucket it has created for CloudTrail logs. It uses CloudTrail to log S3 data events in order to analyse the activity to your bucket.

Upload the files to the bucket.

It may take time before any useful data is seen. Maybe leave it for an hour.

Meanwhile, have a look at the Settings menu to see the content types, file extensions, themes and regular expressions that it will use to classify data. Note that at the time of writing, there are some limitations. For now it only works with S3 although it may support other data sources in the future, for example EBS, EFS, RDS, DynamoDB.

The classifications are U.S. centric, for example US format driving licenses are supported but not UK format.

For now you can’t customise things like the regular expressions it uses to classify data.

After leaving it for a while, choose the Alerts menu. In my case, I see an alert to do with the pem file, with a  description as follows:

“RSA Private Key uploaded to AWS S3. An RSA key is the private encryption key that will be used to protect sensitive information. Please verify that the storage of credential material in this S3 bucket is in compliance with your organization’s policies and that properly locked down access control mechanisms are in place to protect these credentials”

Try and drill down to how it has classified the data and note that it has used a regex to identify it.

Choose the Research menu and  select “s3 Objects” from the drop down list. It should have identified some of the PII and other secret data in the files you uploaded.

Have a look at the pricing and make a decision on whether to disable Macie or leave it enabled. Pricing is based on volume of data and frequency of access.

To clean up, Choose Integrations and remove the bucket from the list.

Choose the logged in username at the top of the screen, Macie General Settings and disable Macie.