S3 Pre-Signed URLs

In the course “Security Engineering on AWS” there is a section on S3. On one of the slides there is a bullet point:

Use pre-signed URLs for applications that refer to S3 objects

The course notes say:

“Make use of pre-signed URLs for applications that refer to S3 objects with anonymous access, e.g. downloading of restricted content. Authentication and authorization must be done in the application.”

One use case is as follows. You have an application which requires the customer to register with the application in order to access content, for example some videos in S3. You do not want to make the videos public as then anyone who knows the URL would be able to access them without paying.

The application would typically use the SDK to request a pre-signed URL. This is a URL with a temporary security token appended to it. The token expires in a configurable time. The token is validated by S3 when the object access is requested.

To test it:

Create a role and attach the policy S3FullAccess.

Launch an AMZ Linux EC2 Instance with the role attached.

SSH in to the instance, use aws configure as usual to choose  the a default region. You may need to research this bit if you don’t already have such an instance)

Issue the following commands (you will have to replace the bucket name, and you can use any file. I copied a jpg to the instance for this test)

aws s3 mb s3://bucket-presigned-urls
aws s3 cp file.jpg s3://bucket-presigned-urls

Now switch to the console, navigate to the object and click on the Object URL which will look something like this:

Object URL


You will get Access Denied. This is because the object does not have public permissions.

Back in the SSH terminal window, create a pre-signed URL:

aws s3 presign s3://bucket-presigned-urls/file.jpg --expires-in 60 

The –expires-in is optional. The default is 1 hour.

The output of the command will be similar to:

https://bucket-presigned-urls.s3.amazonaws.com/file.jpg?AWSAccessKeyId=ASIA26NEMI3H7GPEO2D5&Expires=1562947328&x-amz-security-token=XXXX <several lines of text ommitted>

Copy and paste the URL into a browser. It should work.

Wait 1 minute and try again. You should get “Access Denied. Request has expired”


Amazon Macie

In the “Security Engineering on AWS” course there is an overview of Macie.

Amazon Macie is a security service that uses machine learning to automatically discover, classify, and protect sensitive data in AWS.

As usual for these blogs, I assume you have seen a basic introduction, possibly having attended a course, and want to spend a little time getting hands on with the service.

At the time of writing, Macie is only available in selected regions, including N. Virginia.

To get started with Macie, create a bucket in one of the supported regions.

Create some content with some dummy PII data. Here are some ideas:

Create a spreadsheet with  a name, address, phone, email, credit card number. Use only fictitious data.

Create an EC2 keypair. Do not use it to launch an instance. Download the pem file to your local machine.

Create a dummy IAM user with CLI credentials. Do not give the user any permissions. Download the credentials.csv file to your local machine.

Select the Macie service and enable it. It shows you the service role it will use. Enabling it takes a few seconds.

Choose Integrations, Add and select your bucket. You will also see a bucket it has created for CloudTrail logs. It uses CloudTrail to log S3 data events in order to analyse the activity to your bucket.

Upload the files to the bucket.

It may take time before any useful data is seen. Maybe leave it for an hour.

Meanwhile, have a look at the Settings menu to see the content types, file extensions, themes and regular expressions that it will use to classify data. Note that at the time of writing, there are some limitations. For now it only works with S3 although it may support other data sources in the future, for example EBS, EFS, RDS, DynamoDB.

The classifications are U.S. centric, for example US format driving licenses are supported but not UK format.

For now you can’t customise things like the regular expressions it uses to classify data.

After leaving it for a while, choose the Alerts menu. In my case, I see an alert to do with the pem file, with a  description as follows:

“RSA Private Key uploaded to AWS S3. An RSA key is the private encryption key that will be used to protect sensitive information. Please verify that the storage of credential material in this S3 bucket is in compliance with your organization’s policies and that properly locked down access control mechanisms are in place to protect these credentials”

Try and drill down to how it has classified the data and note that it has used a regex to identify it.

Choose the Research menu and  select “s3 Objects” from the drop down list. It should have identified some of the PII and other secret data in the files you uploaded.

Have a look at the pricing and make a decision on whether to disable Macie or leave it enabled. Pricing is based on volume of data and frequency of access.

To clean up, Choose Integrations and remove the bucket from the list.

Choose the logged in username at the top of the screen, Macie General Settings and disable Macie.



In the course “Security Engineering on AWS” there is slide on MFA for AWS CLI.

The scenario is that you are using CLI, your long term credentials allow certain actions, but for more destructive actions like StopInstance, MFA is required.

For example, the user may have a policy like this:

    "Version": "2012-10-17",
    "Statement": [
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "aws:MultiFactorAuthPresent": "true"
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"

A user with this policy can Describeinstances with their long term credentials, but Stop or Terminate require the use of MFA.

To test this, create a user with CLI access and attach the above policy. Use aws configure to configure the long term credentials as normal for CLI access.

To test the scenario, check the users permissions before we set up MFA

aws ec2 describe-instances

It should succeed.

Try to stop an instance. It should fail:

aws ec2 stop-instances –instance-ids i-006564f1892d9be34

An error occurred (UnauthorizedOperation) when calling the StopInstances operation: You are not authorized to perform this operation. Encoded authorization failure message: TWOM9LYAGS…

Associate an MFA device such as Google Authenticator for iPhone with the user. The steps to do this are easy to follow in the management console.

The sts command in the slide requires the arn of the MFA device. This can be found on the Security Credentials tab for the user:

The command also requires the token that is displayed in the authenticator app.
The command will output the temporary credentials as in the slide above.

One way to supply the credentials to the CLI command is to use a profile.

Edit the .aws/credentials file and copy and paste the temporary credentials into a new profile.


output = json

region = eu-west-1

aws_access_key_id = ASIA26NEMI3H7NH3LCNB aws_

secret_access_key = g4nixFvyjO/13xF6qLLVnw7hmiSZFKYCBTKMCDt8

aws_session_token = FQoGZ…

Now try the CLI command again, supplying the profile name.

aws ec2 stop-instances –instance-ids i-006564f1892d9be34 –profile mfa

It should work.

The credentials last an hour by default. As you can see above, the process is cumbersome. Some people have written scripts to automate the update of the credentials file. The script typically prompts for the MFA token. I leave that for your own research.

Transit Gateway

In “Architecting on AWS”, as of July 2019, there are some slides on Transit Gateway, which was introduced as an alternative to VPC peering, especially when there are large number of peerings.

The scenario is that all 3 VPCs need to connect to an on premise site via the VPN, but are isolated from each other. The on premise site is part of network 10, hence the route to 10/8 in the VPC routing tables, of which only one is drawn. The Transit Gateway has a route to the VPN and all 3 VPCs, but it is not obvious why the VPCs are isolated from each other.

The link between the VPCs and the TGW and between the VPN and the TGW are called attachments. Attachments are associated with a route table. From the console: “Associating an attachment to a route table allow traffic to be sent from the attachment to the target route table. One attachment can only be associated to one route table”. So in the diagram traffic from the bottom VPC is send to the green route table, and traffic from the VPN is sent to the red route table.

To demonstrate this kind of connectivity, I set up a lab. For simplicity, I didn’t use a VPN.

To start with, create 3 VPCs in the same region, with non overlapping CIDR ranges. I used, and For simplicity, I created one public subnet in each, using a /24 mask, with an Internet Gateway in each VPC, and modified the route table for each VPC to add a default route to the IGW. Launch an Amazon EC2 instance in each VPC, using a security group which allows SSH and ICMP from anywhere, to keep it simple. I used public subnets simply to make it easy to login to the instances.

To start with, lets establish full mesh connectivity.

Create a Transit Gateway called TGW, leaving all the defaults. It is in the pending state for about 2 minutes. Notice that it creates a route table, with no associations, propogations or routes.

Create an attachment called ATT1 between the TGW and VPC1, leaving all the defaults. It is pending for about 1 minute. Repeat for ATT2 between the TGW and VPC2, and ATT3 between the TGW and VPC3.

Look at the route table again. There is a route to the VPC1 CIDR range via ATT1, and similarly routes to VPC2 and VPC3 via their respective attachements. The routes have automatically propogated from the VPCs

Click the associations tab. The route table is associated with each attachment, in other words traffic from each attachment is using this route table.

Click the propogations tab. This is why the VPC CIDR ranges were propogated to the route table.

We are note quite set up for full mesh connectivity yet. Logon to the instance in VPC1 and try to ping the private IP address of the other instances. The pings fail. This is because the VPC route tables have no route other than the local route and the default route to the IGW.

For each VPC routing table, add a route with target TGW.

Repeat the ping test. They should all work. We have achieved full connectivity between all 3 VPCs.

Now to set up partial connectivity, similar to the graphic above, but without the VPN to keep it simple. I want VPC1 to be able to communicate with the other VPCs, but VPC2 and VPC3 not to communicate with each other.

Select the TGW route table and delete all 3 associations. This takes a few seconds.

Create a TGW route table called RT1. It is pending or about 1 minute. Associate it with ATT1.

Create a TGW route table called RT2-3. Associate it with ATT2 and ATT3

For RT1, add a route to the CIDR range of VPC2 and choose attachment ATT2 (this is similar to a target). Add another route to VPC3 via ATT3.

For RT2-3, add a route to the CIDR range of VPC1 via ATT1.

Repeat the ping test. The instance in VPC2 can ping the instance in VPC1 but cannot ping the instance in VPC3.

To clean up, delete everything as TGW costs about $0.05 per attachment per hour. Delete the route table associations, then the route tables, then the attachments, then the TGW.