X-Ray

In the “Security Engineering on AWS” course there are two slides on X-Ray.

From the course notes:

AWS X-Ray is a service that collects data about requests that your application serves, and provides tools you can use to view, filter, and gain insights into that data to identify issues and opportunities for optimization. For any traced request to your application, you can see detailed information not only about the request and response, but also about calls that your application makes to downstream AWS resources, microservices, databases and HTTP web APIs.

There is a very cool sample application to get started.

From the X-Ray console, click “Launch a sample application (Node.js)”

It will take a few minutes. It uses Cloud Formation to create an Elastic Beanstalk application, and Elastic Beanstalk creates a second stack.

When both stacks are complete, in the Elastic Beanstalk click on the Elastic Beanstalk application URL to see the sample application:

Click on the sign-up form and supply a Name and Email address (it dosn’t have to be real). This will be a POST request to the application.

In the X-Ray console, click on “Service Map”, to see a screen similar to this:

The leftmost circle represents the HTTP requests to the web page. The other circles are:

  • A call to the metadata service to retrieve the security credentials supplied by the Instance Profile, used to make the Dynamo DB and SNS API calls.
  • A Dynamo DB table. The application is writing the users details to a Dynamo DB table. If you go to the Dynamo DB console you will see it.
  • An SNS Topic. The application is publishing the users details to an SNS topics. If you go the SNS console, you will see an unconfirmed subscription to the email address you used, and a confirmed subscription from an SQS queue.

In the X-Ray console, click “Traces” to see the URLs accessed by the client.

It will include the initial GET to the web application and a POST to the signup page.

You may also see a GET to the favicon, with a 404 not found.

“A favicon /ˈfæv.ɪˌkɒn/ (short for favorite icon), also known as a shortcut icon, website icon, tab icon, URL icon, or bookmark icon, is a file containing one or more small icons, associated with a particular website or web page.”

Drill down to the details of the trace for the signup page.

You see a timeline including the total time for the POST.

This is broken down into the DynamoDB putitem followed by the SNS Publish with the times for each.

You may also see the calls to the metadata service to retrieve security credentials.

Back in the application, click “Start” and leave it for 2 minutes to make about 10 automated signups per minute. Now we start to see average figures for each of the circles in the service map.

The application intentionally includes signups with a duplicate email address, which causes Dynamo DB to return a 400 error, and the POST to return a 409 error. These errors can be seen in the traces.

“An HTTP 400 status code indicates a problem with your request, such as authentication failure, missing required parameters, or exceeding a table’s provisioned throughput”

Just for fun, I removed the sns:publish permission from the policy attached to the Role that the instance is using.

The service map starts to display orange circles, and you can drill down the traces to see the detail:

AuthorizationError: User: arn:aws:sts: <output ommitted> is not authorized to perform: SNS:Publish on resource: arn:aws:sns:<output omiited>

The POST returns a 500 error.

In summary, X-Ray is helping us to indentify both latency issues and intermittent errors returned by a service.

To clean up, delete the X-ray cloud formation stack, which will in turn delete the Elastic Beanstalk stack.