"Who stopped the server?" "It wasn't me." When several people touch the same AWS account, sooner or later someone does something nobody owns up to. The good news is that AWS writes down everything: every action taken from the console, from a program or from an external tool is recorded with who, when, from which address and on what. That log is called CloudTrail and it is on by default, without you having done anything.
What is not so simple is finding what you are looking for, because the names are technical and the log is only kept for 90 days. This guide shows you how to search and how to keep the history.
Before you start: what each thing means
- CloudTrail
- The AWS service that records every action taken in your account: who, what, when, from where and on which resource.
- Event
- Each recorded action. It has a technical name in English (StopInstances to stop a machine, DeleteObject to delete a file...).
- Event history
- The CloudTrail screen where you can search the events of the last 90 days without setting anything up and at no cost.
- Trail
- A configuration that saves the events permanently in an S3 bucket of yours, to keep them for more than 90 days.
- User or role
- Who took the action: an IAM user (a person or a program), an assumed role (for example, an external tool) or the root account.
- Source IP
- The internet address the action was taken from. It lets you tell the office apart from an unknown place.
- Region
- The history is shown per region; an action taken in Paris does not appear if you are looking at Ireland.
Step by step
Open the event history
In the AWS console search for CloudTrail and go to Event history. You will see a list of the most recent actions: time, event name, who and on what. At the top is the region selector: choose the region where what you are investigating happened. Some global actions (IAM ones, for example) appear in the account's main region.
Search by what you know
Above the list there is a filter with a dropdown. The most useful searches:
- Resource name: paste the machine's identifier (
i-0abc...), the bucket name or the Lightsail machine name. Everything that has been done to it appears. - Event name: if you know what happened. Stopping an EC2 machine is
StopInstances; in Lightsail,StopInstance; deleting an S3 file,DeleteObject; changing a DNS record,ChangeResourceRecordSets; deleting a snapshot,DeleteSnapshot. - User name: to see everything a specific person did.
Narrow the date range to the day in question so you do not get lost in the list.
- Resource name: paste the machine's identifier (
Read the event details
Click the event. In the record you will see the exact user (and if it was through a role, which role), the source IP address, the agent (whether it was from the web console, from the command line or from an external tool) and, in the JSON block below, the technical details. With the time, the IP and the user it is usually clear who it was and from where, even if it was by accident.
Work out whether it was a person or a program
If the user is someone on the team and the IP is the office one, it was that person from the console. If the user is one created for a tool (a panel, a backup system) or an assumed role, it was that tool: look at what was scheduled for that time. If the user is the root account or the IP is from a country where you have nobody, treat it as a possible intrusion: change passwords and keys and go over the guide on protecting the account.
Keep the history for more than 90 days
The event history only keeps three months. For a business that is not much: the question "who deleted this" may come the following year. In CloudTrail, Trails menu, Create trail: give it a name, let it create a new S3 bucket to store it, tick that it applies to all regions and create. From then on every event is saved as files in that bucket; the first trail of management events has no CloudTrail cost, although there is the small cost of the space in S3.
Search the archive when you need to
The files in the bucket are compressed JSON, awkward to read by hand. To search them, CloudTrail offers event data stores with SQL-style queries (there is a cost) or you can download the specific day and open it in an editor searching for the resource identifier. For most people, knowing they are there and can be retrieved is enough.
What can go wrong
I cannot find the event even though I know it happened
Wrong region (change it at the top), date range too short, or the event has a different name from the one you think. Try filtering only by the resource name and looking at everything that has happened to it.
Events take a while to appear
CloudTrail is not instant: an action can take several minutes to show up in the history. If it has just happened, wait a little.
Everyone signs in with the same user and I do not know who it was
Then the log cannot help you. Create one user per person with two-factor (guide in this section) and, from then on, every action will have a name attached.
I do not see reads, only changes
By default the history shows management events (create, change, delete). Data reads (opening an S3 file, for example) are only recorded if you enable them in a trail, and they generate a lot of volume.
In Simplio
In Simplio every action someone takes from the application (stopping a machine, deleting a file, changing the firewall, opening the terminal) goes into the activity log with user, date, IP and object, written in plain language and searchable by machine or by person. Sensitive actions ask for two-factor before running, so there is no "it wasn't me". And the monthly report gathers that activity to hand to whoever needs it.
The same task, in AWS and in Simplio
| In the AWS console | In Simplio | |
|---|---|---|
| Know who stopped a machine | CloudTrail, choose region, filter by resource, read the JSON: 4 steps, 10 minutes, if you know the event name | Machine activity log: one line with user, date and IP, in plain language |
| Keep the history | Create a trail and an S3 bucket, pay for storage, know how to read JSON | The activity log is kept in Simplio and exported to the monthly report |
| What was done outside the tool | CloudTrail has it all, raw | What is done from the AWS console is still in CloudTrail; Simplio records everything done from Simplio |