The worst thing about a server going down is not the outage: it is finding out from a customer two hours later. AWS can warn you as soon as something goes wrong, but out of the box it warns about nothing: you have to ask for it machine by machine. This guide sets up the two alerts most worth having, "the machine is not responding" and "the processor has been flat out for a long time", so that they reach your inbox.
Lightsail and EC2 do it differently. Lightsail has it built in and it is simpler; EC2 uses two AWS services with names of their own, CloudWatch and SNS, which we explain below.
Before you start: what each thing means
- Metric
- A figure AWS measures on your machine every few minutes: processor usage, network traffic, whether it responds to status checks...
- Alarm
- A rule of the kind "if this metric goes above such a value for so long, warn me".
- CPU
- The processor. If it is at 100 % for a long time, the website is slow or something has hung.
- Status check
- In EC2, an automatic test AWS runs every minute to see whether the machine responds. If it fails, the machine is down or hung.
- CloudWatch
- The AWS service that stores metrics and manages alarms for EC2 and the other services.
- SNS
- Simple Notification Service, Amazon's notification service: it receives a message from an alarm and sends it to whoever is subscribed, for example to your email.
- SNS topic
- The "channel" notifications are sent to and that you subscribe to with your email address.
Step by step
Lightsail: open the machine's metrics
In the Lightsail console, click the machine and then the Metrics tab. You will see graphs for CPU, network and status checks. Below each graph there is an Add alarm link.
Lightsail: create the "not responding" alarm
On the Status check failures graph, click Add alarm. Condition: greater than or equal to 1 for 2 periods of 5 minutes (that way you avoid an alert for a one-minute blip). Under Notifications, tick Email and enter your address; the first time, Lightsail sends you a confirmation email you have to accept. If you prefer, in some regions it also supports SMS. Save.
Lightsail: create the CPU alarm
On the CPU utilization graph, Add alarm. Condition: greater than 90 % for 3 periods of 5 minutes, that is, a quarter of an hour at the limit. Same email. If your machine is one of the small ones, on the graph itself you will also see the "burstable" zone: small machines can run at 100 % for a while, but not continuously.
Optional and very useful: on the same tab, the high outgoing network traffic alarm warns you that the website is getting far more visits than usual or that something is sending data it shouldn't.
EC2: create the notification channel (SNS topic)
For EC2 you first need the "channel". Open the SNS service, Topics, Create topic, type Standard, name
avisos-servidores. Inside the topic, Create subscription, protocol Email, your address. You get an email with a confirmation link: click it. Until it says "Confirmed", you won't receive anything. You can add several addresses.EC2: create the two alarms from the machine
In the EC2 console, select the machine, Monitoring tab, Create alarm button (or Actions, Monitor and troubleshoot, Manage CloudWatch alarms). First alarm: metric Status check failed, condition greater than or equal to 1 for 2 periods of 5 minutes, notification to the
avisos-servidorestopic. If you want AWS to reboot the machine on its own when the status check fails, tick the "Reboot" action: for a website, it is usually what you would want anyway.Second alarm: metric CPUUtilization, greater than 90 % for 3 periods of 5 minutes, same notification. Repeat on every machine you care about.
Test that it arrives
Don't wait for the day of the problem. In CloudWatch, open the alarm and use Actions, Set alarm state to force it for a moment: the email should arrive. In Lightsail, stop the machine from the console for a few minutes (if you can afford to) and check that it warns you. Put everything back as it was.
What can go wrong
No email arrives
The subscription is unconfirmed or the confirmation email went to spam. In SNS, the subscription must say "Confirmed". In Lightsail, the email contact must appear as verified on the Metrics tab.
It warns me every five minutes
The thresholds are too tight. Raise the percentage or lengthen the time (for example 90 % for 30 minutes). An alert that gets ignored is worse than none: fine-tune until every alert means something.
The machine went down and didn't warn me
If the machine was stopped from the console (someone clicked Stop), the status check doesn't count it as a failure: the machine is stopped, not broken. For that you need separate monitoring that compares the actual state with the one it should have.
I want the alert on my phone, not by email
In SNS, besides email, you can subscribe a phone number (SMS, at a cost and not in every region) or an application of your own. And there are on-call services that connect to SNS. To start with, email on your phone does the job.
In Simplio
In Simplio there is no need to create alarms one by one: on each machine you tick what you want to watch (unexpected stop, processor at the limit, disk full, backup not made) and Simplio checks all your AWS accounts every few minutes and warns you by email and in the app, one alert when it appears and another when it is resolved. Each person on your team chooses which alerts they receive. And since Simplio knows whether a machine should be running, it also warns you when someone stops it from the console.
The same task, in AWS and in Simplio
| In the AWS console | In Simplio | |
|---|---|---|
| Alert for machine down or high CPU | Lightsail: 2 alarms per machine. EC2: SNS topic, subscription, confirmation, 2 alarms per machine: 6 steps, 15 minutes per machine | Tick what to watch on each machine: 2 boxes |
| If someone stops the machine from the console | The alarms don't detect it | Simplio knows whether it should be running and warns you |
| Several accounts and regions | Repeat in each one | All accounts watched from the same place, each person chooses their alerts |