Think for a moment about what would happen if tomorrow you could not get into one of your machines. The SSH key was on the laptop of someone who has left; a technician changed the Administrator password and nobody wrote it down; a change to the SSH configuration went wrong and the door closed with you outside. The machine keeps running and the website stays up, but you are no longer in charge of it, and in AWS nobody opens it for you: there is no "I forgot my password" and no support team that logs in on your behalf. Lightsail gets you out of trouble on Linux (its web console logs in with a temporary key), but on Windows only if nobody ever touched the password, and on EC2 the official route involves stopping the machine. When that does not work, what remains is hours of snapshots and new machines, at the worst possible moment. This guide shows you how to leave an emergency door ready before you need it, built so that only you can use it.
Before you start: what each thing means
- Rescue user
- One more administrator user on the machine, not used for everyday work: only to recover access to the rest.
- SSH certificate
- A "key with an expiry date" signed by an authority the machine trusts. Unlike a normal key, it does not have to be copied to the machine and can be valid for only five minutes.
- Certificate authority (CA)
- The key that signs those certificates. The machine accepts whoever brings a certificate signed by it; without the CA none can be made.
- from=
- A condition in the SSH configuration: that user can only log in from one specific IP address.
- sudo / Administrators
- How you are an administrator on Linux (sudo) and on Windows (the Administrators group).
Step by step
Decide what you want the emergency door to be like
A badly made rescue user is worse than none: a password written down in a shared document is the first door an attacker tries. What we want: that it has no password, that it only accepts certificates signed by you and short-lived, and if possible that it only accepts connections from a fixed IP of yours. That way, even if someone knows the user name, there is nothing for them to try.
Create a certificate authority (once only)
On your computer, not on the machine:
ssh-keygen -t ed25519 -f ca_rescate -C "CA de rescate"Keep
ca_rescate(the private key) somewhere safe: a password manager or an encrypted disk. Never copy it to the machines. Theca_rescate.pubfile is the one that will go to each machine.Prepare the Linux machine
Log in over SSH as usual (you still can) and, with sudo:
sudo useradd -m -s /bin/bash -c "Usuario de rescate" rescate sudo usermod -p "*" rescate # sin contraseña valida, sin bloquear echo "rescate ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/90-rescate sudo chmod 440 /etc/sudoers.d/90-rescate sudo cp ca_rescate.pub /etc/ssh/ca_rescate.pub echo 'from="TU.IP.FIJA" rescate' | sudo tee /etc/ssh/principales_rescateAnd in
/etc/ssh/sshd_config(or a file in/etc/ssh/sshd_config.d/), at the top:TrustedUserCAKeys /etc/ssh/ca_rescate.pub AuthorizedPrincipalsFile /etc/ssh/principales_rescateCheck and reload:
sudo sshd -t && sudo systemctl reload ssh. Therescateuser has no password and only accepts certificates from your CA, and only from your IP if you setfrom=.Prepare the Windows machine
Windows Server 2019 and later include OpenSSH as an optional feature. In PowerShell as administrator:
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0, start thesshdservice and open port 22 in the Windows firewall and in the Lightsail or EC2 one (only for your IP). Create the user with a long random password that you do not write down anywhere (do not create it without a password: by default Windows blocks network access for accounts with a blank password, and OpenSSH counts as network access), add it to Administrators, copyca_rescate.puband the principals file toC:\ProgramData\ssh\and, in thesshd_configin that same folder, put the same two lines as on Linux (with the paths as__PROGRAMDATA__/ssh/). A detail that costs hours: Windows OpenSSH rejects authentication files that other users can read, so remove permission inheritance from them and leave only Administrators and SYSTEM:icacls FICHERO /inheritance:r /grant Administrators:F /grant SYSTEM:F. Restartsshd. If you want PowerShell instead of cmd when you log in, set the registry keyHKLM:\SOFTWARE\OpenSSH\DefaultShelltopowershell.exe.Test the door before you need it
Generate a normal key and sign it with the CA for five minutes:
ssh-keygen -t ed25519 -f temporal -N "" ssh-keygen -s ca_rescate -I "prueba" -n rescate -V +5m temporal.pub ssh -i temporal rescate@IP.DE.LA.MAQUINA sudo -n true && echo OKIf you see OK, the door works. After five minutes the certificate stops being valid on its own: there is nothing to revoke or delete.
If you have already lost access: what to do once the door is open
Log in with a new certificate and fix the administrator: on Linux
sudo passwd ubuntu(or whichever user it is) or add your new public key to its~/.ssh/authorized_keys; on Windowsnet user Administrator NUEVA-CONTRASEÑA. If the account was locked,sudo passwd -u usuarioornet user usuario /active:yes.And if you had no emergency door
Linux on Lightsail: the Lightsail web console logs in with a temporary key, so from there you can do what the previous step describes. Windows on Lightsail: the console gives you the Administrator password only if nobody changed it; otherwise you will have to create a snapshot and a new machine. EC2: the official AWSSupport-ResetAccess procedure in Systems Manager resets the password or the key by stopping and starting the machine.
And if there is no snapshot either
This is the end of the chain and it is worth looking at it squarely: if you cannot get in and no copy of the machine exists, everything inside it is lost. The application, the database, the files, the configurations. AWS cannot log in for you or recover what was never copied, and there is no technical service to call. That is why the emergency door and automatic snapshots are not two options to choose between: they are the two things that separate an hour-long scare from losing a client's business.
What can go wrong
"Permission denied" even though the certificate is correct
It is almost always the from=: you are connecting from a different IP. Or the user's password field is locked with ! (which useradd leaves by default on some systems): set it to * with usermod -p "*".
"Certificate invalid: not yet valid" in the sshd log
The machine's clock is behind. A certificate valid for minutes has a start time, and if the server believes it has not arrived yet, it rejects it with a Permission denied that explains nothing: it only shows up with LogLevel DEBUG. On Linux, timedatectl tells you whether it is synchronised and chronyc makestep fixes it; on Windows, w32tm /resync. When signing, always leave a margin backwards (-V -15m:+5m): an earlier start extends nothing dangerous, because the expiry stays the same.
The machine was already using TrustedUserCAKeys
This is the case on every Lightsail machine: its web console works with its own authority, and there can only be one TrustedUserCAKeys directive. Do not change it, or you will lose the Lightsail console. Instead, put the authority in the rescue user's own authorized_keys, which coexists with any other: cert-authority,principals="rescate",from="TU.IP.FIJA" ssh-ed25519 AAAA… (the contents of ca_rescate.pub after the options).
Someone deleted the rescue user
Whoever is an administrator can remove it, that is unavoidable. The defence is to check it every day (a test connection) and alert if it fails, so you can rebuild it while you still have access.
What if my CA is stolen?
Whoever has the CA can make certificates for all your machines. That is why it goes in a password manager and not in an email, and that is why the from= matters: without your IP, the CA alone is useless. If you suspect anything, generate another CA and repeat the preparation step on each machine.
In Simplio
In Simplio this is called the spare key: every machine card has that button. "Create spare key" leaves on the machine the simplio-rescate user with no usable password, which only accepts five-minute certificates signed by Simplio and only from Simplio's IP; on Linux Simplio does it over SSH, on Windows it gives you a script to paste once. If you lose access, "Rescue with the spare key": with your two-factor code (even if you are an administrator) you choose the user to recover (or ask to see which users there are) and what to do: a new password shown only once and not stored, adding your public key, unlocking it, or a console as simplio-rescate. Simplio checks every day that the key still opens the door and alerts you if someone has removed it. How the spare key works and how secure it is.
The same task, in AWS and in Simplio
| In the AWS console | In Simplio | |
|---|---|---|
| Preparing the door | Create the CA, the user, sudo, the two files and the SSH directives on each machine, by hand | A "Create spare key" button on the card: Simplio does it over SSH (Linux) or gives you the script to paste once (Windows) |
| Storing the CA | You, in a password manager | On the Simplio server, in the secrets file; the machines only accept the rescue user from Simplio's IP |
| Using it | Sign a certificate by hand and log in over SSH | Two-factor, automatic five-minute certificate, and guided actions: new password (shown once), public key, unlock |
| Knowing it still works | Remembering to test it | Daily check and alert if the spare key stops working |
| Not knowing which user to recover | Log in and look at /home or the Windows accounts | "See which users are on the machine": list with folder, administrator and last login |