How to copy a folder from your server to S3 every night with a single command (aws s3 sync)

Machine snapshots are the "big" backup: the whole disk, for when something breaks. But sometimes you want something finer and cheaper: a daily copy of one folder (the website uploads, the documents, a database dump) somewhere off the machine, from which you can recover a single file without restoring anything. That is S3, and the AWS command-line tool has a command, aws s3 sync, that copies only what has changed. Scheduled every night, it is the cheapest and most reliable file backup you can have.

Step by step

1. Create the bucket

   S3 console, Create bucket, unique name (copias-miempresa-web), same region as the machine (so transfer between the two does not use up your allowance or get charged), Block all public access on, versioning on and default encryption. Create.

2. Create a key that can only write to that bucket

   In IAM, create a policy with this content (change the bucket name) and a user copias-web with that policy and an access key, as explained in the guide on an IAM user with minimal permissions:

   {
     "Version": "2012-10-17",
     "Statement": [
       { "Effect": "Allow",
         "Action": ["s3:ListBucket"],
         "Resource": "arn:aws:s3:::copias-miempresa-web" },
       { "Effect": "Allow",
         "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
         "Resource": "arn:aws:s3:::copias-miempresa-web/*" }
     ]
   }

   If that key ever leaked, the only thing anyone could do is touch that bucket. Nothing else.

3. Install the AWS tool on the machine

   Over SSH, on Ubuntu or Debian:

   sudo apt update && sudo apt install -y awscli

   (On Amazon Linux it comes preinstalled.) Configure it with the key from the previous step:

   aws configure

   It asks for the key ID, the secret, the region (eu-west-1, for example) and the output format (leave json). They are saved in the user's folder, readable only by that user.

4. Test the command by hand

   aws s3 sync /var/www/html/uploads s3://copias-miempresa-web/uploads --delete

   The first time takes a while, because it uploads everything; you will see each file go by. The following runs only upload what has changed. The --delete option deletes from the bucket what is no longer in the folder, so the copy is a mirror; with versioning on, what was deleted is still recoverable. Check in the S3 console that the files are there.

5. Add the database if you have one

   So the backup also includes the database, dump it first to a file in the same folder that gets synchronised, for example with a small script /usr/local/bin/copia.sh:

   #!/bin/bash
   mysqldump -u usuario -pCONTRASEÑA nombre_bd | gzip > /var/copias/bd-$(date +%F).sql.gz
   find /var/copias -name "bd-*.sql.gz" -mtime +7 -delete
   aws s3 sync /var/copias s3://copias-miempresa-web/bd
   aws s3 sync /var/www/html/uploads s3://copias-miempresa-web/uploads --delete

   Give it execute permission (chmod 700) and test it with sudo /usr/local/bin/copia.sh. The find line deletes dumps older than a week from the machine; in S3 they stay.

6. Schedule it every night

   sudo crontab -e

   Add a line at the end for 3:30 in the morning (machine time, which is usually UTC):

   30 3 * * * /usr/local/bin/copia.sh >> /var/log/copia.log 2>&1

   Save. The next day check /var/log/copia.log and the date of the files in S3. Also set a lifecycle rule on the bucket that deletes old versions after 30 or 90 days, so it does not grow forever (guide in this section).

What can go wrong