Introduction

Today I implemented a automated backup system for my CentOS server using Duplicity and Amazon S3. After I recently launched my smartphone app Positive thinking, which uses MongoDB as the database/datastore I started to concern myself that I was responsible for my users data and it was no longer just my personal loss if something ever happened.

After researching a few options I went with the above solution, because S3 is highly scalable, reliable, secure, fast and relatively inexpensive (when pared with a solution like Duplicity, which only sends file deltas).

Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup.

So lets get started with a quick guide to getting daily backups of your CentOS server to Amazon S3. With the help of Duplicity and Duply.

Installation

First lets install the software on our server (all commands are run as the root user)

Set up

Next lets create a new backup config in Duply (Duply is a frontend wrapper for Duplicity, making setup easier and manages running the backup easier and better)

Now edit via vim/emacs/nano the config the above command generated

Comment out the GPG_KEY value and enter any password you like into the GPG_PW value

Go to S3 in your AWS console and generate a new S3 bucket,  called ‘our_backup’.  Replace the buckets endpoint domain with the appropriate domain (in my example below s3.amazonaws.com is the US standard region endpoint). Refer to here for the other region endpoints

Enter your AWS access and secret keys.

Set SOURCE to the root path /

uncomment MAX_AGE and set its value to 1 month

Next we want to configure which folders and files we want backed up

Heres a list of my folders I backup. Copy the syntax of + space <folder_path>. Dont forget the ** at the bottom, which tells Duply to only include the files in the list.

I am backing up all my nginx and php configs, my web sites, my golang code and my mongodb and mysql data files

Run the first initial backup

Now lets run the first backup which will backup everything in full (note the below is the log from a sequential backup, not the first backup but it gives you a idea of what to look for)

Check that its working

Now check S3 to see if the backups got uploaded correctly.

Check which files and folders have been backed up

Scheduling a daily backup and email notifications

Running our backup daily (2am) with Emails

Enter the following

Every day you will get emailed a log like the above when we ran the initial first backup.

Make a backup of our Duply config (at the very least you should securely store the backup password). Download duply_conf_back.tar.gz to a USB drive or something, if you need to completely restore your backups on a new server having the full config will be handy.

Restoring a backup

We can fetch a single file with the following

I will write more on restoring in the future, for now refer to

Should you get your self into a sticky situation