Jason Whatson

  • Git hub
  • Twitter
  • Skype
  • Linkedin
  • Instagram
  • STAY CONNECTED

Software engineer - Web & Mobile development, cloud computing : Sydney, AU;
  • About me
  • My resume
  • Contact
  • My Blog
  • Quotes
  • My portfolio’s
    • Software portfolio
    • Graphics portfolio
    • Music production & recording

Incremental backups to Amazon S3 on CentOS using duplicity

  • Feb 13, 2014

    Introduction

    Today I implemented a automated backup system for my CentOS server using Duplicity and Amazon S3. After I recently launched my smartphone app Positive thinking, which uses MongoDB as the database/datastore I started to concern myself that I was responsible for my users data and it was no longer just my personal loss if something ever happened.

    After researching a few options I went with the above solution, because S3 is highly scalable, reliable, secure, fast and relatively inexpensive (when pared with a solution like Duplicity, which only sends file deltas).

    Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup.

    So lets get started with a quick guide to getting daily backups of your CentOS server to Amazon S3. With the help of Duplicity and Duply.

    Installation

    First lets install the software on our server (all commands are run as the root user)

    1
    yum install duplicity duply python-boto

    Set up

    Next lets create a new backup config in Duply (Duply is a frontend wrapper for Duplicity, making setup easier and manages running the backup easier and better)

    1
    duply create our_backup

    Now edit via vim/emacs/nano the config the above command generated

    1
    vim /etc/duply/our_backup/conf

    Comment out the GPG_KEY value and enter any password you like into the GPG_PW value

    1
    2
    #GPG_KEY='_KEY_ID_'
    GPG_PW='YOUR_PASSWORD_HERE'

    Go to S3 in your AWS console and generate a new S3 bucket,  called ‘our_backup’.  Replace the buckets endpoint domain with the appropriate domain (in my example below s3.amazonaws.com is the US standard region endpoint). Refer to here for the other region endpoints

    Enter your AWS access and secret keys.

    Set SOURCE to the root path /

    1
    2
    3
    4
    5
    6
    7
    8
    TARGET='s3://s3.amazonaws.com/our_backup/main'
    # optionally the username/password can be defined as extra variables
    # setting them here _and_ in TARGET results in an error
    TARGET_USER='<your AWS access key>'
    TARGET_PASS='<your AWS secret key'
     
    # base directory to backup
    SOURCE='/'

    uncomment MAX_AGE and set its value to 1 month

    1
    MAX_AGE=1M

    Next we want to configure which folders and files we want backed up

    1
    vim /etc/duply/our_backup/exclude

    Heres a list of my folders I backup. Copy the syntax of + space <folder_path>. Dont forget the ** at the bottom, which tells Duply to only include the files in the list.

    1
    2
    3
    4
    5
    6
    7
    + /var/lib/mongo_data
    + /var/lib/mysql_data
    + /etc/nginx
    + /usr/share/web_sites
    + /etc/php*
    + /golang_code
    **

    I am backing up all my nginx and php configs, my web sites, my golang code and my mongodb and mysql data files

    Run the first initial backup

    Now lets run the first backup which will backup everything in full (note the below is the log from a sequential backup, not the first backup but it gives you a idea of what to look for)

    1
    duply our_backup backup

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    Start duply v1.5.11, time is 2014-02-13 02:00:01.
    Using profile '/etc/duply/full_backup'.
    Using installed duplicity version 0.6.22, python 2.6.6, gpg 2.0.14 (Home: ~/.gnupg), awk 'GNU Awk 3.1.7', bash '4.1.2(1)-release (i386-redhat-linux-gnu)'.
    Signing disabled. Not GPG_KEY entries in config.
    Test - Encryption with passphrase (OK)
    Test - Decryption with passphrase (OK)
    Test - Compare (OK)
    Cleanup - Delete '/tmp/duply.10415.1392256802_*'(OK)
     
    --- Start running command PRE at 02:00:02.867 ---
    Skipping n/a script '/etc/duply/full_backup/pre'.
    --- Finished state OK at 02:00:02.885 - Runtime 00:00:00.018 ---
     
    --- Start running command BKP at 02:00:02.895 ---
    Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
    Reading globbing filelist /etc/duply/full_backup/exclude
    Local and Remote metadata are synchronized, no sync needed.
    Last full backup date: Wed Feb 12 05:38:53 2014
    --------------[ Backup Statistics ]--------------
    StartTime 1392256804.12 (Thu Feb 13 02:00:04 2014)
    EndTime 1392256815.70 (Thu Feb 13 02:00:15 2014)
    ElapsedTime 11.58 (11.58 seconds)
    SourceFiles 14635
    SourceFileSize 440491579 (420 MB)
    NewFiles 72
    NewFileSize 3688548 (3.52 MB)
    DeletedFiles 26
    ChangedFiles 19
    ChangedFileSize 80165809 (76.5 MB)
    ChangedDeltaSize 0 (0 bytes)
    DeltaEntries 117
    RawDeltaSize 18935063 (18.1 MB)
    TotalDestinationSizeChange 5888919 (5.62 MB)
    Errors 0
    -------------------------------------------------
     
    --- Finished state OK at 02:00:18.777 - Runtime 00:00:15.881 ---
     
    --- Start running command POST at 02:00:18.791 ---
    Skipping n/a script '/etc/duply/full_backup/post'.
    --- Finished state OK at 02:00:18.809 - Runtime 00:00:00.018 ---

    Check that its working

    Now check S3 to see if the backups got uploaded correctly.

    Screen Shot 2014-02-13 at 2.15.30 PM

     

    Check which files and folders have been backed up

    1
    2
    3
    4
    5
    duply full_backup list
     
    #or
     
    duply full_backup list | grep quote.go

    Scheduling a daily backup and email notifications

    Running our backup daily (2am) with Emails

    1
    crontab -e

    Enter the following

    1
    2
    MAILTO=jason@mindfsck.net
    0 2 * * * duply our_backup backup

    Every day you will get emailed a log like the above when we ran the initial first backup.

    Make a backup of our Duply config (at the very least you should securely store the backup password). Download duply_conf_back.tar.gz to a USB drive or something, if you need to completely restore your backups on a new server having the full config will be handy.

    1
    tar czvf duply_conf_back.tar.gz /etc/duply

    Restoring a backup

    We can fetch a single file with the following

    1
    2
    cd /tmp
    duply full_backup fetch usr/share/web_sites/mindfsck.net/wp-content/plugins/wordpress-seo/readme.txt x.txt

    I will write more on restoring in the future, for now refer to

    1
    man duply

    Should you get your self into a sticky situation

    backup, Centos
Jason Whatson

Jason Whatson

Jason is a software developer from Sydney Australia who loves to write software to build great products and help businesses succeed with their goals. Jason is a aspiring entrepreneur who subscribes to the lean startup ideology.

23 Comments

  1. Arjan

    −Reply
    April 15th, 2014

    Thank you very much for this clear explanation. Works like a charm!!!

    Greetz from Holland

  2. vishnu

    −Reply
    April 20th, 2014

    Hello,
    I get this error,

    when i try take backup.

    sudenly after “duply uvlog backup” step

    Failed to create bucket (attempt #3) ‘uvlog’ failed (reason: error: [Errno 111] Connection refused)

    Please help me

    • Jason Whatson

      Jason Whatson

      −Reply
      April 21st, 2014

      Hi Vishnu, log into the aws console and make sure the bucket exists… Also make sure the aws credentials you are using have the correct read, write, create permissions for s3

      Hope that helps

  3. Thach Pham

    −Reply
    September 28th, 2014

    Oh well! this script has saved my life 🙂

  4. Spongedoc

    −Reply
    December 10th, 2014

    Hi!

    Great tutorial 🙂

    I found error in the profile create command:
    Not duply create our_backup
    Then: duply our_backup create

    THX

  5. Mike

    −Reply
    January 24th, 2015

    Cracking write-up mate, thanks.

  6. thomas

    −Reply
    February 28th, 2015

    i want to ask the centos version and YUM source configuration, thank you

  7. cheap kicks online

    −Reply
    July 12th, 2015

    Real air jordan retro for sale, real air jordan websites, cheap.Shop for free shipping and 100% authentic jordan shoes,authentic retro.welcome to our authentic jordan shoes page, We offer a variety of styles of authentic jordans 1.2.3 etc. 2014 jordan shoes sale collection now!

    • ảnh đẹp

      −Reply
      November 4th, 2015

      Written. tks you for sharing !!

  8. ảnh đẹp

    −Reply
    November 4th, 2015

    Bài viết hay. tks bạn đã chia sẽ!!

  9. athreyaKN

    −Reply
    December 11th, 2015

    I constantly get the “BackendException: No connection to backend
    03:01:57.801 Task ‘BKP’ failed with exit code ’23’.
    — Finished state FAILED ‘code 23’ at 03:01:57.801 – Runtime 00:00:00.373 —”
    error.

  10. peter

    −Reply
    March 28th, 2017

    really helpful

  11. hình nền đẹp

    −Reply
    June 8th, 2017

    Thank you very much for this clear explanation. this script has saved my life

  12. Sarah78

    −Reply
    July 5th, 2017

    This post is really giving me exciting information. There is something new that is related to my hobby. This post is considerable use for me. Thanks for sharing it.

  13. anh dep

    −Reply
    July 5th, 2017

    Bài viết rất hữu ích cảm ơn bạn!

  14. thuốc chữa mụn

    −Reply
    April 8th, 2019

    hồ hết cạc nốt mụt đền trường đoản cú biến khuất nhưng chẳng
    cần điều trị. tuy rằng nhiên, phải bạn bị rất giàu mụn, nhiều
    trạng thái vì chưng các tình trạng béng da
    hoi ra, cạc nốt mụt nè thường giò trường đoản cú biến tạ thế.
    Nếu bạn giò điều trị những kiểu mụn nào,
    ngơi sẽ đặng lại thẹo.
    Nếu điều trừng trị bằng thuốc trừng phạt mụn, bạn vì thế đến gặp thầy thuốc đa dương liễu đặng thắng hướng dẫn giống ngày tiết.
    các loại thuốc trừng trị mụt rất da trạng thái như thuốc hòn, thể kem, dung nhếch gel năng
    xà gian nhiều chứa chấp các vách phần trừng phạt mụn. Tùy vào
    tình yêu trạng da mực tàu bạn, bác sĩ sẽ chỉ định đơn loại thuốc hợp.

  15. niềng răng kim loại

    −Reply
    May 9th, 2019

    Niềng răng mắc cài kim loại là phương pháp chỉnh sử dụng mắc cài và dây cung
    đưa răng về đúng vị trí trên cung hàm, đảm bảo khớp cắn và vấn đề thẩm mỹ.

    Có 3 loại niềng răng mắc cài kim loại là Niềng răng mắc cài kim loại bình thường, Niềng răng mắc cài
    kim loại tự khóa và niềng răng mắc cài kim loại mặt trong.

    Niềng răng mắc cài kim loại dù là mắc cài thông thường
    hay mắc cài tự khóa đều có thể áp dụng cho tất cả các đối tượng khách
    hàng. Những ví như bị các khiếm khuyết về răng như
    hô, móm, thưa, lệch lạc dù là trẻ nhỏ hay người lớn đều
    có thể dùng niềng răng mắc cài kim loại.

  16. boc seo

    −Reply
    June 22nd, 2019

    sẹo rỗ là một loại sẹo phổ biến ở nhiều
    người, tuy nhiền chẳng hề ai cũng biết cách
    điều trị sẹo rỗ hiệu quả nhất,
    an toàn nhất.
    Ngày nay sẹo rỗ tuy không gây ra nguy
    hiểm hay ảnh hưởng đến tình trạng sức khỏe nhưng nó
    làm mất đi tính thẩm mỹ trên làn da, khiến nhiều người e sợ, không tự tin khi giao dịch.

  17. sẹo do mụn

    −Reply
    September 2nd, 2019

    sẹo rỗ là một loại sẹo phổ biến ở nhiều người, tuy nhiền chẳng hề ai cũng biết
    cách điều trị sẹo rỗ hiệu quả nhất, an toàn nhất.

    Ngày nay sẹo rỗ tuy không gây ra nguy hiểm hay
    ảnh hưởng đến tình trạng sức khỏe nhưng
    nó làm mất đi tính thẩm mỹ trên làn da, khiến nhiều người e ngại, không tự tin khi giao dịch.

  18. Minny Cyrillus Eurydice

    −Reply
    September 2nd, 2020

    Coding is a lifestyle for me!

  19. Dil okulu

    −Reply
    October 20th, 2020

    I am really delighted to glance at this website posts which consists of lots of valuable information, thanks for providing such
    statistics.

  20. discover more here

    −Reply
    December 2nd, 2020

    Wooden furniture has one thing very all natural regarding it.
    There is this feeling of coziness, of attributes as well as of luxury that may
    be be found in wood furniture. Wood is birthed from the planet.
    It feeds the fire, breaks down in to blows as well as ashes away.
    It is actually extremely near the human presence in the world.
    Might be actually that is why it reverberates a lot along with our
    team. When you touch a wealthy mahogany workdesk, might
    be actually that is actually why you still get that
    cozy feeling.

Leave A Reply

Leave a Reply Cancel reply

Recent Posts

  • Book Review: Unnatural Selection: Why the Geeks will Inherit the Earth – Mark Roeder
  • Incremental backups to Amazon S3 on CentOS using duplicity
  • An example of how Golang makes concurrent programming easy and awesome
  • Installing NGINX + PHP 5.5 with opcache + MySql on CentOS 6 (LEMP)
  • Basic software engineering principles – Don’t Repeat Yourself (DRY)

Recent Comments

  • Speakerzdn on Book Review: Unnatural Selection: Why the Geeks will Inherit the Earth – Mark Roeder
  • http://daftarnova88.info on Basic software engineering principles – Don’t Repeat Yourself (DRY)
  • Donald Webb on Golang vs JAVA vs PHP: Comparing productivity
  • Donald Webb on Golang vs JAVA vs PHP: Comparing productivity
  • Lensamovie on Basic software engineering principles – Don’t Repeat Yourself (DRY)

Archives

  • March 2014
  • February 2014
  • January 2014
  • November 2013
  • June 2013
  • April 2013
  • May 2010

Categories

  • Book Reviews
  • Programming
  • Software engineering
  • Web hosting

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

About me

I am a software developer from Sydney Australia who loves to write software to build great products and help businesses succeed with their goals.

I am an aspiring entrepreneur who subscribes to the lean startup ideology and creator of “positive affirmations”, a app for Android and iPhone.

Contact

Phone: 0439372293
Email: jason@mindfsck.net

Random Quote

If I remain focus I will succeed at life and the poker table — Joe hatchem

© Copyright 2021 Jason Whatson. All Rights Reserved.