Home » Code, Headline

Amazon s3 backup for web server files and SQL using bash

6 July 2008 10 Comments

I needed a solid backup method for my clients websites, where I wanted to improve from the previous method of backing up to a separate host in the US Hostmonster. Hostmonster expires in 80 days, so why renew for $100+/year just to house a few GB of client backups.

I wanted to use the Amazon S3 service for my backups with an amazing reliability and a low monthly cost, so I  went ahead crawling the web for bash scripts that would suit my situation. The one I liked the most was Here, but I have adapted it to suit what I want.

Project is now sponsored at sourceforge at http://sourceforge.net/projects/des3bashwebsite/

My Requirements:

Environment: Redhat Cpanel X | WHM Server (VPS and shared - both)
Language: BASH
Automation: Cron Script
Files: public_html directory and linked mysql database
Desired Output:  A single blob housing the www files and the sql database on my s3
Other: No archives left on server
Frequency: Weekly or Monthly

I wanted to improve on the above script by being compatible with CPanel, include the SQL data for the website files, tar’ing both the www files and the sql files into one blob, and making the final filename more descriptive about the site. I want to use this script for a large number of websites that will all backup into a single bucket, so the file name was to be of the type: url-YYYYMMDD.tar.gz as to easily distinguish between backups.

This was to be a single dump of the whole website, and with filesize not an issue this is ideal towards the websites I create for clients.

1. Setup the File Structure

All my sites reside on separate cpanel accounts where the common file structure is:

/home/$USER/ (home)
/home/$USER/backup/    (website backup folder)
/home/$USER/public_html/  (website files here)

Setup the new base structure as

/home/$USER/
/home/$USER/s3/
/home/$USER/backup/
/home/$USER/backup/sql/
/home/$USER/public_html/

2. Add Files

copy the de-s3-web-backup.sh file into /home/$user/backup/ and set permissions as 760
copy the s3-bash files into /home/$USER/s3 and set permissions 760 for all

3. Create Key file

Paste your amazon aws secret key into a text file and name it s3.key, and save in /home/$USER/s3/s3.key with permissions 760

4. Create a new s3 Bucket

Create a new amazon s3 for all your new downloads. S3 Firefox Organizer is great for this.

5. Get the Script

Download the script (from below) and upload it to your webserver in the /backup/ folder created above. Make sure the file privileges are 760. If you have a different file scheme, don’t put this in your public_html file

6. Customise Script

You need to input your server parameters into the script in order for it to run.

AMAZONID is your amazon identification key
BUCKET is your amazon aws s3 bucket
SITE is a quick description of your website: example duivesteyn.net
USER is the name of the user account. In CPanel this is your username
mysqlusername is the MYSQL username for the website you are backing up
mysqlpassword is your MYSQL password for the above username
databases is the list of databases you are backing up example: “main” or “main1 main2″ for 2 databases

7. Add to cron

Schedule the task to run automatically through cron. This is also available in your websites CPanel.
Full Backup recommended weekly, and SQL backup recommended nightly.

0 0 * * * sh /home/USER/de-website-backup.sh

And thats it. be sure to test it by running it in a terminal first, and remember to checkup on your bucket from time to time.

Download Here (GPL)

Version 1.01

https://sourceforge.net/projects/des3bashwebsite/
References: Bnee.com s3-bash

10 Comments »

  • tman said:

    Hi there. Great script. I can get the script to run, it works except if I have more than one DB in databases=”dbname”. If I only have one everything is good, full data and everything. If I add a second by putting a space between them, I get the files created but no data in them. Any idea’s?

    # 26 July 2008 at 3:58 am
  • themandril said:

    I’m confused - where do we get the de-s3-web-backup.sh file? Also, how would I customize it to only do a particular directory under html_public and a specific sql database that goes with it?

    # 1 September 2008 at 7:55 pm
  • ebrodeur said:

    Thanks, the script worked perfectly!

    @themandril: look under “Download Here” which has a non-hyperlinked URL to the download; it begins with https://.

    You’ll need the s3-bash scripts which can be found by clicking the “s3-bash” link next to “References” at the bottom of the post.

    Lastly, the s3.key file cannot contain any trailing characters such as newline. If so, the s3-bash scripts will die (with an appropriate error displayed at the console).

    As suggested by the author test via SSH to ensure the script is working properly.

    # 8 September 2008 at 5:57 am
  • ebrodeur said:

    Be sure to check where your TMP files are going. If your environment uses /tmp the last line of the de-website-backup-public.sh file must be changed to:

    rm -rf /tmp/s3-*

    If your environment uses a TMP folder located in your user home folder (ie: ~/tmp) the .sh file does not need to modified.

    If you don’t confirm this you will leave sensitive files in your TMP folder which may be compromised.

    Another side effect is the script will not run successfully on the same host because the folders already exist.

    Lastly, for multiple sites on a single host you should cron them sufficiently far apart to avoid concurrency problems with the TMP files.

    # 8 September 2008 at 6:46 am
  • Amazon S3 - The Beginner’s Guide | Web Tricks said:

    [...] Amazon s3 backup for web server files and SQL using bash [...]

    # 22 September 2008 at 12:05 pm
  • Admin (author) said:

    Cheers for all the comments so far guys, love the feeback.

    # 26 September 2008 at 11:09 am
  • Amazon S3 - The Beginner’s Guide | Web Burning Blog said:

    [...] Amazon s3 backup for web server files and SQL using bash [...]

    # 19 October 2008 at 11:49 am
  • Week in Review (October 20 through October 26) | GrandmasterB dot com said:

    [...] I should be able to grow the GMB Network on Media Temple, and if I get too big for the Grid Service I can move up to a Dedicated Virtual plan. One other note on moving my hosting, I also setup an account on the Amazon AWS to remotely backup my sites. After reading a tutorial by Christina Warren, I setup the necessary software and a shell script to do automatic backups of my site daily. The script took about 10 minutes to setup and another five minutes to setup my S3 account…after that it was all butter. Now every night at 1am my script runs and backs up the three sites I have on it, and uploads the zipped files to the Amazon S3 hosting. I’ve been running the S3 account for about two weeks now, and my bill is only 33 cents. If you want to backup your site automatically and have CPanel hosting checkout this tutorial. [...]

    # 27 October 2008 at 8:45 pm
  • Rob Anderson said:

    Hi there, this is a wonderful script and I have it working … almost :)

    It works fine from the command line via SSH running as root, but not via cron when it is presumably running under my cpanel account (iotaco in this case). This is the error I get emailed to me when the cron job runs:

    …
    /home/iotaco/backup/sql/20081107-iotaco_wceet.sql
    which: no curl in (/usr/bin:/bin)
    s3-put: Environment Error: curl not found on the path or not executable
    …

    So it works normally until it has to run curl in s3-put. I can confirm that curl is in the /usr/bin so not sure what is going wrong here.

    I am a complete Linux noob so hoping somebody can give me some ideas to get this working!!

    Here is my cron command:

    0 0 * * * sh /home/iotaco/backup/de-website-backup-public.sh

    BTW, I found I had to tweak the backup script to get this far, i.e. remove blank lines and modify the code to get multiple databases working, i.e.

    $mysqldumpcmd $userpassword $dumpoptions ${database} > ${backupdir}/sql/${DOW}-${database}.sql

    Before I changed the above line it wouldn’t work for multiple databases, now it does :)

    Please help!

    # 8 November 2008 at 6:53 am
  • Rob Anderson said:

    @Rob Anderson (me :)

    If you run the cron job under the root account it works fine. I guess when running it under the cPanel account, it doesn’t have enough access to run curl (hey - I told you I was a noob).

    This solution rocks!

    # 13 November 2008 at 8:31 am

Leave your response!

You must be logged in to post a comment.