Remote backup tool READY!

Remote Backup 1.0
Features (12-17-2005)
:
This util use native interworx backup
Transfer files using secure ssh to remote server
Little util for key server included
Report by mail when backup start, and report when finsih, with time date, and backup size included in the report

Auto rotate old backups:
Save copy last backup or save only current backup.
Optional safe mode, if backup fail, old backup not rotated or deleted.

Remote Backup 1.1
New features (05-01-2006):
Now you can configure folder where backup its saved.
Skiplist available, now you can make your own list of site or sites to skip in the backup. (selective backup)

Next version features ?

Just place your request and i try add feature requested.

About Rsync feature
Available in the future? NO
… sorry! Why?
Simple: rsync its a little complex to backup and restore, and not really usefull when have 100% automatized hosting company with multiple domains per server, and system for enable, disable and delete accounts in automatic way.
With rsync, the backup continue saving domains already deleted of the servers, and need manual deletion of backups every few days for have correct and real copy of sites hosted on the servers. (this can be fixed but need extra code for check list domain hosted per servers, for compare with the domains on the backup etc etc)

Everybody say rsync its light backup system, yes !! that its real, specially for sueper big sites, if you have super big site, the best its use rsync, but you can continue using my backup tool with that domain skiped form the backup, and just configure rsync for that super big site.

The native backup Iworx used by my frontend its really light and not overload the server, in all my servers, the usual load its 1 to 1.2 runing the iworx backup the load just go to 1.86 to maximum 2 for about 1:30hour, and about rsync save band width, yes that its real, but at this moment server providers around the world offer great BW quota transfer to cheap price, so i really no think BW its a real trouble.
BW its your trouble? well i think the solution its second HARD DISK for daily backups, and one weekly or montly remotebackup.
For this reason i decide use Iworx native backup and create this frontend for improve the IWORX native backup usage. : )


The idea of this backup is make this task automatic, programed in cron for save backup to remote server using ssh transfer.
You can configure daily in crontab or weekly, or montly.

Backup 100 normal sites, take about 1 hour (this time can change based on network speed, size of sites, number of files, cpu etc etc)

Install:

You need key your server using ssh, in this way backup program make automatic conection using ssh without need you enter root password or save roor password on any file.
This how to install, supose you have 2 servers and need backup server 1 to server 2 and server 2 to server 1.
Or backup server to any other server with ssh acces and root access.
Or backup to self server.

Step1
Install script on server you want perform backup:

mkdir /root/.bk
cd /root/.bk
wget http://ns3.zero-spam-tolerance.com/remotebackup.tar
tar xvf remotebackup.tar
nano -w config

IPREMOTE=
enter IP of remote server where backup be saved or self server IP.

EMAIL=
Enter email where you want recive reports of backup.

LASTBK=
With this option you can save one copy of last backup, or can disable and just only save fresh and current backup.
If you have free space for save copy, i recomend not disable and use default value, in this way you have oldbackup saved and current backup saved.

SAFEOLDBK=
This option save your old backup in folder named SAFEoldBACKUP before backup start, and delete when backup end.
If server crash in under the backup process, you have a copy of your old backup in SAFEoldBACKUP folder.
I add this option because rotation scheme i use is very simple and rotation fail if server crash.
If you not recive mail reporting backup finished, i recomend enter to folder where files backup are saved and chek if folder SAFEoldBACKUP exist.
MV folder to any other name, and rerun backup, if backup its completed then delete that folder.

PATHBK=
Just configure the pat you want use for save the backup on remote server, you no need cerate folder, the backup create the folder if no exist.

Step2
This util make by default complete server backup (all domains) if you need skip/exclude domain or multile domains from the backup, just edit file skiplist, and add the doamins you want skip, one per line, example:

domain1.com
domain2.com

Aditional to domains list, you can filter the backup, based on the TLD, if you no want backup some specific TLD domain, you can add to skiplist that TLD, example, just add to skiplist:

.tv
.biz

This, make backup skip/exlude all domains .tv and .biz

Step 3
Now is time key your servers:

Run this command on server want backup and server where you the backup its saved.

ssh-keygen -t rsa

Press enter 3 times, use empty frace for key.
(If you already have key created just overwrite)

Now o run the util for key server:

./keyserver

This util just copy they key to remote server, the util ask for root password of remote server where you save backup, enter password and press enter.
For check if the server is correct keyed, just run:

ssh root@IPofremoteserverhere

Run hostname for see if you stay loged on remeote server correctly, if all is ok, just run exit for back.

(you can key self server if no plain make remotebackup)

Step4:
Add to cron the backup, daily or weekly:

nano /etc/crontab

For daily at 1 AM add:

02 1 * * * root /root/.bk/remotebackup > /dev/null 2>&1

For weekly at 1 AM add:

22 1 * * 0 /root/.bk/rootremotebackup > /dev/null 2>&1

I sugest not run at 4am because server perform other task, and you can probably overload your server.
Dont forgot change path /root/.bk/ to path where you decide locate the backup util.

If want run your fist backup manually just type:

./remotebackup

By default, backup files are saved on /root/.bk folder of remote server, or self server if you decide make backup to self server, remember you can change that path or can use the default.

Well i hoppe this be util for somebody. I not are programer, so this is very simple but work ok.
This backup make complete backup of all sites, and no use rsync, take care with BW used if run daily.
My native language not is english, sorry if my spell no is correct !

[B]Paypal donations: groovedj@hotmail.com : )

[/B] Best regards! and enjoy !

If anyone download and use script prior day 23 december, please download script again, because i find error in rotation logic.
Now its fixed.
Extract and overwrite only file remotebackup

Best regards.

Bravo Dj-Grobe

I did explain how to do this manualy
http://www.interworx.com/forums/showthread.php?t=493

But it’s very cool to have this done automaticly and to have the auto rotate ability.

I’ll test it soon…

Did you thing about the few boxes backup to ONE remote place ? (I think about the key merge on the remote boxe )

Pascal

You cant backup multiple boxes to only one remote box.
The trouble its the backup system i create, save backup on /root/.bk / folder of remote box, if you plain multiple boxes to only one, that can be a trouble.

Gimme few days an i add option for choice in configuration, the name of folder and complete path where the backup be saved on remote server.
I know that limitation when i start, but no have a lot of time, for that reason publish the backup on this stage.

The first week of 2006 i add that features, if have some free time i make this week.

About the key, i add the merge option soon for case you coment.

Best regards.

Well for me everything already work (less the auto rotate, I first delete everything on the remote server, then backup) but for the community and a general use, it could be great to have this, yep.

Thanks Dj-Grobe

Pascal

This is very interesting to me - this is exactly what I was looking for.

I just got backup storage space through RSYNC or SCP and I was looking for a script that allowed for a full backup of only relevant info from my IWorx box.

Is the method outlined above a quick, safe, thorough way of backing up my box? Or is using the .pex backup method that I saw on the forums a better way (http://interworx.com/forums/showthread.php?t=1047&highlight=backup)

Any advice would be appreciated!

Thanks.

Int,

I think if you do Rsync its better to just do it by file and not use the InterWorx backup.pex. This way you dont…

  1. Waste CPU time TGZ’n each siteworx account
  2. Waste BW to transfer unchanged files

My ideal situation will be to do weekly (or even daily since it should be rather quick) SiteWorx “structure” backups. This way I could restore SiteWorx accounts if I had a complete hard drive failure. I would then use my daily Rsync backup to copy back over the data files (ie. /html, etc.) and the /var/lib/mysql files. Now this would work restoring to the same system (Same version of software like mysql, iworx, etc.). If you need a backup to move to a new system it would be better to just do a full Iworx backup.

So hopefully in the next 2.2 release the structure only backup will be incorporated. Also, if you need anyone to test this structure only backup I wouldn’t mind getting a preview of the new backup.pex :slight_smile:

New version soon.

New version soon, but not rsync, i continue using native interworx backup.

Soon i work on rsync version…but i need time !!..

The new version its a little modification for enable selective backup, not just all backup.
With selective backup you can skip domains you add to skiplist.

Aditional to that better rotation scheme and some changes for improve the backup.

I try back soon with new version, but have lot of work here, so please have patience.

I can’t download http://www.interworxutils.cjb.net/remotebackup.tar :frowning:

Ups!!!, the link of download its down, after this weekend i have the new version working and make public new link for download.
Sorry for delay to repply WebXtrA

Okay, no problem. I got a work arround thanks to Socheat.

Thanks

Sorry for this big delay, very bussy here.
Well, the new backup version its ready, original post updated please read !.
Download link updated and working.

Enjoy!

Hello,

I was wondering if it’s possible to backup 1/3, 1/4 etc. of all the accounts.
This looks like a wierd question, but I’ll explain:

We are certified by “Hostmerk”, it’s a Dutch entity that ensures that a HSP/ISP has been checked that it has certain hardware, procedures etc. so that prospects/customers know that they are dealing with a company that has a certain quality level.

One of the things is that every single account has to backed up to an external server every day.

At this moment our cluster is backing up sites every day from 0:00h untill +/- 12:00h, so that takes about 12 hours to backup all the accounts.

I would like to have all the nodes do a portion of the backup, so that the backup will be done much quicker and because it doesn’t put a heavy load on our Cluster Manager for almost 12 hours.

By the way, this is what I use now to backup:

[SIZE=1]#!/bin/sh
/home/interworx/bin/listaccounts.pex | awk ‘{print “/home/interworx/bin/backup.pex --ftp --transfer-ini=/root/transfer.ini --domains=” $2}’ >> /root/scripts/backuptoday.sh
sh /root/scripts/backuptoday.sh
rm -f /root/scripts/backuptoday.sh

backuptoday.sh then has lines like these, to backup all the accounts:
/home/interworx/bin/backup.pex --ftp --transfer-ini=/root/transfer.ini --domains=domain1.com
/home/interworx/bin/backup.pex --ftp --transfer-ini=/root/transfer.ini --domains=domain2.com
/home/interworx/bin/backup.pex --ftp --transfer-ini=/root/transfer.ini --domains=domain3.com

etc.
[/SIZE]
Greets,

R?mon

In the next release of Iworx they will offer a structure only backup which will just backup the SiteWorx account settings. This then leaves you to backup the data itself.

Intead of using the Iworx full backup to do the nightly backup I suggest you use Rsync. Doing this will save time b/c only files that are changed (and I think just the bits that are changed) are sent to the remote backup server. Plus there is no compression (tar.gz) so it will have a lighter CPU load while running. This could dramatically cut down on the backup time. But you may have to wait until the structure backup is out so you can also get the siteworx settings like the email boxes etc.

Thanks for your reply.
The only problem with rsync is that it doesn’t delete files which were deleted in the source folders. Meaning that the backup will contain every single file that once was there.

I don’t know if the structure backup will also backup the databases of a siteworx account? Because these change every day and sometimes it is nessicary to have a backup of 2 days ago because a customer did something wrong the day before yesterday and finds it out today.

Also the ease of restoring a site is much better with a full backup.

I’m looking into if I can just count the lines in backuptoday.sh devide that trough X and then make multiple backuptoday.sh (backuptoday1.sh, backuptoday2.sh etc.) scripts containing only the first lets say 200 accounts, second 200 accounts etc.

Node 1 will excecute backuptoday1.sh; backing up the first 200 accounts, Node 2 will excecute backuptoday2.sh backing up the second 200 accounts etc.

It should be possible, but don’t know how yet.

–delete: delete files that don’t exist on the sending side

I use “rsync -a --delete source dest” or “rsync --rLptgoD --delete source dest”. The difference between -a and -rLptgoD is that -a is equal to -rlptgoD (little L) and the big L will copy symlinks as files and where the little l will copy just the symlinks.

I believe this is what a structure backup does, it backs up all the settings (ie database) of a particular siteworx account. So first you would restore the structure backup and then restore data from the rsync backup.

Iworx Question
If you restore a siteworx structure backup on top of an existing siteworx account will it only update the database and leave all current files in place? Like lets say they deleted an email account and then you restored the struct backup, this wouldn’t effect the other mailbox data thats there or HTML stuff correct? Just after restoring the struct backup you would have to copy over the /home/site/var/site/mail/mailboxname folder correct?

I couldn’t agree more. I keep 5 days of backups with rsync. Here’s a sample of what I do:

echo "1 -> 2"
rsync -a --delete /backup/1/* /backup/2

echo "0 -> 1"
rsync -a --delete /backup/0/* /backup/1

rsync --rLptgoD --delete source /backup/0

The way I look at it its not really that much more difficult or time consuming using rsync for data, but this is something that is more of personal preference rather than fact. As it stands right now there is no doubt that its easier with Iworx backup vs rsync, but with the introduction of a structure only backup then all you will have to do is restore the data which can easily be done with cp or rsync. You could even right a restore script to make it easier.

Also, what I really like about rsync data backup is something you pointed out earlier. If a client messes something up on lets say a HTML file they have. You can go to the rsync backup for whichever of the “5” days you have and restore that single file. You would be able to do this with Iworx full backup, but you’d have to do it manually by unpacking the tgz file.

As to your idea of splitting it up it sounds like it should work, but I dont know so I will let someone else address that.

OK, this is what I did, works for me (maybe DJ-Grobe can implement something similat in his script :wink:

I changed the script a bit and now its this:
############################
[SIZE=1]
#!/bin/sh
mkdir /tmp/backuptoday
/home/interworx/bin/listaccounts.pex | awk ‘{print “/home/interworx/bin/backup.pex --ftp --transfer-ini=/root/transfer.ini --domains=” $2}’ >> /tmp/backuptoday/backuptoday.sh
split -l 200 /tmp/backuptoday/backuptoday.sh /tmp/backuptoday/
sh /tmp/backuptoday/aa
rm -rf /tmp/backuptoday

############################

the only thing I have to change on the nodes is the “sh /tmp/backuptoday/aa” line, since split, splits the files and writes them away as aa, ab, ac, ad etc.

It’s not really automatic and I have to keep track of the amount of accounts etc. but it works for me :wink:
[/SIZE]

I would be interested to know how this helps out with your overall time to backup.

Well since it takes 2 - 3 hours to backup 200 sites (approx.) every server is busy for 2 - 3 hours in stead of one for 12 hours, it will now take 2 -3 hours.

But it doesn’t work… Since the databases are on the cluster manager and the nodes try to connect to the local mysql database when it is making a backup.

So I guess I’m going to try your suggestion.

What is a working download link for this backup script?
-a