Interworx 3 and backup, documentation?

Something its wrong with backups?
I testing transfer one simple site with just 11.31 MB storage !! and take more of 20 minutes !!!
Is use the panel… after that 20 minutes ahve error failiure, but o clear on any place what is the error !!

I test with other sites and find the same!!

Now go to shell, create backup using:

/home/interworx/bin/backup.pex --domains=domanname.com --file-path=/root/

This simple task before upgarde to 3.0 take just 2 minutes for little sites…now take long time.

After backup its generated i move the file to new server where i wnat restore, i move manually…
I try restore and have failiure again !!

I stay desperate have to move really high number of sites this weekend !!

Something its wrong?
Why backup take absolutly incredible long time !!! with sites with just 20mb 15 mb and no high number of files …before version 3 this take just 1 or 2 minutes per site !!

On this way make backup of server with 200 sites or more its imposible !!

If sombody from iworx, read, please tellme !! i think have to take a look on this toruble, please !

I try with more sites… all sites cant be imported return failure error !! when try import in other server, with same iworx version !

I asume now this its a bug ; (

I opened emergency ticket for this !!

EDIT:

I CANT RESTORE SITES due aprently bug, i have ticket for tat opened right now.
Anyway i found aparently why the backup runing slow, i detect some sites are very little, but are very very very old…and have 100000 of STATS FILES using 300 400 500mb !!
The backup should add option for skip, any log or stat data…this in high percent of sites its not critical information, and create big torubels delaying amazing the backup !!!

Please add that feature asap !! its needed…on server with old sites, the backup can be a nightmare !!

Imagine right now i testing with site with just 13mb, but the stats are 545 mb !!! ad thousand of files !!

Anyway not understand why have error when try restore sites on my new servers !
Please iworx guy check te ticket !

[QUOTE=IWorx-Socheat;12712]I personally like structure-only backups for really large accounts.

Also, it’ll be far quicker, and less resource intensive, to backup all accounts and grab the data I can’t easily backup (i.e., internal iworx data). Imagine that you have 300 accounts on your box, each between few hundred MB and few GB in size. Doing a full backup on all accounts could take a whole day to finish.

If instead you used rsync or rsnapshot to backup the data (which would be very quick after the first time it runs, since only changed files are copied), and you then did structure-only backups of all the accounts, this would finish much quicker.

But yes, you are right. There is more work to restore a structure-only backup, because after restoring the structure-only backup, you then have to copy the data over. But, at least you have the choice. :)[/QUOTE]

You absolutly right about the time and resource usage the full backup take. We have 300 accounts and it takes 6 hours (100GB of data)

As soon V3 will be out, we’ll try the structure only backups + Rsnapshot + Mysql backup script, if it is fine we’ll give here the config of rsnapshot and why not a SRPM of a script that do this job

Just a last question about structure only backups. Tell we want to do a full backup, so we do :
1- A rsync or rsnapshot of /home (in /backups for example)
2- A backup of the Mysql database with a mysqldump (with rsnapshot and in /backups)
3- Perform a backup.pex with --structure-only in /backups (like /home/interworx/bin/backup.pex --domains=all --structure-only --output-dir=/backups )

Is that correct ?
Do you suggest to perform first the backups.pex or the rsync ?

Now we’d like to import all the accounts (tell we have the same Ip shem)

We just have to do something like

for backup_file in /backups/*.tgz
do
~iworx/bin/import.pex --create-reseller --control-panel=siteworx --ip-address=inifile --archive=$backup_file
done

This way, even if it was a structure-only backup ALL the reseler account will be created and all reseler’s siteworx accounts will be created and attach to the good reseler ? Is it correct ?

I think about a new feature that could be very usefull.

The import command with --ip-address=inifile allows to restore accounts in the same IP defined in the backup file which is great, good !

You can also define your own ip adress with --ip-address=192.1.1.1, fine !

Indeed, if you are restoring all accounts on the same server the IP are the same, so the inifile is great, but if you are restoring/migratiing all accounts on a new server with different IP then you have to define which accounts has to be set on an IP.

It could be great to have the ability to backup only the accounts that are set on one IP. Like this it could be possible to backup these accounts in a special directory and so more easy to restore these accounts on an other IP, I mean :

Tell we have :
100 accounts on IP 111.168.1.1
100 accounts on IP 111.168.1.2
50 accounts on IP 111.168.1.3

It could be great to be able to backups these accounts like this :

/home/interworx/bin/backup.pex --domains=111.168.1.1 --structure-only --output-dir=/backups/ip-111-168-1-1

or

/home/interworx/bin/backup.pex --ip=111.168.1.1 --structure-only --output-dir=/backups/ip-111-168-1-1

or

/home/interworx/bin/backup.pex --ip=111.168.1.1 --structure-only --output-dir=/backups --filename-format “111-168-1-1-%D”

This way it would be very easy to know the IP of every accounts backuped and so more easy to restore/migrate these accounts on a new IP by doing something like this

for backup_file in /backups/ip_group1/*.tgz
do
~iworx/bin/import.pex --create-reseller --control-panel=siteworx --ip-address=1.2.3.4 --archive=$backup_file
done

or

for backup_file in /backups/111-168-1-1*.tgz
do
~iworx/bin/import.pex --create-reseller --control-panel=siteworx --ip-address=1.2.3.4 --archive=$backup_file
done

In fact, you give a good way to restore an account on a defined IP, so I think it could be a good idea to give the the ability to backup ALL accounts set on one IP, what do you think about this ?

Maybe it is already the case and I have forgotten something ?

Pascal

Imposible import sites form any of my old server, udpated to rc3 on my new servers with iworx rc3 too

Failure
Archive file : /tmp/cpbfn7Fjg…tgz
Working directory : /tmp/siteworximportx6mOEA
Archive extracted successfully
Setting IP address : 64.******
Setting old base directory : /home/*****
Loaded Domain : ******
Setting backup options : ALL
Loaded package option : E-mail Boxes => 50
Loaded package option : MySQL Databases => 10
Loaded package option : Subdomains => 0
Loaded package option : FTP Accounts => 50
Loaded package option : CGI Access => 1
Loaded package option : E-mail Aliases => 999999999
Loaded package option : E-mail Groups => 0
Loaded package option : Bandwidth => 4.93
Loaded package option : Storage Space => 222.6
Loaded package option : Save Web Transfer Logs => 0
Loaded package option : Burstable => 0
Loaded package option : MySQL Users => 1
Loaded package option : Resolve DNS Names => 0
Loaded package option : E-mail Autoresponders => 999999999
Loaded package option : Pointer Domains => 0
Loaded package option : Crontab Access => 0
Loaded package option : SSL Access => 0
Loaded package option : ScriptWorx => 0
Loaded package option : Account Backup => 1
Loaded package option : Secondary Domains => 0
Loaded master user *************
Master language loaded : en-us
Loaded unix user : onetwoth
Shell is invalid
Loaded e-mail account : ***********
Loaded email account preferences : *************
Loaded spam status : Disabled
Loaded MySQL user : ************
Skipping default FTP user
Quota Sanity Checking
Setting Reseller ID : 1

Some idea?
What cause the error?

[QUOTE=Dj-Grobe;12716]The backup should add option for skip, any log or stat data…this in high percent of sites its not critical information, and create big torubels delaying amazing the backup !!!

Please add that feature asap !! its needed…on server with old sites, the backup can be a nightmare !![/QUOTE]

Great idea! That’s why we already did it :wink:

 --backup-options all no-logs no-stats 

Will tell it to do a full backup, but skip the logs and stats data. We’re looking into your import issues right now from your support ticket.

[QUOTE=pascal;12717]In fact, you give a good way to restore an account on a defined IP, so I think it could be a good idea to give the the ability to backup ALL accounts set on one IP, what do you think about this ?

Maybe it is already the case and I have forgotten something ?

Pascal[/QUOTE]

This is a good idea, and I’ve come up with a solution. ~iworx/bin/listaccounts.pex lists all the accounts on the box, and was improved in 3.0 to add some nice features for 3.0 -> 3.0 mass transfers. For example, right now in the v.3.0 RC, you can run:

 ~iworx/bin/listaccounts.pex --domain --username --email --storage

And this will list all the accounts on the box, 1 per line, displaying the domain name, username, email address, and the storage used of the account, separated by tabs:

$ ~iworx/bin/listaccounts.pex --domain --username --email --storage
domain1.com	domainon	domain1@hotmail.com	0.7890625
aaa.domain.com	aaadomai	domain@domain.com	0.15625
aab.domain.com	aabdomai	domain@domain.com	0.15625
...

It will print the columns in the order you specify. For example, in the above case, it will print the domain name first, TAB, username, TAB, email, TAB, storage.

I’ve just added the --ip-address paramater to the code, which will allow you to list the IP address of the account:

$ ~iworx/bin/listaccounts.pex --domain --ip-address
domain1.com	192.168.1.166
aaa.domain.com	192.168.1.155
aab.domain.com	192.168.1.155

You could then incorporate this into a bash script using grep and awk to do what you need:

for domain in `~iworx/bin/listaccounts.pex --domain --ip-address | grep 192.168.1.155 | awk '{print $1}'`
do 
  ~iworx/bin/backup.pex --domains=$domain --output-dir=/backups/ip-192.168.1.155
done

Sorry, I should clarify: the --ip-address parameter has been added to the devel code and will be in the next RC (if we do another RC) or the final release.

[QUOTE=Dj-Grobe;12716]Anyway not understand why have error when try restore sites on my new servers !
Please iworx guy check te ticket ![/QUOTE]

This issue has been resolved via support ticket. For some strange reason, all his accounts had no theme info set in the database (a blank field for each user’s theme). This was causing problem with imports. After fixing the missing fields, imports are now working correctly again, and the import code has been tweaked to set a default theme if no theme information is found in the backup file.

Thankyou socheat, importcode its patched? on my new server? how can translate that fix to my other new servers?

ABout
–backup-options all no-logs no-stats I dfont know about that options already exist that its wonderfull, any way i start thinkinin no use shell, due this ist masive import i plain use panel, due is the best way to see when some site fail in the import. if i use sell, and import 100 sites, and just one fail, i should make visaul revision site per site for see what site fail, form the panel no way to skip logs and stats?

If you have other servers with the same problem, let us know, and we’ll fix them. The fix needs to go on the old server, not the new server. The old server was giving the new server incomplete information.

No, not at this time. As you probably found, the mass import page is already pretty full with options and is complex enough as it is. We may add the ability to control backup options on the mass transfer page in a future release, if we can do so without turning the mass import page into an airplane dashboard :wink:

Well another solution for use page import, be delete the stats and logs on old servers…
But i not have idea how make that with script, i trying figure one script can do that in faster way.

Some sugestion?

Right now, i make the list of old servers on the ticket system, in few minutes i repply that ticket, …some idea about how delete on easy and faster way logs and stats, in masive way?

Here’s how you would list all the log dirs and stats dirs:


ls /home/*/var/*/logs
ls /home/*/var/*/stats

You could probably change “ls” to “rm -rf”, but I would be very careful with that. A simple typo and you could delete a lot more than you wanted. You have been warned. :smiley:

Actually, you’d probably want to do this:


ls /home/*/var/*/logs/*

You don’t want to delete the actual logs dir, just the contents.

Oh lol, i think * not work on taht way …jijiji
Really easy way :slight_smile:
Thankyou

rm -rf /home//var//logs/*
rm -rf /home//var//stats/*

ANyway please post for the future, import from webpage need feature for skip logs and stats if needed :slight_smile:

Just edited : )

[QUOTE=Dj-Grobe;12739]More bugs !!! pleas esombody form iworx check !!!

I have ticket opened !![/QUOTE]

It’s not a bug in InterWorx… :slight_smile: You are trying to create/import a reseller on an IP that isn’t available to resellers. On the IP management page, switch “IP Available to Resellers” to “Yes” and your “bug” will be fixed. :wink:

Puf, jaaja
Thankyou jaja i fell like dunceeeeeee
Just error, when i setup the new server i fogort sahre the ip !! for resellers lol !!
Thanks : )

socheat? when server its SLAVE dns, just need one IP correct? no need 2 ips, correct?

The short answer is yes, you only need one IP.