zsync support in backups

zsync (http://zsync.moria.org.uk)

I think this could be another alternative for rsync backups, as this doesn’t require the daemon part of the rsync. So, the clients could do rsync like backups without having their iworx host/server running rsyncd. (a windows binary of zsync clinet is not available on their site; a cygwin port might be available, don’t know about that)

zsync currently works only on single files. And in compressed files only for .gz files. So, ideal for siteworx backups? In the site (http://zsync.moria.org.uk/server), it is said that creating the .gz file using “zsyncmake -z” is more efficient than to have the normal already compressed .gz file.

Could this be (easily) built into iworx?

It’s just a thought, I don’t know you will be considering this or not. :wink: :wink:

I thought I would atleast get a reply (even if it was like “No, we won’t do this”, “it won’t work”, “nobody will use it”, etc)

I apologize Tiger, this one slipped through. No, we don’t plan on adding zsync support to the backup system. We don’t even have rsync in the backup system. Since our backups are tar-gzipped, it doesn’t really make sense to rsync/zsync the archive file. However, if you use the new, upcoming, structure-only feature, you are free to use whatever method you want to backup your site data yourself, be it rsync, zsync, cpio, or whatever. :slight_smile:

Socheat

Yes, I know it and I didn’t request that. zsync seems to be a simple program (runs by itself) and the backup is just doing a download.

Since our backups are tar-gzipped, it doesn’t really make sense to rsync/zsync the archive file.

At http://zsync.moria.org.uk/server , it is said that zsync works with .gz files (then why not for tar.gz?)

However, if you use the new, upcoming, structure-only feature, you are free to use whatever method you want to backup your site data yourself, be it rsync, zsync, cpio, or whatever. :slight_smile:

It is fine if clients could run zsyncmake on their backups to download(zsync) the backups.

Sorry, I misspoke. In InterWorx 3.0, we tar.bzip2 the archive. :frowning:

Why bzip2 instead of gz? Due to better compression?

Better if you could provide an option for it to be in bzip2 or gz (or simply tar).
So that users have the option to rsync (the tar) or zsync (the tar.gz).

As you said users can create the tar/tar.gz of (mail & web data) themselves along with structure-only backup. But, what is the case for database backups? Need to do a mysqldump and then compressing it?

Instead of all the above, better to have an option for tar.gz/tar/tar.bzip2
Let the user decide, and I think it won’t be much difficult to implement this.
(may be not in 3.0, but for a future release)

Just my thoughts.

I agree, that would be the ideal situation. However, for various technical reasons, we had to choose a different compression than gzip.

We contemplated an “importer auto-detect” feature, where you no longer had to pass --control-panel= to the import.pex command. This would allow you to do what you propose, let the user decide the compression type. However, it was complex enough that we decided to table it for a future release.

Socheat

Just to understand.

What is the utility to rsync a tar file ?

I mean the advantage of rsync is to detect which files have been updated between source and destination and to upload only the updated files (differences between source and destination). With a tar file it will be always the case ? no ? the tar file will always be updated after a siteworx account backup.

So why simply not use scp ?

just to understand what you mean :slight_smile:

Pascal

Obviously, to transfer only the changed data. :slight_smile:

I mean the advantage of rsync is to detect which files have been updated between source and destination and to upload only the updated files (differences between source and destination).

Not only that.

the tar file will always be updated after a siteworx account backup.

Yes, it would. But, as rsync transfers only the changes within the file, there is no problem.

And another thing is, I have read that the memory requirements of rsync directly depends on the number of files it operates on. Being a single (tar) file, you are not affected with that (assume you are low/(not high) on memory). But, not sure the tar (a single file) could go beyond 2GB on some servers.

Of course, you could delete the tar, after you rsync it, to gain disk space. But the problem is you need 2 times the space that of your site’s disk requirements, at least temporarily.

I guess TARing the files won’t make the backup not rsyncable. But, I am not sure. (as long as the order of files getting added to the tar is not changed, it should be OK; even in that case the subsequent backup should get back to the optimal transfer)

:slight_smile: :slight_smile:
Anyways, InterWorx people are not going to choose tar/tar.gz and they are with tar.bz2 , so you are on your won (Luckily, they haven’t chosen .7z/.zip which is not yet widely known/used in the linux/unix world).
:slight_smile: :slight_smile:

We use tar because it preserves file permissions and ownerships and because it is so common.

I guess you meant tar.bz2 (as of iworx 3.0) which is not suitable for rsync/zsync.

I think we need to be clear on our terms here, I meant tar – tar is a packaging format which gathers all the files up into one huge file. THIS (as opposed to gzip, bzip or zip) is used to preserve the file perms and ownership, but doesn’t really reduce the overall size (at least not that much). gzip (currently) and bzip (in 3.0) are used to compress the file size to something more manageable. Even thouth they are generally used together tar and g/bzip are two different things. :slight_smile:

OK, I just wanted to make sure you didn’t mean simple .tar as the backup format (i.e without gz/bz2) as Socheat said that tar.bz2 for 3.0.

I know that gzip/bzip2 can compress only one file and 7zip could add multiple files. I guess .7z on Unix doesn’t yet save the file owner/perms (it wasn’t some time ago). Anyways, 7zip would require more memory/time to compress (of course depends on options) than the others (but better compression) and also not widely know as gzip/bzip2. Another one is RAR available for all OSes, but that isn’t free and ( also not widely know :slight_smile: ).

tar is a packaging format which gathers all the files up into one huge file. THIS (as opposed to gzip, bzip or zip) is used to preserve the file perms and ownership, but doesn’t really reduce the overall size.

I knew this. :slight_smile:

gzip (currently) and bzip (in 3.0) are used to compress the file size to something more manageable. Even thouth they are generally used together tar and g/bzip are two different things. :slight_smile:

I knew this too. :slight_smile: :slight_smile:

haha, glad to hear it – just wanted to make sure we were on the same page. :smiley: