I currently use a mix of rsync and rdiff-backup to backup my home system.
The basic backup is an rdiff-backup of all of /home to the NAS server in the garage.
*However* (and it's a big however) this has one huge hole in it, rdiff-backup does *not* copy the 'contents' of symbolic links. All it does (if you tell it to) is to copy the symbolic link leaving it on the resulting backup as a symbolic link which points at nothing.
This is all very well if you live in a perfect world where all of your system resides in one neat partition etc. but my home directory has symbolic links to old (and not so old) data on other filesystems and partitions. Thus rdiff-backup doesn't back these up.
I'm looking around for something like rdiff-backup that does incremental backups but *also* copies *everything* that is part of /home/chris whether it's via a symbolic link or not. Can anyone recommend anything?
Hi,
2008/5/7 Chris G cl@isbd.net:
I currently use a mix of rsync and rdiff-backup to backup my home system.
*However* (and it's a big however) this has one huge hole in it, rdiff-backup does *not* copy the 'contents' of symbolic links. All it does (if you tell it to) is to copy the symbolic link leaving it on the resulting backup as a symbolic link which points at nothing.
Is there not a dereference symlink option?
Can anyone recommend anything?
Grab the sources, add dereference symlink option and then can even mail the patch to the authors. :)
I havn't looked at the sources, but I suspect that there is a function that deals with one file, doing all the diffing and revision control. All you'd really need to do is to pass targets of symlinks to the same function. Should be fairly easy to do I would guess.
- Srdj
On Wed, May 07, 2008 at 10:54:42AM +0100, Srdjan Todorovic wrote:
Hi,
2008/5/7 Chris G cl@isbd.net:
I currently use a mix of rsync and rdiff-backup to backup my home system.
*However* (and it's a big however) this has one huge hole in it, rdiff-backup does *not* copy the 'contents' of symbolic links. All it does (if you tell it to) is to copy the symbolic link leaving it on the resulting backup as a symbolic link which points at nothing.
Is there not a dereference symlink option?
No. I only discovered the problem when I lost a (trivial, fortunately) file that was 'down a symlink' and tried to recover it. I then checked the rdiff documentation pretty thoroughly and the only options it has to do with symlinks are to copy them or not to copy them.
Can anyone recommend anything?
Grab the sources, add dereference symlink option and then can even mail the patch to the authors. :)
Well, it's written in python, so that's a possibility I suppose.
I havn't looked at the sources, but I suspect that there is a function that deals with one file, doing all the diffing and revision control. All you'd really need to do is to pass targets of symlinks to the same function. Should be fairly easy to do I would guess.
Yes, I suppose so, I may take a look tonight when I get home.
On Wed, 2008-05-07 at 10:28 +0100, Chris G wrote:
I currently use a mix of rsync and rdiff-backup to backup my home system.
The basic backup is an rdiff-backup of all of /home to the NAS server in the garage.
*However* (and it's a big however) this has one huge hole in it, rdiff-backup does *not* copy the 'contents' of symbolic links. All it does (if you tell it to) is to copy the symbolic link leaving it on the resulting backup as a symbolic link which points at nothing.
Yes assuming that even with the --enable-symbolic-links option it only does what you say and copies the link then it does appear that rdiff-backup does not follow links nor is there an option to allow this.
However I think rdiff-backup uses librsync and rsync itself does have the capability to transform/follow links so I would imagine it would be a pretty trivial change to rdiff-backup to make it do this.
On Wed, May 07, 2008 at 11:03:15AM +0100, Wayne Stallwood wrote:
On Wed, 2008-05-07 at 10:28 +0100, Chris G wrote:
I currently use a mix of rsync and rdiff-backup to backup my home system.
The basic backup is an rdiff-backup of all of /home to the NAS server in the garage.
*However* (and it's a big however) this has one huge hole in it, rdiff-backup does *not* copy the 'contents' of symbolic links. All it does (if you tell it to) is to copy the symbolic link leaving it on the resulting backup as a symbolic link which points at nothing.
Yes assuming that even with the --enable-symbolic-links option it only does what you say and copies the link then it does appear that rdiff-backup does not follow links nor is there an option to allow this.
However I think rdiff-backup uses librsync and rsync itself does have the capability to transform/follow links so I would imagine it would be a pretty trivial change to rdiff-backup to make it do this.
It's not *that* trivial to change, I've just taken a look at the code.
At first glance it's not too bad as there's a copy() function which essentially switches on the file type (regular file, directory, symlink, etc.) and acts accordingly. So one could change the symlink action to copy what it points at rather than just the symlink.
However it gets messy because ACLs, permissions, etc. are done separately and each has a check for symlink, handling symlinks (which are no longer symlinks) for these cases is difficult.
rdiff-backup only uses librsync at the very lowest level to work out the minimum data transfer to update a file as far as I can see.
On Wed, 2008-05-07 at 11:19 +0100, Chris G wrote:
rdiff-backup only uses librsync at the very lowest level to work out the minimum data transfer to update a file as far as I can see.
Ahh in my magical little oversimplified world it was just a wrapper for rsync that added versioning :)
Of course you could just try and replicate the multiple backup versions in a script around rsync (or perhaps unison if it supports the following links thing) but I think it would be potentially quite tricky. I am not sure how rdiff-backup works, does it only keep one full copy of each file and then do cleverness with hard links and compression etc in the backup pool like backuppc does ?
If it just snapshots to a different cyclic location then that would be pretty easy to implement, but would mean you essentially need numberofsnapshots x size of volume to be backed up.
The other option is something like backuppc as this just calls rsync directly and therefore (I have never needed to test it) should be able to follow links if you make the necessary mods to the hosts command file. However if you are only backing up one host then backuppc is a tad overkill.
Wayne Stallwood wrote:
On Wed, 2008-05-07 at 11:19 +0100, Chris G wrote:
rdiff-backup only uses librsync at the very lowest level to work out the minimum data transfer to update a file as far as I can see.
Ahh in my magical little oversimplified world it was just a wrapper for rsync that added versioning :)
I've used rsnapshot, which is just a script that wraps around rsync and gives snapshots. Two settings near the top: rsync_short_args and rsync_long_args Any use? Bill
On Wed, May 07, 2008 at 12:58:22PM +0100, Wayne Stallwood wrote:
On Wed, 2008-05-07 at 11:19 +0100, Chris G wrote:
rdiff-backup only uses librsync at the very lowest level to work out the minimum data transfer to update a file as far as I can see.
Ahh in my magical little oversimplified world it was just a wrapper for rsync that added versioning :)
No, it's rather more than that! :-)
Of course you could just try and replicate the multiple backup versions in a script around rsync (or perhaps unison if it supports the following links thing) but I think it would be potentially quite tricky. I am not sure how rdiff-backup works, does it only keep one full copy of each file and then do cleverness with hard links and compression etc in the backup pool like backuppc does ?
I'm not sure what it does but it's much more sophisticated than a backup pool that rotates. Unless you delete old stuff it keeps a history back to when you started using it and you can restore anything from any time. Obviously it does eventually run out of space but it's pretty efficient. Currently my backup of my home directory is 30.7Gb and my home directory is 30.5Gb (the bulk is a Digikam photo album so that explains a bit).
I think in fact a better approach is probably to find the critical symbolic links and make sure I'm backing up what they point at. It turns out that I only needed to make one change to do this.
Hi,
2008/5/7 Chris G cl@isbd.net:
Unless you delete old stuff it keeps a history back to when you started using it and you can restore anything from any time. Obviously it does eventually run out of space but it's pretty efficient. Currently my backup of my home directory is 30.7Gb and my home directory is 30.5Gb (the bulk is a Digikam photo album so that explains a bit).
Makes me wonder how suitable cvs/svn/git would be for a backup solution across machines. There was an article somewhere in one of the Linux Format issues, detailing how to setup subversion do just that. And I do like how git is decentralised and seems to pack the diffs as well.
Has anyone done anything like this as a backup before?
- Srdj
-
Bill Hill -
Chris G -
Srdjan Todorovic -
Wayne Stallwood