Well I have implemented my simple incremental/snapshot backup strategy and it works pretty well. The whole thing comprises an rsync backup run daily by anacron in /etc/cron.daily, an rsync server/daemon on the backup NAS and a script on the NAS that rotates the backups. The whole lot is probably 100 lines of code in total - and that's *with* lots of comments! :-)
I have appended the three main component files for those who want to see. The 'clever' bit is that all of the backups on the NAS except the two latest ones are completely inaccessible from the outside, you can only get at them if you have a login on the NAS. That's the bit that is rare in most backup systems I've looked at. You could maybe get a similar result with a 'pull' system run from the backup server but then that would be very slow on the underpowered NAS processor, on my system most of the work is done at the sending end (my desktop).
... oh and the *BIG* advantage is that all the backups are complete mirrors of the backed up files, no compression, no tar archives, you can navigate around the backups exactly as you can navigate around your current home directory. It means it's *very* eay to check that backups are OK and no special tools at all are needed to restore files, just copy the file you want.
So, here is the rsync bit that is run by anacron on my desktop every day. The password is *only* for the rsync server on the backup machine, so access to the script only allows someone to run backups, nothing more.
#!/bin/sh export RSYNC_PASSWORD='XXXXXX' # # # Daily backups, copy everything important to the backup machine # EXCLUDE1='--exclude tmp/ --exclude 'Mail/??/' --exclude .VirtualBox/ --exclude .gvfs/' EXCLUDE2='--exclude .cache/ --exclude lost+found' # # # Home directories # rsync -a --port=874 $EXCLUDE1 $EXCLUDE2 --link-dest=/old/home /home/ chris@dlink::chris2day/new/home # # # /etc # rsync -a --port=874 --link-dest=/old/etc /etc/ chris@dlink::chris2day/new/etc
This is probably the most difficult bit, the rsyncd configuration file on the backup NAS. I could relax the security somewhat and it would be shorter and simpler. The 'post-xfer exec' runs the rotate.sh script after rsync has transferred the data.
hosts allow = 192.168.1.4 hosts deny = * use chroot = no uid = chris gid = allaccount secrets file = /mnt/HD/HD_a2/chris/rsyncd/secrets pid file = /mnt/HD/HD_a2/chris/rsyncd/pid timeout = 300 strict modes = false # log file = /mnt/HD/HD_a2/chris/rsyncd/log
[ chris2day ] path = /mnt/HD/HD_a2/chris/today read only = false list = yes post-xfer exec = /mnt/HD/HD_a2/chris/bin/rotate.sh auth users = chris
... and finally the rotate script which is the bit I was asking about previously. I had the algorithm pretty much right. It's a bit clumsy in the shell (ash, it's a busybox based NAS) but it's not quite messy enough to want to move to something better like Python.
#!/bin/sh # # # Script to 'rotate' snapshots, runs after rsync creates a new # backup in the today/new directory # cd /mnt/HD/HD_a2/chris/bak # # # Get day, month and year # day=`date +%d` month=`date +%m` year=`date +%Y` # # # Check for first of the month # if [ $day = "1" ] then if [ $month = "1" ] then mv month12 $year else rm month12 fi # # # Move all the months up one # m=11 while [ $m != "0" ] do mv month$m month`expr $m + 1` m=`expr $m - 1` done else rmdir day31 fi # # # Move all the days up one # d=30 while [ $d != "0" ] do mv day$d day`expr $d + 1` d=`expr $d - 1` done # # # and copy the new day1 from today # # # Move the older backup from 'today' to 'bak' and change the # name of the newer one # mv ../today/old day1 mv ../today/new ../today/old mkdir ../today/new chown chris:allaccount ../today/new
Like I said I really don't quite understand why rsnapshot et al manage to make it so complicated. OK, the above has 31 days and 12 months hard-coded but given that they're hard-linked snaphots that many backups doesn't take much space at all.
There were of course a few 'deliberate' mistakes in that! :-)
There's a couple of fairly trivial ones in the rotate.sh script in paths that will only get tested on the first of the month. There's a rmdir that should be an 'rm -fr' and also an rm that should be 'rm -fr'.
The bigger and more subtle error is that since there are two calls to rsync in the backup script run on my desktop machine the rotate.sh script gets called twice on the backup NAS. That screws things up rather thoroughly! I've fixed it by calling rsync only once:-
#!/bin/sh export RSYNC_PASSWORD=XXXXX # # # Daily backup, copy everything important to the backup machine # rsync -a --port=874 \ --exclude tmp/ --exclude 'Mail/??/' --exclude .VirtualBox/ --exclude .gvfs/ \ --exclude .cache/ --exclude lost+found/ \ --link-dest=/old /etc /home chris@dlink::chris/new/
In a sense this, to my mind, exemplifies the "it just growed" complexity that many fundamentally simple programs get to. My immediate reaction to fixing the problem was to have some sort of flag passed from client to backup server to say 'do the rotate now'. Or I considered splitting up the rsyncd modules with only one executing the rotate.sh script.
However a bit of thought revealed that I could *simplify* things to make it work correctly. It isn't always possible of course but as sume guru somewhere said effort spent on design is well worth it though it often doesn't feel productive.
-
Chris Green