On 19 September 2017 at 07:47, Dennis Dryden ddryden@gmail.com wrote:

I really like Dirvish for small/medium system backups: http://www.dirvish.org/

Dirvish strikes me as ideal for creating backups from within a running Pi on a regular basis, but less suited to taking an SD card out of the Pi, putting it into my desktop and taking a backup of the SD card to go back to later, or for creating half a dozen copies of the SD card to identically configure several Pi's.

Not that I have ever used Dirvish (I looked at it many years ago but can't remember why I didn't adopt it).

On 19 September 2017 at 17:58, steve-ALUG@hst.me.uk wrote:

Could you not just use the built-in sd-card backup/cloning utility?

Which one do you mean? I'm aware of one that clones a running Pi to an SD card in a USB port connected to the Pi, but that doesn't really lend itself to keeping backups off the Pi.

If you don't want to back-up to a separate sd card or USB stick, you could use a mock drive (I think you create a fixed-size file, then format it, then mount it) and backup to that.

That would create a 4GB backup of a 4GB SD card though, which I'm tying to avoid. (Yes it compresses, but not as well or as quickly as using tar to backup just the files in the first place).

Alternatively, grab the source for the new sd-card backup utility and bodge it to backup to where you want it to.

I did look at this (if it's the same tool) and took some inspiration from it, but it was easier to write something from scratch in the end.

Re your backup utility, can I ask a stupid question? If backing up a windows filesystem, in order to create a faithful bootable backup, certain files have to be in certain locations. Is the same true with a PI-backup.

I believe not. The Pi doesn't use a boot sector the way a PC does, and certainly my own experiments so far have confirmed that creating the partition and copying the files across works fine. However I do note that a lot of tools seem to use DD to copy the partition table and the first partition across so I'm not 100% sure I'm not missing something. (So far I've done lots of runs without issue though.)

If not, is there any point in backing up the sfdisk stuff? More fundamentally, if not, just back up the files with tar, gzip, zip or other.

Well I still need to create partitions on the new SD card (the FAT one for /boot and the Ext4 one for /). sfdisk was one way to do that although I'm looking to change that as a direct copy isn't appropriate if restoring a 4GB backup to an 8GB card. Ideally I want the FAT partition to be kept the right size, and the ext4 partition to be whatever is left. And also I want to support more than 2 partitions, as for my own use I want to have a read only / partition and a read/write partition for /var, so that loss of power can't stop it booting, and nor can runaway log files. So I want to support N partitions (N>=2), and have the Nth partition take up whatever space is left with the others copied faithfully.

For what it's worth, I occasionally backup using DD to an image file, which I then zip to save space.

DD is fine for occasional use, but it's slow - especially when you have maybe 1GB used on a 16GB card but DD has to copy the whole thing, and empty space isn't necessarily zeroed so won't always compress well. A dd backup also isn't easy to extract files from (not compared with a .tar.gz file anyway). Tar also has the advantage of being able to ignore files (so the backup can optionally skip log files, apt archives, /tmp, etc). Basically: dd is designed to take an accurate copy of the disk, warts and all, without any knowledge of the filesystem(s), and it does that well. But having something that understands the filesystem brings additional benefits.

I also have a daily Backuppc backup. Backuppc is a pig to set up, but, when it works, it works well. It has the advantage (and possible pitfall), that it backs up only one copy of a file - so if you have 2 Pis and 2 linux machines, and they all have the same version of a library file installed, only one copy will be saved. This saves tons of space. The disadvantage of course is that if that copy of the file gets corrupted, then all the copies will be corrupted.

I use backuppc for other things here in the office. However I wanted something much smaller and simpler to work only with the Pi.

I do now have early but working versions of rpi-backup.sh and rpi-restore.sh if anyone is interested (one day I'll try and get them onto github if there's any interest but I need to stop playing with this now it's working and do some real work!) With hindsight I should have used python as it's more portable (eg to Windows, especially if compiled) but hindsight is a wonderful thing!

{snip}

...

...

I also have a daily Backuppc backup. Backuppc is a pig to set up, but, when it works, it works well. It has the advantage (and possible pitfall), that it backs up only one copy of a file - so if you have 2 Pis and 2 linux machines, and they all have the same version of a library file installed, only one copy will be saved. This saves tons of space. The disadvantage of course is that if that copy of the file gets corrupted, then all the copies will be corrupted.

I use backuppc for other things here in the office. However I wanted something much smaller and simpler to work only with the Pi.

Hmm,  once set up, the backuppc pi backup is quick, reliable and only backs up what you want it to, and what has changed.  If you have already got a backuppc server set up,  all you'd have to set up is the client config.  To be honest I'd have thought that's a lot easier than writing your own backup system!

I'm coming into this late but......

I have my own incremental backup system which I use across several computers (desktop, two laptops and a Raspbery Pi). I wrote it myself because I was using rdiff-nackup and wanted to improve on that.

My backup is written (mostly) in python and uses rsync in server mode for the underlying file copying.

It does daily (can be other intervals, I do some more frequent ones across disks on my desktop) incremental backups of a selection of directory trees to a 'remote' backup system in the garage. It comprises only a hundred lines or so of code so is simple and easy to maintain. The resulting backups are all simply copies of the original diretory trees and files can be restored simply by copying them back.

Configuration is pretty simple comprisng a standard rsync '--files-from' file list and a standard rsync server setup on the backup machine. The python scripts that do the 'clever' stuff are linked from the rsync server configuration as 'pre-xfer' and 'post-xfer' scripts.

I just looked, the python scripts are 112 lines and 41 lines, 72 lines of those are comments.

rsync in server mode is a very powerful beastie! :-)

On 21 September 2017 at 09:46, Chris Green cl@isbd.net wrote:

I have my own incremental backup system which I use across several computers (desktop, two laptops and a Raspbery Pi). I wrote it myself because I was using rdiff-nackup and wanted to improve on that.

I think this or Steve/s approaches would be the ways to go it I wanted to do that kind of backup.

However, for my needs a dd-style backup is more appropriate (but slow, creates large files, etc, hence looking for better ways). As I mentioned in my last email, I think I badly described my needs in the OP (maybe backup/restore is better described in terms of disk imaging, albeit that actually imaging a disk is exactly what I'm trying to avoid).

My use case is more about setting one Pi up the way I like it and then creating a dozen clones of it, than it is about not losing the contents of the original Pi. (To be fair, that is a separate requirement so this has been a very useful discussion all the same.) Or it's about setting up a Pi the way I think I like it, then backing up the whole thing because I'm about to play with it and break it horribly or use it for some other purpose. (I tend to just swap uSD cards cards around but given they're too small to label that's not working out too well!)

On 22 September 2017 at 13:08, Chris Green cl@isbd.net wrote:

As a general rule for myself I do my very best not to customise things too much and to keep installations as standard as possible. Thus all my installations, as far as possible, only have differences from 'as installed' in /etc and /home so I just back those up.

Ditto, although increasingly I'm starting to use /opt as well.

One thing I like about my tar-based backup is that in principle I can add additional "incremental" [*] tarballs to the base quite easily. So I can, for example, take a current Raspbian, apt upgrade it to get latest packages (and maybe a standard set of non-default packages I use), then back it up as a snapshot. I can then take my "application" code and include that in the backup so that it gets restored over the top and ends up with a finished system, but in a way which is more-or-less trivial to manage the OS updates separately from the application.

[*] It's not a true incremental backup as it would only replace or add files, not delete them. But it's close enough for my purposes, and I'm sure the extra deletion step could be brought in if needed.

On 24 September 2017 at 10:06, Nev Young alug@nevilley.org.uk wrote:

I'm a bit late to the discussion but my set-up may help.

Thanks for that, it is interesting to see how other people do things. It doesn't suit my purposes (which is more about creating images of Pi SD cards for the purposes of creating duplicate systems, rather than backing up in case of mishap), but it's a useful rsync example nonetheless.