I really like Dirvish for small/medium system backups: http://www.dirvish.org/

Cheers, Dennis

On Wed, 13 Sep 2017 at 10:44, Mark Rogers mark@more-solutions.co.uk wrote:

I've been experimenting with backing up and restoring Raspberry Pi SD cards.

There is of-course a very easy way: use "dd" or similar to create a disk image. However it has three disdvantages; it is slow, and it creates very large backups. And it's hard to extract individual files from the backups.

I've written up a quick process [1] for doing it manually, which is essentially:

1. Backup (or restore) the partitions using sfdisk
2. (Restore only) Create the filesystems on the destination
3. Backup (or restore) the files using tar.

However I'd like some advice on scripting and improving it.

In particular, this process creates three files (one for the partition info, and one each for the files from the two partitions). I'd like to end up with a single file, which I can do by just tar'ing the three at the end, but given the size of the files I want to keep the use of temporary files to a minimum.

Also, at the moment it's quite inflexible in terms of restoring to a different sized card, but ought not be.

But regardless, I figured there'd be enough Pi users around here who might find it useful.

(Use at your own risk, it's had very limited testing so far.)

[1] https://docs.google.com/document/d/e/2PACX-1vRk_zldu0Ra6MEEepTSKCzD55ANeasQO...

-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER https://maps.google.com/?q=21+Drakes+Mews,+Milton+Keynes,+MK8+0ER&entry=gmail&source=g

---

main@lists.alug.org.uk http://www.alug.org.uk/ https://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!

On 19 September 2017 at 17:58, steve-ALUG@hst.me.uk wrote:

Could you not just use the built-in sd-card backup/cloning utility?

Which one do you mean? I'm aware of one that clones a running Pi to an SD card in a USB port connected to the Pi, but that doesn't really lend itself to keeping backups off the Pi.

If you don't want to back-up to a separate sd card or USB stick, you could use a mock drive (I think you create a fixed-size file, then format it, then mount it) and backup to that.

That would create a 4GB backup of a 4GB SD card though, which I'm tying to avoid. (Yes it compresses, but not as well or as quickly as using tar to backup just the files in the first place).

Alternatively, grab the source for the new sd-card backup utility and bodge it to backup to where you want it to.

I did look at this (if it's the same tool) and took some inspiration from it, but it was easier to write something from scratch in the end.

Re your backup utility, can I ask a stupid question? If backing up a windows filesystem, in order to create a faithful bootable backup, certain files have to be in certain locations. Is the same true with a PI-backup.

I believe not. The Pi doesn't use a boot sector the way a PC does, and certainly my own experiments so far have confirmed that creating the partition and copying the files across works fine. However I do note that a lot of tools seem to use DD to copy the partition table and the first partition across so I'm not 100% sure I'm not missing something. (So far I've done lots of runs without issue though.)

If not, is there any point in backing up the sfdisk stuff? More fundamentally, if not, just back up the files with tar, gzip, zip or other.

Well I still need to create partitions on the new SD card (the FAT one for /boot and the Ext4 one for /). sfdisk was one way to do that although I'm looking to change that as a direct copy isn't appropriate if restoring a 4GB backup to an 8GB card. Ideally I want the FAT partition to be kept the right size, and the ext4 partition to be whatever is left. And also I want to support more than 2 partitions, as for my own use I want to have a read only / partition and a read/write partition for /var, so that loss of power can't stop it booting, and nor can runaway log files. So I want to support N partitions (N>=2), and have the Nth partition take up whatever space is left with the others copied faithfully.

For what it's worth, I occasionally backup using DD to an image file, which I then zip to save space.

DD is fine for occasional use, but it's slow - especially when you have maybe 1GB used on a 16GB card but DD has to copy the whole thing, and empty space isn't necessarily zeroed so won't always compress well. A dd backup also isn't easy to extract files from (not compared with a .tar.gz file anyway). Tar also has the advantage of being able to ignore files (so the backup can optionally skip log files, apt archives, /tmp, etc). Basically: dd is designed to take an accurate copy of the disk, warts and all, without any knowledge of the filesystem(s), and it does that well. But having something that understands the filesystem brings additional benefits.

I also have a daily Backuppc backup. Backuppc is a pig to set up, but, when it works, it works well. It has the advantage (and possible pitfall), that it backs up only one copy of a file - so if you have 2 Pis and 2 linux machines, and they all have the same version of a library file installed, only one copy will be saved. This saves tons of space. The disadvantage of course is that if that copy of the file gets corrupted, then all the copies will be corrupted.

I use backuppc for other things here in the office. However I wanted something much smaller and simpler to work only with the Pi.

I do now have early but working versions of rpi-backup.sh and rpi-restore.sh if anyone is interested (one day I'll try and get them onto github if there's any interest but I need to stop playing with this now it's working and do some real work!) With hindsight I should have used python as it's more portable (eg to Windows, especially if compiled) but hindsight is a wonderful thing!

On 20/09/17 12:53, Mark Rogers wrote:

On 19 September 2017 at 17:58, steve-ALUG@hst.me.uk wrote:

...

Could you not just use the built-in sd-card backup/cloning utility?

Which one do you mean? I'm aware of one that clones a running Pi to an SD card in a USB port connected to the Pi, but that doesn't really lend itself to keeping backups off the Pi.

SD Card copier, described here: https://www.raspberrypi.org/blog/another-update-raspbian/

...

If you don't want to back-up to a separate sd card or USB stick, you could use a mock drive (I think you create a fixed-size file, then format it, then mount it) and backup to that.

That would create a 4GB backup of a 4GB SD card though, which I'm tying to avoid. (Yes it compresses, but not as well or as quickly as using tar to backup just the files in the first place).

The SD Card copier can and does create an image suitable for the media you are copying too.  If you copy from a 4GB card to an 8GB, it will expand the partitions to use all the free space.  Similarly, it will down-size.  If you work out how much actual space is used and create the "fake drive" of just that size plus a bit of leeway, then you, create a minimal-ish backup.

You say above that sd-sd copying doesn't lend itself to keeping backups.  I respectfully disagree.  Back in the day I had loads of backup disks, so backup SDs is hardly different, but more importantly, that's why I suggested making a fake drive from a file.  Once created just move the file to a backup location, or you may be able to create the file for a fake drive on a network drive, and mount and backup to it without having to move it at all.

{snip}

...

I also have a daily Backuppc backup. Backuppc is a pig to set up, but, when it works, it works well. It has the advantage (and possible pitfall), that it backs up only one copy of a file - so if you have 2 Pis and 2 linux machines, and they all have the same version of a library file installed, only one copy will be saved. This saves tons of space. The disadvantage of course is that if that copy of the file gets corrupted, then all the copies will be corrupted.

I use backuppc for other things here in the office. However I wanted something much smaller and simpler to work only with the Pi.

Hmm,  once set up, the backuppc pi backup is quick, reliable and only backs up what you want it to, and what has changed.  If you have already got a backuppc server set up,  all you'd have to set up is the client config.  To be honest I'd have thought that's a lot easier than writing your own backup system!

Steve

On 20 September 2017 at 21:09, steve-ALUG@hst.me.uk wrote:

SD Card copier, described here: https://www.raspberrypi.org/blog/another-update-raspbian/

OK, that looks like the one I had looked at.

The SD Card copier can and does create an image suitable for the media you are copying too. [...] If you work out how much actual space is used and create the "fake drive" of just that size plus a bit of leeway, then you, create a minimal-ish backup.

That's a thought, although calculating that minimum size wouldn't be easy would it? (It's not just how much space is used on the disk, unless the sectors have been used consecutively.)

You say above that sd-sd copying doesn't lend itself to keeping backups. I respectfully disagree.

If you want a faithful bit-copy then that's fine, but how do you adapt the image approach to not include (for example) log or temporary files?

Hmm, once set up, the backuppc pi backup is quick, reliable and only backs up what you want it to, and what has changed.

If what I wanted was rolling backups of a live system that would make sense, but what I'm after are snapshots that I can use to "clone" the SD card, and when I say "I" I also mean other people if needed.

I think the problem is in how I framed the original post. My objectives are: - To have something I run offline (not from the Pi). I need to be able to clone multiple cards etc without having a Pi up and running. (As an aside, that also means I can do the work on a USB3 port on my desktop which speeds things up.) - For the backups to be as small as possible and not include irrelevant files (eg logs) - For backups and restores to be as fast as possible. In particular, not reading from/writing to unused parts of the SD cards - For backup files to be "accessible", in that it is as trivial as possible to extract files from them, compare backups etc. - For the backup/restore process to be self contained (not needing other tools) so that other people can use it with minimal overhead.

As an aside, I mentioned yesterday that I should have used Python for cross-platform, forgetting that 90% of what the script does is calling external tools like mkfs.ext4 which aren't going to be present on a non-Linux system. Doh!

On 21 September 2017 at 09:46, Chris Green cl@isbd.net wrote:

I have my own incremental backup system which I use across several computers (desktop, two laptops and a Raspbery Pi). I wrote it myself because I was using rdiff-nackup and wanted to improve on that.

I think this or Steve/s approaches would be the ways to go it I wanted to do that kind of backup.

However, for my needs a dd-style backup is more appropriate (but slow, creates large files, etc, hence looking for better ways). As I mentioned in my last email, I think I badly described my needs in the OP (maybe backup/restore is better described in terms of disk imaging, albeit that actually imaging a disk is exactly what I'm trying to avoid).

My use case is more about setting one Pi up the way I like it and then creating a dozen clones of it, than it is about not losing the contents of the original Pi. (To be fair, that is a separate requirement so this has been a very useful discussion all the same.) Or it's about setting up a Pi the way I think I like it, then backing up the whole thing because I'm about to play with it and break it horribly or use it for some other purpose. (I tend to just swap uSD cards cards around but given they're too small to label that's not working out too well!)

On 22 September 2017 at 13:08, Chris Green cl@isbd.net wrote:

As a general rule for myself I do my very best not to customise things too much and to keep installations as standard as possible. Thus all my installations, as far as possible, only have differences from 'as installed' in /etc and /home so I just back those up.

Ditto, although increasingly I'm starting to use /opt as well.

One thing I like about my tar-based backup is that in principle I can add additional "incremental" [*] tarballs to the base quite easily. So I can, for example, take a current Raspbian, apt upgrade it to get latest packages (and maybe a standard set of non-default packages I use), then back it up as a snapshot. I can then take my "application" code and include that in the backup so that it gets restored over the top and ends up with a finished system, but in a way which is more-or-less trivial to manage the OS updates separately from the application.

[*] It's not a true incremental backup as it would only replace or add files, not delete them. But it's close enough for my purposes, and I'm sure the extra deletion step could be brought in if needed.

On 24 September 2017 at 10:06, Nev Young alug@nevilley.org.uk wrote:

I'm a bit late to the discussion but my set-up may help.

Thanks for that, it is interesting to see how other people do things. It doesn't suit my purposes (which is more about creating images of Pi SD cards for the purposes of creating duplicate systems, rather than backing up in case of mishap), but it's a useful rsync example nonetheless.