I'm using rsync to copy some files across my LAN. There are various ways I can do this and I'd like to know which is likely to be fastest.
The (most obvious) ways I can see of doing it are:-
1 - Run rxvt in conventional fashion across the network to the remote system, e.g. with syntax like:- rxvt -a /thisDir/ thatSystem:/thatDir/
2 - Use the (already mounted) NFS mount of the remote system and, from rxvt's point of view copy locally:- rxvt -a /thisDir/ /mnt/thatDir/
3 - sshfs? Doesn't really seem a very sensible candidate.
My *guess* is that probably option 1 is best but I'm not really sure. The object is more to minimise system load (and network load) than to get the absolute best speed. The copies (backups) will likely run when I'm using the system so I want them not to overload things too much.
On Sun, Jan 03, 2010 at 02:38:43PM +0000, Chris G wrote:
I'm using rsync to copy some files across my LAN. There are various ways I can do this and I'd like to know which is likely to be fastest.
The (most obvious) ways I can see of doing it are:-
1 - Run rxvt in conventional fashion across the network to the remote system, e.g. with syntax like:- rxvt -a /thisDir/ thatSystem:/thatDir/
2 - Use the (already mounted) NFS mount of the remote system and, from rxvt's point of view copy locally:- rxvt -a /thisDir/ /mnt/thatDir/
3 - sshfs? Doesn't really seem a very sensible candidate.
My *guess* is that probably option 1 is best but I'm not really sure. The object is more to minimise system load (and network load) than to get the absolute best speed. The copies (backups) will likely run when I'm using the system so I want them not to overload things too much.
..., option 1 does have a disadvantage though, it requires that there is a passwordless (unattended passwordless that is) root login from my system to the server. The file copies are backups run as root from /etc/cron.daily.
Chris G wrote:
On Sun, Jan 03, 2010 at 02:38:43PM +0000, Chris G wrote:
I'm using rsync to copy some files across my LAN. There are various ways I can do this and I'd like to know which is likely to be fastest.
Not entirely sure if this is helpful or not, but there is a program called Unison which (according to the blurb in Ubuntu's add/remove programs) " Unison offers several advantages over various synchronization methods such as CVS, Coda, rsync, Intellisync, etc. Unison can run on and synchronize between Windows and many UNIX platforms. Unison requires no root privileges, system access or kernel changes to function. Unison can synchronize changes to files and directories in both directions, on the same machine, or across a network using ssh or a direct socket connection. Transfers are optimised using a version of the rsync protocol, making it ideal for slower links. Unison has a clear and precise specification, and is resilient to failure due to its careful handling of the replicas and its private structures.
Homepage: http://www.cis.upenn.edu/~bcpierce/unison/"
HTH
On Tue, Jan 05, 2010 at 04:03:44PM +0000, steve-ALUG@hst.me.uk wrote:
Chris G wrote:
On Sun, Jan 03, 2010 at 02:38:43PM +0000, Chris G wrote:
I'm using rsync to copy some files across my LAN. There are various ways I can do this and I'd like to know which is likely to be fastest.
Not entirely sure if this is helpful or not, but there is a program called Unison which (according to the blurb in Ubuntu's add/remove programs) " Unison offers several advantages over various synchronization methods such as CVS, Coda, rsync, Intellisync, etc. Unison can run on and synchronize between Windows and many UNIX platforms. Unison requires no root privileges, system access or kernel changes to function. Unison can synchronize changes to files and directories in both directions, on the same machine, or across a network using ssh or a direct socket connection. Transfers are optimised using a version of the rsync protocol, making it ideal for slower links. Unison has a clear and precise specification, and is resilient to failure due to its careful handling of the replicas and its private structures.
Homepage: http://www.cis.upenn.edu/~bcpierce/unison/"
I'm after backup rather than synchronisation so Unison doesn't really offer what I want. What I'm actually doing is to rsync files to the backup system and then rdiff-backup from the rsync image to what is effectively an incremental backup. Since only the rsync image is visible to the outside world unless an intruder actually breaks into the backup system itself they can't remove the incremental backup and thus even if the original or rsync image are destroyed I should be able to restore files still.
I have compared speed of rsync copying using 'local' rsync copy to an NFS mounted partition as opposed to rsync connecting to a remote rsync (via ssh) and copying that way. The rsync to rsync copying is *much* faster, it's almost ten times faster where there are few files that actually need copying (i.e. rsync is doing its job of only copying changed files). I guess this is because file opening to get size etc. is rather slow on NFS.
Anyway I'm finally just about sorted. I have set up an rsync daemon on the backup system (which thus avoids the need for any passwordless login) and make daily copies of the areas I want to back up in the desktop systems. A script in /etc/cron.daily does this which means it gets run by anacron soon after the system is switched on, on the desktop systems rsync running is only a very small load and isn't noticeable while using the system.
Then, on the backup system, I run rdiff-backup to do an incremental backup of the files copied by rsync. I'm using normal cron to do this as the backup system is left running all the time. It's a Western Digital My Book World Edition II NAS which consumes less than 10 watts when idle. I'm pretty impressed with it in general, it runs Linux which they have made accessible by allowing ssh login to the system, thus it's pretty easy to add utilities like rsync and rdiff-backup.
Chris G wrote:
I'm after backup rather than synchronisation so Unison doesn't really offer what I want. What I'm actually doing is to rsync files to the backup system
Sorry - don't mean to be thick here, but surely your first step is to ensure that your main machine and the backup machine are synchronised, which is (AFAIK) what both Unison and rsync do. If you're not synchronising the two machines, then surely you'd be using copy (cp) not rsync?
The rsync to rsync copying is *much* faster, it's almost ten times faster where there are few files that actually need copying (i.e. rsync is doing its job of only copying changed files). I guess this is because file opening to get size etc. is rather slow on NFS.
Also, you have a processor at each end working on the problem; in the attached NFS route, all the processing (apart from file-serving) will be done at one end.
Anyway I'm finally just about sorted.
Good! :-)
On Tue, Jan 05, 2010 at 08:39:58PM +0000, steve-ALUG@hst.me.uk wrote:
Chris G wrote:
I'm after backup rather than synchronisation so Unison doesn't really offer what I want. What I'm actually doing is to rsync files to the backup system
Sorry - don't mean to be thick here, but surely your first step is to ensure that your main machine and the backup machine are synchronised, which is (AFAIK) what both Unison and rsync do. If you're not synchronising the two machines, then surely you'd be using copy (cp) not rsync?
I'm not synchronising, no changes on the backup machine will ever get back to the client machine. If you read the manual pages for rsync it announces itself as "a fast and extraordinarily versatile file copying tool.", it *can* synchronise but it's also a very good file copying tool. It's better than cp because it optimises the copy so that only changes are copied, if you're backing up several (hundreds of?) gigabytes this is very useful as a simple copy would take many hours.
On 7 Jan 2010, at 11:25, Chris G wrote:
On Tue, Jan 05, 2010 at 08:39:58PM +0000, steve-ALUG@hst.me.uk wrote:
Chris G wrote:
I'm after backup rather than synchronisation so Unison doesn't really offer what I want. What I'm actually doing is to rsync files to the backup system
Sorry - don't mean to be thick here, but surely your first step is to ensure that your main machine and the backup machine are synchronised, which is (AFAIK) what both Unison and rsync do. If you're not synchronising the two machines, then surely you'd be using copy (cp) not rsync?
I'm not synchronising, no changes on the backup machine will ever get back to the client machine.
Perhaps this is just a language/communication issue.
rsync can be used to synchronise one folder TO the other i.e. make the target folder the same as the current, in an efficient manner (very much like patch)
On Thu, Jan 07, 2010 at 11:33:04AM +0000, James Taylor wrote:
On 7 Jan 2010, at 11:25, Chris G wrote:
On Tue, Jan 05, 2010 at 08:39:58PM +0000, steve-ALUG@hst.me.uk wrote:
Chris G wrote:
I'm after backup rather than synchronisation so Unison doesn't really offer what I want. What I'm actually doing is to rsync files to the backup system
Sorry - don't mean to be thick here, but surely your first step is to ensure that your main machine and the backup machine are synchronised, which is (AFAIK) what both Unison and rsync do. If you're not synchronising the two machines, then surely you'd be using copy (cp) not rsync?
I'm not synchronising, no changes on the backup machine will ever get back to the client machine.
Perhaps this is just a language/communication issue.
rsync can be used to synchronise one folder TO the other i.e. make the target folder the same as the current, in an efficient manner (very much like patch)
... but that's simply copy isn't it? :-) OK, rsync does it more efficently than cp does but the result is identical.
Synchronising is, to my mind, bi-directional. It's the sort of thing you do with your PDA/phone when you 'synchronise' it with your desktop calendar. New changes on the desktop ar copied to the PDA, new changes on the PDA are copied to the desktop.
Similarly you might want to synchronise your home desktop (or part of it) with your work desktop, changes are copied *both* ways.
Unison most definitely does synchronise in the sense I mean. Rsync *can* do this but it's not really its primary use I don't think.
On 7 Jan 2010, at 11:52, Chris G wrote:
On Thu, Jan 07, 2010 at 11:33:04AM +0000, James Taylor wrote:
On 7 Jan 2010, at 11:25, Chris G wrote:
On Tue, Jan 05, 2010 at 08:39:58PM +0000, steve-ALUG@hst.me.uk wrote:
Chris G wrote:
I'm after backup rather than synchronisation so Unison doesn't really offer what I want. What I'm actually doing is to rsync files to the backup system
Sorry - don't mean to be thick here, but surely your first step is to ensure that your main machine and the backup machine are synchronised, which is (AFAIK) what both Unison and rsync do. If you're not synchronising the two machines, then surely you'd be using copy (cp) not rsync?
I'm not synchronising, no changes on the backup machine will ever get back to the client machine.
Perhaps this is just a language/communication issue.
Synchronising is, to my mind, bi-directional.
I'll say again, I think this is a communication/language issue. Synchronisation is the act of making synchronous - and synchronous only means "the same", that is *after* the process both sides are the same, no matter what method it used to get there:
Whether this means:
a) changes on one are passed to the other *and* vice-a-versa, OR b) if the changes are only passed from one to the other*,
is dependant upon context, and not a fixed rule. Quite often, synchronise only means one way** - consider synchronising to time servers, dns slaves, and other contexts like "manage.py syncdb" in django, both of which are one way copies, so I would be very wary of thinking that when people, programs and documentation say synchronise that this always means bi-directional.
Best Regards
James
* For completeness, just passing the changes isn't enough for a one way synch - any changes on the target would get lost in a true synch. ** One way synchronisation is trivial compared to two way - look at the difficulties many SCM apps have with merges for an example.
On 05/01/10 16:03, steve-ALUG@hst.me.uk wrote:
Not entirely sure if this is helpful or not, but there is a program called Unison
We use unison extensively, it's a handy tool although I'm not sure it'll do what Chris wants.
That said, I'm not entirely sure what Chris wants; copying files across a network where they don't already exist at the destination would presumably incur an overhead for rsync, so I'm assuming that there is already a reasonably full set at the destination and the idea is to update the changes?
I think we need to know much more about the size and number of files, and whether the destination already has copies which need to be updated, etc. The quickest way to copy a single 2GB text file almost certainly isn't the same as the quickest way to copy 2 million 1K zip files!
-
Chris G -
James Taylor -
Mark Rogers -
steve-ALUG＠hst.me.uk