Hello ALUG,
I'm trying to write a bash script to mirror some files on my server to another machine (to be run periodically, not a dynamic mirror).
First it creates a file which contains the full path of all the files that need to be copied (they are in lots of different directories) separated by newlines (I'd like to keep this aspect of the script as it is also used to create an ISO to backup to DVD sometimes).
Next I want to copy all the files in the list to the remote computer. (I've been reading up on scp and have managed to add my key the remote machine so that the script isn't interrupted by password prompts.) The problem I've got is that each item in the list is the name of a regular file including its path and scp won't allow me to copy to the remote machine into a path which doesn't exist:
$ROOT=/var/server-mirror ..... for f in `cat FILES` do scp $f $USER@$HOST:$ROOT/$f done
[where each $f will be the absolute path of the local file, e.g. '/var/www/index.html' which should copy to '/var/server-mirror/var/www/index.html' on the remote machine]
The -r switch, of course, isn't the solution because that only applies to directories at the local end and still doesn't allow you to copy to non-existant directories at the remote end. I suppose what I need is a sort of 'rmkdir' with the -p switch:
rmkdir -p $HOST:$ROOT/`dirname $f`
Any ideas?
Cheers, Richard
PS. Theres no need to use scp if a solution exists using something else....
Richard Lewis wrote:
I suppose what I need is a sort of 'rmkdir' with the -p switch:
rmkdir -p $HOST:$ROOT/`dirname $f`
If you can scp to that machine, I presume you can ssh to it too, right? Then you can use ssh to do the mkdir:
# prepare test mak@yoda$ mkdir -p foo/bar; touch foo/bar/baz mak@yoda$ ssh taurus mkdir a_dir
# copy mak@yoda$ ssh taurus mkdir -p a_dir/`dirname /home/mak/foo/bar/baz` mak@yoda$ scp /home/mak/foo/bar/baz \ taurus:a_dir/`dirname /home/mak/foo/bar/baz` baz 100% 0 0.0KB/s 00:00 mak@yoda$ ssh taurus ls a_dir/home/mak/foo/bar/baz a_dir/home/mak/foo/bar/baz
PS. Theres no need to use scp if a solution exists using something else....
rsync will also do what you want, without the mkdir:
mak@yoda$ ssh taurus mkdir some_dir
mak@yoda$ rsync -e `which ssh` --relative \ /home/mak/foo/bar/baz taurus:some_dir mak@yoda$ ssh taurus ls some_dir/home/mak/foo/bar/baz some_dir/home/mak/foo/bar/baz
-- Martijn
On Thu, 20 Jan 2005 17:41:57 +0000, Martijn Koster mak-alug@greenhills.co.uk wrote:
Richard Lewis wrote:
PS. Theres no need to use scp if a solution exists using something else....
rsync will also do what you want, without the mkdir:
Richard -
I was struggling writing a bunch of scp/rsync/shell scripts, then someone pointed me at a backup program called dirvish. Took me a few days to try and work out the settings, but I'm very happy with it now it is established.
Jen
On Thu, 20 Jan 2005 17:41:57 +0000, "Martijn Koster" mak-alug@greenhills.co.uk said:
Richard Lewis wrote:
I suppose what I need is a sort of 'rmkdir' with the -p switch:
rmkdir -p $HOST:$ROOT/`dirname $f`
If you can scp to that machine, I presume you can ssh to it too, right? Then you can use ssh to do the mkdir:
Good idea.
I've split the operation in two, first it generates the remote directory tree then it copies all the files: for f in `sort FILES` do d=`dirname $f` if [[ $last != $d ]] then ssh $MIRROR_USER@$MIRROR_IP "[ -d $MIRROR_PATH$d ] || mkdir -p $MIRROR_PATH$d" echo $MIRROR_PATH$d fi last=$d done # copy files....
PS. Theres no need to use scp if a solution exists using something else....
rsync will also do what you want, without the mkdir:
mak@yoda$ ssh taurus mkdir some_dir
mak@yoda$ rsync -e `which ssh` --relative \ /home/mak/foo/bar/baz taurus:some_dir mak@yoda$ ssh taurus ls some_dir/home/mak/foo/bar/baz some_dir/home/mak/foo/bar/baz
What I've got now is /very/ slow.
I had a quick look at the rsync man page and it seems that it makes intelligent descisions about what to copy. I might have to try that.
Thanks for the suggestions.
Cheers, Richard
If you're just trying to copy a directory structure over a network I've used rcmd (on UNIX) like...
rcmd -l user host "(cd /usr/rob ; find . -print -depth | cpio -ocdu) " | (cd /usr/somewhere | cpio -icvduAm)
Regards, Rob.
On 20 Jan 2005, at 5:06 pm, Richard Lewis wrote:
Hello ALUG,
I'm trying to write a bash script to mirror some files on my server to another machine (to be run periodically, not a dynamic mirror).
First it creates a file which contains the full path of all the files that need to be copied (they are in lots of different directories) separated by newlines (I'd like to keep this aspect of the script as it is also used to create an ISO to backup to DVD sometimes).
Next I want to copy all the files in the list to the remote computer. (I've been reading up on scp and have managed to add my key the remote machine so that the script isn't interrupted by password prompts.) The problem I've got is that each item in the list is the name of a regular file including its path and scp won't allow me to copy to the remote machine into a path which doesn't exist:
$ROOT=/var/server-mirror ..... for f in `cat FILES` do scp $f $USER@$HOST:$ROOT/$f done
[where each $f will be the absolute path of the local file, e.g. '/var/www/index.html' which should copy to '/var/server-mirror/var/www/index.html' on the remote machine]
The -r switch, of course, isn't the solution because that only applies to directories at the local end and still doesn't allow you to copy to non-existant directories at the remote end. I suppose what I need is a sort of 'rmkdir' with the -p switch:
rmkdir -p $HOST:$ROOT/`dirname $f`
Any ideas?
Cheers, Richard
PS. Theres no need to use scp if a solution exists using something else.... -- Richard Lewis richardlewis@fastmail.co.uk
-
Jenny Hopkins -
Martijn Koster -
Richard Lewis -
Robert Tillyard