On 06/06/13 18:26, Srdjan Todorovic wrote:

Hi,

On 6 June 2013 16:32, Mark Rogers mark@quarella.co.uk wrote:

...

I need to download a set of backup files from a (Windows) server, deleting all but the newest from the server once successfully downloaded.

Something like Expect [0] could really help you here. You'd have to write all the file age, already-have-a-copy-of-a-file (etc) logic yourself though.

Wow - Expect looks interesting! I'd not seen that before.

Mark, I can think of several possibilities, all of which involve some sort of automation - shell script or batch file. These would require checking the status of a command to see if it suceeded: e.g. Linux: Not entirely sure how, but perhaps something along the lines of what's described here see http://stackoverflow.com/questions/3822621/how-to-exit-if-a-command-failed

Windows Batch File: Somecommand if errorlevel==1 goto SomeCommandFailed

With the windows batch file, you must check that "Somecommand" actually sets the error level if the command fails. I think from memory, that means success, and >0 means failure. That example, I think would goto SomeCommandFailed for any return value greater than 0 - but it's been a while - check!

I'm assuming you don't know in advance the names of the files you're copying.

You could a) use expect as Srdjan suggested

b) Install cygwin on the windows server. Use rsync to copy all the relevant files from Windows to your server. Rsync will only copy files that need copying, omitting files that are the same on both sides. (alternatively use scp) If controlling this from your server's side, use ssh to send commands to the windows server to delete the old/copied files. If controlling this from the Windows server, use DOS command to delete the files you've already copied - not entirely sure how to calculate the date of the files in as Windows Batch file. Alternatively, you could use a vbscript program on Windows - this would give you more flexibility in calcuating the dates.

c) Windows Centric: Launch the process from the windows machine. Set up a scheduled task to launch a batch file, vbs program or some other program to do stuff for you. If you try to do a mass ftp transfer off all files in one go (I'm not sure that you can), then if it fails, you'll probably have to re-transfer the whole lot as you won't know which file failed. So, you'll probably have to copy one file at a time.

A very very simple set of batch files to do this would look something like

MainFile.BAT CD MyDirectory

|FOR %%A IN (/*.BAK/) DO CALL C:\BATCHDirectory\MYFTP %%/A /IF ERRORLEVEL == 1 GOTO FAILED GOTO SUCCESS

:FAILED Echo "Failure" END 1

:SUCCESS Echo "Success" END 0

||MYFTP|.BAT IF "%1"=="" GOTO NOPARAMS

FTP %1 REMOTESERVERPATH 'or whatever the format of the FTP command is... IF ERRORLEVEL == 1 GOTO COPYFAILED ECHO "Successfully copied %1" END 0

:COPYFAILED ECHO "Failed to copy copied %1" END 1

:NOPARAMS ECHO "Must specify a file name" END 2

You could do the same in vbscript or some higher level program, but then you'd have to work out how to call a dos command from vbs (it is do-able). You then need to calculate which files to delete and delete them - this would be much easier in a vbscript program. I *Think* you can calculate dates of files in a batch file, but off the top of my head, I can't remember how!

d) Windows centric Just throwing this out there. Investigate if there are winscp or wsftp can do the copying for you and be scripted adequately.

e) Linux Centric Install the FTPCopy (debian/ubuntu) package Use ftpls to generate a listing of the windows server's files. Use ftpcopy to copy the files from the remote server (copying whole directories is supported) Compare the remote file list with the current directory and see if all files have transferred, if yes, work out which ones to delete. Use ftp to delete the remote files, probably one at a time

f) linux cetnric Use straight ftp command with get, mget or reget to fetch remote files. If they transfer OK, use mdelete to delete the ones on the remote machine.

Hope that helps in some way!

I think where you go depends on your familiarity with the various environments, e.g. do you do Batch files or vbscript, or are you more comfortable with linux scripts or perl, php etc?

My inclination would be to do the cygwin way. Then a) you're using linux at both ends b) you have the linux tools you're familiar with at both ends c) you can use rsync to efficiently transfer the files (do it over ssh if you need them to be secure) d) you can issue a command locally something like this to delete the old files on the windows server

|ssh user@WindowsServer|find /path/to/files* -mtime +2 -exec rm {} ;

See: http://www.howtogeek.com/howto/ubuntu/delete-files-older-than-x-days-on-linu... http://stackoverflow.com/questions/305035/how-to-use-ssh-to-run-shell-script...

That's enough for now! Good luck!

Cheers Steve

Thanks everyone for the suggestions!

Some comments, in no particular order:

expect: Long time since I used that - long enough to have forgotten it existed! I'll have another look, for nostaligia if nothing else!

cygwin: I think I'd have problems convincing the server owner to all this to be installed. To be honest, there are simpler solutions, for example we could script having multiple copies of the backups on the host; one which is maintained by the host (keep files for 2 days then delete them), the other which is maintained by my box (download all files and delete them from the host). It just feels like the "wrong" way too do things - I don't like having to keep multiple copies of large files around just because I can't find a better solution! On Linux I could just use hard links to duplicate the files without actually using disk space, but as far as I know doing that on Windows isn't an option?

ncftp: Good point, I have used these tools before on Windows but forgotten about them (or at least not remembered they were cross-platform). Scripting something with those ought to be do-able. Indeed I have a feeling that ncget can download a file and delete it on success so if that can cope with not downloading a file it already has, but still delete it, that may be the solution.

I am a little surprised that this isn't a fairly standard requirement with a standard solution though. Keeping recent backups on the host that created them, and a full archive elsewhere, seems fairly "normal".

-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG

On 07/06/13 09:16, Nick Heppleston wrote:

Hi Mark,

You might want to take a look at SyncBack (you're probably looking at the Pro version IMHO): http://www.2brightsparks.com/syncback/sbpro.html. Although I've never used it myself, I heard good thing about it several times in the past.

It Includes the features you are looking for, including FTP & SFTP support and compression of files as they are synced. There are also a number of scripts that can be used to enhance the product (http://www.2brightsparks.com/syncback/scripts/index.html) which don't look particularly hard to engineer if you need something custom (i.e. deleting of files once synced). Beware though, scripts are written in Visual Basic. Sigh.

Just looked at the last link, and it says scripts are typically written in vbscript. VBScript is not the same as Visual Basic, well not quite the same anyway.

And why the sigh. BASIC ROX! :-) BASIC FTW!

Steve

On 07/06/13 08:49, Mark Rogers wrote:

I am a little surprised that this isn't a fairly standard requirement with a standard solution though. Keeping recent backups on the host that created them, and a full archive elsewhere, seems fairly "normal".

Well there's a business opportunity for you then! :-)

Steve