Not strictly a Linux question, but it could be just as relevant on a Linux system.
I have a client with a SCO SYSV Unix Box running a legacy Accounting system. It is a korn shell.
They have a Windows based Print Workflow system (called StreamServe I believe) that takes print jobs from a spool folder and prints or emails Credit account statements generated by the Accounting package.
Unfortunately the way the queue works is that once a job has been processed by streamserve it is deleted from the queue folder on the Unix box by the streamserve application. The problem is that it is very difficult and time consuming to regenerate the reporting run if something fails upstream of the Streamserve box resulting in lost prints. (this is what happened the other day)
I can't tell the streamserve application to not delete the processed jobs as it simply prints whatever is in the queue every time it polls.
The Streamserve box looks for a particular file extension before processing the file. I can change the file extension that the Streamserve looks for.
So my plan is to change this extension and then have a script that looks for jobs (of the original extension) in the queue and copies them to a archive area before renaming them to a pattern that the (modified) streamserve application is looking for. It is important that I don't introduce any delay of more than a minute or so into the workflow but the print files rarely exceed a couple of MB so that shouldn't be a problem.
I could just knock together a script that does this from Cron but in order to not introduce too much of a delay it would have to run about once a minute.
This box is quite heavily loaded sometimes and it is of limited performance anyway. So my question is-
What is the most efficient way to "watch" a folder for a particular file pattern, if I have a looping "if exist" script is this likely to consume lots of resources as surely it would be continuously polling the disk to check for that file.
Is it good practise to do this (perhaps with some sort of introduced delay in the loop) or is there a better/cleaner/more efficient way.
Regards Wayne
Wayne Stallwood writes:
What is the most efficient way to "watch" a folder for a particular file pattern, if I have a looping "if exist" script is this likely to consume lots of resources as surely it would be continuously polling the disk to check for that file.
You could use `sleep' to suspend your script for a few seconds within the `if exists' loop.
I had to do something similar in the past and my preferred solution was to use tclsh as I was able to time how long the file copying took and then sleep (`after' in Tcl) for an amount of time that made the copying happen at regular intervals.
(Please: no rants about my choice of scripting language!)
HTH.
..Adrian
Wayne Stallwood ALUGlist@digimatic.plus.com writes:
What is the most efficient way to "watch" a folder for a particular file pattern, if I have a looping "if exist" script is this likely to consume lots of resources as surely it would be continuously polling the disk to check for that file.
Every time a file is created in a directory the directory's modification time will be updated. So the answer is to poll that, and only do a full scan when it changes.
So the logic would be:
latest_time = mtime of directory scan for pattern while true: new_time = mtime of directory if current_time != new_time: sleep 1 second scan for pattern latest_time = new_time else: sleep 1 second (or 10s or 1m or whatever)
The 1 second sleep is necessary for fully reliable operation.
It will hit the physical disk less than you imagine, as usually the directory's inode will be cached.
This would happen in a single long-running process, rather than from cron.
-
Adrian F. Clark -
Richard Kettlewell -
Wayne Stallwood