Richard Kettlewell rjk@terraraq.org.uk wrote:
the biggest danger spots for pipelining, as sometimes authentication involves handing off the connection to another process for a bit.
Surely that delay must not drop any data from the socket, else the server is in error. I've done some POP3 pipelining and the only problem has been some non-compliant servers that try to rely on an alternating dialogue.
[...]
If you just want to download a single file I'm not sure I see why one would look at robots.txt.
If you're doing it automatically, you're supposed to, IIRC.
If you want many files then I'd expect the cost of the control connection to rapidly beat a robots.txt lookup.
You're quite possibly correct there, but you have to do it.
The headers probably fit in a single packet and don't require any back-and-forth between the client and the server, so aren't going to contribute much to the time taken.
I'm not so sure. Headers seem to be getting more and more verbose. Are they just adding things back in to emulate FTP's richness?
Perhaps we should stop speculating and start profiling l-)
Oh, the real world will screw up all the numbers, for sure ;-)