Richard Kettlewell rjk@terraraq.org.uk wrote:
MJ Ray markj@cloaked.freeserve.co.uk writes:
Surely that delay must not drop any data from the socket, else the server is in error. [...]
It's nothing to do with delays. The problem arises when something on the server side uses stdio or some other buffered I/O library.
If they're connecting buffered I/O directly to the network socket and collecting more data than they need from it, then surely they are in error and need fixing.
Sendmail and INN are real examples of this.
If that's what they did, I'd probably write that last word with a different letter order.
[...]
If you just want to download a single file I'm not sure I see why one would look at robots.txt.
If you're doing it automatically, you're supposed to, IIRC.
I must have missed the bit where we said we were only talking about automated requests. (I bet the majority of downloads are manual...)
I must have missed the bit where I didn't say "possible" for the download of robots.txt. Whether it's one file or many is irrelevant. Whether it's manual or automatic is relevant. That's what I said.
[...]
The headers on my home page come to under 500 bytes (including three lines of rubbish from proxies).
Congratulations. Do you have the "light headers award" PNG?