Richard Kettlewell rjk@terraraq.org.uk wrote:
Why so defensive? If we're discussing the relative performance of FTP and HTTP then concentrating on unusual usage cases tells you nothing of any use; rather, one must look at the mainstream.
It wasn't defensive, it was merely imitating your tone in reply. Maybe it gives you the same feeling that it did me? I thought I already adequately covered the possibility that robots.txt wouldn't be downloaded in the earlier message wording.
As I think you mentioned elsewhere in other words, in the absence of numbers or OP input, appealing to "the mainstream" case is meaningless.