Progress! With a <Proxy> block to Allow from all, I get 500 rather than 403 ;-) It looks like an Apache issue still, although I can't be sure. I saw a neat trick somewhere about trapping errors in your proxy (so that you can serve a different page if your backend application server is down); maybe I can find the instructions again and use that to work out where the fault is.
Curiously, I have a set of other virtual servers that proxy heavy requests to another instance of Apache and I don't recall having declared a <Proxy> with Allow for them. They use mod_rewrite instead of ProxyPass, but I get the same results either way when passing requests to Usermin.
I'll try some more tweaking; thanks for the help.
I believe you need to add something like
<Proxy *> Order deny,allow Allow from all
</Proxy>
To allow access to the proxy.
I've just been reliably informed (by Brett on IRC) that <Proxy *> is not a good idea and that you should bind it to a particular ip:port
So..
<Proxy ip:port> Order deny,allow Allow from all </Proxy>