One thing that would improve my security is to restrict ssh access without a password to specific IP addresses, I wonder if it's possible to do this as well as allowing password based access from other systems.
Yes, of course it is, you just add only the systems from which you want to allow passwordless acces to the authorized_keys file.