I have just received: Warning: the ECDSA host key for '...' differs from the key for the IP address '...'
.. when trying to SSH to a web server I manage.
I expect these kind of warnings if I've changed IPs, or reinstalled SSH, or reinstalled the OS, but I'm not aware of having done anything that could provoke this warning.
Googling just gives me endless ways to fix the problem by removing the cached key, but I'm finding precious little advice on what to do if the warning has no obvious cause.
Suggestions?
On 6 Dec 2019, at 12:01, Mark Rogers <mark@more-solutions.co.uk mailto:mark@more-solutions.co.uk> wrote:
I have just received: Warning: the ECDSA host key for '...' differs from the key for the IP address '...'
.. when trying to SSH to a web server I manage.
I expect these kind of warnings if I've changed IPs, or reinstalled SSH, or reinstalled the OS, but I'm not aware of having done anything that could provoke this warning.
Is it changing IP addresses itself (behind a load balancer, or muti-valued dns records, or does it get its IP via DHCP and got a different one, or is some other misconfigured remote system claiming your IP address, or whatever)? Is the machine doing unattended updates that may have re-generated the key? Is the machine regenerating the key from some cloud-init mechanism which has an issue that causes it to regenerate the key when it shouldn’t?
Have a look at "ls -l /etc/ssh/*” to see what got changed when, and dig through the system logs from that time, and compare against the last boot time.
Googling just gives me endless ways to fix the problem by removing the cached key, but I'm finding precious little advice on what to do if the warning has no obvious cause.
Suggestions?
The host key checking is designed to alert you to MITM attacks, which would be anywhere between you or your network. Are you connecting from your usual place where it worked before? Try connecting from elsewhere (some remote machine on some other network) to exclude some of those intercept possibilities.
On the host, check for signs of a compromise: miners or other malware running, unexpected files, unexpected logins, unexpected outbound connections, unexpected content being served from your web server etc. If you have console access, you might use that rather in reference to ssh
— Martijn
On Fri, 6 Dec 2019 at 13:00, Martijn Koster mak-alug@greenhills.co.uk wrote:
Is it changing IP addresses itself (behind a load balancer, or muti-valued dns records, or does it get its IP via DHCP and got a different one, or is some other misconfigured remote system claiming your IP address, or whatever)?
Static IP (hosted VPS at DigitalOcean).
Is the machine doing unattended updates that may have re-generated the key?
Not that I'm aware of.
Is the machine regenerating the key from some cloud-init mechanism which has an issue that causes it to regenerate the key when it shouldn’t?
Not that I'm aware of.
Have a look at "ls -l /etc/ssh/*” to see what got changed when, and dig through the system logs from that time, and compare against the last boot time.
That was my first thought and nothing has changed recently (I connect at least once a month, usually more; the newest file was Feb this year.
I only have a week's worth of syslog and it only has two references to ssh ("systemd[xxx]: Listening on GnuPG cryptographic agent (ssh-agent emulation)." from the twice I've connected today.)
The host key checking is designed to alert you to MITM attacks, which would be anywhere between you or your network. Are you connecting from your usual place where it worked before?
Yes
Try connecting from elsewhere (some remote machine on some other network) to exclude some of those intercept possibilities.
I just tried from home, and that works fine. But that could be that any cached key has been added more recently (it's a fairly new laptop) so I'm not sure what that really tells me?
I have SSH'd to several other servers (also DigitalOcean) without any key issues, so if there's a MITM attack going on it's targeted at that one server (a Wordpress server so it's probably one of the most "visible" of the servers I have much to do with).
On the host, check for signs of a compromise: miners or other malware running, unexpected files, unexpected logins, unexpected outbound connections, unexpected content being served from your web server etc.
I'm not really sure what to look for. The Wordpress code has Wordfence installed which is pretty good at finding unexpected changes in the site's code, and I ran a scan on that this morning without finding anything. "last" (looking at /var/log/wtmp) only has stuff from today, wtmp.1 from November 8th (quite possibly the last time I logged in). "lastb" has a nice long list of failed attempts for various users but I'm not sure I can draw much from that.
last does show my office IP as the source of the last login but would a MITM attack mask that?
If you have console access, you might use that rather in reference to ssh
Hmm, good point, until you said that I'd forgotten I do have console access.
On 6 Dec 2019, at 13:45, Mark Rogers mark@more-solutions.co.uk wrote:
On Fri, 6 Dec 2019 at 13:00, Martijn Koster mak-alug@greenhills.co.uk wrote:
Is it changing IP addresses itself (behind a load balancer, or muti-valued dns records, or does it get its IP via DHCP and got a different one, or is some other misconfigured remote system claiming your IP address, or whatever)?
Static IP (hosted VPS at DigitalOcean).
https://www.digitalocean.com/community/questions/why-my-droples-s-ssh-key-ch... https://www.digitalocean.com/community/questions/why-my-droples-s-ssh-key-changed?comment=58664 Seems to suggest this can happen, though it doesn’t go into great detail why.
Have a look at "ls -l /etc/ssh/*” to see what got changed when, and dig through the system logs from that time, and compare against the last boot time.
That was my first thought and nothing has changed recently (I connect at least once a month, usually more; the newest file was Feb this year.
Hm, that is odd then. If cloud-init regenerated the key, that’s where I’d expect to see it.
Try connecting from elsewhere (some remote machine on some other network) to exclude some of those intercept possibilities.
I just tried from home, and that works fine. But that could be that any cached key has been added more recently (it's a fairly new laptop) so I'm not sure what that really tells me?
Right, you’d have to copy the appropriate line from your .ssh/known_hosts to your new host (after removing the one that just got newly added there).
Which reminds me — you haven’t just restored your .ssh directory from backup or something I assume?
On the host, check for signs of a compromise: miners or other malware running, unexpected files, unexpected logins, unexpected outbound connections, unexpected content being served from your web server etc.
I'm not really sure what to look for.
Most recent compromises I’ve seen have been miners, which show up on “top” and “ps". But I kinda doubt that a compromise is behind it.
— Martijn
On Fri, 6 Dec 2019 at 15:14, Martijn Koster mak-alug@greenhills.co.uk wrote:
https://www.digitalocean.com/community/questions/why-my-droples-s-ssh-key-ch... Seems to suggest this can happen, though it doesn’t go into great detail why.
Interesting, thanks. Never thought to narrow my Google search down to a specific host.
Which reminds me — you haven’t just restored your .ssh directory from backup or something I assume?
No. But it is possible that it's been longer than I think since I last connected from my office PC; I tend to do the updates out of hours and that means I'm usually at home. (I can't think of any good reason why they'd have changed since first install, but the longer it's been the more likely there was a decent reason.)
Most recent compromises I’ve seen have been miners, which show up on “top” and “ps".
Cool, checked those. Nothing obvious, CPU not being pushed hard. Also disk space not looking any more utilised than I'd expect.
But I kinda doubt that a compromise is behind it.
Ditto (it was a curiosity unless someone gave me reason to panic), but it's been an interesting investigation.
Blindly ignoring a warning wasn't a great plan, but I think I've done enough now to be happy it's nothing untoward (and it's not a critical server, it has no confidential info on it).
Thank you for your help. It really is appreciated.
Mark
-
Mark Rogers -
Martijn Koster