0

The situation is that I've had a VPS created previously. It was all set up, private-public key authentication, root login turned off, password login turned off. Everything was set up.

Then this server gets destroyed and a new server gets spun-off.

So I'm using ssh -v root@new_server_ip_number to log into this newly installed linux instance and this is what I get:

PS C:\Users\roeslermichal> ssh -v root@10.32.81.216
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3
debug1: Reading configuration data C:\\Users\\roeslermichal/.ssh/config
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 10.32.81.216 [10.32.81.216] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ecdsa_sk type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ed25519_sk type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_ed25519_sk-cert type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\roeslermichal/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.32.81.216:22 as 'root'
debug1: load_hostkeys: fopen C:\\Users\\roeslermichal/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:5OrjMYiYdmoRTDgjsmBfOXun/4FpiClOU6L21gBDPSk
debug1: load_hostkeys: fopen C:\\Users\\roeslermichal/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:5OrjMYiYdmoRTDgjsmBfOXun/4FpiClOU6L21gBDPSk.
Please contact your system administrator.
Add correct host key in C:\\Users\\roeslermichal/.ssh/known_hosts to get rid of this message.
Offending RSA key in C:\\Users\\roeslermichal/.ssh/known_hosts:15
Host key for 10.32.81.216 has changed and you have requested strict checking.
Host key verification failed.

What is this SHA256:5OrjMYiYdmoRTDgjsmBfOXun/4FpiClOU6L21gBDPSk. line? What does it mean?
Because clearly this SHA256:5OrjMYiYdmoRTDgjsmBfOXun/4FpiClOU6L21gBDPSk. number/id is not the same number that identifies the linux server in my Windows known_hosts file.

I'm using Windows laptop and PowerShell to log into this server.
There's a C:\Users\roeslermichal\.ssh\known_hosts file on this Windows machine and I've been expecting that some IDs won't match, because the old server got destroyed and the new one got created. I've already deleted the 10.32.81.216 ssh-rsa key and I've replaced it with this 10.32.81.216 ssh-rsa key of the newly installed linux server.
But the ssh-client won't let me in.
This is how my current :\Users\roeslermichal\.ssh\known_hosts file looks like:

10.32.81.216 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGh8fEmrCov7TLbiKgGasUV3fxbrKmh4Ai/RWixt41Fl
10.32.81.216 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr43mfkweJAaHQ4kw88b0y5OShnQl91jR1eoUIcnaMRvBEi3X7McVuA+cB+MWk4Rj9EX2hnq6tyB+26weQX0GXWf95CL/yqX5p39b+j8c43CR9/3gHbU5aV+exGBbj2rEL4JgmQD58fHHEsL1r6EMcpTUgY8JqfG0F52XUJrF7KdpxlW4vtgOaHdqooBMHuMi+bR7LRq/moAHLv3svB5PPhIfSbM5CW/Eke4H4qiAwKCVUjyXxKCoKkYVDyfQur+nBMxJssUHy03385hxV0gKo8WGQKlSNvI3B1vP85ij5zCYViYUfs05lXPkpsUqosGqHDOJhPnVRM4OacMQVkj2e0MKHs/cXA1GneBiY99tPMaEL2qZ0UJoaYcnG0krc0owKE6Ufx+84VVqLG7hJHPnNRI3UrFjG/C7lAzAogz5eDiYoQvkko7mLuwRob27fIB39oH2cbH4a4DCcIDekS0WwCPeA+uwaHrmhKJluqP8r7qvDluWax3cVzDGojD7I6cU=
10.32.81.216 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCMktfR/tBD8GYWRWpo8DsoIPPxos+Rt/C1Is04S0Dglm6UbQqQQUW9m9GfDWHZn3j37ZWPGeUwTcWEojKi70yk=

10.32.81.218 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFBuju3Gav0s6Uj8XFQToa/qU7gxsxvKqtUCctWaC4FC
10.32.81.218 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6OCnBNeCfiLcYo7FAmopNBxWS5No+Locw+dxujELxhXn/zAEEnsMv+fZYP8JT8Jj+bYFX1jVAxBubqaz7swK3GCYkkL4C/dI2p7MV0E0ogznbZEZS0GHU3wA69R7s4F56oR3ZeCIas+gfe3mckB4i9UtZMy2IsGSVl974wletCXfdXxhkyRzHlgovoCnAYu9qOS/X2X2yuUNKKfL3VGQNkAih/Hjqh7Iwi36sLS8+WB/sYOk5cxJfycWewTEl1Wt5fB5bbc7Fu0Wmjn2IpMHspoR6YEw2lK/GuFIFjcVoHJ8+7JAuY9BnUdyuAbHLZ8vgrymcGw/ZP8GIhgRq1nOseAQrOzZMFtcGCS953a+L5gP9shX2ZwF/MS7h8+EYPxMNFZP6DbU++c4ZmOlb0lPkUJDhTnSbOoDZA+bfDl5jBlKtfF2V7n+V9Dwuwwbsp/qJyULIeMAdCrpjPhmKhnQASloZsEN5LLjh2gVN+YM7jACHe6ZyFD4/gpEE6N6MUG8=
10.32.81.218 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBonnCuOeQpc7CSRzbps8sLnPYMphNrfqs9h7Hz5I+Ml8QxPBUnlNw749EzqC29KFtyB8XE2SnbOK/CuUnghj5E=

But I don't really know what are those host keys, because I haven't created any keys on this new linux server. Login approach as root was the first action I've made regarding this newly created linux server. And there are already some host_keys on this server.??? And these are not private-public ssh keys, because I haven't created them yet, so what are those keys, that got generated and identify my new linux server in the Windows known_hosts file.

I've read this thread carefully few times, but don't quite understand the answers given there and why they work. Even more I don't understand why I can't log into my newly created linux server, although I've replaced the old sever rsa host key with a new server rsa host key.

PS C:\Users\roeslermichal> ssh-keyscan -t rsa 10.32.81.216
# 10.32.81.216:22 SSH-2.0-OpenSSH_8.0
10.32.81.216 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr43mfkweJAaHQ4kw88b0y5OShnQl91jR1eoUIcnaMRvBEi3X7McVuA+cB+MWk4Rj9EX2hnq6tyB+26weQX0GXWf95CL/yqX5p39b+j8c43CR9/3gHbU5aV+exGBbj2rEL4JgmQD58fHHEsL1r6EMcpTUgY8JqfG0F52XUJrF7KdpxlW4vtgOaHdqooBMHuMi+bR7LRq/moAHLv3svB5PPhIfSbM5CW/Eke4H4qiAwKCVUjyXxKCoKkYVDyfQur+nBMxJssUHy03385hxV0gKo8WGQKlSNvI3B1vP85ij5zCYViYUfs05lXPkpsUqosGqHDOJhPnVRM4OacMQVkj2e0MKHs/cXA1GneBiY99tPMaEL2qZ0UJoaYcnG0krc0owKE6Ufx+84VVqLG7hJHPnNRI3UrFjG/C7lAzAogz5eDiYoQvkko7mLuwRob27fIB39oH2cbH4a4DCcIDekS0WwCPeA+uwaHrmhKJluqP8r7qvDluWax3cVzDGojD7I6cU=

Although I haven't replaced the 10.32.81.216 ssh-ed25519 nor 10.32.81.216 ecdsa-sha2-nistp256 key. Can this be the reason I'm unable to login?

michal roesler
  • 165
  • 1
    What does it mean? it's the server host key - is not the same number that identifies the linux server in my Windows known_hosts file correct, because it's a key that is generated when the server is first created - simply remove line 15 in C:\\Users\\roeslermichal/.ssh/known_hosts – Jaromanda X May 29 '23 at 12:35
  • 1
    @Jaromanda+ Actually the "what does it mean" value is the fingerprint of the host key, not the actual host key; that's why it says SHA256: at the beginning. However since you (OP) didn't update the ed25519 key in known_hosts, that key is still the one for the old server instance and now wrong and harmful, so yes you should replace it. And you probably should either replace or remove the ecdsa key to avoid possible future problems or confusion, although it's not your current problem. – dave_thompson_085 May 30 '23 at 03:56
  • the fingerprint of the host key correct, but not important to the understanding of the issue – Jaromanda X May 30 '23 at 03:58
  • I forgot to mention it in the question, but I've tampered with a ~/.ssh/known_hosts file a little. I've deleted some lines referring to the other online servers I got. They were earlier in the file, before 10.32.81.216 and 10.32.81.218 entries. That's why there's Offending RSA key in C:\\Users\\roeslermichal/.ssh/known_hosts:15' line 15, while there isn't that many lines in theknown_hosts` contents presented in the question. – michal roesler May 30 '23 at 06:11
  • Although I've accepted the answer, because the deleting server host keys using ssh-keygen -R 10.32.81.216 worked fine (I was able to log in), there is a problem I don't understand. Before I've deleted those host keys, I've tried to REPLACE the 10.32.81.216 ecdsa-sha2-nistp256 key using the ssh-keyscan -t rsa,ecdsa 10.32.81.216 output and manually REPLACED the value in known_hosts file and it didn't work. How can it be, that REPLACING host keys in ~/.ssh/known_hosts file doesn't work? Is this file free to edit using copy/paste method or not? – michal roesler May 30 '23 at 08:46
  • 1
    This file is free to edit. Probably you put in wrong key, or put it in a wrong way. It is a perfectly supported way to build a known_hosts file using the output of ssh-keyscan. – Nikita Kipriyanov May 30 '23 at 08:51
  • @NikitaKipriyanov If what you write is correct, can I perform this kind of experiment? First, delete the whole contents of the ~/.ssh/known_hosts file, but keep the file itself. Second, run the ssh-keyscan -t ecdsa 10.32.81.216 in powershell and copy the output to the clipboard. Third, Paste the clipboard contents to the empty file and save it. Should this work? – michal roesler May 30 '23 at 09:47
  • 2
    There are subtle things like UTF-8 BOM, line endings etc, which Windows editors might do incorrectly (Notepad is notorious for that) and OpenSSH tools might be sensitive to that. Probably this was the problem with your manual editing of the file. I believe shell redirection (like ssh-keyscan host > known_hosts) instead of manual copy-paste can avoid at least some of such problems. – Nikita Kipriyanov May 30 '23 at 11:04

1 Answers1

1

There is a mutual authentication in SSH.

First, the client authenticates the server is indeed the one it wanted to connect. For that, it remembers the public part of the host keypair in the ~/.ssh/known_hosts file. It learns that either during the first connection (this is exactly the part where it asks you to type "yes"), or may learn from DNS if it contains the SSHFP record for the host and the zone is DNSSEC-protected. If client finds out that the server is presenting the wrong key, it will normally refuse to connect claiming an ongoing MitM attack.

This is what SSH host key pair is for. Roughly speaking it is the SSH version of the PKI infrastructure albeit not a CA-based one (or it uses DNSSEC to implement a CA-like trust chain); it's like HTTPS certificate/key pair (with the purpose "web server authentication") and serves the same purpose. It is the asymmetric ("public-private") key pair that belongs to a server.

Secondly, the server authenticates the client is really who it claims to be. For that, it can use username/password pair or whatever complex chat-based authentication, or the asymmetric key pair stored on the server in users ~/.ssh/authorized_keys. This time, the key pair belongs to an user. Again, traditional CA-based PKI has an "analogue", client certificates (with the purpose "web client authentication").

Well, do you see this line in your output, Someone could be eavesdropping on you right now (man-in-the-middle attack)!? This is how it reports the MitM attack it designed to prevent. Probably it is overly cautious, but this is your security.

If you're absolutely sure there is no attack and the new fingerprint is the correct one, simply remove the offending line from your client's known_hosts file. You can follow the
@JaromandaX advice in the comment or you can remove all offending records with something like

ssh-keygen -R 10.32.81.216

Then you'll have to agree by typing literal "yes" to the question about you are sure, or by using a ssh-keyscan utility as described in the other answer. Notice, while this way of building the file avoids the interactive warning about MitM, it is still susceptible to the attack for the same reason, which is also noted in the manual page of ssh-keyscan (and also mentioned many times in comments to the answers under the link).

Nikita Kipriyanov
  • 12,569
  • 2
    No, known_hosts remembers the actual host public key, as a base64 blob plus some metadata. But if the received key differs from the remembered value -- as when the server is recreated -- ssh displays the fingerprint. And SSHFP in DNS (not KEY) can contain that fingerprint, hence the name. Also you can run ssh-keygen -R hostid without -f anything and it automatically uses the same file as ssh does. – dave_thompson_085 May 30 '23 at 03:56
  • So what exactly is this line for The fingerprint for the ED25519 key sent by the remote host is SHA256:5OrjMYiYdmoRTDgjsmBfOXun/4FpiClOU6L21gBDPSk. - how am I supposed to use it? This SHA256 value is not of the same type, as the values in the C:\Users\roeslermichal\.ssh\known_hosts file, so how am I supposed 2 use it if it's not the right fingerprint/id to be copied into the C:\Users\roeslermichal\.ssh\known_hosts file in my Windows laptop? – michal roesler May 30 '23 at 05:45
  • @dave_thompson_085 thank you for corrections! I was too lazy to look for the exact DNS RR type. @michalroesler Ideally you must use some prior secure channel to verify the key presented over the network is correct and there is no ongoing MitM attack. You are supposed to go to that host, find out the fingerprint of the key (by using ssh-keygen) and compare whatever you see over the network and at the terminal. In many cases, however, you "just assume" there is no attack right now (heuristics: you control the network, attack is unlikely and so on) and trust whatever you see on the screen. – Nikita Kipriyanov May 30 '23 at 07:00
  • Coming back this question: https://serverfault.com/questions/321167/add-correct-host-key-in-known-hosts-multiple-ssh-host-keys-per-hostname - the accepted answer - Is there any way to edit this command: $ ssh-keyscan -t rsa 10.32.81.216 in such a way, that it provides a key, I can copy into the ~/.ssh/known_hosts file and log in passing host key identification? It looks like the client doesn't look for this host's rsa key, because I've updated it, and it doesn't matter. The client checks and is offended by the ecdsa-sha2-nistp256 key. – michal roesler May 30 '23 at 07:07
  • 1
    Yes, use not "-t rsa" but the type you are interested in. You can also omit -t parameter, it will receive rsa, ecdsa and ed25519 keys (see man ssh-keyscan for details). However, this is no more secure than just typing "yes" during first connection, because the same implication holds: if there's ongoing MitM you'll see the key from attacker host, not from your target host (there is a warning about this in the man page). – Nikita Kipriyanov May 30 '23 at 07:12