"REMOTE HOST IDENTIFICATION HAS CHANGED" is reported each time the server hosting Bitbucket is restarted
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Users get the REMOTE HOST IDENTIFICATION HAS CHANGED! warning message when performing git operations whenever the server hosting Bitbucket is restarted.
git clone ssh://git@localhost:7999/proj1/repo1.git
Cloning into 'repo1'...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:LWdsVRrm2RZ41Ft2CxwAyiNR2ouPt99wIyJwIfXWNL8.
Please contact your system administrator.
Add correct host key in /Users/bbuser/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/bbuser/.ssh/known_hosts:41
RSA host key for [localhost]:7999 has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.This can occur if the user used to start up Bitbucket has no write permissions on the $BITBUCKET_HOME/shared/config/ssh-server-keys.pem file, which contains the SSH private key for Bitbucket.
If the machine that hosts Bitbucket was not restarted but the warning message is still encountered, please see the KB, REMOTE HOST IDENTIFICATION HAS CHANGED when accessing Bitbucket Server git repo over ssh, which provides diagnosis and resolution steps if the warning is received in the following scenarios:
- while attempting to access the machine Bitbucket is hosted on via
ssh - while accessing Bitbucket hosted repositories over ssh (clone, push, fetch)
Diagnosis
Scenario 1: The following entries showing AccessDeniedExceptions are logged in $BITBUCKET_HOME/log/atlassian-bitbucket.log file:
2021-03-05 02:02:01,254 WARN [sshd-SshServer[1b57567](port=7999)-nio2-thread-1] c.a.b.i.s.s.DefaultHostKeyPairProvider resolveKeyPair(/var/atlassian/application-data/bitbucket/shared/config/ssh-server-keys.pem) Failed (AccessDeniedException) to load: /var/atlassian/application-data/bitbucket/shared/config/ssh-server-keys.pem
...
2021-03-05 02:02:01,274 INFO [sshd-SshServer[1b57567](port=7999)-nio2-thread-1] c.a.b.i.s.s.DefaultHostKeyPairProvider generateKeyPair(RSA) generating host key - size=2048
...
2021-03-05 02:02:01,715 WARN [sshd-SshServer[1b57567](port=7999)-nio2-thread-1] c.a.b.i.s.s.DefaultHostKeyPairProvider writeKeyPair(/var/atlassian/application-data/bitbucket/shared/config/ssh-server-keys.pem) failed (AccessDeniedException) to write key /var/atlassian/application-data/bitbucket/shared/config/ssh-server-keys.pem: {}
...
2021-03-05 02:02:01,731 WARN [sshd-SshServer[1b57567](port=7999)-nio2-thread-1] c.a.b.i.s.s.DefaultHostKeyPairProvider Could not restrict file permissions on key /var/atlassian/application-data/bitbucket/shared/config/ssh-server-keys.pem
java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/shared/config/ssh-server-keys.pem: Operation not permitted
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238)
at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260)
at java.nio.file.Files.setPosixFilePermissions(Files.java:2045)
at com.atlassian.bitbucket.util.MoreFiles.setPermissions(MoreFiles.java:377)
at com.atlassian.bitbucket.internal.ssh.server.DefaultHostKeyPairProvider.writeKeyPair(DefaultHostKeyPairProvider.java:121)
at org.apache.sshd.server.keyprovider.AbstractGeneratorHostKeyProvider.resolveKeyPairs(AbstractGeneratorHostKeyProvider.java:214)
at org.apache.sshd.server.keyprovider.AbstractGeneratorHostKeyProvider.loadKeys(AbstractGeneratorHostKeyProvider.java:139)
at org.apache.sshd.server.keyprovider.AbstractGeneratorHostKeyProvider.loadKeys(AbstractGeneratorHostKeyProvider.java:60)
at org.apache.sshd.common.keyprovider.KeyPairProvider.getKeyTypes(KeyPairProvider.java:131)
at org.apache.sshd.server.session.AbstractServerSession.resolveAvailableSignaturesProposal(AbstractServerSession.java:372)
at org.apache.sshd.common.session.helpers.AbstractSession.resolveAvailableSignaturesProposal(AbstractSession.java:2173)
at org.apache.sshd.common.session.helpers.AbstractSession.sendKexInit(AbstractSession.java:2098)
at org.apache.sshd.server.session.AbstractServerSession.readIdentification(AbstractServerSession.java:483)
at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:342)
at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63)
at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:368)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:346)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:343)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:555)
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:277)
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:298)
at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:398)
at org.apache.sshd.common.io.nio2.Nio2Session.doReadCycle(Nio2Session.java:338)
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:330)
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:326)
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:322)
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:318)
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:314)
at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:311)
at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:266)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
... 1 frame trimmed
Scenario 2: In case, there is no relevant error message in the logs - it is good to validate following points:
- Whether Port forwarding is enabled for SSH communication.
We can get the clue either from git command (git clone ssh://git@localhost/proj1/repo1.git) where port is not defined (so default port as 22) as well as from git clone command output with port details (whether communication port is 22). If yes, need to further check whether Proxy Server or Load Balancer would have the same fingerprint as direct to the node on 7999 by using ssh-keyscan command.
From the Proxy Server or Load Balancer machine:ssh-keyscan -p 22 <ip address of Load Balancer node> | ssh-keygen -lf -From all the Bitbucket nodes:
ssh-keyscan -p 22 <ip address of Bitbucket node> | ssh-keygen -lf - ssh-keyscan -p 7999 <ip address of Bitbucket node> | ssh-keygen -lf -
Cause
Scenario 1: The Bitbucket user (e.g. atlbitbucket) does not have write permissions to the $BITBUCKET_HOME/shared/config/ssh-server-keys.pem file, which contains the SSH private key for Bitbucket.
This can happen if another user, such as root owns the file or the parent directory.
Scenario 2: The possible cause of this scenario is usage of IPTABLES (in which rules are defined for ports) where configuration has been missed or got cleaned somehow which need to be corrected.
Solution
Scenario 1: Ensure that the $BITBUCKET_HOME/shared/config/ssh-server-keys.pem file is owned by the user that starts up Bitbucket.
Scenario 2: Ensure that IPTABLES entries are configured correctly or need to be corrected.