Confluence Datacenter reports Hazelcast port already in use for separate nodes on different hosts
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Configuring Confluence Datacenter on separate hosts with different IP addresses appears to start correctly. The administrative User Interface shows that the two nodes are successfully joined into a cluster. Nothing appears to be wrong with the nodes, and the cluster appears to be operating normally. There is no visible indication of an error in the front-end of the instance. However, reviewing the logs, there is a serious and concerning ERROR thrown within the logs of your atlassian-confluence.log . The ERROR is a serious concern as the logs display the following messages:
2017-02-10 06:29:33,986 ERROR [localhost-startStop-1] [com.hazelcast.instance.DefaultAddressPicker] log [LOCAL] [dev] [3.4.2-atlassian-14] Port [5801]
is already in use and auto-increment is disabled. Hazelcast cannot start.
java.net.BindException: Address already in useDiagnosis
Environment
This will occur in instance when customizations have been put into place while performing a Custom Build of Confluence. Specifically, if you have been using the WAR build deployment method in Confluence, you may be likely to run into this issue. If you use the standard distribution from Atlassian (Executable,Binary, Zip, or Tar.gz files and not the WAR method) you will not encounter this issue.
Diagnostic Steps
Deploy an Atlassian compiled version of Confluence and then compare the folder layout for the Atlassian-compiled instance of Confluence to the folder layout of your customized build of Confluence. You should note an immediate difference in the folder layout in the Apache Tomcat location and folder location.
Cause
Atlassian has dropped the WAR build file process from its application in favor of the pre-compiled Tomcat distribution layout. This allows us to deliver a consistent build to our clients and increase the supportability of the application as files and folders are placed in the expected locations. This does not prevent you from customizing the application, but does make it easier to deliver a consistent and tested build for our clients.
Resolution
In order to resolve the issue, modify your build process to match the Atlassian standard Apache Tomcat folder locations as found in the pre-compiled Confluence instance. This will resolve the problem and the ERROR message is no longer thrown. The cluster will continue to operate normally without issue.