Confluence Data Center fails to start with Hazelcast phone home error
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
When running Confluence Data Center on Amazon Web Services, with cluster discovery mode configured to "AWS", Confluence may fail to start.
The following appears in the catalina.out
2019-06-21 00:43:33,520 ERROR [localhost-startStop-1] [com.hazelcast.instance.Node] log [x.x.x.x]:5801 [Cluster] [3.8.6] Could not join cluster in 300000 ms. Shutting down now!
2019-06-21 00:43:33,524 WARN [localhost-startStop-1] [com.hazelcast.instance.Node] log [x.x.x.x]:5801 [Cluster] [3.8.6] Terminating forcefully...
2019-06-21 00:43:33,585 WARN [localhost-startStop-1] [com.hazelcast.util.PhoneHome] log [x.x.x.x]:5801 [Cluster] [3.8.6] Could not schedule phone home! Most probably Hazelcast is failed to start.
2019-06-21 00:43:33,591 ERROR [localhost-startStop-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
Node failed to start!
java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:135)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:218)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:176)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:126)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:337)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:309)
at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:407)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:860)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:75)
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:190)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:36)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:133)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:64)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4643)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5109)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1429)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)This error is usually preceded by a java.net.ConnectException.
Diagnosis
Environment
Amazon Web Services hosted environment
Diagnostic Steps
Configure Confluence Data Center on Amazon Web Services with AWS cluster discovery configured
When starting Confluence, the above error may appear in the logs, and Confluence will fail to start.
Cause
Due to the architecture of AWS' network, the multicast discovery technique cannot be used. AWS runs a regional registration endpoint that allows Hazelcast instances to register, and discover other registered nodes from within the same security group.
This error occurs when Hazelcast is unable to reach this regional register.
Possible causes for this issue are
- Outbound connections are blocked by security rule or firewalls
- The AWS Host Header is incorrectly configured
- The AWS Region Name is incorrectly configured
Resolution
Ensure that outbound connectivity is possible from the AWS instance running Confluence, and confirm that the AWS Host Header (if configured) and AWS Region Name fields are correctly configured in confluence.cfg.xml.