The nodes that show as active but not alive (that is with no heartbeat) or offline can clutter up your cluster information. In Jira 8.9 we improved the way cluster information is displayed however, you still needed to manually remove stale or offline nodes from the cluster.

Removing an offline node 

If you run Jira 7.13 and later you do not need to stop or restart all running nodes to remove an offline one. It takes a few seconds for the running nodes to respond to changes introduced with REST (If you are not familiar with REST API, here are some examples) or using SQL directly on the database. 

Remove offline nodes using REST (for Jira 8.1 and later)

1. Make the following request for Jira to return a list of all nodes:

GET /rest/api/2/cluster/nodes

[
 {
  "nodeId": "node3",
  "state": "ACTIVE",
  "lastStateChangeTimestamp": 1551080754927,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": true
 },{
  "nodeId": "node1",
  "state": "OFFLINE",
  "lastStateChangeTimestamp": 1551080840469,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": false
 }
]

2. If you have nodes showing as OFFLINE, delete each offline node from the cluster
DELETE /rest/api/2/cluster/node/{your_offline_node_Id}

3. Double check that the offline node is now deleted and does not show up

GET /rest/api/2/cluster/nodes

For more on these REST API calls, see Jira REST API. 

Remove offline nodes with database queries

1.  (optional) Double-check the number of offline nodes by running the command: 

select NODE_ID from clusternode where NODE_STATE ='OFFLINE'; 

2. Copy the node_id  of the offline node and run the following command to delete it:

delete from clusternode where node_id=<id from first query>;

Removing a stale node with no heartbeat

Nodes might show up active but in fact, have no heartbeat if they have been killed abruptly. We advise stopping Jira gracefully to avoid that situation. 

You can either remove a stale node using REST API (available for Jira 8.1 and later) or do it manually. 

Remove stale nodes using REST (for Jira 8.1 and later)

1. Make the following request for Jira to return a list of all nodes:
   GET /rest/api/2/cluster/nodes

   Example response...

   [
    {
     "nodeId": "node3",
     "state": "ACTIVE",
     "lastStateChangeTimestamp": 1551080754927,
     "ip": "localhost",
     "cacheListenerPort": 40001,
     "nodeBuildNumber": 801000,
     "nodeVersion": "8.1.0-SNAPSHOT",
     "alive": true
    },{
     "nodeId": "node1",
     "state": "ACTIVE",
     "lastStateChangeTimestamp": 1551080840469,
     "ip": "localhost",
     "cacheListenerPort": 40001,
     "nodeBuildNumber": 801000,
     "nodeVersion": "8.1.0-SNAPSHOT",
     "alive": false
    }
   ]

2. Copy the nodeId's of the nodes that are "ACTIVE" and "alive"=false. 
3. Ping the nodes to make sure they are not running in your environment and will not back to the cluster in the future.
4. Make the following request to move the stale node to the offline state
   PUT /rest/api/2/cluster/node/{your_stale_node_Id}/offline
5. Run GET /rest/api/2/cluster/nodes to make sure the node is offline. 
6. Delete the stale node from the cluster:
   DELETE /rest/api/2/cluster/node/{your_stale_node_Id}
7. Double check that the stale node is now deleted and does not show up:
   GET /rest/api/2/cluster/nodes
   
   For more on these REST API calls, see Jira REST API. 

Remove old nodes with database queries

1. Check the database tables and find all rows related to old nodes:

   select * from clusternode;
   select * from clusternodeheartbeat;

2. Copy the node_id of the stale node and run the command:

   delete from clusternode where node_id = '<node_id>';
   delete from clusternodeheartbeat where node_id = '<node_id>';

3. Clean old replication records:

   // validate if this is necessary 
   select count(id) from replicatedindexoperation where node_id = '<node_id>';
   // delete
   delete from replicatedindexoperation where node_id = '<node_id>';

   
   Related tickets

• Jira 8.10 improvement -  JRASERVER-42916 - Getting issue details... STATUS