How to upgrade Remote Mesh Nodes
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When upgrading Bitbucket, the version of the Remote Mesh nodes may also need to be upgraded to maintain compatibility. Relevant Bitbucket-Remote Mesh version compatibility chart is given on the page Bitbucket Mesh compatibility matrix.
Environment
Bitbucket Data Center 8.x and newer
Remote Mesh 1.0.0 and newer
At least one of the repositories has been migrated to Mesh nodes.
General procedure for upgrading Remote Mesh nodes
When upgrading Remote Mesh nodes, it is important to upgrade them one node node at a time, following this procedure:
Drain and disable the node from the Bitbucket Web UI.
Stop the Remote Mesh process.
Upgrade the node.
Detailed description of this step is given in the section “Details on upgrading the Remote Mesh node“ below.Start the Remote Mesh process.
Enable the node from the Bitbucket Web UI.
Before proceeding to the next node, make sure all the replicas have become consistent!
Use the page How to check and restore replica consistency after the mesh node restart in Bitbucket Mesh as a guide on how to check the replica consistency.
- Repeat the procedure for every node.
How to minimize downtime when upgrading Remote Mesh nodes
The most important thing to consider when upgrading is to maintain compatibility between Bitbucket version and Remote Mesh version. The compatibility chart is given on the page Bitbucket Mesh compatibility matrix.
Here are other important notes:
When upgrading Remote Mesh nodes, upgrade to the new bugfix version of Remote Mesh, one node at a time.
Make sure all replicas have become consistent between proceeding with the upgrade of the next node.If you use a Remote Mesh version that is too old with a Bitbucket that requires a newer version, the setup will break!
Bitbucket will work with the Remote Mesh with the bugfix versions higher than the version required by the Bitbucket Mesh compatibility matrix.
For example, Bitbucket 8.19.0 requires Remote Mesh 2.5.0, but it will also work with Remote Mesh 2.5.4.When upgrading both Bitbucket and Remote Mesh, upgrade first all Remote Mesh nodes, then upgrade main Bitbucket application.
Follow standard Bitbucket upgrade documents for main Bitbucket application.
Details on upgrading the Remote Mesh node
n this example, we are upgrading from Mesh v2.0.8 to Mesh v2.3.0. The Mesh installation directory is /opt/atlassian/mesh/current where current is a symlink to the existing version:
# ls -l
/opt/atlassian/mesh/current -> /opt/atlassian/mesh/atlassian-bitbucket-mesh-2.0.8Download the new version
Download the package for the new version from the Bitbucket Mesh Downloads page and place it in the installation directory.
We can use this opportunity to update the scripts to match the configuration currently running:
| File | Use |
|---|---|
set-jmx-opts.sh | Add JMX settings |
set-jre-home.sh | Define JRE_HOME or JAVA_HOME |
set-mesh-home.sh | Set the MESH_HOME directory |
set-mesh-user.sh | Define the process user (aka MESH_USER ) |
_start-mesh.sh | Configure the JVM heap or set JVM_SUPPORT_RECOMMENDED_ARGS |
Update the Symlink
Using a symlink allows us to stage the files and quickly switch between versions to minimize downtime. We can update the symlink using the following command:
# ln -sfnv /opt/atlassian/mesh/atlassian-bitbucket-mesh-2.3.0 /opt/atlassian/mesh/current
...
# ls -l
/opt/atlassian/mesh/current -> /opt/atlassian/mesh/atlassian-bitbucket-mesh-2.3.0Restart Mesh
With the symlink updated, we can use the stop and start scripts to restart Mesh with the updated version. Check log files to verify the version was updated:
# cat /var/atlassian/application-data/mesh/log/atlassian-mesh.log
...
INFO [main] - c.a.b.mesh.app.BuildInfoLogger Starting Mesh 2.3.0
...H2 database update may be necessary
Remote Mesh upgrade may require update of mesh H2 database. Indication that H2 database upgrade is needed is that Mesh node does not start and this message is written to the atlassian-mesh.log file:
***************************
APPLICATION FAILED TO START
***************************
Description:
The storage format for the internal database that you are using has changed and requires manual migration.
Action:
Please perform the manual migration of the database. See https://confluence.atlassian.com/bitbucketserver088/migrate-h2-database-1216583295.html.
In this case, follow the procedure described in the “Solution” section of the page Bitbucket Mesh Sidecar H2 database migration.
When is it safe to add or remove nodes
You should not remove nodes if that would leave the system with less nodes than the replication factor (default and minimum 3). In fact, the system should block this action entirely!
Assuming you have more than 3 nodes then the best time to add or remove them is outside business hours. This is to reduce the load on existing mesh nodes and lessen the chances of a replica being unable to catch up due to concurrent commits being made during the partition migration.
On large, busy repositories it is possible that the repair job is unable to catch up to the latest changes. This can happen if the frequency of repository updates is high, and mesh repair process takes longer time so it can’t catch up. Currently, there is no other solution than waiting for the repository to be quiet long enough so repair process can finish.