Summary

In Bamboo, ActiveMQ is responsible for managing JMS API messaging, encompassing logs, build queuing, and results. This guide outlines best practices for optimizing and tuning ActiveMQ usage to enhance workload efficiency.

Environment

• Bamboo Data Center
• Large build environments
• Network volumes

Diagnosis

The Bamboo server and its agents utilize JMS for communication via an ActiveMQ messaging system operating within the same JVM as Bamboo, sharing resources with the main application. The ActiveMQ broker facilitates:

• Agent Registration
• Build Queuing
• Transfer of build logs (when live logs are enabled)

During excessive ActiveMQ transaction events, Bamboo may exhibit the following behaviors:

• Significant build delays or lost build triggers, whether manual or automatic via commits
• Backend and UI instability due to excessive memory usage
• Unexpected ActiveMQ errors and system crashes
• Difficulties with agent communication, including "connection refused" or "connection reset" errors in logs

Cause

Bamboo may present several reasons for unstable JMS communication through ActiveMQ, including:

Slow or unstable network connection

Communication between Bamboo and an Agent requires minimal latency. Delays in communication may compromise message processing via the broker, impacting the application's behavior and necessitating retransmission due to expired messages and general application failure.

Slow or high-latency disk

ActiveMQ is known for its requirement for low-latency and fast disk access. Bamboo uses a storage-based persistence driver that writes ActiveMQ's KahaDB to a disk partition. If the filesystem or network volume does not provide the necessary throughput, ActiveMQ may miss messages and report inconsistent results.

Non-optimized hardware resources

As Bamboo runs on a JVM, not enough memory (heap) or CPUs may not be available to process transactions in a timely manner. It is crucial to monitor CPU usage and validate if the heap size is adequate for your workload. We advise customers to enable Monitoring of Memory Usage and Garbage Collection in Bamboo. If your Bamboo instance is running out of memory or CPU, please contact our Support for proper guidance.

Solution

Align your Bamboo workloads with the Bamboo build log sizes guardrail limits

Bamboo has a certified upper limit for build log sizes that are transmitted over ActiveMQ. Please refer to the following article for more information:

• Guardrails for Bamboo build log sizes

Use high-speed, low-latency storage

ActiveMQ, like any messaging broker system, relies heavily on storage speed and latency. When the number of requests increases, ActiveMQ may require exclusive locks on specific parts of its KahaDB persistence adapter, necessitating quick responses to ensure further transactions can be completed within the expected time. For enterprise customers using Bamboo on a shared network volume such as NFS, there is a higher chance of latency issues due to the nature of storage communication on such systems. Therefore, guarantee that the <bamboo-shared-home> volume is hosted on high-performance storage to ensure low latency and high-speed transactions.

Set the bamboo.jms.store.directory System Property

In situations where it is not feasible to point the entire <bamboo-shared-home> folder to a high-speed storage volume with low latency, you can set a custom system property called bamboo.jms.store.directory to specify the high-speed volume for ActiveMQ. In a non-clustered Bamboo setup, this property can point to a local disk or even a RAM filesystem. In a clustered environment, ensure that the remote jms-store volume is mounted as a shared/replicated resource allowing concurrent access by additional Bamboo secondary nodes.

It is also possible to distribute several parts of the Bamboo application into distinct mount points to alleviate storage concurrency. For information on which locations can benefit from this, please refer to the documentation below.

• Locating important directories and files

If you decide against a local volume just for ActiveMQ, consider other ways to alleviate load on the shared storage, such as having different volumes for Java temp, artifacts, and Git cache, etc.

Choose the right storage tiering strategy

Whenever log entries from builds are requested, Bamboo must read the entire logs to calculate the number of lines. Depending on the type of storage used by Bamboo, this may trigger storage tiering actions by moving older logs from lower tiers to higher tiers as they have been archived earlier. Consequently, this can temporarily slow down the request until the files are fully managed, especially if the build logs are large. And, in case the ActiveMQ partition is hosted on that same storage, it may be impacted by that action. Therefore, it is recommended to analyze access patterns of log files by Bamboo users and developers before implementing any storage auto-tiering strategies.

Storage auto-tiering can is implemented by multiple storage vendors. We recommend consulting with your local storage team to verify if any auto-tiering policies are in use to discuss how the old log files can be affected by such a strategy.

Update to the latest Bamboo release

The latest Bamboo releases include recent bug fixes and performance enhancements to address the majority of ActiveMQ performance issues. Whenever you encounter issues related to ActiveMQ, upgrade to the latest version (or LTS release) and verify if the issue is resolved.

• Bamboo Release Notes

Review and apply recommended broker timeout settings for ActiveMQ

Starting with Bamboo 9.4, new default ActiveMQ broker timeouts were implemented. These settings are configured by default on new installations but may not be present if your Bamboo instance was installed with an earlier version, even if you are running a version later than 9.4.

Review the properties listed in the following feature request and adjust your Bamboo settings accordingly:

• BAM-20858: Reduce the default Inactive timeout of remote agents

Monitor the KahaDB logs location for unusual file count increase

You can run a script such as the one below to monitor if KahaDB files are increasing over time. Though this does not indicate the source of the excessive number of message, it may be used to match a specific period of the day which can be used for later investigation. If that file count increases and remains elevated it means Bamboo is struggling to keep on top of it.

# Set this to the full path to your Bamboo Home directory
#!/usr/bin/env bash
 
# Set this to the full path to your Bamboo Home directory
BAMBOO_HOME=/storage/data/bamboo
 
BASE_DIR=$(dirname "$0")
 
while true
do
  date >> $BASE_DIR/jmslog.txt
  du -sh $BAMBOO_HOME/jms-store >> $BASE_DIR/jmslog.txt
  FILE_COUNT=`find $BAMBOO_HOME/jms-store -type f | wc -l`
  echo -e "File count: $FILE_COUNT\n" >> $BASE_DIR/jmslog.txt
  sleep 10
done

You can also locate and identify matching Plans that are producing large build logs by checking the instructions outlined on this KB article:

• Disk space hotspots and cleanup best practices in Bamboo - Build results and logs

Use third-party tools to monitor ActiveMQ

It is possible to use third-party tools to follow Bamboo's ActiveMQ queue status, please check the following page for some examples and suggested solutions.

• How can I monitor ActiveMQ Classic?