Summary

The purpose of this article is to provide the mitigation options for the risks associated with exceeding the Guardrails number for excessive test case count in Bamboo.

Please refer Bamboo guard rails for more details. 

Environment

The solution has tested in Bamboo 9.6.4 but it will be applicable for other supported Bamboo version as well. 

Solution

Based on what kind of problems are being highlighted, you can try the below solutions. 

Solution 1

If you are having issues with large number of test cases in the DB and your end goal is to reduce the amount of records in the table, please refer the below solution.

To mitigate the risk associated with excess data in the DB tables related to test case processing please refer How to reduce the size of test_case, test_case_result, test_class, test_class_result,test_error tables in Bamboo

Solution 2

If the problem is related to Bamboo taking a lot of time to process the buildresults and because of which the buildresults are getting into inconsistent state ( refer the risk section of Guardrails for the exact error message ), then the below JVM property can be added to the Bamboo application.

-Dbamboo.max.concurrent.expensive.messages=2

Please refer system property to understand how to add this parameter as a JVM argument.

The above parameter increases the parallelism of what we call "Expensive threads" which include the Build results messaging processing on the Server and JMS messages sent in and out of Agents, by default the value is 1, making it 2 means Bamboo will process these results parallelly.

As the name suggest this thread is expensive in nature, which means it will consume more CPU + Heap memory, if the thread count is increased to 2, please monitor your Bamboo Instance CPU, Heap and Physical memory usage and if the limit is reached please remove this parameter from the JVM argument as it may bring your Bamboo Instance down. 

Solution 3

If Bamboo is taking a lot of time to process a particular buildresult for a specific Job, then the below solution can be applied. 

From Bamboo 10.x.x, there is a new feature introduced to reduce the amount of time taken to process the test results by not tracking history of individual test cases. This change can be done in two ways

1) From Bamboo GUI  - By going to the individual Job configuration and selecting the below check box

2) From Specs

• YAML - bamboo.build.test.skip.history.enabled: 'true'

Default Job:
  key: JOB1
  other:
    statsd-report:
      enabled: 'false'
    job-cache: []
# Plugin com.atlassian.buildeng.bamboo-isolated-docker-plugin:agentSafeguard does not support export to YAML Specs
    all-other-apps:
      custom:
        com.atlassian:
          bamboo.build.test.skip.history.enabled: 'true'
        auto: {}
        buildHangingConfig.enabled: 'false'
  artifact-subscriptions: []

• Java - "bamboo.build.test.skip.history.enabled", "true"

jobs(new Job("Default Job",
                            new BambooKey("JOB1"))
                            .pluginConfigurations(/* Add com.atlassian.buildeng:bamboo-statsd-specs:2.1.4 as dependency to your specs pom.xml to make use of this custom entity.
                                new StatsdReport().enabled(false)  */
                                new AllOtherPluginsConfiguration()
                                    .configuration(new MapBuilder()
                                        .put("custom.plugin.statsd.plan.enabled", "false")
                                        .build()),
                                new AllOtherPluginsConfiguration()
                                    .configuration(new MapBuilder()
                                            .put("custom", new MapBuilder()
                                                .put("com.atlassian", new MapBuilder()
                                                    .put("bamboo.build.test.skip.history.enabled", "true")
                                                    .put("buildeng.vault.config.keys.job", "")
                                                    .build())
                                                .put("auto", new MapBuilder()
                                                    .put("regex", "")
                                                    .put("label", "")
                                                    .build())
                                                .put("buildHangingConfig.enabled", "false")
                                                .put("ncover.path", "")
                                                .put("clover.path", "")
                                                .build())
                                            .build())))

Solution 4

The idea of this solution is to reduce the amount of test cases being proceed in parallel.

1) Consider reducing the number of tests in the plan(s) / Job (s) causing problems, you'll need to connect with your plan owner to check and remove any redundant test cases which may not be in use.

2) Split your job(s) with the highest number of tests into many smaller ones. For Java based application this can be achieved using Maven test groups, for example. Although this may extend the overall duration of the build, it will distribute the processing of test cases over a longer time period and reduce the need for many resources. This will lower the chances of causing performance issues. Using groups you can also skip any group if their processing is not required, refer the below 2 external articles for more details on how to do that

• How to Run Specific TestNG Test Groups Via Maven?
• TestNG – Run Tests and Suites using Maven

3) Stop parsing test results for the plan(s) causing problems. For instance, convert the test results to an HTML report during the build and return it to Bamboo as an artifact.

• • This approach will result in a loss of capability to view the test results within the build result summary page and the ability to quarantine tests.