Summary

In some scenarios when a database backup is being taken, all the local agents can become offline.

Environment

Bamboo 7.2.5 or older versions

Diagnosis

The following message can be found in the atlassian-bamboo.log file:

[DefaultBuildAgent] 9 attempts at building without a successful run. The agent will stop after reaching 10 exceptions.
[DefaultBuildAgent] Maximum number of errors has been reached (10). Agent will now stop.

Cause

During the database backup process, the database becomes offline, but the local agents keep trying to connect to the database in order to check if there are any new builds on the queue to be taken. This process requires a database connection to be established, therefore once the agent verifies the database if offline it throws an exception in the logs.

The local agents go offline because Bamboo is designed to have a maximum number of errors that a local agent can hit before going offline:

private static final int MAX_ERROR_COUNT = 10;

if (errorCount.intValue() >= MAX_ERROR_COUNT) {
    log.fatal("Maximum number of errors has been reached (" + MAX_ERROR_COUNT + "). Agent will now stop.");
    stop();
} else {
    log.warn(errorCnt + " attempts at building without a successful run. The agent will stop after reaching " + MAX_ERROR_COUNT + " exceptions.");
}

Solution

1. Restart Bamboo so all local agents will become available again
2. Restart agents manually by going to Bamboo administration panel >> Agents >> Local agent >> Restart button

 Currently, there are three possible actions that can be taken to void this to happen:

• If possible, stop Bamboo before running any database backup process.
• Instead of using a database backup directly, use Bamboo scheduled backups.
• Instead of using local agents, use Bamboo remote agents