Problem

A build plan has been queued for building for an extended duration and is not showing on the Build activity dashboard as queued. There are Agents capable of processing the build available but it remains queued for an excessive period.

Cause

A large delay between queueing and it appearing in the build queue suggests that the server-side change detection process is taking longer (see further down for more information on the build workflow). It's helpful to understand the basics of the Bamboo build plan workflow to understand where the issue might be when the symptoms present:

1. A build plan is triggered.
2. Plan changes status to queued.
3. Change detection happens on the server-side. This is where it will reach out to the repository to determine if there are any changes it needs for the build and is the most likely breakdown point when observing queued builds that aren't yet in the build queue.
4. Plan is then added to the build queue. This is when it shows up on the Build activity dashboard.
5. Server assigns an agent for it and sends an event to the agent.
6. Agent receives the event and starts building.

It's important to note that this behavior is normal however we've observed a few causes for abnormally long server-side change detection:

Cause 1: PlanExec thread contention

Bamboo has 4 threads that are responsible for moving builds through the early execution phases. These threads can get tied up servicing change detection operations if VCS operations are slow or your Bamboo instance has a high count of repository polling occurring at high frequency.

Cause 2: Older versions of Git for Windows

There have been a number of large performance improvements to Git for Windows in the more recent versions which can speed up change detection significantly.

Cause 3: Quiet repository

The Enable Quiet repository setting on repositories is a configurable delay after a single commit is detected before the build is started. This allows multiple commits to be aggregated into a single build. While the quiet period wait time is in affect for a build, it will not show up in the Build activity dashboard as it's not yet in the Build Queue. The build status will however appear as 'Queued' which can cause some confusion.

Large Quiet period durations are also known to cause issues on busy Bamboo Servers in delaying plans by consuming plan execution threads for the duration of the sleep as well as blocking manual builds and deployment plans. See the following Bug Report:  BAM-7320 - Getting issue details... STATUS

Cause 4: Git Credential Manager for Windows

When attempting to authenticate with a public GitHub repository by supplying a username and no password, if Git Credential Manager for Windows is installed it will launch an interactive child process in the background that waits for you to enter your GitHub credentials. Since Bamboo is being running the git command for change detection in an automated, non-interactive fashion – this process will never receive the credentials and terminate, so Bamboo will keep the build in the 'queued' state until the configured command timeout on the repository is reached with a log entry similar to the following:

2017-02-15 01:47:54,686 INFO [9-BAM::PlanExec:pool-16-thread-1] [GitCommandProcessor] Command was canceled: command 'C:\Program Files\Git\cmd\git.exe' ls-remote https://user@github.com/repository-owner/repo.git failed with code -1. Working directory was [.].

Cause 5: SSH timeout

Bamboo have a hardcoded limit of 2 minutes for idle SSH connections to stay alive, if your repository is considerably big this limit is might be being reached, causing the change detection to fail and, consequently, the pre-build actions not to finish. This situation will prevent the build to be placed in the queue. If this is the case, you'll see the following message in your logs:

BAMBOO-SSH-PROXY: [User session has timed out idling after 120000 ms.]

We have an open improvement request asking for this timeout value limit to be configurable,  you can track the implementation of this improvement at the following ticket:  

BAM-19670 - Getting issue details... STATUS

Workaround

Cause 3: Quiet repository

To minimize the impacts of the Quiet Repository setting on busy servers, please consider:

1. Reducing the amount of plans using the Enable Quiet repository so that multiple builds sleeping for the Quiet period duration do not consume all the plan execution threads.
2. Reducing the duration of the Quiet period.

3. Increasing the amount of plan execution threads by adding the following start-up argument, if you have >= 8 CPU cores on your Bamboo Server:

   We don't recommend using values higher than 8.

   • -Dbamboo.plan.exe.threads=8

   • See Configuring your system properties for instructions based on your environment.

Cause 4: Git Credential Manager for Windows

1. First check to see if any credentials have been stored for the repository under Windows Control Panel > Credential Manager > Generic Credentials. Depending on previous authentication attempts, sometimes the credential manager can store just the username here and all subsequent attempts will launch the interactive git-credential-manager.exe with the GitHub login prompt UI – so it's best to clear this entry first.

2. From command-line, run a git command to the repo. I.E: git ls-remote https://user@github.com/repository-owner/repo.git

3. When prompted for a password, enter it. This will store your user/pass in the credential vault and it won't prompt for it again when the same command is executed from Bamboo.

4. Alternatively, you can also add your password to the repository configuration in Bamboo to bypass the credential manager prompt.

The other workaround you can use, is instead of using a GitHub type repository in Bamboo to access a public repository, you can use a repository of type 'Git' and point it to the public GitHub URL. The difference being that in the GitHub repository in Bamboo, the Username is a mandatory field, so you cannot authenticate with a public repository anonymously as you can with the Git repository.

Resolution

Cause 1: PlanExec thread contention

There are multiple things you can look at to reduce the contention. Cause 2, 3 and 4 are some root causes that can ultimately just tie up these threads, but there's a few other things to look at too:

1. Reduce contention for these threads by reducing the amount of repository polling operations occurring on your instance. The below knowledge base article will guide you through the analysis and identification of excessive polling. Ultimately the resolution is to reduce the frequency at which repository polling occurs.
   1. How do I work with triggering builds

2. Try to understand if basic VCS operations are taking longer than they should due to network factors and investigate the external cause. For example, with Git you can check the Bamboo Application logs at <bamboo-home>/logs/atlassian-bamboo.log for log error and warning messages that Git operations are slow. Example below:

   2017-11-20 22:02:04,861 ERROR [12-BranchDetectionBackgroundThread:pool-20-thread-1] [GitCommandProcessor] ls-remote http://bitbucket:7990/bitbucket/scm/res/stored-spec.git took 40.11 s
   2017-11-20 22:02:04,862 WARN [12-BranchDetectionBackgroundThread:pool-20-thread-2] [GitCommandProcessor] ls-remote http://bitbucket:7990/bitbucket/scm/res/stored-spec2.git took 15.12 s
   2017-11-20 22:02:04,867 ERROR [12-BranchDetectionBackgroundThread:pool-20-thread-3] [GitCommandProcessor] ls-remote http://bitbucket:7990/bitbucket/scm/sim/simple-npm-bamboo.git took 40.13 s

   1. Simple grep command to identify these errors and warnings: grep -E "(ERROR|WARN).*GitCommandProcessor.*took"
3. If Bamboo is interacting heavily with Bitbucket Server, consider enabling ref-advertisement caching on the Bitbucket side to speed up operations: Scaling Bitbucket for Continuous Integration Performance

4. Increase the amount of plan execution threads by adding the following start-up argument, if you have >= 8 CPU cores on your Bamboo Server

   We don't recommend using values higher than 8.

• • -Dbamboo.plan.exe.threads=8

  • See Configuring your system properties for instructions based on your environment.

Cause 2: Older versions of Git for Windows

Upgrade your version of Git for Windows to the most recent supported version. 

At time of writing this article we've observed issues with versions older than 2.8. See Supported platforms for more info.