Commit build status is always red even if other builds succeed

Symptoms

If successive builds are run against the same commit using a CI other than Atlassian Bamboo and one of then is red (meaning 'fails') the overall status of the commit remains red.

For example, see the screen below:

Although newer builds were successfully run against that commit, as you can see on the screen above, the failed build is the one that prevails on the commit screen.

Cause

Although this sounds like a problem, it is not. We've found out in 2 occasions so far that this is caused by the way the CI system is reporting the build status to Stash as well as a misinterpretation of why Stash shows a list of build status for the same commit.

In order to understand what is causing this, we need to clarify how Stash works and why it has a list of builds for a single commit.

The intention with having a list of builds per commit in the build history popup is to capture multiple different test/build/deploy runs, rather than successive attempts at the same build – which Stash expects to always have the same result as a commit is an immutable set of files. For example, you might see that your separate unit test job and functional test job is passing, but your job to deploy to a staging server is not, and this would show as 3 different build statuses no matter how many attempts were made for a given commit. That is the reasoning behind why when at least one of the builds failed, Stash will report that commit as having failed in the overall build status.

This page explains how the build notification REST API works in more details: https://developer.atlassian.com/stash/docs/latest/how-tos/updating-build-status-for-commits.html

Stash is designed this way because, for a given commit, a build result should (in theory) be the same every time for a given state of the code base. Of course, things can go wrong in a build environment that aren't related to the code itself, so Stash will happily accept an updated status notifications for the same build again, showing the most recent result. With Atlassian's build system (Bamboo), we include the build number in the status description to provide a visual indication, but not in the build key (see link above for more details on "build number" and "build key"). This design is also influenced by the behaviour of pull requests where restrictions can be implemented on build status.

We've found this incorrect behaviour in two scenarios:

• Scenario 1: With Stash notifier plugin - a Jenkins plugin to notify Stash of build status
• Scenario 2: With Mendhak/Teamcity plugin - a plugin for TeamCity which updates Stash with build statuses
• Scenario 3: you are using a different CI product and you've got a script that does a REST call to the Stash API to add the build status.

Resolution

• Scenario 1:
  In order to get around this, configure your Jenkins plugin with "keep repeated builds in stash" checked see this discussion for more details.
  
  
• Scenario 2:
  The equivalent option here is the "Only show latest build status for each commit". Check this option for the expected behaviour. Refer to Teamcity plugin page to view that option.
  
  
• Scenario 3:
  

  You have to stop sending the build key. This behaviour will happen if your CI is passing different build keys every time, which generates a history on Stash. 

  As explained before:

  With Atlassian's build system (Bamboo), we include the build number in the status description to provide a visual indication, but not in the build key (see this link for more details on "build number" and "build key").

  From our documentation, this is what you seem to be doing:

  Multiple builds

  If a single commit was built by multiple build plans, you can POST multiple results for the same commit. Each result must have a different key attribute.

  What you are trying to achieve is:

  Updating build results

  Stash stores one build result for each key per commit, so you can update a previous build result by sending a request with:

  • the same commit hash; and
  • the same key attribute.

  The new result will replace the previously submitted build result for that commit hash and key.