Bamboo customized builds fail to run with git tag rejected messages (would clobber existing tag)
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Plans fail to start when building a custom revision (customized build) with the following error messages inside the <Bamboo server home directory>/logs/atlassian-bamboo.log files:
2021-08-25 09:25:56,832 ERROR [7-DelayedChangeDetectionThread:pool-13-thread-4] [ChainExecutionManagerImpl] Errors getting changes for PROJ-PLAN-4
com.atlassian.bamboo.plugins.stash.repository.StashRepositoryException: com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/*' from 'ssh://git@bitbucket7120:7999/tags/tags.git' to source directory '/var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/c1c01d6bea07ceebadac984e0e50c29c566502b3'. command [/usr/bin/git fetch --tags ssh://da9d7051-b4e3-42ac-89a6-ee5996fdbdd6@127.0.0.1:40807/tags/tags.git +refs/heads/*:refs/heads/* --update-head-ok --progress --verbose] failed with code 1. Working directory was [/var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/c1c01d6bea07ceebadac984e0e50c29c566502b3]., stderr:
Warning: Permanently added '[127.0.0.1]:40807' (RSA) to the list of known hosts.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1)
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
From ssh://127.0.0.1:40807/tags/tags
= [up to date] master -> master
! [rejected] v1.0 -> v1.0 (would clobber existing tag)
at com.atlassian.bamboo.plugins.stash.v2.BitbucketServerChangeDetector.collectChangesForRevision(BitbucketServerChangeDetector.java:44)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.lambda$createBuildRepositoryChanges$2(DefaultChangeDetectionManager.java:473)
at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:118)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuildInternal(DefaultChangeDetectionManager.java:439)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:287)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:220)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectAllChangesSinceLastBuild(DefaultChangeDetectionManager.java:164)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectAllChangesSinceLastBuild(DefaultChangeDetectionManager.java:190)
at com.atlassian.bamboo.v2.trigger.ManualBuildDetectionAction.performDelayedChangeDetection(ManualBuildDetectionAction.java:154)
at com.atlassian.bamboo.chains.ChainExecutionManagerImpl$1.getChainState(ChainExecutionManagerImpl.java:250)
at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.tryStartChainState(ChainExecutionManagerImpl.java:322)
at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.delayedStart(ChainExecutionManagerImpl.java:242)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
...Diagnosis
Access the remote repository you're trying to build/ checkout code from and run the following command:
git rev-list -n 1 "replaceWithTagName"Take note of the commit hash you'll get in return (e.g. 250612675cd6ec74ac93d0526dc96ad52a75aa67)
You must replace replaceWithTagName with the actual tag name showing up in the logs. In the example above the tag name is v1.0.
Access the directory inside the server where Bamboo is running where it keeps a local cache/ copy of your Git repository and run the following command:
git rev-list -n 1 "replaceWithTagName"Use the same tag name from step 1 and take note of the commit hash (e.g. 8774260ec7e9dcfaa03497dbcb7e1d2d648ade5b)
In the example above the directory is located at /var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/c1c01d6bea07ceebadac984e0e50c29c566502b3.
The commit hashes you get from steps 1 and 2 should be the exact same but they won't be -- this is what's causing the problem.
Cause
In order to understand the cause we need to take a step back and understand how Bamboo works with triggering builds. To be able to trigger builds automatically, identify changes between builds and etc Bamboo keeps a cache of your Git repositories locally under <Bamboo server home directory>/xml-data/build-dir/_git-repositories-cache. There's a folder inside this directory for every Git repository configured in Bamboo. Every time you trigger a build Bamboo has to update the local Git cache of the repository associated to the plan you're trying to build before dispatching the build to an agent. If this operation (updating the local Git cache) fails the build will not get dispatched and the plan will fail.
If we look closely at the log snippet mentioned above we will see that the process fails when Bamboo is trying to update the local Git cache and it encounters an issue/ conflict with git tags:
2021-08-25 09:25:56,832 ERROR [7-DelayedChangeDetectionThread:pool-13-thread-4] [ChainExecutionManagerImpl] Errors getting changes for PROJ-PLAN-4
com.atlassian.bamboo.plugins.stash.repository.StashRepositoryException: com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/*' from 'ssh://git@bitbucket7120:7999/tags/tags.git' to source directory '/var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/c1c01d6bea07ceebadac984e0e50c29c566502b3'. command [/usr/bin/git fetch --tags ssh://da9d7051-b4e3-42ac-89a6-ee5996fdbdd6@127.0.0.1:40807/tags/tags.git +refs/heads/*:refs/heads/* --update-head-ok --progress --verbose] failed with code 1. Working directory was [/var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/c1c01d6bea07ceebadac984e0e50c29c566502b3]., stderr:
Warning: Permanently added '[127.0.0.1]:40807' (RSA) to the list of known hosts.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1)
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
From ssh://127.0.0.1:40807/tags/tags
= [up to date] master -> master
! [rejected] v1.0 -> v1.0 (would clobber existing tag)
Bamboo is trying to execute "git fetch --tags ssh://..." inside <Bamboo server home directory>/xml-data/build-dir/_git-repositories-cache/c1c01d6bea07ceebadac984e0e50c29c566502b3 (local Git cache for the bitbucket7120:7999/tags/tags.git repository) when the operation gets rejected with the following message:
! [rejected] v1.0 -> v1.0 (would clobber existing tag)
This is a Git error, not a Bamboo error. In a nutshell Git is telling us that there's a conflict with the tag v1.0 because the one that exists locally (in the Git cache) and is pointing to a different commit when compared to the same tag coming from your remote repository. This usually happens when moving a tag between commits.
For example, let's say you originally have tag v1.0 pointing to commit a123. You connect your Git repository to Bamboo and start building your plan. Bamboo creates a local Git cache that also has v1.0 pointing to commit a123. Later you decide to remove that tag and create another one with the same name pointing to commit b456. You push the changes to your remote repository. Before starting a new custom build Bamboo will attempt to fetch changes (including tags!) from your remote repository to keep the local cache up-to-date but the update gets rejected because the tag v1.0 is pointing to commit a123 (locally) and not commit b456 (remote repository). There's a conflict. At this point the "git fetch --tags ssh://..." operation fails and your build is greyed out.
Resolution
In order to solve this problem you have to remove the local Git cache of the affected repositories. Bamboo will recreate the cache for those repositories next time a build is run against each one of them. The new Git cache should have the updated tags (e.g. from the example above the new cache will have v1.0 pointing to b456), so you should be able to build custom revisions of your plans again.
Please note that when you remove a repository cache the first builds checking out code from that repository might take longer than usual to run because they have to wait for the cache to be recreated before getting dispatched to an agent. Depending on how large your repository is this might take several minutes. Your builds will be placed in the queue and sent to an agent after the repository cache is recreated.
As a Bamboo admin you can remove the cache of the affected repositories inside the Bamboo administration > Overview > Build resources > Repository settings page.
Repository caches will always be stored on the Bamboo server side but not necessarily on the agent side. Those that are cached on the agent are labelled with (agent caching) inside the repository settings page. Deleting a repository cache from the repository settings page should be enough to have it removed from the Bamboo server as well as the agents (Bamboo will send a message to the agents). If this problem starts (or continues) to happen on the cache located on the agent side you may have to manually remove the folder holding the cache of your Git repository on the agent located under <Bamboo agent home directory>/xml-data/build-dir/_git-repositories-cache.