Git LFS pull hangs indefinitely in Bamboo with Git LFS 2.6.0
Problem
Git LFS pull hangs indefinitely in Bamboo when the repository has > ~100 LFS objects and the build eventually fails after reaching the Git command timeout (default 180 minutes).
Diagnosis
Environment
Git LFS 2.6.0 client is installed on the Bamboo Server (local agent) or Remote Agent performing the build.
Diagnostic Steps
With verbose Git logging enabled in Bamboo: Edit Repository >> Advanced Options >> Enable verbose logging the build hangs at the below line in the build log:
simple 29-Nov-2018 17:32:16 /usr/bin/git remote set-url origin ssh://d0cbc387-4ab8-444a-81bd-19700ad67444@127.0.0.1:46388/lfs/lfsrepotest.git
simple 29-Nov-2018 17:32:16 /usr/bin/git lfs pull
command 29-Nov-2018 17:39:12 Request to stop 'LFS-LFS2-JOB1-5' received from charlie
simple 29-Nov-2018 17:39:12 /usr/bin/git remote set-url origin file:///var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/e8904ea3b8469d750cc8e102ce3b1f91ae7fb3bastrace -p <pid> of the Git LFS pull shows the process is stuck in a FUTEX_WAIT loop:
[pid 2331] 17:34:04.472561 nanosleep({0, 20000}, NULL) = 0
[pid 2331] 17:34:04.472903 futex(0xce2530, FUTEX_WAIT_PRIVATE, 0, {60, 0}) = -1 ETIMEDOUT (Connection timed out)
[pid 2331] 17:35:04.473506 nanosleep({0, 20000}, NULL) = 0
[pid 2331] 17:35:04.473956 futex(0xce2530, FUTEX_WAIT_PRIVATE, 0, {60, 0}Cause
There's an issue with Git LFS 2.6.0 that causes problems when pulling from a clone with Git alternates --shared or --reference and this is how Bamboo leverages benefits of it's local Git cache:
Workaround
If using remote agents, you can disable repository caching at Edit Repository >> Advanced Options >> Enable repository caching on remote agents which will workaround the issue (not recommend as the repository cache is very beneficial to speed up repository operations, reducing load on your repository server and network).
Resolution
Upgrade to a fixed version of Git LFS, 2.6.1 and above: https://github.com/git-lfs/git-lfs/releases/tag/v2.6.1