Summary

Bamboo stops responding with the following errors being thrown inside the <bamboo-home>/logs/atlassian-bamboo.log files:

2022-12-07 18:01:18,134 ERROR [http-nio-8085-exec-195 url: /bamboo/agentServer/message] [ArtifactServlet] Exception when storing the artifact
java.io.IOException: No space left on device
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
        at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280)
        at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
        at java.base/java.nio.channels.Channels.writeFully(Channels.java:97)
        at java.base/java.nio.channels.Channels$1.write(Channels.java:172)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1310)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
        at com.atlassian.bamboo.v2.build.agent.messages.ArtifactStreams.deserialiseStreamToDir(ArtifactStreams.java:89)
        at com.atlassian.bamboo.agent.messaging.ArtifactServlet.storeArtifact(ArtifactServlet.java:97)
        at com.atlassian.bamboo.agent.messaging.ArtifactServlet.doPost(ArtifactServlet.java:85)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        ...
2022-12-07 18:01:17,978 ERROR [http-nio-8085-exec-256 url: /bamboo/agentServer/message] [ArtifactServlet] Exception when storing the artifact
java.io.IOException: Directory '<bamboo-shared-home>/artifacts/tmp/com.atlassian.bamboo.agent.messaging.ArtifactServlet5229193036404846737/<artifact-name>' could not be created
        at com.atlassian.bamboo.utils.BambooPathUtils.openOutputStream(BambooPathUtils.java:221)
        at com.atlassian.bamboo.utils.BambooPathUtils.openOutputStream(BambooPathUtils.java:197)
        at com.atlassian.bamboo.v2.build.agent.messages.ArtifactStreams.deserialiseStreamToDir(ArtifactStreams.java:87)
        at com.atlassian.bamboo.agent.messaging.ArtifactServlet.storeArtifact(ArtifactServlet.java:97)
        at com.atlassian.bamboo.agent.messaging.ArtifactServlet.doPost(ArtifactServlet.java:85)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

Environment

This was observed on Bamboo 8.2.5 but can happen to other versions of Bamboo.

Diagnosis

The following actions can be taken to diagnose the issue and confirm that you're running into the problem described in this article:

1. Look for threads with the /agentServer/message?resultKey endpoint that might've been reported as stuck by the StuckThreadDetectionValve for several hours and have only completed when Bamboo ran out of disk space inside the <bamboo-install>/logs/catalina.out file:

   07-Dec-2022 15:55:30.466 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8085-exec-66 url: /bamboo/agentServer/message] (id=[520]) has been active for [63,210] milliseconds (since [11/16/22, 3:54 PM]) to serve the same request for [https://<bambooBaseURL>/agentServer/message?resultKey=MYPROJ-PLAN-MYJOB-12&entityType=PLAN] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [3] thread(s) in total that are monitored by this Valve and may be stuck.
   ...
   07-Dec-2022 18:01:43.760 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8085-exec-66] (id=[520]) was previously reported to be stuck but has completed. It was active for approximately [7,629,750] milliseconds. There is/are still [2] thread(s) that are monitored by this Valve and may be stuck.

   In this example the thread was running for 7,629,750 milliseconds (approx. 2.1 hours) before Bamboo failed with "No space left on device" errors.

2. Once you identify the thread(s) reported by the StuckThreadDetectionValve take note of the plan(s) and job(s) keys they were processing. From the example above this is MYPROJ-PLAN-MYJOB. With this in mind leverage the use of the commands mentioned inside the Disk space hotspots and cleanup best practices in Bamboo article to confirm whether the artifact(s) produced by the plan(s) and job(s) in question are indeed large:
   

   The following command will list the disk space used by each artifact within a plan. Please check the how to match an artifact with its respective build section of the article mentioned above to identify the plan(s):

   Find large artifacts

   # BAMBOO_HOME=/var/atlassian/application-data/bamboo
   # du -h ${BAMBOO_HOME}/artifacts/ --max-depth 1 | sort -rh
   60K	/var/atlassian/application-data/bamboo/artifacts
   32K	/var/atlassian/application-data/bamboo/artifacts/plan-15106049
   12K	/var/atlassian/application-data/bamboo/artifacts/globalStorage
   4.0K	/var/atlassian/application-data/bamboo/artifacts/plan-6062084
   0	/var/atlassian/application-data/bamboo/artifacts/tmp
   0	/var/atlassian/application-data/bamboo/artifacts/plan-1638401
   (...)

3. If there is(are) no artifact(s) created by the plan(s) in question stored inside Bamboo yet there's another way to confirm whether the artifact(s) that the plan(s) and job(s) are trying to produce are indeed large. Review the artifact definition(s) inside the job configuration and access the <bamboo-agent-home>/xml-data/build-dir/PROJ-PLAN-JOB folder and see if you can sort out the size of the artifact that's getting created and transferred back to Bamboo by leveraging commands like du -h.

Cause

The disk/partition holding the <bamboo-shared-home>/artifacts folder is running out of space due to large artifacts. If you're using Bamboo's artifact remote handler this means artifacts are always stored inside the <bamboo-shared-home>/artifacts folder (the number of artifacts kept inside this folder may vary depending on how you have set up global expiry). Depending on how large the artifact(s) you're producing are compared to the size of the disk/partition holding the <bamboo-shared-home>/artifacts folder it may fill up rather quickly.

Solution

Solution 1

Increase the size of the disk/partition holding the <bamboo-shared-home>/artifacts folder.

Solution 2

Leverage the use of the "Disk space hotspots and cleanup best practices in Bamboo" documentation to identify artifacts and build logs that are or might start causing trouble due to their size and configure global expiry to avoid running out of disk space.

Solution 3

Limit the sizes of artifact(s) produced by your plans.

1. Stop Bamboo.

2. Add the -Dbamboo.artifact.size.limit.per.build property to the JVM by following the instructions inside the "Configuring your system properties". This property has to be set in bytes. For example, if you want to limit the size of any single artifact file to 200 MB you will need to use 209715200:

   -Dbamboo.artifact.size.limit.per.build=209715200

3. Start Bamboo.

After that a warning message will start showing up inside the artifact definition page of every plan in Bamboo with the following message:

Builds will not fail due to this limitation but deployment projects and other jobs that require such artifact to run will fail if they can't download the artifact from the server. Users will see an error on the build result summary page with the following message when an artifact exceeds the size limit: