Summary

When using ephemeral agents, it is expected that the build initiating the startup of the ephemeral agent will run immediately. However, when using Security Context for container in Ephemeral Agent Template the affected build remains in the queue for 5 minutes before starting.

Environment

Bamboo DC Ephemeral agents

Diagnosis

1. Create an Ephemeral Agent Template with a container security context

   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: '{{NAME}}'
     labels:
         '{{RESOURCE_LABEL}}': ephermeral
   spec:
     automountServiceAccountToken: false
     containers:
       - image: atlassian/bamboo-agent-base:BAMBOO_VERSION
         name: '{{BAMBOO_AGENT_CONTAINER_NAME}}'
         env:
           - name: BAMBOO_EPHEMERAL_AGENT_DATA
             value: '{{BAMBOO_EPHEMERAL_AGENT_DATA_VAL}}'
         securityContext:
           allowPrivilegeEscalation: false
           readOnlyRootFilesystem: true
           runAsUser: 2005
           capabilities:
             add: ["SETGID","SETUID","CHOWN"]
             drop:
               - ALL
         volumeMounts:
           - name: bamboo-agent-lib
             mountPath: /var/atlassian/application-data/bamboo-agent  
     volumes:
       - name: bamboo-agent-lib
         emptyDir: {}     
     restartPolicy: Never

2. Start a build with this template

   
   

3. An ephemeral agent will be started, but the build will remain in the queue and start running after 5 minutes.

4. Navigate to the ephemeral agent Pod to view the log

   In the ephemeral agent startup logs, you will find the following permission around the starting stage of the container.

   2024-05-21T20:31:37.123276958Z INFO:root:Generating /var/atlassian/application-data/bamboo-agent/conf/wrapper.conf from template wrapper.conf.j2
   2024-05-21T20:31:37.133963582Z WARNING:root:Permission problem writing '/var/atlassian/application-data/bamboo-agent/conf/wrapper.conf'; skipping
   ---
   2024-05-21T20:31:37.309793140Z Installing file: /var/atlassian/application-data/bamboo-agent/conf/wrapper.conf

Cause

When utilising pod security context with readOnlyRootFilesystem: true, it is essential to mount the container on a volume to enable write access to the agent home. However, because the Agent wrapper config directory is not mounted, the startup script failed to utilise the wrapper template wrapper.conf.j2 (located in the agent base image) to create the wrapper.conf property file.

The creation of the wrapper config file from the wrapper template (wrapper.conf.j2) is crucial because certain properties needed to run as an Ephemeral agent must be added to the agent's wrapper.conf file,

wrapper.java.additional.4=-DDISABLE_AGENT_AUTO_CAPABILITY_DETECTION=false # exception of this.
...
wrapper.java.additional.7=-Dbamboo.agent.ephemeral.for.key=TP-UNTAR-JOB1-5
wrapper.java.additional.8=-Dbamboo.agent.ephemeral.template.id=589825
wrapper.java.additional.9=-Dbamboo.agent.ephemeral.pod.name=tp-untar-job1-5-nmjmqvtm 

and these properties are retrieved by the startup script from the Pod environment variable BAMBOO_EPHEMERAL_AGENT_DATA.

Environment:
      BAMBOO_EPHEMERAL_AGENT_DATA:  bamboo.agent.ephemeral.for.key=TP-UNTAR-JOB1-5#bamboo.agent.ephemeral.template.id=589825#bamboo.agent.ephemeral.pod.name=tp-untar-job1-5-nmjmqvtm...
...

Due to a permission error, the startup script failed to create the wrapper.conf file from the template. As a result, the agent had to obtain the default wrapper.conf file from its class path while installing the agent wrapper, and this default file does not include the ephemeral agent properties.

2024-05-21T20:31:37.260379446Z Installing agent wrapper
....
2024-05-21T20:31:37.309793140Z Installing file: /var/atlassian/application-data/bamboo-agent/conf/wrapper.conf

The effect of this is that the agent will be started as a Remote agent, and not an Ephemeral agent.

 Note that if remote agent authentication is enabled in Bamboo, you need to authenticate the agent as a remote agent. Furthermore, after the build runs, the Pod is not automatically removed, so you will need to remove the pod manually.

Solution

When creating the Ephemeral Agent Template template, add an additional volume mount for the wrapper config directory e.g

 spec:
  automountServiceAccountToken: false
  containers:
    - image: atlassian/bamboo-agent-base:BAMBOO_VERSION
      ...
      volumeMounts:
        - name: bamboo-agent-lib
          mountPath: /var/atlassian/application-data/bamboo-agent
		- name: wrapper-file
          mountPath: /var/atlassian/application-data/bamboo-agent/conf
  volumes:
    - name: bamboo-agent-lib
      emptyDir: {}
     - name: wrapper-file
      emptyDir: {}