Bamboo agent go offline with message "Quit and let the Wrapper resynch"
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
A Bamboo remote agent will go offline unexpectedly showing in the logs the message "Quit and let the Wrapper resynch"
Environment
Observed (although not exclusively) in Windows hosted agents
Diagnosis
Bamboo agent will go offline and then restart automatically. In the logs we will observe something similar to:
INFO | jvm 86 | 2020/12/03 06:50:51 | Read Timed out. (Last Ping was XXXXX milliseconds ago)
INFO | wrapper | 2020/12/03 06:50:51 | Wrapper Process has not received any CPU time for XX seconds. Extending timeouts.
...
INFO | jvm 86 | 2020/12/03 06:52:07 | Read Timed out. (Last Ping was XXXXX milliseconds ago)
INFO | jvm 86 | 2020/12/03 06:52:07 | Wrapper Manager: The Wrapper code did not ping the JVM for XX seconds. Quit and let the Wrapper resynch.
INFO | jvm 86 | 2020/12/03 06:52:07 | Send a packet RESTART : restartCause 1
The time out messages in the logs mean that neither the wrapper now the JVM have access to the CPU for the seconds mentioned in the log entry. The first way that this can happen is when the Wrapper is competing for system resources with another process that has the habit of consuming 100% of the CPU for extended periods of time without yielding to other processes. Most modern operating systems are fairly good about managing multitasking. But there are still cases where it can fail. One example of this on Windows is when the machine is very low on memory, leading to lots of disk swapping. If the total memory is not large enough, the entire system can freeze up for as long as a minute before any applications are again given any CPU cycles.
Cause 2
Bamboo uses the Tanuki wrapper library. And the older version of the Tanuki wrapper library had issues with unexpected JVM shutdowns where it would show error messages like the below.
INFO | jvm 25 | 2022/05/25 14:20:59 | 2022-05-25 14:20:40,446 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] ExecutableQueueUpdate: invalidate executables: REMOTE_AGENT, agents known to be affected: [223543297]
INFO | wrapper | 2022/05/25 14:20:59 | Wrapper Process has not received any CPU time for 58 seconds. Extending timeouts.
ERROR | wrapper | 2022/05/25 14:21:00 | JVM exited unexpectedly.
STATUS | wrapper | 2022/05/25 14:21:05 | Launching a JVM...
INFO | jvm 26 | 2022/05/25 14:21:06 | Wrapper (Version 3.2.3-atlassian-1) http://wrapper.tanukisoftware.org
INFO | jvm 26 | 2022/05/25 14:21:06 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
INFO | jvm 26 | 2022/05/25 14:21:06 |
INFO | jvm 26 | 2022/05/25 14:21:06 | WARNING - The Wrapper jar file currently in use is version "3.2.3-atlassian-1"
INFO | jvm 26 | 2022/05/25 14:21:06 | while the version of the Wrapper which launched this JVM is
INFO | jvm 26 | 2022/05/25 14:21:06 | "3.5.37".
INFO | jvm 26 | 2022/05/25 14:21:06 | The Wrapper may appear to work correctly but some features may
INFO | jvm 26 | 2022/05/25 14:21:06 | not function correctly. This configuration has not been tested
INFO | jvm 26 | 2022/05/25 14:21:06 | and is not supported.
INFO | jvm 26 | 2022/05/25 14:21:06 |
INFO | jvm 26 | 2022/05/25 14:21:06 | Wrapper code received an unknown packet type: -122
INFO | jvm 26 | 2022/05/25 14:21:06 | 2022-05-25 14:21:06,428 INFO [WrapperSimpleAppMain] [AgentBootstrap] Starting AgentThere were significant bug fixes in the Tanukisoftware service wrapper library from version 3.5.0. Please refer to: https://wrapper.tanukisoftware.com/doc/english/release-notes.html#3.5.0 for more detail.
Solution 1
It's worth mentioning that this is not a Bamboo problem but a hardware limitation based on the usage given. Addressing this is our official recommendation.
The wrapper also offers a way to extend the time out value.
This property is ignored unless wrapper.use_system_time=TRUE.
The property to set the time out is
wrapper.cpu.timeoutThis property sets the number of seconds without CPU before the JVM will issue a warning and extend timeouts. In order for this property to have any effect, it must have a value less than the following properties:
wrapper.startup.timeout
wrapper.ping.timeout
wrapper.shutdown.timeoutThe default value for wrapper.cpu.timeout is 10 (seconds). Setting this property value to "0" (zero) means never extend time out.
While the ability is there, be aware that setting this property value to "0" (zero) (= disable Timeout) or a larger value than another timeout could cause that timeout to be falsely triggered in cases of heavy load. This can lead to the JVM being restarted when a restart is not really necessary.
Solution 2
To address some known issues in the Tanuki wrapper library, we updated the Tanuki wrapper used for the Bamboo remote agent. Bamboo 6.10 is released with the new version of the wrapper. We strongly recommend you reinstall the wrapper to benefit from all bug fixes.
To update the wrapper on a remote agent:
- Stop the original agent.
- Backup the BAMBOO_AGENT_HOME/conf/wrapper.conf file.
- Remove the BAMBOO_AGENT_HOME/conf directory.
- Download a new Remote Agent JAR from your Bamboo Server: http://<BAMBOO_URL>/admin/agent/addRemoteAgent.action.
- Launch the remote agent.
- This will create a new wrapper.conf file. If you want to re-enact your custom configuration in that file, you can do it now.