Elastic Agents can't decode user-data while using specific Java versions
Summary
Elastic Agent is not started on the Elastic Instance, because it's unable to decode user data passed by Bamboo to the EC2. The following error is present in /tmp/testIfStartedFromBamboo.log file on the Elastic Instance:
Wed Feb 16 12:34:19 UTC 2022
Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: Unexpected character ('�' (code 65533 / 0xfffd)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (String)"�XG��ȑ�+��ʞ����{ƒ $�w$��
�[;�ؕ��@V֫�YY�����(�S�R�~|�2����æ����i��鴥���k^�iY�l�~=�q�%���}��K
�s�D5y?�K���6=���1�<�/�:/���P$�ti�%d�%���]����s�����eV�ߓs����r�ϩ3r2���ۇ�oSڤgh���o1ߡ����DG�L
#� �4B�k|�����į��O�߳�;�S�-�b8�\I�1�aE�$v���';i�O}KN�3�����5�i/�w�J��c4͐
�I��p��$z�p�h�I��{�!L"��E$! �$��!���{@ewr���?���ʯ_dN��������>_����Ω�v]���$��������v���̻~J�tZދ�K�i\��Y���xڄ�R�R{&�黬����HJ�~�$���Y<`���]�qF�}�֘����1��U��o
�~
�^��~��k����5IR�]���a��{l)BX���"[truncated 2443 chars]; line: 1, column: 2]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:591)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1902)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4141)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004)
at com.atlassian.aws.utils.JsonUtils.fromJson(JsonUtils.java:43)
at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:49)
at com.atlassian.bamboo.agent.elastic.metadata.RetrieveMetaData.main(RetrieveMetaData.java:24)
Marker /root/.startedFromBamboo not found. This instance was started manually.
/opt/bamboo-elastic-agent/bin/testIfStartedFromBamboo.sh: 17: exit: Illegal number: -1On the Agent startup logs you can also observe the following message:
Fri Feb 16 12:35:02 UTC 2022 - This instance has not been started by Bamboo. The agent will not be started.Environment
Bamboo using Elastic Agents.
Diagnosis
On the Elastic Agent, create the following Java code:
import com.atlassian.aws.utils.JsonUtils;
import com.atlassian.aws.utils.URLFetcherUtils;
import com.atlassian.bamboo.agent.elastic.ElasticAgentUserData;
import com.atlassian.bamboo.agent.elastic.ElasticAgentUserDataImpl;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.SortedMap;
import java.util.TreeMap;
public class getUserData {
private static <T> T fetchData(final Class<T> aClass) throws IOException {
String USER_DATA_URL = "http://169.254.169.254/latest/user-data";
boolean shouldAttemptDecompression = true;
return JsonUtils.fromJson(URLFetcherUtils.fetchString(USER_DATA_URL, StandardCharsets.UTF_8, shouldAttemptDecompression), aClass);
}
public static void main(final String[] args) throws IOException {
final ElasticAgentUserData agentUserData = fetchData(ElasticAgentUserDataImpl.class);
final SortedMap<String, String> userData = new TreeMap<>(agentUserData.getMetaData());
if (args.length == 0) {
userData.forEach((_key, _value) -> System.out.println(_key + "=" + _value));
}
else {
String value = userData.get(args[0]);
System.out.println(userData.get(args[0]) + "=" + value);
}
}
}Then run it as root. A failed Server x Agent JDK combination will result in the following exception:
# java -cp /opt/bamboo-elastic-agent/bin/atlassian-bamboo-agent-elastic-installer.jar getUserData.java
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize keystore
at [Source: (String)"{"baseURL":"https://bamboo.mydomain.net/agentServer/","startupTimeoutSeconds":600,"keyManagerAlgorithmName":"SunX509","trustManagerAlgorithmName":"PKIX","metaData":{"bamboo.version.specific.data.location":"s3://bamboo-agent-release-ap-se2/9.1.0-rc2/4049d092bc97001cafb7b38f7465fa9e2ce987e1","bamboo.server.fingerprint":"-3826544862034641465","aws.startedFromBamboo":"true","startupScript_0":""[truncated 4731 chars]; line: 1, column: 5135] (through reference chain: com.atlassian.bamboo.agent.elastic.ElasticAgentUserDataImpl["keyStore"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1821)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:315)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
at com.atlassian.aws.utils.JsonUtils.fromJson(JsonUtils.java:43)
at getUserData.fetchData(getUserData.java:15)
at getUserData.main(getUserData.java:19)
Caused by: com.fasterxml.jackson.core.JsonParseException: Could not deserialize keystore
at [Source: (String)"{"baseURL":"https://bamboo.mydomain.net/agentServer/","startupTimeoutSeconds":600,"keyManagerAlgorithmName":"SunX509","trustManagerAlgorithmName":"PKIX","metaData":{"bamboo.version.specific.data.location":"s3://bamboo-agent-release-ap-se2/9.1.0-rc2/4049d092bc97001cafb7b38f7465fa9e2ce987e1","bamboo.server.fingerprint":"-3826544862034641465","aws.startedFromBamboo":"true","startupScript_0":""[truncated 4731 chars]; line: 1, column: 5135]
at com.atlassian.aws.utils.JsonUtils$KeyStoreDeserialiser.deserialize(JsonUtils.java:107)
at com.atlassian.aws.utils.JsonUtils$KeyStoreDeserialiser.deserialize(JsonUtils.java:82)
at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:313)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
at com.atlassian.aws.utils.JsonUtils.fromJson(JsonUtils.java:43)
at getUserData.fetchData(getUserData.java:15)
at getUserData.main(getUserData.java:19)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:404)
at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:179)
at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:119)
Caused by: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2167)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
at java.base/java.security.KeyStore.load(KeyStore.java:1479)
at com.atlassian.aws.utils.JsonUtils$KeyStoreDeserialiser.deserialize(JsonUtils.java:101)
... 18 more
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
at java.base/javax.crypto.Mac.getInstance(Mac.java:191)
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2145)
... 21 moreA successful Server x Agent JDK combination will print the expected user data:
# java -cp /opt/bamboo-elastic-agent/bin/atlassian-bamboo-agent-elastic-installer.jar getUserData.java
aws.agentAssemblyLocation=bamboo-agent-release-ap-se2/9.1.0-rc2/5d2d537148a7<...>215248b24f4c11c145e9d72
aws.bambooServerVersion=9.1.0-rc2
aws.startedFromBamboo=true
bamboo.instance.specific.data.locations=
bamboo.server.fingerprint=-3826544862034641465
bamboo.version.specific.data.location=s3://bamboo-agent-release-ap-se2/9.1.0-rc2/4049d092bc9700<...>e2ce987e1
startupScript_0=Cause
User data is compressed on the Server and then decompressed on the Elastic Instance using Java’s built-in compression mechanism. If the Java version on the Bamboo server is significantly newer than the Java version on the Elastic Image, the decompression process may fail. Check this JDK bug for more details.
Sample malfunctioning combinations:
Server | Agent |
|---|---|
JDK 11.0.14.1+1 | JDK 1.8.0_202 |
JDK 11.0.14.1+1 | JDK 1.8.0_275 |
| JDK 11.0.17 | JDK 1.8.0_312 |
JDK 11.0.17+8 | JDK-11.0.11+9 |
Sample working combinations:
Server | Agent |
|---|---|
JDK 1.8.0_202 | JDK 11.0.14.1 |
JDK 1.8.0_262 | JDK 1.8.0_202 |
JDK 1.8.0_262 | JDK 11.0.14.1 |
JDK 1.8.0_322 | JDK 1.8.0_202 |
JDK 1.8.0_322 | JDK 11.0.14.1 |
JDK 11.0.9.1+1 | JDK 1.8.0_202 |
JDK 11.0.9.1+1 | JDK 11.0.14.1 |
JDK 11.0.14.1 | JDK 1.8.0_312 |
JDK 11.0.14.1+1 | JDK 11.0.14.1 |
| JDK 11.0.17+8 | JDK 11.0.15+10 |
Solution
Upgrade the Java version installed on your Elastic Image, or downgrade the Java version used by the Bamboo server. Ideally, we recommended using the same JDK version on both Bamboo Server and Agents to avoid any future compatibility issues.
Last modified on Jun 7, 2024
Powered by Confluence and Scroll Viewport.