Summary

This article covers a scenario where post Bamboo upgrade we see lots of "Connection reset by peer" errors in the Bamboo application logs . The problem is related to updating proxy settings.

Environment

The issue was seen on Bamboo 9.2.13 with Ngnix as the reverse proxy but the solution will be applicable for other Bamboo versions as well.

Diagnosis

From the Bamboo application log (atlassian-bambboo.log)  errors indicate that there is something odd with the Reverse Proxy. We see "ClientAbortException: java.io.IOException: Connection reset by peer"  :-

2024-04-19 00:06:08,332 INFO [http-nio-8085-exec-30] [RemoteAgentManagerImpl] A remote agent is loading on ueclu-dgnbld001.hce.escriptioncolo.com (127.0.0.1).
2024-04-19 00:06:08,376 INFO [http-nio-8085-exec-1] [GetJarServlet] Sending updated resource: bootstrap-with-deps, remote checksum is: a234be03 local checksum is: 4cd2754d
2024-04-19 00:06:08,382 ERROR [http-nio-8085-exec-1] [AgentServerServlet] Error copying stream
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:345) ~[catalina.jar:9.0.87]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:780) ~[catalina.jar:9.0.87]
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:677) ~[catalina.jar:9.0.87]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:380) ~[catalina.jar:9.0.87]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:358) ~[catalina.jar:9.0.87]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:103) ~[catalina.jar:9.0.87]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488) ~[commons-io-2.15.1.jar:2.15.1]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1107) ~[commons-io-2.15.1.jar:2.15.1]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1460) ~[commons-io-2.15.1.jar:2.15.1]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1085) ~[commons-io-2.15.1.jar:2.15.1]

Here's the equivalent Tomcat access logs. Observe the timestamps match the application logs:

"" 127.0.0.1 [19/Apr/2024:00:06:08 -0400] "GET /bamboo/agentServer/GetFingerprint.action?hostName=ueclu-dgnbld001.hce.escriptioncolo.com&version=4&agentUuid=bd7f3c66-d003-4d84-b840-5debbcaff168&agentId=233373707 HTTP/1.0" 200 1013 23 "" "Apache-HttpClient/4.5.14 (Java/1.8.0_312)"
"" 127.0.0.1 [19/Apr/2024:00:06:08 -0400] "GET /bamboo/agentServer/GetJar?name=bootstrap-with-deps&checksum=2721365507 HTTP/1.0" 200 3858432 114 "" "Apache-HttpClient/4.5.14 (Java/1.8.0_312)"

Cause

As we can see the connection is reset really quickly, but only after Tomcat had responded with an HTTP GET 200 meaning the "Connection reset by peer" is coming from the other side of Bamboo, either the Reverse Proxy or the Agent or something between both.

As this is happening everywhere after Bamboo had been upgraded, it appears that the reverse proxy or firewall had reached a limit on the size or volume of requests as all Agents are now trying to get an updated component at the same time .

Looks like a Broken pipe situation.In simple term, Broken Pipe means that a machine is attempting to read or write data from/to a pipe, while the machine on other end of the pipe has died or been terminated. Now, since the connection is closed, new connection should be established for further transfer of data, or else the data transfer ceases.

One of the most common cause for this issue is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment). It also happens when forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download) or when the connection is slow.

Solution

Please check the reverse proxy end and if it may be imposing limits on the number of connections. You can try Increasing Timeout of proxy Server.

Nginx has a variables named proxy_read_timeout and proxy_send_timeout which has default value of 60s, increasing this value will also minimize the chances of Broken Pipe. 

Add the below two lines in the nginx.conf file 

proxy_send_timeout                      3600s;
proxy_read_timeout                      3600s;

To know how nginx  is configured as reverse proxy for Bamboo please can refer this KB .