Connection reset while Directory Sync with Jira due to longer keep-alive
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
While logging in to Confluence or during directory synchronisation we see the following errors in the <Confluence-Home>/logs/atlassian-confluence.log file:
2020-10-09 22:34:22,245 ERROR [http-nio-8090-exec-18] [crowd.manager.application.ApplicationServiceGeneric] authenticateUser Directory 'JIRA Server' is not functional during authentication of 'user1'. Skipped.
.
2020-10-09 22:34:22,246 WARN [http-nio-8090-exec-18] [atlassian.confluence.user.ConfluenceAuthenticator] authenticate OperationFailedException caught while authenticating user <user1>.
-- url: /rest/api/content | traceId: f0283aa5b096c112
com.atlassian.crowd.exception.runtime.OperationFailedException
at com.atlassian.crowd.embedded.core.CrowdServiceImpl.convertOperationFailedException(CrowdServiceImpl.java:676)
at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:76)
at sun.reflect.GeneratedMethodAccessor2902.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
.
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)The issue is intermittent and sometime users are able to login and directory sync is happening successfully as well.
Environment
This is usually seen when Confluence is connected to Jira and Jira is running behind a reverse proxy. Especially noted in AWS/ELB environments with Jira on TLS.
Diagnosis
Enable DEBUG logging for HTTP traffic between Confluence and Jira.
- In Confluence, navigate to General Configuration > Logging and Profiling
- Under Class/Package Name add org.apache.http and set Level to DEBUG
- Select Save from the bottom of the page
This will give us more verbose logging and we can see traffic like follows,
2020-08-10 21:11:57,724 DEBUG [http-nio-8090-exec-22] [http.client.protocol.RequestAddCookies] process CookieSpec selected: default
.
2020-08-10 21:11:57,726 DEBUG [http-nio-8090-exec-22] [http.client.protocol.RequestAuthCache] doPreemptiveAuth Re-using cached 'basic' auth scheme for https://jira.com:443
.
2020-08-10 21:11:57,730 DEBUG [http-nio-8090-exec-22] [http.impl.conn.DefaultManagedHttpClientConnection] setSocketTimeout http-outgoing-1067525: set socket timeout to 0
2020-08-10 21:11:57,730 DEBUG [http-nio-8090-exec-22] [http.impl.conn.DefaultManagedHttpClientConnection] setSocketTimeout http-outgoing-1067525: set socket timeout to 600000
.
2020-08-10 21:11:57,730 DEBUG [http-nio-8090-exec-22] [http.impl.execchain.MainClientExec] execute Executing request POST /rest/usermanagement/1/authentication?username=user1 HTTP/1.1
.
2020-08-10 21:11:57,734 DEBUG [http-nio-8090-exec-22] [org.apache.http.headers] onRequestSubmitted http-outgoing-1067525 >> Via: 1.1 localhost (Apache-HttpClient/4.5.5 (cache))
.
2020-08-10 21:11:57,739 DEBUG [http-nio-8090-exec-22] [org.apache.http.wire] wire http-outgoing-1067525 << "[read] I/O error: Connection reset"With longer keep-alive settings, org.apache.http library runs into issues on establishing the next connection. Keep-alive headers are being returned from reverse proxy - so we will want to look at its configuration.
Solution
Reduce the keep-alive settings to a reasonable value like 5 seconds in reverse proxy in front of Jira. For AWS ELB, we can use this guide.
In some instances the keep-alive is already reasonably set and you will have to decrease the AWS ELB timeout to a reasonable value like 5 seconds.