Confluence Stuck Threads due to oracle.net.nt.TimeoutSocketChannel.read
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
It has been observed that the Confluence application experiences multiple stuck threads on method oracle.net.nt.TimeoutSocketChannel.read
As a result, the application's performance is adversely affected.
Environment
- Oracle JDBC driver version 12.2.0.0.0 and later
Diagnosis
If Confluence is impacted by this issue, we would observe the following stack for multiple stuck threads:
15-Jun-2023 00:04:29.587 WARNING [Catalina-utility-1] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio2-8090-exec-457 url: /display/CTH/xys, /pages/viewpage.action; user: gobster] (id=[1313082]) has been active for [60,841] milliseconds (since [6/15/23 12:03 AM]) to serve the same request for [https://confluence.com/display/CTH/xyz] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [66] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at java.base@11.0.19/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base@11.0.19/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base@11.0.19/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base@11.0.19/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base@11.0.19/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base@11.0.19/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:353)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:910)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
at oracle.jdbc.driver.OraclePreparedStatement.executeLargeUpdate(OraclePreparedStatement.java:3865)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3845)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1061)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:135)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:135)
...
...This indicates that there is an issue with the database connection itself, causing all Confluence connections using JDBC to get stuck at the same point (oracle.net.nt.TimeoutSocketChannel.read)
Cause
According to the following Oracle Document, the JDBC Driver version 12.2.0.1 and above may encounter failures with the error: "IO Error: Socket Read Interrupted."
The document (attached PDF below) explains that the 12.2.0.1 and above, JDBC driver uses Java NIO calls in blocking mode, which can be impacted by any interrupt() calls made by the application. This differs from previous versions of the JDBC driver that used stream-based I/O API calls unaffected by calls to interrupt() . Please note that this is a deliberate and intentional change beginning in the Oracle 12.2.0.1 JDBC driver, rather than a bug.
Further details can be found in Oracle Support website, under document:
Solution
With Oracle JDBC driver version 12.2.0.x and above, set the following connection property: oracle.jdbc.javaNetNio=false (in the startup parameters).
- Add the following System Property to add in the
setenv.shfile. (Check the Configure System Properties document for more information on how to add the system properties. )
This change will revert the driver behavior so it does not blockinterrupt()calls in Java.-Doracle.jdbc.javaNetNio=false - Restart the Confluence will fix the issue.