Problem

Windows sometimes will exhaust all ephemeral ports (default of 5000 or 16384 depending on O/S version) when an enterprise application is hosted on the machine. This is most common when you have a database server hosted on the Windows machine, but can also happen when applications such as Confluence or Jira are installed. 

When all ephemeral ports are exhausted, communications on the node as a whole will be at a standstill until more connections slowly become available as they are recycled. When this happens, it may not even be possible to connect to the machine via RDP to troubleshoot. This also means the hosted application will be intermittently or fully unavailable to the user base until the machine recovers.

The following appears in the atlassian-confluence.log

java.net.SocketException: No buffer space available (maximum connections reached?)

Diagnosis

Environment

• Window Server

Diagnostic Steps

• To check the total number of allowed ephemeral ports via command line:

  netsh int ipv4 show dynamicportrange tcp

  
  

  • Example Result showing 16384 total available ephemeral ports: 
  • 

• A full example error in the atlassian-confluence.log files, showing problems connecting to the database due to ephemeral port exhaustion: 

  WARN [C3P0PooledConnectionPoolManager[identityToken->15dbdf89w41cuy21sy4wz2|ab4738d]-HelperThread-#0] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@23b61339 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
  java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
  	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:774)
  	at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:688)
  ...
  Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
  	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523)
  ...
  Caused by: java.io.IOException: No buffer space available (maximum connections reached?): connect, socket connect lapse 0 ms. wikidb.example.com/10.2.3.4 1521 0 1 true
  	at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209)
  	at oracle.net.nt.ConnOption.connect(ConnOption.java:161)
  	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470)
  	... 17 more
  Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
  	at sun.nio.ch.Net.connect0(Native Method)
  	at sun.nio.ch.Net.connect(Net.java:454)
  	at sun.nio.ch.Net.connect(Net.java:446)
  	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
  	at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
  	at oracle.net.nt.TimeoutSocketChannel.<init>(TimeoutSocketChannel.java:81)
  	at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:169)
  	... 19 more

Cause

Ephemeral ports are a range of ports that are used for outgoing communications over the TCP/IP network protocol. By default, when an outgoing connection is finished, the port that is associated with the connection is put into a TIMED_WAIT state for two minutes. This enables any lingering packets on the network to be ignored.

Some applications and services such as web, DB, or proxy servers can be very "chatty" and might actually use all ports in a two-minute time period, before they can by recycled.

Ephemeral Port exhaustion affects all versions of Windows, not just the older versions. The Windows defaults are not high enough for high-volume web servers such as Confluence & Jira, DB servers, or proxies. This can also occur in linux machines, but is less likely to happen from what we have seen due to the differences in the way the OS handles connections.

Confluence 6.x differs from 5.x and before, in that the Synchrony service is used for Collaborative Editing. This requires additional (websockets or xhr) connections to be maintained while editing, on top of the normal connections a browser makes to Confluence.

When you have a large environment, or highly-used Confluence instances, more ephemeral ports will be needed to accommodate the increased load. This is not specific to Confluence, rather any highly-used application which requires multiple connections.

Resolution

We recommend increasing to 65534 ports, and lowering the recycle timer as you have now done, in order to both allow for more ephemeral connections as well as a faster recycling of unused connections. 

1. Start Registry Editor.
2. Backup the registry

3. Browse to, and then click the following key in the registry: 

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

4. On the Edit menu, click New, DWORD Value, and then add the following registry value to increase the number of ephemeral ports that can by dynamically allocated to clients: 

   Value name
   MaxUserPort
   Value data
   <Enter a DECIMAL value of 65534 here>

5. On the Edit menu, click New, DWORD Value, and then add the following registry value to reduce the length of time that a connection stays in the TIME_WAIT state when the connection is being closed. While a connection is in the TIME_WAIT state, the socket pair cannot be reused: 

   Value name
   TcpTimedWaitDelay
   Value data
   <Enter a DECIMAL value of 30 here>

6. Close the Registry Editor.
7. Reboot the machine for the changes to take effect

8. Confirm the changes after the restart using this command: 

   netsh int ipv4 show dynamicportrange tcp 

Reference Material:

• https://support.microsoft.com/en-us/help/3014399/various-network-and-computer-issues-occur-when-tcp-ephemeral-ports-are
• https://support.microsoft.com/en-us/help/196271/when-you-try-to-connect-from-tcp-ports-greater-than-5000-you-receive-t
• http://dbaktiar-on-java.blogspot.com/2010/03/hudson-shows-buffer-space-available.html
• http://support.pitneybowes.com/VFP06_KnowledgeWithSidebarTroubleshoot?id=kA280000000PEE1CAO&popup=false&lang=en_US