Symptoms

Bitbucket server's mail queue keeps getting overloaded or is not able to connect to a specific mail server. This causes each message to take a long time to leave the queue. Test emails will still work.

Diagnosis

Looking at atlassian-bitbucket-mail.log logs, we can see the mail queue contantly full:

ERROR [Caesium-1-4]  bitbucket.mail-log Email could not be queued: mail queue is full;

Taking thread dumps while the problem is happening will reveal the mail threads are taking a long time in the method java.net.Inet6AddressImpl.lookupAllHostAddr:

java.lang.Thread.State: RUNNABLE
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
	at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
	at java.net.InetAddress.getLocalHost(Unknown Source)
	- locked <0x00000007802e1cc8> (a java.lang.Object)
	at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:235)
	- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:646)
	- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
	at javax.mail.Service.connect(Service.java:295)
	- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
(...) 

Cause

The IPv6 lookup method can enter an infinite loop when a certain race condition occurs. This occurs infrequently, however can cause deadlocks where some threads end up blocked.

Please check the Java bug: https://bugs.openjdk.java.net/browse/JDK-8158038 for more information.

Resolution

• Add the following parameter int the "<bitbucket-install>/bin/_start-webapp.sh" (.bat in windows) file under JVM_SUPPORT_RECOMMENDED_ARGS:

  -Djava.net.preferIPv4Stack=true
  

  With the parameter stated above set to true, Java will not execute IPv6 lookup methods and the problem should be avoided.

  
  

• Restart Bitbucket