Summary

Bitbucket application freezes after upgrading from version 7.x.x to version 8.9.x very frequently. Users are unable to access Bitbucket UI and GIT operations start failing. 

Environment

Tested on Bitbucket Server and Data Center 8.9.x (Recently upgraded from 7.x.x)
Applies to Bitbucket Server and Data Center 8.x.x

Diagnosis

• Check if third-party plugins are on the latest version supported by Bitbucket Server or Data Center.

• A lot of HTTP threads with similar stack traces can be seen in the atlassian-bitbucket.log file.

  2023-06-01 10:31:52,968 ERROR [http-nio-7990-exec-3145] admin @P0QVQUx638x34136x38 123f9aa x.x.x.x,127.0.0.1 "POST /rest/tagdetails/1.0/projects/xyz/repos/testrepo/notes HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: TODO: Handle interrupts better
  java.lang.RuntimeException: TODO: Handle interrupts better
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.asResult(AbstractFutureResponseObserver.java:167)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.asResult(AbstractFutureResponseObserver.java:139)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFragmentResponseObserver.asResult(AbstractFragmentResponseObserver.java:147)
  	at com.atlassian.stash.internal.scm.git.mesh.BidirectionalFragmentResponseObserver.asResult(BidirectionalFragmentResponseObserver.java:66)
  	at com.atlassian.stash.internal.scm.git.mesh.GrpcPlumbingClient.call(GrpcPlumbingClient.java:68)
  	at jdk.internal.reflect.GeneratedMethodAccessor1228.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  	at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  	at com.atlassian.stash.internal.scm.git.mesh.MeshGitFreeFormCommandBuilder$1.call(MeshGitFreeFormCommandBuilder.java:102)
  	at jh.atlassian.bitbucket.tagdetails.RestService.getNotesByCommitIds(RestService.java:78)
  	at jdk.internal.reflect.GeneratedMethodAccessor1811.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
  	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
  	at com.atlassian.bitbucket.internal.ratelimit.servlet.filter.RateLimitFilter.doFilter(RateLimitFilter.java:75)
  	at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
  	at com.atlassian.troubleshooting.thready.filter.AbstractThreadNamingFilter.doFilter(AbstractThreadNamingFilter.java:46)
  
  Caused by: java.lang.InterruptedException: null
  	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
  	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver$ResponseFuture.get(AbstractFutureResponseObserver.java:233)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.asResult(AbstractFutureResponseObserver.java:151)
  	... 47 common frames omitted

• From the above stack trace notice that jh.atlassian.bitbucket.tagdetails.RestService.getNotesByCommitIds is seen and causing the similar HTTP threads to wait forever and eventually saturating all HTTP threads which causes Bitbucket application to freeze. 

Cause

The third-party plugins may not be compatible or supported with the latest Bitbucket version. For example, Tag Details And Git Notes plugin version 5.0.x is not supported on Bitbucket 8.x and causes application to freeze. 

Solution

• Disable the offending plugin and monitor the application.
• Update the third-party plugin such as Tag Details And Git Notes before upgrading Bitbucket Server/ Data Center from old version to 8.x. Once plugins are compatible, proceed with the upgrade. 
• If the issue still persist, contact Atlassian Support.