Applications connected with Crowd SSO sending huge amount of requests to Crowd
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Atlassian application such as Confluence, JIRA which are connected using Crowd Single Sign-On (SSO) are observed to be generating a huge amount of requests to Crowd on every user action. Depending on the number of concurrent users, applications with a high user base may be observed to have send 10000+ calls per minute to Crowd to validate user sessions. This can result in Crowd becoming a bottleneck, resulting in negative performance impact for Crowd-connected applications downstream.
Diagnosis
Check for misconfiguration in crowd.properties
Open the file "crowd.properties" file inside the Atlassian Application connected to Crowd. They should be located in the following path:
For Confluence: CONFLUENCE/confluence/WEB-INF/classes/crowd.properties
For JIRA: JIRA/atlassian-jira/WEB-INF/classes/crowd.properties
Check the value of the parameter
session.validationintervalinside this file. This should be set to a non-zero value.
Validation through thread dumps
When the problem performance problems occur in downstream applications due to this issue, thread dumps (JIRA, Confluence) from these applications will show that most or all HTTP threads in RUNNABLE status are waiting on Crowd for authentication. Most or all threads in Crowd itself appear to be validating tokens.
Cause
If the "session.validationinterval" is being set to 0, each individual request made by users in the SSO-participating application (such as JIRA or Confluence) will result in requests to Crowd to check for the validity of the user's session. This can lead to Crowd becoming flooded and overwhelmed with repeated requests to validation user tokens. Please also note, that session validity check will significantly increase page loading time for the application, since check will be done in Tomcat filter for each request before executing application code.
The following is the description for the session.validationinterval parameter, according to the document for the crowd.properties file:
The number of minutes to cache authentication validation in the session. If this value is set to 0, each HTTP request will be authenticated with the Crowd server.
Resolution
Change session.validationinterval in crowd.properties for each downstream application to a non-zero value. By default, JIRA and Confluence ships with the value of "2", which means the session will be validated against Crowd Server every 2 minutes. A higher value means lower load on Crowd, but there is a security factor. If a Crowd administrator expires a user's session on the Crowd side, the user will still have access to downstream applications until the session needs to be re-validated.