Re-Indexing fails on Jira Data Center 8.7.1 due to tasks can not be cleaned too early
Symptoms
When trying to re-index JIRA, the below error is displayed:
Task Not Found
Errors
A task could not be found for the given task id
This happens even in Safe mode and the following appears in the atlassian-jira.log
:
2020-07-22 09:26:07,663+0000 JiraTaskExecutionThread-2 ERROR username 564x2242x1 1tsxpsv X.X.X.X,X.X.X.X /secure/admin/IndexReIndex!reindex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All FAILED. Indexer: SharedEntityIndexManager: paths: []
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
at com.atlassian.jira.task.TaskManagerImpl.withTaskMap(TaskManagerImpl.java:132)
at com.atlassian.jira.task.TaskManagerImpl.refreshTaskInTaskCache(TaskManagerImpl.java:573)
at com.atlassian.jira.task.TaskManagerImpl.onProgressMade(TaskManagerImpl.java:481)
at com.atlassian.jira.task.TaskProgressAdapter.makeProgress(TaskProgressAdapter.java:44)
at com.atlassian.jira.task.context.TaskProgressPercentageContextSink.updateProgress(TaskProgressPercentageContextSink.java:41)
at com.atlassian.jira.task.context.CompositeSink.updateProgress(CompositeSink.java:36)
at com.atlassian.jira.task.context.PercentageContext$Progress.update(PercentageContext.java:64)
at com.atlassian.jira.task.context.PercentageContext.setName(PercentageContext.java:34)
at com.atlassian.jira.sharing.index.DefaultSharedEntityIndexManager.reIndex(DefaultSharedEntityIndexManager.java:118)
at com.atlassian.jira.sharing.index.DefaultSharedEntityIndexManager.reIndexAll(DefaultSharedEntityIndexManager.java:93)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:66)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:27)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:18)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:533)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.atlassian.jira.task.TaskManagerImpl.withTaskMap(TaskManagerImpl.java:130)
... 22 more
Caused by: java.lang.NullPointerException
at com.atlassian.jira.task.TaskManagerImpl.lambda$withTaskMap$1(TaskManagerImpl.java:128)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Cause
This happens due to a known bug documented in JRASERVER-70663 - Tasks can be cleaned too early in Jira DC 8.7.1
Workaround
Either:
- Disable the feature by setting the "jira.dc.cleanup.cluser.tasks.disabled" feature flag and restart the node. You can follow our documentation to learn how to set it.
- Set the cluster.task.cleanup.offline.node.threshold property to a high value, e.g. 1800000 (which equals to 30 minutes).
- Remember to remove this override before upgrading Jira, or else the mechanism will only clean-up tasks from nodes that are offline for ~3.5 years.
- Upgrade Jira to a version containing a fix once it's available, i.e. 8.7.2, 8.8.0 or higher.
Please note that the workaround needs to be applied to every node in the cluster, because the tasks are being removed by other nodes, not the node that's executing the task.
To make sure that the first workaround works, you can enable DEBUG logs for com.atlassian.jira.cluster.ClusterTaskCleanupService.
During the service startup (when the node starts or after a plugin system restart) you should see the "ClusterTaskCleanupService is disabled" message logged, instead of the "Registering ClusterTaskCleanupService" message if the feature flag is still enabled.