Summary

Jira 9 index is incompatible with Jira 8's, so after the upgrade, a full reindex must be kicked off and finished before Jira 9 is usable — this adds up to the actual "upgrade downtime" for the end-user perspective (i.e. even if Jira's technically up, having no Index means it's not really usable yet). Also, as this is a Platform Release upgrade, no ZDU upgrade's available , and a cluster downtime is required anyway.

Jira 9 index is also more reliable than Jira 8, which implies more data (or metadata) having to be created and read from the database to ensure this higher reliability. This extra data/metadata creation is also performed during the first Jira 9 startup upgrade tasks, also adding more time to the "upgrade downtime" (though considerably less than the full reindex).

This article offers an alternative to greatly reduce the upgrade downtime by:

1. Manually running some DB changes prior to the upgrade
2. Performing the full reindex in an auxiliary temporary instance with production-copied data

These tasks can be performed shortly before the upgrade maintenance window while Jira 8 remains fully operational and in use by end-users.

Environment

This article applies for the following upgrades:

* Hypothetical ".99" versions just to illustrate the version range.

Solution

General steps overview

Please see the main steps and their subtasks for this procedure. You will find detailed explanations and links to other documentation on how you are going to perform each of the steps in the following sections.

1. Manually run DB inserts into Prod DB
2. Performing the full reindex in an auxiliary temporary instance (Steps to be performed just prior to actual the Downtime)
   1. Copy the Production DB to a temporary new DB instance
   2. Manually run DB inserts (same as step #1)
   3. Overwrite some external-related data
   4. Setup a temporary Jira instance
   5. Overwrite some configs
   6. Startup auxiliary Jira and upgrade to Jira 9
   7. Kick-off full reindex and copy snapshot
3. Running Upgrade Procedure Production Jira (Requires downtime)
   1. Set configs to bypass automatic reindex
   2. Upgrade Jira in one node
   3. Copy the index snapshot to Production and restart
   4. Validate and upgrade the other nodes
   5. Change configs and restart Production Jira

Detailed steps

1. Manually run DB inserts into Prod DB

On the Production database, you may run these commands to insert the missing records into the 3 _version tables that Jira 9 requires (and Jira 8 didn't, so there may be lots of "missing records"):

INSERT INTO schema.worklog_version
  SELECT id, issueid, updated, 1, 'N'
  FROM schema.worklog
  LEFT JOIN schema.worklog_version
    ON worklog_version.worklog_id = worklog.id
  WHERE worklog_version.worklog_id is null;

INSERT INTO schema.issue_version
  SELECT id, null, updated, 1, 'N'
  FROM schema.jiraissue
  LEFT JOIN schema.issue_version
    ON issue_version.issue_id = jiraissue.id
  WHERE issue_version.issue_id is null;

INSERT INTO schema.comment_version
  SELECT id, issueid, updated, 1, 'N'
  FROM schema.jiraaction
  LEFT JOIN schema.comment_version
    ON comment_version.comment_id = jiraaction.id
  WHERE actiontype = 'comment'
    AND comment_version.comment_id is null;

You may need to remove or replace the schema to each table, depending on your database setup.

From our benchmarks, we estimate around 2 minutes to insert 10,000,000 rows, though times may vary depending on DB load and specs.
Even though the production cluster will still run the upgrade task that serves the same purpose it will have no work to perform and will conclude almost immediately.
For details about upgrade time values on different databases, please check the section Upgrade process on this page.

2. Performing the full reindex in an auxiliary temporary instance

2.a. Copy the Production DB to a temporary new DB instance

Create a copy of Jira database. How to do that depends on the engine used. Please follow the engine-specific documentation to complete this step.

2.b. Manually run DB inserts (same as step #1)

This is an optional step — you may let these inserts be executed by Jira itself when it's upgrading or do it manually like in step #1 in Prod.

2.c. Overwrite some external-related data

When copying data from production to a non-prod environment, we advise overwriting some data to prevent users from inadvertently navigating from the non-prod instance to the Prod instance (like following links that point to Production URL).

These data overwriting steps are described in our Creating a test environment for Jira article.

For the purpose of this index generation, though, it's important to have data as close as possible to Production's, so you may overwrite only some integration-specific data like WebHooks, DVCS configs, etc.

These are some overwrites you can perform on the Production DB copy once it's restored to a temporary DB instance:

delete from clusternode;
delete from clusternodeheartbeat;
delete from serviceconfig where clazz='com.atlassian.jira.service.services.mail.MailFetcherService'
update "AO_4AEACD_WEBHOOK_DAO" set "ENABLED" = 'f'; /* OR, if it fails: "ENABLED" = 0; */
update "AO_E8B6CC_PROJECT_MAPPING" set "REPOSITORY_URI" = 'https://any.invalid.hostname';

And the commands listed on Remove an application link from Jira server using SQL.

2.d. Setup a temporary Jira instance

With the DB copied, you may now work on creating a temporary Jira instance to run the upgrade and the full reindex on the copied DB.

Again you can rely on the steps outlined on Creating a test environment for Jira article:

Replicate Jira

To replicate Jira, make a copy of your Jira installation and point it to your test database.

1. Copy your entire production installation directory to your test server. 
2. Copy your entire production home directory to your test server. 
3. Edit <installation-directory>/atlassian-jira/WEB-INF/classes/jira-application.properties to point to your test home directory. 
   CLUSTER  For DC, make this change on every node.

4. Edit <home-directory>/dbconfig.xml or  <installation-directory>/server.xml (older versions) to point to your test database. 

Manage shared home directory

1. Copy the production shared home directory to the test server. 
2. Edit <local-home-directory>/cluster.properties to point to your test shared home directory. Make this change on every test node. 

For simplification, the shared-home folder of the temporary instance can even be configured in the local filesystem as we won't even spin up other nodes — just a single node to run the upgrade and full reindex.

2.e. Overwrite some configs

Make sure you have a jira-config.properties file in the new instance's jira-home with at least this line below:

upgrade.reindex.allowed=false

Also, add these flags to the JVM startup properties:

-Dcom.atlassian.jira.startup.allow.full.reindex=false 
-Dcom.atlassian.jira.startup.rebuild.local.index=false
-Datlassian.notifications.disabled=true
-Datlassian.mail.senddisabled=true
-Datlassian.mail.fetchdisabled=true
-Datlassian.mail.popdisabled=true

These properties are usually set through the setenv.sh/bat or in the service configuration:

#
#  Occasionally Atlassian Support may recommend that you set some specific JVM arguments.  You can use this variable below to do that.
#
JVM_SUPPORT_RECOMMENDED_ARGS="-Dcom.atlassian.jira.startup.allow.full.reindex=false 
-Dcom.atlassian.jira.startup.rebuild.local.index=false -Datlassian.notifications.disabled=true -Datlassian.mail.senddisabled=true -Datlassian.mail.fetchdisabled=true -Datlassian.mail.popdisabled=true"

Note all these configs should be present both in the Jira 8 and Jira 9 files.

2.f. Startup auxiliary Jira and upgrade to Jira 9

You may now startup Jira on the temporary node. It should still be Jira 8, just like Production.

It's advised you perform a full reindex as soon as the node comes up.

Then, you may proceed to upgrade this auxiliary node to Jira 9 following our upgrade guides:

• Jira 9.4 Long Term Support release upgrade guide
• Practical guide on upgrading Jira Data Center to version 9 and LTS

Since this is a non-prod instance with Production data, keep in mind the system properties from step 2.e should also be configured when bringing up Jira 9 (you may need to re-apply them after the upgrade and before bringing Jira 9 up for the first time on this temporary instance).

2.g. Kick off full reindex and copy snapshot

Once Jira's upgraded following the guides above and all incompatible apps have also been upgraded to compatible versions and re-enabled, you may kick off a full reindex. 

[c.a.j.index.request.DefaultReindexRequestManager] Re-indexing finished

If Jira's Data Center licensed and configured with a shared-home in the cluster.properties, the index snapshot should've been created in the /<shared-home>/caches/indexesV2/snapshots/ folder.

If Jira's Server licensed or a Data Center without the cluster.properties, a snapshot must be manually created through System/Indexing and by setting the Index Recovery setting to a daily schedule in a near time in the future. The index snapshot will be created as scheduled in the /<jira-home>/caches/indexesV2/snapshots/ folder.

Once the index snapshot's created, you may bring down this instance and the DB to avoid inadvertent usage. Also copy the index snapshot to a safe location as backup.

3. Upgrade Production Jira

This is now during the Production maintenance window with Jira downtime.

Follow the same upgrade steps described in the Practical guide on upgrading Jira Data Center to version 9 and LTS.

3.a. Set configs to bypass automatic reindex

You should add these System Properties to the Jira 9 setenv.sh/bat or service parameters:

-Dcom.atlassian.jira.startup.allow.full.reindex=false 
-Dcom.atlassian.jira.startup.rebuild.local.index=false
-Dcom.atlassian.jira.startup.max.age.of.usable.index.snapshot.in.hours=178

As well as the jira-config.properties:

upgrade.reindex.allowed=false

3.b. Upgrade Jira in one node

In this step, you will run the upgrade procedure described in the linked document below. Jira will be upgraded without a compulsory re-indexing step with the help of the above parameters. Using JQL/Boards etc will not be possible as you will start the upgraded node without Jira 9 index files but administrative actions can be performed.

Perform the upgrade steps 1 to 8 in  Practical guide on upgrading Jira Data Center to version 9 and LTS document. Don't kick full foreground indexing.

Remember to disable incompatible apps before bringing down Jira 8 in Production, and stop all other nodes before doing so as described in the document. In step 8, upon Jira 9's successful startup, you may update the incompatible apps and enable them. Be sure the versions of the applications are the same as the applications on the temporary instance. Then, stop Jira on that node.

3.c. Copy the index snapshot to Production and restart

You may now copy the index snapshot generated by the temporary instance into the /<shared-home>/caches/indexesV2/snapshots/ folder.

You may start up Jira 9 on the first node again and it should pick up the index snapshot from the shared home. It'll re-index the "delta" of changes from the snapshot until the most recent operation in the database in multiple threads (Jira 8 would do this in a single thread), thus the closer the DB copy and index snapshot creation are to the Production upgrade, the better.

[c.a.jira.index.DefaultIndexFetcher] [INDEX_FETCHER]  Fresh enough snapshot: IndexSnapshot_10300_220225-080500.tar.sz exists. Jira will try to use it during startup.

[c.a.jira.cluster.DefaultClusterManager] Current node: mycluster1 Done recovering indexes from snapshot found in shared home.

3.d. Validate and upgrade the other nodes

You may now perform JQL searches in Production to confirm the Issues are showing up on filter results. This indicates the Index is present and operational.

You may now perform the upgrade on all the other nodes and they'll automatically pick up the same index snapshot and process the delta.

Alternatively, you may also stop this node again and copy the /local-home/caches/indexesV2 entire folder to the other nodes to speed up even more the startup as the other nodes won't need to process the delta.

Bring the node up again after copying the indexesV2 folder from it, if you choose to do so, then upgrade the others.

3.e. Change configs and restart Production Jira

Once the entire cluster is upgraded, we advise removing these configs so Jira 9 benefits from its default index startup strategy:

-Dcom.atlassian.jira.startup.allow.full.reindex=false 
-Dcom.atlassian.jira.startup.rebuild.local.index=false
-Dcom.atlassian.jira.startup.max.age.of.usable.index.snapshot.in.hours=178

This can be done in the next maintenance opportunity.