JIRA 9 UPGRADE INDEX

Index health-related changes in Jira 9.0

In Jira 9.0 we’ve introduced some changes related to improved protection of indexes health. They are mostly internal but affect the upgrade process and some parts of index-related features.

Internally enforcing the existence of index versions for all Issues, Comments, and Worklogs

Issue indexes in Jira 9.0 are no longer backward compatible. We’re enforcing the previously optional presence of issue and related entities index versions (first introduced in Jira 8.9.0) as part of the issue lifecycle. From now on all issues and related entities are expected to have an index version both in the database and in the index.

This allows us to remove the cost of extra consistency protective measures previously required to work with unversioned entities (created before upgrading to Jira version 8.9.0 and later).

Note: This is an internal change only(does not affect any APIs and does not require any external changes from users and vendors), but has some practical consequences:

• upgrade to Jira 9.0 requires downtime, the start of the first node performs some potentially lengthy operations to fully complete. See the Upgrade process section for details

• Full reindex process durations might slightly increase(our tests estimate this increase can be up to 3 minutes per 1M issues)

Upgrade process

The upgrade process includes steps to generate and reindex versions for all Issues and related entities in the system. Doing this requires performing two potentially time-consuming upgrade tasks, that will perform one time and one time only, during the startup of the first node in the cluster upgraded to 9.0. Other nodes can't be started during this time.

Generation of missing versions of all issues/comments/worklogs

The duration of this step depends on the number of unversioned entities in the database. Our load tests have shown the following times of operations:

MS SQL Server 13:

Dataset:  1M issues, 55M comments, 500k worklogs

DB hardware: db.r5.xlarge

Time to generate all versions: 9-10 minutes

MySQL 5.7.34:

Dataset: 1M issues, 55M comments, 500k worklogs

DB hardware: db.r5.xlarge

Time to generate all versions: 10 minutes

Oracle DB 19:

Dataset: 1M issues, 55m comments, 0 worklogs

DB hardware: db.r5.xlarge

Time to generate all versions: 8-9 minutes

PostgreSQL 10:

Dataset: 10M issues, 101M comments, 14M worklogs 

DB hardware: db.r5.xlarge

Time to generate all versions: 30-35 minutes

To assess the number of missing versions and estimate the time this step will take, administrator can run the following queries (the relation between the total number of objects and the duration of the upgrade is expected to change proportionally, so for data half of the example size the duration should be ~0.5 of the results).

--- number of unversioned issues
select count(*)
FROM jiraissue
         LEFT JOIN issue_version ON issue_version.issue_id = jiraissue.id
         WHERE issue_version.issue_id is null;
--- number of unversioned worklogs
SELECT count(*)
FROM worklog
        LEFT JOIN worklog_version on worklog_version.worklog_id = worklog.id
        WHERE worklog_version.worklog_id is null;
--- number of unversioned comments
select count(*)
FROM jiraaction
        LEFT JOIN comment_version ON comment_version.comment_id = jiraaction.id
        WHERE actiontype = 'comment'  AND comment_version.comment_id is null;

Full reindex

This step performs full foreground reindex according to new Jira 9.0 index requirements. As mentioned in the Internal Issue indexes versioning section, the duration of this operation will take roughly the time usual for the instance + 3 minutes per each 1M issues in the system.

Other index-related upgrade implications

Legacy index paths deleted

Because of the new structure of index-related directories (see New directory structure for indexes and index snapshots) the following directories will be deleted along with their contents during the upgrade.

• 8.0 index directories

  • cluster only: [sharedhome]/caches/indexesV1

  • [localhome]/caches/indexesV1

• pre 8.0 index directories

  • cluster only: [sharedhome]/caches/indexes

  • [localhome]/caches/indexes

Manual removal of legacy snapshots required

Legacy index path deletion will not affect index snapshots present in pre-9.0 paths.

Snapshots remaining in these paths:

• [sharedhome]/export/snapshots

• [sharedhome]/caches/

will remain at their current locations but will be ignored by any 9.0 snapshot-related operations and require manual removal.

Note: these snapshots are not compatible with Jira 9.0 indexing logic and using them to restore the index might lead to serious performance/consistency problems.

Changes in locations and naming for indexes and index snapshots

Both index file location and index snapshot locations are changed in Jira 9.0.

The path used by the Damage Recovery path to trigger restoring manually copied index snapshots ([sharedhome]/import/indexsnapshots) remains unchanged.

Also worth noting: since Jira 9.0 all index snapshots regardless of their origin are named following the same pattern:

IndexSnapshot_[uniqueNumber]_[yyMMdd-HHmmss].[tar.sz|tar|zip]

A new way to configure the maximum number of snapshots to preserve in the snapshot directory (requires manual migration of override)

At end of index snapshot creation, the snapshot directory is cleaned from older snapshots until only 3 recent snapshots remain. In pre-9.0 Jira, this value could be overridden for Jira Index Snapshot Service by the setting Number of backups to retain service property.

As of Jira 9.0, this property affects all snapshots and because of that is set independently of the Index snapshot service, as application property jira.index.snapshot.count.

Admins of instances that had this value overridden, need to migrate the overridden value manually to the new application property.

Nodes requested for snapshots no longer produce a snapshot if their index is corrupted

To prevent the spread of corrupted indexes in clusters, nodes perform their index health check when triggered to create a snapshot. This affects all snapshot-created operations:

• Admin panel request

• Scheduled index creation

• On request by node joining a cluster.

See more: Indexing inconsistency troubleshooting