Insight producing "InvalidCacheLoadException: loadAll failed to return a value for xxx" or "This attribute needs to be indexed" errors
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After upgrading Insight, we start noticing missing objects from the Insight Schema and also not showing up as options to be selected on Insight Custom Fields, and/or we cannot change the Attribute options, such as cardinality.
InvalidCacheLoadException
When triggering an Insight Reindex, we see the following errors on the atlassian-jira.log:
2021-05-06 14:27:55,871 insight-InsightThreadGroup-worker-thread-6 ERROR admin 863x2041x1 1gmljw3 127.0.0.1 /rest/insight/1.0/index/reindex/start [i.r.i.index.model.ObjectIndexImpl] Index not completed successfully. objectIds: ([1, 2, 3, 4, 5])
com.google.common.cache.CacheLoader$InvalidCacheLoadException: loadAll failed to return a value for 3
at com.google.common.cache.LocalCache.getAll(LocalCache.java:4025)
at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:4972)
at io.riada.insight.index.model.InsightIndexBase.loadAll(InsightIndexBase.java:81)
at io.riada.insight.index.model.ObjectIndexImpl.reindexObjects(ObjectIndexImpl.java:318)
at com.riadalabs.jira.plugins.insight.services.core.index.ReindexServiceImpl$ReindexObjectsJob.executeTask(ReindexServiceImpl.java:543)
at com.riadalabs.jira.plugins.insight.services.core.index.ReindexServiceImpl$ReindexObjectsJob.executeTask(ReindexServiceImpl.java:520)
at com.riadalabs.jira.plugins.insight.services.core.multithreadservice.InsightServiceJob.call(InsightServiceJob.java:42)
at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)- The "objectIds" list shows all of the Object IDs that Insight was attempting to re-index at that specific operation (in the customer situation, the list may be way longer).
- The "return a value for" will indicate the Object ID where Insight first encountered problems from the above list.
This attribute must be indexed
When attempting to edit the configuration of an Object-referencing attribute we see:
Environment
This problem has been detected on Insight major versions 8.6.x and 8.7.x.
Cause
Both errors mentioned here may be caused by having an attribute that must be indexed, such as the 'Label' or an 'Object-Referencing' attribute, not being marked to be indexed by Insight.
Since this is not an operation that is permitted through the UI, this misconfiguration can be caused by an incomplete upgrade task by Insight, or direct database tampering.
Attributes that must be indexed
Although the 'Label' and an Object type attribute, are the main causes for the error messages above, there are other attributes that Insight expects to be indexed, and cannot be disabled through the UI, that we should also take note of if running into this scenario, those being:
- The 4 fields that are created automatically for each object type (Key, Created, Updated & Label field).
- Any attribute that references another object.
Diagnosis
To diagnose this problem, we might look for the above-mentioned errors in the Jira logs, or run the below query on the Jira database:
SELECT "ID", "NAME", "REMOVABLE", "TYPE", "LABEL", "INDEXED" FROM "AO_8542F1_IFJ_OBJ_TYPE_ATTR" WHERE ("REMOVABLE" = 'false' OR "TYPE" = 1) AND ("INDEXED" != 'true' OR "INDEXED" is NULL);SELECT ID, NAME, REMOVABLE, TYPE, LABEL, INDEXED FROM AO_8542F1_IFJ_OBJ_TYPE_ATTR WHERE (REMOVABLE = false OR TYPE = 1) AND INDEXED != true;
The REMOVABLE column informs which are the 4 default fields for an Object Type, and TYPE 1 is the type assigned to an attribute that references an object.
If any rows are returned, proceed with the Solution steps below.
If none rows are returned, please refer to another solution:
Solution
To fix this problem in the affected environment, since we cannot enable or disable the indexing of Label field (or any of the other fields mentioned at the Diagnostics section) through the Insight UI, we will need to apply the changes directly to the database, for all the required fields that are not yet indexed.
Important Note
The default behavior for Insight is to index all attributes with the exception of the 'textarea', which is a very costly field to index.
For this reason, we've included two options on the instructions below, one to only configure the necessary fields to be indexed, and the other to set all attributes to be indexed with the exception of the 'textarea' ones.
To better understand which is the option that you'd like to follow, you can check the explanation around indexing provided by Adding attributes to object types
"The Indexed feature on every attribute essentially helps you to retrieve your search results faster when you do a basic search or an advanced search using IQL anywhere (e.g, post functions, imports, custom fields etc). So the recommended practice is that for all IQL queries used at these multiple places, you should index your attributes.
If your attribute is indexed, Insight will not retrieve results from the database every time but instead retrieve them from an index/cache which yields results quicker. However, this comes at the cost of a higher memory consumption."
Before proceeding with the steps below, we always recommend that you create a native database backup, in case the changes need to be rolled back.
Detailed steps:
- Access the Jira database.
- Run one of the queries below to adjust the indexing flag for the Insight attributes, depending on which path you'd like to follow, and database type.
Set only the necessary attributes to be indexed:
For Postgres:UPDATE "AO_8542F1_IFJ_OBJ_TYPE_ATTR" SET "INDEXED" = 'true' WHERE ("REMOVABLE" = 'false' OR "TYPE" = 1) AND ("INDEXED" != 'true' OR "INDEXED" is NULL);MySQL:UPDATE AO_8542F1_IFJ_OBJ_TYPE_ATTR SET INDEXED = true WHERE (REMOVABLE = false OR TYPE = 1) AND INDEXED != true;Set all attributes but 'textarea' types to be indexed:
For Postgres:UPDATE "AO_8542F1_IFJ_OBJ_TYPE_ATTR" SET "INDEXED" = 'true' WHERE "DEFAULT_TYPE_ID" != 9 AND ("INDEXED" != 'true' OR "INDEXED" is NULL);MySQL:UPDATE AO_8542F1_IFJ_OBJ_TYPE_ATTR SET INDEXED = true WHERE DEFAULT_TYPE_ID != 9 AND INDEXED != true;
- Commit the changes if necessary.
- On the Jira UI, go to Jira administration (gear icon) > Manage Apps Indexing Insight.
- Run a Clean Insight Reindex
- Repeat the Insight Re-Index on each of your nodes if you are in a DC environment.
No downtime should be necessary here.