Summary

When performing a code search, no results are returned or you are prompted that search is not available.

Diagnosis

1. For bundled Elasticsearch, open the bitbucket_search.log located in BITBUCKET_HOME/shared/log. 
2. For remote Elasticsearch, refer to the logging location specified during your install.
3. Search for all shards failed.

2021-02-27T14:28:50,191][WARN ][r.suppressed             ] [bitbucket_bundled]path: /bitbucket-project/project/_search, params: {index=bitbucket-project, type=project}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.6.jar:6.8.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] 

Causes

There are multiple potential causes that have been identified for this issue.

Cause 1:  Stuck or corrupted index

As noted on our KB Bitbucket Data Center Code Search is unable to find results from some or all repositories, if there are failures to index a repository, the max retry count can be reached and the repository will not attempt a future index. (The max retry has been reached and is stored in the index itself) The failure must first be identified as you can encounter the issue again.

Cause 2: Disk space

Low disk space can trigger a failure to search and write to the index. If the data store for the shard allocation reaches a max of 95 percent, Elasticsearch will force a read only index block. This is due to the flood_stage watermark being exceeded. For details, see Elasticsearch 6.8 disk allocator:

Property: cluster.routing.allocation.disk.watermark.flood_stage

Solutions

Solution 1: Corrupted Index

To resolve this problem you first need to delete the Bitbucket index on the Elasticsearch server and then restart indexing in Bitbucket.

Step 1. Delete Elasticsearch index

The Elasticsearch REST API provides a way to delete an index using an HTTP request. 

• If you are using a Bitbucket version before 6.0 you can use the following curl command:

curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search-v1

• If you are using Bitbucket 6.0 or newer run all of the following commands:

1. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search
2. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-state
3. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-version
4. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-project
5. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-repository

Replace the <Elasticsearch-URL> with the URL where Elasticsearch is running. For example, in the case of the bundled Elasticsearch with Bitbucket, the URL would be http://localhost:7992 and these commands need to be run where the Bitbucket is hosted. Here, 7992 is the default port used by Elasticsearch when bundled with Bitbucket. If you are using your own installation of Elasticsearch (for instance because you're using Bitbucket Data Center), be sure to change the URL accordingly.

You will be prompted for a password when you send the request. You can retrieve the password to use from the buckler.yml file by copying the value of the auth.basic.password property contained in that file.

• If you are on a Bitbucket instance, this file can be found at <BitbucketHome>/shared/search/buckler/buckler.yml.
• If you are on a Bitbucket Data Center instance, the file can be found in the remote ElasticSearch node at the location elasticsearch/config/buckler/buckler.yml.

If the request is successful, no output will be returned by the command.

Step 2. Restart indexing

The Bitbucket REST API provides a way to restart indexing by making an HTTP request. You can use the following curl command to make such a request:

curl -u <username> -X POST -v -H 'Content-Type: application/json' -H 'Accept: application/json' <BaseUrl>/rest/indexing/latest/sync

where <username> is the username of a Bitbucket user with administrative privileges, and <BaseUrl> is the base URL of your Bitbucket instance. 

If the request is successful, no output will be returned by the command.

The indexing process will now begin. Depending on the number of repositories this can take a significant amount of time, but at some stage, you should be able to find code from all of your repositories.

Solution 2: Disk space

Increase disk space.