Bitbucket search API results do not go beyond 999 records
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When searching for a string that happens to be present in more than 1000 files in all of Bitbucket repositories, results can be seen for only the first 1000. Subsequent results are not fetched.
Environment
Bitbucket Data Center and Server
Diagnosis
While trying to fetch results above 1000, the QueryInvalidSearchOffsetException may be seen in the response.
curl -u <BITBUCKET_USER>:<BITBUCKET_PASSWORD> -X POST -d '{"query":"linux","entities":{"code":{"start":999, "limit": 5 }}}' -H "Content-Type: application/json" <BITBUCKET_BASE_URL>/rest/search/latest/search | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 303 0 239 100 64 2987 800 --:--:-- --:--:-- --:--:-- 3787
{
"errors": [
{
"context": null,
"message": "Unable to display more results. Try adding more search terms to refine your results. (Elasticsearch error type: search_phase_execution_exception)",
"exceptionName": "QueryInvalidSearchOffsetException"
}
]
}Note: the sample commands on this page use the jq command line JSON processor for parsing and formatting the JSON output.
Cause
This is because of the Pagination settings which have set the max_result_window to only 1000 results. Read more about this in their official documentation.
Solution
Atlassian does not support the manipulation of Elasticsearch/OpenSearch directly. Therefore please perform this on a test instance first.
Increasing to index.max_result_window value to a higher number may impact the performance of the system.
We can update the max_result_window to a number high enough to accommodate the search results.
INCREASE MAX_RESULT_WINDOW TO 2000
Replace the <SEARCH_SERVER_URL> with your search instance base URL, <SEARCH_USER> with username and <SEARCH_PASSWORD> with the password. In the below example, we are increasing the value of max_result_window to be 2000.
curl -u <SEARCH_USER>:<SEARCH_PASSWORD> -X PUT "<SEARCH_SERVER_URL>/bitbucket-search/_settings" -H 'Content-Type: application/json' -d '{ "index" : { "max_result_window" : 2000 } }'
{"acknowledged":true}Confirming if the window size was changed to 2000
Replace the <SEARCH_SERVER_URL> with your search instance base URL, <SEARCH_USER> with username and <SEARCH_PASSWORD> with the password.
curl -u bitbucket -X GET "<SEARCH_SERVER_URL>/bitbucket-search/_settings" | jq .
Enter host password for user 'bitbucket':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2283 100 2283 0 0 445k 0 --:--:-- --:--:-- --:--:-- 445k
{
"bitbucket-search": {
"settings": {
"index": {
"codec": "best_compression",
"refresh_interval": "30s",
"number_of_shards": "5",
"provided_name": "bitbucket-search",
"query": {
"default_field": "content"
},
"max_result_window": "2000",
"creation_date": "1623235773828",
"analysis": {
"filter": {
"filter_remove_empty_tokens": {
"type": "length",
"min": "1"
},