Summary

The KB describes some of the common issues that can occur for Elastic Search. Three of such issues are:

• GC overhead issues 
• Low Max file descriptors
• Low  vm.max_map_count

Environment

7.7.1

Diagnosis

• GC overhead issues 

  2020-11-17 00:09:11,962 ERROR [search-indexing:thread-1]  c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository BAUFI/xjenkins (id: 7784) with error: Unexpected error occurred when requesting index state document (id=7784) the HTTP Status code is: 429
  com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=7784) the HTTP Status code is: 429
  Also , following events were recorded as well
  {code:java}
  2020-11-16T04:10:42,184][INFO ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][260] overhead, spent [612ms] collecting in the last [1.3s]
  [2020-11-16T04:10:45,186][WARN ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][263] overhead, spent [588ms] collecting in the last [1s]
  [2020-11-16T04:11:03,190][WARN ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][281] overhead, spent [500ms] collecting in the last [1s]
  [2020-11-16T04:11:04,661][INFO ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][282] overhead, spent [545ms] collecting in the last [1.4s]
  [2020-11-16T04:11:08,198][INFO ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][285] overhead, spent [577ms] collecting in the last [1.5s]
  [2020-11-16T04:11:12,218][WARN ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][289] overhead, spent [609ms] collecting in the last [1s]
  [2020-11-16T04:11:15,582][INFO ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][292] overhead, spent [610ms] collecting in the last [1.3s]
  [2020-11-16T04:11:18,093][INFO ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled] [gc][294] overhead, spent [580ms] collecting in the last [1.5s]
  

• Low Max file descriptors

  [bitbucket_bundled] max file descriptors [10240] for elasticsearch process is too low, increase to at least [65535]
  

• Low  vm.max_map_count

  [2020-11-16T04:06:22,012][WARN ][o.e.b.BootstrapChecks    ] [bitbucket_bundled] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Cause

• GC overhead issues : This issue is caused because Elasticsearch heap memory is not properly size to the size of the content available to index.
  
  
• Low Max file descriptors:
   Elasticsearch uses a lot of file descriptors or file handles. Running out of file descriptors can be disastrous and will most probably lead to data loss. Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65,536 or higher.
  
  
• Low  vm.max_map_count: Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.
  

Solution

Low Max file descriptors: 

• In order to increase the limit for the max file descriptors, please update the following parameter on both files /etc/systemd/user.conf and /etc/systemd/system.conf

  DefaultLimitNOFILE=65536 

•  If the parameter does not exist in these files you simply need to add a new line. Update /etc/security/limits.conf with the following lines:

           *  soft    nofile  65536
           *  hard    nofile  65536
           elasticsearch   soft    nofile  65536
           elasticsearch   hard    nofile  65536
           elasticsearch   memlock unlimited

•  Do not forget to re-login into the shell in order to get the limits applied.

Low  vm.max_map_count

• Update /etc/sysctl.conf with:

  vm.max_map_count = 262144
  vm.swappiness = 1
  followed by:
  sudo sysctl -p 

GC overhead issues : 

• This can be done by increasing the heap size for the Elasticsearch JVM, Please read Elasticsearch index fails due to garbage collection overhead for more information on this.