Guide to running Elasticsearch in production

If your drive tolerates parallel writes well (=SSD) having multiple primary shards per node helps scale indexing.

I'm curious why that is the case?

Is there any articles/benchmark/notes or anything that you would be willing to share?

We have considered trying out 64g+ heaps for our cluster but we are concerned about very long gc pauses impacting the search performance.

That seems high, no?

Unless you were planning on scaling 20x, it seems you could easily have half the number.

The hierarchy breakdown in the article is misleading. Lucene indexes fields, not documents. Understanding this is key. More fields == more files. Segments are per field not per ES index. A lucene index is not the same as an Elasticsearch index.

Segments are not immutable but an append only file structure. Lucene creates a new segment every time you create a new writer instance or when the lucene index is committed, which is something that happens every second in ES by default and something you could configure to something higher. So, new segment files are created frequently but not on a per document basis. ES/Lucene indeed constantly merge segment files as an optimization. Force merge is not something you should need to do often and certainly not while it is writing heavily. A good practice with log files is to do this after you roll over your indices. With modern setups, you should be reading up on index life cycle management (ILM) to manage this for you.

The notion that ES crashes at ~140 shards is complete bullshit that is based on a misunderstanding of the above. It depends on what's in those shards (i.e. how many fields). Each field has its own sets of files for storing the reverse index, field data, etc. So, how many shards your cluster can handle depends on how your data is structured and how many of them you have. This also means you needs lots of file handles.

Understanding how heap memory is used in ES is key and this article does not mention the notion of memory mapped files and even goes as far as to recommend the filesystem cache is not important!! This too is very misguided. The reality is that most index files are memory mapped files (i.e. not stored in the heap) and they only fit in memory if you have enough file cache memory available. Heap memory is used for other things (e.g. the query cache, write buffers, small data-structures with metadata about fields, etc.) and there are a lot of settings to control that which you might want to familiarize yourself with if you are experiencing throughput issues. Per index heap overhead is actually comparatively modest. I've had clusters with 1000+ shards with far less memory. This is not a problem if those shards are smallish.

The 32GB memory limit is indeed real if you use compressed pointers (which you need to configure on the JVM, ES does this by default). Far more important is that garbage collect performance tends to suffer with larger heaps because it has more stuff to do. The default heap settings with ES are not great for large heaps. ES recommends having at least (as a minimum) half your RAM available for caching. More is better. Having more disk than filecache means your files don't fit into ram. That can be OK for write heavy setups and might be OK for some querying (depending on which fields you actually query on). But generally having everything fit into memory results in more predictable performance.

GC tuning is a bit of a black art and unless you have mastered that, don't even think about messing with the GC settings in ES. It's one of those things where copy pasting some settings somebody else came up with can have all sorts of negative consequences. Most clusters that lose data do so because of GC pauses cause nodes to drop out of the cluster. Mis-configuring this makes that more likely.

CPU is important because ES uses CPU and threadpools for a lot of things and is very good at e.g. concurrently writing to multiple segments concurrently. Most of these threadpools configure based on the number of available CPUs and can be controlled via settings that have sane defaults. Also, depending on how you set up your mappings (another thing this article does not talk about) your write performance can be CPU intensive. E.g. geospatial fields involve a bit of number crunching and some more advanced text analysis can also suck up CPU.

[1] https://opendistro.github.io/for-elasticsearch/

I'm sorry - I have never seen that specific post on our forums before, but the poster was mistaken and by not correcting it at the time, we have allowed incorrect information to perpetuate.

In versions where security was a paid feature, if a trial license expired, security would remain enabled, but certain operations would be rejected for all users (per the warning text "Cluster health, cluster stats and indices stats operations are blocked")

We intentionally did not open the cluster to be world readable/writable. The administrator would be left with a cluster that was secure, but blocked access to some functions that are necessary for running a production cluster. It was up to them to explicitly upgrade to a paid license and re-enable those APIs, or downgrade to a "basic" license which required acknowledgement that security would be disabled.

An example (this is from 6.7.0 because it's the newest version I have installed at the moment, where security was not free. This was true at the time the original forum post was written - I just tested 5.2.0 as well, and the results are the same, with slightly different error messages):

License state:

   license [fc5bee69-f086-4989-a32a-5db329692363] mode [trial] - valid
  license [fc5bee69-f086-4989-a32a-5db329692363] - expired
  recovered [1] indices into cluster_state
  LICENSE [EXPIRED] ON [SATURDAY, SEPTEMBER 28, 2019].

  curl http://localhost:9200/
  {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":["ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":["ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}%

  curl -u elastic http://localhost:9200/
  Enter host password for user 'elastic':
  {
    "name" : "node1",
    "cluster_name" : "es-670",
    ...

  curl -u elastic http://localhost:9200/_cluster/health
  Enter host password for user 'elastic':
  {"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}

  blocking [indices:monitor/stats] operation due to expired license. ...

(Disclosure, as mentioned at the top, I work for Elastic)

But x-pack security is actually free from some point release in version 7. Though x-pack is not open source, just free to use. So our nginx Kibana auth is still there.

Notice how it also doesn't talk about system architecture, load balancers, disaster recovery, etc? It's because the author chose to focus the post on cluster configuration. The topic of security could be its own standalone writeup and I highly doubt that its omission is an endorsement for running an ES cluster totally exposed and unsecured.

* [Making security bits available with the ("free") basic license](https://www.elastic.co/blog/security-for-elasticsearch-is-no...) * [Releasing Kubernetes Operators with security enabled by default](https://www.elastic.co/guide/en/cloud-on-k8s/current/index.h...)

This has the effect of making most "getting started" guide setups more secure by default, which is good.

Unfortunately this is a new change and those bits are not in the Apache licensed core offering, but it's still a big improvement IMHO.

/s

https://www.elastic.co/guide/en/cloud-on-k8s/current/index.h...

Two general approaches to security:

- Upgrade to a paid Elastic cluster, and use their own feature-full security suite.

- Put a reverse proxy server in front of Elastic (like nginx), and configure that to handle security.