Hello all!
We had an interesting discussion yesterday with David about the way we
do sharding of our indices on elasticsearch. Here are a few notes for
whoever finds the subject interesting and wants to jump in the
discussion:
Context:
We recently activated row aware shard allocation on our elasticsearch
search clusters. This means that we now have one additional constraint
on shard allocation: spread copies of shards across multiple
datacenter rows, so that if we loose a full row, we still have a copy
of all the data. During an upgrade of elasticsearch, another
constraint comes into play: a shard can move from a node with an older
version of elasticsearch to a node with a newer version, but not the
other way around. This leads to elasticsearch struggling to allocate
all shards during the recent codfw upgrade to elasticsearch 2.3.5.
While it is not the end of the world (we can still server traffic if
some indices don't have all shards allocated), this is something we
need to improve.
Number of shards / number of replicas:
An elasticsearch index is split at creation in a number of shards. A
number of replica per shard is configured [1]. The total number of
shards for an index is "number_of_shards * (number_of_replicas + 1)".
Increasing the number of shards per index allow to execute read
operation in parallel over the different shards and aggregate the
results at the end, improving response time Increasing the number of
replicas allow to distribute the read load over more nodes (and
provides some redundancy in case we loose one server). As term
frequency [2] is calculated over a shard and not over the full index,
There is some black magic involved in how we shard our indices, but
most of it is documented [3]
The enwiki_content example:
enwiki_content index is configured to have 6 shards and 3 replicas,
for a total number of 24 shards. It also has the additional constraint
that there is at most 1 enwiki_content per node. This ensures a
maximum spread of enwiki_content shards over the cluster. Since
enwiki_content is one of the index with the most traffic, this ensure
that the load is well distributed over the cluster.
Now the bad news: for codfw, which is a 24 node cluster, it means that
reaching this perfect equilibrium of 1 shard per node is a serious
challenge if you take into account the other constraint in place. Even
with relaxing the constraint to 2 enwiki shards per node, we have seen
unassigned shards during elasticsearch upgrade.
Potential improvements:
While ensuring that a large index has a number of shards close to the
number of nodes in the cluster allows for optimally spreading load
over the cluster, it degrade fast if all the stars are not aligned
perfectly. There are 2 opposite solutions
1) decrease the number of shards to leave some room to move them around
2) increase the number of shards and allow multiple shards of the same
index to be allocated on the same node
1) is probably impractical on our large indices, enwiki_content shards
are already ~30Gb and this makes it impractical to move them around
during relocation and recovery
2) is probably our best bet. More smaller shards means that a single
query load will be spread over more nodes, potentially improving
response time. Increasing number of shards for enwiki_content from 6
to 20 (total shards = 80) means we have 80 / 24 = 3.3 shards per node.
Removing the 1 shards per node constraint and letting elasticsearch
spread the shards as best as it can means that in case 1 node is
missing, or during an upgrade, we still have the ability to move
shards around. Increasing this number even more might help keep the
load evenly spread across the cluster (the difference between 8 or 9
shards per node is smaller than the difference between 3 or 4 shards
per node).
David is going to do some tests to validate that those smaller shards
don't impact the scoring (smaller shards mean worse frequency
analysis).
I probably forgot a few points, but this email is more than long
enough already...
Thanks to all of you who kept reading until the end!
MrG
[1]
https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_conc…
[2]
https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.…
[3]
https://wikitech.wikimedia.org/wiki/Search#Estimating_the_number_of_shards_…
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST