Search - Wikitech
Wikitech
Search
This page covers the usage of Elastica, CirrusSearch, and Elasticsearch, in Wikimedia projects.
Wikimedia infrastructure
Data centres and PoPs
Search
For search use in Toolforge, see Help:Toolforge/Elasticsearch​. For codesearch, see mw:codesearch.
CirrusSearch components diagram
In case of emergency
Here is the (sorted) emergency contact list for Search issues on the WMF production cluster:
You can get contact information from officewiki.
Overview
This system has three components: Elastica, CirrusSearch, and Elasticsearch.
If you want to extend the data that is available to CirrusSearch, have a look here.
Elastica
Elastica is a MediaWiki extension that provides the library to interface with Elasticsearch. It wraps the Elastica library. It has no configuration.
CirrusSearch
CirrusSearch is a MediaWiki extension that provides search support backed by Elasticsearch.
Turning on/off
$wmgUseCirrus will set the wiki to use CirrusSearch by default for all searches. Doing so would break searches and search updates. You probably don't want to do this.
Configuration
The canonical location of the configuration documentation is in the CirrusSearch.php file in the extension source. It also contains the defaults. A pool counter configuration example lives in the README in the extension source.
WMF runs the default configuration except for elasticsearch server locations but overrides would live in the CirrusSearch-{common|production|labs}.php files in the mediawiki-config files.

Local Build
See docs: https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom
Install
To install Elasticsearch on a node, use role::elasticsearch::server in puppet. This will install the elasticsearch Debian package and setup the service and appropriate monitoring. It should automatically join the cluster.
Configuration
role::elasticsearch::server has no configuration.
In labs a config parameter is required. It is called elasticsearch::cluster_name and names the cluster that the machine should join.
role::elasticsearch::server use ::elasticsearch to install elasticsearch with the following configuration:
EnvironmentMemoryCluster NameMulticast GroupServers
labs2Gconfigured224.2.2.4
beta4Gbeta-search224.2.2.4deployment-elastic0[5678]
eqiad30Gproduction-search-eqiad224.2.2.5elastic10([012][0-9]*or*3[01]) (eqiad)
codfw30Gproduction-search-codfw224.2.2.7elastic20(0[0-9|1[0-9]|2[0-4])
Note that the multicast group and cluster name are different in different production sites. This is because the network is flat from the perspective of Elasticsearch's multicast and it is way easier to reserve separate IPs than it is to keep up with multicast ttl.
Files
/etc/default/elasticsearchSimple settings that must be initialized either on the command line (memory) or really early (memlockall) in execution
/etc/elasticsearch/logging.ymlLogging configuration
/etc/logrotate.d/elasticsearchLog rolling managed by logrotate
/etc/elasticsearch/elasticsearch.ymlEverything not covered above, e.g. cluster name, multicast group
Puppet
The elasticsearch cluster is provisioned via the elasticsearch module of the operations/puppet repository. Configuration for specific machines is in the hieradata folder of the same operations/puppet repository. A few of the relevant files are linked here, but a grep for 'elasticsearch' in hieradata and poking around the elasticsearch puppet module are the best ways to understand the current configuration.
Provisioning
Configuration
SSL certificates
Elasticsearch exposes HTTPS endpoints via nginx. Puppet CA is used for certificates. Certificates have to be regenerated after the first installation to ensure they contain the correct SAN entry. The procedure is documented on Puppet CA page.
WMF Production setup
Hardware
The WMF, as of September 2015, operates a single elasticsearch cluster which serves all search requests for WikiMedia sites. This is hosted on elastic{01..31}.eqiad.wmnet.
Up to date rack placement can be found in ops/puppet here
serverscoresmemorydisk
elastic1017-103132128G2x300GB SSD RAID0 *
elastic1032-105232128G2x800GB SSD RAID1
elastic2001-202432128G2x800GB SSD RAID1
Software RAID by partition is raid1 for the OS but elastic data will not survive a disk loss
Data distribution
As far as elasticsearch is concerned all of these servers are exactly the same. The hardware within the machines is not taken into account by the shard allocation algorithms. This does lead to some issues where the older machines have a higher load than the newer machines, but fixing it is not yet supported by elasticsearch.
Our largest user of elasticsearch resources is, by far, queries to enwiki. The data for enwiki is split into 7 shards with 1 master plus 3 replica. This means that each individual enwiki query is answered by 7 machines and at any given moment 28 (7*4) of the 31 machines in the cluster are serving enwiki queries. Other popular wikis are included in the table below(wikis can be listed twice, there is a 'content' index and a separate 'everything' index for each wiki):
wikiprimariesreplicastotal
itwiki9227
dewiki, enwiki7328
zhwiki, ukwiki, svwiki, nlwiki, frwiki, wikidatawiki, frwikisource, eswiki, enwikisource, ruwiki, itwiki, plwiki, jawiki, ptwiki7221
viwiki, jawiki, eswiki6218
arwiki, cawiki, commonswiki, enwiktionary, fawiki, ptwiki, ruwiki, zhwiki zhwikisource5215
Many many more wikis served by < 15 machines
Load balancing requests
MediaWiki talks to the elasticsearch cluster through LVS. Application servers talk to a single LVS ip address and this balances the requests out across the cluster. The read and write requests are distributed evenly across all available elasticsearch instances with no consideration for data locality.
Shard balance across the cluster
The production configuration sets index.routing.allocation.total_shards_per_node to 1 for all indexes. This means that each server will only contain a single shard for any given index. This combined with setting the number of shards and replicas to an appropriate number ensure that the indices with heavy query volume are spread out across the entire cluster.
The elasticsearch shard distribution algorithm is relatively naive with respect to our use case. It is optimized for having shards of approximately equal size and query volume in all indexes. The WMF use case is very different. As of September 2015 there are 6419 shards with approximately the following size breakdown:
sizenum shardspercent
x > 50gb330.5%
50gb > x > 10gb560.8%
10gb > x > 1gb130720.3%
1gb > x > 100mb111417.3%
100mb > x390960.8%
Due to this load across the cluster needs to be occasionally monitored and shards moved around. This is further discussed in #Trouble.
Indexing
CirrusSearch updates the elasticsearch index by building and upserting almost the entire document on every edit. The revision id of the edit is used as the elasticsearch version number to ensure out-of-order writes by the job queue have no effect on the index correctness. There is, as of September 2015, only one property that is not upserted as normal on every edit. The CirrusSearch\OtherIndex functionality which adds information to the commonswiki index about duplicate files only updates the commonswiki index when the duplicated file, on the other wiki, is indexed.
Realtime updates
The CirrusSearch extension updates the elasticsearch indexes for each and every mediawiki edit. The chain of events between a user clicking the 'save page' button and elasticsearch being updated is roughly as follows:
Other processes that write to elasticsearch (such as page deletion) are similar. All writes to elasticsearch are funneled through the CirrusSearch\Updater class, but this class does not directly perform writes to the elasticsearch database. This class performs all the necessary calculations and then creates the CirrusSearch\Job\ElasticaWrite job to actually make the request to elasticsearch. When the job is run it creates CirrusSearch\DataSender which transforms the job parameters into the full request and issues it. This is done so that any updates that fail (network errors, cluster maintenance, etc) can be re-inserted into the job queue and executed at a later time without having to re-do all the heavy calculations of what actually needs to change.
Batch updates from the database
CirrusSearch indices can also be populated from the database to bootstrap brand new clusters or to backfill existing indices for periods of time where updates, for whatever reason, were not written to the elasticsearch cluster. These updates are performed with the forceSearchIndex.php maintenance script, the usage of which is described in multiple parts of the #Administration section.
Batch updates use a custom job type, the CirrusSearch\Job\MassIndex job. The main script iterates the entire page table and inserts jobs in batches of 10 titles. The MassIndex job kicks off CirrusSearch\Updater::updateFromPages() to perform the actual updates. This is the same process as CirrusSearch\Updater::updateFromTitle​, updateFromTitle simply does a couple extra checks around redirect handling that is unnecessary here.
Scheduled batch updates from analytics network
Jobs are scheduled in the WMF analytics network by the search platform airflow instance to collect together various information collected there and ship it back to elasticsearch. The airflow jobs build one or more files per wiki containing elasticsearch bulk update statements, uploads them to swift, and sends a message over kafka indicating availability of new information to import. The mjolnir-msearch-daemon running on search-loader instances in the production network recieve the kafka messages, download the bulk updates from swift, and pipe them into the appropriate elasticsearch clusters. This includes information such as page popularity and ml predictions from various wmf projects (link recommendation, ores, more in the future).
Job queue
CirrusSearch uses the mediawiki job queue for all operations that write to the indices. The jobs can be roughly split into a few groups, as follows:
Primary update jobs
These are triggered by the actions of either users or adminstrators.
Secondary update jobs
These are triggered by primary update jobs to update pages or indices other than the main document.
Backend write jobs
These are triggered by primary and secondary update jobs to represent an individual write request to the cluster. One job is inserted for every cluster to write to. In the backend the jobqueue is configured to partition the jobs by cluster into a separate queues. This partitioning ensures slowdowns indexing to one cluster do not cause similar slowdowns in the remaining clusters.
ElasticaWrite
Logging
CirrusSearch logs a wide variety of data. A few logs in particular are of interest:
Administration
All of our (CirrusSearch's) scripts have been designated to run on mwmaint1002.eqiad.wmnet​.
Hardware failures
Elasticsearch is robust to losing nodes. In case maintenance is required on a node (failed disk, RAM issues, whatever...), the node can be depooled and shutdown without any further action. There is no need to synchronize this shutdowns or restart with the Search Platform team (a ping to tell us it is happening is always welcomed).
Tuning shard counts
Optimal shard count requires making a tradeoff between several few competing factors.
Quick background: Each Elasticsearch shard is actually a Lucene index which requires some amount of file descriptors/disk usage, compute, and RAM. So, a higher shard count causes more overhead, due to resource contention as well as "fixed costs".
Now, since Elasticsearch is designed to be resilient to instance failures, if a node drops out of the cluster, shards must rebalance across the remaining nodes (likewise for changes in instance count, etc). Shard rebalancing is rate-limited by network throughput, and thus excessively large shards can cause the cluster to be stuck "recovering" (rebalancing) for an unacceptable amount of time.
Thus the optimal shard size is a balancing act between overhead (which is optimized via having larger shards), and rebalancing time (which is optimized via smaller, more numerous shards). Less importantly, due to the problem of fragmentation, we also don't want a given shard to be too large a % of the available disk capacity.
Currently (01/07/2020 DD/MM/YY), in most cases we don't want shards to exceed 50GB, and ideally they wouldn't be smaller than 10GB (but note that for small indices this is unavoidable). Once our Elasticsearch cluster has 10G networking, we can increase our desired shard size, due to higher networking throughput decreasing the time required to redistribute large shards.
The final consideration is that different indices receive different levels of query volume. As of July 2020, enwiki and dewiki are hammered the hardest. This means nodes that have, for example,`​enwiki_content shards assigned to them, will receive disproportionate load relative to nodes that lack those shards. As such, we want to set our total shard count in relation to the number of servers we have available. For example, a cluster with 36 servers would ideally have a total shard count that is close to a multiple of 36. We generally like to go slightly "under" to be able to weather losing nodes. In short, we want to maintain the invariant that most servers have the same number of shards for a given heavy index, with a few servers having 1 less shard so that we have headroom to lose nodes.
In conclusion: we want shards close to 50GB, but we also need our shard count set to avoid having any particularly hot servers, while making sure the shard count isn't completely even so that we can still afford to lose a few nodes. We absolutely want to avoid having a ridiculous number of extremely tiny shards, since the thrash/resource contention would grind the cluster to a halt.
Adding new wikis
All wikis have Cirrus enabled as the search engine. To add a new Cirrus wiki:
  1. Estimate the number of shards required (one, the default, is fine for new wikis).
  2. Create the search index
  3. Populate the search index
Estimating the number of shards required
If the wiki is small then skip this step. The defaults are good for small wikis. If it isn't, use one of the three methods below to come up with numbers and add them to mediawiki-config's InitializeSettings.php file under wmgCirrusSearchShardCount.
See #Resharding if you are correcting a shard count error.
If it hasn't been indexed yet
You should compare its content and general page counts with a wiki with similarly sized pages. Mostly this means a wiki in a different language of the same type. Enwikisource and frwikisource are similar, for example. Wiktionaries pages are very small. As are wikidatawiki's. In any case pick a number proportional to the number for the other wiki. Don't worry too much about being wrong - resharding is as simple as changing the value in the config and performing an in place reindex. To count the pages I log in to tools-login.pmtpa.wmflabs then
sql $wiki SELECT COUNT(*) FROM page WHERE page_namespace = 0; SELECT COUNT(*) FROM page WHERE page_namespace != 0; sql $comparableWiki SELECT COUNT(*) FROM page WHERE page_namespace = 0; SELECT COUNT(*) FROM page WHERE page_namespace != 0;
Wiki's who's indexes are split other ways then content/general should use the same methodology but be sure to do it once per index and use the right namespaces for counts.
If it has been indexed
You can get the actual size of the primary shards (in GB), divide by two, round the result to a whole number, and spit ball estimate for growth so you don't have to go do this again really soon. Normally I add one to the number if the primary shards already total to at least one GB and it isn't a wiktionary. Don't worry too much about being wrong because you can change this with an in place reindex. Anyway, to get the size log in to an elasticsearch machine and run this:
curl -s localhost:​$port​/_stats?pretty | grep 'general\|content\|"size"\|count' | less
Count and size are repeated twice. The first time is for the primary shards and the second includes all replicas. You can ignore the replica numbers.
We max out the number of shards at 5 for content indexes and 10 for non-content indexes.
Create the index
mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki $wiki --cluster=all
That'll create the search index on all necessary clusters with all the proper configuration.
Populate the search index
mkdir -p ~/log clusters='eqiad codfw cloudelastic'for cluster in eqiad codfw cloudelastic; do mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --skipLinks --indexOnSkip --queue | tee ~/log/​$wiki​.​$cluster​.parse.log mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --skipParse --queue | tee ~/log/​$wiki​.​$cluster​.links.log done
If the last line of the output of the --skipLinks line doesn't end with "jobs left on the queue" wait a few minutes before launching the second job. The job queue doesn't update its counts quickly and the job might queue everything before the counts catch up and still see the queue as empty. If this wiki is a private wiki then cloudelastic should be removed from the set of clusters. No harm if it's included, but it will (should?) throw exceptions and complain.
Health/Activity Monitoring
We have nagios checks that monitor Elasticsearch's claimed health (green, yellow, red) and ganglia monitoring on a number of attributes exposed by Elasticsearch.
Eqiad
Codfw
Expected eligible masters check and alert
There is an icinga check that checks that the expected number of eligible masters is exactly what it is. If this deviates, we should see icinga alerts. Check
We have nagios checks that monitor Elasticsearch's claimed health (green, yellow, red) and ganglia monitoring on a number of attributes exposed by Elasticsearch.
Check the actual eligible masters with:
curl -s localhost:9200/_cat/nodes?h​=​node.role,name | grep mdi
and then compare with puppet configuration.
If an expected eligible master node is down, we can either bring it up or promote another node to a master. Also make sure the masters are spread out across the rack.
Monitoring the job queue
This information is outdated.​showJobs.php prints no output, presumably due to the switch to the Kafka Job Queue. Someone please update this with the correct commands!
The current state of the jobqueue is visible, but only for individual wikis. Enwiki is almost always the busiest index, so we can monitor the state with:
mwscript showJobs.php --wiki=enwiki --group | grep ^cirrus
Under normal operation this will output something like:
ebernhardson@mwmaint1001:~$ mwscript showJobs.php --wiki=enwiki --group | grep ^cirrus cirrusSearchIncomingLinkCount: 19028 queued; 0 claimed (0 active, 0 abandoned); 234 delayed cirrusSearchLinksUpdatePrioritized: 35 queued; 4 claimed (4 active, 0 abandoned); 0 delayed
This output is explained in the manual. Most CirrusSearch jobs are normal to see here, but there is one exception. The cirrusSearchElasticaWrite is typically created and run in-process from other jobs. The only time cirrusSearchElasticaWrite in inserted into the job queue is when the write operations cannot be performed to the cluster. This may be due to writes being frozen, it could be a network partition between the job runners and the elasticsearch cluster, or it could be that the index being written to is red.
Waiting for Elasticsearch to "go green"
Elasticsearch has a built in cluster health monitor. red means there are unassigned primary shards and is bad because some requests will fail. Yellow means there are unassigned replica shards but the masters are doing just fine. This is mostly fine because technically all requests can still be served but there is less redundancy then normal and there are less shards to handle queries. Performance may be effected in the yellow state but requests won't just fail. Anyway, this is how you wait for the cluster to "go green".
until curl -s localhost:9200/_cluster/health?pretty | tee /tmp/status | grep \"status\" | grep \"green\"; do cat /tmp/status echo sleep 1 done
So you aren't just watching number scroll by for fun I'll explain what they all mean:
{ "cluster_name" : "labs-search-project", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 4, "number_of_data_nodes" : 4, "active_primary_shards" : 30, "active_shards" : 55, "relocating_shards" : 0, "initializing_shards" : 1, "unassigned_shards" : 0 }
cluster_name
The name of the cluster.
status
The status we're waiting for.
timed_out
Has this requests for status timed out? I've never seen one time out.
number_of_nodes
How many nodes (data or otherwise) are in the cluster? We plan to have some master only nodes at some point so this should be three more than number_of_data_nodes
number_of_data_nodes
How many nodes that hold data are in the cluster?
active_primary_shards
How many of the primary shards are active? This should be indexes * shards. This number shouldn't change frequently because when the primary shards go off line one of the replica shards should take over immediately. It should be too fast to notice.
active_shards
How many shards are active? This should be indexes * shards * (1 + replicas). This will go down when a machine leaves the cluster. When possible those shards will be reassigned to other machines. This is possible so long as those machines don't already hold a copy of that replica.
relocating_shards
How many shards are currently relocating?
initializing_shards
How many shards are initializing? This mostly means they are being restored from a peer. This should max out at cluster.routing.allocation.node_concurrent_recoveries​. We still use the default of 2.
unassigned_shards
How many shards have yet to be assigned? These are just waiting in line to become initializing_shards.
See the Trouble section for what can go wrong with this.
Pausing Indexing
Various forms of cluster maintenance, such as rolling restarts, take significantly less time when the content of the indices stays static and is not being constantly written to. To support this CirrusSearch includes the concept of 'frozen' indices. When an index is frozen no ElasticaWrite jobs are executed. When an ElasticaWrite job tries to run it sees the index it wants to write to is frozen and re-inserts itself to the job queue with an exponential backoff. Individual indic
To freeze all writes to the cluster. It does not matter which --wiki=XXX you use, this is a cluster wide setting stored in an elasticsearch index. The cluster to freeze is selected with --cluster=XXX:
mwscript extensions/CirrusSearch/maintenance/freezeWritesToCluster --wiki=enwiki --cluster=eqiad
Once run all writes will backup into the job queue. The current state of the jobqueue is visible, but only for individual wikis. Enwiki is almost always the busiest index, so we can monitor the state with:
mwscript showJobs.php --wiki=enwiki --group | grep ^cirrus
To thaw out writes to the cluster re-run the initial script with the --thaw option
mwscript extensions/CirrusSearch/maintenance/freezeWritesToCluster --wiki=enwiki --cluster=eqiad --thaw
After thawing the cluster you must monitor the job queue to ensure the number of cirrusSearchElasticaWrite jobs is decreasing as expected. These jobs include very little calculation and quickly drain from the queue.
Ensure you do not leave the cluster in the frozen state for too long. As of Sept 2015 too long is six hours. In WMF production all jobs need to fit within the redis cluster's available memory. If cirrus jobs are allowed to fill up all available redis memory there will be an outage and it will not be limited to search.
In some cases (as yet unexplained) removing the "freeze-everything" document which controls the freeze fails, and the document reappears randomly. Re-creating and re-deleting this document fixes the issue. The force-unfreeze cookbook can be used for that.
Restarting a node
Elasticsearch rebalances shards when machines join and leave the cluster. Because this can take some time we've stopped puppet from restarting Elasticsearch on config file changes. We expect to manually perform rolling restarts to pick up config changes or software updates (via apt-get), at least for the time being. There are two ways to do this: the fast way, and the safe way. At this point we prefer the fast way if the downtime is quick and the safe way if it isn't. The fast way in the Elasticsearch recommended way. The safe way keeps the cluster green the whole time but is really slow and can cause the cluster to get a tad unbalanced if it is running close to the edge on disk space.
The fast way:
es-tool restart-fast
This will instruct Elasticsearch not to allocate new replicas for the duration of the downtime. It should be faster then the simple way because unchanged indexes can be restored on the restarted machine quickly. It still takes some time.
The safe way starts with this to instruct Elasticsearch to move all shards off of this node as fast as it can without going under the appropriate number of replicas:
ip=$(facter ipaddress) curl -XPUT localhost:9200/_cluster/settings?pretty -d "{ \"transient\" : { \"cluster.routing.allocation.exclude._ip\" : \"$ip\" } }"
Now you wait for all the shards to move off of this one:
host=$(facter hostname) function moving() { curl -s localhost:9200/_cluster/state | jq -c '.nodes as $nodes | .routing_table.indices[].shards[][] | select(.relocating_node) | {index, shard, from: $nodes[.node].name, to: $nodes[.relocating_node].name}' } while moving | grep $host; do echo; sleep 1; done
Now you double check that they are all off. See the advice under stuck in yellow if they aren't. It is probably because Elasticsearch can't find another place for them to go:
curl -s localhost:9200/_cluster/state?pretty | awk ' BEGIN {more=1} {if (/"nodes"/) nodes=1} {if (/"metadata"/) nodes=0} {if (nodes && !/"name"/) {node_name=$1; gsub(/[",]/, "", node_name)}} {if (nodes && /"name"/) {name=$3; gsub(/[",]/, "", name); node_names[node_name]=name}} {if (/"node"/) {node=$3; gsub(/[",]/, "", node)}} {if (/"shard"/) {shard=$3; gsub(/[",]/, "", shard)}} {if (more && /"index"/) { index_name=$3 gsub(/[",]/, "", index_name) print "node="node_names[node]" shard="index_name":"shard }} ' | grep $host
Now you can restart this node and the cluster with stay green:
sudo /etc/init.d/elasticsearch restart until curl -s localhost:9200/_cluster/health?pretty ; do sleep 1 done
Next you tell Elasticsearch that it can move shards back to this node:
curl -XPUT localhost:9200/_cluster/settings?pretty -d "{ \"transient\" : { \"cluster.routing.allocation.exclude._ip\" : \"\" } }"
Finally you can use the same code that you used to find stuff shards moving from this node to find shards moving to this node:
while moving | grep $host; do echo; sleep 1; done
Rolling restarts
Rolling restarts are done with multiple servers in parallel to speed up the operation. Servers from the same availability zone (datacenter row) are restarted at the same time to ensure that all indices still have sufficient shards. 3 nodes in parallel is conservative, 4 has been tested multiple times and works if the cluster isn't under high load.
To optimize recovery, writes are paused during restart and a sync flush is issued. Pausing writes will increase backlog in the kafka queues and writes will be dropped if writes are paused for too long (at this point ~3 hours). Writes are re-enabled between each group of restarts to allow the queue to be processed.
Each server runs multiple elasticsearch instances (2 at this point), both need to be restarted.
The full restart logic is embedded in cookbooks for the different specific use cases:
Those cookbooks are well tested on the main elasticsearch clusters, but not as much on the smaller ones (relforge, cloudelastic). The small clusters are simple enough to restart manually. Those cookbooks are idempotent, they can be stopped and restarted, servers already restarted will be ignored by the next run. To be on the safe side, stop the cookbook while they are waiting on the cluster to go back to green (context managers are used for all operations that needs to be undone, but a force kill might not leave them time to cleanup).
Example run: On one of the cumin host:
sudo -i cookbook sre.elasticsearch.rolling-restart search_eqiad "restart for JVM upgrade" --start-datetime 2019-06-12T08:00:00 --nodes-per-run 3
where:
During rolling restarts, it is a good idea to monitor a few elasticsearch specific things:
Things that can go wrong:
Cold restart
A cold restart (shutting down all nodes and restarting all of them at the same time) is governed by the following settings:
Our current understanding is that elasticsearch will wait for expected_nodes to be available, and recover anyway after recover_after_time if at least recover_after_nodes are present.
Recovering from an Elasticsearch outage/interruption in updates
The same script that populates the search index can be run over a more limited list of pages:
mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --from <YYYY-mm-ddTHH:mm:ssZ> --to <YYYY-mm-ddTHH:mm:ssZ>
or:
mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --fromId <id> --toId <id>
So the simplest way to recover from an Elasticsearch outage should be to use --from and --to with the times of the outage. Don't be stingy with the dates - it is better to reindex too many pages than too few.
If there was an outage its probably good to also do:
mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --deletes --from <YYYY-mm-ddTHH:mm:ssZ> --to <YYYY-mm-ddTHH:mm:ssZ>
This will pick up deletes which need to be iterated separately.
This is the script I have in my notes for recovering from an outage:
function outage() { wiki=$1 start=$2 end=$3 mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --from $start --to $end --deletes | tee -a ~/cirrus_log/$wiki.outage.log mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --from $start --to $end --queue | tee -a ~/cirrus_log/$wiki.outage.log } while read wiki ; do outage $wiki '2015-03-01T20:00:00Z' '2015-03-02T00:00:00Z' done < /srv/mediawiki/dblists/all.dblist
In place reindex
Some releases require an in-place reindex. Usually in-place reindex is needed if any part of mapping: fields, analyzers, etc. - has changed. This mode of reindexing creates a new index with up-to-date mappings, but does not change the source data already stored in the index, only the interpretation of the data. All reindex operations are very expensive, it can take multiple weeks to sequentially work through the wikis and clusters.
Sometime the change may be only for wikis in particular languages so only those wikis will need an update. In any case, this is how you perform an in-place reindex:
function reindex() { cluster=$1 wiki=$2 mkdir -p "$HOME/cirrus_log/" reindex_log="$HOME/cirrus_log/$wiki.$cluster.reindex.log" if [ -z "$cluster" -o -z "$wiki" ]; then echo "Usage: reindex [cluster] [wiki]" return 1 fi TZ=UTC export REINDEX_START=$(date +%Y-%m-%dT%H:%m:%SZ) echo "Started at $REINDEX_START" > $reindex_log mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki $wiki --cluster $cluster --reindexAndRemoveOk --indexIdentifier now 2>&1 | tee -a $reindex_log && \ mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --deletes | tee -a $reindex_log && \ mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --queue | tee -a $reindex_log mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --archive | tee -a $reindex_log }
If you only need to reindex a certain elasticsearch index/wiki type (for example, just dewiki_content rather than all of dewiki, use the following altered version of the reindex function:
function reindex_single_es_index() { cluster="$1" wiki="$2" es_index_suffix="$3" mkdir -p "$HOME/cirrus_log/" reindex_log="$HOME/cirrus_log/$wiki.$cluster.reindex.log" if [ -z "$cluster" -o -z "$wiki" -o -z "es_index_suffix" ]; then echo "Usage: reindex [cluster] [wiki] [es_index_suffix]" return 1 fi TZ=UTC export REINDEX_START=$(date +%Y-%m-%dT%H:%m:%SZ) echo "Started at $REINDEX_START" > "$reindex_log" mwscript extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php --wiki $wiki --cluster $cluster --indexType $es_index_suffix --reindexAndRemoveOk --indexIdentifier now 2>&1 | tee -a "$reindex_log" && \ mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --deletes | tee -a "$reindex_log" && \ mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --queue | tee -a "$reindex_log" mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --archive | tee -a "$reindex_log" }

Every cluster is isolated, so it is suggested to run a reindex process per-cluster. Essentially create a tmux session with a pane for each cluster, and run the following in each (with CLUSTER adjusted as appropriate). This must be done for eqiad, codfw and cloudelastic clusters.
Because the CirrusSearch reindex process involves hitting the SQL databases (data is fed from SQL into Elasticsearch), it's helpful to have each tmux pane run the reindex in a slightly different order to avoid thrash. (This is a recommended performance optimization but is not mandatory, to be clear)
cluster=eqiad expanddblist all | while read wiki ; do reindex $cluster $wiki done

Don't worry about incompatibility during the update - CirrusSearch is maintained so that queries and updates will always work against the currently deployed index as well as the new index configuration. Once the new index has finished building (the second command) it'll replace the old one automatically without any interruption of service. Some updates will have been lost during the reindex process. The third command will catch those updates.
Full reindex
If for some reason a wiki index has to be recreated from scratch (bad backups, no other cluster to copy from, some other sort of total failure) CirrusSearch can regenerate the index from the SQL databases. Historically this was also used to add new fields or apply changes to how cirrussearch renders fields, but that use case has been replaced by the Saneitizer background process. This process is expensive and will take significant time, it is a last resort.
First, do an in-place reindex as above. This is necessary due to the fact that in-place indexing updates the index mappings, while full reindex uses existing mappings to bring in the new data. Then, use this to make scripts that run the full reindex:
function make_index_commands​() { wiki=$1 mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --queue --maxJobs 10000 --pauseForJobs 1000 \ --buildChunks 250000 | sed -e 's/^php\s*/mwscript extensions\/CirrusSearch\/maintenance\//' | sed -e 's/$/ | tee -a cirrus_log\/'​$wiki​'.parse.log/' } function make_many_reindex_commands​() { wikis=$(pwd)/$1 rm -rf cirrus_scripts mkdir cirrus_scripts pushd cirrus_scripts while read wiki ; do make_index_commands $wiki done < $wikis | split -n r/5 for script in x*; do sort -R $script > $script.sh && rm $script; done popd }
Then run the scripts it makes in screen sessions.
Dumps
Dumps of all indices are created weekly by a cron as part of the snapshot puppet module. The dumps are available on dumps.wikimedia.org and can be reimported via the elasticsearch _bulk API.
Adding new nodes
Add the new nodes in puppet making sure to include all the current roles and account, sudo, and lvs settings.
Once the node is installed and puppet has run, you should be left with large RAID 0 ext4 /var/lib/elasticsearch mount point. Two things should be tweaked about this mount point:
service elasticsearch stop umount /var/lib/elasticsearch # Remove the default 5% blocks reserved for privileged processes. tune2fs -m 0 /dev/md2 # add the noatime option to fstab. sed -i 's@/var/lib/elasticsearch ext4 defaults@/var/lib/elasticsearch ext4 defaults,noatime@' /etc/fstab mount /var/lib/elasticsearch service elasticsearch start
Add the node to lvs.
You can watch he node suck up shards:
curl -s localhost:9200/_cluster/state?pretty | awk ' BEGIN {more=1} {if (/"nodes"/) nodes=1} {if (/"metadata"/) nodes=0} {if (nodes && !/"name"/) {node_name=$1; gsub(/[",]/, "", node_name)}} {if (nodes && /"name"/) {name=$3; gsub(/[",]/, "", name); node_names[node_name]=name}} {if (/"RELOCATING"/) relocating=1} {if (/"routing_nodes"/) more=0} {if (/"node"/) {from_node=$3; gsub(/[",]/, "", from_node)}} {if (/"relocating_node"/) {to_node=$3; gsub(/[",]/, "", to_node)}} {if (/"shard"/) {shard=$3; gsub(/[",]/, "", shard)}} {if (more && relocating && /"index"/) { index_name=$3 gsub(/[",]/, "", index_name) print "from="node_names[from_node]" to="node_names[to_node]" shard="index_name":"shard relocating=0 }} '
Removing nodes
Push all the shards off the nodes you want to remove like this:
curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.exclude._ip" : "10.x.x.x,10.x.x.y,..." } }'
Then wait for there to be no more relocating shards:
echo "Waiting until relocating shards is 0..." until curl -s localhost:9200/_cluster/health?pretty | tee /tmp/status | grep '"relocating_shards" : 0,'; do cat /tmp/status | grep relocating_shards sleep 2 done
Use the cluster state API to make sure all the shards are off of the node:
curl localhost:9200/_cluster/state?pretty | less
Elasticsearch will leave the shard on a node even if it can't find another place for it to go. If you see that then you have probably removed too many nodes. Finally, you can decommission those nodes in puppet. The is no need to do it one at a time because those nodes aren't hosting any shards any more.
Standard decommissioning documentation applies. When decommissioning nodes, you should ensure that elasticsearch is cleanly stopped before the node is cut from the network.
Deploying plugins
Plugins (for search) are deployed using the wmf-elasticsearch-search-plugins debian package. This package is built using the ops-es-plugins repository and deployed to our apt repo like any other debian packages we maintain.
The list of plugins we currently maintain is:
They are currently released using the classic maven process (release:prepare|perform) and deployed to maven central. The repository name used for deploying to central is ossrh and thus the engineer performing the release must have its ~/.m2/settings.xml with the following credentials set:
<settings> <servers> <server> <id>ossrh</id> <username>​username​</username> <password>​password​</password> </server> </servers></settings>
To request write permission to this repository one must create a request like this one and obtain a +1 from a person already granted.
Please see discovery-parent-pom for generic advises on the build process of the java projects we maintain.
Inspecting logs
There is two different types of log, we have elasticsearch logs on each node and cirrus logs.
Elasticsearch logs
The logs generated by elasticsearch are located in /var/logs/elasticsearch/ :
NOTE: the log verbosity can be changed at runtime, see elastic search logging for more details.
Cirrus logs
The logs generated by cirrus are located on mwlog1001.eqiad.wmnet:/a/mw-log/ :
Useful commands :
See all errors in realtime (useful when doing maintenance on the elastic cluster)
tail -f /a/mw-log/CirrusSearch.log | grep -v DEBUG
WARNING: you can rapidly get flooded if the pool counter is full.
Measure the throughput between cirrus and elastic (requests/sec) in realtime
tail -f /a/mw-log/CirrusSearchRequests.log | pv -l -i 5 > /dev/null
NOTE: this is an estimation because I'm not sure that all requests are logged here. For instance: I think that the requests sent to the frozen_index are not logged here. You can add something like 150 or 300 qps (guessed by counting the number of "Allowed write" in CirrusSearch.log)
Measure the number of prefix queries per second for enwiki in realtime
tail -f /a/mw-log/CirrusSearchRequests.log | grep enwiki_content | grep " prefix search " | pv -l -i 5 > /dev/null
Using jstack or jmap or other similar tools to view logs
Our elasticsearch systemd unit sets PrivateTmp=true, which is overall a good thing. But this prevents jstack / jmap / etc. from connecting to the JVM as they expect a socket in the temp directory. The easy/safe workaround to viewing these logs is via nsenter (See T230774). Example:
gehel@elastic2050:~$ sudo systemctl status elasticsearch_6@production-search-codfw.service | grep "Main PID" Main PID: 1110 (java) gehel@elastic2050:~$ sudo nsenter -t 1110 -m sudo -u elasticsearch jstack 11102019-08-20 08:58:53 Full thread dump OpenJDK 64-Bit Server VM (25.222-b10 mixed mode): "Attach Listener" #9275 daemon prio=9 os_prio=0 tid=0x00007f9c48002000 nid=0x31f81 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "elasticsearch[elastic2050-production-search-codfw][fetch_shard_store][T#15]" #7796 daemon prio=5 os_prio=0 tid=0x00007f9b58025000 nid=0x2a438 waiting on condition [0x00007f8a64a5e000] java.lang.Thread.State: WAITING (parking)[...]
or We can go further with:
systemctl-jstack() {if [ -z $1 ] { echo "Please provide a service name." echo "Usage: systemctl-jstack elasticsearch_6@production-search-codfw.service" exit 1}PID=$(systemctl show $1 -p MainPID | sed -e 's/^MainPID=//'​)​USER​=​$(​systemctl show $1 -p User | sed - e 's/^User=//') sudo nsenter -t $PID -m sudo -u $USER jstack $PID}
Multi-DC Operations
Status
We have deployed full size elasticsearch clusters in eqiad and codfw. The cluster are queried via LVS at search.svc.{eqiad,codfw}.wmnet:9243. The logic for ongoing updates was implemented in https://gerrit.wikimedia.org/r/#/c/237264/​. Switch of Elasticsearch to codfw is tested as part of the Datacenter Switch tests. By default application servers are configured to query the elasticsearch cluster in their own datacenter. All traffic can be shifted to a single datacenter for maintenance operations when necessary.
Overview
There is no current plan to utilize a second job queue in codfw for the secondary elastic cluster. As update jobs are spawned in eqiad they will be spawned in duplicate with one for each cluster. This will keep both clusters in sync. Prior to that we need to do initial index population.
DC Switch
Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster​InitialiseSettings.php​. The default value is "local", which means that if mediawiki switches DC, everything should be automatic.
Having search traffic flow between 2 datacenters increases the privacy risks. HTTPS has been deployed (on port 9243) to mitigate this risk.
Removing Duplicate Indices
When index creation bails out, perhaps due to a thrown exception or some such, the cluster can be left with multiple indices of a particular type but only one active. CirrusSearch contains a script, scripts/check_indices.py​, that will check the configuration of all wikis in the cluster and then compare the set of expected indices with those actually in the elasticsearch clusters.

/srv/mediawiki/php/extensions/CirrusSearch/scripts/check_indices.py | jq .
When everything is expected the output will be an empty json array:
[]
When something is wrong the output will contain an object for each cluster, and then the cluster will have a list of problems found:
[ { "cluster_name": "production-search-eqiad", "problems": [ { "reason": "Looks like Cirrus, but did not expect to exist.Deleted wiki?", "index": "ebernhardson_test_first" } ], "url": "​https://search.svc.eqiad.wmnet:9243​" } ]
Indicies still have to be manually deleted after reviewing the output above. As we gain confidence in the output through manual review the tool can be updated with options to automatically apply deletes.
curl -XDELETE https://search.svc.eqiad.wmnet/ebernhardson_test_first
Rebalancing shards to even out disk space use
Viewing free disk space per node:
curl -s localhost:9200/_cat/nodes?h=d,h | sort -nr
To move shards away from the nodes with the least disk free, we can lower cluster.routing.allocation.disk.watermark.high settings temporarily. The high watermark is the limit at which Elasticsearch will start moving shards out of the node to free up space. High watermark can be modified by:
curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.disk.watermark.high" : "70%" } }'
Keep an eye on the number of indices on a few nodes
When banning nodes to prepare for some maintenance operations, it is useful to see how many shards are left on those nodes:
watch -d -n 10 \ "curl -s localhost:9200/_cat/shards \ | awk '{print \$8}' \ | egrep 'elastic10(21|22|23|24)' \ | sort \ | uniq -c"
Elasticsearch Curator
elasticsearch-curator is a python tool which provides high level configuration for some maintenance operations. Its configuration is based on action files. Some standard actions are deployed on each elasticsearch node in /etc/curator. For example, you can disable shard routing with:
sudo curator --config /etc/curator/config.yaml /etc/curator/disable-shard-allocation.yaml
Curator must be run as root to access its log file (/var/log/curator.log).
Explain unallocated shards
Elasticsearch 5.2 introduced a new API endpoint to explain why a shard is unassigned:
curl -s localhost:9200/_cluster/allocation/explain?pretty
Shards stuck in recovery
We've seen cases where some shards are stuck in recovery and never complete the recovery process. Reducing temporarily the number of replicas and increasing it again once the cluster is green seems to fix the issue. So far, this has happen only during the 5.6.4 to 6.5.4 upgrade.
curl -k -X PUT "​https://localhost:9243/$index/_settings​" -H 'Content-Type: application/json' -d' { "index" : { "auto_expand_replicas" : "0-2", "number_of_replicas" : 2 } } '
No updates
If no updates are flowing, the usual culprit can be:
Stuck in old GC hell
When memory pressure is excessive the GC behaviour of an instance will switch from primarily invoking the young GC to primarily invoking the old GC. If the issue is limited to a single instance it's possible the instance has an unlucky set of shards that require more memory than the average instance. Resolving this situation involves:
  1. Ban the node from cluster routing
  2. Wait for all shards to drain
  3. Restart instance
  4. Unban the node from cluster routing
Draining all the shards away ensures the instance will get a new selection of shards when it is unbanned. Restarting the intsance is not strictly required, but it's nice to start with a fresh JVM after the old one has been misbehaving.
Pool Counter rejections (search is currently too busy)
Sometimes users' searches may fail with the message: "An error has occurred while searching: Search is currently too busy. Please try again later."
This is generally due to a spike in traffic which triggers MediaWiki's PoolCounter, which in the case of CirrusSearch will drop requests if the traffic spike is too large.
There is an alert mediawiki_cirrus_pool_counter_rejections_rate which should warn if a concerning number of CirrusSearch PoolCounter rejections are detected. The alert can be checked in icinga.
If the alert fires, take some time to consider if the current PoolCounter thresholds should be increased to increase the allowable queue size on MediaWiki's end. If the threshold cannot be increased, then all there is to do is verify that the traffic spike is organic as opposed to the result of some bug upstream.
Other elasticsearch clients
Cirrus is not the only application using our "search" elasticsearch cluster. In particular:
Constraints on elasticsearch clients
Reading this page should already give you some idea of how our elasticsearch cluster is managed. A short checklist that can help you:
Trouble
If Elasticsearch is in trouble, it can show itself in many ways. Searches could fail on the wiki, job queue could get backed up with updates, pool counter overflowing with unperformed searches, or just plain old high-load that won't go away. Luckily, Elasticsearch is very good at recovering from failure so most of the time these sorts of problems aren't life threatening. For some more specific problems and their mitigation techniques, see the /Trouble subpage.
Tasks Api
Sometimes pathological queries can get "stuck" burning cpu as they try and execute for many minutes. We can ask elasticsearch what tasks are currently running with the tasks api:
curl https://search.svc.eqiad.wmnet:9243/_tasks?detailed > tasks
This will dump a multi-megabyte json file that indicates all tasks (such as shard queries, or the parent query that issued shard queries) and how long they have been running. The inclusion of the detailed flag means all shard queries will report the index they are running against, and the parent query will report the exact text of the json search query that cirrussearch provided. For an extended incident this can be useful to track down queries that have been running extended periods of time, but for brief blips it may not have much information.
For a quick look at tasks on a node, you can use something like:
curl -s 'localhost:9200/_tasks/?detailed&nodes=elastic1022' | \ jq '.nodes | to_entries | .[0].value.tasks | to_entries | map({ action: .value.action, desc: .value.description[0:100], ms:(.value.running_time_in_nanos / 1000000) }) | sort_by(.ms)' | \ less
Other Resources
Information about specific CirrusSearch extra functionality:
Last edited on 3 May 2021, at 22:57
Wikitech
Content is available under CC BY-SA 3.0 unless otherwise noted.
Privacy policy
Terms of Use
Desktop
HomeRandomLog inSettingsDonateAbout WikitechDisclaimers
WatchEdit