Memcached for MediaWiki
This page is about Memcached for MediaWiki.
Wikimedia infrastructure
Data centres and PoPs
This page is not about other Memcached clusters in production, such as those for Thumbor, Wikimedia Cloud Services, Wikitech wiki, and Swift.
Wikipedia's Memcached configuration as of November 2020. Detailing the use of additional "onhost" memcached instances, local to MW web servers (handled via mcrouter route prefixes).
Service
There are two logical pools of memcached servers for MediaWiki:
MediaWiki connects to memcached through a proxy called mcrouter, which provides a number of benefits such as such as connection pooling and failover functionality. See #Mcrouter for how these proxies are used.
Magic numbers
WANObjectCache
WANObjectCache is the abstraction layer in MediaWiki PHP that deals with multi-datacenter concerns and mcrouter. It builds on top of BagOStuff which is the generic key-value class that abstracts the memcached protocol itself.
See also:
High level
Memcached commands
Intra-dc:
Cross-dc:
Purge traffic uses the /*/mw-wan/ prefix to tell mcouter to broadcast this to other pools and clusters as well. The actual command is generally SET as it needs to induce a "hold-off" period using the tombstone (per the above). In rare cases where a hold-off is not needed (e.g. if the purge is not related to a DB write), then the broadcasted event will use DELETE
Getting revision/page from WANObjectCache key
If you're trying to track down the specific revision text given an SqlBlobStore key, the somewhat convoluted procedure is documented at mw:Manual:Caching#Revision_text​.
Mcrouter
For Mcrouter runbooks, see Memcached for MediaWiki/mcrouter
Service
There is a local mcrouter instance on every app server.
There is also a cluster of 4-shard "Proxies" pool of mcrouter instances in each data center for the purpose of receiving cross-dc memcached commands to then proxy further to the dc-local app servers accordingly.
TODO: Do these proxies also consider their dc-local gutterpool?
Routes
Each MediaWiki api/appserver sees memcached through a local proxy called Mcrouter[1]. Mcrouter introduces the concepts of routes and pools and each route applies consistent hashing on the key name to know where to send it, i.e. which of the 18 shards for memcached.
There are several routes available in our configuration, which are addressable via a route prefix that mcrouter strips from the key before forwarding the memcached command.
  1. Main route. This route is declared as /$region/mw/ but is not addressed by MediaWiki as such. It routes to the dc-local "Main" pool shards. If a shard is perceived as unavailable from an appserver ("TKO") the local mcrouter forwards all commands (incl gets, sets, and locks) to a shard of the "Gutter" pool instead (see T240684, T244852).
    • This route is used by the majority of traffic, through WANObjectCache::getWithSet calls in MediaWiki.
    • MediaWiki doesn't use the /$region/mw/ prefix. Instead /$region/mw/ is the default route and MediaWiki sends these commands without any routing prefix.
    • Switchover to and from the gutterpool is decided by Mcrouter locally (per-appserver), it is not centrally coordinated. The keys stored in a gutter server have a reduced TTL.
  2. WAN route. This route is declared as /$region/mw-wan/. It routes to the dc-local "Main" pool shards as well as the "Proxies" for all non-local DCs.
    • This route is for internal use by MediaWiki's WANObjectCache to broadcast its purges ("tombstones"). This happens from calls to WANObjectCache::purge (invalidates a single key) or WANObjectCache::touchCheckKey (effectively invalidate many keys, through a shared "check" key; somewhat like the Varnish XKey mechanism).
    • This route is not used for storing "regular" values is not exposed to any generic WANObjectCache::getWithSet or BagOStuff calls.
Example
The memcached key WANCache:v:metawiki:translate-groups (belongs to the Translate extension) is formatted by the WANObjectCache library. When Translate wants to get the value of this key, WANObjectCache will send a GET command from MediaWiki to localhost:11213, where mcrouter is listening. The command is then further routed to mc1022 (based on key hashing). MediaWiki it totally ignorant about the mc[1,2]0XX host, it only knows about sending commands to a localhost port. A mcrouter admin command helps figure out where keys are hashed/routed to:
elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2 VALUE __mcrouter__.route​(​get,WANCache:v:metawiki:translate-groups​) 0 1610.64.0.83:11211 END elukey@mw1345:~$ dig -x 10.64.0.83 +short mc1022.eqiad.wmnet.
Some things to notice:
To get a key and dump it to a file it is sufficient to:
elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt elukey@mw1345:~$ du -hs dump.txt 380K dump.txt
In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content.
Runbooks
Memcached server failure
See also
github.com/memcached/memcached
Introducing mcrouter: A memcached protocol router for scaling memcached deployments
Last edited on 10 August 2021, at 14:01
Wikitech
Content is available under CC BY-SA 3.0 unless otherwise noted.
Privacy policy
Terms of Use
Desktop
HomeRandomLog inSettingsDonateAbout WikitechDisclaimers
WatchEdit