In T221389: setup/install WMF7426 as phab1003.eqiad.wmnet we requested (and received) a temporary server so that we could more comfortably migrate to debian stretch.
We would like to now request that this hardware become permanent in order to have phabricator redundancy in eqiad (warm standby).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • mmodell | T182832 Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state | |||
| Resolved | Paladox | T125357 /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded | |||
| Resolved | Dzahn | T151070 Move Phabricator from PHP 7.0 to PHP 7.2 | |||
| Resolved | Dzahn | T238956 switch prod Phabricator from phab1003 to phab1001 | |||
| Resolved | Dzahn | T190568 Reimage both phab1001 and phab2001 to stretch / buster | |||
| Declined | Dzahn | T232887 The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent |
permanent allocation of the system phab1003/WMF7426 will require the sign off of either @faidon or @mark.
@mark please approve per OKR "Improve resilience and modernize devtools services" -> "Phabricator can be failed over in common failure scenarios"
I'm a bit confused; as far as I know the old plan was always to have HA of Phabricator between eqiad and codfw, and the linked task T190572 also talks about that. So is that no longer the case, and if so, why is that? I believe there have been blockers & complications for that deployment, but are they documented anywhere? How does this task relate to those plans, why do we feel failover within eqiad is (also) needed?
Also, to satisfy the OKR KR "Phabricator can be failed over in common failure scenarios" we should probably list which failure scenarios we consider common and how these plans help with that.
After further discussion with Mukunda and Mark we are focusing on eqiad/codfw failover and take back the request to permanently keep the eqiad server.
Databases used by the codfw server have been switched to codfw, avoid cross-DC connections. The codfw db server is readonly. This prevents the phd service from starting.
But we can still failover and make the server writable if needed.
We will give this server back to the pool in T238957
There are further comments on failure scenarios at T190572#5668444.