Several things are true:

• We want to be confident that we can deploy "master" of our code to production.
• We want to cross-check our various deployed code bases with each other prior to them hitting production.
• We want "gating" code checks to not take too long (~10 mins total), as that reduces developer productivity.
• We want to be able to merge patches quickly and with confidence, especially for incident response.

This means we want all our code to be covered "adequately" by tests, that we test all our extensions / skins / etc. together, and that these tests not take forever. These are in moderate opposition to each other.

As of mid-April 2019, the interlock "gate" only covers 31 of the 190 extensions and skins in production, and yet is already far too slow (~25 minutes for a full test run; 12 minutes for the wmf-quibble-core-vendor-mysql-hhvm-docker job, of which 5 minutes is the actual PHP unit run). Requests to add new, critical extensions to the gate (like Scribunto in T125050) are objected to on the grounds that this would be the straw that would break the camel's back. Even with no further additions of new code bases, the slowness of gate increases as teams increase the depth and breadth of their code coverage, either as maintenance and response to discovered bugs, or particularly as they add features and configuration options.

Very roughly, we can approximate that every test you write steals time from every other developer, so you should have a good reason to add that twelfth edge case test.

From the most recent merge of MediaWiki core's PHP unit test report numbers, we have 77 test classes that take more than 1000ms, an entirely arbitrary round number:

Obviously these are not directly comparable for a variety of reasons:

• These are shown based on class; some extensions, like Cirrus, merge all their tests into few classes, whereas others like Wikibase spread them out, making Cirrus look massive and flattering the huge number of tests that it creates.
• Individual tests can be trivial (e.g. ResourcesTest::testFileExistence runs all 2947 file checks in 3.7 sec) or very complex (the solitary Wikibase\Repo\Tests\Store\StoreTest::testRebuild takes 5.7 sec).
• Some tests are for features/behaviour that is particularly critical and it's reasonable to test every edge case we can think of; similarly, some tests are in areas where we've had repeated major issues, and we want to avoid further regressions.
• Some tests are driven by use (e.g. ResourcesTest is a structure test for files registered by extensions and skins, and though might well be made more efficient is mostly driven by the rest of us).

However, it's worth reflecting on whether we're over-testing in the areas most likely to break and most important for users.