I'd like to note that the latest version of xdebug (2.9.0), released one month ago, claims to be up to 250% faster than 2.7.0 (the version we're using) for generating coverage [1]. I've upgraded locally, and while I didn't get a 250% speedup, it's certainly faster (I didn't measure it, though). I believe this upgrade should be performed as soon as possible, as it seems fairly straightforward.

That said, phpdbg or pcov may still be faster. If we want to try the latter, then this task should have T192167 as subtask.

[1] - https://xdebug.org/announcements/2019-12-09

While switching to pcov / phpdbg (pcov seems better after a quick look?) is a good idea, another consideration is to use only unit tests for generating coverage reports (cc @Gehel), which would speed things up by virtue of those tests running much faster and there also being fewer of them, for now.

But that would mean instead of recording 29% code coverage for core we would instead show 1.9%.

Now that we're on PHPUnit 8, I've fiddled a bit with pcov and, oh dear, that's just astonishing.

XDebug 2.9.1

$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilterParser --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ 461 tests here ]

Time: 16.09 seconds, Memory: 70.00 MB

OK (461 tests, 1040 assertions)

Generating code coverage report in HTML format ... done [4.87 seconds]

Pcov

$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilterParser --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ Same 461 tests here ]

Time: 3.88 seconds, Memory: 70.00 MB

OK (461 tests, 1040 assertions)

Generating code coverage report in HTML format ... done [2.1 seconds]

Multiple attempts result in similar times. As noted above, there are some differences in the reports. One that I could spot quickly is that PCOV considers the following to be not coverable:

• lines containing only global $foo
• lines containing only switch ( expr ) {

But I don't think it's a big deal.

Then I decided to execute more tests, including non-unit tests as well as database tests:

Xdebug

$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilter --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ 716 tests and 6 failures ]

Time: 4.19 minutes, Memory: 140.00 MB

FAILURES!
Tests: 716, Assertions: 1460, Failures: 6.

Generating code coverage report in HTML format ... done [4.95 seconds]

Pcov

$ php tests/phpunit/phpunit.php --coverage-html ../coverageTest --filter AbuseFilter --configuration ../../AbuseFilterSuite.xml
Using PHP 7.4.1
PHPUnit 8.5.2 by Sebastian Bergmann and contributors.

[ Still 716 tests ]

Time: 2.02 minutes, Memory: 178.00 MB

FAILURES!
Tests: 716, Assertions: 1460, Failures: 6.

Generating code coverage report in HTML format ... done [2.22 seconds]

Not too bad, huh?

Change 567938 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[integration/config@master] dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+

https://gerrit.wikimedia.org/r/567938

As per T243847#6567940, I wanted to try xdebug 3.0.0 (released 5 days ago) and see what kind of speedup we can expect. The following measerements refer to running all tests in AbuseFilter.

Xdebug 2.9.1

Using PHP 7.4.1
[...939 tests]

Time: 4.51 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.96 seconds]

Xdebug 3.0.0, mode=coverage

Using PHP 7.4.1
[...939 tests]

Time: 3.07 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.12 seconds]

pcov

Using PHP 7.4.1
[...939 tests]

Time: 2.33 minutes, Memory: 176.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [3.87 seconds]

So yes, it does seem that xdebug brought a massive performance improvement, although it's still slower than pcov. I think that upgrading xdebug, if it can be done easily (i.e. without having to get pcov in apt or building from source), would be great.

Is pcov as accurate as xdebug? I remember that used to be a problem with phpdbg when I last looked into it a few years ago.

In T234020#6657625, @Legoktm wrote:

Is pcov as accurate as xdebug? I remember that used to be a problem with phpdbg when I last looked into it a few years ago.

It's very similar. According to my previous tests (T234020#5826901, which also match the documentation on github), the only notable difference is that pcov considers

• lines containing only global $foo ...
• lines containing only switch ( expr ) {

as not coverable. Which I think is not a problem, especially if it comes with that massive speedup.

In T234020#6657575, @Daimona wrote:

As per T243847#6567940, I wanted to try xdebug 3.0.0 (released 5 days ago) and see what kind of speedup we can expect. The following measerements refer to running all tests in AbuseFilter.

Xdebug 2.9.1

Using PHP 7.4.1
[...939 tests]

Time: 4.51 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.96 seconds]

Xdebug 3.0.0, mode=coverage

Using PHP 7.4.1
[...939 tests]

Time: 3.07 minutes, Memory: 98.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [7.12 seconds]

pcov

Using PHP 7.4.1
[...939 tests]

Time: 2.33 minutes, Memory: 176.00 MB

OK (939 tests, 1819 assertions)

Generating code coverage report in HTML format ... done [3.87 seconds]

So yes, it does seem that xdebug brought a massive performance improvement, although it's still slower than pcov. I think that upgrading xdebug, if it can be done easily (i.e. without having to get pcov in apt or building from source), would be great.

@Daimona what is the command you used for this? Have you edited phpunit.xml to modify the <filter><whitelist> section for just AbuseFilter? It should be faster than that, I would think.

In T234020#6658340, @kostajh wrote:

@Daimona what is the command you used for this? Have you edited phpunit.xml to modify the <filter><whitelist> section for just AbuseFilter? It should be faster than that, I would think.

Yes, I'm using a custom config file with only AbuseFilter whitelisted. The problem is that there are several heavy, slow, integration tests. Running without coverage, I get

Time: 1.85 minutes, Memory: 72.00 MB

Change 567938 merged by jenkins-bot:

[integration/config@master] dockerfiles: [quibble*] Add pcov and use it for test coverage

https://gerrit.wikimedia.org/r/567938

Change 678141 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] jjb: Switch quibble jobs to images using pcov not xdebug

https://gerrit.wikimedia.org/r/678141

Memo: I missed some usages. I guess these can be migrated later. I also haven't checked whether pcov should be installed in more images for those jobs to be migrated.

Change 678373 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov

https://gerrit.wikimedia.org/r/678373

Change 678373 merged by jenkins-bot:

[integration/config@master] dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov

https://gerrit.wikimedia.org/r/678373

Mentioned in SAL (#wikimedia-releng) [2021-04-11T00:52:12Z] <James_F> dockerfiles: [quibble-buster-php73-coverage] Switch from xdebug to pcov T234020

Change 678141 merged by jenkins-bot:

[integration/config@master] jjb: Switch quibble jobs to images with pcov (no-op)

https://gerrit.wikimedia.org/r/678141

Change 678376 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] jjb: Switch coverage jobs to images using pcov not xdebug

https://gerrit.wikimedia.org/r/678376

https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-patch-docker/42183/console is a re-build of https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-patch-docker/42030/console and fails:

Before:

00:04:05.404 Generating code coverage report in Clover XML format ... done [311 ms]
…

With the pcov image:

00:03:04.645 In CloverXml.php line 69:
00:03:04.645                                      
00:03:04.645   String could not be parsed as XML  
00:03:04.645                                      
00:03:04.645 
00:03:04.645 check [--sha1 [SHA1]] [--test-dir TEST-DIR] [--html [HTML]] [--command COMMAND]

Can you pastebin the generated clover.xml file? It should just be going through new SimpleXMLElement( file_get_contents( $fname ) ); - see https://gitlab.com/legoktm/clover-diff/-/blob/master/src/CloverXml.php

I think the actual issue is a few lines above:

03:07:54 $ php -d zend_extension=pcov.so -d pcov.enabled=1 -d pcov.directory=/workspace/src/extensions/WikiLambda -d pcov.exclude='@tests@' "$MW_INSTALL_PATH"/tests/phpunit/phpunit.php --coverage-clover /tmp/cloverJQCYA7 --filter '/ApiZObjectEditorTest/'
03:07:54 /usr/lib/php/20180731/pcov.so doesn't appear to be a valid Zend extension

which leads to:

Error:         No code coverage driver is available

So it might be producing a garbage XML file. Still haven't verified why.

Oh, I just realized why. It's not a zend extension. It should be loaded with just "extension". Local testing confirms, and on my local install, I've always had "extension" too, so I'm not sure how I couldn't get it right.

Change 678383 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[integration/config@master] dockerfiles: [quibble-buster-php73-coverage] pcov doesn't work with zend_extension

https://gerrit.wikimedia.org/r/678383

In T234020#6987347, @Daimona wrote:

Memo: I missed some usages. I guess these can be migrated later. I also haven't checked whether pcov should be installed in more images for those jobs to be migrated.

And also composer-package-phpXX, which run with xdebug on by default. I guess pcov would need to be installed in a more base dockerfile for that, e.g. phpXX. That is also only doable on PHP >= 7.3 since 7.2 doesn't use sury-php. That would resolve T269489, and more generally speed up the affected jobs.

Change 678383 merged by jenkins-bot:

[integration/config@master] dockerfiles: [quibble-buster-php73-coverage] pcov isn't a zend extension

https://gerrit.wikimedia.org/r/678383

Mentioned in SAL (#wikimedia-releng) [2021-04-14T16:19:54Z] <James_F> Docker: Publish quibble-buster-php73-coverage fixing loading of pcov T234020

Change 678376 merged by jenkins-bot:

[integration/config@master] jjb: Switch coverage jobs to images using pcov not xdebug

https://gerrit.wikimedia.org/r/678376

A quick preview of the improvement (for GrowthExperiments):

• This build with xdebug ran 308 tests in 4.19 minutes
• This build with pcov ran 312 tests in 7.94 seconds

I'm waiting for the next build of https://integration.wikimedia.org/ci/blue/organizations/jenkins/mwcore-phpunit-coverage-master/activity/ to see results on a larger scale, but I think this looks nice.

Change 679410 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[integration/config@master] jjb: Use pcov for mwcore-phpunit-coverage-patch

https://gerrit.wikimedia.org/r/679410

Change 679410 merged by jenkins-bot:

[integration/config@master] jjb: [mwcore-phpunit-coverage-patch] Use pcov not xdebug

https://gerrit.wikimedia.org/r/679410

Change 679422 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] dockerfiles: [quibble-buster-php73-coverage] Run core on $MW_INSTALL_PATH directly

https://gerrit.wikimedia.org/r/679422

Change 679422 merged by jenkins-bot:

[integration/config@master] dockerfiles: [quibble-buster-php73-coverage] Performance tuning config tweaks

https://gerrit.wikimedia.org/r/679422

Mentioned in SAL (#wikimedia-releng) [2021-04-16T23:20:09Z] <James_F> Docker: Publishing quibble-buster-php73-coverage version with performance tuning config tweaks T234020 T280167

Change 680481 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] jjb: Switch quibble coverage jobs to 0.0.46-s3

https://gerrit.wikimedia.org/r/680481

Change 680481 merged by jenkins-bot:

[integration/config@master] jjb: Switch quibble coverage jobs to 0.0.46-s3

https://gerrit.wikimedia.org/r/680481

In T234020#7041017, @hashar wrote:

I note that the MediaWiki core code coverage run went from a few hours run down to just 13 minutes! https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ Well done @Daimona

Bravo @Daimona! 👏 👏