I agree it is overdue. Next week is supposedly calm, I guess we can send an announce to wikitech-l / engineering folks and do the switch in our morning then catch up whatever is hit by race condition / parallelism indeed.

For the php 7.3, 7.4, 8.0 and 8.1 images we should probably move the whole Apache system up in the image layers to quibble-buster and it will thus be included in all the variants.

In T285649#7563333, @hashar wrote:

I agree it is overdue. Next week is supposedly calm, I guess we can send an announce to wikitech-l / engineering folks and do the switch in our morning then catch up whatever is hit by race condition / parallelism indeed.

For the php 7.3, 7.4, 8.0 and 8.1 images we should probably move the whole Apache system up in the image layers to quibble-buster and it will thus be included in all the variants.

Sounds good. Let me know how I can help. @awight will you be around? It sounds like the patches / work needed are:

• quibble-buster, for moving the apache config to that image
• build the new docker image
• send the announcement to wikitech-l
• update the jobs to use the new image, and adding the --web-backend=external flag
• update breaking tests with it.skip and file tasks to fix them later

Another thing that will reduce the risk of this change is to roll out T296826: specFileRetries does not seem to apply in extension wdio.conf.js to extensions/skins first; this will retry a failed test in the event of a failure.

In T285649#7563424, @kostajh wrote:

Sounds good. Let me know how I can help. @awight will you be around?

Yes, I'll be around this week and next. Great to see this movement, let's coordinate more to find ways I might be helpful :-)

I had a side track with docker-pkg (seed_images must be set in the config Gerrit 747060).

From a quick analysis and some discussion today with Kosta, we need Apache for php 7.2 (production) and 7.3 (for release branches) and I really want to avoid copy pasting between images or creating a bunch of new child images.

We can move the Apache config from the child image quibble-buster-php72-apache up to quibble-buster and that will then be included in the child. We can keep the entrypoint.sh with a new name such as with-apache.sh or whatever then the jobs can be switched using --entrypoint=with-apache.sh.

The devil is the image quibble-buster-php72-apache hardocdes 7.2 in the Dockerfile and php-fpm.conf and we might have to vary some bits.

[global]
pid = /tmp/php/php7.2-fpm.pid
error_log = /dev/stderr

include=/etc/php/7.2/fpm/pool.d/*.conf

The pid can be set to anything else really.

supervisord.conf also hardcode 7.2:

command = php-fpm7.2 --nodaemonize

So I don't know how to unhardcode 7.2 without having to copy paste a bunch of files between images :-\

Change 747815 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: quibble-apache make it php version agnostic

https://gerrit.wikimedia.org/r/747815

Change 747815 merged by jenkins-bot:

[integration/config@master] dockerfiles: quibble-apache make it php version agnostic

https://gerrit.wikimedia.org/r/747815

Change 747896 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update quibble-apache job

https://gerrit.wikimedia.org/r/747896

Mentioned in SAL (#wikimedia-releng) [2021-12-16T18:44:14Z] <hashar> Building refactored releng/quibble-buster-php72-apache image for T285649

Mentioned in SAL (#wikimedia-releng) [2021-12-16T19:04:52Z] <hashar> Updating jenkins job for quibble-apache # T285649

docker-registry.wikimedia.org/releng/quibble-buster-php72-apache:1.2.0-s5 has the refactor which should make it php version independent. The Apache/fpm/supervisord configuration should be moved up in quibble-buster

Then in each of the child images we will have to add:

ENV PHP_VERSION=7.2
# Convenience symlink to make php-fpm.conf version agnostic
RUN ln -s /etc/php/$PHP_VERSION /etc/php/current

Replace PHP_VERSION=7.2 with whatever is the one being installed.

And I think that would do it.

Change 747896 merged by jenkins-bot:

[integration/config@master] jjb: update quibble-apache job

https://gerrit.wikimedia.org/r/747896

Change 748104 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: move apache up to quibble-buster

https://gerrit.wikimedia.org/r/748104

Change 748104 merged by jenkins-bot:

[integration/config@master] dockerfiles: move apache up to quibble-buster

https://gerrit.wikimedia.org/r/748104

Change 751392 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: quibble full run to generic image

https://gerrit.wikimedia.org/r/751392

Change 751396 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: move wmf-quibble-apache* to generic image

https://gerrit.wikimedia.org/r/751396

Change 751422 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] api-testing: rename jobs to shorter forms

https://gerrit.wikimedia.org/r/751422

Change 751427 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: add entrypoint to Quibble job template

https://gerrit.wikimedia.org/r/751427

Change 751428 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] api-testing: switch Quibble jobs to Apache

https://gerrit.wikimedia.org/r/751428

Change 751422 merged by jenkins-bot:

[integration/config@master] api-testing: rename jobs to shorter forms

https://gerrit.wikimedia.org/r/751422

Change 751427 merged by jenkins-bot:

[integration/config@master] jjb: add entrypoint to Quibble job template

https://gerrit.wikimedia.org/r/751427

Change 751444 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/tools/api-testing@master] Add Quibble Apache config

https://gerrit.wikimedia.org/r/751444

Change 751444 merged by jenkins-bot:

[mediawiki/tools/api-testing@master] Add Quibble Apache config

https://gerrit.wikimedia.org/r/751444

Change 751428 merged by jenkins-bot:

[integration/config@master] api-testing: switch Quibble jobs to Apache

https://gerrit.wikimedia.org/r/751428

Change 751392 merged by jenkins-bot:

[integration/config@master] jjb: quibble full run to generic image

https://gerrit.wikimedia.org/r/751392

Change 751396 merged by jenkins-bot:

[integration/config@master] jjb: move wmf-quibble-apache* to generic image

https://gerrit.wikimedia.org/r/751396

Change 751684 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Migrate mediawiki-quibble-apitests-vendor-php72-docker to Apache

https://gerrit.wikimedia.org/r/751684

Change 751685 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update Quibble jobs to latest image

https://gerrit.wikimedia.org/r/751685

Change 751685 merged by jenkins-bot:

[integration/config@master] jjb: update Quibble jobs to latest image

https://gerrit.wikimedia.org/r/751685

Change 751684 merged by jenkins-bot:

[integration/config@master] Migrate mediawiki-quibble-apitests-vendor-php72-docker to Apache

https://gerrit.wikimedia.org/r/751684

Change 751689 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] zuul: mediawiki/core add experimental job with apache

https://gerrit.wikimedia.org/r/751689

Change 751689 merged by Hashar:

[integration/config@master] zuul: mediawiki/core add experimental job with apache

https://gerrit.wikimedia.org/r/751689

Change 751690 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] zuul: allow wmf-quibble-apache job for core or vendor

https://gerrit.wikimedia.org/r/751690

Change 751690 merged by Hashar:

[integration/config@master] zuul: allow wmf-quibble-apache job for core or vendor

https://gerrit.wikimedia.org/r/751690

Change 751691 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] zuul: fix wmf-quibble-apache filtering

https://gerrit.wikimedia.org/r/751691

Change 751691 merged by jenkins-bot:

[integration/config@master] zuul: fix wmf-quibble-apache filtering

https://gerrit.wikimedia.org/r/751691

Change 751697 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Migrate Quibble jobs to use Apache

https://gerrit.wikimedia.org/r/751697

Change 751697 merged by jenkins-bot:

[integration/config@master] Migrate Quibble jobs to use Apache

https://gerrit.wikimedia.org/r/751697

Mentioned in SAL (#wikimedia-releng) [2022-01-05T17:31:40Z] <hashar> Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697 to get rid of the wmf-quibble-apache jobs # T285649

Change 751997 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Skip a flaky SiteStats test

https://gerrit.wikimedia.org/r/751997

Change 751997 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Run jobs for tests dependent on deferred updates

https://gerrit.wikimedia.org/r/751997

Change 753431 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Run jobs for tests dependent on deferred updates (part II)

https://gerrit.wikimedia.org/r/753431

Change 753431 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Run jobs for tests dependent on deferred updates (part II)

https://gerrit.wikimedia.org/r/753431

Change 759241 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Add wiki.runAllJobs() for two more tests

https://gerrit.wikimedia.org/r/759241

Change 759242 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Disable flaky test

https://gerrit.wikimedia.org/r/759242

Change 759241 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Add wiki.runAllJobs() for two more tests

https://gerrit.wikimedia.org/r/759241

Change 759242 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Disable flaky test

https://gerrit.wikimedia.org/r/759242