Page MenuHomePhabricator
Create Task
Maniphest T299389

Wikibase CI broken due to missing PHP extensions: dom, intl, mbstring, xml, xmlreader, xmlwriter
Closed, ResolvedPublic
Actions

Description

gate-and-submit for change 754495 had four failing jobs: php73-noselenium #45876, php74-noselenium #44397, client #26479, and repo #26475. The first two jobs complain about the xmlwriter extension being missing; the latter two complain about missing dom, intl, mbstring, xml, and xmlreader.

Reason

We have rebuild the CI image for Quibble 1.3.0 which caused a reinstall of the PHP packages. Those come from sury.org. php-xdebug and php-apcu depends on the 8.1 flavor of the package which brings in php8.1-cli but not the other extensions. Essentially the images are broken.

docker-registry.wikimedia.org/releng/quibble-buster-php73:1.3.0
docker-registry.wikimedia.org/releng/quibble-buster-php74:1.3.0

Details

SubjectRepoBranchLines +/-
jjb: fix php 8.1 packages used ini php73 imagesintegration/configmaster+14 -14
dockerfiles: fix php 8.1 being used in php73 imageintegration/configmaster+61 -5
jjb: switch Quibble jobs to 1.3.0 (take 2)integration/configmaster+30 -30
Revert "jjb: switch Quibble jobs to 1.3.0"integration/configmaster+30 -30
Revert "jjb: Pass --parallel-npm-install to Selenium jobs"integration/configmaster+3 -3
Customize query in gerrit

Revisions and Commits

rCICF integration-config
rCICF1053c9d35905 dockerfiles: quibble images to use soname xdebug/apcu

Event Timeline

Lucas_Werkmeister_WMDE created this task.Jan 18 2022, 11:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2022, 11:38 AM
Stashbot added a comment.Jan 18 2022, 11:39 AM

Mentioned in SAL (#wikimedia-releng) [2022-01-18T11:39:44Z] <hashar> Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389

hashar triaged this task as High priority.Jan 18 2022, 11:44 AM
hashar added a project: Continuous-Integration-Infrastructure.
hashar added subscribers: kostajh, hashar.

I have upgraded the Jenkins jobs this morning to use Quibble 1.3.0 which has patches by @kostajh . I have thus rebuild the images yesterday.

One of the build uses docker-registry.wikimedia.org/releng/quibble-buster-php73:1.3.0 but complains with:

Composer is operating significantly slower than normal because you do not have the PHP curl extension enabled.

And

00:01:39.198 To enable extensions, verify that they are enabled in your .ini files:
00:01:39.198     - /etc/php/8.1/cli/php.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/10-opcache.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/10-pdo.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-apcu.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-calendar.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-ctype.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-exif.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-ffi.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-fileinfo.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-ftp.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-gettext.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-iconv.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-phar.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-posix.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-readline.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-shmop.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-sockets.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-sysvmsg.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-sysvsem.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-sysvshm.ini
00:01:39.198     - /etc/php/8.1/cli/conf.d/20-tokenizer.ini

Those are php 8.1 configuration file when the image should have php 7.3 and indeed Quibble starts with:

php --version
00:00:30.469 INFO:quibble.commands:php --version: PHP 8.1.1 (cli) (built: Dec 20 2021 21:33:24) (NTS)

TLDR: the image is broken

gerritbot added a comment.Jan 18 2022, 11:45 AM

Change 754895 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Revert \"jjb: Pass --parallel-npm-install to Selenium jobs\"

https://gerrit.wikimedia.org/r/754895

gerritbot added a project: Patch-For-Review.Jan 18 2022, 11:45 AM

Change 754896 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Revert \"jjb: switch Quibble jobs to 1.3.0\"

https://gerrit.wikimedia.org/r/754896

Stashbot added a comment.Jan 18 2022, 11:46 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-18T11:46:04Z] <hashar> Rolled back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389

hashar added a comment.Jan 18 2022, 12:04 PM

From apt list:

php-apcu5.1.21+4.0.11-7+0~20220117.31+debian10~1.gbp9a887a
php-common2:92+0~20220117.43+debian10~1.gbpe0d14e
php-xdebug3.1.2+2.9.8+2.8.1+2.5.5-1+0~20220117.45+debian10~1.gbp75d182
php7.3-bcmath7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-cli7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-common7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-curl7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-fpm7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-gd7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-gmp7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-intl7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-json7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-ldap7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-mbstring7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-mysql7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-opcache7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-pcov1.0.11-4+0~20220117.23+debian10~1.gbp34fff9
php7.3-pgsql7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-readline7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-sqlite37.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-tidy7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-xml7.3.33-1+0~20211119.91+debian10~1.gbp618351
php7.3-zip7.3.33-1+0~20211119.91+debian10~1.gbp618351
php8.1-apcu5.1.21+4.0.11-7+0~20220117.31+debian10~1.gbp9a887a
php8.1-cli8.1.1-1+0~20211220.11+debian10~1.gbp519568
php8.1-common8.1.1-1+0~20211220.11+debian10~1.gbp519568
php8.1-opcache8.1.1-1+0~20211220.11+debian10~1.gbp519568
php8.1-phpdbg8.1.1-1+0~20211220.11+debian10~1.gbp519568
php8.1-readline8.1.1-1+0~20211220.11+debian10~1.gbp519568
php8.1-xdebug3.1.2+2.9.8+2.8.1+2.5.5-1+0~20220117.45+debian10~1.gbp75d182

So more details:

apt-mark showmanual|grep php
php-apcu
php-xdebug
php7.3-bcmath
php7.3-cli
php7.3-curl
php7.3-fpm
php7.3-gd
php7.3-gmp
php7.3-intl
php7.3-ldap
php7.3-mbstring
php7.3-mysql
php7.3-pcov
php7.3-pgsql
php7.3-sqlite3
php7.3-tidy
php7.3-xml
php7.3-zip
apt-mark showauto|grep php
php-common
php7.3-common
php7.3-json
php7.3-opcache
php7.3-readline
php8.1-apcu
php8.1-cli
php8.1-common
php8.1-opcache
php8.1-phpdbg
php8.1-readline
php8.1-xdebug

We explicitly install php-apcu and php-xdebug however they are marked to depends on the 8.1:

$ apt-cache depends php-apcu php-xdebug
php-apcu
  Depends: php8.1-apcu

php-xdebug
  Depends: php8.1-xdebug

That is an issue with the sury.org packages we are using for php 7.3.

hashar updated the task description. (Show Details)Jan 18 2022, 12:08 PM
kostajh added a comment.Jan 18 2022, 12:10 PM

Can/should we use pecl to install apcu and xdebug? e.g. pecl install apcu & pecl install xdebug

gerritbot added a comment.Jan 18 2022, 12:25 PM

Change 754895 merged by jenkins-bot:

[integration/config@master] Revert \"jjb: Pass --parallel-npm-install to Selenium jobs\"

https://gerrit.wikimedia.org/r/754895

gerritbot added a comment.Jan 18 2022, 12:43 PM

Change 754896 merged by jenkins-bot:

[integration/config@master] Revert \"jjb: switch Quibble jobs to 1.3.0\"

https://gerrit.wikimedia.org/r/754896

hashar added a comment.Jan 18 2022, 12:45 PM
In T299389#7627900, @kostajh wrote:

Can/should we use pecl to install apcu and xdebug? e.g. pecl install apcu & pecl install xdebug

That is a bit of a pain, we would need various developments dependencies to be setup and then teardown and that exposes us to a bunch of compile failure when building the image. It is way easier to consume the binary packages maintained by sury.org, the person behind it is the developer maintaining the official PHP Debian package.

The bulk of it is they have split the apcu and xdebug packages to no more be version independent so we just need to install the proper versions directly, for example: php7.3-apcu and php7.3-xdebug. Following patches will address it for Quibble and for other CI images using those packages.

hashar added a commit: rCICF1053c9d35905: dockerfiles: quibble images to use soname xdebug/apcu.Jan 18 2022, 12:51 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 1:11 PM
gerritbot added a comment.Jan 18 2022, 1:24 PM

Change 754932 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch Quibble jobs to 1.3.0 (take 2)

https://gerrit.wikimedia.org/r/754932

gerritbot added a project: Patch-For-Review.Jan 18 2022, 1:24 PM
gerritbot added a comment.Jan 18 2022, 2:34 PM

Change 754951 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: fix php 8.1 being used in php73 image

https://gerrit.wikimedia.org/r/754951

Stashbot added a comment.Jan 18 2022, 3:00 PM

Mentioned in SAL (#wikimedia-releng) [2022-01-18T15:00:03Z] <hashar> Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389

gerritbot added a comment.Jan 18 2022, 3:02 PM

Change 754932 merged by jenkins-bot:

[integration/config@master] jjb: switch Quibble jobs to 1.3.0 (take 2)

https://gerrit.wikimedia.org/r/754932

hashar added a comment.Jan 18 2022, 3:02 PM

It is fixed by https://gerrit.wikimedia.org/r/c/integration/config/+/754905 which would stop getting php8.1 being installed. I have verified the new images have the proper php version:

Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php81:1.3.0-s1
Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php73-coverage:1.3.0-s1
Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php73:1.3.0-s1
Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php80:1.3.0-s1
Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php74:1.3.0-s1

All Quibble jobs have been switched again to the updated images, should be good now.

Some other images have to be rebuild as well which will be done via https://gerrit.wikimedia.org/r/754951

gerritbot added a comment.Jan 18 2022, 7:56 PM

Change 754951 merged by jenkins-bot:

[integration/config@master] dockerfiles: fix php 8.1 being used in php73 image

https://gerrit.wikimedia.org/r/754951

Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 8:10 PM
gerritbot added a comment.Jan 18 2022, 8:14 PM

Change 755016 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: fix php 8.1 packages used ini php73 images

https://gerrit.wikimedia.org/r/755016

gerritbot added a project: Patch-For-Review.Jan 18 2022, 8:14 PM
hashar closed this task as Resolved.Jan 18 2022, 8:17 PM
hashar claimed this task.

Solved! Apparently the Quibble 1.3.0 images were the sole affected, I still have rebuild the rest of the images for good measure :D

gerritbot added a comment.Jan 18 2022, 8:19 PM

Change 755016 merged by jenkins-bot:

[integration/config@master] jjb: fix php 8.1 packages used ini php73 images

https://gerrit.wikimedia.org/r/755016

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL