If there is some aspect of this that is not addressed by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1010344/ , feel free to reopen.

I forgot that I had opened this task and almost went to open another one today. Anyway, a little more information: the problem still exists in 1.39.7. In addition to ignoring $wgUseImageMagick, the hardcoded getScalerType for webp also ignores $wgUseImageResize.

my understanding is that this worked after retrying.

Marking this resolved as you just confirmed a big file upload going through correctly. Thanks for your help in debugging this!

Seems like this is fixed! We just had File:OBR_Hafignnover_6-2024.webm (4.88 GB) successfully uploaded (req-id: c1a5aab7-6e6b-4ef5-b20d-f9ddb095577f ). The job took 5 minutes 20 seconds to complete, so went beyond the previous 202 second limit.

Can you try uploading again. The issue might be fixed now

In T358308#9724785, @Bawolff wrote:

[...]
So it seems like two separate issues.

I guess sometimes the job runner pod gets terminated in the middle of a job. That would be fine if something like https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1008403 got merged)

request_terminate_timeout for mw-jobrunners should now be set to 86400, as it was on bare metal.

Change #1021427 merged by jenkins-bot:

[operations/deployment-charts@master] mw-debug: fix php.timeout

https://gerrit.wikimedia.org/r/1021427

Mentioned in SAL (#wikimedia-operations) [2024-04-18T11:29:19Z] <cgoubert@deploy1002> Finished scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308 (duration: 37m 04s)

Change #1021427 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-debug: fix php.timeout

https://gerrit.wikimedia.org/r/1021427

Mentioned in SAL (#wikimedia-operations) [2024-04-18T10:52:15Z] <cgoubert@deploy1002> Started scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308

Change #1021418 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix php.timeout

https://gerrit.wikimedia.org/r/1021418

Mentioned in SAL (#wikimedia-operations) [2024-04-18T10:45:15Z] <claime> Rebuild php7.4-fpm production images - T358308

Change #1021415 merged by Clément Goubert:

[operations/docker-images/production-images@master] php7.4-fpm: Actually use FPM__request_terminate_timeout

https://gerrit.wikimedia.org/r/1021415

Change #1021418 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mediawiki: Fix php.timeout

https://gerrit.wikimedia.org/r/1021418

Change #1021415 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/docker-images/production-images@master] php7.4-fpm: Actually use FPM__request_terminate_timeout

https://gerrit.wikimedia.org/r/1021415

I think I found it

In T358308#9724446, @Ladsgroup wrote:

How can I get into a pod in job runners namespace(?) via shell.php? I want to try some stuff

Yes, this is due to T358308. Until that is fixed, you can upload large (>3GB) files via https://commons.wikimedia.org/wiki/Help:Server_side_upload

Ok, i did some more digging in logstash.

How can I get into a pod in job runners namespace(?) via shell.php? I want to try some stuff

In T358308#9711611, @Bawolff wrote:

Looking at recent logs, it does seem like some of the failures might be caused by things other than a timeout (There was one recently that failed after 17 seconds). However i still think the larger files are failing due to some sort of timeout as they tend to fail at around 202 second mark, which is pretty suspicious. Sample size is pretty low since users know large uploads don't work so they don't try and upload large files.

We can exclude a bad setting of the async trait for mw-jobrunner.
From a pod in production via shell.php

> use Wikimedia\MWConfig\ClusterConfig;
> ClusterConfig::getInstance()->isK8s()
= true

It means that a request was sent but then the server didn't respond within x seconds and the servers cut off the connection, failing the request.

Still cannot upload, I think this is a serious problem.

Change #1019823 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: set more php-specific settings to match metal instances

https://gerrit.wikimedia.org/r/1019823

Note there are several kinds of perceptual hashes and before we do any actual development we need to first compare the effectiveness and performance of different kinds of hashes.

Not sure if we would be able to do that kind of comparison in a SQL query

This one too T362352

Dropping a note here after a chat with @Cparle that we'd ideally want to add the proposed new column for perceptual hashes (T121797 / T251026) as part of this work, rather than migrating the tables twice.

Change #1019823 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: set more php-specific settings to match metal instances

https://gerrit.wikimedia.org/r/1019823

Looking at recent logs, it does seem like some of the failures might be caused by things other than a timeout (There was one recently that failed after 17 seconds). However i still think the larger files are failing due to some sort of timeout as they tend to fail at around 202 second mark, which is pretty suspicious. Sample size is pretty low since users know large uploads don't work so they don't try and upload large files.