Page MenuHomePhabricator

<Code Stewardship Review> Thumbor
Open, In Progress, MediumPublic

Description

This task will be filled out later, but getting it as a place-holder for the on-going internal conversations.

Stewardship Acceptance Criteria form (gdoc)

Related Objects

StatusSubtypeAssignedTask
In ProgressGoal DAbad
In Progress DAbad
Resolvedhnowlan
Resolvedhnowlan
Resolvedhnowlan
Resolvedhnowlan
Resolvedhnowlan
Resolvedhnowlan
ResolvedVlad.shapik
Openhnowlan
Resolvedhnowlan
Resolvedjijiki
ResolvedVRiley-WMF
ResolvedJclark-ctr
Resolvedhnowlan
ResolvedJhancock.wm
ResolvedVlad.shapik
Resolvedhnowlan
InvalidNone
Resolvedfgiunchedi
Resolved Gilles
Resolvedfgiunchedi
Resolvedjijiki
Resolvedjijiki
Resolvedroman-stolar
ResolvedVlad.shapik
Resolvedroman-stolar
InvalidNone

Event Timeline

greg moved this task from Backlog to Radar on the Thumbor board.

Will:

  • need to update python
  • old hardware

Can we define a product owner?

  • not clear on if any product team has owned this in the past
  • originally owned by the multimedia team
  • we plan to ask for a multimedia team for 2022 fiscal

Greg to have conversation with SRE on what a potential timeline is

From my perspective, the largest threats to the continued stability of Thumbor are:

  1. Old hardware (T233196)
  2. Debian Stretch deprecation (T216815)
  3. Python 2 deprecation (T252719)

1 and 2 are relatively simple, and it makes sense to fix them at the same time (preferably with T233196: Migrate thumbor to Kubernetes ). It's pretty much only blocked on resourcing. Some work toward this goal was done in T267327. Most of the tasks in Thumbor are upstream library issues, and many are fixed in the versions of the libraries packaged in Buster or Bullseye.

The Python 3 upgrade is trickier. Upstream released an alpha version with Python 3 support (Thumbor 7) in February 2020, but has had very little activity since (none in the last 6 months and no releases of any kind in the last 18 months). T267327 got the Wikimedia customizations working with Thumbor 7, at least well enough to pass the Wikimedia test suite. From my perspective, most of the work remaining there is testing and dealing with whatever upstream may decide to do.

To be clear, thumbor is just one part of the problem, the entire file management stack needs a dedicated owner. Other people have just been covering for issues when it gets really bad. File metadata had been causing significant problems until Tim and Amir moved it into external storage. MediaWiki had been uselessly creating thumbnails of all images that Thumbor served, until Tim disabled it after realizing this was happening during Shellbox work. I ended up doing all of the file-related Shellbox porting in MediaWiki even though it's really not "ServiceOps" responsibility.

There's legacy cruft like T290759: Undeploy VipsScaler from Wikimedia wikis, which is both unused and used at the same time. T40010: RFC: Re-evaluate librsvg as SVG renderer on Wikimedia wikis is a very sad story of people putting in a lot of work to the point of building a new SVG renderer that is really just blocked on someone packaging stuff up and getting it installed (and then dealing with ensuing bugs and updates, etc.).

I believe the bug counts listed in the Google Doc are underselling the scope of the work, because many issues are scattered across MediaWiki-File-management and Wikimedia-SVG-rendering too.

To be clear, thumbor is just one part of the problem, the entire file management stack needs a dedicated owner.

For emphasis, let me add that the effects of the lack of ownership there is visible to the community / end users. Speaking from the perspective of a community member, having a Multimedia team again would be good but there also needs to be somewhere the buck stops on the entirety of the file management stack. That can certainly be a "Multimedia team", but then the scope needs to be very clear so that the necessary resources are allocated. Because it's not obvious just from the name that things like Thumbor, or file metadata storage, or rearchitecting file operations to not require a database lock, or… fall within the scope of a "multimedia" team.

This stuff is fundamentals, and can't be left to random acts of kindness by people on other teams.

greg added a subscriber: EChetty.

(removing @EChetty as assignee, was that a mis-click assignment? I can't find your name on https://office.wikimedia.org/wiki/Contact_list)

DAbad renamed this task from Code Stewardship Review - Thumbor to <Code Stewardship Review> Thumbor.Dec 6 2021, 3:54 PM

This is also blocking the work on adding support for chemical markup files: T18491: Support for Chemical Markup Language

The Python 3 upgrade is trickier. Upstream released an alpha version with Python 3 support (Thumbor 7) in February 2020, but has had very little activity since (none in the last 6 months and no releases of any kind in the last 18 months).

7.0.0 was upgraded from Alpha to Beta in November 2021, and was upgraded to a full release about a week ago.

DAbad triaged this task as Medium priority.
DAbad moved this task from Backlog to Investigate on the Foundational Technology Requests board.

March 11, 2022 Update

  • As per internal discussions, the PET team will take ownership of Thumbor until we resolve the larger media support question with Product Ownership falling Platform PM team.

We have outlined the following items to be worked on:

We are currently aligning resources to pick up this work and will provide update on when we expect to start.

Change 770910 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update point of contact for Thumbor

https://gerrit.wikimedia.org/r/770910

Change 770910 merged by Muehlenhoff:

[operations/puppet@production] Update point of contact for Thumbor

https://gerrit.wikimedia.org/r/770910

DAbad changed the task status from Open to In Progress.Apr 28 2022, 6:47 PM

April 28, 2022 Update

  • Migrating test suite from Nosetests to pytest
  • Removed dependency on defunct Thumbor plugin lib
  • Working on helm chart for k8s

For everyone's info, currently no Code-Stewardship-Reviews are taking place as there is no clear path forward and as this is not prioritized work.
(Entirely personal opinion: I also assume lack of decision authority due to WMF not having a CTO currently. However, discussing this is off-topic for this task.)