Page MenuHomePhabricator

Jgiannelos (jgiannelos)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
May 5 2020, 11:24 AM (207 w, 3 d)
Availability
Available
IRC Nick
nemo-yiannis
LDAP User
Jgiannelos
MediaWiki User
JGiannelos (WMF) [ Global Accounts ]

Recent Activity

Tue, Apr 23

Jgiannelos closed T349972: REST API `mobile-html` end-point thumbnails resolution are too big as Resolved.

PCS now serves mobile-html with data-file-original-src to indicate that the image was upscaled.

Tue, Apr 23, 3:39 PM · Essential-Work, Content-Transform-Team-WIP, Mobile-Content-Service, Page Content Service, affects-Kiwix-and-openZIM, API Platform
Jgiannelos closed T362876: Mobile-html doesn't render the expected leading paragraph as Resolved.
Tue, Apr 23, 3:28 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos closed T312667: Articles have incorrect horizontal scrolling with blank space. as Resolved.
Tue, Apr 23, 3:28 PM · Essential-Work, Content-Transform-Team-WIP, ios-app-v7.0, Wikipedia-Android-App-Backlog (Android Release - FY2023-24), Page Content Service, Wikipedia-iOS-App-Backlog, Product-Infrastructure-Team-Backlog-Deprecated
Jgiannelos added a comment to T355730: Provide developer access to the cassandra-dev cluster.

Hi, is there any update with dev access for PCS devs?

Tue, Apr 23, 3:16 PM · Cassandra
Jgiannelos moved T363168: Upgrade mobileapps to node 18 from Backlog to In Progress on the Content-Transform-Team-WIP board.
Tue, Apr 23, 2:08 PM · Platform Engineering, Trust and Safety Product Team (Engineering), Essential-Work, Content-Transform-Team-WIP, Page Content Service, MediaWiki-Engineering, [DEPRECATED] wdwb-tech, Wikidata, Citoid, CX-cxserver, Wikidata-Termbox, Wikimedia-Portals, Data-Engineering, serviceops
Jgiannelos updated the task description for T363168: Upgrade mobileapps to node 18.
Tue, Apr 23, 2:08 PM · Platform Engineering, Trust and Safety Product Team (Engineering), Essential-Work, Content-Transform-Team-WIP, Page Content Service, MediaWiki-Engineering, [DEPRECATED] wdwb-tech, Wikidata, Citoid, CX-cxserver, Wikidata-Termbox, Wikimedia-Portals, Data-Engineering, serviceops
Jgiannelos claimed T363168: Upgrade mobileapps to node 18.
Tue, Apr 23, 2:07 PM · Platform Engineering, Trust and Safety Product Team (Engineering), Essential-Work, Content-Transform-Team-WIP, Page Content Service, MediaWiki-Engineering, [DEPRECATED] wdwb-tech, Wikidata, Citoid, CX-cxserver, Wikidata-Termbox, Wikimedia-Portals, Data-Engineering, serviceops
Jgiannelos created T363168: Upgrade mobileapps to node 18.
Tue, Apr 23, 2:06 PM · Platform Engineering, Trust and Safety Product Team (Engineering), Essential-Work, Content-Transform-Team-WIP, Page Content Service, MediaWiki-Engineering, [DEPRECATED] wdwb-tech, Wikidata, Citoid, CX-cxserver, Wikidata-Termbox, Wikimedia-Portals, Data-Engineering, serviceops
Jgiannelos updated the task description for T362988: CTT tasks week of 2024-04-19.
Tue, Apr 23, 1:38 PM · Essential-Work, MW-1.43-notes (1.43.0-wmf.2; 2024-04-23), Content-Transform-Team-WIP
Jgiannelos moved T343387: /page/summary/ and /page/mobile-html/ do not get the latest article description in zhwiki from To Deploy to To Verify on the Content-Transform-Team-WIP board.
Tue, Apr 23, 1:36 PM · Essential-Work, Page Content Service, Chinese-Sites, Content-Transform-Team-WIP
Jgiannelos moved T312667: Articles have incorrect horizontal scrolling with blank space. from To Deploy to To Verify on the Content-Transform-Team-WIP board.
Tue, Apr 23, 1:36 PM · Essential-Work, Content-Transform-Team-WIP, ios-app-v7.0, Wikipedia-Android-App-Backlog (Android Release - FY2023-24), Page Content Service, Wikipedia-iOS-App-Backlog, Product-Infrastructure-Team-Backlog-Deprecated
Jgiannelos moved T362876: Mobile-html doesn't render the expected leading paragraph from To Deploy to To Verify on the Content-Transform-Team-WIP board.
Tue, Apr 23, 1:36 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos moved T362876: Mobile-html doesn't render the expected leading paragraph from Code Review to To Deploy on the Content-Transform-Team-WIP board.
Tue, Apr 23, 12:26 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service

Fri, Apr 19

Jgiannelos claimed T362982: CTT tasks week of 19/04/2024.
Fri, Apr 19, 1:54 PM · Content-Transform-Team-WIP
Jgiannelos added a project to T362982: CTT tasks week of 19/04/2024: Content-Transform-Team-WIP.
Fri, Apr 19, 1:45 PM · Content-Transform-Team-WIP
Jgiannelos created T362982: CTT tasks week of 19/04/2024.
Fri, Apr 19, 1:44 PM · Content-Transform-Team-WIP
Jgiannelos committed rIVDI7a0445ba75fc: Reduce noise on visualdiff testing.
Reduce noise on visualdiff testing
Fri, Apr 19, 11:42 AM

Thu, Apr 18

Jgiannelos moved T362876: Mobile-html doesn't render the expected leading paragraph from Backlog to Code Review on the Content-Transform-Team-WIP board.
Thu, Apr 18, 3:33 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos added a project to T362876: Mobile-html doesn't render the expected leading paragraph: Essential-Work.
Thu, Apr 18, 3:33 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos added a watcher for Content-Transform-Team-WIP: Jgiannelos.
Thu, Apr 18, 3:30 PM
Jgiannelos added a comment to T362876: Mobile-html doesn't render the expected leading paragraph.

From a quick look this is the <p> element that its chosen instead of the actual first paragraph:

image.png (409×2 px, 131 KB)

Thu, Apr 18, 1:00 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos added projects to T362876: Mobile-html doesn't render the expected leading paragraph: Page Content Service, Content-Transform-Team-WIP.
Thu, Apr 18, 12:44 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos created T362876: Mobile-html doesn't render the expected leading paragraph.
Thu, Apr 18, 12:43 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service

Mon, Apr 15

Jgiannelos updated the task description for T362559: Add dependency for GeoData extension required to get images info for summary.
Mon, Apr 15, 4:55 PM · Page Content Service
Jgiannelos updated the task description for T362558: Add dependency for PageImages extension required to get images info for summary.
Mon, Apr 15, 4:50 PM · Page Content Service
Jgiannelos updated the task description for T362558: Add dependency for PageImages extension required to get images info for summary.
Mon, Apr 15, 4:49 PM · Page Content Service
Jgiannelos updated the task description for T362557: Add dependency injection for RevisionLookup required to generate the summary output.
Mon, Apr 15, 4:44 PM · Page Content Service
Jgiannelos updated the task description for T362556: Add dependency injection for PageProps required to generate the summary output.
Mon, Apr 15, 4:43 PM · Page Content Service
Jgiannelos created T362560: Find a way to get description information from Wikibase .
Mon, Apr 15, 4:41 PM · Page Content Service
Jgiannelos created T362559: Add dependency for GeoData extension required to get images info for summary.
Mon, Apr 15, 4:40 PM · Page Content Service
Jgiannelos created T362558: Add dependency for PageImages extension required to get images info for summary.
Mon, Apr 15, 4:40 PM · Page Content Service
Jgiannelos created T362557: Add dependency injection for RevisionLookup required to generate the summary output.
Mon, Apr 15, 4:39 PM · Page Content Service
Jgiannelos created T362556: Add dependency injection for PageProps required to generate the summary output.
Mon, Apr 15, 4:38 PM · Page Content Service
Jgiannelos created T362555: [prototyping] Create a MW extension that exposes the page summary output.
Mon, Apr 15, 4:37 PM · Page Content Service

Fri, Apr 12

Jgiannelos added projects to T362418: visualdiff: Kartographer overlay colour on maps has different tone : Maps (Kartographer), Parsoid.
Fri, Apr 12, 2:35 PM · Parsoid-Read-Views (Phase 1 - DiscussionTools support), Content-Transform-Team-WIP, Regression, Parsoid, Maps (Kartographer)
Jgiannelos added a comment to T362418: visualdiff: Kartographer overlay colour on maps has different tone .

After a bit of debugging this looks like its the issue:

image.png (196×1 px, 32 KB)

image.png (124×1 px, 19 KB)

The svg renders 2 elements for parsoid but one with legacy parser output.

Fri, Apr 12, 2:34 PM · Parsoid-Read-Views (Phase 1 - DiscussionTools support), Content-Transform-Team-WIP, Regression, Parsoid, Maps (Kartographer)
Jgiannelos updated the task description for T362418: visualdiff: Kartographer overlay colour on maps has different tone .
Fri, Apr 12, 2:32 PM · Parsoid-Read-Views (Phase 1 - DiscussionTools support), Content-Transform-Team-WIP, Regression, Parsoid, Maps (Kartographer)
Jgiannelos created T362418: visualdiff: Kartographer overlay colour on maps has different tone .
Fri, Apr 12, 2:31 PM · Parsoid-Read-Views (Phase 1 - DiscussionTools support), Content-Transform-Team-WIP, Regression, Parsoid, Maps (Kartographer)
Jgiannelos committed rIVDI465d88c1a75f: Force lazy elements to be loaded.
Force lazy elements to be loaded
Fri, Apr 12, 1:55 PM
Jgiannelos committed rIVDI71d7763c642e: puppeteer: Wait until network is idle before diffing.
puppeteer: Wait until network is idle before diffing
Fri, Apr 12, 8:49 AM

Thu, Apr 11

Jgiannelos closed T361938: Decide on prototyping next steps as Resolved.
Thu, Apr 11, 2:50 PM · Page Content Service
Jgiannelos closed T361938: Decide on prototyping next steps, a subtask of T361936: [EPIC] Port page summary endpoint to MW, as Resolved.
Thu, Apr 11, 2:50 PM · Page Content Service
Jgiannelos claimed T361938: Decide on prototyping next steps.

For now it looks like working on an extension with good abstractions so it can easily be implemented in core looks like the most feasible way forward.
I boostrapped the extension code for the prototyping work here:
https://gitlab.wikimedia.org/repos/content-transform/pagesummary

Thu, Apr 11, 2:49 PM · Page Content Service

Wed, Apr 10

Jgiannelos moved T343387: /page/summary/ and /page/mobile-html/ do not get the latest article description in zhwiki from Backlog to To Deploy on the Content-Transform-Team-WIP board.
Wed, Apr 10, 6:07 PM · Essential-Work, Page Content Service, Chinese-Sites, Content-Transform-Team-WIP

Tue, Apr 9

Jgiannelos committed rIVDI219154fbdd39: Fix MW API IP blocking issues.
Fix MW API IP blocking issues
Tue, Apr 9, 6:12 PM

Mon, Apr 8

Jgiannelos closed T361095: Configure and run visualdifftesting for hebrew wikivoyage, a subtask of T361025: Deploy parsoid read views for hebrew wikivoyage, as Resolved.
Mon, Apr 8, 3:40 PM · Patch-For-Review, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Parsoid-Rendering, Platform Team Legacy (Designing), MediaWiki-General, Product-Infrastructure-Team-Backlog-Deprecated, Services (designing), Notice, Roadmap, Epic, VisualEditor-Performance, Blocked-on-Services, RESTBase, Service-Architecture, VisualEditor, MediaWiki-User-Interface
Jgiannelos closed T361095: Configure and run visualdifftesting for hebrew wikivoyage as Resolved.
Mon, Apr 8, 3:40 PM · Patch-For-Review, Parsoid, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP
Jgiannelos moved T361095: Configure and run visualdifftesting for hebrew wikivoyage from In Progress to To Verify on the Content-Transform-Team-WIP board.
Mon, Apr 8, 3:40 PM · Patch-For-Review, Parsoid, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP
Jgiannelos added a comment to T361608: RESTBase scap deployment failed.

It failed again but with a different error:

:* restbase1030.eqiad.wmnet
09:48:11 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'restbase/deploy', '-g', 'canary', 'fetch', '--refresh-config'] (ran as deploy-service@restbase1030.eqiad.wmnet) returned [70]: Registering scripts in directory '/srv/deployment/restbase/deploy-cache/revs/c4d19d7c4e2b4e8d16cd12677d44b42962c05917/scap/scripts'
Fetch from: http://deploy1002.eqiad.wmnet/restbase/deploy/.git
Running ['git', 'remote', 'set-url', 'origin', 'http://deploy1002.eqiad.wmnet/restbase/deploy/.git'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Running ['git', 'fetch', '--tags', '--jobs', '38', '--no-recurse-submodules'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Running ['git', 'config', 'lfs.url', 'https://gerrit.wikimedia.org/r/p/mediawiki/services/restbase/deploy.git/info/lfs'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Update submodules
git submodule sync
Running ['git', 'submodule', 'sync', '--recursive'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Fetch submodules
Remapping submodule /srv/deployment/restbase/deploy-cache/cache to http://deploy1002.eqiad.wmnet/restbase/deploy/.git
Updating .gitmodule: /srv/deployment/restbase/deploy-cache/cache
Running ['git', 'checkout', '.gitmodules'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Running ['git', 'config', '--list', '--file', '.gitmodules'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
git submodule sync
Running ['git', 'submodule', 'sync', '--recursive'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Running ['git', 'submodule', 'update', '--init', '--recursive', '--jobs', '38'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache', 'env': {'SHELL': '/bin/bash', 'PWD': '/var/lib/deploy-service', 'LOGNAME': 'deploy-service', 'XDG_SESSION_TYPE': 'tty', 'MOTD_SHOWN': 'pam', 'HOME': '/var/lib/deploy-service', 'LANG': 'en_US.UTF-8', 'SSH_CONNECTION': '10.64.32.28 43578 10.64.48.228 22', 'XDG_SESSION_CLASS': 'user', 'USER': 'deploy-service', 'SHLVL': '0', 'XDG_SESSION_ID': '47018', 'XDG_RUNTIME_DIR': '/run/user/497', 'SSH_CLIENT': '10.64.32.28 43578 22', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/games', '_': '/usr/bin/scap', 'PHP': 'php7.4', 'SSH_AUTH_SOCK': '/run/keyholder/proxy.sock', 'GIT_LFS_SKIP_SMUDGE': '1'}, 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Setting lfs.url of restbase to https://gerrit.wikimedia.org/r/mediawiki/services/restbase/info/lfs
Running ['git', 'config', 'lfs.url', 'https://gerrit.wikimedia.org/r/mediawiki/services/restbase/info/lfs'] with {'cwd': '/srv/deployment/restbase/deploy-cache/cache/restbase', 'stdout': -1, 'stderr': -1, 'text': True, 'stdin': -3}
Unhandled error:
deploy-local failed: <FileNotFoundError> {}
Mon, Apr 8, 9:50 AM · Scap, Release-Engineering-Team, RESTBase

Fri, Apr 5

Jgiannelos created T361941: Design an architecture for the PHP implementation of summary on MW.
Fri, Apr 5, 1:29 PM · Page Content Service
Jgiannelos created T361939: Map current summary dependencies to MW.
Fri, Apr 5, 1:27 PM · Page Content Service
Jgiannelos updated the task description for T361938: Decide on prototyping next steps.
Fri, Apr 5, 1:19 PM · Page Content Service
Jgiannelos created T361938: Decide on prototyping next steps.
Fri, Apr 5, 1:16 PM · Page Content Service
Jgiannelos added a parent task for T214000: Evaluate difficulty of porting PCS summary logic to PHP: T361936: [EPIC] Port page summary endpoint to MW.
Fri, Apr 5, 1:08 PM · Essential-Work, Content-Transform-Team-WIP, Page Content Service
Jgiannelos added a subtask for T361936: [EPIC] Port page summary endpoint to MW: T214000: Evaluate difficulty of porting PCS summary logic to PHP.
Fri, Apr 5, 1:08 PM · Page Content Service
Jgiannelos created T361936: [EPIC] Port page summary endpoint to MW.
Fri, Apr 5, 1:08 PM · Page Content Service
Jgiannelos added a comment to T361095: Configure and run visualdifftesting for hebrew wikivoyage.

I added 4800 new titles from hewikivoyage:

MariaDB [parsoid_rv_deploy_targets]> SELECT prefix, COUNT(*) FROM pages GROUP BY prefix;                                                                                                                                                                                                                                    +--------------+----------+
| prefix       | COUNT(*) |
+--------------+----------+
| enwikivoyage |    24615 |
| hewikivoyage |     4800 |
+--------------+----------+
2 rows in set (0.137 sec)
Fri, Apr 5, 11:10 AM · Patch-For-Review, Parsoid, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP
Jgiannelos added a comment to T361608: RESTBase scap deployment failed.

Thanks @hashar

jgiannelos@deploy1002:/srv/deployment/restbase/deploy$ git tag -d scap/sync/2024-02-19/0011
Deleted tag 'scap/sync/2024-02-19/0011' (was d3425717)
Fri, Apr 5, 10:57 AM · Scap, Release-Engineering-Team, RESTBase

Thu, Apr 4

Jgiannelos added a comment to T361187: Issue with Stadium Locations Displayed Incorrectly on German Wikipedia.

Should be fixed in prod:

image.png (1×2 px, 1 MB)

Thu, Apr 4, 3:40 PM · Patch-For-Review, Content-Transform-Team-WIP, Page Content Service, Wikipedia-iOS-App-Backlog
Jgiannelos closed T361187: Issue with Stadium Locations Displayed Incorrectly on German Wikipedia as Resolved.
Thu, Apr 4, 3:39 PM · Patch-For-Review, Content-Transform-Team-WIP, Page Content Service, Wikipedia-iOS-App-Backlog
Jgiannelos added a comment to T328036: MCS decommission (2023).

MCS is still used by kiwix. I think wikiwand has stopped using the endpoint.

Thu, Apr 4, 2:13 PM · Essential-Work, Content-Transform-Team-WIP, Mobile-Content-Service
Jgiannelos moved T361095: Configure and run visualdifftesting for hebrew wikivoyage from Backlog to In Progress on the Content-Transform-Team-WIP board.
Thu, Apr 4, 1:28 PM · Patch-For-Review, Parsoid, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP
Jgiannelos added a comment to T350507: Update mobileapps k8s deployment chart for Cassandra credentials.

Things look better on staging:

Thu, Apr 4, 10:39 AM · Content-Transform-Team, Patch-For-Review, Page Content Service, serviceops, RESTBase Sunsetting
Jgiannelos created P59454 (An Untitled Masterwork).
Thu, Apr 4, 10:38 AM
Jgiannelos created P59452 (An Untitled Masterwork).
Thu, Apr 4, 10:36 AM
Jgiannelos added a comment to T350507: Update mobileapps k8s deployment chart for Cassandra credentials.

From staging:

{
  "status": 500,
  "type": "internal_error",
  "title": "ArgumentError",
  "detail": "Datacenter eqiad was not found. Available DCs are: [codfw]",
  "method": "GET",
  "uri": "/en.wikipedia.org/v1/page/mobile-html/Dog"
}
Thu, Apr 4, 10:06 AM · Content-Transform-Team, Patch-For-Review, Page Content Service, serviceops, RESTBase Sunsetting

Wed, Apr 3

Jgiannelos added a comment to T350507: Update mobileapps k8s deployment chart for Cassandra credentials.

I think the problem is on the nodejs cassandra client TLS initialization and more specifically on how we pass the config options.

Wed, Apr 3, 1:29 PM · Content-Transform-Team, Patch-For-Review, Page Content Service, serviceops, RESTBase Sunsetting
Jgiannelos added a comment to T350507: Update mobileapps k8s deployment chart for Cassandra credentials.

From staging:

Wed, Apr 3, 11:28 AM · Content-Transform-Team, Patch-For-Review, Page Content Service, serviceops, RESTBase Sunsetting
Jgiannelos added a comment to T350507: Update mobileapps k8s deployment chart for Cassandra credentials.

I am testing things on staging and I am getting this error (and a CrashLoopBackOff from the pod):

ENOENT: no such file or directory, open '/etc/ssl/certs/wmf-ca-certificates.crt'",
Wed, Apr 3, 10:58 AM · Content-Transform-Team, Patch-For-Review, Page Content Service, serviceops, RESTBase Sunsetting

Tue, Apr 2

Jgiannelos added projects to T361608: RESTBase scap deployment failed: RESTBase, Release-Engineering-Team.
Tue, Apr 2, 3:37 PM · Scap, Release-Engineering-Team, RESTBase
Jgiannelos created T361608: RESTBase scap deployment failed.
Tue, Apr 2, 3:37 PM · Scap, Release-Engineering-Team, RESTBase

Thu, Mar 28

Jgiannelos added a comment to T361187: Issue with Stadium Locations Displayed Incorrectly on German Wikipedia.

I verified the ticket on mobile-html endpoint. Indeed the pins don't match between desktop and mobile-html.
This is also not reproducible in parsoid either so it must be a transformation on the PCS level that moves the pins to the wrong position.

Thu, Mar 28, 11:03 AM · Patch-For-Review, Content-Transform-Team-WIP, Page Content Service, Wikipedia-iOS-App-Backlog
Jgiannelos edited projects for T361187: Issue with Stadium Locations Displayed Incorrectly on German Wikipedia, added: Page Content Service; removed Maps.
Thu, Mar 28, 10:27 AM · Patch-For-Review, Content-Transform-Team-WIP, Page Content Service, Wikipedia-iOS-App-Backlog

Mar 27 2024

Jgiannelos added a comment to T351931: With parsoid categories are not shown.

Pending from initial parsoid patch:

  • Tests for
    • [[Category:Foo|{{1x|}}]] (missing sort key)
    • [[Category:Foo|Category:Foo]] (category as sort key)
    • [[Category:Foo|\n]] (newline as sort key)
    • Sort key that is language converted
Mar 27 2024, 12:26 PM · Patch-For-Review, Content-Transform-Team-WIP, Parsoid (Tracking), Parsoid-Read-Views (Phase 2 - testwiki Main namespace support)
Jgiannelos claimed T361095: Configure and run visualdifftesting for hebrew wikivoyage.
Mar 27 2024, 12:22 PM · Patch-For-Review, Parsoid, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP
Jgiannelos created T361095: Configure and run visualdifftesting for hebrew wikivoyage.
Mar 27 2024, 12:21 PM · Patch-For-Review, Parsoid, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP

Mar 25 2024

Jgiannelos added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

So the root cause looks like is the following:

Mar 25 2024, 6:56 PM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

It looks like errors/latency are stabilized after depooling some nodes:

image.png (1×3 px, 507 KB)

Mar 25 2024, 5:18 PM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

It looks like this path was not deployed using scap:
https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1009842

Mar 25 2024, 1:29 PM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos created P58908 (An Untitled Masterwork).
Mar 25 2024, 1:22 PM
Jgiannelos moved T360597: Increased latency, timeouts from wikifeeds since march 10th from Backlog to In Progress on the Content-Transform-Team-WIP board.
Mar 25 2024, 1:05 PM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

From the URLs from logstash as @hnowlan pointed out it looks like the main cause of timeouts is outgoing requests to v1/<domain>/page/summary/<title> and more specifically to media files (svg, png, webm).
From restbase:

jgiannelos@deploy1002:~$ curl -v restbase.svc.codfw.wmnet:7233/fr.wikipedia.org/v1/page/summary/Fichier%3ACleopatra_poster.jpg
Mar 25 2024, 1:05 PM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

I was trying to see if there is a correlation between this issue and switching over parsoid from restbase to MW core but it doesn't look something is related.

Mar 25 2024, 11:45 AM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos added a comment to T360597: Increased latency, timeouts from wikifeeds since march 10th.

From logs I think there are 2 things to investigate:

  • What happened since ~10th March ?
  • Why traffic was completely dropped on the 19th of March on codfw?
    • Could be related to the datacenter switchover ?
    • Meanwhile eqiad kept having traffic before and after that time.
Mar 25 2024, 11:26 AM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team
Jgiannelos added a project to T360597: Increased latency, timeouts from wikifeeds since march 10th: Content-Transform-Team-WIP.
Mar 25 2024, 10:59 AM · Content-Transform-Team-WIP, Patch-For-Review, serviceops, Content-Transform-Team

Mar 15 2024

Jgiannelos created P58804 (An Untitled Masterwork).
Mar 15 2024, 2:34 PM
Jgiannelos created P58803 (An Untitled Masterwork).
Mar 15 2024, 2:20 PM

Mar 14 2024

Jgiannelos closed T359509: REST API calls suddenly all returning 400 as Resolved.
Mar 14 2024, 2:01 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MW-Interfaces-Team, Content-Transform-Team-WIP, Patch-For-Review, RESTBase-API
Jgiannelos closed T359509: REST API calls suddenly all returning 400, a subtask of T354439: 1.42.0-wmf.21 deployment blockers, as Resolved.
Mar 14 2024, 2:01 PM · Release-Engineering-Team (Now this 🫠), Release, Train Deployments
Jgiannelos added a comment to T359509: REST API calls suddenly all returning 400.

Metrics looks better now after the backport fix:

image.png (1×1 px, 377 KB)

Mar 14 2024, 2:01 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MW-Interfaces-Team, Content-Transform-Team-WIP, Patch-For-Review, RESTBase-API
Jgiannelos raised the priority of T360105: Increase on 400 status code on page/html (parsoid) responses from High to Unbreak Now!.
Mar 14 2024, 12:33 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos added a comment to T360105: Increase on 400 status code on page/html (parsoid) responses.

It looks like this fix was backported last week as a hotfix: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009542
But never merged to master so now the same bug got triggered.

Mar 14 2024, 12:30 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos triaged T360105: Increase on 400 status code on page/html (parsoid) responses as High priority.
Mar 14 2024, 12:24 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos updated the task description for T360105: Increase on 400 status code on page/html (parsoid) responses.
Mar 14 2024, 12:23 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos updated the task description for T360105: Increase on 400 status code on page/html (parsoid) responses.
Mar 14 2024, 12:22 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos updated the task description for T360105: Increase on 400 status code on page/html (parsoid) responses.
Mar 14 2024, 12:21 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos created T360105: Increase on 400 status code on page/html (parsoid) responses.
Mar 14 2024, 12:19 PM · RESTBase Sunsetting, RESTBase, Parsoid
Jgiannelos closed T347034: RESTBase /v1/related endpoint should call the MW action API with a GET not a POST as Resolved.
Mar 14 2024, 10:24 AM · API Platform, RESTBase Sunsetting, Essential-Work, Wikifeeds, Sustainability (Incident Followup), Discovery-Search
Jgiannelos closed T347034: RESTBase /v1/related endpoint should call the MW action API with a GET not a POST, a subtask of T346945: 2023-09-20 Elasticsearch unavailable incident, as Resolved.
Mar 14 2024, 10:23 AM · Discovery-Search (Current work), Wikimedia-Incident, SRE-OnFire, Data-Platform-SRE

Mar 12 2024

Jgiannelos created T359964: Move page lint job outside of restbase request/response cycle to allow completely disabling cassandra writes.
Mar 12 2024, 5:09 PM · Content-Transform-Team-WIP, RESTBase Sunsetting, Epic