Page MenuHomePhabricator
Projects SRE

SREGroup
ActivePublic
Watch Project

Members (28)

Watchers (15)

Details

Description

Homepage: https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering

If you're filing a task that needs to be handled by the SRE team, add a tag from the list below. If you're not sure which one to pick, add the SRE tag and the on-duty person will redirect it as appropriate.

Cross-team tags:

Tags by team:

Recent Activity
View All

Today

Dzahn reassigned T363360: Requesting membership in airflow-analytics-product-admins for hghani from OSefu-WMF to BCornwall.
Thu, Apr 25, 4:10 AM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Dzahn moved T363360: Requesting membership in airflow-analytics-product-admins for hghani from Awaiting User Input to Patch in Review on the SRE-Access-Requests board.
Thu, Apr 25, 4:10 AM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Dzahn added a comment to T363360: Requesting membership in airflow-analytics-product-admins for hghani.
In T363360#9742998, @Hghani wrote:

My contract expiry date is June 30th 2024. I believe the contact should be @OSefu-WMF.

Thu, Apr 25, 4:09 AM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
OSefu-WMF added a comment to T363360: Requesting membership in airflow-analytics-product-admins for hghani.

Approved

Thu, Apr 25, 12:33 AM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests

Yesterday

BCornwall reassigned T363360: Requesting membership in airflow-analytics-product-admins for hghani from BCornwall to OSefu-WMF.

All we're waiting on is @OSefu-WMF 's approver

Wed, Apr 24, 10:46 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Dzahn added a comment to T349402: eqiad: 1 VM requested for community-crm.

https://community-crm.wikimedia.org/ is now online 🎉

Wed, Apr 24, 10:28 PM · Patch-For-Review, fundraising-tech-ops, vm-requests, Infrastructure-Foundations, SRE
gerritbot added a project to T363360: Requesting membership in airflow-analytics-product-admins for hghani: Patch-For-Review.
Wed, Apr 24, 10:27 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a comment to T363360: Requesting membership in airflow-analytics-product-admins for hghani.

Change #1023965 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] admin: Move hghani to airflow-analytics-product-admins

https://gerrit.wikimedia.org/r/1023965

Wed, Apr 24, 10:27 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
BCornwall updated the task description for T363360: Requesting membership in airflow-analytics-product-admins for hghani.
Wed, Apr 24, 10:25 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
BCornwall closed T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf as Resolved.

@nshahquinn-wmf Give it a few minutes to propagate and your access should be all set. Thanks!

Wed, Apr 24, 10:20 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a comment to T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf.

Change #1023963 merged by BCornwall:

[operations/puppet@production] admin: Add nshahquinn-wmf to airflow-analytics-product-admins

https://gerrit.wikimedia.org/r/1023963

Wed, Apr 24, 10:18 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Hghani added a comment to T363360: Requesting membership in airflow-analytics-product-admins for hghani.

Hi,
My contract expiry date is June 30th 2024. I believe the contact should be @OSefu-WMF.

Wed, Apr 24, 10:16 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Dzahn moved T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf from Untriaged to Patch in Review on the SRE-Access-Requests board.
Wed, Apr 24, 10:14 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Dzahn moved T363377: Requesting access to deployment shell access for Jsn.sherman from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Wed, Apr 24, 10:13 PM · SRE, SRE-Access-Requests
Dzahn moved T363360: Requesting membership in airflow-analytics-product-admins for hghani from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Wed, Apr 24, 10:13 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
Dzahn added a comment to T363360: Requesting membership in airflow-analytics-product-admins for hghani.

The email address contains the -ctr suffix. For contractors please provide an expiry_date and expiry_contact. On that date we will ask if the access should be removed or extended.

Wed, Apr 24, 10:13 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023964 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] wmnet: add CNAME records for commons-impact-analytics (k8s ingress)

https://gerrit.wikimedia.org/r/1023964

Wed, Apr 24, 10:12 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
BCornwall changed the status of T363360: Requesting membership in airflow-analytics-product-admins for hghani from Open to In Progress.
Wed, Apr 24, 10:11 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
BCornwall changed the status of T363377: Requesting access to deployment shell access for Jsn.sherman from Open to In Progress.
Wed, Apr 24, 10:09 PM · SRE, SRE-Access-Requests
BCornwall claimed T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf.
Wed, Apr 24, 10:09 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a project to T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf: Patch-For-Review.
Wed, Apr 24, 10:08 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a comment to T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf.

Change #1023963 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] admin: Add nshahquinn-wmf to airflow-analytics-product-admins

https://gerrit.wikimedia.org/r/1023963

Wed, Apr 24, 10:07 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a comment to T349402: eqiad: 1 VM requested for community-crm.

Change #983951 merged by Dzahn:

[operations/puppet@production] Add CDN configuration for new community-crm

https://gerrit.wikimedia.org/r/983951

Wed, Apr 24, 10:06 PM · Patch-For-Review, fundraising-tech-ops, vm-requests, Infrastructure-Foundations, SRE
BCornwall updated the task description for T363288: Requesting membership in airflow-analytics-product-admins for nshahquinn-wmf.
Wed, Apr 24, 10:06 PM · Patch-For-Review, Movement-Insights, SRE, SRE-Access-Requests
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023962 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] DNM: service: move commons-impact-analytics service to production state

https://gerrit.wikimedia.org/r/1023962

Wed, Apr 24, 10:02 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023961 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] service: add commons-impact-analytics AQS 2.0 service

https://gerrit.wikimedia.org/r/1023961

Wed, Apr 24, 10:02 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023960 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] DNM: cassandra: add commons_impact_analytics user

https://gerrit.wikimedia.org/r/1023960

Wed, Apr 24, 10:00 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023959 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] kubernetes: add usernames for commons-impact-analytics to deployment server

https://gerrit.wikimedia.org/r/1023959

Wed, Apr 24, 10:00 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023958 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] DNM: rest-gateway: route commons-analytics via rest-gateway

https://gerrit.wikimedia.org/r/1023958

Wed, Apr 24, 9:58 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023957 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] DNM: services: add commons-impact-analytics service helmfile configs

https://gerrit.wikimedia.org/r/1023957

Wed, Apr 24, 9:57 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a project to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production: Patch-For-Review.
Wed, Apr 24, 9:57 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production.

Change #1023956 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] admin_ng: add namespace for commons-impact-analytics

https://gerrit.wikimedia.org/r/1023956

Wed, Apr 24, 9:57 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
Scott_French changed the status of T361835: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production from Open to In Progress.

Thanks, all, for the details shared thus far.

Wed, Apr 24, 9:53 PM · Patch-For-Review, Data Products (Data Products Sprint 12), serviceops, Service-deployment-requests, SRE
gerritbot added a comment to T363415: upgrade deployment servers to bullseye / add bullseye support to puppet role.

Change #1023955 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] deployment_server: add bullseye support, python3 package names

https://gerrit.wikimedia.org/r/1023955

Wed, Apr 24, 9:30 PM · Patch-For-Review, serviceops, SRE
gerritbot added a project to T363415: upgrade deployment servers to bullseye / add bullseye support to puppet role: Patch-For-Review.
Wed, Apr 24, 9:21 PM · Patch-For-Review, serviceops, SRE
gerritbot added a comment to T363415: upgrade deployment servers to bullseye / add bullseye support to puppet role.

Change #1023954 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] redis: use python3-redis to support bullseye

https://gerrit.wikimedia.org/r/1023954

Wed, Apr 24, 9:21 PM · Patch-For-Review, serviceops, SRE
gerritbot added a comment to T360439: Phase out cergen for Search Platform services.

Change #1023440 merged by Bking:

[operations/puppet@production] Replace tabs with 4 spaces in tlsproxy nginx.conf

https://gerrit.wikimedia.org/r/1023440

Wed, Apr 24, 9:14 PM · Patch-For-Review, Data-Platform-SRE (2024.04.15 - 2024.05.05), SRE
Dzahn added a parent task for T363415: upgrade deployment servers to bullseye / add bullseye support to puppet role: T360964: replace buster machines in devtools project.
Wed, Apr 24, 9:04 PM · Patch-For-Review, serviceops, SRE
Dzahn created T363415: upgrade deployment servers to bullseye / add bullseye support to puppet role.
Wed, Apr 24, 9:03 PM · Patch-For-Review, serviceops, SRE
Dzahn updated subscribers of T291916: Tracking task for Bullseye migrations in production.

@Muehlenhoff Where does deploy* (deployment_server role both prod and wmcs) fit in? Since we are still on buster there. But want bullseye deployment_servers in cloud VPS projects and production hasn't upgraded the role yet. A legit subtask for here?

Wed, Apr 24, 8:53 PM · Epic, Infrastructure-Foundations, SRE
gerritbot added a comment to T360414: Phase out cergen for Observability services.

Change #1018749 merged by Andrea Denisse:

[operations/puppet@production] prometheus: Ensure TLS certificates are provided by CFSSL

https://gerrit.wikimedia.org/r/1018749

Wed, Apr 24, 8:43 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE
Stashbot added a comment to T360414: Phase out cergen for Observability services.

Mentioned in SAL (#wikimedia-operations) [2024-04-24T20:38:41Z] <denisse> Disabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414

Wed, Apr 24, 8:38 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE
Stashbot added a comment to T360414: Phase out cergen for Observability services.

Mentioned in SAL (#wikimedia-operations) [2024-04-24T20:38:15Z] <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414

Wed, Apr 24, 8:38 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE
Stashbot added a comment to T360414: Phase out cergen for Observability services.

Mentioned in SAL (#wikimedia-operations) [2024-04-24T20:37:53Z] <denisse@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414

Wed, Apr 24, 8:38 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE
Stashbot added a comment to T360414: Phase out cergen for Observability services.

Mentioned in SAL (#wikimedia-operations) [2024-04-24T20:37:20Z] <denisse> Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414

Wed, Apr 24, 8:37 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE
Maintenance_bot added a project to T363399: Q4:rack/setup/install parsoidtest1001: SRE.
Wed, Apr 24, 8:29 PM · SRE, serviceops, ops-eqiad, DC-Ops
Maintenance_bot added a project to T363409: PowerSupplyFailure: SRE.
Wed, Apr 24, 8:29 PM · SRE, ops-eqiad
VRiley-WMF added a comment to T362990: hw troubleshooting: memory DIMM_B3 multi-bit memory errors for prometheus1005.

This was a duplicate ticket that was opened for https://phabricator.wikimedia.org/T360687

Wed, Apr 24, 7:58 PM · Patch-For-Review, SRE, SRE Observability, ops-eqiad, DC-Ops
VRiley-WMF closed T362990: hw troubleshooting: memory DIMM_B3 multi-bit memory errors for prometheus1005 as Resolved.
Wed, Apr 24, 7:57 PM · Patch-For-Review, SRE, SRE Observability, ops-eqiad, DC-Ops
VRiley-WMF claimed T362990: hw troubleshooting: memory DIMM_B3 multi-bit memory errors for prometheus1005.
Wed, Apr 24, 7:56 PM · Patch-For-Review, SRE, SRE Observability, ops-eqiad, DC-Ops
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL