BTW in case it was not clear, my intentions here are basically:

• deploy something ASAP (like next week) that everyone is reasonably happy with for the interim
• don't do anything to get in the way of the badly-needed Envoy upgrade
• don't break anything else

In T363407#9743785, @JMeybohm wrote:

Thanks for the write-up!
What is not very clear to me is what part of the work would need to be done anyways (in case we'd have a envoy version >= 1.24). The reason I'm asking this is that envoy 1.23 is EOL since a year or so, so we need to look at an upgrade anyways.

Anyway I think that all that is needed to unblock VLAN migrations has been done or documented on this ticket? Optimistically closing but please re-open if you disagree.

In T360029#9725627, @Volans wrote:

As for the commit I advocate to add dbctl support in Spicerack but IIRC that requires changes in dbctl as most of its logic is in its CLI part and not exposed as a library, but to be checked.

I largely agree with Arzhel's assessment. At a cursory glance, Uruguay or Paraguay look ideal as first candidates.

I think you should be able to use the existing spicerack interface to confctl to do the set/host_ip=... action -- that should be equivalent to a ConftoolEntity.update call.

@Marostegui As it turns out, plain old confctl can be used to do this already.

In T360029#9722005, @Ladsgroup wrote:

In T360029#9658042, @CDanis wrote:

Actually the idea is that dbctl should not contain the IPs at all. It should look up the IP via DNS, we should store FQDN instead.

This has been fixed with this patch, which I forgot to associate with this bug.

Just to make sure I understand, the request here is an easy-to-automate way of dbctl to change the instance IP address?

Should this ticket really be "deprecate cergen"? :)

This would best be fixed by extending the haproxy bwlim work done in T317799 -- we've talked about having per-ASN limits in addition to the existing and partially-deployed per-file-URI limits.

Sent upstream as https://github.com/jaegertracing/helm-charts/pull/541

As it turns out, this required a change to the upstream chart:

I've verified that oauth2-proxy will silently just serve plain HTTP if you specify https_address but don't provide it with TLS key material. So I think I've provided it with such in this patch?

All pods on k8s-aux-eqiad restarted, thanks @akosiaris for the script.

Per docs, Thanos supports logging when a query is received but before it begins execution:

@Fabfur just wanted to make sure you've seen this task, it is decent documentation of the existing mechanism and probably helpful for doing T353910

The script was added to the wmf-sre-laptop package in May 2023 with this commit

Probably the "best" way to solve this is via the Alt-Svc mechanism, which Traffic means to experiment with at some point (T208242: Investigate using RFC 7838 Alternate Services to better optimize edge connections). That's something we want to do anyway, independent of this issue, and will also require a lot less new infrastructure work to support than the other alternatives here

In T351566#9408853, @akosiaris wrote:

Hey,

So, a couple of questions:

• profile::opentelemetry::collector, has 2 optional parameters, $otel_gateway_fqdn and $otel_gateway_otlp_port. Looking at the puppet code, if we don't supply these, we won't be configuring an otlp exporter (the otlp receiver will be enabled regardless and it's anyway orthogonal to the exporter). Are there plans to enable it in the near future?

hi serviceops, any plans to work on this soon? I/F would be happy to help with an implementation but we kind of want serviceops to figure out the right approach. cc @Kappakayala

Likely offending patch identified and is being reverted https://gerrit.wikimedia.org/r/c/mediawiki/core/+/979079

Conclusion at end of meeting was that o11y would migrate the base profile
to use the new cfssl support ~next week

Users are no longer impacted.

In T349244#9360471, @Fabfur wrote:

Looping in @CDanis as the original author for the cp1075 hiera overrides.
Do you think we can safely remove them or do we need to apply the same hiera data to another (new) cp hosts?

Thanks so much @pmiazga ! This is great to see and the spec chosen for the header sounds good.

This was very very likely a result of https://www.wikimediastatus.net/incidents/jrtsgd7jcpbl

Will be live in half an hour.

We discussed this in our Monday I/F meeting and approved it.

Nothing recent in the SAL.

Hi Issac, sorry this slipped through SRE's process as well -- this should have been taken care of last week.

In T252890#9165519, @ayounsi wrote:

@CDanis Is that still needed now that we have NEL?

The good news: OpenTelemetry tracing support exists as of our currently-deployed version of Envoy (v1.23.10): https://www.envoyproxy.io/docs/envoy/v1.23.10/api-v3/config/trace/v3/opentelemetry.proto.html