Change 667162 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add bullseye-wikimedia to apt.wikimedia.org

https://gerrit.wikimedia.org/r/667162

Change 667162 merged by Muehlenhoff:
[operations/puppet@production] Add bullseye-wikimedia to apt.wikimedia.org

https://gerrit.wikimedia.org/r/667162

Mentioned in SAL (#wikimedia-operations) [2021-03-17T13:58:41Z] <moritzm> added bullseye tftpboot environment T275873

Change 675104 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Add bullseye d-i config

https://gerrit.wikimedia.org/r/675104

Change 675104 merged by Muehlenhoff:
[operations/puppet@production] Add bullseye d-i config

https://gerrit.wikimedia.org/r/675104

Mentioned in SAL (#wikimedia-operations) [2021-03-26T13:02:27Z] <moritzm> reimaging theemin T275873

Change 677279 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] debian: Add an alias for bullseye

https://gerrit.wikimedia.org/r/677279

Change 677279 abandoned by Muehlenhoff:

[operations/puppet@production] debian: Add an alias for bullseye

Reason:

https://gerrit.wikimedia.org/r/677279

Mentioned in SAL (#wikimedia-operations) [2021-04-07T08:58:33Z] <moritzm> imported quickstack for bullseye/main (part of standard packages) T275873

Mentioned in SAL (#wikimedia-operations) [2021-04-07T09:40:50Z] <moritzm> imported git-lfs for bullseye/main (part of standard packages) T275873

Change 677492 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/debs/quickstack@master] Rebuild for bullseye T275873

https://gerrit.wikimedia.org/r/677492

Change 677492 merged by Muehlenhoff:

[operations/debs/quickstack@master] Rebuild for bullseye T275873

https://gerrit.wikimedia.org/r/677492

Change 677807 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only install git-fat for distros up to Buster

https://gerrit.wikimedia.org/r/677807

Change 677807 merged by Muehlenhoff:

[operations/puppet@production] Only install git-fat for distros up to Buster

https://gerrit.wikimedia.org/r/677807

Change 677931 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch to iptables legacy alternative provider on bullseye

https://gerrit.wikimedia.org/r/677931

Change 677931 merged by Muehlenhoff:

[operations/puppet@production] Switch to iptables legacy alternative provider on bullseye

https://gerrit.wikimedia.org/r/677931

Change 678838 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Extend d-i config for bullseye

https://gerrit.wikimedia.org/r/678838

Change 678838 merged by Muehlenhoff:

[operations/puppet@production] Extend d-i config for bullseye

https://gerrit.wikimedia.org/r/678838

Mentioned in SAL (#wikimedia-operations) [2021-04-13T14:08:57Z] <moritzm> updated bullseye d-i image to 2021-04-12 daily build T275873

Change 679768 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Python 2 packages on Bullseye

https://gerrit.wikimedia.org/r/679768

Change 679768 merged by Muehlenhoff:

[operations/puppet@production] Remove Python 2 packages on Bullseye

https://gerrit.wikimedia.org/r/679768

Mentioned in SAL (#wikimedia-operations) [2021-04-16T11:02:31Z] <moritzm> imported ferm 2.5.1-1+wmf1 to bullseye-wikimedia/main T275873

Change 680978 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Adapt name of security suite for bullseye and later

https://gerrit.wikimedia.org/r/680978

Change 680978 merged by Muehlenhoff:

[operations/puppet@production] Adapt name of security suite for bullseye and later

https://gerrit.wikimedia.org/r/680978

Mentioned in SAL (#wikimedia-operations) [2021-04-19T11:33:51Z] <moritzm> imported debdeploy 0.0.99.13-1+deb11u1 to bullseye-wikimedia T275873

Change 681033 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only disable timers for ipmitoo/smartmon timers up to Buster

https://gerrit.wikimedia.org/r/681033

Change 681033 merged by Muehlenhoff:

[operations/puppet@production] Only disable timers for ipmitoo/smartmon timers up to Buster

https://gerrit.wikimedia.org/r/681033

Mentioned in SAL (#wikimedia-operations) [2021-04-23T08:12:59Z] <moritzm> upgrading d-i image for bullseye to RC1 release T275873

Mentioned in SAL (#wikimedia-operations) [2021-04-28T15:00:06Z] <moritzm> imported python-poolcounter 0.0.2-1+deb11u1 to apt.wikimedia.org T275873

Mentioned in SAL (#wikimedia-operations) [2021-04-29T10:59:57Z] <moritzm> updating apt on buster (SUA 198), which eases bullseye upgrades T275873

Mentioned in SAL (#wikimedia-operations) [2021-05-06T08:23:19Z] <moritzm> imported wikimedia-lvs-realserver to apt.wikimedia.org/bullseye T275873

Mentioned in SAL (#wikimedia-operations) [2021-05-26T14:31:30Z] <moritzm> updated bullseye d-i image to 2021-05-26 daily image T275873

Mentioned in SAL (#wikimedia-operations) [2021-06-10T17:11:19Z] <moritzm> updating bullseye installer image to latest daily image (kernel ABI changed again) T275873

Mentioned in SAL (#wikimedia-operations) [2021-06-29T07:05:02Z] <moritzm> upgrading bullseye early installs to the latest state of testing T275873

Mentioned in SAL (#wikimedia-operations) [2021-07-15T07:48:09Z] <moritzm> updated bullseye d-i image for latest daily build T275873

Leaving this here for tracking, I'm seeing a permission error from node-exporter on a Bullseye host (thanos-fe2001). It looks like the values in the file are bogus anyways so we can probably disable the node-exporter module if possible

Jul 15 12:57:39 thanos-fe2001 prometheus-node-exporter[593]: level=error ts=2021-07-15T12:57:39.661Z caller=collector.go:161 msg="collector failed" name=rapl duration_seconds=0.00107328 err="open /sys/class/powerc
ap/intel-rapl:0/energy_uj: permission denied"                                                                                                                                                                        Jul 15 12:57:39 thanos-fe2001 prometheus-node-exporter[593]: level=error ts=2021-07-15T12:57:39.675Z caller=collector.go:161 msg="collector failed" name=rapl duration_seconds=0.010393724 err="open /sys/class/power
cap/intel-rapl:0/energy_uj: permission denied"

In T275873#7215007, @fgiunchedi wrote:

Leaving this here for tracking, I'm seeing a permission error from node-exporter on a Bullseye host (thanos-fe2001). It looks like the values in the file are bogus anyways so we can probably disable the node-exporter module if possible

This is tracked by upstream at https://github.com/prometheus/node_exporter/issues/1892 and their solution is to also mask the RAPL collector (https://github.com/wagdav/homelab/commit/26fc86c6a79a5f1a634c7b313f86c0b6109539c0), so I think we can simply do that fleet-wide (including older distros)? I don't think we're currently using that data in any way.

In T275873#7215066, @MoritzMuehlenhoff wrote:

In T275873#7215007, @fgiunchedi wrote:

Leaving this here for tracking, I'm seeing a permission error from node-exporter on a Bullseye host (thanos-fe2001). It looks like the values in the file are bogus anyways so we can probably disable the node-exporter module if possible

This is tracked by upstream at https://github.com/prometheus/node_exporter/issues/1892 and their solution is to also mask the RAPL collector (https://github.com/wagdav/homelab/commit/26fc86c6a79a5f1a634c7b313f86c0b6109539c0), so I think we can simply do that fleet-wide (including older distros)? I don't think we're currently using that data in any way.

+1 on disabling the collector (on >= bullseye, since it's been introduced in node-exporter 1.0.0)

In T275873#7215311, @fgiunchedi wrote:

This is tracked by upstream at https://github.com/prometheus/node_exporter/issues/1892 and their solution is to also mask the RAPL collector (https://github.com/wagdav/homelab/commit/26fc86c6a79a5f1a634c7b313f86c0b6109539c0), so I think we can simply do that fleet-wide (including older distros)? I don't think we're currently using that data in any way.

+1 on disabling the collector (on >= bullseye, since it's been introduced in node-exporter 1.0.0)

Given that this also applies to Bullseye and there's still time to land the fix in testing, I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991160.

Something else I noticed in node-exporter, node_cpu_frequency_hertz is gone thus the cpu frequency panel in host overview stops working (tracked as T286768)

Mentioned in SAL (#wikimedia-operations) [2021-07-19T09:52:16Z] <moritzm> imported megacli for bullseye-wikimedia T282272 T275873

In T275873#7216558, @MoritzMuehlenhoff wrote:

+1 on disabling the collector (on >= bullseye, since it's been introduced in node-exporter 1.0.0)

Given that this also applies to Bullseye and there's still time to land the fix in testing, I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991160.

This got fixed in https://packages.qa.debian.org/p/prometheus-node-exporter/news/20210727T223327Z.html and should still make it into the initial Bullseye release.

Mentioned in SAL (#wikimedia-operations) [2021-08-03T11:36:45Z] <moritzm> updated bullseye d-i images to rc3 T275873

In T275873#7216558, @MoritzMuehlenhoff wrote:

In T275873#7215311, @fgiunchedi wrote:

This is tracked by upstream at https://github.com/prometheus/node_exporter/issues/1892 and their solution is to also mask the RAPL collector (https://github.com/wagdav/homelab/commit/26fc86c6a79a5f1a634c7b313f86c0b6109539c0), so I think we can simply do that fleet-wide (including older distros)? I don't think we're currently using that data in any way.

+1 on disabling the collector (on >= bullseye, since it's been introduced in node-exporter 1.0.0)

Given that this also applies to Bullseye and there's still time to land the fix in testing, I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991160.

This made is to testing before the initial release and is now upgraded on the early Bullseye installs.

Change 712287 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mtail: On bullseye use the distro default (3.0.0-rc43)

https://gerrit.wikimedia.org/r/712287

Change 712287 merged by Muehlenhoff:

[operations/puppet@production] mtail: On bullseye use the distro default (3.0.0-rc43)

https://gerrit.wikimedia.org/r/712287