Page MenuHomePhabricator

Upgrade Netbox to 3.2
Closed, ResolvedPublic

Description

We're currently running v2.10.4 in prod and -next.

WARNING: Upgrading an existing NetBox deployment to version 3.0 must be done from version 2.11.0 or later.

Next steps:

  1. Upgrade -next to 3.1, . (Through a step on 2.11.12)
  2. Prepare changes to fix automation, reports, scripts, etc
  3. Backup prod data (CSV dumps, etc)
  4. Upgrade prod to 3.2 (with a step on 2.11)

Breaking changes:

  • All objects now use numeric IDs in their UI view URLs instead of slugs. You may need to update external references to NetBox objects. (Note that this does not affect the REST API.)
  • The UI now uses numeric IDs when filtering object lists. You may need to update external links to filtered object lists. (Note that the slug- and name-based filters will continue to work, however the filter selection fields within the UI will not be automatically populated.)
  • The RackGroup model has been renamed to Location (see #4971). Its REST API endpoint has changed from /api/dcim/rack-groups/ to /api/dcim/locations/.
  • The foreign key field group on dcim.Rack has been renamed to location.
  • The foreign key field site on ipam.VLANGroup has been replaced with the scope generic foreign key (see #5284).
  • Custom script ObjectVars no longer support the queryset parameter: Use model instead (see #5995).
  • The default CSV export format for all objects now includes all available data from the object list. Additionally, the CSV headers now use human-friendly titles rather than raw field names. If backward compatibility with the old format is desired, export templates can be written to reproduce it.
  • The display_field keyword argument has been removed from custom script ObjectVar and MultiObjectVar fields. * These widgets will use the display value provided by the REST API.
  • The deprecated display_name field has been removed from all REST API serializers. (API clients should reference the display field instead.)
  • The redundant REST API endpoints for console, power, and interface connections have been removed. The same data can be retrieved by querying the respective model endpoints with the ?connected=True filter applied.
  • The tenant and tenant_id filters for the Cable model now filter on the tenant assigned directly to each cable, rather than on the parent object of either termination.
  • The cable_peer and cable_peer_type attributes of cable termination models have been renamed to link_peer and link_peer_type, respectively, to accommodate wireless links between interfaces.
  • Automatic redirection of legacy slug-based URL paths has been removed. URL-based slugs were changed to use numeric IDs in v2.11.0.
  • The asn field has been removed from the site model. Please replicate any site ASN assignments to the ASN model introduced in NetBox v3.1 prior to upgrading.
  • The asn query filter for sites now matches against the AS number of assigned ASN objects.
  • The contact_name, contact_phone, and contact_email fields have been removed from the site model. Please replicate any data remaining in these fields to the contact model introduced in NetBox v3.1 prior to upgrading.
  • The created field of all change-logged models now conveys a full datetime object, rather than only a date. (Previous date-only values will receive a timestamp of 00:00.) While this change is largely unconcerning, strictly-typed API consumers may need to be updated.
  • A pre_run() method has been added to the base Report class. Although unlikely to affect most installations, you may need to alter any reports which already use this name for a method.
  • Webhook URLs now support Jinja2 templating. Although this is unlikely to introduce any issues, it's possible that an unusual URL might trigger a Jinja2 rendering error, in which case the URL would need to be properly escaped.

New features that could be relevant to us:

  • Journaling Support (#151) - T310583
  • Parent Interface Assignments (#1519) - T296832
  • Pre- and Post-Change Snapshots in Webhooks (#3451)
  • New Site Group Model (#5892)
  • Improved Change Logging (#5913) - T310589
  • Provider Network Modeling (#5986) - T310591
  • Updated User Interface (#5893)
  • GraphQL API (#2007) - T310577
  • IP Ranges (#834)
  • Custom Model Validation (#5963) - T310590
  • New Views for Models Previously Under the Admin UI (#6466)
  • New Housekeeping Command (#6590)
  • Custom Queue Support for Plugins (#6651)
  • Modules & Module Types (#7844) - T305126
  • Custom Object Fields (#7006) - T305126
  • Custom Status Choices (#8054) - T310594
  • Inventory Item Templates (#8118)

Enhancements that could be relevant to us:

  • #5914 - Add edit/delete buttons for IP addresses on interface view
  • #5942 - Add button to add a new IP address on interface view
  • #5953 - Support Markdown rendering for custom script descriptions
  • #6040 - Add UI search fields for asset tag for devices and racks
  • #6157 - Support Markdown rendering for report logs
  • #4833 - Allow assigning config contexts by device type
  • #5344 - Add support for custom fields in tables
  • #5401 - Extend custom field support to device component models
  • #5451 - Add support for multiple-selection custom fields
  • #6149 - Support image attachments for locations
  • #6087 - Improved prefix hierarchy rendering
  • #6527 - Support Markdown for report descriptions
  • #6666 - Show management-only status under interface detail view
  • #6560 - Enable CSV import via uploaded file
  • #6929 - Introduce LOGIN_PERSISTENCE configuration parameter to persist user sessions
  • #2434 - Add option to assign IP address upon creating a new interface
  • #5775 - Enable synchronization of groups for remote authentication backend
  • #6917 - Make IP assigned checkmark in IP table link to interface
  • #7323 - Add serial filter field for racks & devices
  • #7551 - Add UI field to filter interfaces by kind
  • #7561 - Add a utilization column to the IP ranges table
  • #7767 - Add visual aids to interfaces table for type, enabled status
  • #7853 - Add speed and duplex fields to device interface model

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+6 -0
operations/puppetproduction+0 -28
operations/dnsmaster+2 -2
operations/puppetproduction+1 -1
operations/dnsmaster+2 -2
operations/puppetproduction+1 -1
operations/dnsmaster+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+6 -5
operations/dnsmaster+2 -3
operations/puppetproduction+1 -82
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+2 -2
operations/software/homer/deploymaster+20 -27
operations/software/netbox-extrasmaster+1 -1
operations/software/netbox-extrasmaster+0 -170
operations/puppetproduction+0 -47
operations/puppetproduction+13 -0
operations/puppetproduction+0 -22
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+3 -3
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+5 -0
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/cookbooksmaster+3 -4
operations/software/spicerackmaster+4 -3
operations/puppetproduction+9 -0
operations/dnsmaster+3 -0
operations/dnsmaster+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+39 -39
operations/dnsmaster+2 -2
operations/puppetproduction+7 -1
operations/dnsmaster+3 -2
operations/puppetproduction+2 -0
operations/puppetproduction+35 -25
operations/puppetproduction+5 -1
operations/puppetproduction+31 -18
operations/puppetproduction+19 -26
operations/dnsmaster+2 -0
operations/puppetproduction+1 -1
operations/dnsmaster+2 -0
operations/puppetproduction+9 -0
operations/puppetproduction+32 -0
operations/puppetproduction+9 -0
operations/puppetproduction+4 -1
operations/puppetproduction+1 -2
operations/puppetproduction+6 -0
operations/puppetproduction+30 -6
operations/puppetproduction+5 -0
operations/cookbooksmaster+1 -1
operations/puppetproduction+0 -1
operations/puppetproduction+13 -1
operations/puppetproduction+6 -0
operations/puppetproduction+2 -0
operations/dnsmaster+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+69 -67
operations/puppetproduction+0 -15
operations/dnsmaster+1 -1
operations/puppetproduction+9 -0
operations/software/netbox-deploy3-2-2+14 -10
operations/software/netbox-deploy3-1+1 -4
operations/software/netbox-deploy3-1+1 -1
operations/software/netbox-deploy3-1+54 -26
operations/software/netbox-deploy2-11-12+28 -25
operations/puppetproduction+4 -0
operations/puppetproduction+10 -4
operations/software/netbox-deploy2-10-4-bullseye+111 -0
operations/software/netbox-deploy2-10-4-bullseye+2 -1
operations/puppetproduction+1 -0
operations/puppetproduction+2 -0
operations/software/netbox-deploy2-10-4-bullseye+108 -1
operations/puppetproduction+15 -6
operations/puppetproduction+15 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedjbond
ResolvedVolans
DeclinedNone
OpenNone
Resolvedayounsi
Resolvedayounsi
DeclinedNone
Resolvedayounsi
Resolvedayounsi
StalledNone
Resolvedcmooney
Resolvedayounsi
OpenNone
OpenNone
OpenNone
Resolvedayounsi
Resolvedayounsi
Resolvedayounsi
Resolvedayounsi
OpenNone
OpenNone
Resolvedayounsi
StalledNone
Resolvedayounsi
StalledNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 803466 merged by Volans:

[operations/cookbooks@master] sre.dns.netbox: update to use new netbox servers

https://gerrit.wikimedia.org/r/803466

Change 803468 merged by Jbond:

[operations/puppet@production] netbox: move api url to discovery domain name

https://gerrit.wikimedia.org/r/803468

Change 803485 abandoned by Ayounsi:

[operations/puppet@production] Enable Icinga notifications on new netbox frontends

Reason:

change done in a different CR

https://gerrit.wikimedia.org/r/803485

Change 803579 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:netbox: add REQUESTS_CA_BUNDLE variable to systemd timers

https://gerrit.wikimedia.org/r/803579

Change 803579 merged by Jbond:

[operations/puppet@production] P:netbox: add REQUESTS_CA_BUNDLE variable to systemd timers

https://gerrit.wikimedia.org/r/803579

Change 803582 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] netbox: also add CA bundle environment to netbox_dump_run job

https://gerrit.wikimedia.org/r/803582

Change 803582 merged by Jbond:

[operations/puppet@production] netbox: also add CA bundle environment to netbox_dump_run job

https://gerrit.wikimedia.org/r/803582

Change 803858 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Homer: add REQUESTS_CA_BUNDLE for new Netbox endpoint

https://gerrit.wikimedia.org/r/803858

Change 803858 merged by Ayounsi:

[operations/puppet@production] Homer: add REQUESTS_CA_BUNDLE for new Netbox endpoint

https://gerrit.wikimedia.org/r/803858

Change 804345 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] CDN: disable caching for netbox-exports

https://gerrit.wikimedia.org/r/804345

Change 804345 merged by Ayounsi:

[operations/puppet@production] CDN: disable caching for netbox-exports

https://gerrit.wikimedia.org/r/804345

Change 789635 abandoned by Jbond:

[operations/puppet@production] O:netbox::standalone: use netbox-next/deploy scap repo

Reason:

no longer needed

https://gerrit.wikimedia.org/r/789635

Change 805125 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] hieradata: netbox1001 to specify netbox1002 as the active server.

https://gerrit.wikimedia.org/r/805125

Change 805125 merged by Jbond:

[operations/puppet@production] hieradata: netbox1001 to specify netbox1002 as the active server.

https://gerrit.wikimedia.org/r/805125

ayounsi added a parent task: Restricted Task.Jun 14 2022, 8:10 AM

Change 805468 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Netbox: only run CSV dumps on active server

https://gerrit.wikimedia.org/r/805468

Change 805468 merged by Ayounsi:

[operations/puppet@production] Netbox: only run CSV dumps on active server

https://gerrit.wikimedia.org/r/805468

Change 805740 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Netbox: remove Icinga checks for netbox.wikimedia.org

https://gerrit.wikimedia.org/r/805740

Change 805740 merged by Ayounsi:

[operations/puppet@production] Netbox: remove Icinga checks for netbox.wikimedia.org

https://gerrit.wikimedia.org/r/805740

Mentioned in SAL (#wikimedia-operations) [2022-06-15T12:42:31Z] <volans@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on netbox:443 with reason: Netbox upgrade to 3.2 T296452

Mentioned in SAL (#wikimedia-operations) [2022-06-15T12:42:37Z] <volans@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on netbox:443 with reason: Netbox upgrade to 3.2 T296452

Change 790400 merged by Ayounsi:

[operations/puppet@production] Netbox: Add 2.11 configuration knobs

https://gerrit.wikimedia.org/r/790400

Icinga downtime and Alertmanager silence (ID=b899820b-e817-4d4e-af91-553e8854cf5d) set by volans@cumin1001 for 4:00:00 on 1 host(s) and their services with reason: Netbox upgrade to 3.2

netbox1002.eqiad.wmnet

Icinga downtime and Alertmanager silence (ID=1e0ccaf6-24e0-41c6-8aa6-1269763ad443) set by volans@cumin1001 for 4:00:00 on 1 host(s) and their services with reason: Netbox upgrade to 3.2

netbox2002.codfw.wmnet

Change 790681 merged by Ayounsi:

[operations/puppet@production] Netbox: update config file for 3.1

https://gerrit.wikimedia.org/r/790681

Change 805813 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/netbox-extras@master] Removed temporary migration script to 3.2

https://gerrit.wikimedia.org/r/805813

Change 805813 merged by jenkins-bot:

[operations/software/netbox-extras@master] Removed temporary migration script to 3.2

https://gerrit.wikimedia.org/r/805813

Change 790991 merged by jenkins-bot:

[operations/software/netbox-extras@master] ganeti-netbox-sync: Add netbox 3.2 support

https://gerrit.wikimedia.org/r/790991

Change 790975 merged by Ayounsi:

[operations/software/homer/deploy@master] wmf-netbox: Netbox 3.2 compatibility

https://gerrit.wikimedia.org/r/790975

Change 805830 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/netbox-extras@master] CSV dumps: update names for Netbox 3.2

https://gerrit.wikimedia.org/r/805830

Change 805830 merged by Volans:

[operations/software/netbox-extras@master] CSV dumps: update names for Netbox 3.2

https://gerrit.wikimedia.org/r/805830

Change 805869 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/netbox-extras@master] Revert "ganeti-netbox-sync: Add netbox 3.2 support"

https://gerrit.wikimedia.org/r/805869

Change 805881 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/software/netbox-extras@master] CSV dumps: it's actually dcim.cables

https://gerrit.wikimedia.org/r/805881

Change 805881 merged by jenkins-bot:

[operations/software/netbox-extras@master] CSV dumps: it's actually dcim.cables

https://gerrit.wikimedia.org/r/805881

Change 805869 merged by jenkins-bot:

[operations/software/netbox-extras@master] Revert "ganeti-netbox-sync: Add netbox 3.2 support"

https://gerrit.wikimedia.org/r/805869

Change 808197 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] netbox: update netbox service definition so it pages

https://gerrit.wikimedia.org/r/808197

Change 808198 had a related patch set uploaded (by Jbond; author: jbond):

[operations/dns@master] netbox: update netbox so that its active/active

https://gerrit.wikimedia.org/r/808198

Change 808199 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] netbox: update netbox service to active/active

https://gerrit.wikimedia.org/r/808199

cookbooks.sre.hosts.decommission executed by volans@cumin2002 for hosts: sretest2001.codfw.wmnet

  • sretest2001.codfw.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox

Change 815219 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove Netbox 2.10 hosts from Puppet before decom

https://gerrit.wikimedia.org/r/815219

Change 815219 merged by Ayounsi:

[operations/puppet@production] Remove Netbox 2.10 hosts from Puppet before decom

https://gerrit.wikimedia.org/r/815219

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netbox2001.wikimedia.org

  • netbox2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netbox1001.wikimedia.org

  • netbox1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netboxdb2001.codfw.wmnet

  • netboxdb2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netboxdb1001.eqiad.wmnet

  • netboxdb1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netboxdb1001.eqiad.wmnet

  • netboxdb1001.eqiad.wmnet (FAIL)
    • Host steps raised exception:

ERROR: some step on some host failed, check the bolded items above

Change 803460 merged by Jbond:

[operations/dns@master] netbox: increase TTL to 1D

https://gerrit.wikimedia.org/r/803460

Change 890421 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] netbox: force database connections to use TLS.

https://gerrit.wikimedia.org/r/890421

Change 890421 merged by Jbond:

[operations/puppet@production] netbox: force database connections to use TLS.

https://gerrit.wikimedia.org/r/890421

Change 808199 merged by Jbond:

[operations/puppet@production] netbox: update netbox service to active/active

https://gerrit.wikimedia.org/r/808199

Change 808198 merged by Jbond:

[operations/dns@master] netbox: update netbox so that its active/active

https://gerrit.wikimedia.org/r/808198

Change 890380 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/dns@master] netbox: update netbox so that its active/active

https://gerrit.wikimedia.org/r/890380

Change 890381 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/puppet@production] netbox: update netbox service to active/active

https://gerrit.wikimedia.org/r/890381

Change 890381 merged by Jbond:

[operations/puppet@production] netbox: update netbox service to active/active

https://gerrit.wikimedia.org/r/890381

Change 890380 merged by Jbond:

[operations/dns@master] netbox: update netbox so that its active/active

https://gerrit.wikimedia.org/r/890380

Change 890384 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/dns@master] netbox: update netbox so that its active/active

https://gerrit.wikimedia.org/r/890384

Change 890385 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/puppet@production] netbox: update netbox service to active/active

https://gerrit.wikimedia.org/r/890385

during the recent DC switch over netbox got moved to codfw and it was super slow. this means that in the current set up:

  • active/active may not be the best idea
  • we need to update the dc-switch cookbook to also failover the postressdb

Change 890385 abandoned by Jbond:

[operations/puppet@production] netbox: update netbox service to active/active

Reason:

more work required on netbox side, see tasks

https://gerrit.wikimedia.org/r/890385

Change 890384 abandoned by Jbond:

[operations/dns@master] netbox: update netbox so that its active/active

Reason:

more work required on netbox side see tasks

https://gerrit.wikimedia.org/r/890384

Change 803489 abandoned by Jbond:

[operations/puppet@production] hieradata: decommission netbox servers

Reason:

this has since been done

https://gerrit.wikimedia.org/r/803489

Change 803508 abandoned by Jbond:

[operations/puppet@production] P:netbox: Add hosts entry for service address

Reason:

not sure this is needed

https://gerrit.wikimedia.org/r/803508

Change 797320 merged by Jbond:

[operations/puppet@production] P:trafficserver::backend: netbox-next switch to netbox-next.discovery.wmnet

https://gerrit.wikimedia.org/r/797320