Nova Resource:Codesearch/SAL

2024-04-03

• 02:46 Krinkle: Reboot host https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=codesearch&var-instance=All&from=now-3h&to=now

2024-03-15

• 03:53 Krinkle: Deploying fix for T321347

2024-03-01

• 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 11:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 11:38 dcaro: reboot codesearch8 as it's locked out of resources
• 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-07-12

• 17:48 mutante: added new member aokoth

2023-05-24

• 15:11 legoktm: temporarily taking down to free up disk space (T337263)

2023-04-21

• 14:57 legoktm: rebooting codesearch8 to see if that fixes networking issues (T335197)

2023-03-13

• 01:12 Krinkle: Change codesearch.wmcloud.org proxy from port 3002 (Hound UI) to 3003 (new Codesearch frontend), ref T263354

2023-03-12

• 22:50 Krinkle: Create new https://codesearch-backend.wmcloud.org/ webproxy, points to the same location as codesearch-old.

2022-12-02

• 19:41 Krinkle: codesearch8 instance unresponsive to http and ssh, rebooted from horizon

2022-11-01

• 04:35 legoktm: temporarily stopping to fix full disk

2022-06-17

• 20:04 legoktm: added Krinkle as project admin, Gerrit +2

2022-05-20

• 05:16 legoktm: restarting to pick up label changes

2022-04-09

• 03:55 legoktm: restarting everything to add new "wmcs" instance

2022-03-15

• 02:09 legoktm: forcing manual run of codesearch-write-config for T303641

2021-11-03

• 16:06 legoktm: manually triggered codesearch-write-config job to pick up fix for T294915 immediately

2021-10-24

• 22:39 legoktm: one more restart of everything, switched hound container over to bullseye for newer git
• 22:20 legoktm: restarted everything to add new prometheus metrics endpoint: https://codesearch.wmcloud.org/_metrics

2021-10-23

• 16:07 majavah: hard reboot codesearch8 after OOM crash

2021-05-21

• 16:08 legoktm: restarting everything for T283317

2021-05-20

• 23:45 legoktm: restarting to switch docker images

2021-04-14

• 01:38 legoktm: deleting codesearch6 instance (T279763)

2021-04-13

• 20:53 legoktm: switched codesearch.wmcloud.org and codesearch-beta.wmcloud.org to point to codesearch8 (T279763)

2021-04-09

• 08:57 legoktm: deleting codesearch7, creating codesearch8 as the correct size (T279763)
• 08:52 legoktm: creating codesearch7 instance (T279763)
• 08:48 legoktm: moved role::labs::lvm::srv from project puppet to codesearch06's puppet with the goal of moving to cinder

2021-03-05

• 23:45 legoktm: restarting for new shouthow profile (T253597)

2021-01-26

• 02:01 legoktm: restarting for new puppet profile (T272947)

2021-01-09

• 00:52 legoktm: bunch of restarts, repos with non-master default branches should be properly supported

2020-11-25

• 19:03 legoktm: fixed permissions on /srv/codesearch/.git/objects, forcing puppet run which is triggering restarts

2020-11-18

• 22:45 Amir1: restart codesearch to pick up new config (T268187)

2020-11-09

• 11:05 Amir1: restart codesearch

2020-10-30

• 22:43 GarbageMonster: restarting codesearch to pick up new config (T266909)

2020-10-05

• 04:57 legoktm: restarting to deploy T264557

2020-09-22

• 20:33 legoktm: (re)starting all hound instances now that gerrit-replica is back up

2020-08-28

• 19:17 Amir1: restarting codesearch to pick up the new config (T261517)

2020-08-12

• 08:57 legoktm: restarting proxy to implement etag changes

2020-08-11

• 23:48 legoktm: restarting hound-deployed for the same reason
• 21:30 Amir1: restarting hound-search to pick up the new config for services (T260124)

2020-07-26

• 05:56 legoktm: restarting everything (again) for https://gerrit.wikimedia.org/r/616317
• 04:50 legoktm: restarting everything to deploy https://gerrit.wikimedia.org/r/616316

2020-07-24

• 13:39 Amir1: restarting codesearch to pick up the new config (T231572)

2020-07-22

• 14:03 Amir1: restart codesearch to pick up new config (adding mediawiki/vagrant)

2020-07-13

• 21:04 Amir1: create codesearch.wmcloud.org dns proxy (T256206)

2020-06-28

• 09:31 legoktm: root@codesearch6:~# chown codesearch:codesearch -R /srv/codesearch

2020-06-11

• 11:04 legoktm: restarting everything after gerrit-replica 502s fixed T255094 T255125

2020-03-06

• 18:03 bd808: Forced Puppet run on codesearch6 to apply Puppet fix for T246017

2020-02-25

• 21:52 Amir1: restart hound proxy
• 21:50 Amir1: sudo service hound-libraries restart (T246120 T246121)

2020-02-24

• 17:51 bd808: `systemctl restart docker` on codesearch6 (T246017)
• 17:37 bd808: `systemctl restart hound-search.service` on codesearch6 (T246017)

2020-02-22

• 16:54 Amir1: hard reboot of codesearch6 (T245920)

2020-02-19

• 14:48 Amir1: restarting hound_proxy to pick up the new changes

2020-02-13

• 19:02 Reedy: is down "Hound is still starting up, please wait a few minutes for the initial indexing to complete."

2020-01-25

• 04:14 legoktm: restarted for new libraries search profile

2020-01-23

• 02:00 legoktm: shutting down and deleting temporary codesearch5 instance
• 01:59 legoktm: shutting down old codesearch4 instance (but not deleting yet)
• 01:57 legoktm: deleting temporary codesearch6.wmflabs.org domain
• 01:56 legoktm: retargetting codesearch.wmflabs.org to codesearch6 (T242319)

2020-01-19

• 06:08 legoktm: temporarily taking everything down to free up disk space (T243121)

2020-01-17

• 07:37 legoktm: rolling restart of hound instances to test new bootstrapping overload protection

2020-01-16

• 20:17 legoktm: legoktm@codesearch5:~$ sudo crontab -u codesearch -r

2020-01-11

• 03:57 legoktm: added mutante as project admin to help with puppetization efforts

2019-12-22

• 12:29 Amir1: ladsgroup@codesearch4:~$ sudo service hound-armchairgm restart (T241319)

2019-12-15

• 01:30 legoktm: rolling restart of all hound instances because of gerrit-replica outage (T240776, T240763)
• 00:40 Amir1: sudo service hound-search restart (T240776)

2019-10-16

• 10:17 Amir1: ladsgroup@codesearch4:~$ sudo service hound-services restart (T235596)

2019-09-30

• 16:14 Amir1: soft-rebooting codesearch4
• 15:49 Amir1: ladsgroup@codesearch4:~$ sudo MODE=restart /srv/codesearch/manage.sh (T234211)

2019-09-29

• 21:49 Amir1: git pull for puppet (T219248)

2019-09-18

• 00:06 Amir1: another restart

2019-09-17

• 23:48 Amir1: restart to pick up the shiny new "services" section

2019-09-16

• 15:56 Amir1: ladsgroup@codesearch4:~$ sudo MODE=restart /srv/codesearch/manage.sh (It seems it's down, forcing a restart)

2019-09-03

• 22:39 Amir1: "sudo MODE=restart /srv/codesearch/manage.sh"

2019-08-07

• 20:10 legoktm: deleting now unneeded ggmirror-02 instance
• 20:04 legoktm: switched over to using gerrit-replica

2019-07-17

• 00:53 legoktm: manual rolling restart of everything to start using new gerrit mirror

2019-07-16

• 23:04 legoktm: deleted medium sized ggmirror-01 instance, it ran out of disk space. recreated as ggmirror-02 with size large

2019-07-11

• 15:01 legoktm: legoktm@codesearch4:/srv/puppet$ git pull

2019-06-06

• 18:15 legoktm: force updated config and doing a rolling restart (T223771)

2019-05-24

• 21:54 legoktm: maintenance to reduce disk space

2019-03-28

• 08:13 legoktm: legoktm@codesearch4:/srv/puppet$ git pull # T219248

2019-01-03

• 00:35 legoktm: restarting hound instances for T211999

2018-12-03

• 06:17 legoktm: deleted codesearch3 instance (T210372)

2018-11-26

• 18:33 duh: codesearch4: deleted all of /srv/hound/hound-core/data/ and restarted the hound-core service for a fresh clone (T210409)
• 06:36 legoktm: retarget codesearch.wmflabs.org to codesearch4 instance
• 06:14 legoktm: created and setting up codesearch4 image based on Debian Stretch
• 06:03 legoktm: deleted codesearch-sourcegraph1 instance

2018-11-16

• 13:51 andrewbogott: migrating project to eqiad1

2018-11-05

• 07:46 legoktm: Implemented automatic restarts of hound to regularly pick up new config changes

2018-09-27

• 23:35 legoktm: restarting hound for config change (https://gerrit.wikimedia.org/r/463390)

2018-09-21

• 07:36 legoktm: restarted to pick up new config / repository lists

2018-09-05

• 05:25 legoktm: restarted all hound backends after deleting all of their data
• 05:16 legoktm: stopping hound backends to clean up disk space
• 04:52 legoktm: restarted hound for gerrit:458105

2018-08-17

• 18:17 legoktm: rolling restart to pick up new config

2018-07-09

• 13:07 legoktm: restarting everything for config update

2018-04-13

• 17:41 legoktm: booted up codesearch3, it was down for some reason

2018-03-02

• 02:12 legoktm: deleted codesearch2 instance
• 02:09 legoktm: pointed codesearch.wmflabs.org at codesearch3:3002
• 02:05 legoktm: ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t bd808) to quintuple the number of inodes
• 01:52 legoktm: applied role::labs::lvm::srv role to codesearch3
• 01:52 legoktm: created codesearch3 instance (jessie, medium flavor)

2018-02-28

• 04:44 legoktm: restarted hound-search service (T188477), also I restarted hound-extensions earlier (T188450)

2018-02-24

• 08:47 legoktm: added MaxSem as a project admin

2018-02-23

• 22:23 legoktm: added d3r1ck's correct account as project admin

2018-02-14

• 20:19 legoktm: added Addshore, D3r1ck01, Ladsgroup, and Prtksxna to Gerrit labs-codesearch project
• 20:12 legoktm: added Addshore, D3r1ck01, Ladsgroup, and Prtksxna as project admins of codesearch

2018-01-18

• 13:32 legoktm: rebooting codesearch2 to see how well it recovers
• 11:36 legoktm: legoktm@codesearch2:~$ sudo systemctl enable hound_proxy

2018-01-16

• 22:49 chasemp: restarted everything I could find in the stack :)

2017-12-21

• 23:28 legoktm: restarting hound instances to pick up config change