For ulsfo user-facing traffic, the lowest point of the day is approximately a symmetrical dip centered on 20:00 UTC (Noon Pacific). So if they need a 4-hour window, ask for 10AM -> 2PM their time, etc. Given weekend risks and all of that in general, I'd prefer we do this mid-week (Tues, Weds, or Thurs).

And to be clear, what I expect we'll do on our end is depool ulsfo from users in DNS in our config-geo ahead of the window (let's say 2 hours ahead? the TTLs are 10 minutes, but there's always stragglers), and bring them back after they're done. Assuming we don't actually lose the machines or network and purges keep flowing throughout, there's no need to wipe any caches before repooling, either.

Rob Halsell replied via email on Tue, 16 Feb 2016 09:53:06 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at that site,
which is 20:00 GMT / 12:00 Pacific. So for a maintenance window, can we
set it for the two hours before and afterwards, so 10AM to 2PM Pacific;
additionally we would like to keep this work to Tuesday, Wednesday, or
Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thursdays
between 10:00-14:00 Pacific work best for you. All but our management
network have redundant power supplies wired to both towers, so actual
downtime should be minimal. We plan to migrate all traffic away from that
site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

It has come to our attention that your network management card for your
(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor your PDU
because of this issue. We would like to work with you to fix the issue
sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associated with this
work so we would like to schedule a maintenance window to your convenience
so
that we can go ahead and swap out the bad PDU. If we determine that
downtime
is needed, we can schedule this maintenance window during off hours so the
impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If you would
like to
discuss this over the phone please let me know and I can accommodate this
for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

UnitedLayer Support Ticket System replied via email on Tue, 16 Feb 2016 10:59:45 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Dear Rob,=20

Preferably the sooner the better. Tomorrow would be great for us, please let
us know if you can accommodate that maintenance window. We'd like to make t=
his
as easy on you as possible so any of those days will work for us.=20

Thanks,=20
Phu=20=20

Phu/Support,

20

20

We prefer the PDU swap happen during our traffic low-point at that site,
which is 20:00 GMT / 12:00 Pacific. So for a maintenance window, can we
set it for the two hours before and afterwards, so 10AM to 2PM Pacific;
additionally we would like to keep this work to Tuesday, Wednesday, or
Thursday.

20

Please let us know which of the upcoming Tuesday/Wednesday/Thursdays
between 10:00-14:00 Pacific work best for you. All but our management
network have redundant power supplies wired to both towers, so actual
downtime should be minimal. We plan to migrate all traffic away from that
site during the maintenance window as a preventative measure.

20

Please advise,

20

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

20

> It has come to our attention that your network management card for your
> (b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
> malfunctioning. We are not able to remotely access and monitor your PDU
> because of this issue. We would like to work with you to fix the issue
> sooner
> than later so we do not run into any problems.
>
> There may be some downtime with a few of your servers associated with t=

his

work so we would like to schedule a maintenance window to your convenie=

nce

so
that we can go ahead and swap out the bad PDU. If we determine that
downtime
is needed, we can schedule this maintenance window during off hours so =

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If you would
like to
discuss this over the phone please let me know and I can accommodate th=

is

> for
> you.
>
> Thank you in advance and sorry for the inconvenience,
>
>
>
> ---
>
> UnitedLayer Operations:
> Phu Phan
>
> UnitedLayer, LLC
> Main (415) 349-2100
> Toll Free (888) 853-7733
> Support/NOC (415) 349-2102
> (available 24x7)

20

20

20

20

--=20
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

20

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Rob Halsell replied via email on Tue, 16 Feb 2016 11:57:31 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00 to 14:00
Pacific), is fine. We'll migrate our traffic away during the PDU swap,
please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great for us, please
let
us know if you can accommodate that maintenance window. We'd like to make
this
as easy on you as possible so any of those days will work for us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at that site,
which is 20:00 GMT / 12:00 Pacific. So for a maintenance window, can we
set it for the two hours before and afterwards, so 10AM to 2PM Pacific;
additionally we would like to keep this work to Tuesday, Wednesday, or
Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thursdays
between 10:00-14:00 Pacific work best for you. All but our management
network have redundant power supplies wired to both towers, so actual
downtime should be minimal. We plan to migrate all traffic away from

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

It has come to our attention that your network management card for your
(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor your PDU
because of this issue. We would like to work with you to fix the issue
sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associated with

this

work so we would like to schedule a maintenance window to your

convenience

so
that we can go ahead and swap out the bad PDU. If we determine that
downtime
is needed, we can schedule this maintenance window during off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If you would
like to
discuss this over the phone please let me know and I can accommodate

this

for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

UnitedLayer Support Ticket System replied via email on Tue, 16 Feb 2016 14:51:01 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Dear Rob,=20

That sounds great! We will email once have everything ready at 11:00AM PST.=

20

Thank you,=20
Phu=20

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00 to 14:00
Pacific), is fine. We'll migrate our traffic away during the PDU swap,
please advise upon the start and completion of work.

20

Thanks!

20

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

20

> Dear Rob,
>
> Preferably the sooner the better. Tomorrow would be great for us, please
> let
> us know if you can accommodate that maintenance window. We'd like to ma=

ke

this
as easy on you as possible so any of those days will work for us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at that si=

te,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance window, can=

we

set it for the two hours before and afterwards, so 10AM to 2PM Pacifi=

c;

additionally we would like to keep this work to Tuesday, Wednesday, or
Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thursdays
between 10:00-14:00 Pacific work best for you. All but our management
network have redundant power supplies wired to both towers, so actual
downtime should be minimal. We plan to migrate all traffic away from

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

It has come to our attention that your network management card for =

your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor your=

PDU

because of this issue. We would like to work with you to fix the is=

sue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associated wi=

th

this

work so we would like to schedule a maintenance window to your

convenience

so
that we can go ahead and swap out the bad PDU. If we determine that
downtime
is needed, we can schedule this maintenance window during off hours=

so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If you wo=

uld

> > > like to
> > > discuss this over the phone please let me know and I can accommodate
> this
> > > for
> > > you.
> > >
> > > Thank you in advance and sorry for the inconvenience,
> > >
> > >
> > >
> > > ---
> > >
> > > UnitedLayer Operations:
> > > Phu Phan
> > >
> > > UnitedLayer, LLC
> > > Main (415) 349-2100
> > > Toll Free (888) 853-7733
> > > Support/NOC (415) 349-2102
> > > (available 24x7)
> >
> >
> >
> >
> > --
> > Rob Halsell
> > Operations Engineer
> > Wikimedia Foundation, Inc.
> > E-Mail: rhalsell@wikimedia.org
> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> > *0245 D22A*Office: 415.839.6885 x6620
> > Fax: 415.882.0495
> >
>
>
>
> ---
>
> UnitedLayer Operations:
> Phu Phan
>
> UnitedLayer, LLC
> Main (415) 349-2100
> Toll Free (888) 853-7733
> Support/NOC (415) 349-2102
> (available 24x7)
>
>

20

20

--=20
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

20

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Will depool at ~9AM Pacific, 17:00 UTC.

Change 271289 had a related patch set uploaded (by Ema):
Depool ulsfo

https://gerrit.wikimedia.org/r/271289

Change 271289 merged by Ema:
Depool ulsfo

https://gerrit.wikimedia.org/r/271289

Mentioned in SAL [2016-02-17T17:02:48Z] <ema> depooled ulsfo https://phabricator.wikimedia.org/T127094

UnitedLayer Support Ticket System replied via email on Wed, 17 Feb 2016 10:49:13 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Dear Rob,=20

I just wanted to give you guys a heads up that we will be ready to swap out
the pdu at 11AM PST, please let me know when you have diverted traffic.=20

Thank you,=20
Phu=20

Dear Rob,=20

20

That sounds great! We will email once have everything ready at 11:00AM PS=

T.=20

20

Thank you,=20
Phu=20

20

> Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00 to 14:=

00

Pacific), is fine. We'll migrate our traffic away during the PDU swap,
please advise upon the start and completion of work.

20

Thanks!

20

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

20

> Dear Rob,
>
> Preferably the sooner the better. Tomorrow would be great for us, ple=

ase

let
us know if you can accommodate that maintenance window. We'd like to =

make

this
as easy on you as possible so any of those days will work for us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at that =

site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance window, c=

an we

set it for the two hours before and afterwards, so 10AM to 2PM Paci=

fic;

additionally we would like to keep this work to Tuesday, Wednesday,=

or

Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thursdays
between 10:00-14:00 Pacific work best for you. All but our managem=

ent

network have redundant power supplies wired to both towers, so actu=

al

downtime should be minimal. We plan to migrate all traffic away fr=

om

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

It has come to our attention that your network management card fo=

r your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor yo=

ur PDU

because of this issue. We would like to work with you to fix the =

issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associated =

with

this

work so we would like to schedule a maintenance window to your

convenience

so
that we can go ahead and swap out the bad PDU. If we determine th=

at

downtime
is needed, we can schedule this maintenance window during off hou=

rs so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If you =

would

like to
discuss this over the phone please let me know and I can accommod=

ate

> > this
> > > > for
> > > > you.
> > > >
> > > > Thank you in advance and sorry for the inconvenience,
> > > >
> > > >
> > > >
> > > > ---
> > > >
> > > > UnitedLayer Operations:
> > > > Phu Phan
> > > >
> > > > UnitedLayer, LLC
> > > > Main (415) 349-2100
> > > > Toll Free (888) 853-7733
> > > > Support/NOC (415) 349-2102
> > > > (available 24x7)
> > >
> > >
> > >
> > >
> > > --
> > > Rob Halsell
> > > Operations Engineer
> > > Wikimedia Foundation, Inc.
> > > E-Mail: rhalsell@wikimedia.org
> > > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> > > *0245 D22A*Office: 415.839.6885 x6620
> > > Fax: 415.882.0495
> > >
> >
> >
> >
> > ---
> >
> > UnitedLayer Operations:
> > Phu Phan
> >
> > UnitedLayer, LLC
> > Main (415) 349-2100
> > Toll Free (888) 853-7733
> > Support/NOC (415) 349-2102
> > (available 24x7)
> >
> >
>=20
>=20
> --=20
> Rob Halsell
> Operations Engineer
> Wikimedia Foundation, Inc.
> E-Mail: rhalsell@wikimedia.org
> Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> *0245 D22A*Office: 415.839.6885 x6620
> Fax: 415.882.0495
>=20

20

20

20

---

20

UnitedLayer Operations:
Phu Phan

20

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

20

20

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Brandon Black replied via email on Wed, 17 Feb 2016 18:58:12 +0000

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

We diverted traffic early, circa 9AM PST, we're ready whenever you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be ready to swap out
the pdu at 11AM PST, please let me know when you have diverted traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at 11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00 to 14:00
Pacific), is fine. We'll migrate our traffic away during the PDU swap,
please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great for us, please
let
us know if you can accommodate that maintenance window. We'd like to make
this
as easy on you as possible so any of those days will work for us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at that site,
which is 20:00 GMT / 12:00 Pacific. So for a maintenance window, can we
set it for the two hours before and afterwards, so 10AM to 2PM Pacific;
additionally we would like to keep this work to Tuesday, Wednesday, or
Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thursdays
between 10:00-14:00 Pacific work best for you. All but our management
network have redundant power supplies wired to both towers, so actual
downtime should be minimal. We plan to migrate all traffic away from

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

It has come to our attention that your network management card for your
(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor your PDU
because of this issue. We would like to work with you to fix the issue
sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associated with

this

work so we would like to schedule a maintenance window to your

convenience

so
that we can go ahead and swap out the bad PDU. If we determine that
downtime
is needed, we can schedule this maintenance window during off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If you would
like to
discuss this over the phone please let me know and I can accommodate

this

for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

UnitedLayer Support Ticket System replied via email on Wed, 17 Feb 2016 11:02:02 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Thanks, I will update you once I have completed the replacement.=20

--Phu=20

We diverted traffic early, circa 9AM PST, we're ready whenever you are.

20

Thanks,

• Brandon

20

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
<support@unitedlayer.com> wrote:
> Dear Rob,
>
> I just wanted to give you guys a heads up that we will be ready to swap=

out

the pdu at 11AM PST, please let me know when you have diverted traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at 11:00AM=

PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00 to =

14:00

Pacific), is fine. We'll migrate our traffic away during the PDU sw=

ap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great for us, =

please

let
us know if you can accommodate that maintenance window. We'd like =

to make

this
as easy on you as possible so any of those days will work for us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at th=

at site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance window=

, can we

set it for the two hours before and afterwards, so 10AM to 2PM P=

acific;

additionally we would like to keep this work to Tuesday, Wednesd=

ay, or

Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thurs=

days

between 10:00-14:00 Pacific work best for you. All but our mana=

gement

network have redundant power supplies wired to both towers, so a=

ctual

downtime should be minimal. We plan to migrate all traffic away=

from

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket Syst=

em <

support@unitedlayer.com> wrote:

It has come to our attention that your network management card=

for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor=

your PDU

because of this issue. We would like to work with you to fix t=

he issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associat=

ed with

this

work so we would like to schedule a maintenance window to your

convenience

so
that we can go ahead and swap out the bad PDU. If we determine=

that

downtime
is needed, we can schedule this maintenance window during off =

hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If y=

ou would

like to
discuss this over the phone please let me know and I can accom=

modate

>> > > this
>> > > > > for
>> > > > > you.
>> > > > >
>> > > > > Thank you in advance and sorry for the inconvenience,
>> > > > >
>> > > > >
>> > > > >
>> > > > > ---
>> > > > >
>> > > > > UnitedLayer Operations:
>> > > > > Phu Phan
>> > > > >
>> > > > > UnitedLayer, LLC
>> > > > > Main (415) 349-2100
>> > > > > Toll Free (888) 853-7733
>> > > > > Support/NOC (415) 349-2102
>> > > > > (available 24x7)
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Rob Halsell
>> > > > Operations Engineer
>> > > > Wikimedia Foundation, Inc.
>> > > > E-Mail: rhalsell@wikimedia.org
>> > > > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
>> > > > *0245 D22A*Office: 415.839.6885 x6620
>> > > > Fax: 415.882.0495
>> > > >
>> > >
>> > >
>> > >
>> > > ---
>> > >
>> > > UnitedLayer Operations:
>> > > Phu Phan
>> > >
>> > > UnitedLayer, LLC
>> > > Main (415) 349-2100
>> > > Toll Free (888) 853-7733
>> > > Support/NOC (415) 349-2102
>> > > (available 24x7)
>> > >
>> > >
>> >
>> >
>> > --
>> > Rob Halsell
>> > Operations Engineer
>> > Wikimedia Foundation, Inc.
>> > E-Mail: rhalsell@wikimedia.org
>> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
>> > *0245 D22A*Office: 415.839.6885 x6620
>> > Fax: 415.882.0495
>> >
>>
>>
>>
>> ---
>>
>> UnitedLayer Operations:
>> Phu Phan
>>
>> UnitedLayer, LLC
>> Main (415) 349-2100
>> Toll Free (888) 853-7733
>> Support/NOC (415) 349-2102
>> (available 24x7)
>>
>>
>
>
>
> ---
>
> UnitedLayer Operations:
> Phu Phan
>
> UnitedLayer, LLC
> Main (415) 349-2100
> Toll Free (888) 853-7733
> Support/NOC (415) 349-2102
> (available 24x7)
>

20

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

UnitedLayer Support Ticket System replied via email on Wed, 17 Feb 2016 13:11:25 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Dear Brandon,=20

We have swapped out the pdus, but I do not see any power being drawn from a=
ny
of the power supplies in cabinet 123, and some of the servers in cabinet 12=
2.=20

Can you check the servers.=20

Thanks,=20
Phu=20

Thanks, I will update you once I have completed the replacement.=20

20

--Phu=20

20

> We diverted traffic early, circa 9AM PST, we're ready whenever you are.
>=20
> Thanks,
> -- Brandon
>=20
> On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
> <support@unitedlayer.com> wrote:
> > Dear Rob,
> >
> > I just wanted to give you guys a heads up that we will be ready to sw=

ap out

the pdu at 11AM PST, please let me know when you have diverted traffi=

c.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at 11:00=

AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00 t=

o 14:00

Pacific), is fine. We'll migrate our traffic away during the PDU =

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket Syste=

m <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great for us=

, please

let
us know if you can accommodate that maintenance window. We'd lik=

e to make

this
as easy on you as possible so any of those days will work for us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at =

that site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance wind=

ow, can we

set it for the two hours before and afterwards, so 10AM to 2PM=

Pacific;

additionally we would like to keep this work to Tuesday, Wedne=

sday, or

Thursday.

Please let us know which of the upcoming Tuesday/Wednesday/Thu=

rsdays

between 10:00-14:00 Pacific work best for you. All but our ma=

nagement

network have redundant power supplies wired to both towers, so=

actual

downtime should be minimal. We plan to migrate all traffic aw=

ay from

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket Sy=

stem <

support@unitedlayer.com> wrote:

It has come to our attention that your network management ca=

rd for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22=

is

malfunctioning. We are not able to remotely access and monit=

or your PDU

because of this issue. We would like to work with you to fix=

the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers associ=

ated with

this

work so we would like to schedule a maintenance window to yo=

ur

convenience

so
that we can go ahead and swap out the bad PDU. If we determi=

ne that

downtime
is needed, we can schedule this maintenance window during of=

f hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns. If=

you would

like to
discuss this over the phone please let me know and I can acc=

ommodate

> >> > > this
> >> > > > > for
> >> > > > > you.
> >> > > > >
> >> > > > > Thank you in advance and sorry for the inconvenience,
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > ---
> >> > > > >
> >> > > > > UnitedLayer Operations:
> >> > > > > Phu Phan
> >> > > > >
> >> > > > > UnitedLayer, LLC
> >> > > > > Main (415) 349-2100
> >> > > > > Toll Free (888) 853-7733
> >> > > > > Support/NOC (415) 349-2102
> >> > > > > (available 24x7)
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Rob Halsell
> >> > > > Operations Engineer
> >> > > > Wikimedia Foundation, Inc.
> >> > > > E-Mail: rhalsell@wikimedia.org
> >> > > > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> >> > > > *0245 D22A*Office: 415.839.6885 x6620
> >> > > > Fax: 415.882.0495
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > ---
> >> > >
> >> > > UnitedLayer Operations:
> >> > > Phu Phan
> >> > >
> >> > > UnitedLayer, LLC
> >> > > Main (415) 349-2100
> >> > > Toll Free (888) 853-7733
> >> > > Support/NOC (415) 349-2102
> >> > > (available 24x7)
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Rob Halsell
> >> > Operations Engineer
> >> > Wikimedia Foundation, Inc.
> >> > E-Mail: rhalsell@wikimedia.org
> >> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> >> > *0245 D22A*Office: 415.839.6885 x6620
> >> > Fax: 415.882.0495
> >> >
> >>
> >>
> >>
> >> ---
> >>
> >> UnitedLayer Operations:
> >> Phu Phan
> >>
> >> UnitedLayer, LLC
> >> Main (415) 349-2100
> >> Toll Free (888) 853-7733
> >> Support/NOC (415) 349-2102
> >> (available 24x7)
> >>
> >>
> >
> >
> >
> > ---
> >
> > UnitedLayer Operations:
> > Phu Phan
> >
> > UnitedLayer, LLC
> > Main (415) 349-2100
> > Toll Free (888) 853-7733
> > Support/NOC (415) 349-2102
> > (available 24x7)
> >
>=20

20

20

20

---

20

UnitedLayer Operations:
Phu Phan

20

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

20

20

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Rob Halsell replied via email on Wed, 17 Feb 2016 13:24:03 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo (Juniper
SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs are
switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to being in
the offline PDU.

Please advise if you are able to connect a power meter and read power
output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being drawn from
any
of the power supplies in cabinet 123, and some of the servers in cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready whenever you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be ready to

swap out

the pdu at 11AM PST, please let me know when you have diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away during the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great for

us, please

let
us know if you can accommodate that maintenance window. We'd

like to make

this
as easy on you as possible so any of those days will work for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance

window, can we

set it for the two hours before and afterwards, so 10AM to

2PM Pacific;

additionally we would like to keep this work to Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All but our

management

network have redundant power supplies wired to both towers,

so actual

downtime should be minimal. We plan to migrate all traffic

away from

that

site during the maintenance window as a preventative measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet

1.22 is

malfunctioning. We are not able to remotely access and

monitor your PDU

because of this issue. We would like to work with you to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers

associated with

this

work so we would like to schedule a maintenance window to

your

convenience

so
that we can go ahead and swap out the bad PDU. If we

determine that

downtime
is needed, we can schedule this maintenance window during

off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns.

If you would

like to
discuss this over the phone please let me know and I can

accommodate

this

for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

UnitedLayer Support Ticket System replied via email on Wed, 17 Feb 2016 13:42:32 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Dear Brandon,=20

There was a breaker switch on the new pdu, everything should be up. Can you
please verify.=20

Thanks,
Phu=20

Support,

20

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo (Junip=

er

SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs are
switch outlets, might the outlets not be turned on?

20

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to being in
the offline PDU.

20

Please advise if you are able to connect a power meter and read power
output on any of the outlets in the replacement PDU in 1.23?

20

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

20

> Dear Brandon,
>
> We have swapped out the pdus, but I do not see any power being drawn fr=

om

any
of the power supplies in cabinet 123, and some of the servers in cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready whenever you =

are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be ready to

swap out

the pdu at 11AM PST, please let me know when you have diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT (11:=

00

to 14:00

Pacific), is fine. We'll migrate our traffic away during the =

PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great for

us, please

let
us know if you can accommodate that maintenance window. We'd

like to make

this
as easy on you as possible so any of those days will work for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic low-point=

at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance

window, can we

set it for the two hours before and afterwards, so 10AM to

2PM Pacific;

additionally we would like to keep this work to Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All but our

management

network have redundant power supplies wired to both towers,

so actual

downtime should be minimal. We plan to migrate all traffic

away from

that

site during the maintenance window as a preventative measu=

re.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet

1.22 is

malfunctioning. We are not able to remotely access and

monitor your PDU

because of this issue. We would like to work with you to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers

associated with

this

work so we would like to schedule a maintenance window to

your

convenience

so
that we can go ahead and swap out the bad PDU. If we

determine that

downtime
is needed, we can schedule this maintenance window during

off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or concerns.

If you would

like to
discuss this over the phone please let me know and I can

accommodate

this

for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14=

C7

> > > >> > > > *0245 D22A*Office: 415.839.6885 x6620
> > > >> > > > Fax: 415.882.0495
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > ---
> > > >> > >
> > > >> > > UnitedLayer Operations:
> > > >> > > Phu Phan
> > > >> > >
> > > >> > > UnitedLayer, LLC
> > > >> > > Main (415) 349-2100
> > > >> > > Toll Free (888) 853-7733
> > > >> > > Support/NOC (415) 349-2102
> > > >> > > (available 24x7)
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rob Halsell
> > > >> > Operations Engineer
> > > >> > Wikimedia Foundation, Inc.
> > > >> > E-Mail: rhalsell@wikimedia.org
> > > >> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> > > >> > *0245 D22A*Office: 415.839.6885 x6620
> > > >> > Fax: 415.882.0495
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> ---
> > > >>
> > > >> UnitedLayer Operations:
> > > >> Phu Phan
> > > >>
> > > >> UnitedLayer, LLC
> > > >> Main (415) 349-2100
> > > >> Toll Free (888) 853-7733
> > > >> Support/NOC (415) 349-2102
> > > >> (available 24x7)
> > > >>
> > > >>
> > > >
> > > >
> > > >
> > > > ---
> > > >
> > > > UnitedLayer Operations:
> > > > Phu Phan
> > > >
> > > > UnitedLayer, LLC
> > > > Main (415) 349-2100
> > > > Toll Free (888) 853-7733
> > > > Support/NOC (415) 349-2102
> > > > (available 24x7)
> > > >
> > >
> >
> >
> >
> > ---
> >
> > UnitedLayer Operations:
> > Phu Phan
> >
> > UnitedLayer, LLC
> > Main (415) 349-2100
> > Toll Free (888) 853-7733
> > Support/NOC (415) 349-2102
> > (available 24x7)
> >
> >
>
>
>
> ---
>
> UnitedLayer Operations:
> Phu Phan
>
> UnitedLayer, LLC
> Main (415) 349-2100
> Toll Free (888) 853-7733
> Support/NOC (415) 349-2102
> (available 24x7)
>
>

20

20

--=20
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

20

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Rob Halsell replied via email on Wed, 17 Feb 2016 14:15:08 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Support,

I still dont see power output to both of our power supplies on our router,
which leads me to think we still don't have power output to all ports.
Additionally, the mr1-ulsfo management router that I advised on in my last
email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power output on
the outlets? (This was already requested, but was not yet addressed.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be up. Can you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs are
switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to being in
the offline PDU.

Please advise if you are able to connect a power meter and read power
output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being drawn

from

any
of the power supplies in cabinet 123, and some of the servers in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready whenever you

are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be ready

to

swap out

the pdu at 11AM PST, please let me know when you have diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away during the

PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great

for

us, please

let
us know if you can accommodate that maintenance window. We'd

like to make

this
as easy on you as possible so any of those days will work

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance

window, can we

set it for the two hours before and afterwards, so 10AM to

2PM Pacific;

additionally we would like to keep this work to Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All but

our

management

network have redundant power supplies wired to both

towers,

so actual

downtime should be minimal. We plan to migrate all

traffic

away from

that

site during the maintenance window as a preventative

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet

1.22 is

malfunctioning. We are not able to remotely access and

monitor your PDU

because of this issue. We would like to work with you to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers

associated with

this

work so we would like to schedule a maintenance window

to

your

convenience

so
that we can go ahead and swap out the bad PDU. If we

determine that

downtime
is needed, we can schedule this maintenance window

during

off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know and I can

accommodate

this

for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

Rob Halsell replied via email on Wed, 17 Feb 2016 14:16:07 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Phu,

If you need to call me directly, my cell is +1-727-255-4597. I'm normally
the WMF employee who shows up to 200 paul in person. I'm not sure what
else we can suggest at this point, other than what we already have.

Please advise,

On Wed, Feb 17, 2016 at 2:15 PM, Rob Halsell <rhalsell@wikimedia.org> wrote:

Support,

I still dont see power output to both of our power supplies on our router,
which leads me to think we still don't have power output to all ports.
Additionally, the mr1-ulsfo management router that I advised on in my last
email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power output on
the outlets? (This was already requested, but was not yet addressed.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be up. Can
you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs are
switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to being in
the offline PDU.

Please advise if you are able to connect a power meter and read power
output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being drawn

from

any
of the power supplies in cabinet 123, and some of the servers in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready whenever

you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket System
<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be ready

to

swap out

the pdu at 11AM PST, please let me know when you have diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away during

the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be great

for

us, please

let
us know if you can accommodate that maintenance window.

We'd

like to make

this
as easy on you as possible so any of those days will work

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a maintenance

window, can we

set it for the two hours before and afterwards, so 10AM

to

2PM Pacific;

additionally we would like to keep this work to Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All but

our

management

network have redundant power supplies wired to both

towers,

so actual

downtime should be minimal. We plan to migrate all

traffic

away from

that

site during the maintenance window as a preventative

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet

1.22 is

malfunctioning. We are not able to remotely access and

monitor your PDU

because of this issue. We would like to work with you

to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your servers

associated with

this

work so we would like to schedule a maintenance window

to

your

convenience

so
that we can go ahead and swap out the bad PDU. If we

determine that

downtime
is needed, we can schedule this maintenance window

during

off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know and I

can

accommodate

this

for
you.

Thank you in advance and sorry for the inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

UnitedLayer Support Ticket System replied via email on Wed, 17 Feb 2016 14:39:47 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Hi Rod,

The power cable from your firewall to the pdu was a bit loose. I re-seated =
it
properly and it came back up.

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Phu,

20

If you need to call me directly, my cell is +1-727-255-4597. I'm normally
the WMF employee who shows up to 200 paul in person. I'm not sure what
else we can suggest at this point, other than what we already have.

20

Please advise,

20

On Wed, Feb 17, 2016 at 2:15 PM, Rob Halsell <rhalsell@wikimedia.org> wro=

te:

20

> Support,
>
> I still dont see power output to both of our power supplies on our rout=

er,

which leads me to think we still don't have power output to all ports.
Additionally, the mr1-ulsfo management router that I advised on in my l=

ast

email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power output on
the outlets? (This was already requested, but was not yet addressed.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be up. Can
you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs a=

re

switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to bein=

g in

the offline PDU.

Please advise if you are able to connect a power meter and read power
output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being dra=

wn

from

any
of the power supplies in cabinet 123, and some of the servers in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready whenever

you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket Sy=

stem

<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be re=

ady

to

swap out

the pdu at 11AM PST, please let me know when you have divert=

ed

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything ready=

at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away during

the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support Tic=

ket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be gre=

at

for

us, please

let
us know if you can accommodate that maintenance window.

We'd

like to make

this
as easy on you as possible so any of those days will wo=

rk

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a mainten=

ance

window, can we

set it for the two hours before and afterwards, so 10=

AM

to

2PM Pacific;

additionally we would like to keep this work to Tuesd=

ay,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All b=

ut

our

management

network have redundant power supplies wired to both

towers,

so actual

downtime should be minimal. We plan to migrate all

traffic

away from

that

site during the maintenance window as a preventative

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in cab=

inet

1.22 is

malfunctioning. We are not able to remotely access =

and

monitor your PDU

because of this issue. We would like to work with y=

ou

to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your serve=

rs

associated with

this

work so we would like to schedule a maintenance win=

dow

to

your

convenience

so
that we can go ahead and swap out the bad PDU. If we

determine that

downtime
is needed, we can schedule this maintenance window

during

off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know and I

can

accommodate

this

for
you.

Thank you in advance and sorry for the inconvenienc=

e,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75=

ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 1=

4C7

>> > > > > >> > *0245 D22A*Office: 415.839.6885 x6620
>> > > > > >> > Fax: 415.882.0495
>> > > > > >> >
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> ---
>> > > > > >>
>> > > > > >> UnitedLayer Operations:
>> > > > > >> Phu Phan
>> > > > > >>
>> > > > > >> UnitedLayer, LLC
>> > > > > >> Main (415) 349-2100
>> > > > > >> Toll Free (888) 853-7733
>> > > > > >> Support/NOC (415) 349-2102
>> > > > > >> (available 24x7)
>> > > > > >>
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > ---
>> > > > > >
>> > > > > > UnitedLayer Operations:
>> > > > > > Phu Phan
>> > > > > >
>> > > > > > UnitedLayer, LLC
>> > > > > > Main (415) 349-2100
>> > > > > > Toll Free (888) 853-7733
>> > > > > > Support/NOC (415) 349-2102
>> > > > > > (available 24x7)
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > ---
>> > > >
>> > > > UnitedLayer Operations:
>> > > > Phu Phan
>> > > >
>> > > > UnitedLayer, LLC
>> > > > Main (415) 349-2100
>> > > > Toll Free (888) 853-7733
>> > > > Support/NOC (415) 349-2102
>> > > > (available 24x7)
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > > ---
>> > >
>> > > UnitedLayer Operations:
>> > > Phu Phan
>> > >
>> > > UnitedLayer, LLC
>> > > Main (415) 349-2100
>> > > Toll Free (888) 853-7733
>> > > Support/NOC (415) 349-2102
>> > > (available 24x7)
>> > >
>> > >
>> >
>> >
>> > --
>> > Rob Halsell
>> > Operations Engineer
>> > Wikimedia Foundation, Inc.
>> > E-Mail: rhalsell@wikimedia.org
>> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
>> > *0245 D22A*Office: 415.839.6885 x6620
>> > Fax: 415.882.0495
>> >
>>
>>
>>
>> ---
>>
>> UnitedLayer Operations:
>> Phu Phan
>>
>> UnitedLayer, LLC
>> Main (415) 349-2100
>> Toll Free (888) 853-7733
>> Support/NOC (415) 349-2102
>> (available 24x7)
>>
>>
>
>
> --
> Rob Halsell
> Operations Engineer
> Wikimedia Foundation, Inc.
> E-Mail: rhalsell@wikimedia.org
> Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> *0245 D22A*Office: 415.839.6885 x6620
> Fax: 415.882.0495
>
>

20

20

--=20
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

20

Rob Halsell replied via email on Wed, 17 Feb 2016 14:48:32 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Carlos,

I'm now trying to connect to other devices, and it seems that I cannot
connect to the mgmt network yet. I see power to both the power supplies on
our main routers(MX480s), but the others seem to be taking time to come
back online.

Can you advise if the switch labeled msw1-ulsfo (1.23:u38 or
msw2-ulsfo)(1.22:u37) have powered on? mr1-ulsfo also hasn't come back
online yet, and its been long enough that it should have booted by now.

Unfortunately, with those down I cannot troubleshoot, as they are my mgmt
network. You advised mr1-ulsfo was powering on correct? Does it have any
error indicator LEDs illuminated? We've left the end of our downtime
window, so I'd like to try to resolve this before the evening.

Please advise,

On Wed, Feb 17, 2016 at 2:39 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Hi Rod,

The power cable from your firewall to the pdu was a bit loose. I re-seated
it
properly and it came back up.

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Phu,

If you need to call me directly, my cell is +1-727-255-4597. I'm

normally

the WMF employee who shows up to 200 paul in person. I'm not sure what
else we can suggest at this point, other than what we already have.

Please advise,

On Wed, Feb 17, 2016 at 2:15 PM, Rob Halsell <rhalsell@wikimedia.org>

wrote:

Support,

I still dont see power output to both of our power supplies on our

router,

which leads me to think we still don't have power output to all ports.
Additionally, the mr1-ulsfo management router that I advised on in my

last

email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power output

on

the outlets? (This was already requested, but was not yet addressed.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be up.

Can

you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs

are

switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to

being in

the offline PDU.

Please advise if you are able to connect a power meter and read

power

output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being

drawn

from

any
of the power supplies in cabinet 123, and some of the servers in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready whenever

you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket

System

<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be

ready

to

swap out

the pdu at 11AM PST, please let me know when you have

diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything

ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00 GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away during

the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be

great

for

us, please

let
us know if you can accommodate that maintenance window.

We'd

like to make

this
as easy on you as possible so any of those days will

work

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a

maintenance

window, can we

set it for the two hours before and afterwards, so

10AM

to

2PM Pacific;

additionally we would like to keep this work to

Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All

but

our

management

network have redundant power supplies wired to both

towers,

so actual

downtime should be minimal. We plan to migrate all

traffic

away from

that

site during the maintenance window as a preventative

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in

cabinet

1.22 is

malfunctioning. We are not able to remotely access

and

monitor your PDU

because of this issue. We would like to work with

you

to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your

servers

associated with

this

work so we would like to schedule a maintenance

window

to

your

convenience

so
that we can go ahead and swap out the bad PDU. If

we

determine that

downtime
is needed, we can schedule this maintenance window

during

off hours so

the

impact to your business continuity is at a minimum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know and

I

can

accommodate

this

for
you.

Thank you in advance and sorry for the

inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

Rob Halsell replied via email on Wed, 17 Feb 2016 15:00:09 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Actually, if whoever is on the floor troubleshooting can call me back
directly on my cell I'd be happy to stay online with them while we
troubleshoot all the devices that are offline. Carlos called me earlier
with an update, but after we got the update I hung up to check the
devices. My cell is 727-255-4597.

When this was originally scheduled, I hadn't planned on being onsite. If
we cannot get this sorted shortly, I'll head down there to work with
UnitedLayer directly. We would prefer to avoid downtime during the peak
hours for this site.

On Wed, Feb 17, 2016 at 2:48 PM, Rob Halsell <rhalsell@wikimedia.org> wrote:

Carlos,

I'm now trying to connect to other devices, and it seems that I cannot
connect to the mgmt network yet. I see power to both the power supplies on
our main routers(MX480s), but the others seem to be taking time to come
back online.

Can you advise if the switch labeled msw1-ulsfo (1.23:u38 or
msw2-ulsfo)(1.22:u37) have powered on? mr1-ulsfo also hasn't come back
online yet, and its been long enough that it should have booted by now.

Unfortunately, with those down I cannot troubleshoot, as they are my mgmt
network. You advised mr1-ulsfo was powering on correct? Does it have any
error indicator LEDs illuminated? We've left the end of our downtime
window, so I'd like to try to resolve this before the evening.

Please advise,

On Wed, Feb 17, 2016 at 2:39 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Hi Rod,

The power cable from your firewall to the pdu was a bit loose. I
re-seated it
properly and it came back up.

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Phu,

If you need to call me directly, my cell is +1-727-255-4597. I'm

normally

the WMF employee who shows up to 200 paul in person. I'm not sure what
else we can suggest at this point, other than what we already have.

Please advise,

On Wed, Feb 17, 2016 at 2:15 PM, Rob Halsell <rhalsell@wikimedia.org>

wrote:

Support,

I still dont see power output to both of our power supplies on our

router,

which leads me to think we still don't have power output to all ports.
Additionally, the mr1-ulsfo management router that I advised on in my

last

email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power output

on

the outlets? (This was already requested, but was not yet addressed.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be up.

Can

you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-ulsfo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new PDUs

are

switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as the
mr1-ulsfo and the mgmt switches are powered down in 1.23 due to

being in

the offline PDU.

Please advise if you are able to connect a power meter and read

power

output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket System

<

support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being

drawn

from

any
of the power supplies in cabinet 123, and some of the servers in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the

replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready

whenever

you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket

System

<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be

ready

to

swap out

the pdu at 11AM PST, please let me know when you have

diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything

ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00

GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away

during

the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be

great

for

us, please

let
us know if you can accommodate that maintenance

window.

We'd

like to make

this
as easy on you as possible so any of those days will

work

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a

maintenance

window, can we

set it for the two hours before and afterwards, so

10AM

to

2PM Pacific;

additionally we would like to keep this work to

Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. All

but

our

management

network have redundant power supplies wired to both

towers,

so actual

downtime should be minimal. We plan to migrate all

traffic

away from

that

site during the maintenance window as a preventative

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in

cabinet

1.22 is

malfunctioning. We are not able to remotely

access and

monitor your PDU

because of this issue. We would like to work with

you

to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your

servers

associated with

this

work so we would like to schedule a maintenance

window

to

your

convenience

so
that we can go ahead and swap out the bad PDU. If

we

determine that

downtime
is needed, we can schedule this maintenance window

during

off hours so

the

impact to your business continuity is at a

minimum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know

and I

can

accommodate

this

for
you.

Thank you in advance and sorry for the

inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E

75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

{None}

UnitedLayer Support Ticket System replied via email on Wed, 17 Feb 2016 15:05:45 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Hi Rob,

Let me take a look and will call you back with my findings.

--Carlos

Carlos,

20

I'm now trying to connect to other devices, and it seems that I cannot
connect to the mgmt network yet. I see power to both the power supplies =

on

our main routers(MX480s), but the others seem to be taking time to come
back online.

20

Can you advise if the switch labeled msw1-ulsfo (1.23:u38 or
msw2-ulsfo)(1.22:u37) have powered on? mr1-ulsfo also hasn't come back
online yet, and its been long enough that it should have booted by now.

20

Unfortunately, with those down I cannot troubleshoot, as they are my mgmt
network. You advised mr1-ulsfo was powering on correct? Does it have any
error indicator LEDs illuminated? We've left the end of our downtime
window, so I'd like to try to resolve this before the evening.

20

Please advise,

20

On Wed, Feb 17, 2016 at 2:39 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

20

> Hi Rod,
>
> The power cable from your firewall to the pdu was a bit loose. I re-sea=

ted

it
properly and it came back up.

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Phu,

If you need to call me directly, my cell is +1-727-255-4597. I'm

normally

the WMF employee who shows up to 200 paul in person. I'm not sure wh=

at

else we can suggest at this point, other than what we already have.

Please advise,

On Wed, Feb 17, 2016 at 2:15 PM, Rob Halsell <rhalsell@wikimedia.org>

wrote:

Support,

I still dont see power output to both of our power supplies on our

router,

which leads me to think we still don't have power output to all por=

ts.

Additionally, the mr1-ulsfo management router that I advised on in =

my

last

email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power outp=

ut

on

the outlets? (This was already requested, but was not yet addresse=

d.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be up.

Can

you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our mr1-uls=

fo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new PD=

Us

are

switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as =

the

mr1-ulsfo and the mgmt switches are powered down in 1.23 due to

being in

the offline PDU.

Please advise if you are able to connect a power meter and read

power

output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket Syst=

em <

support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being

drawn

from

any
of the power supplies in cabinet 123, and some of the servers =

in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the replacem=

ent.

--Phu

We diverted traffic early, circa 9AM PST, we're ready when=

ever

you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support Ticket

System

<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will be

ready

to

swap out

the pdu at 11AM PST, please let me know when you have

diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything

ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of 18:00-22:00=

GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away du=

ring

the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be

great

for

us, please

let
us know if you can accommodate that maintenance win=

dow.

We'd

like to make

this
as easy on you as possible so any of those days will

work

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a

maintenance

window, can we

set it for the two hours before and afterwards, so

10AM

to

2PM Pacific;

additionally we would like to keep this work to

Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you. A=

ll

but

our

management

network have redundant power supplies wired to bo=

th

towers,

so actual

downtime should be minimal. We plan to migrate a=

ll

traffic

away from

that

site during the maintenance window as a preventat=

ive

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer Supp=

ort

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in

cabinet

1.22 is

malfunctioning. We are not able to remotely acc=

ess

and

monitor your PDU

because of this issue. We would like to work wi=

th

you

to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your

servers

associated with

this

work so we would like to schedule a maintenance

window

to

your

convenience

so
that we can go ahead and swap out the bad PDU. =

If

we

determine that

downtime
is needed, we can schedule this maintenance win=

dow

during

off hours so

the

impact to your business continuity is at a mini=

mum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know =

and

I

can

accommodate

this

for
you.

Thank you in advance and sorry for the

inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7=

E 75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75=

ED

> 14C7
> > >> > > > > >> > *0245 D22A*Office: 415.839.6885 x6620
> > >> > > > > >> > Fax: 415.882.0495
> > >> > > > > >> >
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >> ---
> > >> > > > > >>
> > >> > > > > >> UnitedLayer Operations:
> > >> > > > > >> Phu Phan
> > >> > > > > >>
> > >> > > > > >> UnitedLayer, LLC
> > >> > > > > >> Main (415) 349-2100
> > >> > > > > >> Toll Free (888) 853-7733
> > >> > > > > >> Support/NOC (415) 349-2102
> > >> > > > > >> (available 24x7)
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > ---
> > >> > > > > >
> > >> > > > > > UnitedLayer Operations:
> > >> > > > > > Phu Phan
> > >> > > > > >
> > >> > > > > > UnitedLayer, LLC
> > >> > > > > > Main (415) 349-2100
> > >> > > > > > Toll Free (888) 853-7733
> > >> > > > > > Support/NOC (415) 349-2102
> > >> > > > > > (available 24x7)
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > ---
> > >> > > >
> > >> > > > UnitedLayer Operations:
> > >> > > > Phu Phan
> > >> > > >
> > >> > > > UnitedLayer, LLC
> > >> > > > Main (415) 349-2100
> > >> > > > Toll Free (888) 853-7733
> > >> > > > Support/NOC (415) 349-2102
> > >> > > > (available 24x7)
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > ---
> > >> > >
> > >> > > UnitedLayer Operations:
> > >> > > Phu Phan
> > >> > >
> > >> > > UnitedLayer, LLC
> > >> > > Main (415) 349-2100
> > >> > > Toll Free (888) 853-7733
> > >> > > Support/NOC (415) 349-2102
> > >> > > (available 24x7)
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Rob Halsell
> > >> > Operations Engineer
> > >> > Wikimedia Foundation, Inc.
> > >> > E-Mail: rhalsell@wikimedia.org
> > >> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> > >> > *0245 D22A*Office: 415.839.6885 x6620
> > >> > Fax: 415.882.0495
> > >> >
> > >>
> > >>
> > >>
> > >> ---
> > >>
> > >> UnitedLayer Operations:
> > >> Phu Phan
> > >>
> > >> UnitedLayer, LLC
> > >> Main (415) 349-2100
> > >> Toll Free (888) 853-7733
> > >> Support/NOC (415) 349-2102
> > >> (available 24x7)
> > >>
> > >>
> > >
> > >
> > > --
> > > Rob Halsell
> > > Operations Engineer
> > > Wikimedia Foundation, Inc.
> > > E-Mail: rhalsell@wikimedia.org
> > > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> > > *0245 D22A*Office: 415.839.6885 x6620
> > > Fax: 415.882.0495
> > >
> > >
> >
> >
> > --
> > Rob Halsell
> > Operations Engineer
> > Wikimedia Foundation, Inc.
> > E-Mail: rhalsell@wikimedia.org
> > Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
> > *0245 D22A*Office: 415.839.6885 x6620
> > Fax: 415.882.0495
> >
>
>
>
>

20

20

--=20
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint =3D CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

20

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102=20
(available 24x7)

Rob Halsell replied via email on Wed, 17 Feb 2016 15:41:26 -0800

Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure

Carlos/Support,

Per our phone call just now, it seemed it all was working except
mr1-ulsfo. The hard reboot we had just done didn't seem to work; however,
right after we hung up, it came back online.

It appears that all our devices have nominal power and are working. Please
advise when UnitedLayer has locked up the cabinets (and all work/cleanup is
complete) and we'll repool our services there.

Thanks!

On Wed, Feb 17, 2016 at 3:05 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Hi Rob,

Let me take a look and will call you back with my findings.

--Carlos

Carlos,

I'm now trying to connect to other devices, and it seems that I cannot
connect to the mgmt network yet. I see power to both the power supplies

on

our main routers(MX480s), but the others seem to be taking time to come
back online.

Can you advise if the switch labeled msw1-ulsfo (1.23:u38 or
msw2-ulsfo)(1.22:u37) have powered on? mr1-ulsfo also hasn't come back
online yet, and its been long enough that it should have booted by now.

Unfortunately, with those down I cannot troubleshoot, as they are my mgmt
network. You advised mr1-ulsfo was powering on correct? Does it have

any

error indicator LEDs illuminated? We've left the end of our downtime
window, so I'd like to try to resolve this before the evening.

Please advise,

On Wed, Feb 17, 2016 at 2:39 PM, UnitedLayer Support Ticket System <
support@unitedlayer.com> wrote:

Hi Rod,

The power cable from your firewall to the pdu was a bit loose. I

re-seated

it
properly and it came back up.

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Phu,

If you need to call me directly, my cell is +1-727-255-4597. I'm

normally

the WMF employee who shows up to 200 paul in person. I'm not sure

what

else we can suggest at this point, other than what we already have.

Please advise,

On Wed, Feb 17, 2016 at 2:15 PM, Rob Halsell <rhalsell@wikimedia.org

wrote:

Support,

I still dont see power output to both of our power supplies on our

router,

which leads me to think we still don't have power output to all

ports.

Additionally, the mr1-ulsfo management router that I advised on in

my

last

email is still not powered on, nor does there seem to be power to
msw1-ulsfo, a switch in that rack.

Can you confirm with a multi-meter reading that there is power

output

on

the outlets? (This was already requested, but was not yet

addressed.)

On Wed, Feb 17, 2016 at 1:42 PM, UnitedLayer Support Ticket System

<

support@unitedlayer.com> wrote:

Dear Brandon,

There was a breaker switch on the new pdu, everything should be

up.

Can

you
please verify.

Thanks,
Phu

Support,

We cannot see any power from the new PDU in 1.23, as our

mr1-ulsfo

(Juniper

SRX100B in U36 of 1.23) hasn't powered back up. If these new

PDUs

are

switch outlets, might the outlets not be turned on?

We cannot see any of our mgmt interfaces to check power use, as

the

mr1-ulsfo and the mgmt switches are powered down in 1.23 due to

being in

the offline PDU.

Please advise if you are able to connect a power meter and read

power

output on any of the outlets in the replacement PDU in 1.23?

On Wed, Feb 17, 2016 at 1:11 PM, UnitedLayer Support Ticket

System <

support@unitedlayer.com> wrote:

Dear Brandon,

We have swapped out the pdus, but I do not see any power being

drawn

from

any
of the power supplies in cabinet 123, and some of the servers

in

cabinet

Can you check the servers.

Thanks,
Phu

Thanks, I will update you once I have completed the

replacement.

--Phu

We diverted traffic early, circa 9AM PST, we're ready

whenever

you are.

Thanks,

• Brandon

On Wed, Feb 17, 2016 at 6:49 PM, UnitedLayer Support

Ticket

System

<support@unitedlayer.com> wrote:

Dear Rob,

I just wanted to give you guys a heads up that we will

be

ready

to

swap out

the pdu at 11AM PST, please let me know when you have

diverted

traffic.

Thank you,
Phu

Dear Rob,

That sounds great! We will email once have everything

ready at

11:00AM PST.

Thank you,
Phu

Tomorrow, 2016-02-17 between the hours of

18:00-22:00 GMT

(11:00

to 14:00

Pacific), is fine. We'll migrate our traffic away

during

the PDU

swap,

please advise upon the start and completion of work.

Thanks!

On Tue, Feb 16, 2016 at 10:59 AM, UnitedLayer Support

Ticket

System <

support@unitedlayer.com> wrote:

Dear Rob,

Preferably the sooner the better. Tomorrow would be

great

for

us, please

let
us know if you can accommodate that maintenance

window.

We'd

like to make

this
as easy on you as possible so any of those days

will

work

for

us.

Thanks,
Phu

Phu/Support,

We prefer the PDU swap happen during our traffic

low-point at

that site,

which is 20:00 GMT / 12:00 Pacific. So for a

maintenance

window, can we

set it for the two hours before and afterwards,

so

10AM

to

2PM Pacific;

additionally we would like to keep this work to

Tuesday,

Wednesday, or

Thursday.

Please let us know which of the upcoming

Tuesday/Wednesday/Thursdays

between 10:00-14:00 Pacific work best for you.

All

but

our

management

network have redundant power supplies wired to

both

towers,

so actual

downtime should be minimal. We plan to migrate

all

traffic

away from

that

site during the maintenance window as a

preventative

measure.

Please advise,

On Tue, Feb 16, 2016 at 9:06 AM, UnitedLayer

Support

Ticket

System <

support@unitedlayer.com> wrote:

It has come to our attention that your network

management

card for your

(b-side)PDU in cabinet 1.23 and (a-side) PDU in

cabinet

1.22 is

malfunctioning. We are not able to remotely

access

and

monitor your PDU

because of this issue. We would like to work

with

you

to

fix the issue

sooner
than later so we do not run into any problems.

There may be some downtime with a few of your

servers

associated with

this

work so we would like to schedule a maintenance

window

to

your

convenience

so
that we can go ahead and swap out the bad PDU.

If

we

determine that

downtime
is needed, we can schedule this maintenance

window

during

off hours so

the

impact to your business continuity is at a

minimum.

Please let me know if you have any questions or

concerns.

If you would

like to
discuss this over the phone please let me know

and

I

can

accommodate

this

for
you.

Thank you in advance and sorry for the

inconvenience,

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E

75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED

14C7

*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Phu Phan

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495

---

UnitedLayer Operations:
Carlos Rios

UnitedLayer, LLC
Main (415) 349-2100
Toll Free (888) 853-7733
Support/NOC (415) 349-2102
(available 24x7)

~~
Rob Halsell
Operations Engineer
Wikimedia Foundation, Inc.
E-Mail: rhalsell@wikimedia.org
Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7
*0245 D22A*Office: 415.839.6885 x6620
Fax: 415.882.0495