Severity: ALERT
Subject: Lafayette and Baton Rouge LA down
The telco has found some misoptioned settings for the circuits serving this
facility. They are correcting them. Hopefully it won't be too long now.
Date: 19 Jan 98 at 13:46 US/Eastern
Severity: ALERT
Subject: RTP busies
We are experiencing busies in our RTP facility. This facility serves
the 424 and 425-7500 numbers.
We are investigating.
Date: 19 Jan 98 at 13:13 US/Eastern
Severity: ALERT
Subject: Baltimore MD (410)244-0719 Packet Loss
The date for the switch over has been moved back. It will
occur in the early morning hours of the coming Wednesday. Expect
a period of downtime.
Date: 19 Jan 98 at 13:10 US/Eastern
Severity: ALERT
Subject: Southern Pines NC reduced capacity
This PoP is at half capacity currently. PSI is dispatching
field service to the site. ETA is 5pm EST.
Date: 15 Jan 98 at 17:32 US/Eastern
Severity: DOWNTIME
Subject: San Jose, CA - 01/21/98 - 04:00 EST
PSINet's Mae-West router will have software and hardware upgraded.
The expected outage is 30 - 45 minutes as the router is rebooted and routing
re-converges. The expected outage is 30 minutes as the maintenance is
conducted.
Date: 15 Jan 98 at 17:21 US/Eastern
Severity: DOWNTIME
Subject: Santa Clara - 01/20/98 - 04:01 EST
Dial-up servers in Santa Clara will be down for a few minutes as modems are upgraded.
Date: 14 Jan 98 at 17:54 US/Eastern
Severity: ALERT
Subject: MSPG:Greenville SC (864) 272-6774 Calling Area
Please switch to the new Greenville number as soon as possible. We have
received reports of busy signals and recordings from the old number; there
is ample capacity (so no busies/recordings) at the new number
If you are experiencing difficulties with the new number, please contact us.
-SC
Date: 13 Jan 98 at 21:45 US/Eastern
Severity: UPDATE
Subject: Trunking problems in Atlanta
The following is a note from my manager, George Williams. It applies to
busy signals and telco recording messages our customers received a few weeks
ago and not to recent unrelated difficulties.
--John Butler
Senior NOC Engineer, Network Operations
Mindspring Enterprises, Inc.
---snip---
To all Atlanta MindSpring Customers:
MindSpring experienced great difficulties with it's telephone network
carriers from December 30, 1997 to January 7, 1998. The problems arose
with the amount of call transport capacity that was present in Atlanta's
local telephone network. MindSpring has enough modems to accomodate the
customers in Atlanta, but customers in Atlanta could not take full advantage
of those modems because the telephone companies were unable to even
deliver the call to our equipment. This is not the same situation that
another notable internet provider found itself in, that provider simply
did not have enough modems for it's customer base.
I personally assure all of you that these difficulties have been
addressed and corrected at this time. MindSpring will take the necessary
strategic steps with it's local telephone providers to prevent these
difficulties from occuring again.
Regards,
George B. Williams
Manager, MindSpring Network Operations Center
---snip---
Date: 08 Jan 98 at 06:47 US/Eastern
Severity: ALERT
Subject: Malibu connection problems
It will be yet another 15-20 days before additional trunks are installed
in this area to alleviate busies/recordings. We and our telco are pursuing
this issue as best we can.
-SC
Recently Resolved Problems
Date: 19 Jan 98 at 06:43 US/EasternStatus: resolved
Subject: Monday 0500-0630: Richmond and Norfolk longline maintenance
This maintenance was completed. Only Norfolk was moved. We are pursuing other
options for our Richmond longline.
Date: 19 Jan 98 at 06:42 US/Eastern
Status: resolved
Subject: Network packet loss
Looking back at our usage graphs for the evening it looks like there
was some sort of attack which saturated one of our main Atlanta links
for about 20 minutes. All has been well for several hours now.
Date: 19 Jan 98 at 06:15 US/Eastern
Status: resolved
Subject: Research Park Trg, Durham 919-424-7500/919-425-7500 Busies
We found a few misconfigured modems at this pop that were causing sporadic
busies to our users. We fixed the modems and we think this have fixed the
problem. If this is not the case, please contact our support. We thank you
for your patience.
-idzam
Date: 18 Jan 98 at 19:19 US/Eastern
Status: resolved
Subject: Columbia, SC Busy (803)252-9191
We found that some of our circuits had been prematurely disconnected and our
telco has corrected this problem. Please begin using the new dial up number
(803)217-6774. If you need help doing this, please call or email support. We
will be keeping a close eye on this POP tonight.
Date: 18 Jan 98 at 14:56 US/Eastern
Status: resolved
Subject: Busies and stuff in Louisville
We have overcome the meanies in Louisville. All is running properly. Sorry
for the interruption.
Date: 18 Jan 98 at 14:38 US/Eastern
Status: resolved
Subject: Utica NY busies
Telco fixed a problem with a circuit here which alleviated the busies. Let
us know if you see more troubles here.
Date: 18 Jan 98 at 13:51 US/Eastern
Status: resolved
Subject: MSPG:Southern California Maintanence 1/16 8-10am pst
This maintenance was not carried out today due to some technical difficulties.
We will have to reschedule this work on a different date. Thank you.
-idzam
Date: 18 Jan 98 at 13:48 US/Eastern
Status: resolved
Subject: Tuscon, AZ ISDN (520)620-6152 Down
Tuscon is back up now thanks to a helpful telco account rep.
Date: 18 Jan 98 at 08:19 US/Eastern
Status: resolved
Subject: MSPG: Nashville TN Auth fail/no carrier (possible)
We have seen no further instability with the equipment we believe to
have been causing the problems we saw here. We will continue to
monitor. Please let us know if you see further trouble here.
Date: 18 Jan 98 at 05:50 US/Eastern
Status: resolved
Subject: Sporadic authentication failures/throughput pauses 1/17 10pm - 1/18 6am
This maintenance is completed.
Date: 17 Jan 98 at 18:10 US/Eastern
Status: resolved
Subject: Tampa FL isdn packet loss
Packet loss should no longer be occurring in this area. PSI installed
additional longline capacity. We have not seen packet loss since then.
Date: 17 Jan 98 at 11:02 US/Eastern
Status: resolved
Subject: Albuquerque isdn down
PSI dispatched field service and they restored power to the switch. All
should be operating well now.
Date: 17 Jan 98 at 09:08 US/Eastern
Status: resolved
Subject: Atlanta, GA 404-287-3038 Drops 0045 1/17
The switch maintenance should be over. All circuits were returned
to service and have remained stable for some time now.
Date: 17 Jan 98 at 09:08 US/Eastern
Status: resolved
Subject: New York, New York (212)616-1038 (212)428-2080 (212)428-2040
These are back up. We will be monitoring this closely. Sorry for the
inconvenience.
Date: 16 Jan 98 at 19:07 US/Eastern
Status: resolved
Subject: Commercial web hosting control panel emergency downtime
This maintenance is done. We thank you for your patience.
-idzam
Date: 16 Jan 98 at 14:44 US/Eastern
Status: resolved
Subject: New Haven CT isdn down
Power was restored to the facility about ten minutes ago. Please
let us know if you see further troubles here.
Date: 16 Jan 98 at 10:41 US/Eastern
Status: resolved
Subject: Memphis ISDN outage
Power was restored to this PoP.
Date: 16 Jan 98 at 07:37 US/Eastern
Status: resolved
Subject: Grid: Emergency Maintanence: Pennsylvania, Maryland 6:30am - 7:30am
This maintenance was completed.
Date: 16 Jan 98 at 06:43 US/Eastern
Status: resolved
Subject: Santa Anna, CA - 01/15/98 - 04:00 - 04:15 EST
The window for this maintenace has expired. Please let us know if you
see any further problems.
Date: 16 Jan 98 at 06:41 US/Eastern
Status: resolved
Subject: Bel Air, Maryland 1/15/98 00:00-00:15 EST
The maintenance window is over for this upgrade. Please let us know if
you see further problems.
Date: 16 Jan 98 at 06:40 US/Eastern
Status: resolved
Subject: GRID:Washington DC 202-289-0215 Packet Loss/Peak Hours
Gridnet has completed their backbone upgrade, and we
saw no packet loss tonight. Please let us know if you
see any more packet loss here.
-rick
Date: 16 Jan 98 at 06:39 US/Eastern
Status: resolved
Subject: Grid 301-359-3011 MD_WESTERNPORT failing PPP connections (fixed)
We received and replicated complaints of failing connections
at this site. We opened a ticket with Gridnet, and they fixed
the problem a short time ago. I appologize for the after the
fact post.
-rick
Date: 16 Jan 98 at 06:14 US/Eastern
Status: resolved
Subject: Gainesville FL 1/15/98 0500-0700 RESCHEDULED
this has been completed.
-rick
Date: 16 Jan 98 at 05:45 US/Eastern
Status: resolved
Subject: NC Triangle 1/16/98 0400-0600
This maintanence has been completed successufully.
-rick
Date: 16 Jan 98 at 05:42 US/Eastern
Status: resolved
Subject: Canceled:Atlanta dialups, California, NYC, Harrisburg, nf-vb, Richmond, Charlottesville 0500-0600 15 Jan
We'll let you know when this will be done.
Date: 16 Jan 98 at 05:41 US/Eastern
Status: resolved
Subject: MSPG: RTP , NC Intermittant Authentication Fail /Loss of ip connectivity
All this work is complete. All equipment is fixed and
operating normally.
-rick
Date: 16 Jan 98 at 00:43 US/Eastern
Status: resolved
Subject: MSPG - RTP, NC Giving Busies
We've added additional circuits here. We should no longer be giving busies
as a result of exceding capacity.
-rick
Date: 16 Jan 98 at 00:40 US/Eastern
Status: resolved
Subject: Slow throughput in Atlanta, Greenville, Alabama, South Carolina
We had to reset our equipment here, and have not seen any more
problems since appx 8pm. We'll continue to monitor, please let
us know if you see any more problems.
-rick
Date: 16 Jan 98 at 00:39 US/Eastern
Status: resolved
Subject: Grid: 509-835-0000 WA_SPOKANE Regular Busies at less than capacity(fixed)
This POP has been restored to capacity. It had been working at less than
50%. Now that all modems have been restored, any busies experienced this
evening will be due to having reached capacity.
-rick
Date: 16 Jan 98 at 00:38 US/Eastern
Status: resolved
Subject: Boston, MA - 01/20/98 - 00:00 - 00:15 EST
Dial-up servers in Boston will be down for a few minutes as modems are upgraded.
Date: 15 Jan 98 at 18:15 US/Eastern
Status: resolved
Subject: Atlanta Brief Busies/Drops
This should be resolved. Please let support know if you have any
further problems.
Cheers,
Brian
Date: 15 Jan 98 at 17:46 US/Eastern
Status: resolved
Subject: Mail server and MOTD timeouts
Things are back to normal...
-SC
Date: 15 Jan 98 at 15:15 US/Eastern
Status: resolved
Subject: Internet connection partially down...
This was apparently just a temporary, albeit highly unusual, curcuit
bounce. We're pursuing the matter with the telco to ensure that this
does not happen again, especially at a less-auspicious time. Thanks
for your understanding, as always.
Cheers,
Brian
Date: 15 Jan 98 at 09:44 US/Eastern
Status: resolved
Subject: Harrisburg, PA 15 January 0500-0600
This maintenance has been completed. Thanks for your patience.
Date: 15 Jan 98 at 07:32 US/Eastern
Status: resolved
Subject: Multiple Gridnet Authentication Failures
These all look to be fixed. I'm going to investigate the cause of this
outage now. Thanks for your patience.
Cheers,
Brian
Date: 15 Jan 98 at 06:03 US/Eastern
Status: resolved
Subject: Thursday Jan 15 0500-0630 Knoxville
This maintanence has been completed, let us know if you see
any problems here.
-rick
Date: 14 Jan 98 at 23:59 US/Eastern
Status: resolved
Subject: UPDATE: MSPG - Atlanta busy signals and telco messages
We've not seen any more trunking problems since the last post.
Our vendor has taken action to decrease the possibility that
this problem will occur in the future. Other problems described
in posts in the last 2 days have been unrelated to interoffice
trunking.
-rick
Date: 14 Jan 98 at 23:57 US/Eastern
Status: resolved
Subject: MSPG:RTP, NC Partial loss of ip connectivity
This was restored approximately 1 hour after the original
post, sorry for the late update. Busies here persisted as
we reached capacity. Checking on install of additional
capacity.
Date: 14 Jan 98 at 23:56 US/Eastern
Status: resolved
Subject: Asheville, NC 704 285 2900 14 January 0200-0700
This maintanence has been completed, please let us know
if you experience any problems here.
-rick
Date: 14 Jan 98 at 23:51 US/Eastern
Status: resolved
Subject: MSPG:Atlanta, GA 404-287-3038 Busies at less than capacity
We've seen no problems here since the last post.
-rick
Date: 14 Jan 98 at 17:50 US/Eastern
Status: resolved
Subject: News local posting delay
Local posts to our news server were delayed for approximately 45 minutes.
Articles posted during this time were propogated to the outside world
immediately. Anything posted during this time should show up in our
local server soon. Sorry for the delays, and happy news reading.
Cheers,
Brian
Date: 14 Jan 98 at 07:29 US/Eastern
Status: resolved
Subject: Commercial web hosting - EMERGENCY downtime jan 14 0500-0600 est
This has been completed. Thanks for your patience.
Cheers,
Brian
Date: 14 Jan 98 at 07:17 US/Eastern
Status: resolved
Subject: Atlanta, Montgomery, Triangle, Florida 1/14/98 0500-0800
This maintenance has been completed. Let us know if you run into
any problems that might be related to this maintenance.
Cheers,
Brian
Date: 14 Jan 98 at 05:23 US/Eastern
Status: resolved
Subject: Southern California Possible Drops
this work was completed.
-rick
Date: 13 Jan 98 at 16:59 US/Eastern
Status: resolved
Subject: Dropped connections on Atlanta 404.654.1300
This seems to be back up and operational with no further problems. We're
continuing to watch this very closely to figure out what happened and
to make sure there are no unpleasant aftereffects. Please let support
know if you run into any difficulties on the Atlanta hunts.
Cheers,
Brian
Date: 13 Jan 98 at 09:30 US/Eastern
Status: resolved
Subject: Emergency maintenance for NYC, LA, Stockton, Harrisburg & All VA PoP's @ 3:30am Eastern
This maintenance has been postponed. We'll let you know when it gets
rescheduled.
Cheers,
Brian
Date: 13 Jan 98 at 09:09 US/Eastern
Status: resolved
Subject: Atlanta, GA 404 654 1300 Occational Telco Recordings
These recordings have been eliminated and customers should no longer
be getting them dialing into 1300. Please let us know if you run into
any further problems.
Cheers,
Brian
Date: 13 Jan 98 at 08:14 US/Eastern
Status: resolved
Subject: Atlanta GA Circuit Maintenance Tuesday 1/13/97 0500-0600 EST
This maintenance was completed this morning. Thanks for your patience.
Cheers,
Brian
Date: 13 Jan 98 at 06:34 US/Eastern
Status: resolved
Subject: News problems
There are still some problems with one of our news file servers. In
particular while reading news groups with very high numbers of articles
and/or very large files (alt.binaries) you may experience slower than
normal response times. We have ruled out all of the obvious candidates
for causes in the servers and the network and are working with our file
server vendor to resolve this issue. Watch for a news maintenance alert
in your favorite mindspring.alerts group soon.
Again, sorry for the inconvenience. We are working on it.
+--+
|oo| The Unknown News Administrator
| | jisley@eng.mindspring.net
`^^'
Date: 13 Jan 98 at 00:37 US/Eastern
Status: resolved
Subject: PSI Indianapolis, IN, Cincinatti, OH, and Dayton, OH Down
The fiber cut has been repaired by the telco. All affected sites
are operational. Thanks for your patience.
-rick
Date: 12 Jan 98 at 18:55 US/Eastern
Status: resolved
Subject: MSPG:Atlanta, GA All access numbers: busies at less than capacity
We have not seen any further Atlanta busy problems. Please let support
know if you begin to experience any problems of that nature in Atlanta.
Cheers,
Brian
Date: 12 Jan 98 at 18:38 US/Eastern
Status: resolved
Subject: MSPG 813-354-0077 FL_Tampa Fast Busies
We believe the busies have been eradicated; we're still watching closely.
-SC
Date: 12 Jan 98 at 17:56 US/Eastern
Status: resolved
Subject: Gainesville, FL Thursday, 15 January 0500-0700 EST
We're replacing some equipment in Gaineville, FL on Thursday
morning, necessitating a 2 hour maintenance window. Downtime
should be minimal. This maintenance will disconnect any customers
online at the time it begins.
Cheers,
Brian
Date: 12 Jan 98 at 16:39 US/Eastern
Status: resolved
Subject: PSI DNS 38.9.215.2 38.9.224.2 Not Responding
These have been back up and stable for a few hours now. They should
work fine again. Thanks for your understanding.
Cheers,
Brian