Network Service Providers (Seattle)

February 05, 2013 at 13:40

At approximately 13:00 PST we began to see network saturation on 2 of our 10 upstream connections into of our Seattle market.   During this time, customers may have experienced sub-optimal routing to various destinations in our Seattle network mesh.  At this time, the issue has been mitigated.

We are currently monitoring the issue and will update customers if there are any further issues.

xen004.cl1.dc4

January 26, 2013 at 18:47

This member of cluster 1 has experienced a kernel panic related to stability issues we have a maintenance window to address this coming week:

https://www.networkredux.com/blog/view/1400/planned-evs-cluster-1-maintenance

The member is back online, virtual servers that we're pinned to this member have been booting for the last 10 minutes on the cluster.

Planned EVS Cluster 1 Maintenance (Complete)

January 24, 2013 at 00:00

This is notification of a maintenance window for EVS customers on Cluster 1 in Seattle.

 
START DATE:     01/29/2013
START TIME:      22:00 PST
END TIME:           01/30/2013 04:00 PST
 
SERVICES/EQUIPMENT:     xen.cl1.dc4.sea.networkredux.net
TYPE OF WORK:                   Cluster maintenance
PURPOSE OF WORK:           Stability, security, and performance improvements
IMPACT OF WORK:               Ubuntu 10.04 LTS VMs will require graceful shutdowns
 
We will be upgrading all servers in EVS Cluster 1 to strengthen cluster stability, add performance improvements as well as apply an important update related to xenmotion/livemotion of Ubuntu 10.04 LTS servers
 
We will be performing the procedure on one physical member of the cluster at a time.  CentOS clients should not experience service disruption during the event while they are xenmotion/livemotion'd across other members.
 
Ubuntu 10.04 clients on this cluster will experience graceful shutdowns during this event to handle migrating the Ubuntu servers as well as updating XenServer kernel drivers to repair the ongoing issues with stable xentmotion/livemotion.
 
 
This NOC post will be updated during the maintenance.
 
UPDATE 22:23 PST: This maintenance is now in progress.
 
UPDATE 3:43am PST: This maintenance is now complete.

Adding new member to xen.cl2.dc4.sea.networkredux.net

January 22, 2013 at 14:19

 

START DATE:           01/22/2013
START TIME:            16:30PM PST

SERVICES/EQUIPMENT:    xen.cl2.dc4.sea.networkredux.net
TYPE OF WORK:                New member addition
PURPOSE OF WORK:        Capacity
IMPACT OF WORK:            None expected

We will be adding a new hardware node member to this cluster to expand capacity and performance.  This work is expected to be non-invasive.  We will update this thread when the work is completed.

Update 4:51PM PST:  This work is completed, no service disruptions occurred.

xen001.cl1 (resolved)

January 18, 2013 at 15:31

We are experiencing issues with some EVS customers on cluster 1 and are evaluating the situation, currently only a small segment of clients are impacted on this cluster.  We will update this thread as we have more information.

Update 4:15PM PST:  EVS' impacted by this have been restored.  We suspect a memory module in a cluster member is to blame and will continue to investigate this matter.  Thank you for your patience, if you have further questions please contact our support department.

Update 4:20PM PST:  Evidence suggests that we are seeing a similar pattern in the storage link driver for Citrix XenServer.  We will be scheduling this cluster for patches and will notify clients when this will take place.

Older Status Updates Newer Status Updates

Blog Categories

Recent Posts

Stay Up To Date