NOC System Status

Router Maintenance / Seattle DC4

May 08, 2013 at 14:09

START DATE: 05/12/2013

START TIME: 20:00 PDT

END TIME: 23:59 PDT

SERVICES/EQUIPMENT: cr-0-1.dc4.sea / cr-0-2.dc4.sea

TYPE OF WORK:                    Router Upgrades

PURPOSE OF WORK:            Stability, security, and performance improvements

IMPACT OF WORK:                Routing reconvergence

We will be upgrading software versions for core routers cr-0-1 and cr-0-2 in Seattle DC4.  During these upgrades we will transition preferred paths between core routers, providing for resilient network services throughout this maintenance window.

This NOC post will be updated during the maintenance window.  If you have any questions about this event, please contact our support department via support@networkredux.com.

Update 23:49 PDT: Mainantence upgrades which required routing reconvergence have been completed, we are still working on some configuration changes to these routers and will be extending the window until 02:00.

Update 12:20 PDT: This maintenance window is completed.

Seattle Network Connectivity

April 23, 2013 at 08:55

We are currently investigating network connectivity issues in our Seattle facility.  We will keep this thread updated.

Update 9:05AM PDT:  All services are restored, we are beginning to collect data for a root cause analysis now.  This even impacted clients with Juniper firewalls in primary/secondary mode, all other services were operational during this event.

Final Update:  Preliminary root cause analysis was sent via support tickets to impacted clients.  Maintenance will be scheduled to address the items which caused this failure.

Newyork Maintenance

March 21, 2013 at 00:00

During scheduled maintenance the legacy shard environment newyork.networkphantom.net has experienced incompatibility issues with legacy components.  Our engineers are quickly working toward resolution and will continue to keep this post updated until core services have been restored.

Update 3:05PM PST: While engineers continue working toward resolution on fixing any incompatibility issues we are also actively replicating accounts to a new device for a timely resolution in whichever process is completed first.

Update 2:00PM PST Mar-22: All matters on Newyork have been resolved and core services responsive 1:25AM PST.

Oriskany DNS Failures

March 17, 2013 at 10:38

The Oriskany Reseller server has been experiencing DNS resolution failures and our operations team is attempting to resolve this.  We will continue to work on this problem until resolution.  Should you have any questions please contact support@networkredux.com -- Thank you.

Update 10:00 March 18:  All matters are now resolved for the Oriskany reseller server.

vz003.dc4.sea

February 22, 2013 at 09:53

The hardware node vz003.dc4.sea has been experiencing issues this morning and we are planning to migrate its data to a standby chassis.  We will keep this thread updated as matters progress.

Update 12:14PM PST:  Clients on the legacy vz003.dc4.sea Virtuozzo server are still impacted by this event.  We are in the process of beginning a restoration process to other viable virtuozzo nodes.

Update 3:18PM PST:  We continue to migrate clients off of this hardware, there is no current ETA on completion however we are moving client VPS nodes as quickly as possible and greatly appreciate your patience if you reside on vz003.dc4.sea.

Update 6:03PM PST:  We continue to migrate containers off of this failed chassis, the underlying problem is related to how the virtuozzo kernel is reading the ethernet firmware.  Our vendor, Parallels, has been investigating this as we have been moving clients off of the failed system.  We are also encountering some issues with the client data moves which we are troubleshooting.  We again appreciate your patience if you have a VPS on this server, we have allocated resources to assist with this problem until resolution.

Update 9:00PM PST:  We continue migrations of client data and environments, some which are requiring extensive repair to bring online.  We will continue this work until completion, clients have already begun to see their VPS servers return.  Thank you for your patience.