Dear Customers,
At roughly 6:20am EST the facility where LANKAHOST operates its some of the servers (Prime, formerly Carrier-1 and Evocative) lost utility power. Redundant power sources, UPS and generators, did not operate as designed thus causing the entire facility to lose power. Prime Data Centers has electrical engineers on site at this time working to solve the problem. Hivelocity has technicians on site standing by and ready to get all impacted servers back online the moment power is restored to the facility.
The following servers are impacted by this outage
ultra10
win10
resllerhosting
Also, a number of VPS and Dedicated Servers managed by LankaHost
------------------------
Please find below update from data center on this
~~~~
Dear Customer,At approximately 6:17am CST our NOC began to see alarms in our Dallas, TX POP. As we are still investigating the issue, we do not have a ETA for resolution yet, but hopefully it will be very soon. Meanwhile, your patience is appreciated. We will post updates shortly.
UPDATE: 6:35am CST: Upon further investigation, we suspect the entire facility at Carrier-1/Prime Datacenters in Dallas has gone dark. It is likely power related. We are awaiting updates from onsite.
UPDATE: 7:25AM CST: We have received confirmation from the facility that this is a power outage. They have confirmed that there is no physical damage to the facility from the storm in the area. The generators are running, however the UPS are not receiving any power from the generators. Facilities and generator contractors are enroute with an ETA of 30-45 minutes.
UPDATE: 9:02AM CST: Power has been restored to our POD and office in Dallas. Networking is back up and most servers are back online. If you still have servers or services down, please post a support ticket and we investigate with urgency.
UPDATE: 11AM CST: We are still working on scattered issues related to the earlier outage. Servers in cabinets --- and --- are up but without proper networking. We are working urgently on this as well as remaining hosts that have been reported to us.
A full post mortem will be posted here and as an announcement once we have gathered all of the facts. The underlying issue was that despite redundant power systems that were online and functional, the facility may have had some type of malfunction with the automated transfer system (ATS). Powerful storms and tornados in Texas overnight caused deaths and destruction in the area but the facility should have remained unaffected even with extended loss of utility power, as it has before. These systems are indeed routinely tested. Please have no doubt that our team will ensure that all necessary investigations, and ultimately fixes, will be employed to avoid even the slightest chance of a repeat in the future. We appreciate your patience and understanding.
Thanks for your understanding
