Drive Failure on 8 Mar 2026 18:25:07 (UTC-04:00)
Resolved on 9 Mar 2026 20:41:02 (UTC-04:00)
Redbuild is completed. All monitoring is green.
Redbuild is completed. All monitoring is green.
In progress on 9 Mar 2026 18:05:54 (UTC-04:00)
The drive is replaced. RAID rebuild in progress. Servers are online, but disk speeds will be slow until the data redundancy is rebuilt.
The drive is replaced. RAID rebuild in progress. Servers are online, but disk speeds will be slow until the data redundancy is rebuilt.
In progress on 9 Mar 2026 16:39:49 (UTC-04:00)
Drive replacement has started. All customer servers will be offline while the drive is replaced.
Drive replacement has started. All customer servers will be offline while the drive is replaced.
In progress on 9 Mar 2026 12:37:37 (UTC-04:00)
Downtime window scheduled for drive replacement:
When: 3/9/2026 4:30 PM EDT - 6:30 Pm EDT
What: Cragfury
Why: Replace the failed data drive
Downtime window scheduled for drive replacement:
When: 3/9/2026 4:30 PM EDT - 6:30 Pm EDT
What: Cragfury
Why: Replace the failed data drive
Identified on 8 Mar 2026 18:25:07 (UTC-04:00)
A drive in the data RAID array has failed on Cragfury. We are working to schedule a repair window with the data center technicians. The customers servers and data will be offline during the repair.
A drive in the data RAID array has failed on Cragfury. We are working to schedule a repair window with the data center technicians. The customers servers and data will be offline during the repair.
Stonehammer Offline on 7 Feb 2026 08:33:49 (UTC-04:00)
Resolved on 7 Feb 2026 09:00:33 (UTC-04:00)
Stonehammer services have been restored.
Stonehammer services have been restored.
Investigating on 7 Feb 2026 08:33:49 (UTC-04:00)
Stonehammer went offline. The data center team is investigating a possible hardware failure.
Stonehammer went offline. The data center team is investigating a possible hardware failure.
Node Offline on 20 Jan 2026 22:08:23 (UTC-04:00)
Resolved on 20 Jan 2026 22:20:43 (UTC-04:00)
8:43 Customers have reported issues connecting to the node
9:03 Support staff are unable to access the server. Datacenter technicians are on the way to investigate.
9:06 The system will be manually power-cycled.
9:19 Issue Resolved
8:43 Customers have reported issues connecting to the node
9:03 Support staff are unable to access the server. Datacenter technicians are on the way to investigate.
9:06 The system will be manually power-cycled.
9:19 Issue Resolved
Update on 20 Jan 2026 22:11:22 (UTC-04:00)
8:43 Customers have reported issues connecting to the node
9:03 Support staff are unable to access the server. Datacenter technicians are on the way to investigate.
9:06 The system will be manually power-cycled.
8:43 Customers have reported issues connecting to the node
9:03 Support staff are unable to access the server. Datacenter technicians are on the way to investigate.
9:06 The system will be manually power-cycled.
Investigating on 20 Jan 2026 22:08:23 (UTC-04:00)
8:43 Customers have reported issues connecting to the node
9:03 Support staff are unable to access the server. Datacenter technicians are on the way to investigate.
8:43 Customers have reported issues connecting to the node
9:03 Support staff are unable to access the server. Datacenter technicians are on the way to investigate.
Datacenter Dallas Texas on 28 Dec 2025 20:21:46 (UTC-04:00)
Resolved on 28 Dec 2025 21:57:11 (UTC-04:00)
All services are restored.
All services are restored.
In progress on 28 Dec 2025 20:55:13 (UTC-04:00)
The facility has reached approximately 80* F, we will begin slowly powering services back up.
The facility has reached approximately 80* F, we will begin slowly powering services back up.
In progress on 28 Dec 2025 20:28:12 (UTC-04:00)
The facility is cooling off, but it is still too warm to begin powering equipment back up. We estimate approximately 15 more minutes until the facility reaches acceptable temperatures to begin powering equipment back up.
The facility is cooling off, but it is still too warm to begin powering equipment back up. We estimate approximately 15 more minutes until the facility reaches acceptable temperatures to begin powering equipment back up.
Identified on 28 Dec 2025 20:23:17 (UTC-04:00)
Facilities personnel have arrived at the datacenter, and CRAC units and additional rooftop units are being re-energized. It will take approximately 30 minutes to cool the facility to acceptable levels for servers to power up without damage.
Facilities personnel have arrived at the datacenter, and CRAC units and additional rooftop units are being re-energized. It will take approximately 30 minutes to cool the facility to acceptable levels for servers to power up without damage.
Investigating on 28 Dec 2025 20:22:37 (UTC-04:00)
We are seeing high temperatures across many devices that are still online. We have notified our provider
We are seeing high temperatures across many devices that are still online. We have notified our provider
Investigating on 28 Dec 2025 20:22:11 (UTC-04:00)
We are seeing high temperatures on core network devices; we are continuing to investigate.
We are seeing high temperatures on core network devices; we are continuing to investigate.
Investigating on 28 Dec 2025 20:21:46 (UTC-04:00)
At approximately 15:06 Pacific time, our NOC began to see servers going offline.
At approximately 15:06 Pacific time, our NOC began to see servers going offline.
Networking Connectivity Issue on 1 Nov 2025 19:37:50 (UTC-04:00)
Resolved on 2 Nov 2025 03:59:50 (UTC-04:00)
Due to our team having to restore customer data from Disaster backups, there will be some data loss. This is not the service we want to deliver, and we apologize immediately for this outage. We are currently completing a Root Cause Analysis for this incident and will create a list of items to prevent future outages like this and to make our infrastructure more robust.
More details to come soon.
If you have any questions or need help, please open a ticket and our team will respond as soon as possible.
All Services Restored
Due to our team having to restore customer data from Disaster backups, there will be some data loss. This is not the service we want to deliver, and we apologize immediately for this outage. We are currently completing a Root Cause Analysis for this incident and will create a list of items to prevent future outages like this and to make our infrastructure more robust.
More details to come soon.
If you have any questions or need help, please open a ticket and our team will respond as soon as possible.
Update on 2 Nov 2025 03:46:05 (UTC-04:00)
Cragfury - Restored
FR2 - Restored
Dugholim - Restored
DeepBrewer - Restored
Boulderbeard -Restored
Mountainfury - Restored
All customer servers automatically booted.
2:43 Update
FR1 - RestoredCragfury - Restored
FR2 - Restored
Dugholim - Restored
DeepBrewer - Restored
Boulderbeard -Restored
Mountainfury - Restored
All customer servers automatically booted.
Update on 2 Nov 2025 01:53:42 (UTC-04:00)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending (Hardware fault)
Mountainfury - Restored (Manually boot your servers)
1:53 Update
FR1 - Restored (Manually boot your servers)Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending (Hardware fault)
Mountainfury - Restored (Manually boot your servers)
Update on 2 Nov 2025 01:36:04 (UTC-04:00)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard - In Progress
Mountainfury - Restored (Manually boot your servers)
1:35 Update
FR1 - Restored (Manually boot your servers)Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard - In Progress
Mountainfury - Restored (Manually boot your servers)
Update on 2 Nov 2025 00:46:26 (UTC-04:00)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - In Progress
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending
Mountainfury - In Progress
12:45 Update
FR1 - Restored (Manually boot your servers)Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - In Progress
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending
Mountainfury - In Progress
Update on 2 Nov 2025 00:14:31 (UTC-04:00)
Cragfury - In progress
FR2 - Restored (Manually boot your servers)
Dugholim - Pending
DeepBrewer - In Progress
Boulderbeard -Pending
Mountainfury - Pending
12:15 Update
FR1 - Restored (Manually boot your servers)Cragfury - In progress
FR2 - Restored (Manually boot your servers)
Dugholim - Pending
DeepBrewer - In Progress
Boulderbeard -Pending
Mountainfury - Pending
Update on 1 Nov 2025 23:54:15 (UTC-04:00)
FR1 - Done
Cragfury - In progress
FR2 - In Progress
Dugholim - on deck
DeepBrewer - In Progress
Boulderbeard -on deck
Mountainfury - on deck
Restorations in Progress
FR1 - Done
Cragfury - In progress
FR2 - In Progress
Dugholim - on deck
DeepBrewer - In Progress
Boulderbeard -on deck
Mountainfury - on deck
Update on 1 Nov 2025 21:56:07 (UTC-04:00)
For any node still offline, we are failing over to our DR plan and starting restores of customer data from our last disaster recovery backup. Some data loss up 6 hours may occur.
For any node still offline, we are failing over to our DR plan and starting restores of customer data from our last disaster recovery backup. Some data loss up 6 hours may occur.
In progress on 1 Nov 2025 21:33:49 (UTC-04:00)
Half of the nodes are restored. We are working to repair or enact our DR plan for the other nodes.
Half of the nodes are restored. We are working to repair or enact our DR plan for the other nodes.
In progress on 1 Nov 2025 20:14:23 (UTC-04:00)
Fix has been deployed. Waiting on all nodes to reboot.
Fix has been deployed. Waiting on all nodes to reboot.
Identified on 1 Nov 2025 19:52:34 (UTC-04:00)
We have identified the network misconfiguration. We are fixing it on each node now.
We have identified the network misconfiguration. We are fixing it on each node now.
Investigating on 1 Nov 2025 19:37:50 (UTC-04:00)
We are aware of a network connection issue and are working to resolve it ASAP.
We are aware of a network connection issue and are working to resolve it ASAP.
DDOS Attack Virginia + Dallas on 30 Oct 2025 16:10:40 (UTC-04:00)
Resolved on 31 Oct 2025 06:20:12 (UTC-04:00)
No further attacks occurred overnight. Closing this incident.
No further attacks occurred overnight. Closing this incident.
Monitoring on 30 Oct 2025 19:36:31 (UTC-04:00)
All services and IPs should now be functioning normally. We are continuing to monitor and will close this in the morning EDT if no further attacks arise.
All services and IPs should now be functioning normally. We are continuing to monitor and will close this in the morning EDT if no further attacks arise.
Update on 30 Oct 2025 18:19:17 (UTC-04:00)
Other major transit providers are having possibly related issues even after the attack on our data center has subsided. These are causing outages as traffic is rerouted.
We are still monitoring and mitigating attacks as they come in
Other major transit providers are having possibly related issues even after the attack on our data center has subsided. These are causing outages as traffic is rerouted.
We are still monitoring and mitigating attacks as they come in
Monitoring on 30 Oct 2025 17:36:59 (UTC-04:00)
We have successfully mitigated the attack for now. We are monitoring and will respond as needed.
We have successfully mitigated the attack for now. We are monitoring and will respond as needed.
Investigating on 30 Oct 2025 16:10:40 (UTC-04:00)
At approximately 3:37 PM Eastern time, our NOC detected a carpet bomb style DDOS attack aimed at both our Virginia and Ashburn locations. This is creating short bursts of packet loss.
There is currently no ETTR, as this attack targets multiple data centers in each region rather than our services specifically.
At approximately 3:37 PM Eastern time, our NOC detected a carpet bomb style DDOS attack aimed at both our Virginia and Ashburn locations. This is creating short bursts of packet loss.
There is currently no ETTR, as this attack targets multiple data centers in each region rather than our services specifically.
US East and Central DDoS Attacks on 26 Oct 2025 22:29:53 (UTC-04:00)
Resolved on 28 Oct 2025 07:11:18 (UTC-04:00)
Mitigations have been deployed and no attack has made it through in the past 12 hours.
Mitigations have been deployed and no attack has made it through in the past 12 hours.
Update on 27 Oct 2025 07:32:32 (UTC-04:00)
The data center provider has made progress in stabilizing some regions overnight; the attack remains active and continues to shift patterns. This attack is not just targeting them but all providers in each area.
Starting this morning, a full-scale mitigation plan is being put in place to filter the networks using Cloudflare's Magic Transit. There is no ETA on complete restoration, but stability is improving.
The data center provider has made progress in stabilizing some regions overnight; the attack remains active and continues to shift patterns. This attack is not just targeting them but all providers in each area.
Starting this morning, a full-scale mitigation plan is being put in place to filter the networks using Cloudflare's Magic Transit. There is no ETA on complete restoration, but stability is improving.
In progress on 26 Oct 2025 22:29:53 (UTC-04:00)
Our Virginia and Texas data center providers are experiencing multiple short-duration carpet-bomb DDoS attacks. We are working to mitigate this ASAP.
Our Virginia and Texas data center providers are experiencing multiple short-duration carpet-bomb DDoS attacks. We are working to mitigate this ASAP.