Resolved on 2 Nov 2025 02:59:50 (UTC-05:00)

All Services Restored

Due to our team having to restore customer data from Disaster backups, there will be some data loss. This is not the service we want to deliver, and we apologize immediately for this outage. We are currently completing a Root Cause Analysis for this incident and will create a list of items to prevent future outages like this and to make our infrastructure more robust.

More details to come soon.

If you have any questions or need help, please open a ticket and our team will respond as soon as possible.

Update on 2 Nov 2025 02:46:05 (UTC-05:00)

2:43 Update

FR1 - Restored
Cragfury - Restored
FR2 - Restored
Dugholim - Restored
DeepBrewer - Restored
Boulderbeard -Restored
Mountainfury - Restored

All customer servers automatically booted.

Update on 2 Nov 2025 00:53:42 (UTC-05:00)

1:53 Update

FR1 - Restored (Manually boot your servers)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending (Hardware fault)
Mountainfury - Restored (Manually boot your servers)

Update on 2 Nov 2025 00:36:04 (UTC-05:00)

1:35 Update

FR1 - Restored (Manually boot your servers)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard - In Progress
Mountainfury - Restored (Manually boot your servers)

Update on 1 Nov 2025 23:46:26 (UTC-05:00)

12:45 Update

FR1 - Restored (Manually boot your servers)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - In Progress
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending
Mountainfury - In Progress

Update on 1 Nov 2025 23:14:31 (UTC-05:00)

12:15 Update

FR1 - Restored (Manually boot your servers)
Cragfury - In progress
FR2 - Restored (Manually boot your servers)
Dugholim - Pending
DeepBrewer - In Progress
Boulderbeard -Pending
Mountainfury - Pending

Update on 1 Nov 2025 22:54:15 (UTC-05:00)

Restorations in Progress

FR1 - Done
Cragfury - In progress
FR2 - In Progress
Dugholim - on deck
DeepBrewer - In Progress
Boulderbeard -on deck
Mountainfury - on deck

Update on 1 Nov 2025 20:56:07 (UTC-05:00)
For any node still offline, we are failing over to our DR plan and starting restores of customer data from our last disaster recovery backup. Some data loss up 6 hours may occur.

In progress on 1 Nov 2025 20:33:49 (UTC-05:00)
Half of the nodes are restored. We are working to repair or enact our DR plan for the other nodes.

In progress on 1 Nov 2025 19:14:23 (UTC-05:00)
Fix has been deployed. Waiting on all nodes to reboot.

Identified on 1 Nov 2025 18:52:34 (UTC-05:00)
We have identified the network misconfiguration. We are fixing it on each node now.

Investigating on 1 Nov 2025 18:37:50 (UTC-05:00)
We are aware of a network connection issue and are working to resolve it ASAP.