AleForge Status
Networking Connectivity Issue on 1 Nov 2025 18:37:50 (UTC-05:00)
Resolved on 2 Nov 2025 02:59:50 (UTC-05:00)
Due to our team having to restore customer data from Disaster backups, there will be some data loss. This is not the service we want to deliver, and we apologize immediately for this outage. We are currently completing a Root Cause Analysis for this incident and will create a list of items to prevent future outages like this and to make our infrastructure more robust.
More details to come soon.
If you have any questions or need help, please open a ticket and our team will respond as soon as possible.
All Services Restored
Due to our team having to restore customer data from Disaster backups, there will be some data loss. This is not the service we want to deliver, and we apologize immediately for this outage. We are currently completing a Root Cause Analysis for this incident and will create a list of items to prevent future outages like this and to make our infrastructure more robust.
More details to come soon.
If you have any questions or need help, please open a ticket and our team will respond as soon as possible.
Update on 2 Nov 2025 02:46:05 (UTC-05:00)
Cragfury - Restored
FR2 - Restored
Dugholim - Restored
DeepBrewer - Restored
Boulderbeard -Restored
Mountainfury - Restored
All customer servers automatically booted.
2:43 Update
FR1 - RestoredCragfury - Restored
FR2 - Restored
Dugholim - Restored
DeepBrewer - Restored
Boulderbeard -Restored
Mountainfury - Restored
All customer servers automatically booted.
Update on 2 Nov 2025 00:53:42 (UTC-05:00)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending (Hardware fault)
Mountainfury - Restored (Manually boot your servers)
1:53 Update
FR1 - Restored (Manually boot your servers)Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending (Hardware fault)
Mountainfury - Restored (Manually boot your servers)
Update on 2 Nov 2025 00:36:04 (UTC-05:00)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard - In Progress
Mountainfury - Restored (Manually boot your servers)
1:35 Update
FR1 - Restored (Manually boot your servers)Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - Restored (Manually boot your servers)
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard - In Progress
Mountainfury - Restored (Manually boot your servers)
Update on 1 Nov 2025 23:46:26 (UTC-05:00)
Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - In Progress
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending
Mountainfury - In Progress
12:45 Update
FR1 - Restored (Manually boot your servers)Cragfury - Restored (Manually boot your servers)
FR2 - Restored (Manually boot your servers)
Dugholim - In Progress
DeepBrewer - Restored (Manually boot your servers)
Boulderbeard -Pending
Mountainfury - In Progress
Update on 1 Nov 2025 23:14:31 (UTC-05:00)
Cragfury - In progress
FR2 - Restored (Manually boot your servers)
Dugholim - Pending
DeepBrewer - In Progress
Boulderbeard -Pending
Mountainfury - Pending
12:15 Update
FR1 - Restored (Manually boot your servers)Cragfury - In progress
FR2 - Restored (Manually boot your servers)
Dugholim - Pending
DeepBrewer - In Progress
Boulderbeard -Pending
Mountainfury - Pending
Update on 1 Nov 2025 22:54:15 (UTC-05:00)
FR1 - Done
Cragfury - In progress
FR2 - In Progress
Dugholim - on deck
DeepBrewer - In Progress
Boulderbeard -on deck
Mountainfury - on deck
Restorations in Progress
FR1 - Done
Cragfury - In progress
FR2 - In Progress
Dugholim - on deck
DeepBrewer - In Progress
Boulderbeard -on deck
Mountainfury - on deck
Update on 1 Nov 2025 20:56:07 (UTC-05:00)
For any node still offline, we are failing over to our DR plan and starting restores of customer data from our last disaster recovery backup. Some data loss up 6 hours may occur.
For any node still offline, we are failing over to our DR plan and starting restores of customer data from our last disaster recovery backup. Some data loss up 6 hours may occur.
In progress on 1 Nov 2025 20:33:49 (UTC-05:00)
Half of the nodes are restored. We are working to repair or enact our DR plan for the other nodes.
Half of the nodes are restored. We are working to repair or enact our DR plan for the other nodes.
In progress on 1 Nov 2025 19:14:23 (UTC-05:00)
Fix has been deployed. Waiting on all nodes to reboot.
Fix has been deployed. Waiting on all nodes to reboot.
Identified on 1 Nov 2025 18:52:34 (UTC-05:00)
We have identified the network misconfiguration. We are fixing it on each node now.
We have identified the network misconfiguration. We are fixing it on each node now.
Investigating on 1 Nov 2025 18:37:50 (UTC-05:00)
We are aware of a network connection issue and are working to resolve it ASAP.
We are aware of a network connection issue and are working to resolve it ASAP.
DDOS Attack Virginia + Dallas on 30 Oct 2025 15:10:40 (UTC-05:00)
Resolved on 31 Oct 2025 05:20:12 (UTC-05:00)
No further attacks occurred overnight. Closing this incident.
No further attacks occurred overnight. Closing this incident.
Monitoring on 30 Oct 2025 18:36:31 (UTC-05:00)
All services and IPs should now be functioning normally. We are continuing to monitor and will close this in the morning EDT if no further attacks arise.
All services and IPs should now be functioning normally. We are continuing to monitor and will close this in the morning EDT if no further attacks arise.
Update on 30 Oct 2025 17:19:17 (UTC-05:00)
Other major transit providers are having possibly related issues even after the attack on our data center has subsided. These are causing outages as traffic is rerouted.
We are still monitoring and mitigating attacks as they come in
Other major transit providers are having possibly related issues even after the attack on our data center has subsided. These are causing outages as traffic is rerouted.
We are still monitoring and mitigating attacks as they come in
Monitoring on 30 Oct 2025 16:36:59 (UTC-05:00)
We have successfully mitigated the attack for now. We are monitoring and will respond as needed.
We have successfully mitigated the attack for now. We are monitoring and will respond as needed.
Investigating on 30 Oct 2025 15:10:40 (UTC-05:00)
At approximately 3:37 PM Eastern time, our NOC detected a carpet bomb style DDOS attack aimed at both our Virginia and Ashburn locations. This is creating short bursts of packet loss.
There is currently no ETTR, as this attack targets multiple data centers in each region rather than our services specifically.
At approximately 3:37 PM Eastern time, our NOC detected a carpet bomb style DDOS attack aimed at both our Virginia and Ashburn locations. This is creating short bursts of packet loss.
There is currently no ETTR, as this attack targets multiple data centers in each region rather than our services specifically.
US East and Central DDoS Attacks on 26 Oct 2025 21:29:53 (UTC-05:00)
Resolved on 28 Oct 2025 06:11:18 (UTC-05:00)
Mitigations have been deployed and no attack has made it through in the past 12 hours.
Mitigations have been deployed and no attack has made it through in the past 12 hours.
Update on 27 Oct 2025 06:32:32 (UTC-05:00)
The data center provider has made progress in stabilizing some regions overnight; the attack remains active and continues to shift patterns. This attack is not just targeting them but all providers in each area.
Starting this morning, a full-scale mitigation plan is being put in place to filter the networks using Cloudflare's Magic Transit. There is no ETA on complete restoration, but stability is improving.
The data center provider has made progress in stabilizing some regions overnight; the attack remains active and continues to shift patterns. This attack is not just targeting them but all providers in each area.
Starting this morning, a full-scale mitigation plan is being put in place to filter the networks using Cloudflare's Magic Transit. There is no ETA on complete restoration, but stability is improving.
In progress on 26 Oct 2025 21:29:53 (UTC-05:00)
Our Virginia and Texas data center providers are experiencing multiple short-duration carpet-bomb DDoS attacks. We are working to mitigate this ASAP.
Our Virginia and Texas data center providers are experiencing multiple short-duration carpet-bomb DDoS attacks. We are working to mitigate this ASAP.