On 20 October 2025 – in the early hours local time, at 8:48 in the morning Hungarian time – the DynamoDB database service became unreachable because of a DNS fault in us-east-1, the largest Amazon Web Services region, in Northern Virginia. The fault set off a chain reaction: for hours it wasn’t possible to launch new EC2 instances, some of the load balancers failed, and dozens of AWS services were disrupted. Full recovery took until the evening, and in the meantime many well-known applications and online shops became unavailable as well.
On 23 October, AWS published its official post-event summary. In this article we use it to look at what happened, and what is worth taking from it when designing systems that run on AWS.
What happened on 20 October?
The DNS records of DynamoDB’s regional endpoint (dynamodb.us-east-1.amazonaws.com) are managed by automation: a planner (DNS Planner) continuously produces new record plans, and enactors (DNS Enactor) write them into Route 53. In a rare race condition, one heavily delayed enactor wrote an outdated plan, and another one deleted this plan during cleanup – leaving the endpoint’s DNS record empty. Anything that would have connected to DynamoDB couldn’t find it.
The main timestamps in Hungarian time (CEST):
| Time | Event |
|---|---|
| 20 Oct 08:48 | DynamoDB DNS errors begin |
| 09:38 | Engineers identify the DNS state as the source |
| 11:25 | DNS information is restored |
| 11:40 | DynamoDB recovery completes as caches expire |
| 14:28 | EC2 instances can be launched again, but with heavy throttling |
| 14:30–23:09 | Connection errors on some Network Load Balancers |
| 22:50 | EC2 APIs and launches work normally |
| 23:20 | End of the event |
So DynamoDB recovered in about three hours, but the consequences dragged on until the evening. The reason is that AWS services themselves use DynamoDB in the background. The system that keeps track of EC2’s physical servers (DropletWorkflow Manager) lost contact with the machines during the outage, and was overloaded by the backlog of work during recovery. When new instances finally launched, their network configuration propagated slowly, so the Network Load Balancer health checks saw them as unhealthy and took them out of service. Lambda, ECS, EKS and Fargate were caught up in this as well.
EC2 instances that were already running were not affected by the fault. This is an important detail, and we will come back to it later.
Why didn’t multiple Availability Zones help?
AWS regions consist of several physically separate groups of data centres, called Availability Zones (AZs). The basic rule is that a production system should run in at least two AZs, so that a power cut or network fault in one data centre doesn’t take it down.
This outage, however, didn’t affect a single AZ but a regional service: the DynamoDB endpoint was unreachable across the whole region, and launching new instances was disrupted in every zone. A multi-AZ design doesn’t protect against this kind of fault – only another region does.
Control plane and data plane
AWS divides its services into two parts: the control plane is the creation, modification and deletion of resources (launching a new instance, growing an autoscaling group), and the data plane is the operation of resources that already exist (the running instance, serving an existing database). The control plane is more complex and has more dependencies, so it fails more often.
That is exactly what was visible on 20 October: the running servers kept going, but anyone who wanted to respond to the fault by launching new instances, autoscaling or bringing up a new environment couldn’t do so for hours. AWS’s own recommendation for this is static stability: the system should survive the loss of a zone or a service without needing a control plane operation to recover. In practice:
- Pre-provisioned capacity. If the system runs in three AZs, each should have enough headroom for two of them to carry the full load – don’t expect autoscaling to launch new ones when needed.
- Recovery shouldn’t require new resources. The standby system should already be running, or at least the image, database replica and DNS settings it needs should be ready.
- Avoid unnecessary dependencies. A simple web application can go down just because it requests configuration from another service at start-up; if that service doesn’t respond, have a local default.
The special role of us-east-1
The us-east-1 region is the oldest and largest AWS region, and the control planes of some global services also run here. According to the summary, during the fault IAM users couldn’t sign in to the console, and there were sign-in errors in other regions too; and Redshift queries failed in every region if they used IAM authentication, because they depended on an IAM API in us-east-1.
The lesson: even if your system runs in Frankfurt (eu-central-1), it can still depend on us-east-1. It is worth thinking through:
- whether you have emergency access (break-glass) if the usual sign-in doesn’t work,
- whether your recovery plan relies on IAM changes or on creating new roles,
- whether the certificates used for CloudFront, changes to Route 53 zones and other global settings are needed precisely when they are unavailable.
Multi-region: when is it worth it?
A second region isn’t free: data has to be replicated, the standby system has to be maintained, and failover has to be tested regularly. The common patterns, from cheapest to most expensive:
| Pattern | What runs in the second region? | Recovery time | Cost |
|---|---|---|---|
| Backup and restore | Only the backups | Hours, days | Low |
| Pilot light | Database replica, the rest stopped | Ten minutes to hours | Medium |
| Warm standby | A scaled-down, working copy | Minutes to ten minutes | Higher |
| Active-active | The full system, with traffic | Almost immediate | The highest |
For the online shop of a small or medium-sized business, backup and restore or pilot light is usually enough – provided the backups really are in another region and someone has already tested the restore. Active-active is worth it if an hour of downtime costs more than the annual cost of duplicate infrastructure.
If your system runs in Europe rather than in us-east-1, that in itself is a good decision: your customers are closer, and regional faults like this don’t affect you directly.
How do you know the problem is with AWS?
AWS status information is available in two places:
- the public AWS Health Dashboard page shows known events by region and service,
- in the console, the Your account health view shows what affects your own account and resources; you can also set up alerts for this with an EventBridge rule.
Don’t rely on this alone: an external monitoring service independent of AWS often alerts sooner than an event appears on the status page.
What has AWS fixed?
According to the summary, AWS has disabled the DynamoDB DNS automation worldwide, and will only turn it back on after fixing the race condition and adding further safeguards. For the Network Load Balancer, they are limiting how much capacity a health check failure can remove at once, and for EC2 they are protecting the recovery process with new tests and better throttling.
What is up to you
No single customer can prevent an outage of this size, but you can reduce its impact. We said it after the CrowdStrike outage too, and it still holds: what helps most is a written and tested recovery plan that knows what depends on what. We wrote about infrastructure reliability in general in this article.
We help with designing AWS environments – multi-AZ and multi-region architectures, backup and recovery plans; you can read more on our Cloud and DevOps page.