11-hour incident blocked traffic from reaching rackspace.com and some subdomains

Rackspace says it has recovered from a nasty distributed denial of service attack that it says may have seen “a portion of legitimate traffic to our DNS infrastructure … inadvertently blocked.”

The trouble started just before lunchtime on Monday, US central time, and persisted until 11 hours later.

Over on the company’s Google+ page Rackspace warned of “intermittent periods of latency, packet loss, or connectivity failures when attempting to reach rackspace.com or subdomains within rackspace.com.”

The company’s status report later confirmed it had “… identified a UDP DDoS attack targeting the DNS servers in our IAD, ORD, and LON data centers [North Virigina, Chicago and London]. As a result of this issue, authoritative DNS resolution for any new request to the DNS servers began to fail in the affected data centers. In order to stabilize the issue, our teams placed the impacted DNS infrastructure behind mitigation services. This service is designed to protect our infrastructure, however, due to the nature of the event, a portion of legitimate traffic to our DNS infrastructure may be inadvertently blocked. Our teams are actively working to mitigate the attack and provide service stability.”

Rackspace is now confident things are back in order, as it has blacklisted DNS servers that were “sending both legitimate and DDoS traffic to Rackspace”. Users may not be entirely out of the woods, as its most recent update says “If you continue to experience adverse impact, please reach out to your support teams and provide trace route information for further investigations.”

A full root cause analysis of the incident is under way.

Source: http://www.theregister.co.uk/2014/12/24/rackspace_restored_after_ddos_takes_out_dns/