• Skip to primary navigation
  • Skip to main content
  • Skip to footer

Cisco Umbrella

Enterprise network security

  • Contact Sales
  • Login
    • Umbrella Login
    • Cloudlock Login
  • Why Us
    • Why Cisco Umbrella
      • Why Try Umbrella
      • Why DNS Security
      • Why Umbrella SASE
      • Our Customers
      • Customer Stories
      • Why Cisco Secure
    • Fast Reliable Cloud
      • Global Cloud Architecture
      • Cloud Network Status
      • Global Cloud Network Activity
    • Unmatched Intelligence
      • A New Approach to Cybersecurity
      • Interactive Intelligence
      • Cyber Attack Prevention
      • Umbrella and Cisco Talos Threat Intelligence
    • Extensive Integrations
      • IT Security Integrations
      • Hardware Integrations
      • Meraki Integration
      • Cisco Umbrella and SecureX
  • Products
    • Cisco Umbrella Products
      • Cisco Umbrella Cloud Security Service
      • Recursive DNS Services
      • Cisco Umbrella SIG
      • Umbrella Investigate
      • What’s New
    • Product Packages
      • Cisco Umbrella Package Comparison
      • – DNS Security Essentials Package
      • – DNS Security Advantage Package
      • – SIG Essentials Package
      • – SIG Advantage Package
      • Umbrella Support Packages
    • Functionality
      • DNS-Layer Security
      • Secure Web Gateway
      • Cloud Access Security Broker (CASB)
      • Cloud Data Loss Prevention (DLP)
      • Cloud-Delivered Firewall
      • Cloud Malware Protection
      • Remote Browser Isolation (RBI)
    • Man on a laptop with headphones on. He is attending a Cisco Umbrella Live Demo
  • Solutions
    • SASE & SSE Solutions
      • Cisco Umbrella SASE
      • Secure Access Service Edge (SASE)
      • What is SASE
      • What is Security Service Edge (SSE)
    • Functionality Solutions
      • Web Content Filtering
      • Secure Direct Internet Access
      • Shadow IT Discovery & App Blocking
      • Fast Incident Response
      • Unified Threat Management
      • Protect Mobile Users
      • Securing Remote and Roaming Users
    • Network Solutions
      • Guest Wi-Fi Security
      • SD-WAN Security
      • Off-Network Endpoint Security
    • Industry Solutions
      • Government and Public Sector Cybersecurity
      • Financial Services Security
      • Cybersecurity for Manufacturing
      • Higher Education Security
      • K-12 Schools Security
      • Healthcare, Retail and Hospitality Security
      • Enterprise Cloud Security
      • Small Business Cybersecurity
  • Resources
    • Content Library
      • Top Resources
      • Cybersecurity Webinars
      • Events
      • Research Reports
      • Case Studies
      • Videos
      • Datasheets
      • eBooks
      • Solution Briefs
    • International Documents
      • Deutsch/German
      • Español/Spanish
      • Français/French
      • Italiano/Italian
      • 日本語/Japanese
    • Security Definitions
      • What is Secure Access Service Edge (SASE)
      • What is Security Service Edge (SSE)
      • What is a Cloud Access Security Broker (CASB)
      • Cyber Threat Categories and Definitions
    • For Customers
      • Support
      • Customer Success Webinars
      • Cisco Umbrella Studio
  • Trends & Threats
    • Market Trends
      • Hybrid Workforce
      • Rise of Remote Workers
      • Secure Internet Gateway (SIG)
    • Security Threats
      • How to Stop Phishing Attacks
      • Malware Detection and Protection
      • Ransomware is on the Rise
      • Cryptomining Malware Protection
      • Cybersecurity Threat Landscape
      • Global Cyber Threat Intelligence
    •  
    • Woman connecting confidently to any device anywhere
  • Partners
    • Channel Partners
      • Partner Program
      • Become a Partner
    • Service Providers
      • Secure Connectivity
      • Managed Security for MSSPs
      • Managed IT for MSPs
    •  
    • Person looking down at laptop. They are connecting and working securely
  • Blog
    • News & Product Posts
      • Latest Posts
      • Products & Services
      • Customer Focus
      • Feature Spotlight
    • Cybersecurity Posts
      • Security
      • Threats
      • Cybersecurity Threat Spotlight
      • Research
    •  
    • Register for a webinar - with illustration of connecting securely to the cloud
  • Contact Us
  • Umbrella Login
  • Cloudlock Login
  • Free Trial
Customer Focus Spotlight

How Cisco Umbrella supported customers during the Akamai DNS outage

Author avatar of Chloe WhitakerChloe Whitaker
Updated — March 17, 2022 • 4 minute read
View blog >

It’s been a week since the Akamai Edge DNS Service outage. In that time, a bevy of news articles, op-eds, and think pieces have popped up discussing how this most recent domain name system (DNS) failure proves that major enterprises like Amazon, American Airlines, Oracle Cloud, and UPS need to improve their network infrastructure.

This consensus, while true, is unhelpful for most internet users. Small businesses and large corporations may rely on digital entities Amazon or Oracle Cloud, but they have no power over how these corporations invest their infrastructure dollars. So when DNS outages occur, most internet users resign themselves to waiting.

But, what if there was a better way to deal with DNS outages from the end user side? Well, as it turns out, there is. On July 22, 2021 – while most people on the internet were stuck reading variations on an error message – many companies deploying Cisco Umbrella kept up with critical business operations. That’s because our DNS-layer security platform comes with a feature called SmartCache. SmartCache allows Cisco Umbrella recursive DNS servers to bypass unresponsive authoritative DNS servers and connect to websites using the last known IP address. Few, if any, DNS security providers offer this kind of functionality to their customers. And, for users looking to avoid the downtime caused by DNS failures out of their control, Cisco Umbrella’s SmartCache represents an easy-to-implement, consumer-centric solution.

What caused the Akamai DNS outage?

Akamai Technologies operates a network of authoritative DNS servers, marketed as the Akamai Edge DNS Platform. On July 22, Akamai engineers pushed out a software configuration update that triggered a bug in their DNS system. This bug caused users to experience widespread DNS failures when trying to access thousands of websites. After a little over an hour, Akamai engineers rolled back the update and the Akamai Edge DNS servers resumed normal operation.1

If the connection between a single bug in a DNS server and a massive internet outage feels a bit nebulous to you, you’re not alone – we even put together a blog post to help users understand how the Domain Name System works. But for the sake of understanding the Akamai outage – and planning for similar future outages – there are three things about the Domain Name System you need to know:

  1. Every computer on the internet, including the servers that host websites, has a unique IP address that is used to connect to it.
  2. Authoritative DNS servers, like those operated by Akamai, store a list of regional domain names and their associated IP addresses, much like a phone book.
  3. Recursive DNS servers, like those operated by your Internet Service Provider (ISP) or Cisco Umbrella, query authoritative DNS servers to convert the domain name you input into your browser to an IP address your computer can use to connect to a website server.

When the Akamai authoritative DNS servers went down, the phone book containing IP addresses for sites like Home Depot, Costco, and LastPass became inaccessible. And because recursive DNS servers couldn’t find the IP addresses they were looking for, they provided internet users with an error message informing them of the DNS failure.

Are DNS outages common?

Considering DNS forms the foundation of internet connectivity, DNS outages are surprisingly common. In 2020, a DNS failure at CloudFlare took sites like Discord, Politico, and Shopify offline. In 2016, a Distributed Denial of Service (DDoS) attack took DYN DNS servers offline, keeping users from sites for Squarespace, Verizon, Twitter, and the Swedish Government, among others. And these are just a few large examples. Smaller DNS outages occur more commonly and can affect a single entity – like the DNS failure that took Salesforce offline in 2021 or the DNS public resolver issue at Google that crippled user connectivity in 2018.

In today’s interconnected, cloud-based world, each of these outages represents hours of unplanned and unproductive downtime for companies large and small. This is where Cisco Umbrella comes in.

How Cisco Umbrella can help you prepare for the next DNS outage

Remember when we discussed how recursive DNS servers query authoritative DNS servers to find the IP address associated with a domain name? Most recursive DNS servers, like those used by your ISP, only cache these records for a short period of time. After the time limit expires, the record is deleted and the recursive DNS server has to query the authoritative DNS server again to find the right IP address for a given domain name.

However, the Cisco Umbrella recursive DNS servers don’t delete the expired records. Instead, they mark them as expired and store them in a database containing records gathered from billions of recursive DNS queries. Most of the time, these expired records aren’t used for very much. But during an authoritative DNS outage, Cisco Umbrella recursive DNS servers can use these expired records to connect users to the last known IP address for the domain they’re trying to access.

So, how did this feature play out during the Akamai outage? Our engineers tracked DNS timeouts for Cisco Umbrella users in the graph below during the event:

Image shows a graph. The Y axis indicates timeouts per second, starting at a value of 100K and increasing in increasing in increments of 100K until reaching the value 1.1M. The X axis indicates timestamps from August 22, 2021, beginning at 15:35 and increasing in increments of five minutes until 16:45. The graph shows a red line that indicates observed DNS server failures, which starts at 160K, spikes to around 900K at 15:37, and goes back down to 160K at 16:38. Below the red line is a green line that indicates SmartCache DNS responses. The green line spikes to roughly 490K at 15:39 before dropping to slightly above zero at 16:38.

and the X axis indicates consecutive timestamps from

The red area of the graph indicates DNS failures caused by an inability to obtain IP addresses from authoritative DNS servers. The sudden spike near the left-hand side of the graph was caused by the Akamai outage – you’ll notice the number of failures drops dramatically near the right-hand side of the graph when the issue is resolved.

The green area of the graph is far more interesting, though. It indicates situations where the Cisco Umbrella recursive DNS servers were able to connect users to websites using the last known IP address. This is SmartCache functionality in action. For the duration of the Akamai outage, Cisco Umbrella recursive DNS servers were able to complete anywhere between 40% to 50% of queries using SmartCache data, allowing many customers to connect to sites they regularly use while large portions of the Internet were down. And since the Cisco Umbrella network continues to grow, you can expect the number of records in the SmartCache system to expand.

Don’t let DNS failures slow you down

Of course, safeguarding connectivity during a DNS outage is just one of the ways our team provides value to customers. Want to learn more about the capabilities of Cisco Umbrella? Register for one of our live demos or sign up for a Cisco Umbrella free trial today!

1 @akamai “Akamai Summarizes Service Disruption [Tweet]” Twitter.

Suggested Blogs

  • Hitachi’s SASE: How Umbrella & Duo Delivered Identity and Security December 13, 2022 2 minute read
  • Why Using DNS for Protection Should Be Your First Line of Defense September 1, 2022 2 minute read
  • New Security for a World Where Everyone and Everything Are Connecting August 30, 2022 3 minute read

Share this blog

FacebookTweetLinkedIn

Follow Us

  • Twitter
  • Facebook
  • LinkedIn
  • YouTube

Footer Sections

What we make

  • Cloud Security Service
  • DNS-Layer Network Security
  • Secure Web Gateway
  • Security Packages

Who we are

  • Global Cloud Architecture
  • Cloud Network Status
  • Cloud Network Activity
  • OpenDNS is now Umbrella
  • Cisco Umbrella Blog

Learn more

  • Webinars
  • Careers
  • Support
  • Cisco Umbrella Live Demo
  • Contact Sales
Umbrella by Cisco
208.67.222.222+208.67.220.220
2620:119:35::35+2620:119:53::53
Sign up for a Free Trial
  • Cisco Online Privacy Statement
  • Terms of Service
  • Sitemap

© 2023 Cisco Umbrella