• Skip to primary navigation
  • Skip to main content
  • Skip to footer

Cisco Umbrella

Enterprise network security

  • Contact Sales
  • Login
    • Umbrella Login
    • Cloudlock Login
  • Why Us
    • Why Cisco Umbrella
      • Why Try Umbrella
      • Why DNS Security
      • Why Umbrella SASE
      • Our Customers
      • Customer Stories
      • Why Cisco Secure
    • Fast Reliable Cloud
      • Global Cloud Architecture
      • Cloud Network Status
      • Global Cloud Network Activity
    • Unmatched Intelligence
      • A New Approach to Cybersecurity
      • Interactive Intelligence
      • Cyber Attack Prevention
      • Umbrella and Cisco Talos Threat Intelligence
    • Extensive Integrations
      • IT Security Integrations
      • Hardware Integrations
      • Meraki Integration
      • Cisco Umbrella and SecureX
  • Products
    • Cisco Umbrella Products
      • Cisco Umbrella Cloud Security Service
      • Recursive DNS Services
      • Cisco Umbrella SIG
      • Umbrella Investigate
      • What’s New
    • Product Packages
      • Cisco Umbrella Package Comparison
      • – DNS Security Essentials Package
      • – DNS Security Advantage Package
      • – SIG Essentials Package
      • – SIG Advantage Package
      • Umbrella Support Packages
    • Functionality
      • DNS-Layer Security
      • Secure Web Gateway
      • Cloud Access Security Broker (CASB)
      • Cloud Data Loss Prevention (DLP)
      • Cloud-Delivered Firewall
      • Cloud Malware Protection
      • Remote Browser Isolation (RBI)
    • Man on a laptop with headphones on. He is attending a Cisco Umbrella Live Demo
  • Solutions
    • SASE & SSE Solutions
      • Cisco Umbrella SASE
      • Secure Access Service Edge (SASE)
      • What is SASE
      • What is Security Service Edge (SSE)
    • Functionality Solutions
      • Web Content Filtering
      • Secure Direct Internet Access
      • Shadow IT Discovery & App Blocking
      • Fast Incident Response
      • Unified Threat Management
      • Protect Mobile Users
      • Securing Remote and Roaming Users
    • Network Solutions
      • Guest Wi-Fi Security
      • SD-WAN Security
      • Off-Network Endpoint Security
    • Industry Solutions
      • Government and Public Sector Cybersecurity
      • Financial Services Security
      • Cybersecurity for Manufacturing
      • Higher Education Security
      • K-12 Schools Security
      • Healthcare, Retail and Hospitality Security
      • Enterprise Cloud Security
      • Small Business Cybersecurity
  • Resources
    • Content Library
      • Top Resources
      • Cybersecurity Webinars
      • Events
      • Research Reports
      • Case Studies
      • Videos
      • Datasheets
      • eBooks
      • Solution Briefs
    • International Documents
      • Deutsch/German
      • Español/Spanish
      • Français/French
      • Italiano/Italian
      • 日本語/Japanese
    • Security Definitions
      • What is Secure Access Service Edge (SASE)
      • What is Security Service Edge (SSE)
      • What is a Cloud Access Security Broker (CASB)
      • Cyber Threat Categories and Definitions
    • For Customers
      • Support
      • Customer Success Webinars
      • Cisco Umbrella Studio
  • Trends & Threats
    • Market Trends
      • Hybrid Workforce
      • Rise of Remote Workers
      • Secure Internet Gateway (SIG)
    • Security Threats
      • How to Stop Phishing Attacks
      • Malware Detection and Protection
      • Ransomware is on the Rise
      • Cryptomining Malware Protection
      • Cybersecurity Threat Landscape
      • Global Cyber Threat Intelligence
    •  
    • Woman connecting confidently to any device anywhere
  • Partners
    • Channel Partners
      • Partner Program
      • Become a Partner
    • Service Providers
      • Secure Connectivity
      • Managed Security for MSSPs
      • Managed IT for MSPs
    •  
    • Person looking down at laptop. They are connecting and working securely
  • Blog
    • News & Product Posts
      • Latest Posts
      • Products & Services
      • Customer Focus
      • Feature Spotlight
    • Cybersecurity Posts
      • Security
      • Threats
      • Cybersecurity Threat Spotlight
      • Research
    •  
    • Register for a webinar - with illustration of connecting securely to the cloud
  • Contact Us
  • Umbrella Login
  • Cloudlock Login
  • Free Trial
Security

IP Routing, AWS, and Docker

Author avatar of Rahim LalaniRahim Lalani
Updated — April 10, 2020 • 3 minute read
View blog >

Screen Shot 2014-06-30 at 9.39.14 AM

Operating distributed computing systems at scale brings a variety of challenges. Minor issues like having the wrong version of a small software library can take a whole application offline. To create a smooth transition from development to operations with regard to dependencies, environment, and testing OpenDNS has adopted open-source Docker containerization technology. Interfacing Docker containers with existing dedicated infrastructure across 23 data centers provided a unique set of routing challenges which we solved with clever application of Generic Routing Encapsulation and Border Gateway Protocol.

Containers

Containerization brings operating system-level virtualization. Compared to tools like VMware, this direct access to the system kernel brings improved speed. Docker specifically brings Git-levels of simplicity to managing system images. In short, packaging applications in Docker takes the guess work out of deployments by replicating the exact same environment between development and production.

Impetus at OpenDNS

Containerization at OpenDNS started as a hackathon project in August 2013 when a team built a small platform system to demonstrate its benefits. Since then, an OpenDNS Engineering Infrastructure team has built a comprehensive platform as a service (PaaS) based on Docker. By abstracting away the underlying infrastructure, engineering can focus on development rather than dependency management and monitoring. The PaaS itself implements service methodologies from the Twelve-factor app.
The OpenDNS Global Network includes 23 data centers plus significant cloud infrastructure in Amazon Web Services (AWS). Routing network traffic between these hardware and cloud infrastructures poses a challenge, and the OpenDNS PaaS project began as a way to replicate the performance of dedicated hardware systems in cloud networks like AWS. Described below is an experiment in routing around one of those limitations.

The Problem

OpenDNS sought to be able to run hundreds of containers, each potentially having a privately-routable IP address, on a single EC2 host. The problem is that an EC2 instance can have a limited number of secondary IP addresses.

Two Quick Definitions

  • Generic Routing Encapsulation (GRE) is used to encapsulate a protocol and route it over IP. In our solution we encapsulate an IP packet inside of another IP packet. This allows us to build tunnels through third party networks.
  • Border Gateway Protocol (BGP) is a routing protocol used to announce reachability of IP networks. This protocol is most frequently used between ISPs. Below we’ll be using it between one of our routers and EC2 instances.

A Solution

Our solution was to use GRE to create an overlay network spanning OpenDNS’s network and our AWS Virtual Private Cloud (VPC) to get around routing limitations in EC2.
The first approach was to allocate and statically route /24 prefixes (allocated from a larger block, e.g. 10.0.0.0/16) to individual docker hosts running in EC2. This would give us sufficient IP addresses to support 250 containers with unique IPs per host.

Extending this concept, we used BGP sessions between each docker host and our router. Each host announces its own address space. This allowed us to go from a single large prefix to individual /32s with minimal configuration on the router. Using Quagga as a BGP daemon on our docker hosts allowed us to redistribute /32s from the hosts’ routing table into BGP. Tools like Pipework could be used here to wire up container IPs and the routing table with Quagga automatically announcing these via BGP.
aws-docker-arch

Powerful

Moving a container from one host to another host becomes straightforward in this scenario. Replicate the container on a new host and assign it the prior IP address, then afterward destroy the old container. The BGP routes converge within a couple of seconds to re-route traffic.

What happens if you don’t take the old container down and there are two containers with the same IP address? Load balancing! Most routers can be configured for Equal-Cost Multi-Path (ECMP) routing to accomplish this.

This leads to powerful options such as creating fault tolerant active-active load balancing across multiple hosts, as well as using anycast across our hybrid environment which we discussed in a previous blog post.

Next Steps

Utilizing Docker has made operating distributed computing systems between dedicated and cloud data centers transparent. As our internal PaaS tools evolve, we continue to automate more tasks. For instance, we configure the GRE tunnels and BGP peers on each docker host and router manually. This is not scalable and we need to look at how to automate this. We used BGP as our dynamic routing protocol but we could explore OSPF as an alternative. We have been using private IP address space in our tests but we would also like to experiment with using public IP addresses in the same way. Lastly, we are excited to start using IPv6 addresses in our containers on EC2 and truly remove any address limitations.

We’d love to hear from you if you’re using a similar method or have alternate solutions!

Suggested Blogs

  • Cisco Umbrella Delivered Better Cybersecurity and 231% ROI February 21, 2023 2 minute read
  • Cisco Listed as a Representative Vendor in Gartner® Market Guide for Single-Vendor SASE January 26, 2023 3 minute read
  • How to Evaluate SSE Vendors: Questions to Ask, Pitfalls to Avoid June 23, 2022 5 minute read

Share this blog

FacebookTweetLinkedIn

Follow Us

  • Twitter
  • Facebook
  • LinkedIn
  • YouTube

Footer Sections

What we make

  • Cloud Security Service
  • DNS-Layer Network Security
  • Secure Web Gateway
  • Security Packages

Who we are

  • Global Cloud Architecture
  • Cloud Network Status
  • Cloud Network Activity
  • OpenDNS is now Umbrella
  • Cisco Umbrella Blog

Learn more

  • Webinars
  • Careers
  • Support
  • Cisco Umbrella Live Demo
  • Contact Sales
Umbrella by Cisco
208.67.222.222+208.67.220.220
2620:119:35::35+2620:119:53::53
Sign up for a Free Trial
  • Cisco Online Privacy Statement
  • Terms of Service
  • Sitemap

© 2023 Cisco Umbrella