• Skip to primary navigation
  • Skip to main content
  • Skip to footer

Cisco Umbrella

Enterprise network security

  • Free Trial
  • Contact us
  • Blog
  • Login
    • Umbrella Login
    • Cloudlock Login
  • Products
    • Product
      • Cisco Umbrella Cloud Security Service
      • Cisco Umbrella Investigate
      • Product Packages
      • Support Packages
    • Functionality
      • DNS-Layer Security
      • Secure Web Gateway
      • Cloud Access Security Broker (CASB)
      • Interactive Intelligence
      • Cloud-Delivered Firewall
    •  
    • Webinar signup
  • Solutions
    • By Need
      • Protect Mobile Users
      • Fast Incident Response
      • Web Content Filtering
      • Shadow IT Discovery & App Blocking
      • Unified Threat Enforcement
      • Reduce Security Infections
      • Secure Direct Internet Access
      • Securing Remote and Roaming Users
    • By Network
      • Protect Guest Wi-Fi
      • SD-WAN Security
      • Off-Network Endpoint Security
    • By Industry
      • Higher Education Security
      • K-12 Schools Security
      • Healthcare, Retail and Hospitality Security
      • Enterprise Cloud Security
      • Small Business Cybersecurity
      • Our Customers
      • Customer Stories
    • Ransomware Defense for Dummies book
  • Why Us
    • Fast Reliable Cloud
      • Cloud Security Infrastructure
      • Cloud Network Status
      • Cloud Network Activity
      • Recursive DNS Services
      • Top Reasons to Trial
      • Getting Started
    • Unmatched Intelligence
      • Cyber Attack Prevention
      • Interactive Intelligence
    • Extensive Integrations
      • IT Security Integrations
      • Hardware Integrations
      • Meraki Integration
      • Cisco SD-WAN
    • Navigation-dropdown-promo-free-trial_102820
  • Resources
    • Content Library
      • Top Resources
      • Analyst Reports
      • Case Studies
      • Customer Videos
      • Datasheets
      • eBooks
      • Infographics
      • Solution Briefs
    • International Documents
      • Deutsch/German
      • Español/Spanish
      • Français/French
      • Italiano/Italian
      • 日本語/Japanese
    • Cisco Umbrella Blog
      • Latest Posts
      • Security Posts
      • Research Posts
      • Threats Posts
      • Product Posts
      • Spotlight
    • For Customers
      • Support
      • Customer Success Hub
      • Umbrella Deployment Hub
      • Customer Success Webinars
      • What’s New
      • Cisco Umbrella Studio
  • Trends & Threats
    • Market Trends
      • Rise of Remote Workers
      • Secure Internet Gateway (SIG)
      • Secure Access Service Edge (SASE)
    • Security Threats
      • Ransomware
      • Cryptomining Malware Protection
      • Cybersecurity Threat Landscape
    •  
    • 2020 Cybersecurity trends
  • Partners
    • Channel Partners
      • Partner Program
      • Become a Partner
    • Service Providers
      • Secure Connectivity
      • Managed Security for MSSPs
      • Managed IT for MSPs
    •  
    • Become a partner
  • Free Trial Signup
  • Umbrella Login
  • Cloudlock Login
  • Contact Us
Security

IP Routing, AWS, and Docker

By Rahim Lalani
Posted on July 1, 2014
Updated on April 10, 2020

Share

Facebook0Tweet0LinkedIn0
Screen Shot 2014-06-30 at 9.39.14 AM

Operating distributed computing systems at scale brings a variety of challenges. Minor issues like having the wrong version of a small software library can take a whole application offline. To create a smooth transition from development to operations with regard to dependencies, environment, and testing OpenDNS has adopted open-source Docker containerization technology. Interfacing Docker containers with existing dedicated infrastructure across 23 data centers provided a unique set of routing challenges which we solved with clever application of Generic Routing Encapsulation and Border Gateway Protocol.

Containers

Containerization brings operating system-level virtualization. Compared to tools like VMware, this direct access to the system kernel brings improved speed. Docker specifically brings Git-levels of simplicity to managing system images. In short, packaging applications in Docker takes the guess work out of deployments by replicating the exact same environment between development and production.

Impetus at OpenDNS

Containerization at OpenDNS started as a hackathon project in August 2013 when a team built a small platform system to demonstrate its benefits. Since then, an OpenDNS Engineering Infrastructure team has built a comprehensive platform as a service (PaaS) based on Docker. By abstracting away the underlying infrastructure, engineering can focus on development rather than dependency management and monitoring. The PaaS itself implements service methodologies from the Twelve-factor app.
The OpenDNS Global Network includes 23 data centers plus significant cloud infrastructure in Amazon Web Services (AWS). Routing network traffic between these hardware and cloud infrastructures poses a challenge, and the OpenDNS PaaS project began as a way to replicate the performance of dedicated hardware systems in cloud networks like AWS. Described below is an experiment in routing around one of those limitations.

The Problem

OpenDNS sought to be able to run hundreds of containers, each potentially having a privately-routable IP address, on a single EC2 host. The problem is that an EC2 instance can have a limited number of secondary IP addresses.

Two Quick Definitions

  • Generic Routing Encapsulation (GRE) is used to encapsulate a protocol and route it over IP. In our solution we encapsulate an IP packet inside of another IP packet. This allows us to build tunnels through third party networks.
  • Border Gateway Protocol (BGP) is a routing protocol used to announce reachability of IP networks. This protocol is most frequently used between ISPs. Below we’ll be using it between one of our routers and EC2 instances.

A Solution

Our solution was to use GRE to create an overlay network spanning OpenDNS’s network and our AWS Virtual Private Cloud (VPC) to get around routing limitations in EC2.
The first approach was to allocate and statically route /24 prefixes (allocated from a larger block, e.g. 10.0.0.0/16) to individual docker hosts running in EC2. This would give us sufficient IP addresses to support 250 containers with unique IPs per host.

Extending this concept, we used BGP sessions between each docker host and our router. Each host announces its own address space. This allowed us to go from a single large prefix to individual /32s with minimal configuration on the router. Using Quagga as a BGP daemon on our docker hosts allowed us to redistribute /32s from the hosts’ routing table into BGP. Tools like Pipework could be used here to wire up container IPs and the routing table with Quagga automatically announcing these via BGP.
aws-docker-arch

Powerful

Moving a container from one host to another host becomes straightforward in this scenario. Replicate the container on a new host and assign it the prior IP address, then afterward destroy the old container. The BGP routes converge within a couple of seconds to re-route traffic.

What happens if you don’t take the old container down and there are two containers with the same IP address? Load balancing! Most routers can be configured for Equal-Cost Multi-Path (ECMP) routing to accomplish this.

This leads to powerful options such as creating fault tolerant active-active load balancing across multiple hosts, as well as using anycast across our hybrid environment which we discussed in a previous blog post.

Next Steps

Utilizing Docker has made operating distributed computing systems between dedicated and cloud data centers transparent. As our internal PaaS tools evolve, we continue to automate more tasks. For instance, we configure the GRE tunnels and BGP peers on each docker host and router manually. This is not scalable and we need to look at how to automate this. We used BGP as our dynamic routing protocol but we could explore OSPF as an alternative. We have been using private IP address space in our tests but we would also like to experiment with using public IP addresses in the same way. Lastly, we are excited to start using IPv6 addresses in our containers on EC2 and truly remove any address limitations.

We’d love to hear from you if you’re using a similar method or have alternate solutions!

Previous Post:

Previous Article

Next Post:

Next Article

Follow Us

  • Twitter
  • Facebook
  • LinkedIn
  • YouTube

Footer Sections

What we make

  • Cloud Security Service
  • DNS-Layer Network Security
  • Secure Web Gateway
  • Security Packages

Who we are

  • Cisco Umbrella Blog
  • Cloud Network Status
  • Cloud Network Activity
  • OpenDNS is now Cisco Umbrella

Learn more

  • Events
  • Careers
  • Support
  • Cisco Umbrella Live Demo
  • Contact Sales
Umbrella by Cisco
208.67.222.222+208.67.220.220
2620:119:35::35+2620:119:53::53
Sign up for a Free Trial
  • Cisco Online Privacy Statement
  • Terms of Service
  • Sitemap

© 2021 Cisco Umbrella