• Skip to primary navigation
  • Skip to main content
  • Skip to footer

Cisco Umbrella

Enterprise network security

  • Contact Sales
  • Login
    • Umbrella Login
    • Cloudlock Login
  • Why Us
    • Why Cisco Umbrella
      • Why Try Umbrella
      • Why DNS Security
      • Why Umbrella SASE
      • Our Customers
      • Customer Stories
      • Why Cisco Secure
    • Fast Reliable Cloud
      • Global Cloud Architecture
      • Cloud Network Status
      • Global Cloud Network Activity
    • Unmatched Intelligence
      • A New Approach to Cybersecurity
      • Interactive Intelligence
      • Cyber Attack Prevention
      • Umbrella and Cisco Talos Threat Intelligence
    • Extensive Integrations
      • IT Security Integrations
      • Hardware Integrations
      • Meraki Integration
      • Cisco Umbrella and SecureX
  • Products
    • Cisco Umbrella Products
      • Cisco Umbrella Cloud Security Service
      • Recursive DNS Services
      • Cisco Umbrella SIG
      • Umbrella Investigate
      • What’s New
    • Product Packages
      • Cisco Umbrella Package Comparison
      • – DNS Security Essentials Package
      • – DNS Security Advantage Package
      • – SIG Essentials Package
      • – SIG Advantage Package
      • Umbrella Support Packages
    • Functionality
      • DNS-Layer Security
      • Secure Web Gateway
      • Cloud Access Security Broker (CASB)
      • Cloud Data Loss Prevention (DLP)
      • Cloud-Delivered Firewall
      • Cloud Malware Protection
      • Remote Browser Isolation (RBI)
    • Man on a laptop with headphones on. He is attending a Cisco Umbrella Live Demo
  • Solutions
    • SASE & SSE Solutions
      • Cisco Umbrella SASE
      • Secure Access Service Edge (SASE)
      • What is SASE
      • What is Security Service Edge (SSE)
    • Functionality Solutions
      • Web Content Filtering
      • Secure Direct Internet Access
      • Shadow IT Discovery & App Blocking
      • Fast Incident Response
      • Unified Threat Management
      • Protect Mobile Users
      • Securing Remote and Roaming Users
    • Network Solutions
      • Guest Wi-Fi Security
      • SD-WAN Security
      • Off-Network Endpoint Security
    • Industry Solutions
      • Government and Public Sector Cybersecurity
      • Financial Services Security
      • Cybersecurity for Manufacturing
      • Higher Education Security
      • K-12 Schools Security
      • Healthcare, Retail and Hospitality Security
      • Enterprise Cloud Security
      • Small Business Cybersecurity
  • Resources
    • Content Library
      • Top Resources
      • Cybersecurity Webinars
      • Events
      • Research Reports
      • Case Studies
      • Videos
      • Datasheets
      • eBooks
      • Solution Briefs
    • International Documents
      • Deutsch/German
      • Español/Spanish
      • Français/French
      • Italiano/Italian
      • 日本語/Japanese
    • For Customers
      • Support
      • Customer Success Webinars
      • Cisco Umbrella Studio
    • Get the 2022 Cloud Scurity Comparison Guide
  • Trends & Threats
    • Market Trends
      • Hybrid Workforce
      • Rise of Remote Workers
      • Secure Internet Gateway (SIG)
    • Security Threats
      • How to Stop Phishing Attacks
      • Malware Detection and Protection
      • Ransomware is on the Rise
      • Cryptomining Malware Protection
      • Cybersecurity Threat Landscape
      • Global Cyber Threat Intelligence
      • Cyber Threat Categories and Definitions
    •  
    • Woman connecting confidently to any device anywhere
  • Partners
    • Channel Partners
      • Partner Program
      • Become a Partner
    • Service Providers
      • Secure Connectivity
      • Managed Security for MSSPs
      • Managed IT for MSPs
    •  
    • Person looking down at laptop. They are connecting and working securely
  • Blog
    • News & Product Posts
      • Latest Posts
      • Products & Services
      • Customer Focus
      • Feature Spotlight
    • Cybersecurity Posts
      • Security
      • Threats
      • Cybersecurity Threat Spotlight
      • Research
    •  
    • Register for a webinar - with illustration of connecting securely to the cloud
  • Contact Us
  • Umbrella Login
  • Cloudlock Login
  • Free Trial
Research

ZeroMQ: Helping us Block Malicious Domains in Real Time

By OpenDNS Security Research
Posted on October 4, 2013
Updated on March 5, 2020

Share

FacebookTweetLinkedIn

Every day, we process terabytes of data in order to spot malicious domains based on their network features and how they are accessed.

Our live dataset comes from two major sources:
– The log files of queries sent by users to our resolvers
– The log files of queries sent by our resolvers to authoritative servers.

In this blog post, we will focus on the latter, highlighting how we use it to block suspicious domains immediately after they show up in our log files.

A not-quite-passive DNS database
———————————-

Every time a resolver needs to answer a question that it hasn’t seen before, or that is no longer present in the cache, it must recursively query authoritative DNS servers before eventually being able to forward the final response to a client.

We are logging every single packet received by our resolvers from authoritative servers. This lets us keep historical data on all the domain names that we received queries for.

This kind of database is often referred to as a “Passive DNS database,” but our system works in a different way, and it’s not quite passive. We are not running sensors: the resolvers themselves are directly logging the responses they receive from authoritative servers.

Such a database lets us answer questions like, “What are all the IP addresses that example.com has resolved to in the past 90 days?” and, “What are all the domain names that are using ns1.example.com as an authoritative server?”

Building the database
———————

Records received from authoritative servers are highly redundant, because the majority of records have a short TTL even though their content doesn’t actually change much. The edns-clientsubnet extension also triggers a new upstream query for each client subnet, even though many different subnets are actually going to share the same response. Removing duplicate records is thus an essential preliminary step to build our DNS database. This step drastically reduces the amount of data to store: on average, out of 241 log records, we actually only store one.

We use a Bloom filter in order to remove duplicate records without having to sort them. Hash functions used to build this Bloom filter are created by using SipHash-2-4 with pseudorandomly-generated keys, and the keys are rotated after each batch of data. Thanks to this trick, we can use very small bitmaps without having to worry too much about false positives: if a name is being reported as present in the set while it shouldn’t, it is very unlikely to happen again after a key rotation. Using a secure pseudorandom function also prevents attackers from intentionally triggering false positives.

The output of this deduplication filter is eventually stored as Hadoop HDFS files, and finally into HBase for ad-hoc queries. We sequentially run a dozen Hadoop jobs every day on this data in order to compute different reputation scores for IP addresses and domain names. This eventually lets us find domain names that need to be manually reviewed, or, when combined with the output of other models and third-party services, can be automatically blocked.

The need for real time processing
———————————

Running algorithms once a day on the data is clearly suboptimal. If nytimes.com DNS records are hijacked, we need to spot this as soon as possible to protect our customers, not the next day. Furthermore, domain names that serve exploits are also typically only in use for a short period of time. We want to block them while they are still active, and as soon as possible, not after the baton has been passed to another domain.

Enter ZeroMQ
————

ZeroMQ is a popular, battle-tested message transport protocol and networking library, designed for very low and predictable latency, high throughput, and high reliability.

The ZeroMQ library implements, among other things, the traditional pub/sub pattern: a “producer” generates a stream of data that any number of local or remote clients can simultaneously connect to, in order to receive live updates.

After the deduplication process, and in addition to storing the output into HDFS and Hadoop, we are now streaming this data to a ZeroMQ socket.

This brings a lot of benefits:
– Any authorized machine can join and leave the feed, anytime. This allows for instant testing and parallel processing without any setup. Need to quickly look for domain names matching a specific pattern? That can be done directly on a researcher’s laptop.
– Security: ZeroMQ supports strong encryption and certificate-based authentication out of the box, thanks to libsodium.
– Low CPU impact: a single machine can effortlessly consume our stream of preprocessed authoritative log data.
– Low latency: the data is immediately available for consumption.
– No API required: all it takes is a host name and a port number. Just connect to the socket, and you will start receiving formatted data. The ZeroMQ protocol is widely used, and there are readily available client libraries for more than 40 programming languages, as well as plugins for common tools like Splunk.

As soon as a client connects, it gets a live stream of JSON objects that can be processed right away.

{"name":"a60.w22.akamai.net.","owner":"w22.akamai.net.","rr":"23.59.188.41","server_ip":"96.17.144.41","ts":1380671640,"ttl":20,"type":"A"}
{"name":"a60.w22.akamai.net.","owner":"w22.akamai.net.","rr":"23.59.188.64","server_ip":"96.17.144.41","ts":1380671640,"ttl":20,"type":"A"}
{"name":"www.unitedminds.ie.","owner":"unitedminds.ie.","rr":"81.17.254.44","server_ip":"81.17.254.6","ts":1380671640,"ttl":3600,"type":"A"}
{"name":"2-01-2a01-0007.cdx.cedexis.net.","owner":"cedexis.net.","rr":"ie0101-authconsumer001-1094152544.eu-west-1.elb.amazonaws.com.","server_ip":"69.28.180.4","ts":1380671640,"ttl":20,"type":"CNAME"}
{"name":"2-01-2a01-000a.cdx.cedexis.net.","owner":"cedexis.net.","rr":"us0801asw013.tango.me.","server_ip":"69.28.180.4","ts":1380671640,"ttl":20,"type":"CNAME"}
{"name":"www.facebook-fun.com.","owner":"facebook-fun.com.","rr":"facebook-fun.com.","server_ip":"216.69.185.7","ts":1380671640,"ttl":3600,"type":"CNAME"}
{"name":"scs.retail.fidelity.com.","owner":"retail.fidelity.com.","rr":"155.199.36.26","server_ip":"192.223.177.53","ts":1380671640,"ttl":60,"type":"A"}
{"name":"a3.dscg10.akamai.net.","owner":"dscg10.akamai.net.","rr":"204.2.145.186","server_ip":"96.17.144.40","ts":1380671640,"ttl":20,"type":"A"}
{"name":"hiphotos.wshifen.com.","owner":"wshifen.com.","rr":"185.10.107.162","server_ip":"180.76.8.100","ts":1380671640,"ttl":300,"type":"A"}

Screen Shot 2013-10-02 at 2.50.16 PM

Building streams out of streams
——————————-

A simple use case of this stream is keeping track of new domain names, or rather domain names that we didn’t see traffic for before, or that didn’t resolve to any IP address until now. In order to do so, we once again use Bloom filters that keep track of unique domain names. To provide a sliding window, we simply use a ring buffer of seven Bloom filters, that we are shifting once the most recent filter gets more than one day old, or holds more than 25 million entries. The output of this consumer is another ZeroMQ stream, that we can use to inspect new web sites as soon as they are discovered.

In addition to tracking new domain names, we simultaneously run another consumer tracking new (domain name, IP) tuples.

Our IP reputation systems
————————-

Every day, we run three Hadoop jobs to assign reputation scores to IP addresses. The first is a bayesian average of the number of known malicious domains found on a given IP address. The second is the secure rank score. The third is based on the amount of “disposable” domains that an IP address has been hosting compared to the amount of stable domains seen on the same IP. (We’ll discuss this score in more detail in an upcoming post.)

We use these IP reputation systems to build lists of IP addresses that have been serving a lot of domain names, all of them being known as controlled by cybercriminals.

Putting the pieces together
—————————

Since we already have a stream of new (domain name, IP) tuples, new domain names resolving to one of the highly suspicious IP addresses can be immediately blocked. While the ZeroMQ library itself is fast and provides latency guarantees, the actual producer and consumer code also needs to be equally efficient to process the data at the same rate as it is received.

This was a good opportunity to try Rust, a modern programming language by Mozilla Research aiming at being a safe replacement for C++. Minor changes to the ZeroMQ bindings had to be done in order to make them compatible with the latest Rust version. But overall, our experience with Rust has been absolutely amazing.

We contributed our changes to the ZeroMQ bindings and open sourced our Bloom Filter implementation on Github.

Towards a more real time architecture
———————————————

When it comes to blocking malware, every second counts. A speedy resolution is the only way to limit the number of compromised machines, so models based on stream processing are insanely useful.

We are not going to get rid of our pink friend any time soon,but he just got a new buddy.

Previous Post:

Previous Article

Next Post:

Next Article

Follow Us

  • Twitter
  • Facebook
  • LinkedIn
  • YouTube

Footer Sections

What we make

  • Cloud Security Service
  • DNS-Layer Network Security
  • Secure Web Gateway
  • Security Packages

Who we are

  • Global Cloud Architecture
  • Cloud Network Status
  • Cloud Network Activity
  • OpenDNS is now Umbrella
  • Cisco Umbrella Blog

Learn more

  • Webinars
  • Careers
  • Support
  • Cisco Umbrella Live Demo
  • Contact Sales
Umbrella by Cisco
208.67.222.222+208.67.220.220
2620:119:35::35+2620:119:53::53
Sign up for a Free Trial
  • Cisco Online Privacy Statement
  • Terms of Service
  • Sitemap

© 2023 Cisco Umbrella