• Skip to primary navigation
  • Skip to main content
  • Skip to footer

Cisco Umbrella

Enterprise network security

  • Contact Sales
  • Login
    • Umbrella Login
    • Cloudlock Login
  • Why Us
    • Why Cisco Umbrella
      • Why Try Umbrella
      • Why DNS Security
      • Why Umbrella SASE
      • Our Customers
      • Customer Stories
      • Why Cisco Secure
    • Fast Reliable Cloud
      • Global Cloud Architecture
      • Cloud Network Status
      • Global Cloud Network Activity
    • Unmatched Intelligence
      • A New Approach to Cybersecurity
      • Interactive Intelligence
      • Cyber Attack Prevention
      • Umbrella and Cisco Talos Threat Intelligence
    • Extensive Integrations
      • IT Security Integrations
      • Hardware Integrations
      • Meraki Integration
      • Cisco Umbrella and SecureX
  • Products
    • Cisco Umbrella Products
      • Cisco Umbrella Cloud Security Service
      • Recursive DNS Services
      • Cisco Umbrella SIG
      • Umbrella Investigate
      • What’s New
    • Product Packages
      • Cisco Umbrella Package Comparison
      • – DNS Security Essentials Package
      • – DNS Security Advantage Package
      • – SIG Essentials Package
      • – SIG Advantage Package
      • Umbrella Support Packages
    • Functionality
      • DNS-Layer Security
      • Secure Web Gateway
      • Cloud Access Security Broker (CASB)
      • Cloud Data Loss Prevention (DLP)
      • Cloud-Delivered Firewall
      • Cloud Malware Protection
      • Remote Browser Isolation (RBI)
    • Man on a laptop with headphones on. He is attending a Cisco Umbrella Live Demo
  • Solutions
    • SASE & SSE Solutions
      • Cisco Umbrella SASE
      • Secure Access Service Edge (SASE)
      • What is SASE
      • What is Security Service Edge (SSE)
    • Functionality Solutions
      • Web Content Filtering
      • Secure Direct Internet Access
      • Shadow IT Discovery & App Blocking
      • Fast Incident Response
      • Unified Threat Management
      • Protect Mobile Users
      • Securing Remote and Roaming Users
    • Network Solutions
      • Guest Wi-Fi Security
      • SD-WAN Security
      • Off-Network Endpoint Security
    • Industry Solutions
      • Government and Public Sector Cybersecurity
      • Financial Services Security
      • Cybersecurity for Manufacturing
      • Higher Education Security
      • K-12 Schools Security
      • Healthcare, Retail and Hospitality Security
      • Enterprise Cloud Security
      • Small Business Cybersecurity
  • Resources
    • Content Library
      • Top Resources
      • Cybersecurity Webinars
      • Events
      • Research Reports
      • Case Studies
      • Videos
      • Datasheets
      • eBooks
      • Solution Briefs
    • International Documents
      • Deutsch/German
      • Español/Spanish
      • Français/French
      • Italiano/Italian
      • 日本語/Japanese
    • Security Definitions
      • What is Secure Access Service Edge (SASE)
      • What is Security Service Edge (SSE)
      • What is a Cloud Access Security Broker (CASB)
      • Cyber Threat Categories and Definitions
    • For Customers
      • Support
      • Customer Success Webinars
      • Cisco Umbrella Studio
  • Trends & Threats
    • Market Trends
      • Hybrid Workforce
      • Rise of Remote Workers
      • Secure Internet Gateway (SIG)
    • Security Threats
      • How to Stop Phishing Attacks
      • Malware Detection and Protection
      • Ransomware is on the Rise
      • Cryptomining Malware Protection
      • Cybersecurity Threat Landscape
      • Global Cyber Threat Intelligence
    •  
    • Woman connecting confidently to any device anywhere
  • Partners
    • Channel Partners
      • Partner Program
      • Become a Partner
    • Service Providers
      • Secure Connectivity
      • Managed Security for MSSPs
      • Managed IT for MSPs
    •  
    • Person looking down at laptop. They are connecting and working securely
  • Blog
    • News & Product Posts
      • Latest Posts
      • Products & Services
      • Customer Focus
      • Feature Spotlight
    • Cybersecurity Posts
      • Security
      • Threats
      • Cybersecurity Threat Spotlight
      • Research
    •  
    • Register for a webinar - with illustration of connecting securely to the cloud
  • Contact Us
  • Umbrella Login
  • Cloudlock Login
  • Free Trial
Security

Statistics "Big Data" Architecture

Author avatar of UmbrellaEngineeringUmbrellaEngineering
Updated — March 11, 2020 • 2 minute read
View blog >

The OpenDNS Global Network enforces security worldwide through our 21 data centers, handling 50 billion DNS requests per day. This traffic generates around 300 gigabytes of query logs each hour, which contains information crucial to our customers. Analyzing that amount of data, however, is no easy task. The Statistics Team is responsible for building and maintaining the complex architecture that stores and processes the massive amounts of data collected. The platform enables OpenDNS customers to gain insight into the security of their networks and the security team to study malware infection patterns around the world.
The Statistics infrastructure is responsible for all reports generated within the Umbrella Dashboard. The process begins with collecting the raw query logs as they come in from all of our DNS resolvers, iOS VPN systems, and proxies. Once the logs are filtered for essential information, they are stored in a Hadoop cluster where different MapReduce algorithms run and generate key statistics. Utilizing Hadoop and MapReduce enables a huge dataset to be processed much faster by breaking it into many subsets, processing each subset individually, and then combining the results of each subset. The statistics are then stored in an HBase distributed database for our customers to quickly access. One large cluster is used for processing all query logs to generate report data, which syncs with a backup cluster to keep an active copy as a failsafe. A special cluster was built for the Security Research team, which gives them a sandbox to experiment with new algorithms against real-time data and stay current with the latest types of attacks.

Overview of Statistics Systems
Overview of Statistics Systems

Some recent projects completed by the Statistics Team include real-time analytics and the backend system for our recently renamed Investigate product. Our customers need to see activity within their network as it happens, and waiting for data to be processed doesn’t make that possible. Since it can take up to 40 minutes for all the MapReduce jobs to run, a separate Kafka and Storm cluster was built to enable searching for DNS logs in real time. The OpenDNS Investigate product was initially developed as an internal research tool, but it quickly became evident how valuable it would be to our customers. The Statistics Team took on the task of transferring and scaling a system initially built on a small development box and making it able to handle millions of queries.
The Statistics Team consists mostly of software engineers, but is rounded out with a product manager and several technical support, infrastructure, and front-end engineers. This diversity of roles allows other teams to function without worrying about accurately presenting data. It also gives the Statistics Team the capability to pursue projects of our own. As our 50 million daily users continue to use OpenDNS to safely connect to the Internet, the Statistics Team will be working behind the scenes to process and analyze all of the data that is being generated.

Suggested Blogs

  • Cisco Umbrella Delivered Better Cybersecurity and 231% ROI February 21, 2023 2 minute read
  • Cisco Listed as a Representative Vendor in Gartner® Market Guide for Single-Vendor SASE January 26, 2023 3 minute read
  • How to Evaluate SSE Vendors: Questions to Ask, Pitfalls to Avoid June 23, 2022 5 minute read

Share this blog

FacebookTweetLinkedIn

Follow Us

  • Twitter
  • Facebook
  • LinkedIn
  • YouTube

Footer Sections

What we make

  • Cloud Security Service
  • DNS-Layer Network Security
  • Secure Web Gateway
  • Security Packages

Who we are

  • Global Cloud Architecture
  • Cloud Network Status
  • Cloud Network Activity
  • OpenDNS is now Umbrella
  • Cisco Umbrella Blog

Learn more

  • Webinars
  • Careers
  • Support
  • Cisco Umbrella Live Demo
  • Contact Sales
Umbrella by Cisco
208.67.222.222+208.67.220.220
2620:119:35::35+2620:119:53::53
Sign up for a Free Trial
  • Cisco Online Privacy Statement
  • Terms of Service
  • Sitemap

© 2023 Cisco Umbrella