Poseidon: Real-Time HTTP Log Analyzer

OpenDNS is known for being a cloud-delivered DNS security company, analyzing around 70 billion DNS requests per day. We also monitor about 10.1 million daily HTTP traffic requests on our proxy, which is maintained by our awesome Cloud Enforcement Team. Since this is a relatively new data set for us, building new data pipelines and processing proper metrics is an important step to gaining visibility and building intuitions about the data. With this in mind, we have created Poseidon, a statistics tool to monitor the network behavior on our proxy.
The primary purpose of Poseidon is to process the HTTP logs ingested from our Kafka stream and provide meaningful metrics in order to gain actionable insights over a sliding one-hour window. Some of the statistics it displays are top non-blocked domains, blocked URLs counts, and a trending system over the past hour (updating every five minutes).
Poseidon’s trending system is based on time series analysis algorithms. We take the relative average standard deviation per domain over the last hour, then sort through these domains by trending score to obtain the top trending domains and URLs in the observed window (60 minutes), and identify items with the sharpest trends for deeper inspection.
Figure 1 is an overview of how Poseidon’s windowing system works:
(Figure 1)

Poseidon provides supplemental data for our researchers by retrieving classification data from Investigate, for example specific threat attribution scores (e.g., ASN scores, IP scores, and DGA scores). We have also created a display for time series data in a histogram to see the traffic behavior within the time window. Poseidon also monitors and generates statistics for OpenDNS’s new IP blocking feature.
Figure 2 is an example of time series histogram, where we can observe the overall counts over the last 1 hr. along with each 5 min. intervals, this helps with identifying sharp changes in traffic and network behavior:
(Figure 2)

Some of the other fields in the proxy traffic that we are conducting analysis on are the HTTP Referer and User-Agent fields. For example, if there was a specific referer sending users to multiple malicious pages, this may be suspicious. We have deployed a set of rules to match malicious user-agent strings, and have also tied in useragentstring.com’s API to help identify and analyze unknown user-agents.
Figures 3 and 4 are screenshots of Poseidon displaying top-counts and displaying overall trending. From this data we can examine user behavior, trends in traffic, and identify outliers in the data:
(Figure 3)

(Figure 4)

The next step for the development of Poseidon is to rebuild it with an analytics platform like Apache Spark to distribute the computations as we deploy more rules, build HTTP detection models, and harvest more network statistics. In addition, we can proxy traffic for domains found from our DNS models and gain more information about them at the HTTP level. This will also be attached to our email alert system in order to send out daily digests of proxy statistics to the team for more in-depth analysis.
Using Poseidon, OpenDNS Security Labs increases its ability to detect new and emerging threats. As we discover suspicious domains from our proprietary DNS models, we can selectively proxy the traffic to gain greater insight – including subdomains, paths, filenames, and file extensions. Any identified indicators can then be fed back into our existing threat models, training sets, and alerting systems. The discovered indicators may also serve as the basis for the creation of new threat models or an entirely new vein of research.
These are exciting times in the OpenDNS Security Labs. Keep checking back with us as we continue to discover new and interesting threats using our innovative systems.