Researchers and scientists use data visualizations to better understand data and communicate results. Good visualizations can provide insight into a dataset that might otherwise be overlooked. In this post, we’ll go through the process of creating graphic insight from an abstract dataset by building an actual data visualization step-by-step.
First: The Data
To build this visualization, we start with a 10-minute log chunk of raw DNS data that gets dumped into an Amazon S3. Log chunks are the rawest form of data the OpenDNS research team uses to do analysis, and they make for a good place to start talking about the life of a data visualization. If you want to know more about the process of getting log chunks, check out this post from Josh Pyorre.
Log chunks are text data that won’t be useful for our visualization without some cleaning and parsing. The goal of the visualization is to see what the traffic looks like when connecting resolver requests based on the order they were received. To create latitude and longitude coordinates with IP addresses, I used the Maxmind API available via pip. This is all put together in a browser-consumable .csv file (see some of the python code below).
Next: The Visualization
To make the query patterns visualization, I used Three.js, a JavaScript API used to create 3D computer graphics in a web browser. Because Three.js uses the GPU, which is made for handling pixel processing, it’s perfect for making visualizations that use a lot of data.
After a first step of creating a scene with lighting, next a globe on which all of the lines will sit should be created, which is easy in Three.js. To help make your planet look more realistic, you can find earth textures that contain elevation/bump maps with a quick Google search; GPUs can then sample those 2D texture images and project the texture on your 3D globe object using UV mapping. You can use as many textures as you like to render your earth replica, but only basic earth texture mapping is required. For the sake of clarity, less is definitely more here; the simpler your graph or chart is, the more likely its insight is to be easily understood.
After finding appropriate textures, you can use the handy ImageUtils and loadTexture method to load the image, making sure to pass in the THREE.UV Mapping object. Then, Three.js will map your textures onto a sphere geometry with dimensions you select to create your planet.
