Over the past few decades, the tech world has been investing quite a bit of time and effort into the creation of robust distributed cloud systems, and the increasing need for resources and performance leads us to a decentralized agglomeration of data acting transparently like a whole. This new face of the internet as we know it involves many actors: some focus on storage, some may focus on routing, some are in charge of keeping the information up-to-date and finally, the users of such a system are the key agents that constantly create fresh content. We are building organized structures that are beginning to function more and more like primitive digital brains designed to perform specific tasks.
Visualization concept
These shared knowledge databases have brought a new challenge: how do we monitor the current state of such a dynamic system?
Indeed, it is not an easy task. Modern systems include billions of elements and in most cases users can add, delete, and modify items at any time. And because the data is decentralized, it can be pretty tricky to make sure we’re looking at an up-to-date version of the system. Last but not least, in many situations, we are interested in not only visualizing the current system, but also the evolution of that system over time. Depending on how the databases are built and how the various modifications are stored, this task can become extremely complex in terms of memory usage, processing time and data consistency.
Furthermore, what about the visualization perspective? How do we choose a layout that works for our needs? That is, how do we design a visualization technique that will be able to represent a structure constantly evolving over time?
Of course, we can’t answer those questions all at once; in this article we will focus on one part of the problem. We propose an idea that will help us form a visual representation of a general data flow inside a graph model. The basic idea is very simple indeed: each graph edge has an “Activity” attribute defined as a float value.
Then we setup a particle system to make small textures fly from the first edge node to the second one. Finally we use the “Activity” value to set the speed of each edge particle.
The result is an interactive representation of the activity of the whole system we want to monitor, giving us an easy way to spot the features of the global data flow.
Depending on the system we are trying to monitor, this can be applied to many different problematics:
- Network traffic
- Chat communication inside a social network
- Users page transitions in a web site (Markov chain analysis)
- Average path in a tree / graph (Decision tree monitoring)
- And many others…
Demo: A network activity simulation
Today we present a real-time simulation in WebGL showing a global edge activity in a graph model. For demonstration purposes, the data set and activity are randomized to illustrate the proof of concept.
Controls
- Hit play/pause to control the animation
- Navigate with the mouse and arrows (FPS view)
- Double-click on a node to zoom in and switch to a spherical view.
- Click on an empty area to go back to standard FPS view.
- Switch through the different layout modes in the menu.
- Drag and drop nodes with the mouse to play with the physics engine.
A bigger dataset in action
Of course, this article wouldn’t be complete without a concrete application on our Security Graph.
Here is what the same process looks like on a related domain graph extracted from a given domain.
To put it in a nutshell, when two domains are requested within a small time frame (1 second), we add a “related domain” link between them. This helps us understand which actors and services are at play when we access a given domain.
For example, here are the “www.youtube.com” related domains (click to enlarge):
You may also notice that we count each related domain appearance. We can then calculate a normalized probability which we use to set the edge activity. In other words, it is like visualizing the markov chain of the related domains.
(These are gif animations. Click on them to open a new tab.)
This is what it looks like on a graph containing 1,455 nodes and 2,187 edges.
And the same result on a bigger graph: 11,779 nodes, 16,470 edges.
Conclusion
In conclusion, monitoring a dynamic system leads to many new challenges, and not only in terms of graph activity. It opens the door to several new approaches and brings a lot of fresh questions. We’ll take a closer look at questions like these in the weeks to come:
- How do we extract the global state of the graph activity?
- What patterns can we find to help us improve the graph state?
- And last but not least, how do we monitor and apply automatic retro-action on the system?
