Today, OpenDNS runs many datacenters worldwide, providing content filtering and security services to millions of users. The Network Engineering team continuously ensures the Internet pipes are well connected and packets are flowing as quickly as possible between our users and servers.
One of the systems my team has worked on, and continues to maintain, is OpenDNS’s HTTP proxy. At its core is Nginx. A cool thing about Nginx is how much you can get done just at the config level, but for maximum flexibility moving business logic into modules (C, or maybe Lua and now Javascript) is the way to go. But custom modules equals custom code, and custom code equals testing and debugging.
We test every feature, the developer is responsible for both the feature code and the associated tests. This ranges from unit tests, to integration, and to continuous monitoring in production. Any test that can be run continuously, is. We’ve added another neat feature to Nginx and dubbed it “X-Ray.” It allows us peek into the code flow for a given a request. Then, for example, if a customer has an issue, developers can mimic the scenario with X-Ray to aid debugging without having to shuffle routing configurations or restarting processes.
Normally the error log is expected to have low verbosity in production, with only critical errors being printed to disk to keep performance as high as possible. With X-Ray we basically get the verbose version of error log (for custom code, not the nginx core) inline, appended to the response body.
Here is a very simple redacted example;
Normal request:
> GET /malware.htm HTTP/1.1 > User-Agent: curl/7.24.0 > Accept: */* > < HTTP/1.1 302 Moved Temporarily < Connection: keep-alive < Location: https://malware.opendns.com/... < <html> <head><title>302 Found</title></head> <body bgcolor="white"> <center><h1>302 Moved Temporarily</h1></center> <hr><center>Umbrella Cloud Security Gateway</center> </body> </html>
And with X-Ray, the response has additional info:
< HTTP/1.1 302 Moved Temporarily < Connection: keep-alive < Location: https://malware.opendns.com/... < 8 init_xray: xray level: 9 (NGX_LOG_DEBUG=8) 8 .... : ==> preparing 000000000383E100, "GET /malware.htm HTTP/1.1" 8 ... : no session args 8 ... : header: "User-Agent" 8 ... : header: "Accept" 8 ... : header: "Host" ... 8 ... : ==> evaluating 000000000383E100, "GET /malware.htm HTTP/1.1" 8 check_auth: pass: .... 8 check_auth: ... ... 8 ...: uri-escaped user-agent: "curl/7.24.0" 8 ...: block: as cat '1' 8 ..._redirect: Location: "https://malware.opendns.com/..." ... <html> <head><title>302 Found</title></head> <body bgcolor="white"> <center><h1>302 Moved Temporarily</h1></center> <hr><center>Umbrella Cloud Security Gateway</center> </body> </html>
The code can now be full of useful information, from critical to very verbose all instantly accessible without hindering performance.
And best of all, you can have this feature as well! Check out nginx-xray on our Github account and get coding!
