Here at OpenDNS we’re always working on expanding our global footprint and adding datacenter locations around the world. Understanding why this is important is easy when you know two things:
- The more servers and locations we have in operation, the more capacity we have.
- The closer you are topologically to an OpenDNS server, the faster your DNS.
Today we’re announcing a new datacenter location in Frankfurt, Germany. Adding to our existing London and Amsterdam locations, the Frankfurt location helps make sure our DNS service is the fastest available to Internet users in Europe and provides much more overall capacity to European users (50% more).
Setting up a datacenter for the largest open recursive DNS service in the world is actually a pretty fun part of the job since we don’t get to rack and stack gear very often. While we have brought new sites online without ever setting foot onsite, by drop-shipping gear to a remote site and bringing it up from HQ, I was looking forward to a trip to Germany and volunteered to go and do the install myself. As icing on the cake, David surprised me with a first class upgrade, which was an awesome experience.
Doing a remote install? Some travel tips…
I’ve come up with a few tips for fellow operations folks who are doing remote installs:
- Have a solid deployment checklist of everything you want at the site. That doesn’t just include routers, switches and servers, but everything you might need including screws, cage-nuts, power cables, extra power cables for your laptop, patch cables, couplers, adapters, SFPs, etc.
- Set up all your power at the datacenter and make sure it’s working before you leave. Make sure you send your remotely manageable power distribution unit ahead of time. Power plugs and configurations differ country by country and you want to make sure your power is lit and working long before you fly somewhere to do an install. Despite the increased cost of a remotely managed power distribution unit, it will pay for itself after 1 or 2 remote-hands charges from your datacenter provider.
- If you can avoid it, don’t book a flight until your gear has cleared customs. Customs handling time varies by country and there is nothing worse than arriving on-site and not being able to access your gear. We generally allocate about a week for customs in Europe, and a bit longer in Asia. In more developing nations, it can vary wildly from days to months.
- Always plan for extra days. We prefer to schedule the installs to start on a Saturday or Sunday and then go into a Monday or Tuesday. This way we have the weekend when the datacenter is not busy to do all the physical racking and stacking. The on-site folks are more forgiving about making a mess usually, and often even offer to lend a hand. By the time Monday morning comes around, everything is racked and powered, and we can now work with the various networking providers to bring circuits online with the regular on-duty techs. There’s nothing wrong with finishing your install on a Monday and spending a couple days sightseeing. Having the extra time gives you some buffer to fix any major hiccups that may happen, and to source a replacement part locally.
- The last tip is probably the most important. Take photos along the way, and at the end. We love having photos of our install, which we archive in our Wiki. This way, if we ever need to request remote-hands work from techs in the future, we don’t simply have to rely on labels on gear to figure things out. We can actually describe the location of equipment or annotate instructions for them on complicated remote-hands work.
I’ve included some photos of our install for you below. Hope you enjoy the photos, and our new Frankfurt datacenter!
Our datacenter… Just kidding. 🙂
A photo from the datacenter showing the power working before we fly out.
A mostly complete install. Lots of room for adding more servers.
Good cable organization, and labeling is critical.
Thanks for reading this far – now ditch your boring job and come work with me so we can add more datacenters and continue building the greatest DNS service in the world.
George Patterson is the Director of Operations for OpenDNS and one of the company’s earliest employees. Every day George and his team manage 15 datacenter locations around the world that collectively handle 30 billion DNS queries daily. OpenDNS has never had downtime. George likes to hire ops people who can code.