For many companies, a nightmare scenario is to find that all of their network traffic is suddenly in the hands of an unfriendly power. This happened to Google on Nov. 12, when an employee at a small Nigerian internet service provider configured the border gateway protocol (BGP) filters of one of its network devices so that Google’s traffic went to Nigeria, passing through Russia and China on the way.
The configuration was fixed in a little over an hour, but in the interim, Google’s internal networks were sending their traffic on a world tour. This impacted Google search, as well as the operations of Google Cloud (see image; right-click on it and select “View Image” to see a larger version), and from there the operations of Google’s customers who use its cloud services.
What happened is that MainOne Cable, the Nigerian ISP, was performing a routine software update when the configuration error happened. At that point, a router began advertising to the internet that it was the appropriate pathway for Google’s traffic. ISPs in China and then Russia saw the advertisement and acted on it, which ultimately meant that Google’s traffic, instead of going to Google, went to Russia, where it was passed to China, where it mostly died.
Google’s Data Didn’t Go Anywhere in China
Chinese edge routers that are part of the “great firewall” of China, simply dropped the packets as being unauthorized. Google, and users of its services, simply lost their connections. The initial fear was that Google’s network traffic has been hijacked, but a later investigation revealed that it was simply human error. The investigation also showed that MainOne had not implemented any protections to ensure that its BGP advertisements were proper. As it turned out, neither had the internet services in Russia and China.
Since then, MainOne has fixed the problem and instituted the necessary protections. But that doesn’t mean the risk is gone. What the cable company did by mistake can easily be done on purpose with the effect of sending the internet traffic of one or more users through some place where it shouldn’t go. And in fact it has been done, most recently by the hackers who work for the Chinese army.
What’s concerning is that the BGP misconfiguration is so easy to accomplish, and it’s hard to fix. Fortunately, you can find out that this is happening by watching BGPMON on Twitter. This service, which is part of OpenDNS, quickly spotted the Google redirection, which, in turn, led to it being fixed quickly.
But as an internet end user, there’s little you can do. You can find out that it’s happening to your traffic by using the Tracert utility, watching the path, then watching the latency numbers. But if your ISP or your own set of IP addresses has been hijacked, the best you can do is to quit using those IPs until the hijacking has stopped.
A Separate ISP Is Another Alternative
Another alternative is to have access through a separate ISP. While implementing for services—such as your primary e-commerce site—might be tricky, keeping access to the internet in general and to your cloud services should be relatively transparent. If your e-commerce site is cloud-based, you may be able to keep running there, too.
Of course, such a failover strategy is something that you’ll have to arrange in advance, but it will have more uses than just a BGP hijacking. Your path to the internet can be interrupted by everything from a DDoS attack to a faulty router configuration.
The other step that’s necessary is to make sure that your data is protected. Google wasn’t worried about data loss when the BGP problem happened, because all of its data is encrypted. This can be what saves your company as well. Another step is to use VPNs (virtual private networks) for any data that’s important.
Using a VPN will make sure that the data is encrypted, but it will do more than that. If the network address advertising is wrong, the VPN simply won’t connect, and data won’t be transferred at all. This happens because when you set up your virtual network, you also define a specific IP address on the other end. If you’ve set up your VPN properly, any attempt to change the spot where it terminates simply won’t work because the address isn’t right.
Monitoring Networks Should Be a Given
Of course, you should always be monitoring your network, and not just because of the chance of a network hijacking attempt, intentional or otherwise. A decent network monitoring application will spot changes in your network addressing and alert your IT staff. The Spiceworks Network Monitor can handle tasks such as this, and it’s free, easy to understand and effective.
With an effective monitoring service in place, you will know almost immediately when something that adversely affects your network happens, whether it’s a BGP configuration problem, the appearance of a rogue WiFi access point or an unauthorized user on your internal network. Just watching the latency numbers will tell you that something is amiss.
Border gateway protocol is a legacy of the early days of the internet, when most actions were based on trust. Unfortunately, in this age of malware and spies, trust is a thing of the past, so you’ll need to have a way to confirm that what’s happening on your network is what you want to happen. That challenge will only get more important.