eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Breaches occur on an-all-too-frequent basis, but what is often never reported are the breaches that don’t happen, thanks to organizations taking rapid, proactive measures. One such incident was outlined by Shopify at KubeCon + CloudNativeCon NA 2018 last week.
Thanks to a bug bounty program and the support of its vendor partner Google, Shopify was able to avoid a potentially disastrous flaw that could have enabled an attacker to take over Shopify’s Kubernetes cluster. Shopify provides an e-commerce platform that allows vendors to sell goods and services. The platform is hosted on the Google Kubernetes Engine (GKE), which provides a hosted version of the open-source Kubernetes container orchestration platform.
“If you’re not familiar with Shopify, we’ve got about 600,000 businesses, so there’s a good chance that you’ve purchased something from us without even realizing it,” Shane Lawrence, security infrastructure engineer at Shopify, said. “We processed about $26 billion last year, and during peak hours we get approximately 80,000 requests per second.”
Shopify runs entirely on GKE, said Lawrence; the reason his company chose Kubernetes is to be able to rapidly respond to scaling demands like the recent Black Friday and Cyber Monday shopping events.
“With Kubernetes we can scale down to a single replica if we need to test things, or we can scale up to hundreds across multiple geographical locations,” Lawrence said. “It’s important to us that our application developers spend their time developing applications, not becoming Kubernetes experts.”
To that end, Shopify built a self-serve platform where organizations can literally go in and press the create cloud runtime button and in minutes have web applications serving traffic running on Kubernetes. The Shopify system uses guardrails to warn organizations if they’re doing something that Shopify thinks might not be best practice or might not be secure.
Bug Bounty
While Shopify has done its best to make the platform as secure as it can be, flaws are an unavoidable part of modern software. To help identify unknown flaws, Shopify makes use of a managed bug bounty program on the HackerOne platform. With a bug bounty program, security researchers are rewarded for responsibly and privately disclosing flaws.
“We recognize that it would be infeasible to hire 300 people to sit around all day every day and test every single commit and try to find some vulnerabilities,” Lawrence said. “So instead we just leverage the power of the community, and in doing so we’ve had over 300 hackers over the last three years or so participate in our [bug bounty] program.”
Lawrence said that over the last three years, Shopify has paid out more than $1 million in bug bounties.
The Kubernetes Flaw
Regarding the specific Kubernetes cluster flaw that was detailed at KubeCon, Lawrence said the bug came in at 7:39 p.m. on a Sunday night from security researcher Andre Baptista. Eleven minutes later, at 7:50 p.m., Shopify’s security response team declared that bug was in fact a security incident. At 8 p.m., Shopify’s cloud security and app development teams were fully engaged on the issue, working on a fix.
Lawrence said that just over an hour after the report came in, Shopify’s team made a code commit that disabled the vulnerable feature at 8:43 p.m. By 9:27 p.m., Shopify began the larger effort of investigating the full impact of the bug, cleaning up credentials and contacting Google to make sure nothing had been missed.
For his efforts, Shopify awarded Baptista a $25,000 reward.
SSRF
The security researcher was able to exploit a Server Side Request Forgery (SSRF) to obtain a Google service account token, as well as the Kube-env variable, which provided a Kubelet token, which in turn was used to gain full control of the cluster.
“SSRF is where you convince a web server to make a request on your behalf,” Lawrence explained.
Lawrence said the Google service account and the metadata server that runs with it are used for interacting with other APIs in a cluster. The APIs assume that the token is also being used by other applications running in the same cloud platform, but not to end users, he said. By using the SSRF flaw, the researcher was able to convince the web server to send him the token directly.
According to Greg Castle, Kubernetes security lead at Google, his company had already anticipated that type of attack. The problem was that Shopify was somehow making use of a beta version of the vulnerable API.
Castle said that there was a known issue with the beta API that was fixed when it became stable. The challenge is that many organizations, including Shopify, were using the beta API and had a dependency on it. Castle said that Google has since announced that it will be turning off the beta API by default when Kubernetes 1.12 becomes generally available on GKE. The researcher was also able to get access to Kubernetes metadata leading to the Kube-env variable. Castle said that GKE has an option to conceal metadata, which effectively puts a proxy in between the metadata server and the containers that are running on the machine.
“It filters out sensitive information like Kube-env, and it was actually specifically developed for exactly this style of attack,” Castle said. “That [metadata concealment] was available at the time. Shopify had tried it out, but had some problems with it, so it wasn’t actually running on the cluster that the security researcher had tried. If it had been, it would have prevented this attack.”
Looking forward, Castle said Google doesn’t want organizations to have to opt into security like metadata concealment. As such, the plan is to provide a new method to have each Kubelet bootstrap itself with a cryptographic assertion that comes from a trusted platform module (TPM).
“The idea is that it will be a better way to bootstrap the Kubelet in the future that will replace the static token and fix up this security weakness,” he said.
Generally speaking, Castle suggested that Kubernetes users make sure that Kubernetes service accounts are configured for least privilege, only providing access and privileges for what is needed to function. He also recommended that Kubernetes users follow platform guidelines from their Kubernetes providers for hardening the system.
Sean Michael Kerner is a senior editor at eWEEK and InternetNews.com. Follow him on Twitter @TechJournalist.