eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
It’s entirely possible that the U.S. government has undergone—and is still undergoing—the harshest, most potentially devastating cyber breach in the short history of digital information.
Reuters broke the story last week that foreign entities—the National Security Agency and FBI have identified them as the Russian hacking group APT29, also known as Dark Halo or Cozy Bear (logo pictured)—had infiltrated several federal IT systems, including the Pentagon, National Institutes of Health, Homeland Security and State Department. This has been confirmed by highly regarded security companies that include CrowdStrike, FireEye, Volexity and Microsoft, for starters.
This was not a cyberattack per se. The perpetrators didn’t smash into these super-important systems; they slid into them on the tails of normal software updates that hundreds of IT managers activated themselves. The targeted update is from SolarWinds, which is getting a lot of grief for these issues; however, the fact is that any one of hundreds of similar applications used by the government could have been used in the same manner. The hackers inserted malicious code into SolarWinds Orion software updates that were pushed out to nearly 18,000 customers. Now untold terabytes of stolen data could well be in the hands of U.S. enemies.
SolarWinds has been hacked previously
A second hacking group, different from the suspected Russian team now associated with the major SolarWinds data breach, also targeted the company’s products earlier this year, according to a security research blog posted on Dec. 19 by Microsoft.
“The investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor,” the blog said. So these are ongoing threats.
SolarWinds is a ubiquitous monitoring/network management tool. Per Gartner, as quoted here, SolarWinds is the No. 3 provider of IT operations software, behind only Splunk and IBM. That, combined with SolarWinds’ low corporate profile, likely made it an attractive target for the hackers.
“We don’t think anyone else in the market is really even close in terms of the breadth of coverage we have,” CEO Kevin Thompson said on an earnings call in October 2019. “We manage everyone’s network gear.”
Its wide usage has turned into a serious problem for the Austin, Texas-based company. SolarWinds is a 21-year-old publicly traded monitoring and network management vendor with 300,000-plus customers across the world. It’s a familiar tool for IT operations and monitoring teams across enterprises big and small.
These things have consequences; Thompson was replaced in his job on Dec. 9 by Sudhakar Ramakrishna, the former chief executive of Pulse Secure. To no one’s surprise, SolarWinds also is looking for a new director of security.
The Wall Street Journal summarized what happened: “In the latest incident, hackers appear to have gained a foothold in their victims’ networks by adding ‘back door’ code to SolarWinds Orion software, according to an analysis of the event by Microsoft Corp. Once installed, this software connected to a server controlled by the hackers that allowed them to launch further attacks against the SolarWinds customer and to steal data. The vulnerable updates were delivered to customers between March and June, SolarWinds said.”
The Economist described the hack in stark terms: “Hackers have vaulted into the heart of America’s government.”
This supply-chain attack began months ago—probably in March 2020—when a highly resourceful and determined hacking operation infiltrated SolarWinds’ update build system, and then used it to compromise a routine patch to the highly popular Orion observability tool, hosted on SolarWinds’ own website.
For now, the attack seems localized to roughly 425 of those 18,000 SolarWinds customers, but this number seems sure to grow. For example, the U.S. Department of Homeland Security isn’t wasting time; it has issued an emergency directive to government organizations to check their networks for the presence of the Trojanized component and report back.
How can IT operations leaders avoid this in the future?
If you’re an IT operations leader or even a CIO at one of the affected customers, the news can’t get worse than this. If you’re a customer that hasn’t been affected, you’re thanking your stars and vigorously assessing the other tools in your environment for similar risks.
By the way, Oracle customers need not worry about all of this. SVP Deborah Hellinger on Dec. 21 sent eWEEK the following statement: “Oracle does not use any SolarWinds Orion product as part of any Oracle product or cloud service. Oracle has no deployed instances of affected SolarWinds product versions in its corporate network, and our investigations have found no suspicious activity or indications of compromise.”
So how can you keep this from happening in your organization?
It’s critical to note that SolarWinds Orion is an on-premises product, requiring local resources to install and manage. So in addition to the extra resources (hardware and people) on-premises products require to keep them working, these products carry a whole host of security risks that you and your IT leadership should carefully evaluate.
Here are four key considerations, offered as industry information best practices from William White, security and IT director of BigPanda, for IT operations leaders and stakeholders when evaluating the security of their monitoring, observability and IT operations tooling investments.
- “Supply chain” infiltration risks: When you select a SaaS solution, you retain control by deciding what data you send to the SaaS provider—you don’t need to install complex software locally that could potentially access other systems and data within your corporate network. That’s not the case with on-prem tools. And this was a major problem with the SolarWinds “supply chain” hack. The attackers were able to use the compromised patch to infiltrate other systems, and it’s likely that they used those systems to infiltrate even more systems.
- Elevated permissions and privileged accounts raise risk: With SaaS-based software, you don’t need to install complex third-party software within your on-prem network. With on-prem software, however, you often have to grant elevated permissions or highly privileged accounts for the software to run, which creates risk.
- Compromised patches: With SaaS-based software tools, you don’t have to review vendor patches or hotfixes being pushed to your SaaS-based deployment. That appears to have opened the exposure for SolarWinds Orion when the software build system was compromised starting with the build for version 2019.4 HF 5. Ironically, the most exposed SolarWinds customers were the ones that were actually diligent about installing Orion patches. Anyone running a down-level version of Orion wasn’t impacted by this hack. Sadly, this is an example of IT shops choosing an on-prem solution, “doing everything right” in terms of staying up-to-date on patches and belatedly finding out that those actions actively put them in greater danger.
- Safe harbor for malicious code: With a SaaS solution, you don’t have to exclude directories or policies from antivirus and anti-malware scans. Yet that is how the SolarWinds exploit seems to have avoided detection, since the malicious code had a safe harbor to help it avoid detection.
Security benefits of SaaS-based monitoring, observability and IT operations tools
Well-designed SaaS-based tools provide a superior level of security in almost all cases, White said. Operations leaders and stakeholders considering IT monitoring, observability and IT operations tools should consider the following:
- SaaS-based architectures can be modern and secure: Most SaaS providers use a modern, secure architecture that compartmentalizes data, security, and identity and access management into different cloud accounts. Access to critical systems is restricted to staff based on the “principle of least privilege,” and user access often requires multifactor authentication using a code generator.
- SaaS-based data is encrypted: Data at rest is generally encrypted by default. Encryption keys are secured in a Key Management System (KMS), where keys are encrypted. This helps secure customer data in the rare event of a rogue application gaining access to the SaaS solution.
- SaaS-based tools use information compartmentalization: As discussed above, with on-prem software, when bad actors gain access into your network environment, they can potentially read any of the data flowing on that network. They often move laterally, from one device, host, application or service, to another, and extract everything they can. By design, that’s just not possible with SaaS-based tools.
Combating phishing and social engineering: What’s the plan?
Last but not the least, while we still don’t know for certain how SolarWinds’ build process was compromised in the first place, a filing by the company suggests that an employee’s Office 365 account was possibly compromised, White said.
This is a common attack vector by hackers for data exfiltration and ransomware, which is conducted through a social engineering or phishing attack via email.
While anyone in the organization can be compromised, your most critical staff—think CEO, CIO, head of engineering—are often targeted by hackers since they are likely to be the busiest and have the least amount of time to inspect emails for phishing attack indicators, and they have access to the most informative and critical systems, White said.
An ongoing security training program for all staff, including realistic phishing simulation tests, is key to raising awareness to reduce the risk from social engineering attack vectors, White said.
But in this instance, such a program would not have helped affected organizations because their exposure came from a third-party vendor. That’s why today, every single organization and every group within every organization must have a robust plan in place to combat the risk of phishing attacks and social engineering—both internally and with their vendors. Untrained users are often the weakest link in an otherwise strong environment.
Moving from on-prem to SaaS is easier than in the past
Before cloud computing and SaaS solutions became prominent, on-prem software was the standard for large enterprises and institutions. On-prem applications were considered reliable and secure, and provided customers with a level of control over the infrastructure and data. Often, those customers built dedicated, tightly controlled data centers to host those applications.
During the last few years, that has started to change. Here are some of the ways enterprises that historically preferred on-prem software solutions are adapting to a SaaS-first world as part of their digital transformation initiatives:
- Requirements due to industry regulations: Some companies were required to deploy on-prem solutions because they were in highly regulated industries or subject to additional privacy constraints or compliance reporting, such as healthcare and utilities or when working with the federal government. But that is changing: Large healthcare, utility and financial customers have started to embrace cloud- and SaaS-based solutions. Even the federal government has recognized the benefits of SaaS solutions through its FedRAMP certification program.
- Software licensing preferences: Other companies preferred the licensing model of on-premises software, where they could purchase a perpetual license, deploy the software in-house, and then optimize for performance by provisioning systems and network resources in their data center.
- With SaaS-based tools, customers can typically scale their licenses up or down based on the number of users, the amount of data consumed or processed, the number of API calls, etc., which affords significantly higher operational and cost flexibility. On top of that, with SaaS-based tools, as long as users have access to a web browser and a standard high-speed network connection, the IT department doesn’t need to waste resources and time on optimizing for performance, with load balancers, HA/DR clusters, etc.
- Uniqueness of the business: Many large enterprises believed their business was so unique that only on-prem solutions allowed for the extensive customization required to support their business. They also put in place expert teams who could be responsive to internal customers that were reliant on their heavily-customized on-prem solutions. But as IT applications and systems have become increasingly complex, having a large bench of specialists on staff, with in-depth knowledge and expertise at all levels of the technology stack, has become more difficult and expensive. Additionally, heavily customized applications can become difficult to manage, support and update—because updates and patches can potentially interfere with those customizations.
- Today, many enterprise solutions used by IT departments and different business functions (marketing, sales, operations, etc.) across most industries have moved toward standard operating procedures and away from customized business processes. This movement to standards and best practices makes it easier for companies to make the switch from custom on-prem solutions to SaaS-based tools.
In summary
Customers compromised their systems unwittingly by following standard best practices—downloading and installing updates and patches based on their vendors’ recommendations, which highlights the unacceptably high-security risks associated with on-prem software.
Victims of compromised on-prem software pay the price both publicly and behind the scenes, White said.
IT operations leaders and executives should consider adopting modern SaaS-based tools for monitoring, observability, event correlation and automation and collaboration. Such SaaS-based tools significantly mitigate the security risks that come with on-prem software, while delivering on the benefits associated with SaaS-based products—such as elastic scaling, lower TCO and rapid time to value.