Nine Reasons Linux Rules the Supercomputing Space

by Darryl K. Taft

In 1994, the first Beowulf Cluster was built at NASA, using Linux, as an alternative to the very expensive HPC supercomputers. “Beowulf Clusters are scalable performance clusters based on commodity hardware, on a private system network, with open-source software (Linux) infrastructure. The designer can improve performance proportionally with added machines. The commodity hardware can be any of a number of mass-market, stand-alone compute nodes as simple as two networked computers, each running Linux and sharing a file system, or as complex as 1,024 nodes with a high-speed, low-latency network.”

Until a few years ago, UNIX variants such as AIX, HP-UX, Tru64 UNIX, Solaris and IRIX ruled. The historic cost of HPC or “supercomputers,” including hardware and variants of the UNIX OS, had limited the use of HPC technologies to market segments that could afford expensive systems, and they have been mainly used in academic research, fluid dynamics, oil and gas exploration, computer-aided design and testing, and pharmaceutical and military research. However, Linux has displaced many of these systems in the HPC space. Relatively suddenly, Intel and AMD replaced RISC processors, and the way was paved for Linux to unseat UNIX as the dominant OS, and numerous second-tier vendors established positions in the market.

The evolution of both lower-cost hardware and Linux has dramatically reduced the cost of the HPC systems. The licensing cost of a custom, self-supported Linux distribution is the same, whether you’re using 20 nodes or 20,000,000 nodes. But most organizations, whether in academia, government or commercial environments, have to rely on a 24/7 vendor-supported and certified OS. Even in these cases, Linux reaps significant savings in both up-front subscription and support computing.

While low price was a key argument for Linux in the past, the market has changed, and today customers and organizations are adopting clusters based on Linux to ensure constant uptime, while still leveraging the flexibility and reliability of open source. Reliability and scalability are the features most commonly cited as reasons for choosing Linux clusters. Scalability allows organizations to start with a small cluster and seamlessly add nodes as processing demands increase.

There are a lot of resources available for HPC on Linux; many of them free. Look at the success of other open-source projects, and you see the epitome of strength in numbers.

Setting up and monitoring a cluster can become very difficult; this is especially true for heterogeneous environments that are dependent on multiple generations of technology that support a variety of applications and multiple user groups. Linux on a cluster of commodity x86 servers is much more economical, in part because Linux clusters have become easy to set up and simple to manage, with vendors and tools specifically built for HPC use cases.

Linux has steadily incorporated HPC features over the years. The HPC market is where vendors test out the ideas that will drive tomorrow’s commercial products, such as multi-core processors or power efficiency. The open-source OS very often spearheads technical innovations, a significant role as HPC turns more and more into “high-productivity computing.”

As demand for processing power and speed grows, HPC interests businesses of all sizes, particularly for transaction processing and data warehouses. Commercial HPC use occurs in Hollywood special effects, financial analytics and many other industries. For a growing number of users and vendors, HPC refers not to cores, cycles or flops anymore, but to discovery, efficiency or time to market. For example, an important category within threshold applications is “ultra-scale business computing,” where applications are not traditional HPC workloads, but they require supercomputing levels of scale, thus resulting in using HPC-specific solutions or technologies.

One of the most powerful supercomputers in Spain, the MareNostrum, also considered among the most attractive supercomputers ever, runs on Linux. MareNostrum is a supercomputer in the Barcelona Supercomputing Center, the second most powerful in Spain and one of seven supercomputers in the Spanish Supercomputing Network. It was presented by IBM and Maria Jesus San Segundo, the Spanish Minister of Education and Science.