Intel Offers More Details of Its Omni-Path Interconnect Fabric

Company officials expect the high-performance technology to be used, not only in HPC environments but also in enterprise data centers.

Intel Omni-Path

Intel in 2012 had many in the industry wondering what the company was doing when, within a span of three months, it spent about $265 million to buy QLogic's InfiniBand technology and supercomputer maker Cray's high-speed interconnect assets.

The chip maker already sold InfiniBand products under the umbrella of its True Scale business, and it had a solid footing in the high-performance computing (HPC) space. Where Intel planned on going with the new products it had acquired was unclear at the time.

However, more than a year ago, Intel officials started mentioning the Omni Scale interconnect fabric it was developing for HPC environments, and in late 2014 said the name had been changed to the Omni-Path Architecture. Since then, more details have trickled out, from its 100G-bps capabilities (four "lanes of 25G-bps bi-directional bandwidth each) to its low port-to-port latency to fast MPI messaging rates.

This week, in a paper released during the HotInterconnect 2015 show, and in a briefing with journalists and analysts the week before during the Intel Developer Forum (IDF) 2015 conference, Intel executives released more details of the high-speed interconnect fabric that the company expects will become the dominant fabric technology for large-scale systems in HPC environments as well as some enterprises.

"It's not just a chip, it's not just a host adapter, it's not just a switch," Hugo Saleh, director of technical computing segment marketing and industry development at Intel, said during the briefing at IDF, adding that it's another example of the company's growing push into other parts of computing systems beyond the chip. "It's an over-arching architecture. ... Where we once were thought of as a processor company … we're now looking at it holistically."

The Omni-Path Architecture is part of Intel's larger HPC Scalable System Framework, an initiative first introduced in April that is designed to offer organizations the technology they need to create HPC-level environments, with a focus on the overall system rather than the components. The fabric technology will begin appearing with the upcoming 14-nanometer Xeon Phi "Knights Landing" chips, which will offer as many as 72 cores and will be able to be used as either co-processors or primary processors.

Intel officials said the fabric technology will address a range of issues within the HPC space, from price and performance to scalability and reliability.

As the number of cores grows, interconnect fabrics become increasingly important to ensure high levels of communications within the system as well as between systems. In traditional environments, networking includes top-of-rack switches, end-of-rack switches and core switches, with data moving between them and the servers. Integrated interconnect fabrics enable communication between the chips and systems without a core switch and top-of-rack switch, with data flowing in a more east-west direction.

One of the key challenges in the computing space is that the cost of fabrics is increasing, whether it's InfiniBand, Ethernet or something else, according to Phil Murphy, chief system architect of Omni-Path. Where once the fabric constituted 10 percent of the system cost, that number now is closer to 20 percent to 25 percent. To get to 100G-bps performance, the percentage could go even higher, to as much as 40 percent, Murphy said during the briefing at IDF.

One of the ways to drive down the cost and increase the price/performance ratio is through integration, he said. When Knights Landing rolls out later this year, the Omni-Path controllers will be on the chip packages, but not integrated onto the same piece of silicon as the chip itself. The two will be sharing a PCIe interface. However, the plan for future generations of Xeon Phi and Omni-Path is to integrate the fabric technology onto the chip, he said.

"Performance is one area where we can improve with integration," Murphy said, noting that integration also enhances everything from density to reliability to power consumption.