eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Intel in 2012 had many in the industry wondering what the company was doing when, within a span of three months, it spent about $265 million to buy QLogic’s InfiniBand technology and supercomputer maker Cray’s high-speed interconnect assets.
The chip maker already sold InfiniBand products under the umbrella of its True Scale business, and it had a solid footing in the high-performance computing (HPC) space. Where Intel planned on going with the new products it had acquired was unclear at the time.
However, more than a year ago, Intel officials started mentioning the Omni Scale interconnect fabric it was developing for HPC environments, and in late 2014 said the name had been changed to the Omni-Path Architecture. Since then, more details have trickled out, from its 100G-bps capabilities (four “lanes of 25G-bps bi-directional bandwidth each) to its low port-to-port latency to fast MPI messaging rates.
This week, in a paper released during the HotInterconnect 2015 show, and in a briefing with journalists and analysts the week before during the Intel Developer Forum (IDF) 2015 conference, Intel executives released more details of the high-speed interconnect fabric that the company expects will become the dominant fabric technology for large-scale systems in HPC environments as well as some enterprises.
“It’s not just a chip, it’s not just a host adapter, it’s not just a switch,” Hugo Saleh, director of technical computing segment marketing and industry development at Intel, said during the briefing at IDF, adding that it’s another example of the company’s growing push into other parts of computing systems beyond the chip. “It’s an over-arching architecture. … Where we once were thought of as a processor company … we’re now looking at it holistically.”
The Omni-Path Architecture is part of Intel’s larger HPC Scalable System Framework, an initiative first introduced in April that is designed to offer organizations the technology they need to create HPC-level environments, with a focus on the overall system rather than the components. The fabric technology will begin appearing with the upcoming 14-nanometer Xeon Phi “Knights Landing” chips, which will offer as many as 72 cores and will be able to be used as either co-processors or primary processors.
Intel officials said the fabric technology will address a range of issues within the HPC space, from price and performance to scalability and reliability.
As the number of cores grows, interconnect fabrics become increasingly important to ensure high levels of communications within the system as well as between systems. In traditional environments, networking includes top-of-rack switches, end-of-rack switches and core switches, with data moving between them and the servers. Integrated interconnect fabrics enable communication between the chips and systems without a core switch and top-of-rack switch, with data flowing in a more east-west direction.
One of the key challenges in the computing space is that the cost of fabrics is increasing, whether it’s InfiniBand, Ethernet or something else, according to Phil Murphy, chief system architect of Omni-Path. Where once the fabric constituted 10 percent of the system cost, that number now is closer to 20 percent to 25 percent. To get to 100G-bps performance, the percentage could go even higher, to as much as 40 percent, Murphy said during the briefing at IDF.
One of the ways to drive down the cost and increase the price/performance ratio is through integration, he said. When Knights Landing rolls out later this year, the Omni-Path controllers will be on the chip packages, but not integrated onto the same piece of silicon as the chip itself. The two will be sharing a PCIe interface. However, the plan for future generations of Xeon Phi and Omni-Path is to integrate the fabric technology onto the chip, he said.
“Performance is one area where we can improve with integration,” Murphy said, noting that integration also enhances everything from density to reliability to power consumption.
Intel Offers More Details of Its Omni-Path Interconnect Fabric
Intel also is working on Omni-Path PCIe cards with one or two ports that can be used with chips that are not Xeon Phis, and is developing a 48-port switch that will feature Omni-Path.
Overall, the Omni-Path Architecture will be 25 percent to 40 percent less expensive than InfiniBand, and will be able to be used in data centers as well as HPC environments, something that doesn’t tend to happen with InfiniBand, according to Murphy. In addition, it will offer 73 percent higher switch MPI message rate and 33 percent lower latency than other interconnect fabrics.
The Intel officials noted that one innovation that will help reduce latency is the insertion of a new networking layer between the traditional Layers 1 and 2. Murphy called it Layer 1.5—the link transport layer—which essentially reduces workload packets into smaller 65-bit units called “flits.” The link transport layer initially was being worked on at Cray, and came to Intel after the chip maker bought the networking technology.
A set of 16 flits plus the CRC (Cyclic Redundancy Check) create what officials call a packet. The technology reduces latency by enabling high-priority packets to move larger and lower-priority packets, even if those larger packets have already been sent down the line, Murphy said. This ensures traffic optimization so that the high-priority packets get to their destinations as fast as possible.
The Intel officials also stressed the importance of open-source technology to the Omni-Path Architecture. The chip maker is supporting Open Fabric Interfaces from by the open-source group OpenFabrics Alliance, which the officials said will lead to improved performance and the ability to scale to tens of thousands of nodes, Murphy said.
So far, the interest from systems makers has been good, according to Saleh. There are more than 100 OEM system designs in the works that will feature Omni-Path Architecture, and more than 100,000 nodes are under bid or contract. The first deployments are expected in the fourth quarter, with momentum continuing into next year, he said.