IBM, Microsoft Help Bring Big Sky Supercomputer to Montana

by Jeffrey Burt

This is one of the two enclosures that make up the IBM 1350 cluster. It houses the cluster’s head nodes, the storage controllers and most of the storage arrays. It also houses some of the InfiniBand network. At the very top is a hardware-based wire-line speed deep packet inspection firewall used to allow secure access to the cluster.

The second of the two enclosures of the IBM 1350 cluster houses most of the compute hardware, including the three IBM BladeCenters in the bottom, middle bottom and middle of the rack. Inside each BladeCenter are 14 blade servers—shown with IBM blue tabs—for a total of 42 compute nodes. At top are the rest of the storage array disks that would not fit in the first enclosure.

This is the rear of the cluster with the IBM Cool Blue doors closed but visible on the enclosure. At the bottom is the orange reinforced feed tube to the right door. Distilled water is circulated through this feed tube.

Pictured are the various management, compute and storage networks in the IBM 1350 cluster. At top is the storage array chassis; at the bottom are the head nodes, storage controllers and the scientific visualization graphics rendering nodes. Also at the bottom are the two distilled water feed tubes—one feed, one return—that form the cooling loop through the Cool Blue doors.

In the coolant pumping system, the two brown pumps are fault-tolerant of one another and circulate distilled water through the IBM Cool Blue doors. The doors exchange about 60 percent of the heat generated from the two enclosures to the distilled water loop. The heat is captured in the distilled water and is exchanged with an ethylene glycol loop that goes to the roof cooling tower. En route, the glycol loop passes through heat pumps in the offices on the third, fourth and fifth floors to heat the offices in the office building, saving the building owner about $40,000 a year in heat costs.

The heat exchanges are shown here. Inside is where the heat from the distilled water is exchanged to the glycol loop.

On the left are the two 4-inch pipes that carry the glycol loop that go from the data center location all the way to the cooling tower on the roof of the building. On the right, three other vertical pipes, each of which goes to floors three, four and five to circulate the glycol through the office heat pumps prior to returning to the glycol loop pipe headed for the cooling tower.

The expansion tank (red) and the reserve tank (green/white) are for the distilled water that circulates in the IBM Cool Blue doors. The expansion tank is needed as the temperature within the loop varies, causing more or less volume in the loop. The reserve tank makes sure there is always the correct amount of distilled water present in the distilled water loop.