Magnum
From Vd
Simon See, SUN, in front of a Magnum InfiniBand Switch (to the left) and a Constellation Blade System (to the right)
The SUN InfiniBand switch, magnum, is the core TITAN network switch. It was acquired April 2008 and connects all servers, compute nodes and the xanadu storage systems into a capable compute facility.
Magnum key information:
- Up to 3,456 double data rate (DDR) InfiniBand ports
- Total bisection bandwidth: 110 Tb/s
- The switch has a 6:1 reduction in interconnect cables compared to traditional approaches
- Applied Magnum cables are roughly 50% more area efficient than ordinary 4X cables and more robust
- Three times fewer cables to install and manage
- Magnum IB cables may be 50% longer than ordinary IB cables, enabling a larger clusters to be built
- The Magnum switch consume relatively little power
Magnum quotes:
To do what the Magnum switch does would take 12 core InfiniBand switches and 288 leaf switches, and by moving to this simplified arrangement, Sun can cut down the number of cables in the cluster by a factor of six and cram a 3,456-node cluster into 20 percent less space. The Magnum is a box that is twice as wide and half as tall as a standard rack, and it has a bisection bandwidth of 110 Tbps. Sun is using a 12X InfiniBand cable coming out of the Magnum switch, which splits down to four 4x InfiniBand links as the wire gets closer to the server nodes.
Sun's Magnum switch, based around the InfiniBand high-speed networking technology, is a honker. The largest InfiniBand switches on the market contain 288 ports, according to Bechtolsheim, and require leaf, or helper switches.
Physically, Magnum is huge – it is about half a rack high and two racks wide. The number 3,456 gives a big clue as to the internal architecture: 3456 = 288 * 12. Current InfiniBand switch chips have 24 ports, and the biggest non-blocking switch one can build with two levels (spine and leaf) is 24 * 12 = 288 ports: 24 leaf switches each of which have 12 ports to the outside and 12 ports to the spines (one port to each of the 12 spine switches). Then, using 12 288-port switches as spines, one can take 288 24-port leaf switches that each have 12 ports to the outside and end up with 288 * 12 = 3456 ports, just like Sun’s Magnum switch. From the pictures of the chassis, it looks like Magnum has the spine switches on cards on one side of the midplane and the leaf switches on the other side, using the cute trick of having one set of cards be vertical and one set horizontal to get all-to-all connections between spines and leaves without having too-long midplane traces. All of this sounds quite reasonable until you start to consider putting all of this in one box. Each 288 port switch (which is on one card in this design!) has 36 switch chips on it. At about 30 Watts per switch chip, each of this cards is over 1 kilowatt, and there are 12 of these in a system. In fact, with 720 switch chips in the box, the total system is well over 20 kW!