InfiniBand is the interconnect of choice for the Virginia Tech Terascale Computing Facility. According to VT-TCF assistant director of operations Kevin Shinpaugh, "One reason we chose InfiniBand as our interconnect for the Terascale facility is that it was originally designed to act as a bus replacement fabric. Its RDMA and queue-pair messaging architecture allows for concepts like bridged I/O, redundant paths, and hot swapping of components. When bridged I/O was introduced to the Mac, we were very excited. We've reviewed the technology and are excited by the new capabilities. With this technology, we are looking to add approximately 500 terabytes of redundant disk storage capacity with 10 Gbps of data throughput to disk. Using InfiniBand HCA's dual path and hot-plug capabilities, this should be relatively easy.
"When we first designed the Terascale system, InfiniBand offered the latencies, performance, and cost structure we needed to be successful," Shinpaugh went on to say. "However, at that time we were using the open-source VAPI-based drivers from Mellanox and Mellanox first-generation switches and I/O-Bridged InfiniBand solutions were not available. Using IP over InfiniBand, this allows IP-based protocols, such as NFS, that need a high-performance network topology that can run over your InfiniBand network without requiring the additional expense in creating a separate high-performance Gig-E network. This capability gives you the option of using IP over IP natively or if needed, you can deploy a very low cost Gigabit Ethernet solution."
Shinpaugh is also looking at what is in the future for the Terascale system. "We originally put this system together using large PowerMac systems. After upgrading to Xserves, we have a great deal more space in our facility. The current systems are PCIX and are using 2.3-GHz PowerPC processors. We look forward to being able to implement dual-core processors, PCI Express, and higher bandwidth InfiniBand switches (including hot-plug and dynamic subnet management) so we can vastly improve upon our processing power. Our testing of SilverStorm's switch technology is going well and we hope to have our disk I/O subsystem running over SRP or natively on the InfiniBand network shortly."
—C.S.
Back to Article