PATHS

In parallel and distributed computing, clusters are increasingly used for compute and I/O-intensive applications. As we add computing resources to a parallel application, one of the fundamental questions is how well the application scales, both with regards to speedup and to increasing the problem size. The PATHS project investigates two main issues impacting scaling. The first is the end-to-end communication latency. The other is the configuration and mapping of the application onto a cluster topology and architecture. Several factors were studied to determine their impact on end-to-end latency, including procotols, workload, and locating communication endpoints at user-, kernel and interrupt-level.

The dominating contribution to latency comes from complex protocols, such as TCP/IP, which do not take advantage of properties in the interconnect to reduce the amount of processing for communication. The latency reduction from choosing a protocol with lower overhead is approximately an order of magnitude larger than the reduction gained from moving communication endpoints from user-space to the operating system kernel.

When blocking communication is used, a hardware supported implementation of Virtual Interface Architecture (VIA) using a 1.25 Gbit interconnect has approximately the same latency for small messages as a software implementation of VIA using 100 Mbit Ethernet. The performance advantage of the hardware supported VIA implementation is masked by the operating system overhead for interrupt handling, suspending, and resuming the communicating threads. To avoid this overhead, polling communication can be used. This results in a significant reduction of latency on the hardware supported gigabit VIA implementation. Even in this case, hardware supported user-space communication reduces latency significantly less than using simpler protocols.

To experiment with an applications configuration, an approach and an instrument, PATHS, was developed. Using PATHS, a number of configurations were specified and their performance measured, in order to identify significant behaviours impacting the performance.

A number of benchmarks were used, including a simulation of the Colombia river and estuary. The platform used for experiments was comprised of three clusters, each with 32 processors. The clusters were used individually or together in a number of multi-cluster configurations.

Tuning the performance of distributed and parallel applications is complex. Simple and small changes to a configuration influence scaling and latency. It is hard to compute good configurations. Instead, it is demonstrated that starting with what is believed to be a good configuration, a number of experiments can be run to find configurations with better performance.

The LAM-MPI implementation of the Message Passing Interface (MPI) standard is documented to use configurations which do not scale well, and lacks mechanisms for changing the configuration. A configuration mechanism has been added to LAM-MPI and used to double the performance of the MPI Allreduce function.