Message Passing Interface (MPI)

To obtain maximum performance on parallel computers, one must often use explicit message passing to communicate data between processors. In a distributed-memory environment, each processor has its own local memory and explicit calls to subroutines are needed to send and receive data from other processors. Writing message passing code is often much more complicated and time consuming than using symmetric multi-processing.

Compiling MPI programs

To compile an MPI program on the p690 Regatta, use the commands mpcc (C compiler), mpCC (C++ compiler), or mpxlf90/mpxlf95/mpxlf (Fortran compilers). These commands invoke the normal compilers, the only difference is that the message passing interface libraries are linked in at the end.

To compile MPI C, C++, or Fortran programs that need to be thread-safe, append _r to the command, e.g. mpcc_r, mpCC_r, or mpxlf_r. These commands can also be used to compile non-threaded programs with threaded libraries (e.g. the thread-safe numerical libraries ESSL). More info here.

NOTE: If you wish to use shared memory to communicate between MPI processes within a (Regatta) node of the system, or if your program uses one-sided communication features, you must link your program with the thread-safe libraries. See also batch system.

Running MPI programs

Information on how to submit a parallel job can be found here. Note that the MPI executable is assumed to be accessible with the same path name by the nodes you are going to use.

MPI programs can be invoked from the command line by typing the executable name, followed by options for the poe program. poe is similar to mpirun on other platforms, it sets up the "parallel environment". Assuming your MPI executable is called prog, the following ways to start the program on four cpu's are equivalent:

./prog -procs 4
poe ./prog -procs 4

See the manual page man poe for more options.

We do emphasize that interactive execution (from the command line) is only allowed for very short jobs that do not require much memory, e.g., for debugging purposes. In all other cases, the job has to be submitted through the batch system.

For a job that is submitted through the batch system, it is not known in advance on which of the three Regatta nodes the job will be executed. Therefore, make sure that each of the three nodes can access the executable. If the executable is stored on a local file system, copy the executable to the other nodes that will (or may) participate in parallel execution. Set up the .rhosts file appropriately so that the remote shell command rsh works for the nodes that you are going to use.

Runtime environment variables

MEMORY_AFFINITY. In an MPI application, each process has its own 'local' memory address space that is mapped/allocated in some way onto the compute node's physical memory. In practice, it may be worthwhile to instruct the operating system to allocate the memory as close as possible to the CPU on which the process runs. You can specify 'MCM-memory' affinity via environment variables.

setenv MEMORY_AFFINITY MCM
export MEMORY_AFFINITY=MCM

We note that this setting is not guaranteed to improve performance (it may have an opposite/negative effect) and should be used with care.

MP_SINGLE_THREAD. Whenever each MPI process consists of exactly one thread, it may help to use the environment setting

setenv MP_SINGLE_THREAD yes
export MP_SINGLE_THREAD=yes

MP_EAGER_LIMIT. The communication subsystem of the IBM uses two methods for data transfer between tasks of a parallel application. Short messages are immediately sent to the receiver (asynchronously) and a send operation completes without acknowledgement from a matching receive. Long messages use a rendez-vous protocol (hand-shaking) which requires an acknowledgement from a matching receive in order for the send operation to complete. The variable MP_EAGER_LIMIT is a threshold in the message size that defines the distinction between short and long messages. The following is the default (in bytes) for MP_EAGER_LIMIT:

 total_tasks   MP_EAGER_LIMIT
 ===========   ==============
    1 - 16          4096
   17 - 32          2048
   33 - 64          1024
   65 - 128          512
  129 - 256          256
       >256          128

The largest possible value for MP_EAGER_LIMIT is 65536 (or 64 Kbyte). Some experimentation to find an optimal value for MP_EAGER_LIMIT may be necessary.

We recommend using MP_EAGER_LIMIT=0 during code development, since it can be used to detect unsafe communication within an application.

See the MPI Programming Guide below for more details on MPI_EAGER_LIMIT and related variables.

MP_BUFFER_MEM. This variable defines the buffer size for 'early' arrived messages. The default is 64 Mbyte.

setenv MP_BUFFER_MEM 67108864
export MP_BUFFER_MEM=67108864

Some links to MPI resources

• Online Books for beginners

  • ``Designing and Building Parallel Programs'', by Ian Foster
  • EPCC self study course

• IBM manuals and documentation

  • Hitchhiker's Guide - PDF (357 KB)
  • Hitchhiker's Guide (sample programs) - TAR file (330 KB)
  • MPI Programming Guide - PDF (626 KB)
  • MPI Subroutine Reference - PDF (1382 KB)
  • PVM Programmer's Manual
  • IBM Release notes
  • Online Status with new features

• Standards and Reference

  • MPI 1.1 A Message Passing Standard: PS | HTML
  • MPI 1.1 Errata: PS
  • MPI 2.0 Extensions to the Message Passing Standard: PS | HTML
  • MPI The Complete Reference . (Netlib, 1995)

• Major MPI Web pages

  • MPI home page
  • MPI2 home page
  • MPI forum

• Local resources

• MPI FAQs

  • MPI home page FAQ