ScaMPI
From Vd
Contents |
ScaMPI (Scali MPI Connect)
Scali MPI Connect SMC is thread safe and thread hot interconnect agnostic MPI library.
Scali MPI libraries can be found at /opt/scali/ where the subdirectories include64 and lib64 contain the header files and the libraries. The documentation can be found at /opt/scali/doc/SMC55_UserGuide.pdf.
SMC contain a number of features like tracing and timing, change of collective algorithms at run time, support for multiple interconnects (selection at startup time), support for all known fortran compilers, support for a subset of MPI2 functions, suppprt for MPI-IO through the ROM-IO module in addition to a large range of features to facilitate development and debugging of applications.
usage
MPI programs can be compiled using mpicc and mpif77. This is using the default compilers.
mpif77 -c -O2 file.f
or for C
mpicc -c -O2 file.c
Run compile and link in a single step :
mpicc -o file.x -O2 file.c
To run
mpiexec file.x
This will run your executable file.x on the allocated nodes, assuming you have logged in using qlogin or running in batch using the queue system.
Example :
qlogin -P uio -pe 8 mpi mpicc -o all2all.x all2all.c mpiexec all2all.x Alltoall size 8 [4 procs] 131072k in chunks of 0k4 - 512k0 transfered Size Time Iter Sample time Bandwidth [Resolution 1.91us] ---------------------------------------------------------------------- 4 0.05 512 88.05 0.30 2.43 8 0.04 512 73.39 0.73 5.82
Mpiexec is normally adequate for general MPI usage. It will schedule one MPI process per CPU core. Interconnect is normally given in a system wide file. hence if InfiniBand or other high speed interconnects are present it will be used. Notice the much higher bandwidth :
Alltoall size 8 [4 procs] 131072k in chunks of 0k4 - 512k0 transfered
Size Time Iter Sample time Bandwidth [Resolution 2.15us] ---------------------------------------------------------------------- 4 0.00 512 7.29 3.66 29.32 8 0.01 512 10.95 4.88 39.03
Advanced usage
However, if you really want ot have control over compilers and MPI the paths need to be specified manually. To compile a program using a fortran compiler and ScaMPI the following command line can be used (assume 64bit):
pathf90 -c -O2 -I/opt/scali/include64 file.f
To link the program use the command line
pathf90 -o file.x -I/opt/scali/include64 -L/opt/scali/lib64 -lfmpi -lmpi file.o
To run a program the mpiexec can be used as this will set up the correct paths and run your program on the allocated nodes. However, if you want to run interactively you can use the following command line (where z is the number of MPI processes started per node, for dual core processors z=4 for quad core processors z=8):
mpimon ./progra.x -- node1 z node2 z nodeN z
Running one MPI process per CPU core place a burden on the memory bandwidth. Reducing z while increasing N might produce higher performance as more nodes yield higher total memory bandwidth.
To specify the interconenct with mpimon the -net can be used.
mpimon -net infinipath ./all2all.x -- node1 node2
Mpimon can take a large number of options and all of these are described in detail in the SMC manual.
MPI IO
Scali MPI relies on ROMIO for the MPI-IO functionality. In order to compile using MPI-IO the special MPIIO include file must be called for (this for fortran):
include 'mpif.h' include 'mpiof.h'
The extra line include 'mpiof.h' is often not present in source code as many other MPI implementations put the ROMIO include lines in the mpif.h file.
To compile the include files are specified as usual with the -I option, but linking need an extra library. The the command line will compile and link the fortran program file.f :
pathf90 -I/opt/scali/include64 -L/opt/scali/lib64 -lfmpi -lmpi -lmpio -o file.x file.f