Quickstart guide for programming the Regatta

On this page, we give some examples of commonly used instructions to compile and execute a program on the IBM Regatta. Since many users of the Regatta use Fortran as programming language, we will use IBM's native Fortran compiler 'xlf' for most of the examples below. For C/C++ the examples would look similar, exchange xlf90 with xlc or xlC, and of course also change the file name suffix from .f to .c or .C. Options for parallelism and optimization are generally the same for Fortran and C/C++.

Contents:

   A. Compiling sequential programs
   B. Compiling OpenMP or threaded programs
   C. Compiling MPI programs
   D. Compiler summary
   E. Frequently used compiler options
   F. Running the program

A. COMPILING SEQUENTIAL PROGRAMS

On many systems, source files for Fortran 77, Fortran 90 and Fortran 95 have the suffixes .f, .f90 and .f95, respectively. On the IBM, the Fortran 77 compiler ('xlf'), Fortran 90 compiler ('xlf90') and Fortran 95 compiler ('xlf95') all assume that source files end with .f. IBM's C compiler ('xlc') and C++ compiler ('xlC') expect files with suffices .c and .C, respectively.

Single source file. Assume your Fortran 90 source program is stored in file 'prog.f'. Then you can compile it with the command:

  xlf90 prog.f -o prog

The '-o prog' option instructs the compiler to name the executable program 'prog'. If the '-o' option is not used, the executable program is called 'a.out' by default.

Multiple source files. If your source program is spread over multiple files, one compiles these files in multiple steps. Suppose for example that a subroutine in the file 'prog.f' calls a subroutine that is declared in the file 'prog2.f'. Then one can compile in two steps:

  xlf90 -c prog2.f
  xlf90 prog.f prog2.o -o prog

The '-c' option instructs the compiler to create an object file (prog2.o) instead of an executable file. This object file is used at a later step of the compilation. In the last step of the compilation, all object files (or one source file with other object files) are usually linked together to create the executable file. By default, the compiler links these object files together with several libraries that are present on the system and that contain implementations of the language standard (for example Fortran intrinsics like abs, sin, mod, ...).

Source file suffix. All the IBM Fortran compilers assume that the source files have suffix .f. If you port your program from another platform, your files may have another suffix, for example .f90. In this case, you can rename the source files, but you can also compile and link the files as follows:

  xlf90 -qsuffix=f=f90 prog.f90

Source files that contain cpp preprocessor directives should end with .F.

The make utility. When your code consists of many source files, we recommend you to use the 'make' utility. With 'make', one can automate the maintenance, update, compilation, and regeneration of object and executable files. The make utility requires by default a file called 'Makefile' in which you specify your compiler options, source/object files, and rules for compilation. After you have created the Makefile, you can build your program with the command 'make' or 'gmake' (GNU make). A directory with an example Makefile for GNU make (which we recommend), as well as an example Fortran program can be found here. (The Makefile example ilustrates how to compile Fortran programs with so called C preprocessing directives. The Fortran example also contains a way to measure wall clock time in Fortran 90.)
Writing Makefiles is not always easy but can simplify your life considerably. Contact hpc-support@hpc.uib.no to get help when needed. Tutorials on the Web can be found here and here.

B. COMPILING OPENMP PROGRAMS

Most options for compiling sequential programs can also be used for compiling parallel OpenMP (or threaded) programs. Commands to compile Fortran/OpenMP programs are xlf_r, xlf90_r, and xlf95_r. Example:

  xlf90_r -qsmp=omp:opt prog.f

The main difference between the 'xlf' and 'xlf_r' commands is that they use different default options (which are set in the configuration file /etc/xlf.cfg). The '_r' makes the linker use «thread-safe» versions of the Fortran/C/C++ libraries.

For more information, see our SMP (OpenMP) pages.

NOTE: If you use xlf_r (F77 compiler) to compile your OpenMP program, you should always use the -qnosave option. For xlf90_r (F90 compiler), the -qnosave option is used by default and need not be set by the user.

C. COMPILING MPI PROGRAMS

Most options for compiling sequential programs can also be used for compiling parallel MPI programs. Commands to compile Fortran/MPI programs are mpxlf_r, mpxlf90_r, and mpxlf95_r. Example:

  mpxlf90_r prog.f

The 'mpxlf...' commands are shell scripts that invoke the appropriate xlf compiler. In addition, the Partition Manager, Message Passing Interface (MPI), and/or Message Passing Library (MPL) are automatically linked in. Flags are passed by mpxlf to the xlf command, so any of the xlf options can be used on the mpxlf shell script.

By default, the mpxlf scripts pass the proper MPI header files to the xlf compilers. It is therefore not necessary to specify the directory (normally with the '-I' option) where the MPI header file is located. The MPI compilers also link in the MPI libraries by default (and linking with something like '-L... -lmpi' is not needed).

For more information, see our MPI page.

NOTE: If your MPI program uses shared memory to communicate between processors (and this is true by default), then your MPI executables must be made thread-safe. This is why we add '_r' to the mpxlf commands above.

D. COMPILER SUMMARY

In short, the following compilers exist on the Regatta. Use the manual pages (e.g. 'man xlc') for more variants and specific details.

language	sequential	MPI	OpenMP	MPI thread-safe	file suffix
Fortran 77	xlf	mpxlf	xlf_r	mpxlf_r	.f
Fortran 90	xlf90	mpxlf90	xlf90_r	mpxlf90_r	.f
Fortran 95	xlf95	mpxlf95	xlf95_r	mpxlf95_r	.f
C	xlc	mpcc	xlc_r	mpcc_r	.c
C++	xlC	mpCC	xlC_r	mpCC_r	.C

E. FREQUENTLY USED COMPILER OPTIONS

The compilers accept many options e.g. for debugging, optimization for code size or performance. Use 'man xlf' to get an overview of the options for the Fortran compilers and 'man xlc' for the C/C++ compilers.

Performance run-time optimization. A good starting point is to compile your code with the options

  -O3 -qarch=auto -qtune=auto -qcache=auto

See our page for performance optimization for more information.

Available memory at compile-time. The compiler may generate messages saying that it needs more memory for additional optimization of a specific subroutine. To avoid this, use the option

  -qmaxmem=-1

to give 'unlimited' memory to the compiler for space intensive optimization.

Available memory at run-time. Because of historical reasons, the IBM compilers use by default 32-bit addressing. If your 32-bit executable program uses more than several hundred Megabytes of memory, then you must set (at link time) the maximum size allowed for the user data area (or user heap) when the executable program is run. Use the linker option

  -bmaxdata:[num]

where [num] sets the maximum size in bytes. For a 32-bit program, the maximum value allowed is (hexadecimal) 0x70000000, which corresponds to roughly 2 Gbyte. The maximum stack space for 32-bit code is 256 Megabytes and can be set by linker option

  -bmaxstack:256000000

If 256 Megabyte stack is not enough, use ALLOCATE/DEALLOCATE instead og automatic arrays.

For a 64-bit program (compiled with -q64), the option -bmaxdata should not be used.

NOTE: The available amount of memory for 32-bit executables is limited to circa 2 Gbyte for your sequential or OpenMP job and to 2 Gbyte per MPI process. If your job or processes require more memory, you must compile/link your whole code with 64-bit addressing. Therefore, add the option -q64 to your compiler. Also add option -X64 to the ar command when you generate libraries.

Profiling and debugging. We created separate pages with information on profiling and debugging programs on the Regatta.

F. RUNNING THE PROGRAM

All executable programs should run in the batch system. If you do not do this and the system administrator detects this, he may suspend or kill your jobs without prior warning. Submitting and running your jobs through the batch system is necessary to avoid system overload and it enforces a fair share of the resources among the users.

All executables and temporary input/output should be stored in directory /work/$USER. The I/O performance of the /work filesystem is much better than that of /home. Therefore, use /work for all large data that is temporary and data that can be generated easily. These include binaries, uncompressed data files, and especially intermediate files (that exist only at runtime). The /work area is also much larger than /home.

A minimal batchscript for submitting a sequential program to the batch system could look like as follows. We assume the user has an executable program called «prog» in his/her home directory, and the user is allowed to run programs under the «parallab» account. The script below starts the executable from the users area (/home), stores output in the much faster temporary storage area (/work), and then copies the output generated by «prog» back to the home directory.

#! /bin/bash -
# @ resources        = ConsumableCpus(1) ConsumableMemory(10 mb)
# @ account_no       = parallab
# @ wall_clock_limit = 0:10:00
# @ queue
cd /work/$USER
$HOME/prog > prog.output
cp prog.output $HOME

The first lines in the script (starting with '# @') look like shell comments, but are actually commands to the batch system and must be present. Save the script in the file «job1», then submit the job to the batch system:

  llsubmit job1

After the job is finished the user will be notified by e-mail. See our batch system page for more detailed information and how to submit parallel jobs.