User support
From Hpc
Contents |
General information
Getting help
If you need help using the system, file a support case with the NOTUR online support system
When doing so, please include as much relevant information as possible. Paths to relevant files, description of what you are trying to do and copies of screen output are all of great help to us, and will probably speed up the solution process.
Logging in
The only way to connect to njord.hpc.ntnu.no is by secure shell (ssh), e.g. from a UNIX system
% ssh -l username njord.hpc.ntnu.no
Windows users: For logging in from MS Windows machines, we recommend using the Putty ssh client or similar.
Program development
Notice, the default object mode on njord is 64-bit, which means that you by default will build executables that run in 64-bit address mode. To change bit mode specify 32 for the environment variable OBJECT_MODE.
System compilers
XL Fortran, XL C and XL C/C++ comprise the family of XL compilers installed on the system. Use one of the following commands when invoking the compiler
Language Serial MPI thread-safe OpenMP Fortran 77 xlf mpxlf_r xlf_r Fortran 90 xlf90 mpxlf90_r xlf90_r Fortran 95 xlf95 mpxlf95_r xlf95_r C xlc mpcc_r xlc_r C++ xlC mpCC_r xlC_r
As a set of standard compile line options for production codes we recommend the following, e.g. for a Fortran MPI program
njord> mpxlf90_r -O3 -qstrict -qarch=pwr5 -qtune=pwr5 my_mpi_program.f90
Option Description -O3 Optimization level, including memory intensive and compile-time intensive optimizations -qstrict Ensures that optimizations done by -O3 and greater do not alter the semantics of a program. -qarch=pwr5 Specifies the correct target processor that allows the compiler to select more efficient machine instructions and generate instruction sequences that perform best for the selected architecture -qtune=pwr5 Tunes instruction selection, scheduling, and other implementation-dependent performance enhancements for the selected hardware architecture
For more information see the XL Fortran V10.1 for AIX and XL C/C++ V8.0 for AIX guides.
OpenMP
When compiling code using OpenMP ($OMP) directives specify the -qsmp=omp option, e.g.
njord> xlf90_r -qsmp=omp my_omp_program.f90
If you use xlf_r (F77 compiler) to compile your OpenMP program, you should also use the -qnosave option. For xlf90_r and xlf95_r the -qnosave option is used by default.
For more information see Parallel programming with XL Fortran.
Libraries
ESSL and Parallel ESSL
The Engineering and Scientific Subroutine Library (ESSL) is a collection of subroutines providing a wide range of mathematical functions for many different scientific and engineering applications. Its primary characteristics are performance, functional capability, and usability. To link with the ESSL library specify this at command line
njord> xlf90 filename.f90 -lessl
For more information see the ESSL for AIX V4.2 and Parallel ESSL for AIX V3.1 guides.
MASS
The Mathematical Acceleration Subsystem (MASS) consist of a library of scalar routines and a set of vector libraries tuned for specific architectures. For more information see Using the Mathematical Acceleration Subsystem (MASS).
Optimization
See Optimizing XL compiler applications.
Performance tools
The IBM High Performance Computing Toolkit is a suite of performance-related tools and libraries to assist in application tuning. This toolkit is an integrated environment for performance analysis of sequential and parallel applications using the MPI and OpenMP paradigms.
Software components
- Xprofiler
- used to profile both serial and parallel applications. It uses procedure-profiling information to construct a graphical display of the functions within an application. Xprofiler provides quick access to the profiled data and helps users identify the functions that are the most CPU-intensive.
- Hardware performance monitor (HPM)
- provides reports of events that are critical to performance, e.g. the number of misses on all cache levels.
- MPI Tracer/Profiler
- consists of a set of libraries that collect profiling and tracing data for MPI programs. Performance metrics, such as, the time used by MPI function calls and message sizes, are reported.
- OpenMP Profiler
- dynamically instruments OpenMP applications, and reports performance related information such as timings and the overhead of OpenMP constructs.
Debugging
Batch jobs
Executable programs should run in the LoadLeveler batch system. Submitting and running your jobs through the batch system is necessary to avoid system overload and it enforces a fair share of the resources among the users. Before you can submit a job or perform any other job related tasks, you need to build a job command file. A job command file describes the job you want to submit and include LoadLeveler keyword statements (starting with #@), see LoadLeveler keywords.
To make I/O more efficient always copy inputfiles and temporary store outputfiles in /work/$USER. After job execution move the outputfiles to your home directory.
Files in the /work filesystem which have not been accessed for a certain amount of time, normally 14 days, will be automatically deleted. This is subject to change but it is not expected to be less than a week. There is no backup of data stored in the /work directory.
LoadLeveler commands
Command Description llsubmit Submit a job llclass Returns information about classes llq Query information about jobs in the queues llcancel Cancel job from the queue llstatus Returns status information about nodes in the cluster
Class overview
Class Max slots Min slots * Max runtime Description forecast 864 16 no limit top-priority class dedicated to forecast jobs bigmem 16 16 24h 120GB memory class express 64 16 1h high-priority class for debugging and test runs normal 384 16 96h default parallel class monster 864 16 no limit long-running massive parallel jobs small 16 1 96h serial, or small smp class
Accounting
Except for jobs in the small class, nodes are allocated for the exclusive use of one job. Jobs are accounted for the time a node is reserved, regardless of the actual usage of said node.
If you, for instance, run a 8 processor job in the normal queue, you will thus be accounted for 16 processors for the entire run-length of the job.
To check accounting status, use the 'cost' command. Note that this statistics is not entirely live. Unfinished jobs, and recently finished jobs will not be included.
LoadLeveler keywords
#@ job_name : Specifies the name of the job. #@ account_no : Specifies the account number to charge for the job. #@ class : Specifies the name of the class were the job will be run, see Class overview. Use command llclass to find out information on job classes. #@ job_type : Specifies the type of job step to process. For serial and SMP (OpenMP) programs specify serial, for MPI programs specify parallel. #@ node : Specifies the number of nodes requested by a job step. #@ tasks_per_node : Specifies the number of tasks per node. #@ node_usage : Specifies whether the job shares nodes with other jobs. On njord for parallel jobs specify not to share nodes: #@ node_usage = not_shared
#@ resources : Specifies the resources consumed by each task of a job step, see Consumable resources. #@ network : Specifies communication protocols, adapters, and their characteristics. On njord when running MPI jobs on more than one node specify as follows #@ network.MPI = sn_all,,us
#@ output : Specifies the name of the file to use as standard output (stdout). #@ error : Specifies the name of the file to use as standard error (stderr). #@ wall_clock_limit : Sets the time limit for the job. #@ environment : Specifies login initial environment variables set by LoadLeveler. Set COPY_ALL to copy all environment variables from your shell. #@ env_copy : Specify all to copy environment variables to all nodes. #@ notification : Specifies when to notify user by mail. Default is mail sent when job ends. #@ queue : Marks the end of the job step and is required.
Consumable resources
When running batch jobs use the resources keyword in the job command file to specify the resources to be consumed by each task of a job step.
MPI programs Set the job_type to parallel, specify 1 for the ConsumableCpus and define the number of MPI tasks through the total_tasks keyword. The ConsumableMemory specifies the amount of memory needed per MPI task # @ job_type = parallel
# @ resources = ConsumableCpus(1) ConsumableMemory(832 mb)
# @ total_tasks = 16OpenMP programs Set the job_type to serial, and specify the number of threads for the ConsumableCpus and the OMP_NUM_THREADS variable. The ConsumableMemory should be set to the total amount of memory needed for the job. # @ job_type = serial
# @ resources = ConsumableCpus(16) ConsumableMemory(4 gb)
# @ environment = OMP_NUM_THREADS=16Hybrid MPI/OpenMP programs Set the ConsumableCpus to the number of threads each MPI task will spawn and ConsumableMemory to the amount of memory each MPI task will use. The example specifies 8 MPI tasks with 2 threads each and a total of 12 GB memory for the job. # @ job_type = parallel
# @ resources = ConsumableCpus(2) ConsumableMemory(1536 mb)
# @ total_tasks = 8
# @ environment = OMP_NUM_THREADS=2
Sample batch scripts
Sequential job
#!/bin/ksh # @ job_name = myjob # @ account_no = nn1234k # @ class = small # @ job_type = serial # @ resources = ConsumableCpus(1) ConsumableMemory(1 gb) # @ error = $(job_name).$(jobid).err # @ output = $(job_name).$(jobid).out # @ wall_clock_limit = 01:00:00 # @ environment = COPY_ALL # @ env_copy = all # # @ queue # # Create (if necessary) and move to my working directory w=$WORKDIR/$USER if [ ! -d $w ]; then mkdir $w; fi cd $w # # Copy inputfiles to working directory cp $HOME/what_ever_path/inputfiles . # # Run my program $HOME/what_ever_path/my_program # # Move results back to home directory mv outputfiles $HOME/what_ever_path
MPI job
#!/bin/ksh # @ job_name = myjob # @ account_no = nn1234k # @ class = normal # @ job_type = parallel # @ node = 2 # @ tasks_per_node = 16 # @ node_usage = not_shared # @ resources = ConsumableCpus(1) ConsumableMemory(832 mb) # @ network.MPI = sn_all,,us # @ error = $(job_name).$(jobid).err # @ output = $(job_name).$(jobid).out # @ wall_clock_limit = 01:00:00 # @ environment = COPY_ALL # @ env_copy = all # # @ queue # # Create (if necessary) and move to my working directory w=$WORKDIR/$USER if [ ! -d $w ]; then mkdir $w; fi cd $w # # Copy inputfiles to working directory cp $HOME/what_ever_path/inputfiles . # # Run my program $HOME/what_ever_path/my_mpi_program # # Move results back to home directory mv outputfiles $HOME/what_ever_path
OpenMP job
#!/bin/ksh # @ job_name = myjob # @ account_no = nn1234k # @ class = normal # @ job_type = serial # @ resources = ConsumableCpus(16) ConsumableMemory(4 gb) # @ node_usage = not_shared # @ error = $(job_name).$(jobid).err # @ output = $(job_name).$(jobid).out # @ wall_clock_limit = 01:00:00 # @ environment = COPY_ALL; OMP_NUM_THREADS=16 # @ env_copy = all # # @ queue # # Create (if necessary) and move to my working directory w=$WORKDIR/$USER if [ ! -d $w ]; then mkdir $w; fi cd $w # # Copy inputfiles to working directory cp $HOME/what_ever_path/inputfiles . # # Run my program $HOME/what_ever_path/my_omp_program # # Move results back to home directory mv outputfiles $HOME/what_ever_path
