From Hpc

Introduction to the Torque / MAUI batch system

In order to run programs on a cluster, it is necessary to use the batch system to distribute the work onto the compute nodes. This is a brief introduction to the batch system which states some motivations for using it, and introduces the most basic commands involved in its use.

Motivation

Running on the frontend

It is possible to run commands on the frontend machine, but this usage should be limited to simple file management, text editing, compilation and other tasks which do not represent a prolonged computational load. This is for the following reasons:

• the frontend machine is not very fast
• performance on the frontend will vary with what other users are doing at the time
• the speed of execution will be affected for everyone if one user runs a great load on the frontend

Thus, work which can be performed on the compute nodes should be performed there.

Running without the batch system

It is possible to run on the nodes without submitting jobs to the batch system, but this is actively discouraged. Such jobs may be terminated without notice, and preserving their execution will not be a priority during maintainance work.

Usage

The batch system is controlled using files (henceforth called 'job scripts'), which are simple text files describing the work at hand and its resource requirements. Users create such files for their computations, and submit them to the system for execution.

Lifespan of a job

In brief, what happens when a job is submitted is that

• the job script is accepted by a program which analyzes the resource requirements (the pbs_server program)
• the job script is passed to a program which decides on time and machine(s) for execution (the maui program)
• at the scheduled time, a program on the nodes starts the job on the user's behalf (the pbs_mom program)
• optionally, program and output messages are put into files in the user's home directory at the end of execution

Script format

A job script begins with a description of resource requirements. These are the lines which will be read when pbs_server performs its analysis, and they all begin with #PBS. pbs_server stops reading when it encounters a line consisting only of #. After this line, the remainder is an (optional) shell script which will be executed on one of the compute nodes allocated to the job. A very simple example job script looks like this:

#PBS -l nodes=1
#
hostname

The #PBS -l nodes=1 flag to PBS specifies a resource requirement of one computational node. A number of such flags exist, but they will be detailed elsewhere - specifying the number of nodes will suffice for this example.

Running the script

In order to run this very simple example, use a text editor to enter the above lines into a file, e.g. MyScript.pbs. To submit this script to the batch system, execute the command

qsub MyScript.pbs

To view the status of all presently executing scripts, execute the command

qstat

When script execution has finished, the job will disappear from the queue, and no longer be visible with 'qstat'. At this time, two files will have appeared in the directory from which the script was submitted: MyScript.o####, and MyScript.e#### (where #### symbolizes the job number which was assigned to the execution of the script). MyScript.o#### will contain the output of the program which was run. In this case, all the 'hostname' program does is to print the network name of the machine which executes it; hence, the output file will contain the name of the node your script was run on. MyScript.e#### contains error messages from the program, and should be empty in this case. Note that the job number is also visible in the output of qstat while the job is running; this number will be referred to in connection with a few commands which produce diagnostics during run time. Finally, a job may be terminated before it is finished by using the command

qdel (job-number)

where job number is as described above.

Finding your application program

It should be noted that the above example works because the 'hostname' program is installed as a system command on all nodes. All batch jobs will start directly under your home directory, regardless of which directory you were working in when you executed the qsub command. In case you are running a program of your own making, it is therefore necessary to locate it in the file tree before it is executed. This can be done in several ways, as illustrated by the following 3 examples; all will run a program my_program located in the directory path/to/executable under the home directory of a user called foo.

• This will explicitly specify the entire path to the program to be run:

/home/foo/path/to/executable/my_program

• This will change working directory (relative to foo's home directory), and start the program:

cd $HOME/path/to/executable
./my_program

• This will change working directory to whichever was the working directory at the time the job script was submitted:

cd $PBS_O_WORKDIR
./my_program

The environment variable $PBS_O_WORKDIR will be set to whichever directory was the working directory when the qsub command that started the job script was executed. Thus, the final example will only work if the job script to run my_program is kept in the path/to/executable directory, next to my_program itself.