Batch System Flags

From Hpc

The following text describes a few commonly used flags which control batch system execution. It assumes familiarity with the contents of the documents "Introduction to the Batch System" and "Using multiple nodes with the batch system".

Naming your job

By default, a batch job will be named after the script file it executes. This name will show up in the output of qstat, and will be the base name of the output and error files. If you wish to give your job a more descriptive name, it can be done with the following flag:

#PBS -N My_batch_job_name

Exporting environment variables

The flag

#PBS -V

will export the environment of the shell executing the qsub command into the batch script execution environment. Note that this environment will not be forwarded automatically to the other nodes in a node set. This flag is necessary in order to use X11 forwarding (graphical user interfaces) in batch jobs.

Specifying estimated time

If you specify an upper limit for how long your program will run, your run may be started sooner when the load is high. This can be specified with an additional attribute to #PBS -l, as follows:

#PBS -l nodes=4,walltime=00:45:00

The example will allocate 4 nodes for 45 minutes. Note that jobs which run more than 10 minutes overdue are automatically terminated.

Merging output and error messages

Most programs will not produce error messages when run correctly. In order to cut down on the file clutter which can be produced by multiple runs, it is possible to merge these two message streams into a single file.

• This flag will merge error messages into the output file:

#PBS -j oe

• This flag will merge output messages into the error file:

#PBS -j eo

Specifying queue

The batch system maintains a few different queues at different priorities, according to what sort of work should be submitted to them; the main queues are named default and long. With nothing else specified, jobs will be submitted into the default queue which has a runtime limit of 24 hours. If you require your job to run for longer than this, you may specify the flag

#PBS -q long

to submit to the long queue which has a runtime limit of 240 hours. Note that in periods when the computational load is great, submitting to long may cause a significant delay in how quickly your job is scheduled to be run.

Email updates

The batch system can email you at the occurrence of specified events. Note that email will be sent to your user on norgrid; if you wish to receive system mail to a different address, you must create a file called .forward under your home directory. In order to set up mail forwarding, execute the following commmand (substituting your own email address):

echo "username@somedomain.com" > $HOME/.forward

(This is only required once every time you want to change the email address.) The following flags prompt the batch system to email you:

• Send mail when the batch job is aborted by the batch system

#PBS -m a

• Send mail when the job begins execution

#PBS -m b

• Send mail when the job ends execution

#PBS -m e

The flags can be combined, e.g.

#PBS -m be

sends mail both when the job begins and ends.

Specifying physical rack for allocation

The following only applies to the Norgrid cluster. It is possible to specify that all allocated nodes must be in the same physical machine rack. This slightly reduces the overhead of network communication between nodes, but it will also delay the execution of the job until enough nodes are free in a single rack; this may take time when the load is high. In order to specify that all nodes should be in ONE rack (regardless of which), use the flag

#PBS -W x=\"NODESET:ONEOF:FEATURE:RACK0:RACK1:RACK2\"

To specify that all nodes should all be in either rack 0 or rack 1, use

#PBS -W x=\"NODESET:ONEOF:FEATURE:RACK0:RACK1\"

Similarly, it is possible to ask for a single specific rack. The batch system queues are somewhat coupled to which physical racks they may request; requesting nodes from a rack which is not available for the queue the job is submitted to is likely to result in an indefinitely queued job. The following table maps queues to physical racks:

Note that queue charters and related resource reservations are described in Norgrid Queues.