Running LSF batch jobs

From Hpc

The Origin systems use the Platform Computing LSF batch queuing system to run batch jobs. LSF lets you submit batch jobs to a queue and will run jobs when requested resources are available. You submit jobs with bsub to one of the available queues. Options to bsub can be specified either as a line in a script (known as an embedded option) or on the command line. If embedded options are used, the script must be submitted with

bsub <jobscript

where jobscript contains job submission options (lines beginning with #BSUB) as well as command lines to execute. For some of the most used bsub options see sample batch script below. You can also submit batch jobs with options to bsub command line options

bsub -o job.out -e job.err a.out

which will submit a.out to the (default) normal queue, and store standard output and standard error in the files job.out and job.err respectively. However, we strongly recommend that you submit batch jobs using a job script.

To ensure agreement between the number of processors you specify for your job using the -n option to bsub and the number of processors during execution you are adviced to use the LSB_MCPU_HOSTS environment variable. This is set at job submission and contains the number of CPUs requested from LSF. Using this variable (see sample batch script below) you only have to change the number specified at the option line (#BSUB -n ...) when later running your job with a different number of processors.

To make I/O more efficient always copy your inputfiles to the /work/$LOGNAME directory. Also temporary store outputfiles at this location. After job execution move the outputfiles to your home directory. Notice that files which have not been accessed for a certain amount of time, normally 14 days, will be automatically deleted, this is subject to change but it is not expected to be less than a week.

For a complete description of the bsub command and its options refer to the man page.

Queue overview

LSF batch queue summary (subject to change without prior notice)

Queue	Priority	max #slots	max #slots/user	max time	max stack	Description
forecast	60	-	-	-	4 GB	Top-priority queue dedicated to forecast jobs
express	43	96	64	30 mins	2 GB	High priority short test jobs
normal	30	576	128	3 days	2 GB	The default queue for parallel jobs
monster	20	560	256	-	4 GB	Public long jobs
gaussian	19	96	64	-	512 MB	For gaussian jobs
single	18	96	96	-	4 GB	Public single cpu long jobs
nonpri	1	680	400	-	4 GB	Lowest priority queue for nonpriority quota jobs

The parameters max #slots and max #slots/user may change dynamically depending on system load. Log on and type

bqueues -l

for current settings.

Single-CPU production jobs must be submitted to the single queue.

The queue priority scheme governs the way jobs are run. In general, jobs with a higher queue priority will run ahead of jobs with a lower priority. If a job is running and a higher priority job arrives, the job will if necessary be suspended, to let the higher priority job run. Use the bqueues or qstat commands to learn more about queues.

A sample batch job script

#!/bin/ksh
#BSUB -J myjob        # Assigns the specified name to the job
#BSUB -q normal       # Submits the job to the specified queue
#BSUB -o job.out      # Appends the standard output to the specified file
#BSUB -e job.err      # Appends the standard error to the specified file
#BSUB -N              # Sends the job report by mail when the job finishes
#BSUB -n 4            # Specify the number of processors

set -e

# Get the number of processors requested from LSF
nprocs=$(echo $LSB_MCPU_HOSTS | sed s'/.* //')              # ksh syntax

# Create (if necessary) and move to my working directory
w=$WORKDIR/$LOGNAME
if [ ! -d $w ]; then mkdir $w; fi
cd $w

# Copy inputfiles to working directory
cp $HOME/what_ever_path/inputfiles .

# Run my MPI program
mpirun -cpr -np $nprocs $HOME/what_ever_path/my_mpi_program

# Move results back to home directory
mv outputfiles $HOME/what_ever_path/

exit 0