FAQ
From Vd
|
Please contact hpc@usit.uio.no for suggestions, questions, assistance etc.
Introduction
Titan is a PC cluster consisting of:
- 448 dual-cpu-dual-core (4 cores) SUN X2200 1u nodes, with 16 - 32 GB of RAM and 512 GB - 1 TB of local disk on each node, 150TB shared GPFS storage and InfiniBand
- 96 dual-cpu-dual-core (4 cores) SUN X6220 blade nodes, with 16 GB of RAM and 512 GB of local disk on each node, 150TB shared GPFS storage and InfiniBand
- 8 eight-cpu-dual-core (16 cores) SUN X4600 4u nodes, with 32-256 GB of RAM and 1.5 TB striped local /work on each node, 150TB shared GPFS storage and InfiniBand
- 80 dual-cpu Dell nodes with 4 GB of RAM and 2x80 GB of local disk on each node, 12TB GPFS and InfiniBand
- About 30 servers running portals, services, administration, etc.
Access to the cluster is managed through login-nodes: titan.uio.no
This is the normal gateway to gain access to the resources. To log inn
please do:
ssh titan.uio.no
and type in username/password.
Having logged on to the cluster, each user has access to both a home-directory, /usit/titan/u1/username, and a datacatalog, /data1/d1/username for semi-permanent data. Usually about 10 GB can be used for the home-directory and about 100 GB can be used for semi-permanent data. If you have greater storage needs, please contact us and we will solve this somehow.
A queue system is used to manage jobs. Both batch jobs and interactive jobs can be run through the queue system.
All jobs on the compute nodes must be run through the queuing system. Only computationally non-intensive jobs should be run on the login node (such as file editing and so on).
In the queue system, interactive jobs can be run by using the qlogin-command, while batch jobs can be run using the qsub-command.
Installed applications and libraries are usually found on /site/. If you wonder if we have an application or library installed, please consult our application list (see section below) or log into titan and look at /site.
If this doesn't fit your running profile, you need a new application or have any other special needs, please feel free to contact us at hpc@usit.uio.no.
GETTING STARTED
A short intro on how to getting started on TITAN.
MODULES
To simplify setting up environments for compilers, mpi-versions and some precompiled applications, Titan has installed the Environment Modules package. This provides means for the dynamic modification of a user's environment via modulefiles.
Each modulefile contains the information needed to configure the shell for an environment. The environment can then be modified on a per-module basis using the module command which interprets modulefiles. The most common commands for use are listed below.
- Tip
- Modules can be loaded and unloaded dynamically and atomically, in a clean fashion from bash, ksh, zsh, sh, csh, tcsh as well as perl and python.
A list of available module environments at the moment are:
adf blcr breve compython g03 ibmpich intel local mpich2 mpich2.pgi openmpi openmpiib pgi scampi
The modules can be loaded/unloaded on command line or within your scripts by typing:
module load environment module unload environment
You will get a list of current loaded modules by
module list Currently Loaded Modulefiles: 1) modules 2) local/1.0.0 3) pathscale/3.0 4) scampi/5.4.0
Modules defined and available to you can be listed by
module avail ------------------------- /data1/share/etc/modulefiles ------------------------- R/2.4.1(default) local/1.0.0(default) acml.gfortran/3.6.0 matlab/7.2(default) acml.gnu/3.6.0 mpich2/1.0.5p3(default) acml.ifort/3.6.0 mpich2.intel/1.0.5p3(default) acml.pathscale/3.6.0 mpich2.pgi/1.0.5p3(default) acml.pgi/3.6.0 mvapich2/1.0 adf/2005(default) openmpi/1.2.0(default) adf/2007 openmpiib/1.2.0(default) blcr/0.6.1 paralleloctave/x.x.0(default) breve/2.4.1(default) pathscale/3.0 cmiss/x.x.0(default) perl/1.0(default) compython/2.1(default) pgi/6.2(default) ibmpich/1.0(default) scampi/5.4.0(default) intel/9.1(default)
A help text exists for some of the modules, to view it, type
module help module
To view exactly what a module is doing to your environment, type
module show modulefile
For further documentation, please refer to the Environment Modules homepage
Personalize default modules
You may add loading of modules to your .bash_login file to make sure your favorite module is always loaded when logging in. To always have the intel module load on login, add:
module load intel
to the .bash_login file.
COMPILING AND TESTING
Compiling and testing can now be done on both the login node and the compute nodes. Both nodes resemble the running environment closely. Please see the Interactive logins paragraph on how to access the compute nodes interactively.
COMPILERS
In addition to the gnu compilers gcc, g++, g77 we have the following compilers:
- Portland v6.2.5 (updated february 15. 2007)
- Intel v9.1 (updated february 10. 2007)
- Pathscale
PORTLAND GROUP (PGI)
The Portland compilers need special environment settings. Thus, to use the Portland compilers you need to load the module pgi:
module load pgi
This module must also be loaded prior to running any application compiled with Portland.
INTEL COMPILERS
The Intel compilers need special environment settings. Thus, to use the Intel compilers you need to load the module intel:
module load intel
This module must also be loaded prior to running any application compiled with Intel.
PARALLEL ENVIRONMENTS
Since Titan consists of multicpu and multicore compute nodes, there are several ways of paralellizing code and running in parallel. This can either be by using mpi, openmp and threading. In this section we will explain how to run in parallel in either of these environments. For more information about the queue system parameters, read the queue system section below.
OPENMP and threading
In order to run in parallel on one node using either threading or OPENMP, the only thing you need to remember is to tell the queueing system that you want an entire node. This can be done on the old dual cpu single core and new dual cpu dual core nodes by specifying:
-pe exclusive 2
and:
-pe exclusive 4
respectively in your job-script or interactive login line. See the queue system section for more information about job-scripts or interactive logins.
USEFUL OPENMP OPTIONS
In addition to make as much "stack" available as possible (issue "ulimit -s unlimited") it might be necessary to set up more memory reserved pr. thread (for x86_64 Intel reserves 2MB). This can be done as follows:
KMP_STACKSIZE=32m
where m specifies Megabytes.
One can specify how many threads to use by using exporting
OMP_NUM_THREADS=4
MPI
Titan v.II has several mpi environments installed, which can be used through using modules to set up you environment by invoking:
module load mpi-version
where mpi-version can either be:
- scampi - for Scali MPI
- mpich2 - for MPICH2
- openmpi - for OpenMPI
You need to invoke a mpi module both in order to compile your code and to run it. You also should compile and run in the same mpi-environment. Although some mpi-versions may be compatible, they usually are not.
To run mpi, you first need to specify this to the queueng system. There are sevearal ways to do this, depending on the configuration of your nodes. The most common would be to use:
-pe mpi $numprocs
where $numprocs is the number of processors you would like to use, in your job-script or interactive login line. In addition you need to load the correct environment in your job-script (or after you have logged in interactively). Finally you just invoke:
mpiexec executable
This means that a typical jobscript for an eight processor job using Scali-MPI, would look like:
#!/bin/sh # 8 cpus with mpi #$ -pe mpi 8 # project geo #$ -P geo source /site/bin/jobsetup module load scampi # move the spindoctor program to $SCRATCH on each node (see # SPREADFILE section below. spreadfile $SGE_O_WORKDIR/spindoctor $SCRATCH # move to scratch cd $SCRATCH # start program /site/bin/mpiexec ./spindoctor #END OF SCRIPT
Note that there are several parallel environments available in the queue-system, that are useful for different types of jobs. More on this in the next section
QUEUE SYSTEM OPTIONS
Using the queue option mpi in the previous example, we have already assumed that it doesn't matter how your parallel job is run. Running in this environment you may get 1 processor on a node where there already are running e.g. 3 other processes (somebody elses). Your process will then compete for local disk and network with these other processes. To alleviate these problems you may specify several other mpi-environments to the queue-system, depending on the type of mpi job you want to run.
As previously described you may specify:
-pe mpi $numprocs
where the queue-system tries to allocate as many processes as possible on each node. Although this environment does not require the queue-system to allocate cores in a certain way.
The second possibility is to specify:
-pe mpi1 $numprocs
where the queue system will allocate only one process pr. node. This last option is useful if you have mpi-jobs that needs more than their portion of memory.
The last parallel environment you may use is
-pe exclusive $numprocs
where $numprocs is divisble by 4. This ensures that you get each node for yourself, i.e. exactly 4 cpus on each node.
USEFUL COMMANDS IN MPI-SCRIPTS
We have made several commands available that are useful for setting up mpi-jobs. For instance you may need files on each of the nodes, or you need to run commands on the nodes.
For spreading files you can use the command spreadfile, which has the syntax:
spreadfile file-with-path destination
(see the above example script for an example of use).
The second command dsh, executes a command on each of the nodes. This has the syntax:
dsh command
MPI TYPES
Heres a short description of each of the mpi-types.
Scali-MPI
Scali-MPI or scaMPI, is one of the mpi-types installed. To compile use the wrapper compilers (mpicc, mpif77, mpic++...). These can be accessed by loading the scampi module.
By default these wrapper-compilers use the gnu compilers. In order to for instance use the intel c and fortran compilers, specify:
mpicc -ccl icc
and:
mpif77 -ccl ifort
respectively.
MPICH2
Similar to Scali-MPI, MPICH2 has wrapper compilers (mpicc, mpif77,mpic++...). These can also be accessed by loading a module, the mpich2 module.
By default these wrapper-compilers use the gnu compilers. In order to for instance use other compilers:
mpicc -cc=icc
and:
mpif77 -f90=ifort
respectively.
MVAPICH2
MVAPICH2 is based on MPICH2. Application programmers should not find any differences. The only difference is that MVAPICH2 uses Titan's infiniband interconnect. Not all nodes have infiniband, and you may not have access to them. Make sure you specify -l ib or -l mxib with your submit-command.
NOTE Currently we have not been able to make MVAPICH2 work with our infiniband switch.
Open-MPI
Open-MPI is a project combining technologies and resources from several other projects (FT-MPI, LA-MPI, LAM/MPI, and PACX-MPI). It is a completely MPI-2 compliant implementation. See http://www.open-mpi.org for more information.
Wrappers for compiling within the openmpi environment (mpicc, mpic++, mpiCC, mpicxx, mpif77, mpif90) can be used by loading the openmpi module. Default for these wrappers are using the intel compilers, however these can be used with any compilers and any flags by using the environment variables in the table below.
| Wrapper | Compiler | Preprocessor Flags | Compiler Flags | Linker Flags | Linker Library | Data File |
| mpicc | OMPI_CC | OMPI_CPPFLAGS | OMPI_CFLAGS | OMPI_LDFLAGS | OMPI_LIBS | mpicc-wrapper-data.txt |
| mpic++ | OMPI_CXX | OMPI_CPPFLAGS | OMPI_CXXFLAGS | OMPI_LDFLAGS | OMPI_LIBS | mpic++-wrapper-data.txt |
| mpiCC | OMPI_CXX | OMPI_CPPFLAGS | OMPI_CXXFLAGS | OMPI_LDFLAGS | OMPI_LIBS | mpiCC-wrapper-data.txt |
| mpif77 | OMPI_F77 | OMPI_CPPFLAGS | OMPI_FFLAGS | OMPI_LDFLAGS | OMPI_LIBS | mpif77-wrapper-data.txt |
| mpif90 | OMPI_FC | OMPI_CPPFLAGS | OMPI_FCFLAGS | OMPI_LDFLAGS | OMPI_LIBS | mpif90-wrapper-data.txt |
COMBINATION MPI-OPENMP
In order to run mpi and openmp in combination, you need to specify:
-pe mpiomp $numprocs
in your job-script or interactive login line. For more information about job-scripts or interactive logins, please se queue system section. This will set up one mpi process pr. node, and reserve the rest for running openmp.
Secondly you need to both set up your mpi-environment (by module load) and your openmp environment. After this is done, you just need to execute:
mpiexec executable
COMBINATION MPI-THREADING
This is very similar to the combination mpi-openmp, only you don't have to set up the openmp environment. The same specifications to the queue-system can however be used.
BENCHMARKING
If you need to benchmark you application, you will want to run the code exclusively. To get access to an exclusive node (a node with 4 CPUs), specify the following in your job-script:
#$ -pe exclusive 4
If you want to run larger benchmarks on multiple nodes, you will want to ensure that you get nodes that do not get disturbance from other jobs using the same switches etc. Please email hpc@usit.uio.no and we will help you allocate nodes in a suitable part of the cluster. hpc@usit.uio.no is always interested in a copy of the benchmarking results as benchmarking results may be used as an indication of how well the cluster is functioning and it might pinpoint problems or dictate what future changes is needed in the cluster setup.
INTERPRETED LANGUAGES
With interpreted languages, we mean languages which does not need to be explicitly compiled, e.g., Python, Perl, R, MatLab, Octave and others. Titan has most of the standard interpreted languages, but if you feel that some version is to old or you don't find the language of your choice, don't hesitate to contact us at hpc@usit.uio.no.
Below you will find more information on some of the languages.
R
R is a software environment for statistical computing and graphics. For more information about R itself, see the official documentation.
On Titan you can reach a tuned R version by using qlogin and invoke
module load R
PYTHON
Python is a dynamic object-oriented programming language that can be used for many kinds of software development. In itself Python is mostly used for system administration and file handling. However, with add-on packages like numpy and scipy, Python has proved very useful for scientific programming as well. For more information about Python, see http://www.python.org/ and for scipy, see http://scipy.org/
compython
compython is a package for doing computational tasks using Python. The package consists of Python 2.5 in addition to the following packages:
biopython-1.42 blt2.4z docutils epydoc-2.1 egenix-mx-base-2.0.6 F2PY-2.45.241_1926 gnuplot-4.0.0 gnuplot-py-1.7 HappyDoc-r2_1, Imaging-1.1.5 numarray-1.3.3 Numeric-24.2 pypar_1_9_2, reportlab_1_20 ScientificPython-2.4.9 numpy-0.9.6 scipy-0.4.8 SWIG-1.3.24 tcl8.4.9 tix-8.1.4 tk8.4.9 Pmw.1.2 pycdf-0.6-2-rc1 ipython-0.8.1 pygtk-2.0.0
To use compython you need to load the module compython, i.e., invoke:
module load compython
Then start python in the normal way, or, if you want tab-completion, run
ipython
If you want do use the package pypar_1_9_2, you have to load the module mpich2 as well, see the MPI section.
MATLAB
MATLAB is a high-performance language for technical computing. More information on MatLab can be found at http://www.mathworks.com/ and, to get started, http://www.mathworks.com/access/helpdesk/help/techdoc/matlab.html
To run MatLab on Titan, qlogin to a compute node and invoke:
module load matlab
OCTAVE
GNU Octave is a high-level language, primarily intended for numerical computations. It is mostly compatible with MatLab, but open source and free. More information about octave itself can be found on http://www.gnu.org/software/octave/
PERL
More information about Perl itself can be found on http://www.perl.org
QUEUE SYSTEM
To run a job on TITAN, you put the job into a queue, and it starts when a suitable compute node (or cluster of nodes) is available. The queue is managed by a queue system called SGE. Sun's SGE user's guide.
You submit your job to SGE along with a list of requirements (memory usage, number of CPUs, run-time, etc.), and the queue system will find a machine to run your job on. It will automatically find all the machines which meets your requirements, then remove any heavily-loaded machines from that list. E.g. if machine compute-5-5 has a small job running which leaves open 1200MB of memory and compute-13-13 has a large job running which leaves only 200MB of memory free, then SGE will schedule your large-memory job onto compute-5-5. Of course, this implies that you told SGE how much memory you need (see below).
To see the queue, run the command showq or qstat:
showq options: [-r] show queue utilization [-rp] show queue/project utilization [-ru] show queue/project/user utilization [-rj] show queue/project/user/job utilization [-p] show pending list for each project [-pu] show pending list for each user [-pj] show all pending jobs [-Q qname] show only queues matching qname [-P pname] show only projects matching pname [-U uname] show only users matching uname
Examples:
showq -P norgrid -rj # shows all running jobs for project norgrid showq -U zfc -pj # shows all pending jobs for user zfc showq -Q libero -P astro -rj # shows libero-jobs for project astro
TITAN is divided into projects. Each project has a certain number of nodes assigned to it. To see which projects are available, run the command showq -r.
To run a "job" involves a Unix-shell script which may execute any number of commands, including your main computational task. i.e. you may copy or rename files, cd into the proper directory, etc., all before you do the "real" job. The lines in the script file are the commands to be executed, in order. Lines starting with a "#" are ignored as comments, except lines that start with a "#$" which are not executed, but contain special instructions to the SGE system. If you are not familiar with shell scripting, it is simply a set of commands that you would have typed at the command line.
On TITAN, the jobscript must contain a few lines to import functions and variables, and to load any necessary modules. Instruction parameters to SGE may be specified on the command line and and in the jobs script. There can be as many #$ lines as you need. If a parameter is specified both on the command line and in the jobscript, the parameter specified on the command line takes precedence.
Note: On TITAN, walltime and project must be specified. If walltime or project is not specified, the job will not be started!
A simple jobscript
#!/bin/sh # #Project #$ -P metos # #Walltime #$ -l s_rt=120:0:0 source /site/bin/jobsetup # copy data file to work area # $SGE_O_WORKDIR is the directory you ran qsub from spreadfile $SGE_O_WORKDIR/input.dat $SCRATCH chkfile output.dat cd $SCRATCH ~/mycode/myprogram input.dat cp output.dat ~/mydata/ # end of script
You submit the jobscript with the qsub command:
qsub jobscript.sge
Note that all the commands in the jobscript is done on the compute-node issued by the queue system.
Note again that all jobs must specify a project and a walltime limit. The TITAN queue system will not start jobs lacking the "#$ -P project" or "#$ -l s_rt=hour:min:sec".
Splitting a job into tasks
To run many instances of the same job, use the -t parameter. This is useful if you have a lot of data-sets which you want to process in the same way with the same job-script.
-t 1-100
means to run 100 instances, numbered from 1 to 100-
You can also specify task numbers with a "stride":
-t 1:100:5
means to increase the task number by five for each task. This example will give 20 tasks: The first is number 1, the last is number 96.
The task id is available in the jobscript as the environment variable $SGE_TASK_ID. You could pick up the environment variable and use it for choosing the right data set:
#!/bin/sh #$ -t 1:100 ... cd $SCRATCH DATASET=$SGE_O_WORKDIR/dataset.$SGE_TASK_ID cp $DATASET $SCRATCH runmyprog $DATASET > result.$SGE_TASK_ID cp $SGE_TASK_ID $SGE_O_WORKDIR # end of script
This job will process the datasets dataset.1, dataset.2, ... dataset.100 and leave the results in result.1, result.2, ... result.100
Your job-instances will have names like 196755.1, 196755.2 etc.
You may delete individual instances with e.g.
qdel 196755.1
If you want to delete the whole job, use
qdel 196755
The instances of an array job are independent, they have their own $SCRATCH and are treated like separate jobs. You may also specify a parallel environment in array-jobs, so that each instance gets e.g. 8 processors.
Large memory jobs
Most nodes are equipped with 16GB of memory, a few have 8GB, a few have 32GB, one has 64GB (not publicly available), a few have 128GB and one has 256GB memory. If you need to use a lot of memory, there are two things to remember:
You must specify in your job-script that you need a large memory node, you do this by requesting the mem_total resource with the submit-option
-l mem_total=25G
Next, the default memory limit (data-segment) for jobs have been set to 20GB, you should also increase this limit by requesting the resource
-l h_data=25G
otherwise your job will be killed when it allocates memory above 20GB (see man setrlimit (RLIMIT_DATA)). For mpi-jobs these limits are per process. This means that if you specify
-l mem_total=25G
and
-pe exclusive 4
you try to get 4 processes on a single node, each requesting 25GB of memory. Since we don't have any node with 100GB of memory, the job will never start.
Disk intensive jobs
If your jobs is disk-intensive, you would not want to share the node with anybody else. There is a resource on each node called disk. This resource has a limit of 4 on each node. If your job is disk-intensive you should specify
-l disk=4
This ensures that the processes of your job will run on different nodes.
Jobs using more than 2 gigabytes of memory
If your job needs more than 2 gigabytes of memory, you should specify a flag that tells the queue-system that you have specific memory needs. In these cases you should use
-l mem_total=4G
to get a node with 4 gig available on each core.
Note that there are other flags to specify memory usage, but mem_total is the best, since it does not kill your job if memory goes above the stated limit. However you should write an appropriate limit, since a to low limit may hamper the performance of the calculation.
The libero queue
It's possible to run your program om somebody else's node if they're not using it. To do this you specify that you want to use the libero resource. This is done by specifying:
-l libero=true
in your script.
If the owner of the node you are running on wants to use it again before your job is finished, your job will be suspended and lie dormant until the node becomes available again. Since it's impossible to say if that will take an hour or a month, it's probably better to say that you want your jobs moved to other nodes. You do this by specifying checkpointing.
Checkpointing
Checkpointing a job means that the job can be stopped and started somewhere else, and continues where it left off. TITAN offers two checkpointing methods:
-ckpt user
and
-ckpt blcr
With user, you say that your job will handle saving its data and picking up where it left off by itself, and the queue system will simply restart the job.
The blcr method tries to handle all this for you. When it works, your job will move to another node. What will happen if it doesn't work is impossible to predict. If you want to use checkpointing, always test it on a few jobs before you commit lots of checkpointing jobs. Blcr is documented at http://ftg.lbl.gov/CheckpointRestart/CheckpointRestart.shtml, we have integrated it seamlessly into the queue system.
You can force your jobs to restart by saying:
qmod -rj <jobid>
or
qmod -sj <jobid>
NOTE Unfortunately blcr has been temporarily disabled because it causes infrequent random data corruption for other users.
Common parameters
If you have options which you almost always use, e.g.
-j y -P kjemi -m e -M me@mine.nu -r y
you may put them in the file .sge_request in your home-directory, or even in your working directory. You may anyway override them in the script-file or on the command line. (See also man sge_request)
A more advanced jobscript
Below is a jobscript that starts an MPI job.
qstat -g c #!/bin/sh # 8 cpus with mpi (or other communication) #$ -pe mpi 8 # 100 hours of walltime #$ -l s_rt=100:0:0 #$ -l libero=false # project metos #$ -P metos # we need 512 MB of memory per process #$ -l s_vmem=512M # name of job #$ -N spin # The jobsetup command creates a work-dir named $SCRATCH on each node # This directory is removed upon job exit, unless we have given # the command savework # If you want automatic copying of files back to $SGE_O_WORKDIR (your # current directory when you issued the qsub command) # when the job is terminated, mark them with the command chkfile: # chkfile result.out result.chk # After copying files back, the scratch-directory $SCRATCH-directory is deleted. source /site/bin/jobsetup module load openmpi # savework # move the spindoctor program to $SCRATCH on each node spreadfile $SGE_O_WORKDIR/spindoctor $SCRATCH # On exit cleanup 'if [ -f $SCRATCH/OK ]; then cp $SCRATCH/precious $SGE_O_WORKDIR; fi' # move to scratch cd $SCRATCH # Ensure the checkpoint file is copied back, no matter how # we terminate (unless the node crashes). chkfile $SCRATCH/spin.dat # start program /site/bin/mpiexec ./spindoctor #END OF SCRIPT
Make sure you put the jobscript name at the end of the qsub command line. Everything after the jobscript name becomes command-line parameters to the jobscript. These work the same way as in a regular shellscript:
qsub -N my_job jobscript.sge 100 15 200 dmc
Interactive logins
If you need to test something interactively, you can log in interactively via the queue system with the command qlogin. qlogin takes almost the same arguments as qsub. E.g. you may use the command
qlogin -pe mpi 8 -P metos
and then run your program interactively with
mpiexec ./myprog.x
or have a look at the nodes you got
dsh 'echo $(hostname) $(uptime)'
List of SGE parameters
| Parameter | Description |
|---|---|
| -M user@hostname -m b,e | Sends email to the specified account at the beginning (-m b) and end (-m e) of the job. This way, you know when the job has finally started if it got stuck in the queue. If you do not put a valid email address in with -M then this will not work |
| -o file -e file | Directs the standard output (-o) and standard error-output (-e) to the specified files. Note that this is output or error-output that is not otherwise directed to files; ie. csh redirection (myprog > file.out) takes precedence |
| -j y | Joins the error-output with the standard output, thereby sending both to the same file (given by -o) |
| -S /bin/tcsh | What shell to use, tcsh or sh |
| -N name | Use this name when displaying in the qstat output; defaults to the name of the script file |
| -hold_jid job_id_or_job_name | Ff you have one job that must wait for another to complete (perhaps the first one creates an output file which is needed by the second program), then you can request that the job be held until that first job completes |
| -pe mpi 10-20 | Requests a "parallel environment" that spans several machines, in this case any number of CPUs between 10 and 20 (inclusive), a single number will request exactly that many CPUs |
| -l slots=2 | Requests that the job be given 2 slots (or 2 cpus) instead of 1; you MUST use this if your program is multi-threaded, you should NOT use it otherwise |
| -l mem_free=1.5G | Requests that only machines with 1.5GB (=1536MB) or more be used for this job; ie. the job requires a lot of memory and thus is not suitable for all hosts. Note that 1G is equal to 1024M |
| -l h_cpu=Xh | Requests that X hours be allocated for this job to run |
| -l scr_free=XG | Requests that only machines with X GB or more free disk space in the /work partition to be used for this job; ie. the job requires a lot of temporary file space and thus is not suitable for all hosts. Note that 1G is equal to 1024M |
| -l arch=GenuineIntel | Requests that only the 32bit Intel nodes are assigned to the job. Currently GenuineIntel and AuthenticAMD is the legal values. This may change. |
| -soft | Putting -soft before a -l requirement indicates that it is a "soft" request; SGE will make a best-effort attempt to find a machine with the requested attribute, but the job may be queue with machines that do NOT have the attribute. Note that -soft applies to ALL -l options that come after it! |
| -t start-stop:step | Submit an SGE "array" job, that is, run the same job multiple times, but set SGE_TASK_ID to the value start, then start+step, etc., up through stop (step may be omitted); it is up to your script to do something different for each task-ID |