File systems on the Regatta (tre) and the clusters (fire, fimm)

The following file systems exist on tre, fire and fimm.

/home/parallab -- file system for user home directories on tre and fire. Files are backed up daily. (You have the same home directory on tre and fire). This is a slow but secure place to store data. Do not use this filesystem for I/O intensive jobs.

/home/fimm -- file system for user home directories on fimm. Files are backed up daily, except for folders called "scratch" or "tmp" and their sub-folders.

/local -- for local installations on fimm of popular or general-purpose software. Similar for fire. For use by system administrator only. Contact hpc-support@hpc.uib.no if you would like something installed in /local.

/scratch -- the fastest available (local) file system for each compute node of tre. Similarly, each of the compute nodes on fire and fimm has a local /scratch area. Files are not backed up. Files created in /scratch should be deleted after the job is finished. Use this area for optimal I/O performance. On a node of tre, /scratch is ca 500 GB. On a node of fire, it is ca 11 GB. On a node of fimm, it is ca 70 GB.

/work -- large external storage shared by all compute nodes on tre. Similarly, all nodes on fire share a /work area (same as on tre) and all nodes on fimm share a /work area. Files are not backed up. Not as fast as /scratch, but /work has much larger capacity. Use /work if multiple processors or multiple compute nodes need to access the same file(s), or if /scratch is too small. /work on tre,fire and fimm has automatic deletion scripts that will delete from the oldest files onward when the filesystem goes beyond 80% usage. Please delete files as soon as possible from this filesystem.

/work2 -- another large external storage shared by all compute nodes on fimm only. Same purpose as /work. Files are not backed up. Note that this filesystem does not have automatic deletion, it could therefore go 100% full. Please delete files as soon as possible from this filesystem.

/migrate -- the area for archiving data, i.e., for the automatic migration of data between disk and tape. Only available on "to" (node "to" of the regattas). Can be used upon request only. Transfer of data to or from /migrate is only meant for users that have large collections of data (tens of Gigabytes or more, typically the result of long simulations) that need to be archived and that cannot be stored in the home directories. See here for more details.

/net/bcmhsm -- "/migrate" for Bjerknes-users. Same rules as for /migrate. Is nfs-mounted on tre,to,en from the backup-server. Mounted on fimm as "/bcmhsm". There are symlinks from /migrate/username to /net/bcmhsm for the Bjerknes-users.

Correct usage. These are general rules that you should keep in mind.

• Use your home directory only to store permanent data (program source files, makefiles, scripts, compressed data files, etc.).
• Use /scratch and /work to store files that are needed or exist only temporarily. These include executables, object files, uncompressed data files, etc. Such files are typically needed only during execution of a job and can often be generated from the source files (in your home directory).
• Transfer of data from and to the /scratch area is the fastest. If your program performs intensive I/O, you should ensure that all input files, intermediate files, and output files are read from and written to the /scratch area.
• Files in /work and /scratch may be deleted automatically (without notice) if they haven't been used in the last seven days.

$HOME and $USER -- when your account on one of the machines is created, you get a personal home directory (called $HOME) and one called /work/$USER, where $USER is your account name. The processors on all compute nodes of tre have access to your home and work directory on tre. And similar for fire and fimm. You can create your own /scratch/$USER directories on each of the compute nodes.

Remote access -- on the clusters fire and fimm, a compute node can access another node's local /scratch area via the network:

• fimm: /net/$nodename/scratch, where $nodename is one of compute-0-[0-19].local, compute-1-[0-31].local, compute-2-[0-32].local
• fire: /auto/$nodename/scratch, where $nodename is one of node[2-32]

See the FAQ page on how to copy data from the /scratch areas. On tre, the batch system command 'llq jobid' lists the node on which your job is running. On fire and fimm, the command 'qstat -f jobid' or 'qstat -n jobid' lists the nodes.

Disk quota -- all users get by default a soft quota and a hard quota for their home directory. If the soft quota is exceeded for more than 7 days, or the hard quota is exceeded, you will not be able to create any more files in your home directory. You can check your disk usage (KB), soft quota (quota) and hard quota (limit) with the 'mmlsquota' command.