Frequently Asked Questions

How do I copy my data from the /scratch disks?

• For fimm:
  

  Each local disk has ca 70 Gbyte space that can be used for temporary storage.

  All nodes on fimm can access each other nodes /scratch file system via NFS as /net/$nodename/scratch/, so the beginning of a job script may look like this:

  #PBS -S /bin/bash
  #PBS -N run1
  #PBS -l nodes=2:ppn=2
  #PBS -l walltime=00:05:00
  
  ## create a unique workdir. at each node, and copy the input data there:
  
  for node in `cat $PBS_NODEFILE | sort | uniq` ;
  do
     mkdir -p /net/$node/scratch/$PBS_JOBID
     cp $PBS_O_WORKDIR/indata.dat /net/$node/scratch/$PBS_JOBID ;
     ...
  done
  
  ## start the computation
  
  mpiexec ...
  

• For fire:
  

  Each local disk has ca 11 Gbyte space that can be used for temporary storage. Other from that, the setup of scratch disks is identical to that described for fimm.

My Fortran program segfaults / stack problems

• If this happens to a program that runs fine elsewhere, the program is very likely out of stack space (a memory segment available to the program at run-time). Try increasing the stackspace to unlimited. If you are using bash, execute the command 'ulimit -s unlimited' just before the program is started. For tcsh, the command is 'limit stackspace unlimited'.

  We recommend using Fortran 9X and the ALLOCATE statement for huge arrays, ALLOCATE always puts data on the heap. If, however, you have some "legacy" Fortran code you don't want to change too much, it may help to use a compiler option that allocates local arrays on the heap, instead of on the stack. For Portland pgf90 this option is -Msave. For Intel ifort the option is -save.

• MPI programs started with mpiexec may still segfault because they inherit the systems default stack size instead of the user's default. We propose two alternative workarounds for this problem:
  1. make a wrapper script which sets the stacksize and then call the program:

     /avle@fimm: > cat jobscript.pbs 
     #PBS -N test -l nodes=1,walltime=01:00:00 -A parallab -S /bin/bash
     cd $PBS_O_WORKDIR
     mpiexec ./job
     exit $?
     /avle@fimm: > cat job
     #! /bin/bash -
     ulimit -s unlimited
     exec ./a.out $*
     /avle@fimm: > 
     

  2. Download this function, compile it like e.g.

     /avle@fimm: > gcc -c set_stack_space.c
     

     In the beginning of the Fortran MPI program, just after the call to MPI_INIT, call the function to set the linux kernel stack limits directly, via the SYSV setrlimit library function. see e.g. "man setrlimit":

     ...
     integer(kind=8) :: hard_limit=50000, soft_limit=50000
     ...
     call set_stack_space(hard_limit, soft_limit)
     ...
     

     Compile/link your program like this

     /avle@fimm: > mpif90 program.f90 set_stack_space.o
     

     If the given hard_limit and soft_limit are smaller than 0, the call to set_stack_space will be equivalent to the command

       ulimit -s unlimited
     

     in bash. If successfull, the call will return 0 in hard_limit and soft_limit.