Debugging

TotalView # dbx # pdbx # xldb # compiler options

TOTALVIEW

The best known tool for debugging parallel (OpenMP and MPI) programs is TotalView. It is also a great tool for debugging sequential programs. TotalView comes with a graphical interface and many features. To use it on the IBM p690 Regatta or the Linux cluster, compile and link your program with the -g option. On the Regatta you must first run

  module load totalview

to set the correct environment variables. Then type

  totalview [name of executable]

to start the debugger. If the program takes arguments, these can be appended to this line after the flag -a. On the Linux cluster, if the program is an MPI application, and you are using MPICH (our default), and you want to debug a 2 process job, start the debugger like this

   mpirun -dbg=totalview -np 2 [program name] [program arguments]

We have separate info for how to start Totalview if you use Open MPI (near the end of this page) or LAM (near the end of this page)

Once inside the debugger, if you cannot see any source code, and keep the source files in a separate directory, add the search path to this directory via the main menu item File->Search path.

Source lines where it is possible to insert a breakpoint are marked with a box in the left column. Click on a box to toggle a breakpoint.

Double clicking a function/subroutine name in a source file should open the sourcefile. You can go back to the previous view by clicking on the left arrow on the top of the window.

The button "Go" runs the program from the beginning until the first breakpoint. "Next" and "Step" takes you one line forward. "Out" will continue until the end of the current subroutine/function. "Run to" will continue until the next breakpoint.

The value of variables can be inspected by right clicking on the name, then choose "add to expression list". The variable will now be shown in a pop up window. Scalar variables will be shown with their value, arrays with their dimensions and type. To see all values in the array, right click on the variable in the pop up window and choose "dive". You can now scroll through the list of values. Another option (very useful) is to visualize the array: after choosing "dive", open the menu item "Tools->Visualize" of the pop up window. If you did this with a 2D array, use middle button and drag mouse to rotate the surface that popped up, shift+middle button to pan, Ctrl+middle button to zoom in/out.

click to see full screenshot

Note that the TotalView license only allows two users simultaneously, so please use it efficiently. Terminate your TotalView session as soon as you do not use it or not really need it in order to give other users a fair chance of access.

The license on the Regatta is a 32-cpu license, that is, it allows TotalView to be used on programs that use at most 32 cpus. The license on the cluster is a 4-cpu license.

Getting started: http://www.etnus.com/

DBX

The dbx debugging tool on the IBM Regatta works similar to dbx tools on other systems. Compile and link the files you want to debug with the -g option. Then run

  dbx [executable]

You are now in the interpreter. Some useful commands are (shortcut in braces):

list	(l)	List the source code from the current position
run	(r)	Run the program from the beginning, until the end or a breakpoint
continue	(c)	Run the program from the current position until the end or a user defined break point.
stop	(st)	Set a breakpoint. E.g., 'stop at 120' sets a breakpoint at line 120 in the current source file. 'stop in test3' sets a breakpoint at the first executable statement in the routine test3.
step	(s)	Move a single step forward.

PDBX

The pdbx tool works similarly to dbx, but can debug parallel programs, as one of the few dbx-tools out there. From the pdbx prompt you can single step one thread at a time or a group of threads, set breakpoints, switch context (jump to another thread) etc. see the manual page (type 'man pdbx') for more information. The example session with pdbx below shows what happen when a divide by zero is encountered in pdbx after compiling the code with the option

  -g -qflttrap=ov:und:zero:inv:enable -qsigtrap

the program is started with pdbx as shown below. we first enter process 1 with "on 1" and then print the source from line 17 to 24 with "l 17,24". then we enter all processes and start the program. After crashing, we enter process 2 and gives the "where" command to see where we stopped.

avle@tre $ pdbx ./a.out -procs 4
pdbx Version 3, Release 2 -- Aug 26 2003 22:31:38
   2:reading symbolic information ...
   0:reading symbolic information ...
   3:reading symbolic information ...
   1:reading symbolic information ...
   2:[1] stopped in _main at line 18 in file "bw.f90" ($t1)
   2:   18   call start_mpi()
   3:[1] stopped in _main at line 18 in file "bw.f90" ($t1)
   3:   18   call start_mpi()
   0:[1] stopped in _main at line 18 in file "bw.f90" ($t1)
   0:   18   call start_mpi()
   1:[1] stopped in _main at line 18 in file "bw.f90" ($t1)
   1:   18   call start_mpi()
0031-504  Partition loaded ...
pdbx(all) on 1
pdbx(1) l 17,24
   1:   17   
   1:   18   call start_mpi()
   1:   19   allocate(a(n))
   1:   20   
   1:   21   do target=0, qmpi_num_proc-1
   1:   22       call barrier
   1:   23       if(qmpi_proc_num == target) then
   1:   24           write(*,'(a,i0,a)') 'MPI processor ', target, ' is node:'
pdbx(1) on all
pdbx(all) c
   0:MPI started with 4 processors
...
   1:Trace/BPT trap in crash at line 58 in file "bw.f90" ($t1)
   1:   58     print *,a/b
   3:Trace/BPT trap in crash at line 58 in file "bw.f90" ($t1)
   3:   58     print *,a/b
   2:Trace/BPT trap in crash at line 58 in file "bw.f90" ($t1)
   2:   58     print *,a/b
   0:Trace/BPT trap in crash at line 58 in file "bw.f90" ($t1)
   0:   58     print *,a/b
pdbx(all) where
   0:crash(0x2ff220d8, 0x2ff220dc), line 58 in "bw.f90"
   0:_main(), line 32 in "bw.f90"
   1:crash(0x2ff220d8, 0x2ff220dc), line 58 in "bw.f90"
   1:_main(), line 32 in "bw.f90"
   3:crash(0x2ff220d8, 0x2ff220dc), line 58 in "bw.f90"
   3:_main(), line 32 in "bw.f90"
   2:crash(0x2ff220d8, 0x2ff220dc), line 58 in "bw.f90"
   2:_main(), line 32 in "bw.f90"
pdbx(all) on 2
pdbx(2) where
   2:crash(0x2ff220d8, 0x2ff220dc), line 58 in "bw.f90"
   2:_main(), line 32 in "bw.f90"
pdbx(0) on all
pdbx(all) 
...

you can use pretty much the same commands as dbx to inspect variables. please consult the manual page for more advanced usage.

XLDB

A debugger that comes with IBM's XL compilers is the graphical tool 'xldb'. Compile and link your program with -g and execute

  xldb [executable]

The options are described in more detail in the online help text inside xldb. From inside xldb, left-click Help in the commands window to see an index of help topics. For help on a particular window, press the help key (F1) while the mouse key is in the window.

Note: the installed version only supports 32-bit compiled executables.

COMPILER OPTIONS

The following options are available for the XL compilers on the IBM Regatta system. Most of the compiler options mentioned below generate extra code and degrade performance. They should only be used for error determination. See the manual pages for additional information.

• (Fortran/C/C++) Trapping floating point exceptions

  When you do something 'illegal' in a program, for instance divide by zero, a so-called floating point exception occurs. This temporarily aborts the ongoing operations in hardware to send a signal to the operating system. This generally slows down your program considerably. So it is a good idea to track down and eliminate these bugs, both from an accuracy point of view and speedwise.

  The way to detect such floating point exceptions is to compile with the -qflttrap option. An example for the most common exceptions is (Fortran compiler):

    xlf90 -qflttrap=ov:und:zero:inv:enable -g ...
  

  This example makes your program abort when one of the exceptions OVerflow, UNDerflow, division by ZERO or INValid operation occurs. Remove the exceptions you do not need, or add others. The enable option turns trapping on, and must always be present if you want your program to abort.

  It is also useful to run the program in the debugger dbx/pdbx, these will throw you out of the execution approximately at the line the exception occurs.

  See the user guide on the languages page for details on other exceptions.

• (Fortran/C/C++) Procedure interface checking

  The compiler option -qextchk performs external name type-checking and function call checking.

• (Fortran/C/C++) Initialization of automatic variables

  The compiler option -qinitauto[=hex_value] initializes each byte or word of storage for automatic variables to a specific value (typically 00 or FF), to help locate variables that are referenced before being defined.

• (Fortran) Disable implicit data typing

  The compiler option -u (or -qundef) detects undeclared variables. It disables the implicit data typing feature in Fortran.

• (Fortran) Array bound checking

  The compiler option -C performs run-time checking of array bounds and character substring expressions.

• (C/C++) Various run-time checks

  The compiler option -qcheck[=option] generates code for run-time checks. The value for option can be

  • nullptr - enable runtime checking of addresses contained in pointer variables used to reference storage.
  • bounds - enable runtime checking of addresses when subscripting within an object of known size.
  • divzero - enable runtime checking of integer division. A trap will occur if an attempt is made to divide by zero.
  • all - enable all the above suboptions.