Profiling

Basic profiling - Microprofiling - Hardware counters - Tracing tools - Analysis tools - Other tools

BASIC PROFILING: xprofiler - gprof - prof

The most commonly used profiling tool for AIX is called xprofiler, which easily can be used to profile both parallel MPI programs and serial programs. Example of use:

1. Compile and link your program with the -pg -g options.
2. Run the program as normal, on inspection you will discover new files in the directory the program was run, called gmon.out.X. The X only appears for parallel programs and denote the process number. An MPI program running on 4 processes will e.g. generate 4 files, gmon.out.0 to gmon.out.3.
3. Start xprofiler like this:

     	<user@tre> # xprofiler [executable] [gmon.out.0]
   	

   This is an example for profiling an MPI code, it is most common/natural to look at the master thread, which is the number 0.

The user interface that now pops up should be almost self-explanatory. You will get a call-graph of your program, with green boxes for each subroutine/function. The width of the box is a measure of how much time is spent in the routine and all its descendants, the height denotes the "self-time" - so look out for high boxes! On the menu bar you can find options for hiding library calls through filtering. The flat profile gives the list over routines in which most time is spent, sorted on self-time. Now, in the flat profile window; marking the name of a subroutine another menu option then lets you browse its sourcecode. In the left column of the window that now appears the approximate number of ticks spent on each line is estimated. A tick is like a mark/signal emitted from the program approximately every 1/100 second. Note that this method may be very inaccurate - but it will still give you a good idea on where major time is spent. The method is similar to tprof below.

Most UNIX like systems also have the commands prof and gprof available, AIX no exception. prof gives a basic timing profile for a program. To use it, compile your program with the -p option, execute it, then run the command

  prof -g > prof.txt

File prof.txt will now contain info on how much time was spent in each subroutine. The gprof command works similarly, but gives more detailed information. To use it, compile your code with the -pg option, execute it, and run

  gprof > gprof.txt

MICRO-PROFILING: tprof

With tprof it is possible to profile your programs on a line-by-line basis, getting fine grained control on which source lines most of the time is spent. To do this it is necessary to compile your program with the -g option. Make sure the source files for each routine are available in the same directory as the executable. In that directory execute

  tprof -m -p [executable] -x [executable]

Running a program with tprof will force the program to emit «ticks» a kind of signal to tprof, every 1/100 second, making it possible to count these ticks and generate a profile.

A bunch of report files are produced, for the example program, compile and profile like this:

  xlf90 -O4 -g -qsuffix=f=f90 util.f90 perf.f90
  tprof -m -p ./a.out -x ./a.out

The file __t.perf_perf.f90 will contain the source code marked with the number of ticks spent on each source line. The file __h.perf.f90 will contain a list of «hot-spots», e.g. lines with lots of ticks.

-- the rest of this page is still under construction --

HARDWARE COUNTERS: hpmcount, hpmviz

The POWER cpu series has the ability to emit signals when special kinds of events happen in the processor. E.g. when the program asks for data not residing in the fast memory (cache), an event called a «cache miss» occur. If your program performance is worse than expected, profiling using hardware counters may be a good way to find the problem. Perhaps the most common reason for poor performance is accessing your data (memory) in a non-optimal way.

The hpmcount tool generates a report containing some statistics on how «well» your program runs. For the above example, run hpmcount like this:

  hpmcount ./a.out

This program will by default measure the performance of your complete application. If you wish to look only at specific parts of your application, you must use libhpm, a library with routines that makes it possible to define up to 100 sections of your program, for which you want to analyze the performance. Sections may be nested, in threads. It is also possible to define the set of events you want to 'listen to'. A program with a graphical interface called hpmviz lets you browse the results from libhpm. More info:

• HPM README
  
• HPM README with better description of derived metrics

Tracing tools - pct - Jumpshot - Paraver

Tracing is a technique which is suitable for analyzing communication patterns in parallel programs. The user must first instrument the code; which means to insert signal emitting chunks of code into an object file at almost arbitary places. This can be done via the dpcl library, even run-time. After instrumenting, the code is run, and it will now produce so called trace files. These files contain a time-series of events defined by the instrumentation, and the events can be visualized with tailer made tools such as Jumpshot or Paraver. (Vampir is another package for visualization/analyzis of tracefiles that we had available on the Origin.)

On the Regatta we will provide the following tools in the near future:

• Trace file generation/instrumentation:
  • pct
• Visualize trace files/Analysis:
  • Jumpshot
  • pvt (for HPM tracefiles)
  • Paraver

ANALYSIS TOOLS - paradyn

Paradyn (from the 3.2 info) «Paradyn is a tool for measuring and analyzing the performance of sequential, parallel, and distributed programs. Paradyn can monitor large, long running programs and provides facilities for helping to automatically find performance problems in complex parallel programs. Paradyn operates on executable (a.out or .exe) files by dynamically inserting and customizing measurement code while the program is running. Paradyn can measure programs running on Solaris (SPARC & x86), Linux (x86), Windows NT (x86), IRIX (MIPS), and AIX (Power), and heterogeneous combinations of these systems. Paradyn can also handle MPI (native MPI on IRIX and AIX, MPICH 1.2 on Linux and Solaris)»

OTHER TOOLS

• SvPablo

More ACTC tools come here ...