1. Build

Each component builds independently via GNU makefiles or CMake* (Linux, GNU or Intel toolchains) or an MSVC solution (Windows, for mat/ctt/qTV/ pet*/urpu*). There're also examples shell scripts which build all confiurations at once. You can use them as reference for building the components individually. From mpi.tool/components/:

cd mpi.tool/components
./BuildAll.sh

This builds tractor (the trace collector runtime), mat, ctt, qTV, and pet* and urpu* against the MPI implementations available on your system ( MPICH / MVAPICH / Intel MPI / Open MPI).

For omp.tool*, build with the Intel oneAPI compiler and runtime. GNU (gcc) does not support OMPT yet, so a gcc build compiles but cannot collect anything:

cd omp.tool/src
make

2. Collect a trace with tractor

Set the toolkit root and preload the collector runtime before launching your MPI job:

export PEAT_ROOT=/path/to/peat
export LD_PRELOAD=$PEAT_ROOT/mpi.tool/lib/{intel|gnu}/libtractor.impi.so

mpiexec.hydra -launcher ssh -genvall -genv LD_PRELOAD $LD_PRELOAD \
    -n <ranks> ./your_mpi_app

-genv is the Hydra (MPICH-based) launcher syntax. With Open MPI, pass the variable with -x LD_PRELOAD instead. Pick the library that matches your MPI as described in Capabilities (libtractor[_dbg].<mpi>[.<extra>].so).

For Fortran programs on out-of-the-box MPICH-based MPI, put a libmpifort whose wrappers call MPI_* (not PMPI_*) on LD_LIBRARY_PATH. Patched MPI builds do not need this.

Useful environment variables (all optional unless noted):

3. Analyse the trace

Run mat -h (or ctt -h) for the short help. The defaults are usually fine.

mat

mpi.tool/bin/mat -i tractor_t -o out > mat.log

• -i is the trace file mask (default tractor_t); -w the input directory; -o the output directory (default ./out).
• The processed trace (.mpi, .hwc, .bind) and the .trace.info file that qTV loads are written to the output directory.
• Additional statistics go to stdout — redirect them to a file.
• -m (--matching-only) writes only the .bind files, which is faster when you do not need to rewrite .mpi/.hwc.

If the trace is too big for one machine, use the MPI-parallel build; use at most as many ranks as the traced application had:

mpiexec -n <ranks> mpi.tool/bin/pmat.impi -i tractor_t -o out

ctt

Call-level classification and timing, written to stdout. Restrict the set of ranks and functions and cap the list length (sorted by time consumed):

mpi.tool/bin/ctt -i tractor_t -r 0,1,2 -f MPI_Send,MPI_Recv -m 20 > ctt.txt

• -r rank set, -f function set (both default to all), -m maximum output entries per function.

4. Visualise with qTV

Start qTV, use Load and select the <mask>.trace.info file that mat wrote to its output directory:

mpi.tool/bin/qTV

In the load dialog you can:

• tick Fastest and slowest to load only the N fastest and N slowest ranks or nodes (N is a spin box, default 2), or tick individual ranks/nodes manually;
• choose which functions and message types to ignore while the trace is being read;
• use the bottom sliders to load only a percentage range of the trace;
• set the number of rendering threads — useful for traces with a lot of events.

Navigating

Toolbar

The menus mostly duplicate toolbar actions.

Analyse menu

Point to point and Collectives each offer Least BW (lowest achieved bandwidth) and Most Imbalance; the matching operations are highlighted on the timeline. Region windows list the functions (with calling address) and the messages inside the selected interval.

See qTV UI at a glance for screenshots of the window, dialogs and menus.

Keeping traces small

Work down this list until the trace is a manageable size:

1. Memory exhausted during collection? Reduce the number of steps or the processes per node (PPN).
2. Trace too big? Reduce the number of time steps; do not reduce the node or core count, because that changes the communication pattern.
3. Still too big? Trace only the main loop with MPI_Pcontrol (below).
4. Needed functions fall outside the Pcontrol region? List them in PEAT_TRACTOR_PCONTROL_FUNCS.
5. Nothing helped? Dump events on the fly with PEAT_TRACTOR_USE_REALTIME_DUMP=1.

Typical Pcontrol layout: put the preparation step (MPI_Comm_*, MPI_Cart_*, MPI_Send_init, MPI_Recv_init, …) outside the traced region and add those function IDs to PEAT_TRACTOR_PCONTROL_FUNCS, so that communicators and persistent requests are still known:

/* preparation: communicators, persistent requests (functions listed in PCONTROL_FUNCS) */
MPI_Cart_create(...);
MPI_Send_init(...);  MPI_Recv_init(...);

MPI_Pcontrol(1);                 /* start tracing */
for (step = 0; step < nsteps; step++) {
    /* main loop: collectives and point-to-point */
}
MPI_Pcontrol(-1);                /* stop tracing */

/* finalisation */

export PEAT_TRACTOR_PCONTROL=1
export PEAT_TRACTOR_PCONTROL_FUNCS=<ids of MPI_Cart_create,MPI_Send_init,...>

Tractor adds up the MPI_Pcontrol arguments and traces while the running total is positive, so stop a region with -1 (or the negative of what you started with), not 0. Use PEAT_TRACTOR_PC_LEVELS to react only to particular levels. MPI_Init, MPI_Init_thread, MPI_Finalize and MPI_Pcontrol are always recorded.

5. Estimate with pet*

pet now uses mat as its matching engine, so run mat over the trace first (the matching-only mode, -m, is enough), then model performance under a different network/hardware profile:

mpi.tool/bin/mat -i tractor_t -m
pet --input=tractor_t --transport-file=<path_to_profile_file>

urpu is optional: it identifies user code and regions directly from the tractor trace (no mat run needed) and its result may be used with pet.

See pet --help for the full option list (transport selection, CPU frequency/PPN targets, eager thresholds, etc.).

6. Profile OpenMP regions with omp.tool*

The tool is an OMPT tool library (libompstat.so, built into omp.tool/lib/<compiler>/) and needs an OMPT-capable OpenMP runtime. There are two ways to load it into the application:

A. Standard OpenMP 5 tool interface (recommended)

The runtime finds the tool’s ompt_start_tool entry point itself, so no LD_PRELOAD is needed and the tool is not injected into unrelated child processes:

export OMP_TOOL=enabled                                  # default in OpenMP 5, shown for clarity
export OMP_TOOL_LIBRARIES=omp.tool/lib/intel/libompstat.so   # colon-separated list of tool libraries
./your_openmp_app

B. LD_PRELOAD

Alternatively preload the library; the runtime then picks up ompt_start_tool from the global symbol scope:

LD_PRELOAD=omp.tool/lib/intel/libompstat.so ./your_openmp_app

Either way, per-thread statistic files (*omp_stat*) are written to the current directory. The tool measures the CPU frequency at start-up (about a one-second delay); set OSL_FREQUENCY=<Hz> (e.g. 2000000000) to skip that measurement. Summarise the result with:

omp.tool/bin/omp.stat.pl <raw_stat_output>

Recommended workflow

Validate each stage against the previous one before moving on. If a stage changes behaviour or timing noticeably, fix that before continuing.

1. Make a pure run. Run the unmodified application to get a reference time.
2. Prepare the build. Compile the application and every library you want call stacks from with -fno-omit-frame-pointer if you will unwind more than level 0.
3. Make a “clean” run with the appropriate MPI (patched MPI if you need collective-split or transport-API tracing*). Compare with the pure run.
4. Make a trace run with the matching tractor preloaded. Compare the timing, and look at the size of the trace; if it is too big, reduce it (see keeping traces small).
5. Post-process. mat then qTV for a timeline, or ctt (which reads the tractor trace directly) for a text report. mat is also required before pet*.
6. Estimate the original system*. Pick the pet configuration/transport profile of the system you traced on and run pet. The estimate should reproduce the measured run; if not, adjust before trusting any estimation.
7. Estimate the target system*. Select the configuration files and parameters of the target system, run pet and collect the results.