Components

The individual programs behind PEAT’s capabilities. You normally only need the tools named under each capability; this page is the detail. For how they connect, see how the pieces fit together.

How the pieces fit together

Every analysis starts from a tractor trace. Post-processing branches into a visual path, a text-report path and an estimation path.

Collect → analyse → visualise

Text report

Estimation*

The dashed boxes are inputs to pet: transport and MPI-local profiles (which you can generate with Tune or define yourself), plus, optionally, user-code and region information from urpu, which reads the tractor trace directly.

tractor — Trace Collector Runtime

A low-intrusion, LD_PRELOAD-based shared library that intercepts MPI calls (and optionally OpenMP/OMPT and GPU calls via Level Zero / OpenCL) and records them into trace files consumed by every other component. Drives an optional PTP Daemon (PTPD) to synchronise clocks across nodes for cross-rank timing correction.

• Point-to-point, collective, and GPU activity capture
• Communicator and rank resolution
• Stack unwinding with configurable return-address depth
• Hardware counter sampling (PAPI) and memory allocation profiling*
• Node/CPU topology capture via hwloc
• PControl-aware selective tracing, include/exclude event filters
• Transport-API profiler* — records calls and timings of the transport libraries MPI uses underneath (OFI, PSM2, PSM3, UCX, verbs) into *.transport trace files

“Transport API” here means the user-space communication libraries and interfaces that MPI calls to move data — not the physical fabric or the wire protocol. In patched MVAPICH2 it also covers the shared-memory (SMP/CMA) channel. The corresponding names are *.transport trace files, libtractor_transport_<api>.so libraries and the PEAT_TRACTOR_COLLECT_TRANSPORT environment variable.

Choosing a tractor library

Tractor is preloaded into every rank. All builds live in the lib directory (one sub-directory per compiler) and follow the pattern libtractor[_dbg].<mpi>[.<extra>].so:

Fortran: for out-of-the-box MPICH-based MPI, trace Fortran calls with a custom libmpifort whose wrappers call MPI_* instead of PMPI_*. Patched MPICH-family builds (via the .fortran.patch and its redirect environment variable). See MPI patches.

Trace files

Tractor writes plain-text, colon-separated, gzip-compressed files named <mask>.<type>[.<rank>].gz. The default mask is tractor_t (PEAT_TRACTOR_FILE).

The record layout is described in mpi.tool/components/tractor/doc/Tractor.Trace.Format.Description.pdf in the source tree.

mat — Matching Tool

Matches send/receive and collective calls across ranks from the raw tractor traces and builds per-function timing and message-size statistics (occurrences, durations, transferred bytes). Ships as a serial binary (mat) for traces that fit in one machine's RAM, or as an MPI-parallelised binary (pmat.impi / pmat.mvapich2 / pmat.ompi) for very large traces.

ctt — Calls Tracing Tool

Post-processes tractor traces into call-level reports: classifies MPI calls, computes per-call durations and occurrences per rank, and can select representative ranks (fastest / slowest) for focused analysis.

qTV — Qt Trace Viewer

A Qt-based GUI to visually inspect tractor traces: a timeline view of MPI/collective/user-region events per rank and thread, message legends, cut-off filtering, and a raw-record text viewer. Available on Linux and Windows. more

pet — Performance Estimation Tool*

Consumes tractor traces — with the send/receive and collective matching computed by mat — together with network/hardware profiles to model and estimate MPI application performance — per-rank/thread timelines, communicator, GPU, OpenMP and transport-API events — helping estimate behaviour under different transport/network configurations. Maintained in a compilable state; not under active development.

urpu — User Regions Processing Utility*

Identifies user code and user regions directly from the tractor trace (it does not need mat) and aggregates statistics for them. The result may be used by pet. Same serial / MPI-parallelised build split as mat.

Customised MPI (patches)

Public MPI/PMPI interfaces are not enough to see inside collective algorithms or the transport layer, so PEAT ships source patches for several open-source MPI implementations under mpi.tool/patches/. Apply with patch -p1 from the top of the MPI source tree; the patch files within one implementation are independent and can be applied in any order.

Currently provided: MPICH 3.4.3*, 4.3.2, 5.0.1 and 5.0.2; MVAPICH2 2.3.5 and 2.3.7-1*; MVAPICH 4.1; and Open MPI 5.0.10*.

omp.tool* — OpenMP profiling

OpenMP profiling tool built on the OMPT callback interface, producing per-region OpenMP statistics. Originally based on a customised Intel OpenMP 3.0 runtime (2012–13, pre-OMPT); rewritten to use standard OMPT callbacks.

• ompcalls.cpp — OMPT callback registration and handlers
• omp_stat.cpp / .h — per-region statistic collection and reporting
• offconv.c — offload/region name converter utility
• omp.stat.pl — post-processing / summary script for collected statistics