What PEAT lets you do

PEAT answers three questions about an MPI, OpenMP or GPU application: what is it really doing, where does the time go, and how would it run on a different system. Everything starts from one low-overhead trace of an unmodified binary. The programs behind these capabilities are described on the Components page.

See what happens, rank by rank

Load a trace in the viewer and look at the application the way it actually ran, not the way the source suggests it did.

• A timeline per rank and thread showing MPI calls, collectives, OpenMP regions and user-defined regions.
• Sends are matched with their receives across ranks, and node clocks are synchronised with PTP, so ordering between nodes is meaningful.
• Zoom into any interval to list the functions (with calling address) and messages inside it.
• Too many ranks to show or out of the memory? Load only the fastest and slowest ranks or nodes, or any set you pick, and cut off by function or message type.

Tools: tractor mat qTV

Find where the time is lost

• Per-function statistics: occurrences, durations, bytes transferred and message-size distribution.
• Call-level reports with per-call duration and count per rank, limited to the ranks and functions you choose and sorted by time consumed.
• Built-in analyses for the point-to-point and collective operations with the lowest bandwidth and the most imbalance.
• Call-site attribution: return-address stacks recorded to a depth you choose, so a slow call can be tied back to the code that issued it.
• CPU hardware counters collected alongside the MPI time, where the OS supports it.

Tools: tractor mat ctt qTV

Look below the MPI call

A slow MPI_Allreduce or MPI_Wait is a symptom. These layers show the cause.

• Inside collectives: see the point-to-point transfers a collective performs (patched MPI).
• OpenMP: OpenMP events (via OMPT) in the same trace as MPI; omp.tool* additionally gives stand-alone per-region statistics for an OpenMP-only application, with no tractor involved (Intel oneAPI runtime).
• GPU: Level Zero and OpenCL calls on the same timeline as MPI.
• Transport API: the calls MPI makes into OFI (libfabric), PSM2, PSM3, UCX and verbs, with timings, in their own lane*.
• Fortran: Fortran MPI calls are recorded exactly once, alongside C (with the Fortran patch or a custom libmpifort).

Tools: tractor MPI patches omp.tool* (independent of tractor)

Keep tracing cheap and traces small

• No rebuild of the MPI program: the collector is injected with LD_PRELOAD. Build with frame pointers only if you want deeper call stacks.
• Trace only what matters: gate tracing with MPI_Pcontrol, include or exclude individual MPI functions, and choose which functions get a full call stack.
• Bound memory and size: dump events to disk as they occur, write compressed trace files, or keep statistics only and skip the trace files altogether.
• Pick the right depth of tracing: plain MPI works with unmodified MPI; collective-split and transport-API tracing need a patched MPI.

See Keeping traces small for a step-by-step recipe.

Tools: tractor

Estimate behaviour on another system*

Instead of re-running, take a trace from the system you have and ask what it would do on one you do not — a different network, a different transport, or hardware that does not exist yet.

• Re-time a captured trace (matched by mat) with a different transport / network profile (InfiniBand, Omni-Path, shared memory, …) and get an estimated trace, statistics and wall time.
• Calibrate before estimating: first reproduce the measured run on the original system, then change the profile to the target.
• Optionally identify user code regions with urpu (which works on the tractor trace directly) and use them to get estimates for the sections of the code you care about, not just the whole run.

Intended for pre-silicon studies of future systems. It is not a replacement for measurement on real hardware. pet is now only kept in a compilable state and is not under active development.

Tools: tractor mat pet* urpu*

Where it runs

• MPI: MPICH 4/5, MVAPICH2/4, Intel MPI and Open MPI. Unmodified MPI gives user-level tracing; patched MPICH and MVAPICH add collective-split and transport-API tracing*.
• Accelerators and threads: OpenMP (OMPT), Level Zero and OpenCL on every supported MPI.
• Operating systems: Linux for everything; Windows for mat, ctt, qTV, pet* and urpu* (viewer and post-processing, not collection).
• Toolchains: GNU or Intel; Qt for the viewer.

Tracing capability by MPI implementation

(o) = used out-of-the-box, unpatched. (p) = built against the patches/ tracing hooks (see MPI patches) — required for collective-split and transport-API tracing.