A key feature for code optimisation is performance measurement. TRUST offers different solutions, using TRUST internal features or external libraries.

TRUST counters

In order to obtain statistics on the performance of a test case, TRUST uses counters. Counters are C++ objects.

Their main purpose is to serve as time watches for parts of the code you deem important. They also track additional metrics:

Timing statistics:

Minimum, maximum, average and standard deviation of the measured time per time step

Usage metrics:

An integer called count tracking the number of times the counter has been started and stopped
A custom integer called quantity, mainly used to measure the quantity of bytes exchanged during communication operations, or to store the number of iterations of the linear solver

Identification:

std::string attributes description and family. A description is mandatory for each counter. The family attribute is by default set to "None" but can be used as a key for regrouping the counters you want to parse together after your computation.

Level management: Counters also have a certain level. The level of a counter is represented by an integer. It makes sure that you know when you open your counter. Indeed, if a counter of level 1 is running, you can only start a counter of level 2. Otherwise, the code will stop. In the same logic, you can only close the most recently opened counter. Because of this interlock structure, counters also have another interesting metric called alone_time. It is the elapsed time where the considered counter is the last opened one (i.e., excluding time spent in nested counters). This metric is printed in the CSV output file and helps identify the intrinsic cost of each code section.

Example of the level hierarchy:

statistics().begin_count(STD_COUNTERS::total_execution_time, -1);                   // Level -1 - reserved to the total execution time counter
    statistics().begin_count(STD_COUNTERS::timeloop, 0);                            // Level 0  - reserved to counters of the global steps
        statistics().begin_count(STD_COUNTERS::convection, 1);                      // Level 1  - only possible while Level 0 is running
            statistics().begin_count(STD_COUNTERS::mpi_recv, 2);                    // Level 2  - only possible while Level 1 is running
            statistics().end_count(STD_COUNTERS::mpi_recv, 1 , nb_exchanged_bytes);
        statistics().end_count(STD_COUNTERS::convection);
    statistics().end_count(STD_COUNTERS::timeloop);
statistics().end_count(STD_COUNTERS::total_execution_time);

This ensures proper nesting and helps track where time is spent in your code hierarchy.

Those counter objects are managed by the Perf_counters class. In practice, you will only need to interact with the Perf_counters class (not with individual counter objects directly). The Perf_counters class follows a singleton pattern and a Pimpl pattern, such that the implementation of the class is hidden in the Perf_counters.cpp file. The unique instance of Perf_counters can be called inside the code by using:

statistics()

The unique instance of Perf_counters will be created at the first statistics() call.

The counters managed by the Perf_counters instance are separated in two types:

the standard counters used by default in TRUST and identified by a STD_COUNTERS key (STD_COUNTERS is an enum class),
custom counters that can be created inside the code and that are identified by a std::string.

The basic API for counters in TRUST is as follows:

#include <Perf_counters.h>
 
statistics().begin_count(MY_COUNTER_KEY, level);
{
    // The block of code I want to have statistics about
}
statistics().end_count(MY_COUNTER_KEY, count, quantity);

MY_COUNTER_KEY is either a STD_COUNTERS if you want to open a standard TRUST counter or the std::string that corresponds to the description of the custom counter you try to open. In the statistics().begin_count() function, the level parameter is optional. If omitted, the counter will automatically use the one defined at the creation of the counter. If you are having trouble determining the level of your counter, you can use the function statistics().get_last_opened_counter_level() to know the level of the last opened counter. The count parameter specifies how much to increment the counter's total count (i.e., how many times to record this begin/end cycle). It is set by default to 1. The quantity parameter specifies how much to increment the quantity attribute of your counter and is by default set to 0.

During a TRUST computation, TRUST statistics are measured through 3 main steps:

Computation start-up
Time loop
Post-resolution

At the end of each step, the counters are reset and statistics are printed in the two files:

MY_TRUST_CASE_NAME.TU
MY_TRUST_CASE_NAME_csv.TU

The first file contains aggregated stats that are the most commonly used alongside some information regarding the environment of your computation (date, OS, CPU model, GPU model if you run a GPU computation, number of CPU processors used, ...). It has been designed to be human readable, but it is not easy to parse it informatically. The second file has been created for easily parsing each and every counter's data with your favorite csv parsing tool, for example pandas.

The first time steps can take more time thant the rest, if you want to discard them of your stats, add the following line of code right before the time loop:

statistics().set_nb_time_steps_elapsed(int n) ;

Standard TRUST counters

Here is the list of the standard TRUST counters:

Key	Description	Family	Is_communication	Is_gpu
total_execution_time	Total time	None	False	False
computation_start_up	Computation start-up	None	False	False
timeloop	Time loop	None	False	False
backup_file	Back-up operations	None	False	False
system_solver	Linear solver resolutions Ax=B	None	False	False
petsc_solver	Petsc solver	None	False	False
implicit_diffusion	Solver for implicit diffusion	None	False	False
compute_dt	Computation of the time step dt	None	False	False
turbulent_viscosity	Turbulence model::update	None	False	False
convection	Convection operator	None	False	False
diffusion	Diffusion operator	None	False	False
gradient	Gradient operator	None	False	False
divergence	Divergence operator	None	False	False
rhs	Source terms	None	False	False
postreatment	Post-treatment operations	None	False	False
restart	Read file for restart	None	False	False
matrix_assembly	Nb matrix assembly for implicit scheme:	None	False	False
update_variables	Update ::mettre_a_jour	None	False	False
mpi_sendrecv	MPI_send_recv	MPI_sendrecv	true	False
mpi_send	MPI_send	MPI_sendrecv	true	False
mpi_recv	MPI_recv	MPI_sendrecv	true	False
mpi_bcast	MPI_broadcast	MPI_sendrecv	true	False
mpi_alltoall	MPI_alltoall	MPI_sendrecv	true	False
mpi_allgather	MPI_allgather	MPI_sendrecv	true	False
mpi_gather	MPI_gather	MPI_sendrecv	true	False
mpi_partialsum	MPI_partialsum	MPI_allreduce	true	False
mpi_sumdouble	MPI_sumdouble	MPI_allreduce	true	False
mpi_mindouble	MPI_mindouble	MPI_allreduce	true	False
mpi_maxdouble	MPI_maxdouble	MPI_allreduce	true	False
mpi_sumfloat	MPI_sumfloat	MPI_allreduce	true	False
mpi_minfloat	MPI_minfloat	MPI_allreduce	true	False
mpi_maxfloat	MPI_maxfloat	MPI_allreduce	true	False
mpi_sumint	MPI_sumint	MPI_allreduce	true	False
mpi_minint	MPI_minint	MPI_allreduce	true	False
mpi_maxint	MPI_maxint	MPI_allreduce	true	False
mpi_barrier	MPI_barrier	MPI_allreduce	true	False
gpu_library	GPU_library	GPU_library	false	true
gpu_kernel	GPU_kernel	GPU_kernel	false	true
gpu_copytodevice	GPU_copyToDevice	GPU_copy	false	true
gpu_copyfromdevice	GPU_copyFromDevice	GPU_copy	false	true
gpu_malloc_free	GPU_allocations	GPU_alloc	false	true
interprete_scatter	Scatter_interprete	None	true	false
virtual_swap	DoubleVect/IntVect::virtual_swap	None	true	False
read_scatter	Scatter::read_domaine	None	true	False
parallel_meshing	Parallel meshing	None	False	False
IO_EcrireFicPartageBin	write	IO	False	False
IO_EcrireFicPartageMPIIO	MPI_File_write_all	IO	False	False

You can use them whenever you need to.

Custom TRUST counters

As explained above, on top of standard counters, you can also create and use custom counters. To create a new custom counter, you just need to add the following in your code:

statistics().create_custom_counter(std::string counter_description, int counter_level, std::string counter_family = "None", bool is_comm=false, bool is_gpu=false);

Then, you can open and close your new counter, using std::string counter_description as your new counter key. All of the custom counters will be printed in both TU files.

Note: Warning: if a custom counter already has your std::string counter_description, the function statistics().create_custom_counter() will not create another counter.

External profilers

Some external profilers can be directly used on a TRUST data file. To find the appropriate option to use them, run:

trust -h

Below, we present the three most useful ones.

Perf

First, install the Perf library if you don't have it yet.

Then, you just need to:

trust -perf MY_DATA_FILE.data

Perf is dedicated to CPU profiling. It will enable you to locate the main performance bottlenecks inside your code.

Heaptrack

First, make sure the Heaptrack package is installed on your computer.

Then, you just need to:

trust -heaptrack MY_DATA_FILE.data

Heaptrack is dedicated to monitor memory usage (allocations: their number and size). It will enable you to find excessive allocations and possibly memory leaks.

Note: Perf and Heaptrack work best with Hotspot, a GUI for visualizing profiling data.

Nsight system

Normally, Nsight system is already available in TRUST. You can also install it alone or alongside the Cuda Toolkit.

Then, you just need to:

trust -nsys MY_DATA_FILE.data

Nsight system is dedicated to GPU profiling. It will enable you to locate the main performance bottlenecks inside your code. It detects the code not yet ported on GPU, and helps you visualize the data copy between host and device memory. Nsight system is also useful with CPU-only runs thanks to the rich labeling of the code.