Performance profiling

Since the primary motivator of Celeritas is performance on GPU hardware, profiling is a necessity. Celeritas uses NVTX (CUDA), ROCTX (HIP) or Perfetto (CPU) to annotate the different sections of the code, allowing for fine-grained profiling and improved visualization.

Tracing events in Celeritas

Celeritas includes a number of NVTX, HIP, and Perfetto events that can be used to trace different aspects of the code execution. These events are enabled when the environment variable CELER_ENABLE_PROFILING (see Environment variables) is set. All profiling backends (CUDA, HIP, and Perfetto) support both Timeline and Counter events detailed below, except that HIP does not support Counters.

Profiling backends allow grouping various events into “namespaces” (NVTX/HIP domains, Perfetto categories) so that users can selectively enable events they are interested in. Celeritas groups all events in the “celeritas” namespace.

Slices

Detailed timing of each step iteration is recorded with “slices” events in Celeritas. The step slice contains nested Slices for each action composing the step, some actions such as along-step actions contain more nested slices for fine-grained profiling.

In addition to the slices in the simulation loop, slices events are also recorded when setting up the problem (e.g. detector construction)

Counters

Celeritas provides a few counter events. Currently it writes:

  • active, alive, and dead track counts at each step iteration, and

  • the number of hits in a step.

Profiling a Celeritas example app

A detailed timeline of the Celeritas construction, steps, and kernel launches can be gathered, the example below illustrates how to do it using NVIDIA Nsight systems.

Here is an example invoking the celer-sim app through the NVIDIA utility to generate a timeline:

1$ CELER_ENABLE_PROFILING=1 \
2> nsys profile \
3>   --trace=cuda,nvtx,osrt \
4>   --nvtx-capture="celeritas" \
5>   --osrt-backtrace-stack-size=16384 --backtrace=fp \
6>   -o report.nsys-rep -f true \
7>   celer-sim inp.json

Line 2 specifies the APIs to be captured: in this case, CUDA calls, NVTX ranges, and OS runtime libraries. To use the NVTX ranges, which is strongly recommended for kernel annotations, you must enable the CELER_ENABLE_PROFILING variable in addition to using the NVTX “trace” option (line 3). The capture domain in line 4 restricts profiling to the Celeritas application. Additional frame-pointer-based backtracing is specified in line 5; line 6 writes (and overwrites) to a particular output file; the final line invokes the application.

On AMD hardware using the ROCProfiler, here’s an example that writes out timeline information:

1$ CELER_ENABLE_PROFILING=1 \
2> rocprof \
3> --roctx-trace \
4> --hip-trace \
5> celer-sim inp.json

It will output a results.json file that contains profiling data for both the Celeritas annotations (line 3) and HIP function calls (line 4) in a “trace event format” which can be viewed in the Perfetto data visualization tool.

On CPU, timelines are generated using Perfetto, which is only supported when CUDA and HIP are disabled. Perfetto supports application-level and system-level profiling. To use the application-level profiling, see Diagnostics.

1$ CELER_ENABLE_PROFILING=1 \
2> celer-sim inp.json

The system-level profiling, capturing both system and application events, requires starting external services. Details on how to setup the system services can be found in the Perfetto documentation. Root access on the system is required.

Integration with user applications

When using a CUDA or HIP backend, no additional code is needed in the user application. The commands shown in the previous sections can be used to profile your application. If your application already uses NVTX, or ROCTX, you can exclude Celeritas events by excluding the celeritas domain.

When using Perfetto for CPU profiling, you need to create a :cpp:class:celeritas::TracingSession: instance. The profiling session needs to be explicitly started, and will end when the object goes out of scope, but it can be moved to extend its lifetime.

1#include "corecel/sys/TracingSession.hh"
2
3int main()
4{
5   // Pass with empty/no filename for system profiling
6   celeritas::TracingSession session("out.perfetto");
7}

As mentioned above, Perfetto can either profile application events only, or application and system events. The system-level profiling requires starting external services. Details on how to setup the system services can be found in the Perfetto documentation. Root access on the system is required.

When the tracing session is started with a filename, the application-level profiling is used and written to the specified file. Omitting the filename will use the system-level profiling, in which case you must have the external Perfetto tracing processes started. The container in scripts/docker/interactive provides an example Perfetto configuration for tracing both system-level and Celeritas events.

As with NVTX and ROCTX, if your application already uses Perfetto, you can exclude Celeritas events by excluding the celeritas category.

Kernel profiling

Detailed kernel diagnostics including occupancy and memory bandwidth can be gathered with the NVIDIA Nsight Compute profiler.

The Source tab is especially useful, as it maps diagnostics such as number of instructions executed and number of stalls to individual lines in the source code. To obtain this mapping, Celeritas must be compiled with -lineinfo in CMAKE_CUDA_FLAGS. The following ncu command can be run to obtain an .ncu-rep for use with NVIDIA Nsight Compute.

1 $ ncu \
2 > --set full \
3 > --kernel-name launch_action_impl \
4 > --launch-skip 254 --launch-count 1 \
5 > -f -o output \
6 > celer-sim inp.json

The metric set must be full for detailed source-level profiling information. Note that full metrics require 30+ passes and thus are very time-consuming; use basic to rapidly obtain coarse-grained profiling information. This command will force-overwrite an output file named output.ncu-rep. In this example, only a single kernel is profiled: the 255th launch of the launch_action_impl. Selecting how many kernels must be skipped before reaching the kernel of interest can be done in NVIDIA Nsight Systems prior to running NVIDIA Nsight Compute. This can be done in two ways. The first method is:

  1. Click CUDA HW then [All Streams] then Kernels

  2. Right-click launch_action_impl and click Show in Events View

  3. Select any kernel in the event view

  4. Obtain the launch number of the kernel in the # column

  5. Use # minus 1 as the launch-skip

This method can be used to profile the most expensive kernel by sorting the event view by duration and finding the # of the top kernel. The second method is:

  1. Right-click the kernel of interest in the timeline

  2. Click Analyze the selected kernel with NVIDIA Nsight Compute

  3. Click Display the command line to use NVIDIA Nsight Compute CLI

  4. Copy the launch-skip from the supplied ncu command.

A final option is to profile a kernel using NVTX annotations:

1$ CELER_ENABLE_PROFILING=1 \
2> ncu \
3> --set full \
4> --nvtx --nvtx-include "celeritas@{RANGE}" \
5> --launch-skip 100 --launch-count 1 \
6> -f -o output \
7> celer-sim inp.json

Here, celeritas@RANGE is the name of the NVTX domain and slash-separated range, (e.g. run/step/*/propagate). Note that the domain and range are flipped compared to nsys since the kernel profiling allows detailed top-down stack specification.