Stepping mechanics¶

The core algorithm in Celeritas is to perform a loop interchange [AK84] between particle tracks and steps. The classical (serial) way of simulating an event is to have an outer loop over tracks and an inner loop over steps, and inside each step are the various actions applied to a track such as evaluating cross sections, calculating the distance to the nearest geometry boundary, and undergoing an interaction to produce secondaries. In Python pseudocode this looks like:

track_queue = primaries
while track_queue:
   track = track_queue.pop()
   while track.alive:
      for apply_action in [pre, along, post]:
         apply_action(track)
      track_queue += track.secondaries

There is effectively a data dependency between the track at step i and step i + 1 that prevents vectorization. The approach Celeritas takes to “vectorize” the stepping loop on GPU is to have an outer loop over “step iterations” and an inner loop over “track slots”, which are elements in a fixed-size vector of tracks that may be in flight:

initializers = primaries
track_slots = [None] * num_track_slots
while initializers or any(track_slots):
   fill_track_slots(track_slots, initializers)
   for apply_action in [pre, along, post]:
      for (i, track) in enumerate(track_slots):
         apply_action(track)
         track_queue += track.secondaries
   if not track.alive:
      track_slots[i] = None

The stepping loop in Celeritas is therefore a sorted loop over “step actions”, each of which is usually a kernel launch (or an inner loop over tracks if running on CPU).

Actions¶

Actions can operate on shared parameters and thread-local state collections. All actions inherit from a celeritas::ActionInterface abstract base class, and the hierarchy of actions allows multiple inheritance so that a single “action” class can, for example, allocate states at the beginning of the run and execute once per step.

There are currently two different actions that act as extension points to the stepping loop: BeginRunActionInterface is called once per event (or set of simultaneously initialized events), and StepActionInterface is called once per step, ordered using celeritas::StepActionOrder.

class ActionInterface¶

Pure abstract interface for an action that could happen to a track.

An action represents a possible state point or state change for a track. Explicit actions (see StepActionInterface ) call kernels that change the state (discrete processes, geometry boundary), and implicit actions (which do not inherit from the explicit interface) are placeholders for different reasons to pause the state or mark it for future modification (range limitation, propagation loop limit).

The ActionInterface provides a clean virtual interface for gathering metadata. The StepActionInterface provides additional interfaces for launching kernels. The BeginRunActionInterface allows actions to modify the state (or the class instance itself) at the beginning of a stepping loop, and EndRunActionInterface allows actions to gather and merge multiple state information at the end.

Using multiple inheritance, you can create an action that inherits from multiple of these classes. Note also that the function signatures are similar to other high-level interfaces classes in Celeritas (e.g., AuxParamsInterface, OutputInterface), so one “label” can be used to satisfy multiple interfaces.

The label should be a brief lowercase hyphen-separated string, usually a noun, with perhaps some sort of category being the first token.

The description should be a verb phrase (and not have a title-cased start).

Subclassed by celeritas::ConcreteAction, celeritas::MutableActionInterface, celeritas::StaticConcreteAction, celeritas::StepActionInterface< P, S >

template<class P, template<MemSpace M> class S> class BeginRunActionInterface : public celeritas::ActionTypeTraits<P, S>, public celeritas::MutableActionInterface¶

Interface for updating states at the beginning of the simulation.

This is necessary for some classes that require deferred initialization (either to the class itself or the state), for example because it needs the number of total actions being run.

If the class itself—rather than the state—needs initialization, try to initialize in the constructor and avoid using this interface if possible.

Todo:: This is currently called once per each state on each CPU thread, and it would be more sensible to call a single with all cooperative states.

Warning

Because this is called once per thread, the inheriting class is responsible for thread safety (e.g. adding mutexes).

Subclassed by celeritas::ActionDiagnostic, celeritas::ExtendFromSecondariesAction, celeritas::SortTracksAction, celeritas::StatusChecker

template<class P, template<MemSpace M> class S> class StepActionInterface : public celeritas::ActionTypeTraits<P, S>, public virtual celeritas::ActionInterface ¶

Interface for kernel actions in a stepping loop.

Template Parameters:

P – Core param class
S – Core state class

Subclassed by celeritas::ActionDiagnostic, celeritas::AlongStepGeneralLinearAction, celeritas::AlongStepNeutralAction, celeritas::AlongStepRZMapFieldMscAction, celeritas::AlongStepUniformMscAction, celeritas::ExplicitCoreActionInterface, celeritas::ExtendFromPrimariesAction, celeritas::ExtendFromSecondariesAction, celeritas::InitializeTracksAction, celeritas::Model, celeritas::SlotDiagnostic, celeritas::SortTracksAction, celeritas::StepDiagnostic

enum class celeritas::StepActionOrder

Within-step ordering of explicit actions.

Each “step iteration”, wherein many tracks undergo a single step in parallel, consists of an ordered series of actions. An action with an earlier order always precedes an action with a later order.

Initialization and execution¶

The front end constructs the Problem definition classes and allows user actions and Auxiliary user data to be set up
“Core params”, which reference these classes, are constructed; in the process, certain required implementation actions (e.g., managing primaries and secondaries, initializing tracks, crossing boundaries) are added to the action
Additional user actions and data can be added
The “core state” is created on each CPU thread (or task), simultaneously constructing a vector of auxiliary state data
The celeritas::Stepper constructs a final ordered runtime vector of actions
The Stepper immediately calls the “begin run” actions
Each step calls all the “step” actions

enum class celeritas::TrackStatus : std::uint_least8_t

Whether a track slot is alive, inactive, or dying inside a step iteration.

Each track slot has a state marking its transition between death and life.

A track slot starts as inactive . If not filled with a new track, it is inactive for the rest of the step iteration.
When it is populated with a new particle, it is initializing . If an error occurs during initialization it is errored .
During the pre-step setup, a non-errored active track slot is marked as alive .
During along-step or post-step a track can be marked as errored or killed .

Values:

enumerator inactive: No tracking in this thread slot.

enumerator initializing: Before pre-step, after initialization.

enumerator alive: Track is active and alive.

enumerator begin_dying_

enumerator errored: Track failed during this step.

enumerator killed: Killed physically inside the step.

enumerator size_

template<MemSpace M> class Stepper : public celeritas::StepperInterface¶

Manage a state vector and execute a single step on all of them.

Stepper<MemSpace::host> step(input);

// Transport primaries for the initial step
StepperResult alive_tracks = step(my_primaries);
while (alive_tracks)
{
    // Transport secondaries
    alive_tracks = step();
}

Note

This is likely to be removed and refactored since we’re changing how primaries are created and how multithread state ownership is managed.

Track sort order¶

For performance reasons such as reducing divergence and improving memory access patterns, it is desirable to map similar tracks into similar threads. There will be an upcoming paper describing and analyzing these options in more detail.

enum class celeritas::TrackOrder

Change the ordering or thread mapping of track slots.

There are three categories of track sorting:

No sorting is performed and new tracks are inserted in the nearest empty track slot. (none )
The location of new track slots is biased during track initialization: charged and neutral tracks are inserted at opposite sides of the track slot vacancies. (init_charge )
Tracks are reindexed one or more times per step so that the layout in memory is unchanged but an additional indirection maps threads onto different track slots based on particle attributes (reindex_status, reindex_particle_type ), actions (reindex_along_step_action, reindex_step_limit_action, reindex_both_action ).
As a control to measure the cost of indirection, the track slots can be reindexed randomly at the beginning of execution (reindex_shuffle ).

Values:

enumerator none: Don’t do any sorting: tracks are in an arbitrary order.

enumerator begin_layout_

enumerator init_charge: Partition data layout of new tracks by charged vs neutral.

enumerator end_layout_

enumerator begin_reindex_: Shuffle at the start of the simulation.

enumerator reindex_shuffle

enumerator reindex_status: Partition by active/inactive status.

enumerator reindex_particle_type: Sort by particle type.

enumerator begin_reindex_action_

enumerator reindex_along_step_action: Sort only by the along-step action id.

enumerator reindex_step_limit_action: Sort only by the step limit action id.

enumerator reindex_both_action: Sort by along-step id, then post-step ID.

enumerator end_reindex_action_

enumerator end_reindex_

enumerator size_