Acceleritas#

The accel directory contains components exclusive to coupling Celeritas with Geant4 for user-oriented integration. A simple interface for multithreaded or serial applications is demonstrated in Minimal Geant4 integration, and the more advanced implementation can be inspected in the Integrated Geant4 application (celer-g4) app.

High-level interface#

The SimpleOffload class is an extremely easy-to-use interface for offloading tracks to Celeritas in a multithreaded or serial application. The class names correspond to user actions and ActionInitialization. It requires a few app-owned pieces such as SharedParams and LocalTransporter to be owned by the calling application; the options described below must also be set up and provided.

class SimpleOffload#

Compressed interface for running Celeritas in a multithread Geant4 app.

This class must be a thread-local instance with references to data that exceed the lifetime of the class: e.g. SharedParams can be a global variable, and LocalTransporter can be a global variable with thread_local storage duration.

The CELER_DISABLE environment variable, if set and non-empty, will disable offloading so that Celeritas will not be built nor kill tracks.

The method names correspond to methods in Geant4 User Actions and must be called from all threads, both worker and master.

Public Functions

SimpleOffload() = default#: Construct with celeritas disabled.

SimpleOffload(SetupOptions const *setup, SharedParams *params, LocalTransporter *local)#

Construct from a reference to shared params and local data.

On construction, this will check for the CELER_DISABLE variable and disable offloading if set. Otherwise it will initialize the multithread logging if the run manager is initialized.

inline void Build(SetupOptions const *setup, SharedParams *params, LocalTransporter *local)#: Lazy initialization of this class on a worker thread.

inline void BuildForMaster(SetupOptions const *setup, SharedParams *params)#: Lazy initialization of this class on the master thread.

void BeginOfRunAction(G4Run const *run)#: Initialize celeritas data from setup options.

void BeginOfEventAction(G4Event const *event)#: Send Celeritas the event ID and reseed the Celeritas RNG.

void PreUserTrackingAction(G4Track *track)#: Send tracks to Celeritas if applicable and “StopAndKill” if so.

void EndOfEventAction(G4Event const *event)#: Flush offloaded tracks from Celeritas.

void EndOfRunAction(G4Run const *run)#: Finalize Celeritas.

inline explicit operator bool() const#: Whether offloading is enabled.

The SetupOptionsMessenger can be instantiated with a reference to a global SetupOptions instance in order to provide a Geant4 “UI” macro interface to an app’s Celeritas options.

class SetupOptionsMessenger : public G4UImessenger#

Expose setup options through the Geant4 “macro” UI interface.

The following options are exposed in the /celer/ command “directory”:

Command	Description
geometryFile	Override detector geometry with a custom GDML
outputFile	Filename for JSON diagnostic output
physicsOutputFile	Filename for ROOT dump of physics data
offloadOutputFile	Filename for HepMC3 copy of offloaded tracks as events
maxNumTracks	Number of tracks to be transported simultaneously
maxNumEvents	Maximum number of events in use
maxNumSteps	Limit on number of step iterations before aborting
maxInitializers	Maximum number of track initializers
secondaryStackFactor	At least the average number of secondaries per track

The following option is exposed in the /celer/detector/ command “directory”:

Command	Description
enabled	Call back to Geant4 sensitive detectors

If a CUDA/HIP device is available, additional options are available under /celer/cuda/ :

Command	Description
stackSize	Set the CUDA per-thread stack size for VecGeom
heapSize	Set the CUDA per-thread heap size for VecGeom
sync	Sync the GPU at every kernel for timing
defaultStream	Launch all kernels on the default stream

Warning

The given SetupOptions should be global or otherwise must exceed the scope of this UI messenger.

Celeritas setup#

The setup options help translate the Geant4 physics and problem setup to Celeritas. They are also necessary to set up the GPU offloading characteristics. Future versions of Celeritas will automate more of these settings.

struct SetupOptions#

Control options for initializing Celeritas.

The interface for the “along-step factory” (input parameters and output) is described in AlongStepFactoryInterface .

I/O

GDML filename (optional: defaults to exporting existing Geant4)

std::string geometry_file#: Filename for JSON diagnostic output.

std::string output_file#: Filename for JSON diagnostic output.

std::string physics_output_file#: Filename for ROOT dump of physics data.

std::string offload_output_file#: Filename to dump a HepMC3 copy of offloaded tracks as events.

Celeritas stepper options

Number of track “slots” to be transported simultaneously

size_type max_num_tracks = {}#: Maximum number of events in use.

size_type max_num_events = {}#: Maximum number of events in use.

size_type max_steps = no_max_steps()#: Limit on number of step iterations before aborting.

size_type initializer_capacity = {}#: Maximum number of track initializers (primaries+secondaries)

real_type secondary_stack_factor = {3.0}#: At least the average number of secondaries per track slot.

CUDA options

size_type cuda_stack_size = {}#: Sync the GPU at every kernel for timing.

size_type cuda_heap_size = {}#: Sync the GPU at every kernel for timing.

bool sync = {false}#: Sync the GPU at every kernel for timing.

bool default_stream = {false}#: Launch all kernels on the default stream.

Public Members

IntAccessor get_num_streams#: Set the number of streams (defaults to run manager # threads)

Public Static Functions

static inline constexpr size_type no_max_steps()#: Don’t limit the number of steps.

struct SDSetupOptions#

Control options for initializing Celeritas SD callbacks.

These affect only the HitManager construction that is responsible for reconstructing CPU hits and sending directly to the Geant4 detectors.

Various attributes on the step, track, and pre/post step points may be available depending on the selected options.

Disabling track will leave G4Step::GetTrack as nullptr
Enabling locate_touchable will also set Material and MaterialCutsCouple
Enabling track will set particle the Charge attribute on the pre-step
Requested post-step data including GlobalTime, Position, KineticEnergy, and MomentumDirection will be copied to the Track when the combination of options is enabled
Track and Parent IDs will never be a valid value since Celeritas track counters are independent from Geant4 track counters.

Public Functions

inline explicit operator bool() const#: True if SD is enabled.

Public Members

bool enabled = {false}#: Call back to Geant4 sensitive detectors.

bool ignore_zero_deposition = {true}#: Skip steps that do not deposit energy locally.

bool energy_deposition = {true}#: Save energy deposition.

bool locate_touchable = {false}#: Set TouchableHandle for PreStepPoint.

bool track = {false}#: Create a track with the dynamic particle type and post-step data.

StepPoint pre#: Options for saving and converting beginning-of-step data.

StepPoint post#: Options for saving and converting end-of-step data.

std::unordered_set<G4LogicalVolume const*> force_volumes#: Manually list LVs that don’t have an SD on the master thread.

std::unordered_set<G4LogicalVolume const*> skip_volumes#: List LVs that should not have automatic hit mapping.

struct StepPoint#

Public Members

bool direction = {false}#: AKA momentum direction.

class UniformAlongStepFactory : public celeritas::AlongStepFactoryInterface #

Create an along-step method for a uniform (or zero) field.

The constructor is a lazily evaluated function that must return the field definition and driver configuration. If unspecified, the field is zero.

Public Functions

UniformAlongStepFactory() = default#: Construct with no field (linear propagation)

explicit UniformAlongStepFactory(FieldFunction f)#

Construct with a function to return the field strength.

The function is evaluated whenever Celeritas is set up (which is after Geant4 physics is initialized).

virtual result_type operator()(argument_type input) const final#

Emit an along-step action.

The action will embed the linear propagator if the magnetic field strength is zero (or the accessor is unset).

class RZMapFieldAlongStepFactory : public celeritas::AlongStepFactoryInterface #

Create an along-step method for a two-dimensional (r-z in the cylindical coordinate system) map field (RZMapField).

Public Functions

explicit RZMapFieldAlongStepFactory(RZMapFieldFunction f)#

Emit an along-step action with a non-uniform magnetic field.

The action will embed the field propagator with a RZMapField.

Detailed interface#

These classes are usually integrated into UserActions. The SimpleOffload interface above hides the complexity of these classes, or for more complex applications you can choose to use these classes directly instead of it.

class SharedParams#

Shared (one instance for all threads) Celeritas problem data.

The CeleritasDisabled accessor queries the CELER_DISABLE environment variable as a global option for disabling Celeritas offloading. This is implemented by SimpleOffload

This should be instantiated on the master thread during problem setup, preferably as a shared pointer. The shared pointer should be passed to a thread-local LocalTransporter instance. At the beginning of the run, after Geant4 has initialized physics data, the Initialize method must be called first on the “master” thread to populate the Celeritas data structures (geometry, physics). InitializeWorker must subsequently be invoked on all worker threads to set up thread-local data (specifically, CUDA device initialization).

Some low-level objects, such as the output diagnostics and Geant4 geometry wrapper, can be created independently of Celeritas being enabled.

Internal use only

using SPHitManager = std::shared_ptr<detail::HitManager>#: Hit manager, to be used only by LocalTransporter.

using SPOffloadWriter = std::shared_ptr<detail::OffloadWriter>#: Hit manager, to be used only by LocalTransporter.

using SPOutputRegistry = std::shared_ptr<OutputRegistry>#: Hit manager, to be used only by LocalTransporter.

using SPConstGeantGeoParams = std::shared_ptr<GeantGeoParams const>#: Hit manager, to be used only by LocalTransporter.

inline SPHitManager const &hit_manager() const#: Hit manager, to be used only by LocalTransporter.

inline SPOffloadWriter const &offload_writer() const#: Optional offload writer, only for use by LocalTransporter.

int num_streams() const#: Lazily obtained number of streams.

SPOutputRegistry const &output_reg() const#: Lazily created output registry.

SPConstGeantGeoParams const &geant_geo_params() const#: Lazily created Geant geometry parameters.

SharedParams(SetupOptions const &options, SPOutputRegistry reg)#

Set up Celeritas using Geant4 data and existing output registery.

A design oversight in the GeantSimpleCalo means that the action registry must be created before SharedParams is initialized, and in the case where Celeritas is not disabled, initialization clears the existing registry. This prevents the calorimeter from writing output.

void Initialize(SetupOptions const &options, SPOutputRegistry reg)#

Initialize shared data with existing output registry.

TODO: this is a hack to be deleted in v0.5.

void set_output_filename(std::string const&)#

Save a diagnostic output filename from a Geant4 app when Celeritas is off.

This will be overwritten when calling Initialized with setup options.

TODO: this hack should be deleted in v0.5.

Construction

static bool CeleritasDisabled()#

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

static bool KillOffloadTracks()#

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

static void InitializeWorker(SetupOptions const &options)#

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

SharedParams() = default#

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

explicit SharedParams(SetupOptions const &options)#

Set up Celeritas using Geant4 data.

This is a separate step from construction because it has to happen at the beginning of the run, not when user classes are created. It should be called from the “master” thread (for MT mode) or from the main thread (for Serial), and it must complete before any worker thread tries to access the shared data.

inline void Initialize(SetupOptions const &options)#: Helper for making initialization more obvious from user code.

void Finalize()#

Clear shared data after writing out diagnostics.

This should be executed exactly once across all threads and at the end of the run.

Accessors

inline SPConstParams Params() const#

Access Celeritas data.

This can only be called after Initialize.

VecG4ParticleDef const &OffloadParticles() const#: Get a vector of particles supported by Celeritas offloading.

inline explicit operator bool() const#: Whether Celeritas core params have been created.

class LocalTransporter#

Manage offloading of tracks to Celeritas.

This class must be constructed locally on each worker thread/task/stream, usually as a shared pointer that’s accessible to:

a run action (for initialization),
an event action (to set the event ID and flush offloaded tracks at the end of the event)
a tracking action (to try offloading every track)

Public Functions

LocalTransporter(SetupOptions const &options, SharedParams const &params)#: Construct with shared (MT) params.

inline void Initialize(SetupOptions const &options, SharedParams const &params)#

Helper for making initialization more obvious from user code.

This gives it some symmetry with Finalize, which is provided as an exception-friendly destructor.

void InitializeEvent(int)#: Set the event ID and reseed the Celeritas RNG at the start of an event.

void Push(G4Track const&)#: Convert a Geant4 track to a Celeritas primary and add to buffer.

void Flush()#: Transport the buffered tracks and all secondaries produced.

void Finalize()#

Clear local data.

This may need to be executed on the same thread it was created in order to safely deallocate some Geant4 objects under the hood…

MapStrReal GetActionTime() const#: Get the accumulated action times.

inline explicit operator bool() const#: Whether the class instance is initialized.

Interface utilities#

Logger celeritas::MakeMTLogger(G4RunManager const &runman)#

Construct a logger that will redirect Celeritas messages through Geant4.

This logger writes the current thread (and maximum number of threads) in each output message, and sends each message through the thread-local G4cerr.

In the main of your application’s exectuable, set the “process-local” (MPI-aware) logger:

celeritas::self_logger() = celeritas::MakeMTLogger(*run_manager);

class ExceptionConverter#

Translate Celeritas C++ exceptions into Geant4 G4Exception calls.

This should generally be used when wrapping calls to Celeritas in a user application.

For example, the user event action to transport particles on device could be used as:

void EventAction::EndOfEventAction(const G4Event*)
{
    // Transport any tracks left in the buffer
    celeritas::ExceptionConverter call_g4exception{"celer0003"};
    CELER_TRY_HANDLE(transport_->Flush(), call_g4exception);
}

Public Functions

inline ExceptionConverter(char const *err_code, SharedParams const *params)#

Construct with an error code and shared parameters.

The error code is reported to the Geant4 exception manager. The shared parameters are used to translate internal particle data if an exception occurs.

inline explicit ExceptionConverter(char const *err_code)#: Construct with an error code for dispatching to Geant4.

void operator()(std::exception_ptr p) const#: Capture the current exception and convert it to a G4Exception call.

class AlongStepFactoryInterface#

Helper class for emitting an AlongStep action.

Currently Celeritas accepts a single along-step action (i.e., the same stepper is used for both neutral and charged particles, across all energies and regions of the problem). The along-step action is a single GPU kernel that combines the field stepper selection, the magnetic field, slowing-down calculation, multiple scattering, and energy loss fluctuations.

The factory will be called from the thread that initializes SharedParams. Instead of a daughter class, you can provide any function-like object that has the same interface.

Celeritas provides a few “default” configurations of along-step actions in celeritas/global/alongstep.

Subclassed by celeritas::RZMapFieldAlongStepFactory, celeritas::UniformAlongStepFactory

Classes usable by Geant4#

These utilities are based on Celeritas data structures and capabilities but are written to be usable both by the celer-g4 app and potential other users.

class GeantSimpleCalo : public celeritas::OutputInterface #

Manage a simple calorimeter sensitive detector across threads.

The factory should be created in DetectorConstruction or DetectorConstruction::Construct and added to the output parameters. Calling MakeSensitiveDetector will emit a sensitive detector for the local thread and attach it to the logical volumes on the local thread.

Output interface

Category of data to write

inline virtual Category category() const final#: Return the key in the JSON output.

virtual std::string label() const final#: Return the key in the JSON output.

virtual void output(JsonPimpl*) const final#: Write output to the given JSON object.

Public Functions

GeantSimpleCalo(std::string name, SPConstParams params, VecLV volumes)#: Construct with name and logical volume pointers.

UPSensitiveDetector MakeSensitiveDetector()#: Emit a new detector for the local thread and attach to the stored LVs.

inline VecLV const &Volumes() const#: Get the list of volumes with this SD attached.

VecReal CalcTotalEnergyDeposition() const#

Calculate thread-integrated energy deposition.

This function should only be called after all detector data has been collected.

class HepMC3PrimaryGenerator : public G4VPrimaryGenerator#

HepMC3 reader class for sharing across threads.

This singleton is shared among threads so that events can be correctly split up between them, being constructed the first time instance() is invoked. As this is a derived G4VPrimaryGenerator class, the HepMC3PrimaryGenerator must be used by a concrete implementation of the G4VUserPrimaryGeneratorAction class:

void PrimaryGeneratorAction::GeneratePrimaries(G4Event* event)
{
    HepMC3PrimaryGenerator::Instance()->GeneratePrimaryVertex(event);
}

Public Functions

inline explicit HepMC3PrimaryGenerator(std::string const &filename)#: Construct with a path to a HepMC3-compatible input file.

inline void GeneratePrimaryVertex(G4Event *g4_event) final#

Add primaries to Geant4 event.

Add HepMC3 primaries to a Geant4 event.

This function should be called by G4VUserPrimaryGeneratorAction::GeneratePrimaries .

inline int NumEvents()#: Get total number of events.

class RZMapMagneticField : public G4MagneticField#

A user magnetic field equivalent to celeritas::RZMapField.

Public Functions

inline explicit RZMapMagneticField(SPConstFieldParams field_params)#: Construct with the Celeritas shared RZMapFieldParams.

inline void GetFieldValue(double const point[3], double *field) const#: Calculate the magnetic field vector at the given position.