Geant4 interface

The accel directory contains components exclusive to coupling Celeritas with Geant4 for user-oriented integration. A simple interface for multithreaded or serial applications is demonstrated in Geant4 integration example, and the more advanced implementation can be inspected in the Integrated Geant4 application (celer-g4) app.

High-level interface

The celeritas::SimpleOffload class is an extremely easy-to-use interface for offloading tracks to Celeritas in a multithreaded or serial application. The class names correspond to user actions and ActionInitialization. It requires a few app-owned pieces such as celeritas::SharedParams and celeritas::LocalTransporter to be owned by the calling application; the options described below must also be set up and provided.

class SimpleOffload

Compressed interface for running Celeritas in a multithread Geant4 app.

This class must be a thread-local instance with references to data that exceed the lifetime of the class: e.g. SharedParams can be a global variable, and LocalTransporter can be a global variable with thread_local storage duration.

The CELER_DISABLE environment variable, if set and non-empty, will disable offloading so that Celeritas will not be built nor kill tracks.

The method names correspond to methods in Geant4 User Actions and must be called from all threads, both worker and master.

Public Functions

SimpleOffload() = default

Construct with celeritas disabled.

SimpleOffload(SetupOptions const *setup, SharedParams *params, LocalTransporter *local)

Construct from a reference to shared params and local data.

On construction, this will check for the CELER_DISABLE variable and disable offloading if set. Otherwise it will initialize the multithread logging if the run manager is initialized.

inline void Build(SetupOptions const *setup, SharedParams *params, LocalTransporter *local)

Lazy initialization of this class on a worker thread.

inline void BuildForMaster(SetupOptions const *setup, SharedParams *params)

Lazy initialization of this class on the master thread.

void BeginOfRunAction(G4Run const *run)

Initialize celeritas data from setup options.

void BeginOfEventAction(G4Event const *event)

Send Celeritas the event ID and reseed the Celeritas RNG.

void PreUserTrackingAction(G4Track *track)

Send tracks to Celeritas if applicable and “StopAndKill” if so.

void EndOfEventAction(G4Event const *event)

Flush offloaded tracks from Celeritas.

void EndOfRunAction(G4Run const *run)

Finalize Celeritas.

inline explicit operator bool() const

Whether offloading is enabled.

The SetupOptionsMessenger can be instantiated with a reference to a global SetupOptions instance in order to provide a Geant4 “UI” macro interface to an app’s Celeritas options.

class SetupOptionsMessenger : public G4UImessenger

Expose setup options through the Geant4 “macro” UI interface.

The following options are exposed in the /celer/ command “directory”:

Command	Description
geometryFile	Override detector geometry with a custom GDML
outputFile	Filename for JSON diagnostic output
physicsOutputFile	Filename for ROOT dump of physics data
offloadOutputFile	Filename for HepMC3/ROOT dump of offloaded tracks
geometryOutputFile	Filename for GDML export
maxNumTracks	Number of tracks to be transported simultaneously
maxNumEvents	Maximum number of events in use
maxNumSteps	Limit on number of step iterations before aborting
maxInitializers	Maximum number of track initializers
secondaryStackFactor	At least the average number of secondaries per track
autoFlush	Number of tracks to buffer before offloading
maxFieldSubsteps	Limit on substeps in field propagator

The following option is exposed in the /celer/detector/ command “directory”:

Command	Description
enabled	Call back to Geant4 sensitive detectors

If a CUDA/HIP device is available, additional options are available under /celer/cuda/ :

Command	Description
stackSize	Set the CUDA per-thread stack size for VecGeom
heapSize	Set the CUDA per-thread heap size for VecGeom
actionTimes	Add timers around every action (may reduce performance)
defaultStream	Launch all kernels on the default stream

Warning

The given SetupOptions should be global or otherwise must exceed the scope of this UI messenger.

Celeritas setup

The setup options help translate the Geant4 physics and problem setup to Celeritas. They are also necessary to set up the GPU offloading characteristics. Future versions of Celeritas will automate more of these settings.

struct SetupOptions

Control options for initializing Celeritas.

The interface for the “along-step factory” (input parameters and output) is described in AlongStepFactoryInterface .

I/O

GDML filename (optional: defaults to exporting existing Geant4)

std::string geometry_file

Filename for JSON diagnostic output.

std::string output_file

Filename for JSON diagnostic output.

std::string physics_output_file

Filename for ROOT dump of physics data.

std::string offload_output_file

Filename to dump a ROOT/HepMC3 copy of offloaded tracks as events.

std::string geometry_output_file

Filename to dump a GDML file for debugging inside frameworks.

Celeritas stepper options

Number of track “slots” to be transported simultaneously

size_type max_num_tracks = {}

Maximum number of events in use (DEPRECATED: remove in v0.6)

size_type max_num_events = {}

Maximum number of events in use (DEPRECATED: remove in v0.6)

size_type max_steps = no_max_steps()

Limit on number of steps per track before killing.

size_type max_step_iters = no_max_steps()

Limit on number of step iterations before aborting.

size_type initializer_capacity = {}

Maximum number of track initializers (primaries+secondaries)

real_type secondary_stack_factor = {3.0}

At least the average number of secondaries per track slot.

size_type auto_flush = {}

Number of tracks to buffer before offloading (if unset: max num tracks)

CUDA options

Per-thread stack size (may be needed for VecGeom) [B]

size_type cuda_stack_size = {}

Dynamic heap size (may be needed for VecGeom) [B].

size_type cuda_heap_size = {}

Dynamic heap size (may be needed for VecGeom) [B].

bool action_times = {false}

Sync the GPU at every kernel for timing.

bool default_stream = {false}

Launch all kernels on the default stream for debugging.

Diagnostic setup

Filename base for slot diagnostics

Public Members

IntAccessor get_num_streams

Set the number of streams (defaults to run manager # threads)

SDSetupOptions sd

Sensitive detector options.

Public Static Functions

static inline constexpr size_type no_max_steps()

Don’t limit the number of steps.

struct SDSetupOptions

Control options for initializing Celeritas SD callbacks.

By default, Celeritas connects to Geant4 sensitive detectors so that it reconstructs full-fidelity hits with all available step information.

• If your problem has no SDs, you must set enabled to false.

• By default, steps that do not deposit energy do not generate any hits.

• To improve performance and memory usage, determine what quantities (time, position, direction, touchable, …) are required by your setup’s sensitive detectors and set all other attributes to false.

• Reconstructing the full geometry status using locate_touchable is the most expensive detector option. Disable it unless your SDs require (e.g.) the volume’s copy number to locate a detector submodule.

• Some reconstructed track attributes (such as post-step material) are currently never set because they are rarely used in practice. Contact the Celeritas team or submit a pull request to add this functionality.

Various attributes on the step, track, and pre/post step points may be available depending on the selected options.

• Disabling track will leave G4Step::GetTrack as nullptr .

• Enabling track will set the Charge attribute on the pre-step.

• Requested post-step data including GlobalTime, Position, KineticEnergy, and MomentumDirection will be copied to the Track when the combination of options is enabled.

• Some pre-step properties (Material and MaterialCutsCouple, and sensitive detector) are always updated. Post-step values for those are not set.

• Track and Parent IDs will never be a valid value since Celeritas track counters are independent from Geant4 track counters. Similarly, special Geant4 user-defined UserInformation and AuxiliaryTrackInformation are never set.

The force_volumes option can be used for unusual cases (i.e., when using a custom run manager) that do not define SDs on the “master” thread. Similarly, the skip_volumes option allows optimized GPU-defined SDs to be used in place of a Geant4 callback. For both options, the FindVolumes helper function can be used to determine LV pointers from the volume names.

Note

These setup options affect only the HitManager construction that is responsible for reconstructing CPU hits and sending directly to the Geant4 detectors. It does not change the underlying physics.

Public Functions

inline explicit operator bool() const

True if SD is enabled.

Public Members

bool enabled = {true}

Call back to Geant4 sensitive detectors.

bool ignore_zero_deposition = {true}

Skip steps that do not deposit energy locally.

bool energy_deposition = {true}

Save energy deposition.

bool locate_touchable = {true}

Set TouchableHandle for PreStepPoint.

bool track = {true}

Create a track with the dynamic particle type and post-step data.

StepPoint pre

Options for saving and converting beginning-of-step data.

StepPoint post

Options for saving and converting end-of-step data.

std::unordered_set<G4LogicalVolume const*> force_volumes

Manually list LVs that don’t have an SD on the master thread.

std::unordered_set<G4LogicalVolume const*> skip_volumes

List LVs that should not have automatic hit mapping.

struct StepPoint

Public Members

bool direction = {true}

AKA momentum direction.

std::unordered_set<G4LogicalVolume const*> celeritas::FindVolumes(std::unordered_set<std::string> names)

Find volumes by name for SDSetupOptions.

Example:

setup.sd.force_volumes = FindVolumes({"foo", "bar"});

class UniformAlongStepFactory : public celeritas::AlongStepFactoryInterface

Create an along-step method for a uniform (or zero) field.

The constructor is a lazily evaluated function that must return the field definition and driver configuration. If unspecified, the field is zero.

class RZMapFieldAlongStepFactory : public celeritas::AlongStepFactoryInterface

Create an along-step method for a two-dimensional (r-z in the cylindical coordinate system) map field (RZMapField).

Detailed interface

These classes are usually integrated into UserActions. The SimpleOffload interface above hides the complexity of these classes, or for more complex applications you can choose to use these classes directly instead of it.

class SharedParams

Shared (one instance for all threads) Celeritas problem data.

The CeleritasDisabled accessor queries the CELER_DISABLE environment variable as a global option for disabling Celeritas offloading. This is implemented by SimpleOffload

This should be instantiated on the master thread during problem setup, preferably as a shared pointer. The shared pointer should be passed to a thread-local LocalTransporter instance. At the beginning of the run, after Geant4 has initialized physics data, the Initialize method must be called first on the “master” thread to populate the Celeritas data structures (geometry, physics). InitializeWorker must subsequently be invoked on all worker threads to set up thread-local data (specifically, CUDA device initialization).

Some low-level objects, such as the output diagnostics and Geant4 geometry wrapper, can be created independently of Celeritas being enabled.

Internal use only

using SPHitManager = std::shared_ptr<detail::HitManager>

Hit manager, to be used only by LocalTransporter.

If sensitive detector callback is disabled, the hit manager will be null.

using SPOffloadWriter = std::shared_ptr<detail::OffloadWriter>

Hit manager, to be used only by LocalTransporter.

If sensitive detector callback is disabled, the hit manager will be null.

using SPOutputRegistry = std::shared_ptr<OutputRegistry>

Hit manager, to be used only by LocalTransporter.

If sensitive detector callback is disabled, the hit manager will be null.

using SPState = std::shared_ptr<CoreStateInterface>

Hit manager, to be used only by LocalTransporter.

If sensitive detector callback is disabled, the hit manager will be null.

using SPConstGeantGeoParams = std::shared_ptr<GeantGeoParams const>

Hit manager, to be used only by LocalTransporter.

If sensitive detector callback is disabled, the hit manager will be null.

inline SPHitManager const &hit_manager() const

Hit manager, to be used only by LocalTransporter.

If sensitive detector callback is disabled, the hit manager will be null.

inline SPOffloadWriter const &offload_writer() const

Optional offload writer, only for use by LocalTransporter.

inline SPOutputRegistry const &output_reg() const

Output registry for writing data at end of run.

void set_state(unsigned int stream_id, SPState&&)

Let LocalTransporter register the thread’s state.

unsigned int num_streams() const

Lazily obtained number of streams.

SPConstGeantGeoParams const &geant_geo_params() const

Lazily created Geant geometry parameters.

Construction

static bool CeleritasDisabled()

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

static bool KillOffloadTracks()

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

static void InitializeWorker(SetupOptions const &options)

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

SharedParams() = default

On worker threads, set up data with thread storage duration.

Some data that has “static” storage duration (such as CUDA device properties) in single-thread mode has “thread” storage in a multithreaded application. It must be initialized on all threads.

explicit SharedParams(SetupOptions const &options)

Set up Celeritas using Geant4 data.

This is a separate step from construction because it has to happen at the beginning of the run, not when user classes are created. It should be called from the “master” thread (for MT mode) or from the main thread (for Serial), and it must complete before any worker thread tries to access the shared data.

explicit SharedParams(std::string output_filename)

Set up Celeritas components for a Geant4-only run.

This is for doing standalone Geant4 calculations without offloading from Celeritas, but still using components such as the simple calorimeter.

inline void Initialize(SetupOptions const &options)

Helper for making initialization more obvious from user code.

inline void Initialize(std::string output_filename)

Helper for making initialization more obvious from user code.

void Finalize()

Clear shared data after writing out diagnostics.

This should be executed exactly once across all threads and at the end of the run.

Accessors

inline SPConstParams Params() const

Access Celeritas data.

This can only be called after Initialize.

VecG4ParticleDef const &OffloadParticles() const

Get a vector of particles supported by Celeritas offloading.

inline explicit operator bool() const

Whether Celeritas core params have been created.

class LocalTransporter

Manage offloading of tracks to Celeritas.

This class must be constructed locally on each worker thread/task/stream, usually as a shared pointer that’s accessible to:

• a run action (for initialization),

• an event action (to set the event ID and flush offloaded tracks at the end of the event)

• a tracking action (to try offloading every track)

Todo:

Rename LocalOffload or something?

Warning

Due to Geant4 thread-local allocators, this class must be finalized or destroyed on the same CPU thread in which is created and used!

Public Functions

LocalTransporter(SetupOptions const &options, SharedParams &params)

Construct with shared (MT) params.

inline void Initialize(SetupOptions const &options, SharedParams &params)

Helper for making initialization more obvious from user code.

This gives it some symmetry with Finalize, which is provided as an exception-friendly destructor.

void InitializeEvent(int)

Set the event ID and reseed the Celeritas RNG at the start of an event.

void Push(G4Track const&)

Convert a Geant4 track to a Celeritas primary and add to buffer.

void Flush()

Transport the buffered tracks and all secondaries produced.

void Finalize()

Clear local data.

This may need to be executed on the same thread it was created in order to safely deallocate some Geant4 objects under the hood…

MapStrReal GetActionTime() const

Get the accumulated action times.

inline explicit operator bool() const

Whether the class instance is initialized.

Interface utilities

Logger celeritas::MakeMTLogger(G4RunManager const &runman)

Construct a logger that will redirect Celeritas messages through Geant4.

This logger writes the current thread (and maximum number of threads) in each output message, and sends each message through the thread-local G4cerr.

In the main of your application’s exectuable, set the “process-local” (MPI-aware) logger:

celeritas::self_logger() = celeritas::MakeMTLogger(*run_manager);

class ExceptionConverter

Translate Celeritas C++ exceptions into Geant4 G4Exception calls.

This should generally be used when wrapping calls to Celeritas in a user application.

For example, the user event action to transport particles on device could be used as:

void EventAction::EndOfEventAction(const G4Event*)
{
    // Transport any tracks left in the buffer
    celeritas::ExceptionConverter call_g4exception{"celer0003"};
    CELER_TRY_HANDLE(transport_->Flush(), call_g4exception);
}

class AlongStepFactoryInterface

Helper class for emitting an AlongStep action.

Currently Celeritas accepts a single along-step action (i.e., the same stepper is used for both neutral and charged particles, across all energies and regions of the problem). The along-step action is a single GPU kernel that combines the field stepper selection, the magnetic field, slowing-down calculation, multiple scattering, and energy loss fluctuations.

The factory will be called from the thread that initializes SharedParams. Instead of a daughter class, you can provide any function-like object that has the same interface.

Celeritas provides a few “default” configurations of along-step actions in celeritas/alongstep.

Subclassed by celeritas::RZMapFieldAlongStepFactory, celeritas::UniformAlongStepFactory

Classes usable by Geant4

These utilities are based on Celeritas data structures and capabilities but are written to be usable both by the celer-g4 app and potential other users.

class GeantSimpleCalo : public celeritas::OutputInterface

Manage a simple calorimeter sensitive detector across threads.

The factory should be created in DetectorConstruction or DetectorConstruction::Construct and added to the output parameters. Calling MakeSensitiveDetector will emit a sensitive detector for the local thread and attach it to the logical volumes on the local thread.

Output interface

Category of data to write

class HepMC3PrimaryGenerator : public G4VPrimaryGenerator

HepMC3 reader class for sharing across threads.

This singleton is shared among threads so that events can be correctly split up between them, being constructed the first time instance() is invoked. As this is a derived G4VPrimaryGenerator class, the HepMC3PrimaryGenerator must be used by a concrete implementation of the G4VUserPrimaryGeneratorAction class:

void PrimaryGeneratorAction::GeneratePrimaries(G4Event* event)
{
    HepMC3PrimaryGenerator::Instance()->GeneratePrimaryVertex(event);
}

class RZMapMagneticField : public G4MagneticField

A user magnetic field equivalent to celeritas::RZMapField.

Low-level Celeritas integration

This subsection contains details of importing Geant4 data into Celeritas.

Geant4 geometry utilities

inline G4VPhysicalVolume *celeritas::load_geant_geometry(std::string const &filename)

Load a Geant4 geometry, leaving the pointer suffixes intact for VecGeom.

Do not strip 0x extensions since those are needed to deduplicate complex geometries (e.g. CMS) when loaded separately by VGDML and Geant4. The pointer-based deduplication is handled by the Label and LabelIdMultiMap.

Returns:

Geant4-owned world volume

inline std::unordered_set<G4LogicalVolume const*> celeritas::find_geant_volumes(std::unordered_set<std::string> names)

Find Geant4 logical volumes corresponding to a list of names.

If logical volumes with duplicate names are present, they will all show up in the output and a warning will be emitted. If one is missing, a RuntimeError will be raised.

static std::string_view const labels[] = {"Vol1", "Vol2"};
auto vols = find_geant_volumes(make_span(labels));

class Converter

Create an in-memory VecGeom model from an in-memory Geant4 model.

Return the new world volume and a mapping of Geant4 logical volumes to VecGeom-based volume IDs.

Geant4 physics interfaces

class GeantImporter : public celeritas::ImporterInterface

Load problem data directly from Geant4.

This can be used to circumvent ROOT as a serialization tool, whether to simplify the toolchain or to integrate better with user frameworks. As much data as possible is imported (subject to the data selection); downstream Celeritas classes will validate the imported data as needed.

GeantImporter import(GeantSetup("blah.gdml"));
ImportData data = import();

or to import from an existing, initialized Geant4 state:

GeantImport import(world_volume);
ImportData data = import();

class GeantSetup

Construct a Geant 4 run manager and populate internal Geant4 physics.

This is usually passed directly into GeantImporter . It hides Geant4 implementation details (including header files) from the rest of the code. It is safe to include even when Geant4 is unavailable!

The setup is targeted specifically for physics that Celeritas supports.

Geant4 physics options

struct GeantPhysicsOptions

Construction options for Geant physics.

These options attempt to default to our closest match to G4StandardEmPhysics. They are passed to the detail::CelerEmPhysicsList and detail::CelerFTFPBert physics lists to provide an easy way to set up physics options.

Gamma physics

Enable Compton scattering

bool compton_scattering = {true}

Enable the photoelectric effect.

bool photoelectric = {true}

Enable the photoelectric effect.

bool rayleigh_scattering = {true}

Enable Rayleigh scattering.

bool gamma_conversion = {true}

Enable electron pair production.

bool gamma_general = {false}

Use G4GammaGeneral instead of individual gamma processes.

Electron and positron physics

Enable discrete Coulomb

bool coulomb_scattering = {false}

Enable e- and e+ ionization.

bool ionization = {true}

Enable e- and e+ ionization.

bool annihilation = {true}

Enable positron annihilation.

BremsModelSelection brems = {BremsModelSelection::all}

Enable bremsstrahlung and select a model.

MscModelSelection msc = {MscModelSelection::urban}

Enable multiple coulomb scattering and select a model.

RelaxationSelection relaxation = {RelaxationSelection::none}

Enable atomic relaxation and select a model.

Physics options

Number of log-spaced bins per factor of 10 in energy

int em_bins_per_decade = {7}

Enable universal energy fluctuations.

bool eloss_fluctuation = {true}

Enable universal energy fluctuations.

bool lpm = {true}

Apply relativistic corrections for select models.

bool integral_approach = {true}

See PhysicsParamsOptions::disable_integral_xs.

Cutoff options

Lowest energy of any EM physics process

MevEnergy min_energy = {0.1 * 1e-3}

Highest energy of any EM physics process.

MevEnergy max_energy = {100 * 1e6}

Highest energy of any EM physics process.

double linear_loss_limit = {0.01}

See PhysicsParamsOptions::linear_loss_limit.

MevEnergy lowest_electron_energy = {0.001}

Tracking cutoff kinetic energy for e-/e+.

bool apply_cuts = {false}

Kill secondaries below the production cut.

double default_cutoff = {0.1 * units::centimeter}

Set the default production cut for all particle types [len].

Multiple scattering configuration

E-/e+ range factor for MSC models

double msc_range_factor = {0.04}

Safety factor for MSC models.

double msc_safety_factor = {0.6}

Safety factor for MSC models.

double msc_lambda_limit = {0.1 * units::centimeter}

Lambda limit for MSC models [len].

double msc_theta_limit = {constants::pi}

Polar angle limii between single and multiple Coulomb scattering.

double angle_limit_factor = {1}

Factor for dynamic computation of angular limit between SS and MSC.

MscStepLimitAlgorithm msc_step_algorithm = {MscStepLimitAlgorithm::safety}

Step limit algorithm for MSC models.

NuclearFormFactorType form_factor = {NuclearFormFactorType::exponential}

Nuclear form factor model for Coulomm scattering.

Public Members

GeantMuonPhysicsOptions muon

Muon EM physics.

bool verbose = {false}

Print detailed Geant4 output.

GeantOpticalPhysicsOptions optical = {GeantOpticalPhysicsOptions::deactivated()}

Optical physics options.