Task List

Zoom Link: https://cern.zoom.us/j/7652031579?pwd=HhT6xn28qVuJJ5yhu0TdhctegbjtN7.1

Task List Spreadsheet: https://docs.google.com/spreadsheets/d/159vRkuQ4VSZwLp4ARx5AZhcYsIpUfLfnX82SNIcR_Bw/edit?usp=sharing

Simulation with Dual-Readout Calorimetry

Setting Up the Working Area

Check the Centos version and ensure it is 8. You can do so with either of the 2 commands shown below.

$ rpm -E %{rhel}
$ cat /etc/redhat-release

Once Centos version is set to 8, run the following command. On the Baylor cluster, also do the following module load. When you need to compile codes, use gpu006 (due to motif-devel availability). Job execution can be done on any node.

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/setup.sh
$ module load gl_fix

Then pull DD4hep code from the GitHub and DualTestBeam code from the GitLab. You will need to ask for permission from jon_wilson2@baylorNOSPAM.edu for pull access from the CalVision GitLab.

$ git clone https://github.com/AIDASoft/DD4hep.git
$ cd DD4hep/examples
$ git clone https://gitlab.cern.ch/calvisionsimulation/DualTestBeam.git

Note: If you are brave, determined, or stupid, you can pull the beta release at: git clone git@githubNOSPAM.com:saraheno/DualTestBeam.git. Please do note that the https://github.com/AIDASoft/DD4hep.git repository is for more sophisticated coding. Make sure to start with https://gitlab.cern.ch/calvisionsimulation/DualTestBeam.git.

Next, edit DD4hep/examples/CMakeLists.txt and add " DualTestBeam " after " SimpleDetector " in the following line.

{SET(DD4HEP_EXAMPLES "AlignDet CLICSiD ClientTests Conditions DDCMS DDCodex DDDigi DDG4 DDG4_MySensDet LHeD OpticalSurfaces RICH Persistency DDCAD SimpleDetector DualTestBeam"
CACHE STRING "List of DD4hep Examples to build")

Then follow these commands to compile and make.

$ cd ..
$ mkdir build
$ mkdir install
$ cd build
$ cmake -DDD4HEP_USE_GEANT4=ON -DBoost_NO_BOOST_CMAKE=ON -DDD4HEP_USE_LCIO=ON -DBUILD_TESTING=ON -DROOT_DIR=$ROOTSYS -D CMAKE_BUILD_TYPE=Release -DDD4HEP_BUILD_EXAMPLES=ON -DCMAKE_INSTALL_PREFIX=../install ..
$ make -j4
$ make install
$ cd ..
$ source install/bin/thisdd4hep.sh
$ cd examples/DualTestBeam/compact

Setting Up Each Subsequent Time

$ cd DD4hep
$ source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/setup.sh
$ source install/bin/thisdd4hep.sh
$ (only for Baylor cluster) module load gl_fix

Running Interactively

$ ddsim --compactFile=DRConly.xml --runType=batch -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-1*cm" --gun.direction "0 0 1" --gun.energy "5*GeV" --gun.particle="e-" -N 1

Some DD4hep Documentation

DD4hep Guide for Young Students: https://docs.google.com/document/d/1J1Qx08Ozn8g4XEAypybaJbVPWNe5J576tK-5u2ouMt4/edit?usp=sharing

GitLab containing DualTestBeam code: https://gitlab.cern.ch/snabili/DualTestBeam/-/tree/dev_SN

Test Beam Array

Similar to the above instructions, but get the code from https://github.com/saraheno/DualTestBeam.git. It should also work with https://gitlab.cern.ch/calvisionsimulation/DualTestBeam. Then run interactively with a command as shown below. Note that the z position should be at - [(crystallength+fiberlength+gap+edgeoffset)/2+1]*10 to put in mm.

$ ddsim --compactFile=./DRDualTestBeam.xml --runType=vis -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-780." --gun.direction "0 0 1" --gun.energy "10*GeV" --gun.particle="mu-"

The following command is useful for debugging.

$ ddsim --compactFile=./DRJunk.xml --runType=vis -G --steeringFile junkst.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-780." --gun.direction "0 0 1" --gun.energy "10*GeV" --gun.particle="pi-"

Then initiate with the following command.

$ /control/execute vis2.mac

To run batch jobs, use the following command.

$ ddsim --compactFile=./DRDualTestBeam.xml --runType=batch -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-780." --gun.direction "0 0 1" --gun.energy "10*GeV" --gun.particle="pi-" -N 5

Kludge for now. The first argument below is the number of events to analyze. It must be less than or equal to the number of events in out.root. The last argument below is the particle gun energy in GeV.

$ crystalana(10,"out.root", 10)

Test Beam Array with Tile HCAL

The Tile HCAL has the same code repository as the other Test Beam module. You can run interactively with the following command.

$ ddsim --compactFile=./DRDualSampHcal.xml --runType=vis -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-2400." --gun.direction "0 0 1" --gun.energy "10*GeV" --gun.particle="mu-"

Running Resolution.C

In your output directory, use the following command.

$ root -l -b -q 'Resolution.C(nevt,"electron.root","pion.root","hcalonly.root",energy,doecal,dohcal,hcaltype,doedge,gendet,"hists.root","ECALleaf","HCALleaf")'

A sample command with arguments is shown here.

$ root -l -b -q 'Resolution.C(500,"out_FSCEPonly_e-10gev.root","out_FSCEPonly_pi-10gev.root","out_FSCEPonly_e-10gev.root",10,1,1,0,1,3,"hists_FSCEPonly_3_10gev.root","DRCNoSegment","DRFNoSegment")'

For the latter, you need to change the arguments to match what you want to do.

No.	Argument	Description
6	doecal	True (1) if there is a crystal ECAL in your geometry.
9	doedge	Always set this to be true (1). This is for the edge detector.
7	dohcal	True (1) if there is any detector that is not a crystal ECAL in your geometry.
12	ECALleaf	This is related to how the data is stored in the input root tuple. Always set this to DRCNoSegment.
2	electron.root	File containing electron simulations for the geometry being studied.
5	energy	The energy of the beam in the simulation.
10	gendet	Determines how to calculate deposited energy. 1 counts photons at creation point in media, 2 counts photons that reach kill media, and 3 uses the energy deposit itself.
13	HCALleaf	This is related to how the data is stored in the input root tuple. Set to DRFNoSegment for fiber (spaghetti) HCAL. Set to DRSNoSegment for transversely segmented HCAL.
4	hcalonly.root	Used for geometries where ECAL and HCAL have different technologies. This is often the same as argument 2.
8	hcaltype	If the above is true, 0 for the fiber (spaghetti) HCAL and 1 for the transversely segmented HCAL. If above is false, this does not matter.
11	hists.root	Output file containing relevant histograms created in the Resolution.C code.
1	nevt	Number of events to analyze. Ensure this is less than or equal to the number of events in the input electron and pion files. If testing, set to a small number for fast runtime.
3	pion.root	File containing pion simulations for the geometry being studied.

Running Mass Jobs

There are three programs to run mass jobs. INCOMPLETE SECTION.

$ python massjobs2.py -g FSCEPonly
$ source ./massjobs.sh
$ condor_submit ProcessResvE.jdl
$ condor_submit ProcessResvE2.jdl

At Baylor cluster with pbs instead of condor, do the following. Check various job settings like direction, position, number of events or directory.

$ python3 massjobs_pbsarray.py -g FSCEPonly
$ mkdir -p output
$ cd jobs
$ bash submit.sh
$ cd ..

If there are multiple outputs, you may use the hadd command. For large sizes, you may have to increase the default 100GB file size limit. Refer to https://root-forum.cern.ch/t/root-6-04-14-hadd-100gb-and-rootlogon/24581/2.

Then, to make resolution plots vs energy, open ROOT and type the following.

[0] .L ResvE.C
[1] res()

Single Crystal Simulation with DD4hep

Setting Up the Working Area

Create a subdirectory to hold your work. Then do the following.

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos7-gcc11-opt/setup.sh
$ git clone https://github.com/AIDASoft/DD4hep.git
$ cd DD4hep/examples
$ git clone https://github.com/Mekhpar/SingleCrystal_cosmic_ray.git

Note that it is very important that you do not use any LCG version earlier than 102, because your Geant4 version will be too old! If you are on Rocky/Alma Linux 8, you can instead use the following command.

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/setup.sh

Then, edit CMakeLists.txt and add SingleCrystal_cosmic_ray to

SET(DD4HEP _EXAMPLES "AlignDet CLICSiD ClientTests Conditions DDCMS DDCodex DDDigi DDG4 DDG4_MySensDet LHeD Optica\<br />lSurfaces Persistency DDCAD SimpleDetector SingleDualCrystal"
    CACHE STRING "List of DD4hep Examples to build")

(if you created a new hit class, need to add it in CMakelists.txt in your top area for your checked out code on the line SOURCES $(DD4hep_DIR}include/root/Warnings.h)

$ cd ..
$ mkdir build
$ mkdir install
$ cd build
$ cmake -DDD4HEP_USE_GEANT4=ON -DBoost_NO_BOOST_CMAKE=ON -DDD4HEP_USE_LCIO=ON -DBUILD_TESTING=ON -DROOT_DIR=$ROOTSYS -D CMAKE_BUILD_TYPE=Release -DDD4HEP_BUILD_EXAMPLES=ON -DCMAKE_INSTALL_PREFIX=../install ..
$ make -j4
$ make install -j4
$ cd ..
$ source install/bin/thisdd4hep.sh

Each subsequent time

$ cd dd4hep/DD4hep
$ source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos7-gcc11-opt/setup.sh
$ source bin/thisdd4hep.sh

The same caveats as above about the LCG version vis-a-vis GEANT4 version, and CentOS 7 vs. Rocky/Alma 8 apply to the LCG setup line here.

To make changes

When you make any change to the C++ code, go to DD4hep/build and do make -j4 followed by make install -j4.

To visualize the geometry

go to the "compact" subdirectory geoDisplay DRSingleCrystal.xml

it can be useful to click on somethign and select "SetVisibility"

also to see details on the geometry at the command line:

geoPluginRun -input ../checkout/examples/ClientTests/compact/FiberTubeCalorimeter.xml -volmgr -plugin DD4hep_VolumeDump

can do this with --volmgr to check for duplicates

also geoPluginRun -input ../checkout/examples/ClientTests/compact/FiberTubeCalorimeter.xml -volmgr -plugin DD4hep_DetectorCheck -name all -geometry -structure -volmgr -sensitive

to put the geometry in the form the GEANT4 team will look at

do not do this as it is broken:

geoConverter -compact2gdml -input DRSingleCrystal.xml -output DRSingleCrystal.gdml

instead, start a run, except using a runtype of "shell". (ddsim --steeringFile SCEPCALsteering.py --compact ./DRSingleCrystal.xml --runType shell --part.userParticleHandler='' -G --gun.position="0.,30.,0." --gun.direction "0 -1 0" --gun.energy "1*GeV" --gun.particle="mu-" -O out.root)

once at the prompt, do

/ddg4/ConstructGeometry/writeGDML file-name

You can also run the overlap checking tool

/geometry/test/run

to run

cd examples/SingleDualCrystal/compact

ddsim --steeringFile SCEPCALsteering.py --compact ./DRSingleCrystal.xml --runType batch --part.userParticleHandler='' -G --gun.position="0.,30.,0." --gun.direction "0 -1 0" --gun.energy "1*GeV" --gun.particle="mu-" -N 1 -O out.root

to run in batch mode on the umd hep cluster (see https://sites.google.com/umd.edu/umdt3public/user-guide/submitting-analysis-jobs?authuser=0)

condor_submit condor-job.jdl

check with condor_q

to analyze out.root

root.exe

at root prompt: .L CrystalAna.C

crystalana(10,"out.root",1.,1.) (first argument is the number of events to analyze. must be less than or equal to the number of events in out.root)

(or you can do root -l -b -q CrystalAna.C >> haha.txt after editing the arguments by hand at the very bottom of the file)

then look at the resulting histograms in hist.root

to run interactively

ddsim --compactFile=./DRSingleCrystal.xml --runType=vis -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,30.,0." --gun.direction "0 -1 0" --gun.energy "1*GeV" --gun.particle="mu-"

ddsim --compactFile=./DRSingleCrystal.xml --runType=vis -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-200." --gun.direction "0 0 1" --gun.energy "10*GeV" --gun.particle="pi-"

/control/execute vis.mac

/run/beamOn 1

Then do typical GEANT4 visualization commands like:

http://www.hep.ph.ic.ac.uk/~yoshiu/COMET/comet_g4HTMLdoc/_vis_viewer_.html

/vis/viewer/refresh

/vis/viewer/zoomTo 10

/vis/viewer/pan

/view/viewer/panTo

/vis/viewer/panTo 0.2 0.0

/vis/viewer/set/viewpointThetaPhi 70 20

useful /ddg4/Gun/Particle mu- and /ddg4/Gun/Position (0.,100.,0)

( for debugging, this command is useful because it makes much fewer photons, making it easier to see their trajectories

ddsim --compactFile=./DRJunk.xml --runType=vis -G --steeringFile junkst.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,30.,0." --gun.direction "0 -1 0" --gun.energy "1*GeV" --gun.particle="mu-"

to run with gdb

gdb --args `which python3` `which ddsim` --compactFile=../compact/DRFSCEPonly.xml --runType=batch -G --steeringFile ../compact/SCEPCALsteering.py --outputFile=out_FSCEPonly_10GeV_e-.root --part.userParticleHandler='' -G --runType shell > catch throw std:bad_alloc
> run
# wait for the exception > bt

including the cosmic ray generator

source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos7-gcc11-opt/setup.sh

make a directory to hold your work

mkdir dd4hep

cd dd4hep

git clone --branch standAlone https://github.com/andresailer/SingleDualCrystal.git

cd SingleDualCrystal

mkdir build

mkdir install

cd build/

cmake -D CMAKE_INSTALL_PREFIX=$PWD/../install ..

make -j4

make install

cd ..

source install/bin/thisSingleCrystal.sh

git clone https://github.com/saraheno/DDCry.git

cd DDCry

git clone https://github.com/saraheno/cry.git

make -C cry

mkdir build

cd build

cmake -D CMAKE_BUILD_TYPE=Release -D BUILD_TESTING=OFF -D CMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$PWD/../install ..

make

make install

cd ..

source install/bin/thisDDCry.sh

source /cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos7-gcc11-opt/setup.sh

# now go back to your SingleDualCrystal /compact area

cd ../compact
ddsim --steeringFile SCEPCALcosmicSteering.py --compact ./DRSingleCrystal.xml --runType run -N 1 -O out.root

making digis

mkdir stuff4stuff

cd stuff4stuff

git clone git@githubNOSPAM.com:saraheno/DualTestBeam.git

cd DualTestBeam

mkdir build

mkdir install

cd build

singularity run -B /cvmfs:/cvmfs -B /data:/data docker://gitlab-registry.cern.ch/sft/docker/alma9-core:latest

source /cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc14-opt/setup.sh

make -j4

make install

cd ..

source ./install/bin/thisDualTestBeam.sh

cd compact

ddsim --compactFile=DRConly.xml --runType=batch -G --steeringFile SCEPCALsteering.py --outputFile=junk.root --part.userParticleHandler= -G --gun.position="0. 0. -1*cm" --gun.direction "0. 0. 1." --gun.energy "20*GeV" --gun.particle="pi-" --outputFile=junk.root -v VERBOSE -N 1 >& crash.txt

my fork of Chekanov's code

ddsim --compactFile=./DRSingleCrystal.xml --runType=vis -G --steeringFile SCEPCALsteering.py --outputFile=out.root --part.userParticleHandler='' -G --gun.position="0.,0.,-800." --gun.direction "0 0 1" --gun.energy "20*GeV" --gun.particle="mu-"

-- Sarah Eno - 22 Aug 2022