How to run a cosmology simulation¶
In order to run a cosmology simulation, you’ll need to build enzo.exe, inits.exe and ring.exe (see Obtaining and Building Enzo) inits creates the initial conditions for your simulation, and ring splits up the root grid which is necessary if you’re using parallel IO. Once you have built the three executables, put them in a common directory where you will run your test simulation. You will also save the inits and param files (shown and discussed below) in this directory.
Creating initial conditions¶
The first step in preparing the simulation is to create the initial conditions. The file inits uses is a text file which contains a list of parameters with their associated values. These values tell the initial conditions generator necessary information like the simulation box size, the cosmological parameters and the size of the root grid. The code then takes that information and creates a set of initial conditions. Here is an example inits file:
# # Generates initial grid and particle fields for a # CDM simulation # # Cosmology Parameters # CosmologyOmegaBaryonNow = 0.044 CosmologyOmegaMatterNow = 0.27 CosmologyOmegaLambdaNow = 0.73 CosmologyComovingBoxSize = 10.0 // in Mpc/h CosmologyHubbleConstantNow = 0.71 // in units of 100 km/s/Mpc CosmologyInitialRedshift = 60 # # Power spectrum Parameters # PowerSpectrumType = 11 PowerSpectrumSigma8 = 0.9 PowerSpectrumPrimordialIndex = 1.0 PowerSpectrumRandomSeed = -584783758 # # Grid info # Rank = 3 GridDims = 32 32 32 InitializeGrids = 1 GridRefinement = 1 # # Particle info # ParticleDims = 32 32 32 InitializeParticles = 1 ParticleRefinement = 1 # # Overall field parameters # # # Names # ParticlePositionName = ParticlePositions ParticleVelocityName = ParticleVelocities GridDensityName = GridDensity GridVelocityName = GridVelocities
inits is run by typing this command:
./inits.exe -d Example_Cosmology_Sim.inits
inits will produce some output to the screen to tell you what it is
doing, and will write five files:
first four files contain information on initial conditions for the
baryon and dark matter componenets of the simulation, and are HDF5
files. The last file is an ascii file which contains information on
the power spectrum used to generate the initial conditions.
It is also possible to run cosmology simulations using initial nested subgrids.
Parallel IO - the ring tool¶
This simulation is quite small. The root grid is only 32 cells on a
side and we allow a maximum of three levels of mesh refinement.
Still, we will use the ring tool, since it is important for larger
simulations of sizes typically used for doing science. Additionally,
if you wish to run with 64 or more processors, you should use
ParallelRootGridIO, described in Parallel Root Grid IO.
The ring tool is part of the Enzo parallel IO (input-output) scheme. Examine the last section of the parameter file (see below) for this example simulation and you will see:
# # IO parameters # ParallelRootGridIO = 1 ParallelParticleIO = 1
These two parameters turn on parallel IO for both grids and particles. In a serial IO simulation where multiple processors are being used, the master processor reads in all of the grid and particle initial condition information and parcels out portions of the data to the other processors. Similarly, all simulation output goes through the master processor as well. This is fine for relatively small simulations using only a few processors, but slows down the code considerably when a huge simulation is being run on hundreds of processors. Turning on the parallel IO options allows each processor to perform its own IO, which greatly decreases the amount of time the code spends performing IO.
The process for parallelizing grid and particle information is quite different.
Since it is known exactly where every grid cell in a structured Eulerian grid
is in space, and these cells are stored in a regular and predictable order in
the initial conditions files, turning on
ParallelRootGridIO simply tells
each processor to figure out which portions of the arrays in the GridDensity
GridVelocities belong to it, and then read in only that part of the
file. The particle files (
store the particle information in no particular order. In order to efficiently
parallelize the particle IO the ring tool is used. ring is run on the same
number of processors as the simulation that you intend to run, and is typically
run just before Enzo is called for this reason. In ring, each processor reads
in an equal fraction of the particle position and velocity information into a
list, flags the particles that belong in its simulation spatial domain, and
then passes its portion of the total list on to another processor. After each
portion of the list has made its way to every processor, each processor then
collects all of the particle and velocity information that belongs to it and
writes them out into files called
PVel.nnnn, where nnnn
is the processor number. Turning on the
ParallelParticleIO flag in the Enzo
parameter file instructs Enzo to look for these files.
For the purpose of this example, you’re going to run ring and Enzo on 4 processors (this is a fixed requirement). The number of processors used in an MPI job is set differently on each machine, so you’ll have to figure out how that works for you. On some machines, you can request an ‘interactive queue’ to run small MPI jobs. On others, you may have to submit a job to the batch queue, and wait for it to run.
To start an interactive run, it might look something like this:
qsub -I -V -l walltime=00:30:00,size=4
This tells the queuing system that you want four processors total for a half hour of wall clock time. You may have to wait a bit until nodes become available, and then you will probably start out back in your home directory. You then run ring on the particle files by typing something like this:
mpirun -n 4 ./ring.exe pv ParticlePositions ParticleVelocities
This will then produce some output to your screen, and will
generate 8 files:
PVel.0003. Note that the ‘mpirun’ command may actually be ‘aprun’
or something similar. Consult your supercomputer’s documentation to
figure out what this command should really be.
Congratulations, you’re now ready to run your cosmology simulation!
Running an Enzo cosmology simulation¶
After all of this preparation, running the simulation itself should be straightforward. First, you need to have an Enzo parameter file. Here is an example compatible with the inits file above:
# # AMR PROBLEM DEFINITION FILE: Cosmology Simulation (AMR version) # # define problem # ProblemType = 30 // cosmology simulation TopGridRank = 3 TopGridDimensions = 32 32 32 SelfGravity = 1 // gravity on TopGridGravityBoundary = 0 // Periodic BC for gravity LeftFaceBoundaryCondition = 3 3 3 // same for fluid RightFaceBoundaryCondition = 3 3 3 # # problem parameters # CosmologySimulationOmegaBaryonNow = 0.044 CosmologySimulationOmegaCDMNow = 0.226 CosmologyOmegaMatterNow = 0.27 CosmologyOmegaLambdaNow = 0.73 CosmologySimulationDensityName = GridDensity CosmologySimulationVelocity1Name = GridVelocities CosmologySimulationVelocity2Name = GridVelocities CosmologySimulationVelocity3Name = GridVelocities CosmologySimulationParticlePositionName = ParticlePositions CosmologySimulationParticleVelocityName = ParticleVelocities CosmologySimulationNumberOfInitialGrids = 1 # # define cosmology parameters # ComovingCoordinates = 1 // Expansion ON CosmologyHubbleConstantNow = 0.71 // in km/s/Mpc CosmologyComovingBoxSize = 10.0 // in Mpc/h CosmologyMaxExpansionRate = 0.015 // maximum allowed delta(a)/a CosmologyInitialRedshift = 60.0 // CosmologyFinalRedshift = 3.0 // GravitationalConstant = 1 // this must be true for cosmology # # set I/O and stop/start parameters # CosmologyOutputRedshift = 25.0 CosmologyOutputRedshift = 10.0 CosmologyOutputRedshift = 5.0 CosmologyOutputRedshift = 3.0 # # set hydro parameters # Gamma = 1.6667 PPMDiffusionParameter = 0 // diffusion off DualEnergyFormalism = 1 // use total & internal energy InterpolationMethod = 1 // SecondOrderA CourantSafetyNumber = 0.5 ParticleCourantSafetyNumber = 0.8 FluxCorrection = 1 ConservativeInterpolation = 0 HydroMethod = 0 # # set cooling parameters # RadiativeCooling = 0 MultiSpecies = 0 RadiationFieldType = 0 StarParticleCreation = 0 StarParticleFeedback = 0 # # set grid refinement parameters # StaticHierarchy = 0 // AMR turned on! MaximumRefinementLevel = 3 MaximumGravityRefinementLevel = 3 RefineBy = 2 CellFlaggingMethod = 2 4 MinimumEfficiency = 0.35 MinimumOverDensityForRefinement = 4.0 4.0 MinimumMassForRefinementLevelExponent = -0.1 MinimumEnergyRatioForRefinement = 0.4 # # set some global parameters # GreensFunctionMaxNumber = 100 // # of greens function at any one time # # IO parameters # ParallelRootGridIO = 1 ParallelParticleIO = 1
Once you’ve saved this, you start Enzo by typing:
mpirun -n 4 ./enzo.exe -d Example_Cosmology_Sim.param >& output.log
The simulation will now run. The -d flag ensures a great deal of
output, so you may redirect it into a log file called
for later examination. This particular simulation shouldn’t take
too long, so you can run this in the same 30 minute interactive job
you started when you ran inits. When the simulation is done, Enzo
will display the message “Successful run, exiting.”
Congratulations! If you’ve made it this far, you have now successfully run a cosmology simulation using Enzo!