Dynamic Refinement Regions in Nested Cosmology Simulations

Traditionally, cosomological zoom-in simulations run with Enzo have used static nested regions that are rectangular in shape and surround the region of maximum refinement. These static regions have served two purposes: one, they define the volume in the initial conditions where high and intermediate resolution particles are placed; and two, once the simulation has begun, they define the regions where AMR to high levels is allowed. As particles of different masses move during the simulation, they will move to grids of either higher or lower resolution than their ‘birth’ grids where they were located during the problem initialization.

There are several strategies available for combating the limitations of this static approach, some of which are described below. In addition, it may be sometimes desireable to restrict the region of maximum refinement to a non-rectangular volume or to a subset of the highest resolution particles. Most of these approaches rely on must refine particles in Enzo, which are simply dark matter particles that have a specal type that signals the code to add extra refinement to the hydro mesh surrounding the particle (see Must Refine Particles). Some of these options (but not all) are optimized for use with the initial conditions generator MUSIC.

Non-static Nested Grids

With this method, the static grids used during the simulation initialization are allowed to dynamically change during the simulation itself. Refinement is adjusted to a minimum level appropriate for a dark matter particle’s mass each time the hierarchy is rebuilt. This option can be activated by setting MustRefineParticlesCreateParticles = 3. CosmologySimulationGrid parameters are still necessary because during the problem initialization particles of different masses can only be initialized in rectangular regions. However, the user should not set RefineRegionLeftEdge or RefineRegionRightEdge as one would for a static nested run. Also, StaticHierarchy = 0 should be set.

Within the code, this method works by identifying dark matter particles whose particle density is less than the cosmic dark matter density of the simulation as defined by CosmologySimulationOmegaCDMNow and OmegaMatterNow. Users should recall that particle ‘masses’ stored by Enzo are actually particle densities (see Enzo Particle Masses). When a dark matter particle resides on a grid whose resolution is equal to the resolution of the particle’s birth grid, the particle’s density will be exactly CosmologySimulationOmegaCDMNow/OmegaMatterNow. If a particle is on a grid of coarser resolution, the particle’s density will be less than this fraction; in this case, refinement of the mesh is triggered around the particle until the particle’s density equals CosmologySimulationOmegaCDMNow/OmegaMatterNow.

The user should note that this method gaurentees that a particle will never be on a grid coarser than its generation grid, but a consequence of the CIC flagging method and Enzo’s grid deconstrution method is that particles can be on grids with finer resolution. This method therefore does not prevent contamination of high resolution grids by low resolution particles.

MUSIC Ellipsoidal Masking

The nested cosmological initial conditions generator MUSIC can identify a subset high-resolution particles that lie within an ellipsoidal ‘masked’ region contained within the highest level nested region. MUSIC identifies these particles particles within files called RefinementMask.x. AMR in Enzo can be restricted to the volume containing these particles by setting MustRefineParticlesCreateParticles = 2 or 3, CosmologySimulationParticleTypeName = RefinementMask, CellFlaggingMethod = 8 and MustRefineParticlesRefineToLevel. Typically, MustRefineParticlesRefineToLevel should be set to at least one more level than the highest level of the nested regions. The level set for this parameter will force AMR to be done around the mask particles up to this level. In practice, within the code, masked particles are made must refine particles during the simulation initialization and given a distinct particle type from other dark matter particles, which allows the code to identify them as masked particles.

If additional refinement criteria are set with CellFlaggingMethod (such as refinement on gas density), higher AMR levels can be reached up to the MaximumRefinementLevel, but only for those cells contained within the region flagged by the masked particles.

In the surrounding intermediate resolution nested region, if MustRefineParticlesCreateParticles = 2, then traditional static nested regions are used. If MustRefineParticlesCreateParticles = 3, then non-static nested refinement as described above is conducted.

Non-ellipsoidal Masking

This method uses traditional static grids on intermediate levels, but within the highest level static region, maximum refinement is restricted to a volume surrounding a subset of the highest resolution dark matter particles by tagging them as must refine particles either from a list of particle IDs or within a specified rectangular box. It is activated by setting MustRefineParticlesCreateParticles = 1.

These particles can be selected in one of two ways: (1) by specifiying a rectangular volume with the parameters MustRefineParticlesRegionLeftEdge and MustRefineParticlesRightEdge; or (2) by providing a list of particle IDs. If MustRefineParticlesRegionLeftEdge and MustRefineParticlesRegionRightEdge are not set, but MustRefineParticlesCreateParticles = 1, then the code looks for an ascii file called MustRefineParticlesFlaggingList.in within the run directory. The list method allows for a non-simply connected refinement mask.

The list of particle IDs can be obtained from a prior dark-matter-only simulation of the refined initial conditions. The user should be cautitious when doing this, because particle IDs are not contained within the initial conditions; they are assigned during the problem initialization. Depending on the way in which each simulation is initialized, particle IDs are not gaurenteed to be idential between simulations initialized on different numbers of cores. These methods do not rely on the MUSIC initial conditions generator.

The user should also set CellFlaggingMethod and MustRefineParticlesRefineToLevel. Also, despite using static nested grids, StaticHierarchy = 0 should be set.


One simple strategy for dynamically adjusting intermediatly refined nested regions is to allow the nested rectangular regions to shrink in a regular way with the comological collapse of structures. This can be activated with the parameter RefineRegionAutoAdjust. With this method, nested regions still maintain a regular shape, but the rectangles are allowed to shrink as the highest resolution particles occupy a smaller and smaller volume.