Running Enzo with CUDA

Enzo contains CUDA version of PPM and MHD solver. Supported parameters include

  • PPM: TwoShock, HLL, HLLC, PPMFlatteningParameter, PPMSteepeningParameter, DualEnergyFormalism, RiemannSolverFallback
  • Gravity
  • Color fields: chemistry, etc
  • Driving field
  • Comoving coordinates
  • EOSType: 0

How to compile with CUDA

In order to make Enzo compatible with CUDA, a few changes need to be made in the settings.

Currently Enzo CUDA is only compatible with 32-bit precision. In order to correctly set this, make sure that in src/enzo/ you run the following configuration commands:

make cuda-yes
make integers-32
make precision-32
make particles-32
make particle-id-32
make inits-32
make io-32

Then locate your machine specific Makefile, e.g., Make.mach.cuda for a GCC/CUDA example, and then set the following variables:

MACH_CUDACOMPILER = /your/path/to/nvcc
MACH_LIBS_CUDA = -L/your/cuda/install/dir/lib64 -lcudart
# add -I/your/cuda/install/dir/include to the MACH_INCLUDES variable

YOUR_TARGET_ARCH corresponds to the target GPU architecture, e.g.,

MACH_OPT_CUDA = -arch=sm_60 -O3  # for Pascal P100s
MACH_OPT_CUDA = -arch=sm_70 -O3  # for Volta V100s
MACH_OPT_CUDA = -arch=sm_75 -O3  # for Turing cards such as RTX 2080 or Titan RTX

see CUDA doc for details.

How to run with CUDA

The only thing to do is to set UseCUDA=1 in whichever parameter file. That’s all!

Be sure that each node has at least 1 NVIDIA GPU. Also note that each GPU can be running multiple MPI processes, the performance will typically increase with mulitple MPI processes per GPU. So it’s recommended to set the number of MPI processes per node to be the number of CPU cores to fully use both the CPU and GPU resources. Furthermore, it’s recommended to turn on CUDA MPS (Multi-Process Service), which enables concurrent running of multiple MPI processes on the GPU.