Singularity CMAQ 532 on Atmos

HOST MPI Method

Singularity Container was prepared by Carlie Coats on his machine as root, then packaged to a tar.gz file.

/proj/ie/apps/dogwood/singularity

Uploaded the cmaq.sif.nov24.tar.xz to the google drive and downloaded to atmos.

Dogwood has SLURM as the batch queueing system.

If I try to run cmaq_cctm.csh, I get an error

mpiexec@r1n55 HYDU_create_process (./utils/launch/launch.c:75): execvp error on file srun (No such file or directory)

I checked on the location of srun on atmos:

which srun
/usr/local/bin/srun

One option may be to bind /usr/local/bin I tried that, but it didn't work.

Currently, I can run the openmpi version of the script. I am loading the following modules on atmos.

openmpi-4.0.1/gcc-9.1.0
singularity/3.6

cd /work/MOD3EVAL/eadams02/Singularity/Scripts-BATCH

sbatch cmaq_cctm.csh

I modified cmaq_cctm.csh to use the following:

set HOSTDATA = /work/MOD3EVAL/eadams02/CMAQv5.3.2_Benchmark_2Day_Input
set CONTAINER = /work/MOD3EVAL/eadams02/Singularity/cmaq.sif
set SCRIPTDIR = /work/MOD3EVAL/eadams02/Singularity/Scripts-BATCH

mpirun -np 16 singularity exec \
 --bind ${HOSTDATA}:/opt/CMAQ_532/data \
 --bind ${SCRIPTDIR}/opt/CMAQ_532/logs_hold \
${CONTAINER} /opt/CMAQ_532/scripts/run_cctm.csh

I then modified the run_cctm.csh on line number 732

changed
 (/usr/bin/time -p mpirun -np $NPROCS $(EXEC) ) |& tee buff_${EXECUTION_ID}.txt
to
${EXEC} |& tee buff_$EXECUTION_ID}.txt

When I tried using mpich instead of MPIRUN, I was getting an error

*** ERROR in INIT3/INITLOG3 ***
Error opening log file on unit    99
I/O STATUS = 17
File: CTM_LOG_000.v532_mpich_gcc_2016_12SE1_20160701

It was almost as if it was trying to open this same log file 16 times.

I can't test the mpich properly, as that compiler isn't available on ATMOS.

ATMOS has

intelmpi

and

openmpi

Table of run time

PE Configuration	4x4pe
Wall Time Openmpi	1151 seconds
Wall Time Mvapich	can't test on ATMOS
Wall Time Mpich	can't test on ATMOS