CMAQ version 5.3.1 Compiler Test
Timing Information for CMAQv5.3.1 - Nov 2019
The Community Multiscale Air Quality (CMAQ) modeling system Version 5.3.1 (CMAQv5.3.1) has a planned release date of early December 2019.
Release Testing
The CMAS Center tested the CMAQ release package with the debug and optimized compiler option to compare with the EPA. All tests were conducted with the U.S. EPA SE52 12km domain July 11, 2011 test dataset. The code used in this testing was obtained from EPA's github account, as a pull request to my github account. https://github.com/lizadams/CMAQ/tree/patches2
I ran using 32 processors for the optimized version.
@ NPCOL = 8; @ NPROW = 4
I obtained Fahim's output from atmos and transfered the first day's output to
/proj/ie/proj/CMAS/CMAQ/from_EPA/CMAQv531_2016_12SE1_output
Ran for 2 days, GCC OPT version ran successfully for both days.
NOTE: Obtained the following error on the 2nd day of the run using the INTEL compiler 205250 Nov 26 14:03 CTM_LOG_011.v53_intel_Bench_2016_12SE1_full_CONC_20160702
Processing Day/Time [YYYYDDD:HHMMSS]: 2016184:005500
Which is Equivalent to (UTC): 0:55:00 Saturday, July 2, 2016
Time-Step Length (HHMMSS): 000500
VDIFF completed... 0.2 seconds
COUPLE completed... 0.0 seconds
HADV completed... 0.1 seconds
ZADV completed... 0.0 seconds
HDIFF completed... 0.0 seconds
DECOUPLE completed... 0.0 seconds
PHOT completed... 0.0 seconds
CLDPROC completed... 0.1 seconds
CHEM completed... 0.4 seconds
AERO completed... 0.7 seconds
Master Time Step
Processing completed... 1.6 seconds
MPI_ABORT was invoked on rank 16 in communicator MPI_COMM_WORLD with errorcode 538976288.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.
forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source CCTM_v53.exe 00000000009CB53E Unknown Unknown Unknown libpthread-2.17.s 00002B812638D370 Unknown Unknown Unknown libpthread-2.17.s 00002B812638A213 pthread_spin_lock Unknown Unknown forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source
| Scenario | Compiler | netCDF | I/O API | MPI_YN (#P) | Module | Timing(35 Lay, 220 Var) | Notes | |
|---|---|---|---|---|---|---|---|---|
| Gfortran OpenMPI UNC Debug | Gfortran version 9.1.0 | 4.0.1 | 3.2 | Y (8x4) | openmpi_4.0.1_gcc_9.1.0 | limit stacksize unlimited | ||
| Gfortran OpenMPI EPA Debug | Gfortran version 9.1.0 | 4.0.1 | 3.2 | Y (8x4) | openmpi_4.0.1_gcc_9.1.0 | 3448.4 s/57.47m | limit stacksize unlimited | |
| Gfortran Openmpi EPA O3 OPT | Gfortran version 9.1.0 | 4.0.1 | 3.2 | Y (8x4) | openmpi_4.0.1_gcc_9.1.0 | 692.7/11.5m | limit stacksize unlimited | |
| Gfortran Openmpi UNC O3 OPT | Gfortran version 9.1.0 | 4.0.1 | 3.2 | Y (8x4) | openmpi_4.0.1_gcc_9.1.0 | 766.83s/12.7m | limit stacksize unlimited | |
| Gfortran Openmpi UNC O3 OPT | Gfortran version 9.1.0 | 4.0.1 | 3.2 | Y (4x4) | openmpi_4.0.1_gcc_9.1.0 | 1298.1s/21.6m | limit stacksize unlimited | |
| Intel OpenMPI UNC (OPT) | Intel version 18.2 | 4.0.1 | 3.2 | Y (8x4) | openmpi_3.1.4_intel_18.2 | 533.04s/8.84m | limit stacksize unlimited |
QA plots are generated on longleaf using the interactive queue
R script for comparing model output results from different compilers
- module load r/3.4.3
- setenv R_LIBS /nas/longleaf/home/lizadams/R/x86_64-pc-linux-gnu-library/3.4
- cd /proj/ie/proj/CMAS/CMAQ/CMAQ_v5.3.1_branch2/CMAQ_QA
If you are on the login node, your prompt will contain the login node id:
- [lizadams@longleaf-login2 ~]
Start an interactive queue (don't run on the login node), your environment variables and R module will be set up
- srun -p interact --pty /bin/csh
Your login prompt will be something like
- [lizadams@off03 CMAQ_QA]$
Edit, then Run the R script that compares model output
- R CMD BATCH plot_max_diff_compiler_sens_v5.3beta.R
Plotting the difference in values between the base (EPA debug) and sensitivity (UNC debug) cases at the column, row, layer of the maximum difference between the variables. | Compilation Testing Plots
