WRF-CMAQ v5.3.1 Compiler Test
Latest CMAQv5.3.1 code (CMAQ_REPO_CMAS2way) available from git clone https://github.com/fisidi/CMAQ.git
Public available version git clone -b master https://github.com/USEPA/CMAQ.git CMAQ_REPO
Created instructions for how to build and run WRF4.1.1 https://github.com/lizadams/CMAQ/blob/master/DOCS/Users_Guide/Tutorials/CMAQ-WRF_tutorial_build_gcc.md
Note: above instructions include how to download the latest updates from Carlie for ioapi3.2.
Status of runs.
The Runtime error of the debug compiled standalone CMAQ fails with FPE. /proj/ie/proj/CMAS/WRF-CMAQ/CMAQ_REPO_CMAS2way/openmpi_4.0.1_gcc_9.1.0_debug/CCTM/scripts/slurm-2390504.out mpirun noticed that process rank 7 with PID 0 on node c-206-11 exited on signal 8 (Floating point exception).
The optimized compiled code ran and created output, but then failed with the following message:
>>> WARNING in subroutine SHUT3 <<<
Error closing netCDF file
File name: CTM_CONC_1
netCDF error number -33
*** FATAL ERROR shutting down Models-3 I/O ***
The elapsed time for this simulation was 1337.2 seconds.
I was able to create plots using the output from the optimized version and compare it to Fahim's optimized compile of the CMAQ_REPO_CMAS2way
These show agreement between the two. Fahim provided two sets of outputs, one with ioapi-3.2 older, and one with the latest release. I downloaded and used m3diff, and found no difference between the ACONC file output for the difference between the CCTM_v531 using the CMAQ_REPO_CMAS2way
FILE A: AFILE (./data_04_19_2020/output_CCTM_v531_gcc9.1_Bench_2016_12SE1/CCTM_ACONC_v531_gcc9.1_Bench_2016_12SE1_20160701.nc)
FILE B: BFILE (./data/output_CCTM_v531_gcc9.1_Bench_2016_12SE1/CCTM_ACONC_v531_gcc9.1_Bench_2016_12SE1_20160701.nc)
Date and time 2016183:000000 (0:00:00 July 1, 2016)
A:AFILE/NO2 vs B:BFILE/NO2 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 1.92732E-02@( 31,39, 1) 2.38787E-05@( 94,13, 1) 1.23437E-03 1.58040E-03
B 1.92732E-02@( 31,39, 1) 2.38787E-05@( 94,13, 1) 1.23437E-03 1.58040E-03
A:B 0.00000E+00@( 1, 0, 0) 0.00000E+00@( 1, 0, 0) 0.00000E+00 0.00000E+00
I also found no difference between my run and Fahim's
FILE A: AFILE (data/output_CCTM_v531_gcc9.1_Bench_2016_12SE1/CCTM_ACONC_v531_gcc9.1_Bench_2016_12SE1_20160701.nc)
FILE B: BFILE (/proj/ie/proj/CMAS/WRF-CMAQ/CMAQ_REPO_CMAS2way/openmpi_4.0.1_gcc_9.1.0/data/output_CCTM_v531_gcc_Bench_2016_12SE1_error/CCTM_ACONC_v531_gcc_Bench_2016_12SE1_20160701.nc)
Date and time 2016183:000000 (0:00:00 July 1, 2016)
A:AFILE/NO2 vs B:BFILE/NO2 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 1.92732E-02@( 31,39, 1) 2.38787E-05@( 94,13, 1) 1.23437E-03 1.58040E-03
B 1.92732E-02@( 31,39, 1) 2.38787E-05@( 94,13, 1) 1.23437E-03 1.58040E-03
A:B 0.00000E+00@( 1, 0, 0) 0.00000E+00@( 1, 0, 0) 0.00000E+00 0.00000E+00
So, we are getting matching answers for the optimized CMAQv5.3.1 latest code.
Fahim obtained matching answers for WRF-CMAQ using new and old ioapi-3.2.
FILE A: AFILE (data/output_12km_nf_rrtmg_20_5_1_v411531/CCTM_ACONC_v411531_20160701.nc)
FILE B: BFILE (data_04_19_2020/output_12km_nf_rrtmg_20_5_1_v411531/CCTM_ACONC_v411531_20160701.nc)
Date and time 2016183:000000 (0:00:00 July 1, 2016)
A:AFILE/NO2 vs B:BFILE/NO2 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 2.22875E-02@( 9,37, 1) 2.36311E-05@( 95,13, 1) 1.29369E-03 1.71342E-03
B 2.22875E-02@( 9,37, 1) 2.36311E-05@( 95,13, 1) 1.29369E-03 1.71342E-03
A:B 0.00000E+00@( 1, 0, 0) 0.00000E+00@( 1, 0, 0) 0.00000E+00 0.00000E+00
When I use m3diff to compare my WRF-CMAQ and Fahim's results, I am seeing differences at the first timestep.
FILE A: AFILE (/proj/ie/proj/CMAS/WRF-CMAQ/openmpi_4.0.1_gcc_9.1.0_debug/WRF-4.1.1/output_12km_nf_rrtmg_20_5_1_v41153_2016/CCTM_CONC_v41153_20160701.nc)
FILE B: BFILE (/proj/ie/proj/CMAS/WRF-CMAQ/from_fahim/data/output_12km_nf_rrtmg_20_5_1_v411531_debug/CCTM_CONC_v411531_20160701.nc)
-----------------------------------------------------------
Date and time 2016183:000000 (0:00:00 July 1, 2016)
A:AFILE/O3 vs B:BFILE/O3 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 8.12574E-02@( 35,46, 1) 1.31113E-02@( 53, 4, 1) 3.55808E-02 1.00917E-02
B 8.12574E-02@( 35,46, 1) 1.31113E-02@( 53, 4, 1) 3.55808E-02 1.00917E-02
A:B 0.00000E+00@( 1, 0, 0) 0.00000E+00@( 1, 0, 0) 0.00000E+00 0.00000E+00
Date and time 2016183:010000 (1:00:00 July 1, 2016)
A:AFILE/O3 vs B:BFILE/O3 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 8.16709E-02@( 35,48, 1) 1.18036E-02@( 36,10, 1) 3.20051E-02 8.83139E-03
B 7.84062E-02@( 35,48, 1) 1.11254E-02@( 59,28, 1) 3.18859E-02 8.73194E-03
A:B 4.59358E-03@( 34,48, 1) -1.09005E-02@( 30,41, 1) 1.19151E-04 6.41393E-04
Date and time 2016183:020000 (2:00:00 July 1, 2016)
A:AFILE/O3 vs B:BFILE/O3 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 7.22206E-02@( 35,50, 1) 6.84631E-03@( 59,28, 1) 2.91940E-02 8.49730E-03
B 7.64798E-02@( 35,49, 1) 4.48307E-03@( 59,28, 1) 2.91144E-02 8.45926E-03
A:B 7.65019E-03@( 36,50, 1) -1.17895E-02@( 30,41, 1) 7.96838E-05 6.51846E-04
--
FILE A: AFILE (data/output_12km_nf_rrtmg_20_5_1_v411531/CCTM_ACONC_v411531_20160701.nc)
FILE B: BFILE (/proj/ie/proj/CMAS/WRF-CMAQ/openmpi_4.0.1_gcc_9.1.0/WRF-4.1.1/output_12km_nf_rrtmg_20_5_1_v41153_2016/CCTM_ACONC_v41153_20160701.nc)
Date and time 2016183:000000 (0:00:00 July 1, 2016)
A:AFILE/NO2 vs B:BFILE/NO2 vs (A - B)
MAX @( C, R, L) Min @( C, R, L) Mean Sigma
A 2.22875E-02@( 9,37, 1) 2.36311E-05@( 95,13, 1) 1.29369E-03 1.71342E-03
B 2.22726E-02@( 9,37, 1) 2.36312E-05@( 95,13, 1) 1.29924E-03 1.73567E-03
A:B 1.26635E-03@( 59,28, 1) -5.31193E-03@( 31,39, 1) -5.55327E-06 1.42523E-04
You can see the difference plots in the following link. https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/WRF-CMAQ/fahim_base/plots/compiler_sens/base_from_fahim/layer1_only
I obtained the run scripts that Fahim was using on atmos. With the updated run scripts, I am able to match his results for the debug version, and my optimized version matches Fahim's version.
Here are the updated plots https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/WRF-CMAQ/fahim_debug_base/plots/compiler_sens/base_from_fahim/layer1_only
Turned on the sf option for the code from Fahim and obtained matching answers for the debug version.
Here are the plots https://dataviewer-dept-cempd.cloudapps.unc.edu/index.cfm?back_address=/WRF-CMAQ/wrf_cmaq_sf/plots/compiler_sens/base_from_fahim/layer1_only
Older comparisons:
Plotting the difference in values between the base (EPA gcc_ioapi_3.2_epa) and sensitivity (UNC gcc ioapi_3.2_epa, UNC intel ioapi_3.2_unc, UNC intel ioapi_3.2_epa ) plot of value of each species at the column, row, layer of the maximum difference between the variables.
| Compilation Testing Plots
