The MM5 MCPL() Meteorology-Coupler I/O API Module

This web page is under construction.
As a matter of fact ALL web pages are under construction whether they admit it or not.
--Found on the web page of Do Wong Chu

Best Viewed With Any Browser
These pages are All Browser Enhanced^tm. There are absolutely no <BLINK>ing text, unnecessary and ill-considered <TABLE>s nor <FRAME>s, nor obnoxious <FONT> COLORs, SIZEs, or FACEs at this <CENTER>, nor are there in-line <IMAGE>s that take forever to load.
Please report any problems to the authors, Carlie Coats, coats@emc.mcnc.org, and John McHenry, mchenry@emc.mcnc.org.

Copyright

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License (LGPL) as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the
Free Software Foundation, Inc.,
59 Temple Place, Suite 330,
Boston, MA 02111-1307 USA

The LGPL can also be found at URL http://www.gnu.org/copyleft/lesser.html

MM5 is copyright UCAR 1998. License for MM5 itself is described at http://www.mmm.ucar.edu/mm5/mm5-home.html

Back to Contents

Conceptually, MCPL() is a drop-in MM5 output module designed for coupling MM5 to other environmental models, using the EDSS/Models-3 I/O API. It fits into MM5 as a subroutine called every output time step by the MM5v2 driver or from the SOLVE routine. It can either be called every simulation-hour, as is traditional for MM5 output, or more frequently -- even as frequently as every MM5 advection-step where appropriate for model coupling purposes. It is a netCDF generalization of MM5's OUTTAP() module, and serves as a replacement for most of the functionality of the EPA/NCSC MCIP program(s) used to interface MM5 to air quality models. It also absorbs much INTERP functionality, since it can optionally also produce pressure level interpolated output files. For each MM5 domain, it can optionally produce one or many output grids fitting inside it. These output grids may either be the entire MM5 domain or any proper window into it, and may each have its own output time-step. (NOTE that some output variables -- decoupled wind components, wind speed, divergence, or vorticity -- are only available on proper windows into the MM5 domains (since the first and last horizontal rows and columns fo dot-points don't have PSTAR-values available to do the necessary decoupling). Air-quality boundary files are only available if the output grid is a proper window into the MM5 domain, as well. Some other variables are available only if the corresponding MM5 physics options are chosen.)

MCPL() has a relatively trivial effect upon MM5 performance. Profiling of a 24-hour continental North America 45-kilometer MM5 run on an IBM SP showed a total CPU time of 15,000 seconds, of which 30.5 seconds was spent inside MCPL. Normal run-to-run CPU-time variability on this machine is an order of magnitude larger than the time spent in MCPL, in fact.

When the MM5 Kain-Fritsch cloud package is active, MCPL also can produce (heretofore-nonstandard) files which store the starting conditions for Kain-Fritsch convective cloud events. Data from these files are then used (for example) by air quality models to reproduce exactly the scheduling and behavior of such convective events. This package is not part of the standard release; please contact the MCPL authors for access to it.

For each MM5 domain, the user may select one or more output window grids. For each requested output window, MCPL produces any combination of the following twelve time-independent and time-stepped output files (which are EDSS/Models-3 I/O API selective direct-access files, rather than being sequential Fortran binary (unformatted) files as are "normal" MM5 outputs). For the time-stepped output files, each output window has its own starting date, starting time, and time step (with default values taken from the starting date, starting time, and TAPFREQ of the MM5 run).

The file list for both the "standard" and the non-standard (KF convective cloud event) MCPL() output files is:

GRID_BDY_2D
2-dimensional time-independent grid-boundary variables
GRID_BDY_3D
3-dimensional time-independent grid-boundary variables on MM5 sigma-surfaces
GRID_CRO_2D
2-dimensional time-independent cross-point grid variables
GRID_CRO_3D
3-dimensional time-independent cross-point grid variables on MM5 sigma-surfaces
GRID_DOT_2D
2-dimensional time-independent dot-point grid variables
MET_BDY_2D
2-dimensional grid-boundary meteorology variables
MET_BDY_3D
3-dimensional grid-boundary meteorology variables on MM5 sigma-surfaces
MET_CRO_2D
2-dimensional cross-point grid meteorology variables
MET_CRO_3D
3-dimensional cross-point grid meteorology variables on MM5 sigma-surfaces
MET_DOT_3D
3-dimensional dot-point grid meteorology variables on MM5 sigma-surfaces
PRES_CRO_3D
3-dimensional cross-point grid meteorology variables on standard pressure levels
PRES_DOT_3D
3-dimensional dot-point grid meteorology variables on standard pressure levels
MET_KF_1D
cross-point scalar meteorology variables at the inception of Kain-Fritsch convective cloud events.
MET_KF_3D
cross-point vertical profile meteorology variables at the inception of Kain-Fritsch convective cloud events.

MCPL gives the user the choice of which variables to write to each of the standard files, out of a potential selection which includes the "standard" MCPL-output variables for these files, the lists of ADDR1 and ADDR2 MM5-internal variables, and a selection of other variables used by, e.g., the TOPLATS hydrology model or the MAQSIP or Models-3 air quality model.

The MM5 MCPL/IO API Module also produces IO API-style grid description files, if these do not yet exist. These are sequential formatted (ASCII) Fortran files containong grid and coordinate system descriptions in EDSS/Models-3 IO API-compatible terms.

Module operation is as follows: First, this module gets the output grid names from environment variable OUTGNAMES (where grid name "NONE" means that output for the grid is turned off); then definitions for the AQM cross and dot point grids either from environment variables or the I/O API GRIDDESC file, checks the defining parameters for the AQM dot-point grid, and computes the parameters for the boundary structure. If the GRIDDESC file does not yet exist, it constructs it on the basis of user-supplied grid-descriptive inputs. It also gets default starting date, starting time, time step, and duration from MM5.

Then, for each output grid, it gets the starting date, starting time, and time step fron environment variables, constructs headers for the output files requested by the user, and opens them. The variables-list for each of these files is selected by the user from the complete set of potential output variables for that file, and communicated to the routine by way of a text file containing the requested-variables list for that file, formatted with one variable-name per line.

For each relevant time step, MCPL() copies/computes/re-orders the relevant variables from MM5 internal data structures into I/O API storage order in an internal buffer, and writes the result to the appropriate output file. Additionally if the MCPL-KF module is activated, it detects the inception of each Kain-Fritsch convective cloud event, and writes the starting date, time, duration and starting meteorology data for that cloud event to the KF-files for all the appropriate output grids.

Back to Contents

Interface to MM5

Compilation Flags

For some systems (Sun f90, recent SGI compilers, etc.), you will need to add additional qualifiers to CPPFLAGS in configure.user, in order for the compiler to process some of the following flags correctly.

For MM5V3, you should add the preprocessor definition -DMM5_V3=1 (or its Cray or IBM SP equivalent) to the MM5/mcpl/Makefile compile line for mcpl.F, defining the cpp symbol MM5_V3 so that MCPL will be able to access MM5V3 date conventions and file header data structures instead of the previous versions.

For the Benjamin-Miller sea level pressure algorithm, you should add the preprocessor definition -DMCPL_BMP=1 (or equivalent) to the MM5/mcpl/Makefile compile line for mcpl.F.

For the Models-3 air quality extensions, you should add the preprocessor definition -DMCPL_ADDMDL3=1 (or equivalent) to the MM5/mcpl/Makefile compile line for mcpl.F, as well as adding the so-called "EPA mods" to the relevant land surface and radiation packages.

For OpenMP task-parallel MCPL operation (probably not necessary unless you're running either on a system with lots of processors or unless you're generating output every advection step), you should add the preprocessor definition -DMCPL_OMP=1 (or equivalent) to the MM5/mcpl/Makefile compile line for mcpl.F.
NOTE: current versions of IBM's xlf have an incomplete OpenMP implementation that does not recognize the OpenMP SHARED directive for objects which are COMMONs, so this task-parallel operation should not be turned on for such systems. (As of 12/20/99, the upcoming next release of IBM's compiler system promises such support.)

MM5 Source Code Modifications

The MM5v2 MCPL module has two entries, the initialization entry MCPL_GRID which should be called after coarse-grid initialization in the MM5 main program, and the MCPL_OUT(NEST) output entry, which is most easily called at the end of SOLVE3(), (allowing high-frequency output if desired, e.g., every advection step). MM5's driver.F should be terminated by a call to the I/O API shutdown routine M3EXIT() instead of a STOP statement, in order to flush all netCDF file headers to disk, etc.: ... C STOP 99999 MM5.237 CALL M3EXIT('MM5', 0, 0, 'Normal completion', 0 ) ...
Note that if you want to analyze the output while MM5 is still running, your script must declare the output files to be volatile: setenv <logical name> "<physical path spec> -v"

MCPL() is called through two or three ENTRY points, according to whether Kain-Fritsch convective cloud-event data is requested:

MCPL_GRID
has no arguments, must be called first, and sets up the grid geometry for subsequent calls.
MCPL_OUT( NEST )
creates and writes the output files for all output grids of the specified MM5 nest domain at the current time step.
MCPL_KF2( & NEST, II, JJ, ! what cell? & P300KF, TCKF, RKF, ICEKF, TADVEC, NIC, CLDTP, ! scalar output variables & TKF0, QKF0, UKF0, VKF0, WAV0, PKF0, & DZQKF, DPKF ) ! vertical column output variables
writes the specified cloud-event data to the MET_KF_1D and MET_KF_3D output files for all output grids at the current time step. Should be called at the end of KFPARA; may be called in parallel if the compiler is OpenMP-compliant.

MCPL uses MDATE from common block /VARIOUS/ to determine the default starting date and time, TAPFRQ from /PARAM2/ to determine the default output time step, and XTIME from /VARIOUS/ to determine the current simulation date and time. Note that it uses environment variable MCPL_START_DATE, if available, or the current wall clock (if not) to determine the century component for the EDSS/Models-3 dates and times it uses internally. Consequently, MCPL itself is Y2K-compliant if used with user-specified MCPL_START_DATE, although MM5v2 itself is not. Because of this, there will be problems with default dates when MM5 is used for historical-case runs not for the current century, a difficulty which may be avoided by always explicitly specifying MCPL_START_DATE.

On MCPL Internals

MCPL() uses a variety of worker routines to extract the requested variables from MM5v2 common blocks (particularly /ADDR1/ for 3-D variables, and /ADDR2/ for 2-D variables), decouple them (MM5 keeps most 3-D variables in the form PSTAR * <variable>) and perform units conversions where appropriate, and re-order them into standard EDSS/Models-3 I/O API column-row-layer subscript order.

How to modify MCPL to output additional variables using the existing worker routines (or how to create new worker routines in order to output additional derived variables) is described in the section, "Adding New Output Variables to MCPL()"

Back to Contents

MCPL() Output Files and Their Variables

Back to Contents

Run-Script Control of the Module

Back to Contents

Restrictions and Limitations

NONHYDROSTATIC MM5!!

DOES NOT WORK WITH DISTRIBUTED-MEMORY MM5!!
We have looked at the software engineering task of making MCPL work with the distributed-memory version of MM5, but the task of doing a clean implementation of the necessary "gathers" of MM5 data from all of the machines where the data has been distributed into buffers from which MCPL can perform its calculations and write out the results looks to be exceedingly complex. If you would like this capability, please be prepared to come to us with at least two man-years' funding.

OpenMP on SGI: When running with OpenMP parallelism under SGI IRIX6, the default per-thread stack and memory limits are inappropriately tiny. This difficulty may be controlled by setting appropriate values for environment variables that control this behavior. The following values work for us:

    MP_SLAVE_STACKSIZE=32000000 >or other reasonable limit<
    MP_STACK_OVERFLOW=ON

MM5V2, Historical Simulations, and Y2K-Related Issues If your script sets all of the date-related environment variables for MCPL() explicitly, you should not encounter any Y2K-related problems. However, with MM5V2 (which is itself not Y2K-compliant) if you use default MCPL starting dates (i.e., you have not done a setenv MCPL_SDATE <date-list, ordered by output window>), MCPL attempts to fill in the century field by using the century value obtained from the current real-time wall clock/calendar. With MM5V2, the default behavior leads to errors when the simulation century and the wall-clock century are not the same. For example, if on March 4, 2000, one is doing a historical MM5V2 simulation for July 17, 1995, the disparity between the run-time wall clock century and the simulation century leads MCPL in MM5V2 to think that the requested output dates will be for year 2095 (= century 2000 from the wall-clock + year 95 from MM5V2's MDATE).

For historical cases before 1 AD, there may potentially be problems; the system has not been tested thoroughly for these, although I believe it will behave correctly (Fortran integer division involving negative numbers "does the right thing", unlike C division which is undefined and allowed by the C standard to vary from vendor to vendor).

No MPI/MPP. MCPL follows a single-output-file model rather than a distributed-I/O model. It therefore requires that the variables being output are already gathered into the global arrays pointed to by the pointers in MM5 COMMONs, and that (for KF cloud-event data) the MCPL_KF sub-module be able to receive all of the KF event data and serialize it by means of OpenMP critical sections. Note that since MCPL was first developed for use in coupling MM5 with other environmental models, its structure is in part dictated by the requirements of those models. These other models may not be assumed to have exactly the same data decomposition as MM5 (or even to live on the same machine(s); it is perfectly reasonable to build a real-time forecast system coupling an MM5 running in RTP,NC with a hydrology model running at GA Tech and an air quality model running at UT-Austin).

No Moving Nests. This is an EDSS/Models-3 I/O API requirement, since each file's grid definition is active for the entire life of the file. Workaround: each domain instantiation should be treated as a distinct domain in MM5, and have its own set of output files, which have starting dates and times matching the time at which the domain is activated. Note that MCPL already turns off output for a domain at the time MM5 turns the domain off.

Now completed: Early versions may not have been tested for all potential output variables, or all output files. Variables needed by MAQSIP, SMOKE, and TOPLATS have been validated. Pressure-level files are currently undergoing test and validation.

If MCPL_KF is used, and MM5 is run in parallel, you must use the thread-safe version of the EDSS/Models-3 I/O API library. This is also true if you use the (currently-being-tested) multithreaded version of MCPL().

Back to Contents

MCPL() Module Design Internals

Back to Contents

Adding New Output Variables to MCPL()

Back to Contents

MCPL() Future Plans and Ongoing Extensions

Back to Contents

Download MCPL

NOTICE:

NOTE to Internet Explorer users: IE seems to be brain-damaged in some fashion, and refuses to accept an FTP-URL to the current working directory. Try using an alternate browser such as Netscape for downloads from this page.

The version referenced below is current as of 1/22/2001 includes minor bug-fixes of the code related to grid descriptions, an improved relative humidity calculation which does a linear transition between ice-based relative humidity formulation below 248.16 K and a water-based humidity formulation above 268.16nbsp;K.

The version referenced below is current as of 3/21/2000 includes minor bug-fixes, provision for using the Benjamin-Miller sea level pressure algorithm instead of the INTERP algorithm, and provision for putting out wind components (UX,VX) interpolated to cross-points to files MC2 and MC3.

The version referenced below is current as of 10/18/99 includes minor bug-fixes, as well as adaptations for MM5V3, which may be activated by adding a CPP-definition for symbol MM5_V3 to the compilation-flags. As of the current date, we believe the MM5V3 mods work correctly (they are relatively simple, and involve changes to treatment of date-&-time variables, and grid definitions taken from the new file header structure), but they have not been tested, as we at NCSC are not yet running MM5V3 in production mode. With luck, that will happen during January 2000.

The version current as of 7/29/99 comments out IMPLICIT NONE in mcpl.F and includes re-fixes for OpenMP parallelism and for MAQSIP cloud-type information.

The version referenced below is current as of 7/29/99. It comments out IMPLICIT NONE in mcpl.F and includes re-fixes for OpenMP parallelism and for MAQSIP cloud-type information.

Versions prior to 7/26/99 still had problems when compiled with OpenMP parallelism turned on. There was also a bug in file-header description of cloud parameterization, when MCPL was run with more than one output window per MM5-domain.

Versions prior to 7/21/99 had problems when compiled with OpenMP parallelism turned on.

This version includes a current snapshot of the EDSS/Models3 I/O API. A new release is expected in September, 1999.

This version is for MM5v2.x. We will re-do for MM5v3 as soon as we can.

If you get error messages about undeclared variables: remove the IMPLICIT NONE at the top of mcpl.F: I have had so much trouble dealing with IMPLICIT source codes that I refuse to deal with them any more. I've written versions of the MM5 include-files which declare everything they define and I use them for my own private development. Then I sometimes forget to delete the IMPLICIT NONE when I prepare a new release.

You may download MCPL as a gzipped tar file from here. This package will include the following:

INSTALL_MCPL.txt: which contains the installation instructions
include/addmdl3.incl: with data structures for use in computing output variables used by EPA's CMAQ and NCSC's MAQSIP air quality models.
ioapi: directory containing the source code for the EDSS/Models-3 I/O API
ioapi_tools: directory containing the source code for various I/O API-related analysis programs
mcpl: directory containing the MCPL source code, Makefile, and resource files.
mcpl_env_file: csh-script for setting MCPL-related environment variables.
mcpl_new_routines: directory containing source code of MM5v2 routines modified to work with MCPL, as well as a MCPL-adapted configure.user file. For a basic install, only the SOLVE3 mods are required; the other routines are needed for an ADDMDL3-install.

Note that this released version does not include the KF package nor the modifications for two-way model coupling (which may vary from application to application, depending upon how many data-dependency loops need to be closed up). Users desiring versions with these (more-complex-to-install) capabilities should contact the authors. Due to Microsoft Internet Explorer's mis-implementation of the HTTP standard, you may need to use a more-compliant browser such as Opera, Netscape, or Lynx in order to download MCPL successfully.

In addition to the MCPL source, you will also need the EDSS/Models3 I/O API, which is contained downloadable tar-file described above, as well as from here, and the netCDF library from UCAR, available as a UNIX-compressed tar file from the netCDF home page.

Use of I/O coupling mode with MCPL also requires version 3.4 or later of the PVM library, from Oak Ridge National Laboratories available from the PVM home page .

Back to Contents

Acknowledgements and Disclaimer

Although the research described in this article has been funded wholly or in part by the United States Environmental Protection Agency through grants R825210 and R825211 to MCNC, it has not been subjected to the Agency's required peer and policy review and therefore does not necessarily reflect the views of the Agency and no official endorsement should be inferred.

The authors would also like to thank Dr. Nelson Seaman, Annette Gibbs-Lario, and the Department of Meteorology at the Pennsylvania State University for all their work beta-testing this software during the course of our joint real-time air quality forecasting project, and for all their help and support in developing MCPL.

Back to Contents

Send comments to

Carlie J. Coats, Jr.
coats@emc.mcnc.org