Session Presentations

Migration and performance of CMAQ & WRF-CMAQ in the public cloud with commercial images

Arturo Fernandez (afernandez@odyhpc.com)

Organizations interested in migrating their CMAQ & WRF-CMAQ workloads to the public cloud face several challenges including learning a new environment, cost prediction and performance optimization. The latter point is common when it comes to the migration of HPC (High Performance Computing) apps as public CSPs (cloud service providers) did not originally design their infrastructure with HPC in mind. Given these premises, system administrators and end-users have the choice to either prepare their own set-ups or use commercial images optimized for specific environments; these images integrate the app and its dependencies along with pre-processing and post-processing tools.

The presentation will focus on the following aspects:

• Launching CMAQ & WRF-CMAQ with commercial images and scaling up to hundreds of cores
• Performance and scalability analysis
• Preliminary cost assessment

AWS was the first commercial cloud platform and, since then, it has grown to offer multitude of services, which include several tailored to HPC needs. The first part of the presentation will cover how to run CMAQ with images in AWS infrastructure using either single instances or clusters composed by several instances; the latter option allows end-users to scale up CMAQ workloads up to hundreds or even thousands of cores. The presentation will also address the use of cluster monitoring tools.

The second part of the presentation will cover performance and cost analysis. By using the U.S. Southeast test-case prepared by CMAS, it is possible to measure not only performance and cluster scalability, but also to perform comparisons vs. on-premises hardware. The evaluation of CMAQ performance in AWS infrastructure includes measurements with x86_64 and AArch64 (Graviton2) processors. These measurements provide side-by-side comparisons for both architectures as AArch64 processors are emerging as a cheaper alternative to x86_64 cores in the cloud market. The presentation will also make available data with Microsoft Azure infrastructure to provide measurements from a different CSP. Prediction of cloud costs is a non-trivial task that usually requires a TCO (total cost-of-ownership) analysis. A TCO analysis is beyond the scope of the presentation, but it is however possible to perform a cost assessment of computational power, which is the main contributor to overall cost. As CSPs offer a variety of payment options, the cost assessment will consider on-demand vs. reserved vs. spot prices. The cost assessment will also illustrate potential savings when using AArch64 vs. x86_64 infrastructure.

CMAQ inverse modeling, WRF downscaling, and MPAS simulations on the cloud: Challenges and best practices
Matthew Alvarado, Nicholas Heath, Chase Calkins, Rebecca Adams-Selin, John Henderson, Amy McVey, and Elizabeth Bettencourt
Atmospheric and Environmental Research (AER), Lexington, MA

We will highlight three recent efforts to advance our air quality and meteorological modeling on the Amazon Web Services cloud. The first is our containerized CMAQ- and SMOKE-based inverse modeling system that optimizes NH3 emissions with Cross-track Infrared Sounder (CrIS) satellite observations. The second is our development of a cost-effective WRF-based downscaling system to produce high-resolution (4 km) climatologies from global climate model (GCM) output. The third is our on-going efforts to implementat the Model for Prediction Across Scales (MPAS) on AWS, which includes a discussion of the relative advantages and disadvantages of Amazon Batch and ParallelCluster for multi-node processing. Finally, we will discuss the policy and work flow changes we have made over the years to make cloud computing a regular part of our work, including staff training, investment in infrastructure development, resource tagging, and security policies.

Meteorology and Air Quality Modeling Using the AWS Parallel Cluster Cloud Service
Zac Adelman and Tsengel Nergui
Lake Michigan Air Directors Consortium

LADCO developed Weather Research Forecast (WRF) system and Comprehensive Air Quality Modeling System with Extensions (CAMx) modeling platforms on Amazon Web Services (AWS) for recent ozone and regional haze modeling projects. LADCO's motivation for using AWS was to take advantage of the scalable computing, memory, and data storage capabilities of the AWS cloud. LADCO modelers also envisioned that prototyping and deploying regional modeling capabilities on AWS would enable a new way to transfer modeling and high performance computing (HPC) capabilities to their member states. LADCO developed the AWS modeling platforms using the Parallel Cluster system for provisioning HPC resources. The system was developed to use AWS spot instances, which are as much as 75% less expensive than the on-demand AWS instances. This talk will describe the design of the AWS cluster used by LADCO, present performance statistics for relatively large-scale WRF and CAMx simulations, and show an analysis of the costs of the simulations. LADCO successfully used their AWS configuration for modeling to support and ozone attainment demonstration and regional haze state implementation plans (SIPs).

Implementation and Testing of the Community Multiscale Air Quality (CMAQ) model on the Cloud
Elizabeth Adams, Carlie Coats, Christos Efstathiou and Saravanan Arunachalam
UNC Institute for the Environment

Mark Reed and Robert Zelt
UNC Information Technology Services

Containers are lightweight, standalone, executable packages of software that offer a low-overhead method of building software such as the Community Multiscale Air Quality Modeling System (CMAQ) once, and then running across different cloud platforms and hardware. We developed methods to build, install, and run CMAQ as a Singularity container and Docker container. The container method scales well for the 12-km x 12-km Southeast U.S. domain on single node cloud computing environments including Amazon AWS and Microsoft Azure, using up to 96 processors with hyperthreading turned off.

We also tested multi-node high performance computing on the cloud using the Amazon ParallelCluster command line interface and adapted CMAQ install scripts that mimic the container recipes to install and run CMAQ. The Azure Cycle Cloud benchmark (analogous to Amazon ParallelCluster) also showed good scalability using a containerized version of the 12-km x 12-km Continental U.S. (ConUS) domain benchmark case. The method to build the container required installing the software using the CMAS tutorial build instructions and then creating the container with the MPI libraries native to Azure's Cycle Cloud environment.

This presentation will provide a summary of model performance for the benchmark simulations on the two different cloud computing environments for the two different modeling domains using single node as well as multinode configurations. Creating and sharing reproducible workflow methods for installing and building CMAQ for the 12-km x 12-km Southeast U.S. and ConUS benchmark cases using the Parallel Cluster and Azure Cycle Cloud will allow the CMAS community to learn how to provision resources, accurately forecast CMAQ modeling run time and storage requirements and further create reliable cost estimates for performing air quality modeling in the cloud for their individual use cases.