3. Getting your data into PAVE

PAVE currently supports four input file formats: IO/API (netCDF), UAM-IV, UAM-V, and "chain files" (which contain a list files of one of the other three types). Files of these types are loaded into PAVE from the Add/Delete/Select Dataset_popup window. This window comes up automatically when you start PAVE, and also appears when you choose Edit/Select From Dataset List from PAVE's Datasets menu. See Quick PAVE Jumpstart for more details on loading datasets.

IO/API Format

The Models-3/ EDSS Input/Output Applications Programming Interface (see http://www.baronams.com/products/ioapi for further info) provides an easy-to-learn, easy-to-use programming interface to files for the air quality model developer and model-related-tool developer, in both FORTRAN and C/C++. I/O API files are portable across computing platforms. This means that the same file can be read on a Sun workstation, a DEC Alpha workstation, and on a Cray supercomputer. If your data is not already in IO/API, UAM-IV, or UAM-V format, we recommend that you write a translation program to convert your data into IO/API format.

There is source code for three example conversion programs is included with the PAVE distribution. One of them converts an ASCII ROM dataset with a Lat/Lon map projection, and the other converts a binary SAQM dataset with a Lambert Conformal map projection, the third converts the standard AIRS AMP350 observational data format to the Models-3 I/O API format that PAVE requires. Note that you will need additional libraries, and possibly header files, in order to successfully compile this code. These libraries and header files can be found in:

<top level PAVE dir>/<platform type>/lib/*
The example conversion code is in:

<top level PAVE dir>/doc/pave/convert_to_m3ioapi/rom_to_m3ioapi/*
<top level PAVE dir>/doc/pave/convert_to_m3ioapi/saqm_to_m3ioapi/*
The AIRS AMP350 convertor was first provided with the PAVE 2.1 release. It can be downloaded from the web site.
http://www.ie.unc.edu/cempd/EDSS/updates/updates.html This program requires the following inputs.
  • The input AIRS amp350 print format file name.
  • The time zone conversion file. (provided with the obs2api program - tzt.dat)
  • Additional hour shift variable. The AIRS data are hourly averaged, and a 00 time flag represents the hour 00-01. A user may wish to represent that data segment by the ending hour. In that case, a 1 should be entered here.
  • Starting year, month, day, hour (GMT) (e.g. 1997 07 10 12)
  • Ending year, month, day, hour (GMT) (e.g. 1997 07 16 12)
  • Name of output variable (8 characters max) (e.g. O3_OBS)

    NOTE: these example codes will probably need to be adapted for use with your data files. They were written for specific data files being used for testing purposes, and were NOT originally intended to be cleanly written examples of how to easily convert generic gridded data data into PAVE format. However, they are being included here with the hope that you may find them useful.

    One sometimes confusing concept is the precise meaning of the XORIG, YORIG, XCELL, and YCELL parameters in an IO/API data file. Below is an image of an example of using these parameters with two different data files that need to be co-registered within a PAVE plot. Note that units are in meters.

    dot-vs-cross

    UAM-IV Format

    Most geographically oriented UAM-IV input and output files are directly readable by PAVE. These include DIFFBREAK, REGIONTOP, TEMPERATURE, WIND, AIRQUALITY, BOUNDARY, TOPCONC, EMISSIONS, PTSOURCE, AVERAGE, and INSTANT.

    UAM-V Format

    Some of the UAM-V input and output files are compatible with the UAM-IV file formats. These files are readily visualized by PAVE because it can obtain all the information it needs from inside the files (i.e. they are "self-describing"). The files in this category are: EMISSIONS, PTSOURCE, BOUNDARY, AIRQUALITY, and the coarse grid AVERAGE and INSTANT files.

    However, many UAM-V files have new formats that are not self-describing. PAVE needs additional information in order to read these files, such as the kind of data in the file, the number of rows, columns, and layers in the data, the geographic region covered, etc. The file types that fall into this category are wind, temperature, cloud, water vapor, rain, vertical diffusion, height, fine grid average, and fine grid instant.

    In order to display a file of one of these latter types with PAVE, a UAM-V "metafile" must be used. A metafile is an ASCII file that contains the additional information PAVE needs to read and correctly interpret the data. To visualize data in a file that needs a metafile, select the metafile from the PAVE file browser, instead of the file that contains the data. A description of the contents of UAM-V metafiles follows.

    The very first line of the metafile MUST be the following:

     #! UAMV DESCRIPTION FILE 

    If PAVE does not find the above string, the file is considered to be of unknown type and an error is returned.

    Several keywords must be present in the metafile. Each keyword should be on a separate line and followed by a value. Blank lines are permitted. The following keywords are required (and are listed in the recommended order):

    UAMV_FILE
    UAMV_TYPE
    NCOLS
    NROWS
    LEVELS
    XORG
    YORG
    DX or DLON
    DY or DLAT
    UTM_ZONE (for UTM-based domains only)
    FINE_GRID  (for UAM-V output files only - not for input meteorology files)
    TITLE
    

    Keyword Descriptions

    UAMV_FILE - the name of the file that contains the actual data to be displayed by PAVE. It can be either a full or a relative pathname to the current working directory.

    UAMV_TYPE - the type of data in the UAMV_FILE. Valid choices are: Wind, Temp, Cloud, H2O, Rain, Vdif, Height, FineGridAverage, FineGridInstant

    NCOLS - number of horizontal columns in the grid

    NROWS - number of horizontal rows in the grid

    LEVELS - number of vertical layers in the grid

    XORG - the x-coordinate of the lower left corner of the grid, in km for UTM-based grids and degrees for latitude-longitude grids

    YORG - the y-coordinate of the lower left corner of the grid, in km for UTM-based grids and degrees for latitude-longitude grids

    DX - size of the horizontal cell in the x-direction for UTM-based grids (km)

    DLON - size of the horizontal cell in x-direction for lat-lon grids (degrees)

    DY - size of the horizontal cell in y-direction for UTM-based grids (km)

    DLAT - size of the horizontal cell in y-direction for lat-lon grids (degrees)

    FINE_GRID - logical variable. Possible values: 0 and 1. If the value is 1, the file represents data on a fine grid, otherwise it is on a coarse grid. For meteorological input files, this value should be set to one only if there is a layer of cells outside the domain specified by { XORG, YORG, NROWS, NCOLS, DX, and DY]}.

    TITLE - A title for tile plots. Spaces are allowed.

    Example of a UAM-V temperature metafile on the OTAG coarse grid:

    #! UAMV DESCRIPTION FILE
    UAMV_FILE /home/user/tmpr.cc.20jul93.ld.rams1
    UAMV_TYPE  Temp
    NCOLS      64
    NROWS      63
    LEVELS      5
    XORG       -99.0
    YORG       26.0
    DLON       0.5
    DLAT       0.3333333
    FINE_GRID  0
    TITLE      Coarse Grid Temperature: July 20, 1993
    
    Example of a UAM-V fine grid average metafile on the OTAG fine grid:

    #! UAMV DESCRIPTION FILE
    UAMV_FILE /home/trayanov/testpave/avrg.ff.20jul93-93.mc.93basA1
    UAMV_TYPE  FineGridAverage
    NCOLS      137
    NROWS      110
    LEVELS     7
    XORG       -92.0
    YORG       32.0
    DLON       0.16666667
    DLAT       0.11111111
    FINE_GRID  1
    TITLE      Base Case Fine Grid Average
    
    Example of a UAM-V temperature file on a UTM-based grid:

    #! UAMV DESCRIPTION FILE
    UAMV_FILE /metdata/uamv.t.jun26-28.16km.8.v4
    UAMV_TYPE  Temp
    NCOLS      35
    NROWS      50
    LEVELS     8
    XORG       200.0
    YORG       4400.0
    DX         16.0
    DY         16.0
    UTM_ZONE   16
    FINE_GRID  0
    TITLE      Coarse Grid Temperature File
    
    Creating metafiles with scripts

    It is relatively straightforward to use a script to create metafiles. Examples of scripts that do this follow. However, the specifics (such as parsing file names to determine file type) depend on the particular application. The two examples given below are for the OTAG project. Note that these are hypertext links which are only available if you are reading this document using a Web browser.

    Script to create OTAG input metafiles

    Script to create OTAG output metafiles

    The above scripts are provided with your PAVE installation. They are called make_in_metas and make_out_metas and are located in the scripts/ subdirectory of the top level directory where PAVE is installed. Before using them, be sure to edit the upper portion of the scripts to match your episode and directory structure.

    Viewing multiple sequential files as one dataset using chain files

    Data from multiple files can be concatenated and displayed as if they were from a single file. This is useful if you wish to animate over sequential data that is stored in multiple files. In order to do this you need to supply a "chain file" - an ASCII file that contains a list of the data files to be concatenated.

    The first line of a chain file must be

    #! LIST_OF_CHAINED_FILES

    If this is not found, PAVE will not recognize that the file is a chain file. A list of full path names to the individual files to be concatenated should follow this line with one filename per line. Note that no blank lines or comment lines are permitted and the files MUST be in the order in which you wish the data to be displayed. The individual files can be one of the following types: netCDF, UAM-IV or UAM-V (regular or meta files).

    A script is provided with PAVE to simplify the creation of chain files. The script will currently work only in cases where the files to be chained reside in a single directory. To create a chain file, cd to the directory that contains the files you wish to chain, and type

     chain_files name_of_chain_file list_of_files_to_chain
    For example:

     chain_files avrg.cc.20-30jul93-93.mc.basB avrg.cc.* 

    Limitations: The current version of PAVE does not check the files for consistency (i.e. whether they are of the same type, whether the grid is the same in all the files, or whether the files are listed in the proper sequential order).

    Here is an example of a chain file:

    #! LIST_OF_CHAINED_FILES
    /home/trayanov/testpave/rain.cc.20jul93.ai.meta
    /home/trayanov/testpave/rain.cc.21jul93.ai.meta
    /home/trayanov/testpave/rain.cc.22jul93.ai.meta
    
    Next Chapter: Using formulas

    Return To Table of Contents