AMET Notes

AMET1.1

Last Update: August 30, 2012

These notes document the installation of AMET1.1 on a clean Ubuntu 10.04 Linux server. The notes cover the installation of all necessary components required to install and run AMET.

There is a series of packages and libraries that you need to install for AMET. Instructions on how the Ubuntu server was configured are below. The most important thing about getting AMET to work in a new installation is to maintain consistency with the version numbers of the Tier 2 software and libraries listed in the installation guide. I didn't follow these very closely in my installation and a few of them came back to bite me. Notes below.

Perl

Ubuntu 10 installed with Perl 5.10. For whatever reason, Perl syntax changed from Perl 9 to 10 breaking the AMET1.1 scripts. You have downgrade to Perl 5.8.9 to use AMET1.1.

cd /home; mkdir perl; cd perl
wget http://www.cpan.org/src/5.0/perl-5.8.9.tar.gz
tar xvzf perl-5.8.9.tar.gz
cd perl-5.8.9
./Configure -des -Dprefix=/home/perl
make
make test
make install

Then you need to add the perl libraries required by AMET1.1. The key to these installation/downgrade instructions is that you must specify your downgraded version of perl (5.8.0) so that the libraries get installed in the right location on your system. I used perl 5.8.9 and it seemed to work fine.

/home/perl/perl-5.8.9/bin/perl -MCPAN -e shell
cpan> install PDL
cpan> install PDL::NetCDF
cpan> install DBI
cpan> install Mysql
cpan> install Mail::Sendmaiul
cpan> install Date::Calc
cpan> install Sort::Fields

Note that I actually needed to install PDL from source because the make test was crashing due to some unnecessary components not being built correctly. This build looked something like this:

cd /home; mkdir PDL; cd PDL 
wget http://sourceforge.net/projects/pdl/files/PDL/2.4.11/PDL-2.4.11.tar.gz
tar xvzf PDL-2.4.11.tar.gz
cd PDL-2.4.11
/home/perl/perl-5.8.9/bin/perl Makefile.PL
make
make test
make install 

R

Install R from apt-get

sudo apt-get install r-base-core

The default version of R that was installed was version 2.15.0. Apparently there is syntax in AMET1.1 R scripts that doesn't work with new versions of R. I had to downgrade to R 2.14.0.

cd /home;mkdir R; cd R
wget http://cran.r-project.org/src/base/R-2/R-2.14.0.tar.gz
tar xvzf R-2.14.0.tar.gz
cd R-2.14.0
./configure --prefix=/home/R --with-X=no
make
make test
make install

Install R libraries from source

sudo R 
> install.packages(c("RMySQL", "maps", "mapdata","date","reshape", "rgdal", "sp", "maptools", "xtable", "yaml",
"ncdf", "chron", "DBI", "akima"))

MADIS API

I needed to downgrade to MADIS 3.6. MADIS3.9 definitely did not work. I couldn't find the MADIS3.6 source code out on the web, I had an archive of the code sitting out on UNC's system that I ported over to the new Ubuntu server. The URL for downloading the MADIS API is here http://madis.noaa.gov/madis_api.html.

MySQL

Install MySQL from apt-get

sudo apt-get install mysql-client-5.1 mysql-server-5.1 python-mysqldb

GDAL

Install gdal libraries

sudo apt-get install libgdal1-1.6.0 libgdal1-dev gdal-bin proj-bin libproj-dev

Python

Install python libraries from apt-get

sudo apt-get install python-pyproj python-mysqldb python-numpy python-scipy python-yaml python-argparse python-dev

Texlive

install texlive from apt-get

sudo apt-get install texlive

Setting up MySQL for AMET

Once you have MySQL and the accompanying libraries installed you need to start the server and create the ametsecure user/password.

Start the server

mysqld_safe &

Log into the server and create the user ametsecure; assign a password to ametsecure and give the user full privileges to the database

mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 25338 to server version: 5.5.29-standard
Type 'help;' or '\h' for help. Type '\c' to clear the buffer. 
mysql> CREATE USER 'ametsecure'@'localhost' IDENTIFIED BY 'some_pass';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ametsecure'@'localhost' WITH GRANT OPTION;
mysql> \q

After you create this user, edit the amet-config.pl and amet-conf.R scripts under $AMET_HOME/configure and add the password you created. You'll set $root_pass in the *.pl script and passwd in the *.R script to the ametsecure password that you created.

AMET1.2

Last Update: May 24, 2017

These notes document the installation of AMET1.3 on a clean Red Hat Enterprise Linux server with and without root. The notes cover the installation of all necessary components required to install and run AMET.

Installation without root

Assume the installation directory is /home/AMET

R

If starting from scratch with later versions of R, you'll need to install xz utils, PCRE 8.40, zlib, bzip2, and curl before R would build. Example from the UNC Kill Devil cluster:

wget --no-check-certificate https://www.zlib.net/zlib-1.2.11.tar.gz
tar xvzf zlib-1.2.11.tar.gz
cd zlib-1.2.11; mkdir local
configure --prefix=$cwd/local
make
make install

wget http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xvzf bzip2-1.0.6.tar.gz
cd bzip2-1.0.6; mkdir local
make
make install

wget --no-check-certificate https://tukaani.org/xz/xz-5.2.3.tar.gz
tar xvzf xz-5.2.3.tar.gz
cd xz-5.2.3; mkdir local
configure --prefix=$cwd/local
make
make install

wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre2-10.21.tar.gz
tar xvzf pcre2-10.21.tar.gz
cd pcre2-10.21; mkdir local
configure --prefix=$cwd/local --enable-utf --enable-unicode-properties --enable-jit --disable-cpp
make
make install

Manually download the curl library
tar xvf curl-7.54.0.tar
cd curl-7.54.0; mkdir local
configure --prefix=$cwd/local
make
make install

Then set up the environment with these new packages for building R:

setenv CFLAGS "-I/proj/ie/apps/AMET/AMET_v13/Testing/zlib-1.2.11/local/include -I/proj/ie/apps/AMET/AMET_v13/Testing/bzip2-1.0.6/local/include -I/proj/ie/apps/AMET/AMET_v13/Testing/xz-5.2.3/local/include -I/proj/ie/apps/AMET/AMET_v13/Testing/pcre-8.40/local/include -I/proj/ie/apps/AMET/AMET_v13/Testing/curl-7.54.0/include"
setenv LDFLAGS "-L/proj/ie/apps/AMET/AMET_v13/Testing/zlib-1.2.11/local/lib -L/proj/ie/apps/AMET/AMET_v13/Testing/bzip2-1.0.6/local/lib -L/proj/ie/apps/AMET/AMET_v13/Testing/xz-5.2.3/local/lib -L/proj/ie/apps/AMET/AMET_v13/Testing/pcre-8.40/local/lib -L/proj/ie/apps/AMET/AMET_v13/Testing/curl-7.54.0/lib"

Install R:

wget https://cran.r-project.org/src/base/R-3/R-3.4.0.tar.gz
tar xvzf R-3.4.0.tar.gz
cd R-3.4.0
./configure --prefix=/home/R
make
make test
make install

Install R libraries from source

cd /home/AMET/R-3.4.0/local/bin
R
> install.packages(c("RMySQL", "date", "maps", "mapdata", "stats", "plotrix", "fields"))

MADIS API

I needed to downgrade to MADIS 3.6. MADIS3.9 definitely did not work. I couldn't find the MADIS3.6 source code out on the web, I had an archive of the code sitting out on UNC's system that I ported over to the new Ubuntu server. The URL for downloading the MADIS API is here http://madis.noaa.gov/madis_api.html.

MySQL

Install MySQL from apt-get

sudo apt-get install mysql-client-5.1 mysql-server-5.1 python-mysqldb

Once you have MySQL and the accompanying libraries installed you need to start the server and create the ametsecure user/password.

Start the server

mysqld_safe &

Log into the server and create the user ametsecure; assign a password to ametsecure and give the user full privileges to the database

mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 25338 to server version: 5.5.29-standard
Type 'help;' or '\h' for help. Type '\c' to clear the buffer. 
mysql> CREATE USER 'ametsecure'@'localhost' IDENTIFIED BY 'some_pass';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ametsecure'@'localhost' WITH GRANT OPTION;
mysql> \q

After you create this user, edit the amet-conf.R scripts under $AMET_HOME/configure and add the password you created. You'll set 'passwd in the *.R script to the ametsecure password that you created.

AMET2

Last Update: July 22, 2013

Prepared by Georgia Tech, July 2013. Please contact Yongtao Hu at yh29@mail.gatech.edu for any questions.

Here described are installation notes for amet2 based on a successful installation at Georgia Tech on a RedHat Linux x86_64 system with the kernel of 2.6.18-348.4.1.el5. Root privilege is recommended throughout the installation (and for creating users on the mysql server after installation). In brief, amet2 relies on mysql, python (v2.6 or later versions) and R, and a lot of python and R modules/packages. Many required python and R modules/packages further rely on external libraries. Here all softwares are installed from source codes (consult with your administrator if the auto installation, for example through the “package manager apt” on Ubuntu, is easier on your system). Uncompress the source code package first (with exception of R packages), then go to the source code main directory to start the installation.

(1) Check the versions of existing glibc, mysql, R, python etc. on the system.

   On our system the following versions led to a successful installation of amet2:
   glibc-2.5-107.el5_9.4 (gcc version 4.1.2) 
   mysql v14.12 distribution 5.0.77 (using “yum” can easily upgrade mysql to a later version, for example 5.0.95 on our system, but it is still far from the latest version; 
   we have tried to install 5.6.10 from source code but failed.)
   We had to upgrade R and python.

(2) R-2.15.3: Follow the instructions from the R official website.

   tar xzvf xxx.tar.gz
   ./configure   (using default prefix for installation destination)
   make
   make check-all
   make install 
   A couple of WARNING and tcltk errors in testing results were ignored in our case.
   Installed at: /usr/local/bin/R, R_HOME_DIR = /usr/local/lib64/R

(3) python 2.7.3: Follow the instructions from the python official website. Our experience is that there is no need to go with python 3.

   ./configure
   make
   make install
   In our case, three modules failed: imageop, _tkinter and_sqlite3, ignored.
   Set env PYTHONPATH in ~/.bash_profile. 
   Installed at /usr/local/bin/python
    

(4) PyYAML 3.10

   /usr/local/bin/python setup.py --without-libyaml install
   python setup.py test

(5) proj-4.8.0: Follow instructions.

   ./configure –without-jni (default –prefix /usr/local)
   make; make check, make install

(6) pyproj-1.9.3

   python setup.py build
   python setup.py install 

(7) MySQL-python-1.2.4

   python setup.py build
   python setup.py install

(8) lapack 3.3.1

   create make.inc by copying INSTALL/make.inc-gfortran
   add –fPIC to OPTS and NOOPT
   make lib

(9) ATLAS 3.8.4 (have tried 3.10.0 and 3.10.1, but failed)

   bunzip2 –c xxx.bz2 | tar xfm – 
   cpufreq-selector –g performance
   On our system, no cpufreq support, so turn off throttling. 
   under subdir ATLAS-Linux64 ../configure –Fa alg –fPIC –with-netlib-lapack=/XXX/lapack_LINUX.a (from the installation of lapack 3.3.1)
   make
   cd lib 
   make shared
   make ptshared

(10) numpy 1.6.1

   copy site.cfg.example to site.cfg, edit section headers: DEFAULT, blas_opt and lapack_opt (using defaults for the last two sections).
   python setup.py build –fcompiler=gnu95
   python setup install
   test in python commands mode
   import numpy
   numpy.test()

(11) scipy 0.11.0

   python setup.py config_fc –fcompiler=gnu95 install
   test in python commands mode
   import scipy
   scipy.test()

(12) nose 1.2.1, for testing

   easy_install nose

(13) szip 2.1

   ./configure –prefix=/local-location/szip
   make
   make check 
   make install

(14) zlib 1.2.7

   ./configure –prefix=/local-location/zlib
   make check install

(15) hdf5 1.8.9

   ./configure –withzlib=/xxx/zlib –prefix=/ local-location/hdf5
   make check install

(16) netcdf 4.1.3, follow instructions from nectdf official website

   set CPPFLAGS = and LDFLAGS= correctly for hdf5, zlib, szip, include and lib
   ./configure –prefix=/local-location/netcdf
   make check install

(17) netCDF4 1.0.4, this is a python interface of netcdf4

   copy setup.cfg.template to setup.cfg, uncomment/correct netcdf4.1.3, hdf5 and szip dirs.
   python setup.py build
   python setup.py install
   cd test; python run_all.py
   In our case, error=1, test_select_nc, in function “num2date” failed for calculating milliseconds, ignored.

(18) gdal 1.9.2

   set in ~/.bash_profile, PATH for netcdf4.1.3/bin for nc-config and LD_LIBRARY_PATH for netcdf4.1.3, hdf5 and szip libs. 
   proj libs dynamic link at /usr/local/lib (proj libs should be already there).
   ./configure (default prefix to /usr/local)
   make
   make install

(19) rgdal 0.86

   R CMD INSTALL KeinSmooth-2.23-10.tar.gz
   R CMD INSTALL MASS_7.3-23.tar.gz
   R CMD INSTALL lattice_0.20-15.tar.gz
   R CMD INSTALL RColorBrewer_1.0-5.tar.gz
   R CMD INSTALL sp-1.0-8.tar.gz
   R CMD INSTALL rgdal_0.8-6.tar.gz
   The last installation of rgdal requires gdal-config and gdal libs are on the path (/usr/local/bin and lib). 
   Also, porj libs and include should be on the path too (/usr/local/lib and include), and libs of hdf5, zlib and szip are set by LD_LIBRARY_PATH in ~/.bash_profile.

(20) ncdf4, this is a R interface of netcdf4

   set in ~/.bash_profile, PATH for netcdf4.1.3/bin for nc-config and LD_LIBRARY_PATH for netcdf4.1.3, hdf5 and szip libs. 
   R CMD INSTALL ncdf4_1.6.1.tar.gz

(21) M3 and aqfig, these are two R packages developed by EPA

   R CMD INSTALL maps_2.3-2.tar.gz
   R CMD INSTALL mapdata_2.2-2.tar.gz
   R CMD INSTALL mapproj_1.2-1.tar.gz
   R CMD INSTALL splancs-2.0.1-32.tar.gz
   R CMD INSTALL RandomFields-2.0.66.tar.gz
   R CMD INSTALL geoR-1.7-4.tar.gz
   R CMD INSTALL scatterplot3d_0.3-33.tar.gz
   R CMD INSTALL geoRglm-0.9-2.tar.gz
   R CMD INSTALL maptools_0.8-26.tar.gz 
   R CMD INSTALL M3_0.3.tar.gz
   R CMD INSTALL aqfig-0.5.tar.gz (GaTech’s modified version, if you want to test amet2 with SEMAP’s products)
        

(22) Install AMET-2.0.a1.tar.gz.

   This is the AMET2’s distribution package, after uncompress it, inside the main dir there are python and R portions need further installations.
   Uncompress the package into dir “amet-2.0a1”. 
   Configure amet-2.0a1/config/config.py and config.R. 
   Then either run amet-2.0a1/config/setup.csh or set the path to amet-2.0a1/config with PYTHONPATH in ~/.bash_profile.
   cd amet-2.0a1/loading/AMET-2.0a1/, python setup.py install.
   cd amet_2.0a1/analysis to do further R installations:
   R CMD INSTALL xtable_1.7-1.tar.gz
   R CMD INSTALL DBI_0.2-5.tar.gz
   R CMD INSTALL RMySQL_0.9-3.tar.gz
   R CMD INSTALL gtable_0.1.2.tar.gz
   R CMD INSTALL digest_0.6.3.tar.gz
   R CMD INSTALL proto_0.3-10.tar.gz
   R CMD INSTALL dichromat_2.0-0.tar.gz
   R CMD INSTALL colorspace_1.2-1.tar.gz
   R CMD INSTALL munsell_0.4.tar.gz
   R CMD INSTALL labeling_0.1.tar.gz
   R CMD INSTALL evaluate_0.4.3.tar.gz
   R CMD INSTALL testthat_0.7.1.tar.gz
   R CMD INSTALL scales_0.2.3.tar.gz
   R CMD INSTALL yaml_2.1.7.tar.gz 
   R CMD INSTALL XML_3.9.6-1.1.tar.gz
   geos 3.3.8, this is an external lib, needs to be installed before installing rgeos. 
   bunzip2 –c geos-3.3.8.tar.bz2 | tar xfm –
   ./configure    (requires gcc > 4.0, prefix -> /usr/local) 
   make
   make install
   R CMD INSTALL rgeos_0.2-15.tar.gz
   R CMD INSTALL reshape+0.8.4.tar.gz
   R CMD INSTALL amet_0.5.4.tar.gz (GaTech’s modified version, if you want to test amet2 with SEMAP’s products)     

Now finally you should have amet command (/usr/local/bin/amet) ready for use with amet2’s R functions available too.