README

PARALLEL ANALYSIS OF GEODESIC DATA
**********************************

FOR THE IMPATIENT
=================
The command::

    ./configure && make && make install

should compile the static library libpagoda.a and the programs pgsub, pgra,
pgea, pgbo, pgcmp, and pgdump and install them into /usr/local.  Please refer
to the INSTALL file for generic build instructions.  That would be a good
place to start if you are new to using configure scripts.  Detailed
instructions are covered later in this file.

QUESTIONS/HELP/SUPPORT/BUG-REPORTS
==================================
email: pagoda-dev@googlegroups.com

DEPENDENCIES
============
pagoda uses autoconf/automake/libtool to build and has dependencies on
 * Message Passing Interface
 * Global Arrays (http://www.emsl.pnl.gov/docs/global/)
 * parallel netcdf (http://trac.mcs.anl.gov/projects/parallel-netcdf/).

GLOBAL ARRAYS (GA)
------------------
At the time of pagoda's initial releases, GA transitioned to using configure
and a single library (libga) for version 5.0.  Prior to 5.0, building GA
involved knowing your "TARGET" system and produced as many as five separate
yet inter-dependent libraries (liblinalg, libma, libglobal, libarmci,
libpario, libtcgmsg, libtcgmsg-mpi).  pagoda's configure system attempts to
locate libga first and then the other libraries as needed.  GA was
traditionally built for Fortran programs so by default contains Fortran source
code.  This requires the Fortan libraries to be known at link-time.  Since
this is the most common case, configure will try to determine $FLIBS for
solely the purpose of linking with GA.  GA can be built without dependence on
Fortran source and libraries, but that mode of GA install is not well tested.
In the future there will hopefully be a better way of detecting a GA
installation.

Please refer to GA's own documentation for how to build it, usually located
within their releases' README files.

BUILDING GLOBAL ARRAYS 4.3.x on franklin.nersc.gov
..................................................
Do no use the provided GA 4.x.x on franklin (e.g. module load ga) since it is
usually an older version.  The environment variables we use to build GA 4.3.x
are as follows::

    setenv TARGET LINUX64
    setenv MSG_COMMS MPI
    setenv ARMCI_NETWORK PORTALS
    setenv CC cc
    setenv CXX CC
    setenv F77 ftn
    setenv FC ftn
    setenv XT_SYMMETRIC_HEAP_SIZE 1G
    #setenv MA_USE_ARMCI_MEM 1 # <--- this option is currently broken; don't use
    setenv CRAY_PORTALS_USE_BLOCKING_POLL 1

With the above, you should be able to simply "make" GA.  The last environment
variable is important, apparently, because it toggles a feature of GA
necessary on portals systems or else global operations within GA will crawl.

For the recommended envrionment variables on other platforms, please see the
GA support page http://www.emsl.pnl.gov/docs/global/support.html.

BUILDING GLOBAL ARRAYS 5.0
..........................
For detailed instructions on building GA 5.x or better (now converted to
autotools), please see the README file in the GA distribution.  The following
example will build GA 5.x to use sockets for communication (this is not
optimal -- please see GA's README file)::

    tar -xvzf ga-5-0B.tgz
    cd ga-5-0B
    mkdir bld
    cd bld
    ../configure
    make

OTHER GLOBAL ARRAYS NOTES
.........................
Note: as of this writing, GA's MA_USE_ARMCI_MEM is broken.  It is unclear when 
this broke, but unset MA_USE_ARMCI_MEM in your environment.

PARALLEL NETCDF (PNETCDF)
-------------------------
We recommend using the latest available version of pnetcdf.  We test with
version 1.2.0.  Pagoda may still work with earlier versions of pnetcdf, but in
order to take advantage of the latest pnetcdf IO optimizations, version 1.2.0
or better is required.

Pagoda has been known to work with pnetcdf 1.1.1.  Note that in pnetcdf 1.1.0
there were problems with C++ compilers and the pnetcdf.h header.

BUILDING PAGODA
===============

Salt to taste the following.  Note that we prefer to add --prefix=`pwd` to
our own development configurations::

    mkdir bld
    cd bld
    ../configure --with-pnetcdf=/path/to/pnetcdf/install --with-ga=/path/to/ga5.x/install
    #### OR ####
    ../configure --with-pnetcdf=/path/to/pnetcdf/install --with-ga="-L/path/to/ga4.x/lib/TARGET -I/path/to/ga4.x/include"

By now you should be able to "make".  The "make check" target is currently not
implemented and will eventually contain our test suite.  The "make checkprogs"
target will build all test programs as they become available.

There is a "sandobox" in which small test programs exist, but it should be
ignored by end users.  It exists mainly for testing certain third party
library functionality or for developing new ideas outside of the main source.
You must --enable-sandbox to build the programs (if you are a developer).

TESTING PAGODA
==============
The test suite uses autoconf's autotest capabilities.  Running "make check"
will build all test programs and requires the ability to call the MPI process
launcher.  The tests confirm, among other things, that the pagoda tools
produce the same results as their corresponding NetCDF Operators for any
number of processes.  By default, we test running on 1, 2, 3, and 4 processes.
The influential environment variables are MPIRUN, NP_START, NP_INC, and
NP_STOP which control the MPI launcher and the arguments passed to the "seq"
command-line tool when generating the number of procs to use in the tests' for
loops.

The input datasets used for the test suite must come from you, the user.
Place any input files or symbollicaly link to them from within the ./data
directory.  If you see many skipped tests, make sure you put some input files
into ./data before running the suite.