Copyright (C) 1996-2000

Object Oriented MPI (OOMPI): A C++ class library for MPI
Authors: Jeffrey M. Squyres, Jeremiah J. Willcock, Brian C. McCandless, 
	 Peter W. Rijks, Andrew Lumsdaine

This file is part of the Notre Dame Object Oriented MPI (OOMPI)

You should have received a copy of the License Agreement for the Notre
Dame Object Oriented MPI along with the software; see the file LICENSE.
If not, contact Office of Research, University of Notre Dame, Notre
Dame, IN 46556.

Permission to modify the code and to distribute modified code is
granted, provided the text of this NOTICE is retained, a notice that
the code was modified is included with the above COPYRIGHT NOTICE and
with the COPYRIGHT NOTICE in the LICENSE file, and that the LICENSE
file is distributed with the modified code.

LICENSOR MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.
By way of example, but not limitation, Licensor MAKES NO
REPRESENTATIONS OR WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY
PARTICULAR PURPOSE OR THAT THE USE OF THE LICENSED SOFTWARE COMPONENTS
OR DOCUMENTATION WILL NOT INFRINGE ANY PATENTS, COPYRIGHTS, TRADEMARKS
OR OTHER RIGHTS.

--------------------------------------------------------------------

Developed and implemented by:
  Jeffrey M. Squyres
  Jeremiah J. Willcock
  Brian C. McCandless
  Peter W. Rijks
  Andrew Lumsdaine

  Laboratory for Scientific Computing
  Department of Computer Science and Engineering
  University of Notre Dame
  http://www.lsc.nd.edu/

--------------------------------------------------------------------

OOMPI: Object Oriented MPI
1.0.3
Installation and Execution Instructions
=======================================


I. INSTALLATION INSTRUCTIONS
============================

There are three basic steps to the installation of OOMPI:

	1. ./configure <options>
	2. make
	3. make install

If you wish to use the defaults for a given architecture (which should
be sufficient for most situations) and have the vendor-supplied MPI
implementation installed in its default location, you can probably
leave <options> blank and do the three steps listed above.  

If you wish to use LAM, MPICH, or want to change some of the default
settings, you should continue reading.

These three steps are covered in detail below.


1. ./configure <options>
------------------------

OOMPI has been compiled and tested on the combinations listed in Table
1.

OS and version      Comp    MPI impl
========================================
AIX 4.1.4           g++     LAM 6.3.2   
                            MPICH 1.2.0 
                    mpCC    Native      
                    xlC     LAM 6.3.2   
                            MPICH 1.2.0 
AIX 4.2.1           g++     LAM 6.3.2   
                            MPICH 1.2.0 
                    mpCC    Native      
                    xlC     LAM 6.3.2   
                            MPICH 1.2.0 
----------------------------------------
HP-UX 11.00         aCC     LAM 6.3.2   
                            MPICH 1.2.0 
                    mpiCC   Native      
----------------------------------------
Irix 6.5            CC      LAM 6.3.2   
                            MPICH 1.2.0 
                            Native      
----------------------------------------
Linux 2.2.12 Alpha  g++     LAM 6.3.2   
                            MPICH 1.2.0 
Linux 2.2.14 x86    KCC     LAM 6.3.2   
                            MPICH 1.2.0 
                    g++     LAM 6.3.2   
                            MPICH 1.2.0 
                    pgCC    LAM 6.3.2   
                            MPICH 1.2.0 
----------------------------------------
Solaris 2.5.1       CC      LAM 6.3.2   
                            MPICH 1.2.0 
                    KCC     LAM 6.3.2   
                            MPICH 1.2.0 
                    g++     LAM 6.3.2   
                            MPICH 1.2.0 
Solaris 2.6         CC      LAM 6.3.2   
                            LAM 6.3.3b17
                            MPICH 1.2.0 
                    KCC     LAM 6.3.2   
                            MPICH 1.2.0 
                    g++     LAM 6.3.2   
                            MPICH 1.2.0 
Solaris 7           CC      LAM 6.3.2   
                            MPICH 1.2.0 
                    KCC     LAM 6.3.2   
                            MPICH 1.2.0 
                    g++     LAM 6.3.2   
                            MPICH 1.2.0 
----------------------------------------
		Table 1: Supported architectures / OSs for OOMPI

*** LAM Users: 

The LAM Team does not support any version of LAM prior to LAM 6.3.2.
If you do not have this version, you should upgrade to it.  See the
LAM home page: http://www.mpi.nd.edu/lam/

OOMPI depends on using the hcp compiler wrapper script; it must be in
your path.  hcp requires that the LAMHCP environment variable be set
to the C++ compiler that you want to use.  See the LAM documentation
for details.


*** MPICH Users:

The MPICH team does not support any version prior to MPICH 1.2.0.  If
you do not have this version, you should upgrade to it.  See the MPICH
home page: http://www.mcs.anl.gov/mpi/mpich/

Architecture specific notes on MPICH:
 - OOMPI must be compiled in 64 bit mode on IRIX systems when used
   with MPICH.


*** GNU g++ Users: 

Unfortunately, the GNU compilers don't seem to handle exceptions
properly.  "-fhandle-exceptions" (for versions before 2.8.x) generates
"Internal compiler errors" when compiling OOMPI, so exceptions are
disabled when compiling with GNU.  See below for important information
regarding the test suite and the [mis]handling of exceptions with g++.

It is strongly recommended that all GNU user upgrade to the latest
version of the compilers (2.95.2 at the time of this writing).

Also, in order for exceptions to work correctly with g++, every
library that passes exceptions (even if it does not generate or catch
them) must be compiled with the -fexceptions flag.  In particular,
your MPI implementation must be compiled with this flag.  An "Abort"
error in the exception test in the OOMPI test suite is the main
symptom that this was not done.


*** SGI MPI/IRIX Users: 

The default behavior is that OOMPI will be built using the n32 version
of libmpi.so if it is available (in /usr/lib32).  If the n32 version
is not available, it will be built using the 64 bit version from
/usr/lib64.  This behavior can be overridden by forcing OOMPI to be
compiled in 64 bit mode with a command line option to the
"./configure" script (see below).  

Additionally, the test suite makes use of the MPI_CHECK_ARGS
environment variable.  Set this variable to "1" *before* invoking
mpirun on the test suite for proper checking of exception handling
upon MPI errors.  If it is not set, some tests will be skipped.


*** HP-UX Users: 

You may get warnings from the CC C++ compiler when OOMPI is compiling:

CC: "<some_path>/mpi.h", line 134: warning: "volatile" not implemented (ignored) (163)
CC: "<some_path>/mpi.h", line 135: warning: "volatile" not implemented (ignored) (163)

This is normal, and is due to a limitation in the HP-UX C++ compiler.
It has been reported to HP.


*** Other Users:

Shortcuts for the systems listed above have been built into the
configure script.  As more hardware/OS/MPI combinations are tested,
more shortcuts will be added to the script.  See the contact
information in the README file if you wish to provide information for
another hardware/OS/implementation combination in the OOMPI package.

Although the GNU configure script is fairly intelligent in how it
attempts to find your MPI implementation, it still may need a bit of
help.  Listed below are the flags that configure accepts.  You should
only need to specify one of the MPI implementation flags (unless you
are using MPICH; see below).  Additionally, the CXX environment
variable may be set to override the default C++ compiler that OOMPI
will use to build itself (except for LAM and MPICH, which will use the
"hcp" and "mpiCC" wrapper compiler scripts, respectively.  The
underlying C++ compiler that LAM/hcp will use can be overridden with
the LAMHCP environment variable.  The underlying compiler for
MPICH/mpiCC can be overridden when MPICH is configured; see the MPICH
documentation for more details).


*** Note on optimization:

Please note that all of the OOMPI components are built with high
levels of optimization.  If you are confident that the optimizer of
your compiler is functioning properly, you can disable the optimizer
in the compilation of the test suite and example programs (which can
save a *lot* of time in compiling!) with the "--with-testopt" and
"--with-examopt" flags to "./configure".  See Table 2 for a list of
"./configure" options, below.


*** Note on exceptions: 

If configure determines that your C++ compiler does not correctly
handle exceptions (e.g. g++ 2.7.2), OOMPI can still be built but its
exception handling capabilities will default to OOMPI_ERRORS_RETURN
even if you set OOMPI_ERRORS_EXCEPTION.  An error message will be
printed to stderr when an exception would have been thrown, but
returns instead.  See the description of the test suite below, under
"GNU g++ users" for important information about what [normal] error
messages to expect when you run the test suite.


-------------------------------------------------------------------------
Generic flags:
--prefix=dir		Specifies the top-level directory where OOMPI
			should be installed (i.e. dir/lib,
			dir/include, and dir/bin).  If no prefix is 
			specified, /usr/local is used.

Specific MPI implementation switches:
--with-cray		Specifies to use the native Cray MPI implementation.
--with-hp		Specifies to use the native HP MPI implementation.
--with-lam=dir		Specifies the top-level directory where LAM 6.x 
			was installed.  If "hcp" is already in your
			path, you do not need to specify "dir".
--with-mpich=dir	Specifies the top-level directory where MPICH 
			1.2.0 was installed.
--with-sun=dir		Specifies to use the native Sun HPC/MPI.  You
			can optionally specify dir if HPC was not
			installed into its default location.
--with-sp		Specifies to use the native IBM POE/MPI
			implementation.
--with-sgi=dir		Specifies to use the native SGI MPI implementation,
			possibly in a different directory.
--with-sgi64=dir	Specifies to use the native SGI MPI implementation,
			forcing OOMPI to compile in 64 bit mode, possibly
			in a different directory.


Unlisted MPI implementation switches (see note *1* below):
--with-mpidir=dir	Specifies a top-level directory where dir/mpi.h
			and dir/libmpi.a may be found.

--with-mpiinc=dir	Specified a top-level directory where dir/mpi.h
			may be found.
--with-mpilib=dir	Specified a top-level directory where dir/libmpi.a
			may be found.

Optimization flags:
--with-noopt            Use no optimization flags in compiling any part
                        of OOMPI.  Useful for debugging.
--with-libopt=flags     Use <flags> when compiling liboompi.a.
--without-libopt	Do not optimize liboompi.a.
--with-testopt=flags    Use <flags> when compiling the test suite.
--without-testopt	Do not optimize the test suite.
--with-examopt=flags    Use <flags> when compiling the examples.
--without-examopt=flags	Do not optimize the examples.


Miscellaneous switches:
--with-extralibs=libs	Extra libraries that may be necessary to link with
			MPI programs (see note *2* below).
-------------------------------------------------------------------------
			Table 2: "./configure" flags

Table 2 notes:
~~~~~~~~~~~~~~

*1*: These flags are only necessary if you do not have one of the MPI
implementations listed above.  That is, if you have an MPI
implementation that has a specific "--with-" option (e.g.,
"--with-mpich"), you do *not* need to use the "--with-mpidir",
"--with-mpiinc", or "--with-mpilib" directories.

*2*: This switch is meant for unlisted MPI implementations, but can be
used in conjunction with any of the listed MPI implementations.

For unlisted MPI implementations, if libraries other than libmpi.a are
necessary to link MPI executables, they should be specified with this
switch.  For example:

./configure --with-mpidir=/home/mpi --with-extralibs="-lmpi -latmlib -lotherlib"


2. make
-------

"make" by itself will compile the OOMPI library.  The OOMPI test suite
(found in contrib/test_suite) and several example programs (found in
contrib/examples) can be compiled with "make examples".  "make" can
actually take several arguments, most of which are similar to the GNU
coding standards:

	Argument		Function
	----------------------- ------------------------------------
	lib			Compiles the OOMPI library.
	examples		Compiles everything in contrib/.
	docs			Compiles (latex) the OOMPI
				documentation in the doc/
				subdirectory. 
	install			Installs OOMPI (see below).
	uninstall		Opposite of install; removes OOMPI
				from its installed locations.
	clean			Removes .o and associated files.
				It is safe to do this after you
				install OOMPI (to conserve disk
				space).
	distclean		Restores the OOMPI tree to the same
				state as it was when you un-tarred
				the distribution.
	----------------------- ------------------------------------
		Table 3: Targets for the default Makefile

"make" by itself actually invokes "make clean lib examples".


3. make install
---------------

"make install" installs the following OOMPI files:

	Source			Destination
	----------------------- ------------------------------------
	src/liboompi.a		$prefix/lib/liboompi.a
	src/oompi.h		$prefix/include/oompi.h
	src/[many_files].h	$prefix/include/oompi-1.0.3/[many_files].h
	oscar/oscar		$prefix/bin/oscar
				Symbolic link from 
				  $prefix/include/oompi-1.0.3 to
				  $prefix/include/oompi
	----------------------- ------------------------------------
	     Table 4: Destination for files in a "make install"

Additionally, $prefix/include/oompi-1.0.3 is symbolically linked
to $prefix/include/oompi.  The only purpose for this symbolic link is
a reminder to the user what version of OOMPI is installed -- oscar
will correctly -I$prefix/include/oompi-1.0.3, regardless of
whether the symbolic link is present or not.

The destination directories are created if they do not already exist.
OOMPI is now ready to use.


4. make uninstall
-----------------

This is the opposite of "make install".  All files are removed from
their installed locations (from within $prefix), but the $prefix/lib,
$prefix/include, and $prefix/bin directories are not removed.  

The $prefix/include/oompi-1.0.3 directory *is* removed, and the
$prefix/include/oompi symbolic link is removed *only* if it still
points to the $prefix/include/oompi-1.0.3 that you are
uninstalling.


5. make clean
-------------

This removes all .o and associated files from the OOMPI distribution
tree.  This is safe to do after a "make install"; its purpose is to
conserve disk space after OOMPI has been fully installed.


6. make distclean
-----------------

As per GNU standards, "make distclean" will restore the distribution
tree to the same state that it was when you unzipped and untarred it
(barring any additional files or changes that the user may have
made).  This is also safe to do after a "make install", but you will
lose your configure information.  If a re-build is necessary, you
will need to run configure again.


II. OOMPI DOCUMENTATION
=======================

The docs/oompi.ps (and docs/oompi.pdf) file provides complete
documentation on the design and implementation of OOMPI.  Manual pages
for all OOMPI classes an member function are also provided in the .ps
document.  This distribution does not include pages suitable for the
"man" command.  The latest documentation and information are always
available from the OOMPI WWW site (see the README document); the WWW
site should always have the most current and up-to-date information on
OOMPI documentation, patches, and distributions.

Adventuresome individuals may re-make the documentation with a "make
docs" in the top-level OOMPI directory.


III. EXECUTION INSTRUCTIONS
===========================

1. Source code
--------------

Programs that use OOMPI classes and functions only need to include
<oompi.h> in their respective source files.  


2. OOMPI Simplified Compiler for Awesome Results (oscar)
--------------------------------------------------------

oscar is a simple shell script which adds all the MPI and OOMPI
include and library directories to the C++ compiler command line for
you (in fact, the script is remarkably similar to the mpiCC script
from MPICH 1.0.12; many thanks MPICH!).  For example, if my_prog.cc is
a program that uses OOMPI, the following will produce a valid
executable:

	unix% oscar my_prog.cc -o my_prog

"my_prog" is now a valid OOMPI/MPI program that can be run in your 
MPI environment.

After a successful "make install", oscar is installed into
$prefix/bin/oscar; it is probably most convenient if you install
"$prefix/bin" into your path.


3. Test suite
-------------

OOMPI comes with a test suite that tests every method of every class
in the OOMPI library.  It is highly recommended that you run this
program to ensure that OOMPI has installed correctly, your C++
compiler is functioning properly, and your MPI environment works
according to the MPI 1.2 standard.  While the test suite does not
perform stress tests on any of the subsystems listed above, it
verifies the basic functionality of OOMPI itself.

In order to build the test suite, first install OOMPI.  This is now
required, as the test suite is built using the oscar wrapper compiler
that is part of OOMPI.  After OOMPI is installed, run "make examples"
to build the test suite and example programs.

The test suite lists each method as it is tested.  There are four
possible outcomes of each test:

	Outcome		Meaning
	--------------- --------------------------------------------
	PASS		The test passed.

	FAIL		The test failed, and the test suite will
			abort.  A brief error message will describe
			the error.

	POSTPONED	The test on this method is being deferred
			until other tests have successfully tested.
			This method will show up later in the
			test suite.

	SKIPPED		The test is skipped because of a particular
			known problem with the underlying MPI 
			implementation.
	--------------- --------------------------------------------
	Table 5: Outcome of individual tests in the OOMPI test suite

The test suite executable is located at contrib/test_suite/oompi_test.
It is designed to run on an even number of processors, and can take
the following command line switches:

	Switches	Meaning
	--------------- --------------------------------------------
	-help		Print out a short description of the command
			line switches.

	-hp0102		Skip tests of MPI functions are are faulty
			in the HP MPI release 01.02

	-ibm21014	Skip tests of MPI functions that are faulty
			in the IBM POE maintenance version 2.1.0.14
	-ibm21015	Same, IBM POE 2.1.0.15
	-ibm21016	Same, IBM POE 2.1.0.16
	-ibm21017	Same, IBM POE 2.1.0.17
	-ibm21018	Same, IBM POE 2.1.0.18
	-ibm23012	Same, IBM POE 2.3.0.12

	-sgi20		Skip tests of MPI functions that are faulty
			in the SGI MPI release 2.0
	-sgi30		Same, SGI MPI 3.0 and 3.1
	-sgi32		Same, SGI MPI 3.2
	--------------- --------------------------------------------
	Table 6: Command line switches to the OOMPI test suite

*** IBM SP Users: 

There are several functions that fail on the SP, most of which are not
commonly used.  For versions before the fixes, the test suite will always
fail unless one of the -ibm* flags is used.  Skipped tests are noted in the
right hand column.  Additionally, it seems that MPI_DIMS_CREATE() is rather
fragile -- it works for some cases, and not with others.  By 2.1.0.18, all
bugs except MPI_CART_GET appear to have been fixed.  Also, in 2.1.0.14, the
MPI_Request_free call does not appear to work, and so the
OOMPI_Request::Free() and OOMPI_Request_array::Freeall() methods are no-ops
on this platform.

We currently do not have access to an IBM POE version later than 2.3.0.12;
the MPI_CART_GET bug has not yet been fixed in that version.


*** SGI Users: 

MPI_TESTALL, MPI_GRAPH_GET, and MPI_CANCEL do not work properly in
release 2.0 of SGI's MPI.  Most problems have been fixed in SGI's MPI
release 3.1.x, which is available from the SGI website
(http://www.sgi.com/Products/Evaluation/evaluation.html).  While there
is still a "-sgi20" command line flag for the oompi_test test suite,
it is *highly* recommended that you upgrade to the latest version.  

However, the test suite makes use of the MPI_CHECK_ARGS environment
variable (for SGI MPI versions 3.0 and later).  It *must* be set to
"1" *before* mpirun is used to invoke the oompi_test program.  If it
is not, the test checking for proper exception handling will be
skipped, or the test suite may abort with a segmentation fault (or
some other nasty error).


*** GNU g++ users:

GNU users are strongly recommended to upgrade to the latest version of
the GNU compilers.

The test suite checks several functions which are expected to fail.
These tests are supposed to be handled internally by OOMPI.  Normally,
an exception would be thrown, but the latest version of g++ (2.7.2.3
at the time of this writing) doesn't handle exceptions properly.  So
you should expect to see several messages of the form:

	OOMPI detects MPI error 00000: MPI_func: some error message
	MPI error in MPI_COMM_WORLD rank x of y.
	OOMPI set for exception handling (OOMPI_ERROR_EXCEPTIONS),
	but OOMPI was compiled with no exception handling.
	Defaulting to OOMPI_ERRORS_RETURN.

These messages are normal, and will be internally handled by the test
suite.  Depending on the IO scheme of your MPI implementation, you may
receive one message for each rank in MPI_COMM_WORLD, or just one for
rank 0.  Regardless, be aware to make your programs
exception-resilient, and mail the g++ authors to get their act
together with exceptions.


*** Errors / bugs:

Please report any errors or problems with the test suite to
oompi-bugs@mpi.nd.edu.  Please include the version of OOMPI that you
are working with as well as the output of configure, make, make
install, and the test suite.


4. Example code
---------------

Several example OOMPI programs are provided in contrib/examples.
These examples are OOMPI-fied versions of popular MPI example
programs.  They are well commented, and intended to demonstrate OOMPI
techniques to MPI programmers.  They also demonstrate use of the oscar
wrapper compiler, and (like the test suite) must not be built until
OOMPI has been installed.
