MPI programming using mpich2

Check the User's Guide for more details: http://www-unix.mcs.anl.gov/mpi/mpich2/index.htm#docs

NOTE: all MPI programs should be run using the N1 Grid Engine 6 scheduler.

All executables are located in /usr/local/mpich2-1.0.3/bin which is in your login PATH. Before getting started make sure you have set up your account with ssh passwordless logins. See How to set up SSH Passwordless Logins.

To compile:
C programs - mpicc
FORTRAN 77 - mpif77
FORTRAN 90 - mpif90

The MPI libraries were configured using Absoft. If you want to compile a simple program simply type 

mpif90 myprog.f90
If you want to use the IMSL or LAPACK, however, you will have to link to those libraries explicitly. I've created an alias 
mpiaf90
that should take care of this for you. Type 
which mpiaf90
to see exactly what it does. If something fails or you use other modules and libraries you may have to modify this compilation. Click here for a sample Makefile I use.

Simple Examples:
C - download mpihello.c and run: 

mpicc mpihello.c 
mpiexec -n 2 a.out
FORTRAN 90 - download hello.f90 and run: 
mpif90 hello.f90
mpiexec -n 2 a.out
For a rough estimate of how many flops courtney is capable of download flop.c

Further Refinement: machinefile:
This step should really be unnecessary since the N1 Grid Engine 6 scheduler creates a machinefile for you automatically.
To tell your program which computers to run on, create a machinefile. For instance a machinefile MF.dat with the contents

janelle:1
carrie:2
will run the master process (of rank 0) on janelle and run the other two processes (of rank 1 and 2) on carrie. To execute with the machinefile type 
mpiexec -machinefile MF.dat -n 3 a.out
To see which machines are booted into the ring type 
mpdtrace
To see what MPI jobs are currently running type 
mpdlistjobs
To kill one of your own job numbers given in mpdlistjobs type 
mpdkilljob #
To see a detailed status report (need Pless SSH to use this) of one or more nodes type 
getstat   OR   getstat nodename1 nodename2 etc...