Brief notes for configuring, installing and using PETSc, ScaLAPACK
and MUMPS with Fortran 95 (assuming you're running linux
(preferably Debian) and have Intel compilers (7.1) and the MKL
(7.21) already installed
in /opt/intel).
MPICH
# tar -xzf mpich-1.2.6.tar.gz
# cd mpich-1.2.6
# setenv CC icc
# setenv CXX icpc
# setenv FC ifc
# setenv F90 ifc
# ./configure --with-arch=LINUX --prefix=/opt/mpich-1.2.6
--with-device=ch_p4 --enable-debug --without-mpe
--enable-sharedlib
# make
# sudo mkdir /opt/mpich-1.2.6
# sudo make install
# ssh-keygen -t dsa
# cd $HOME/.ssh
# cp id_dsa.pub authorized_keys
{To build mpich2 configure with "--with-device=ch3:sock
--enable-fast --enable-debuginfo --enable-sharedlibs=gcc
--without-mpe --prefix=/opt/mpich2" (the "--enable-sharedlibs"
option in not needed with GNU compilers)}
PETSc[a]
# tar -xzf petsc-2.3.3.tar.gz
# cd petsc-2.3.3
# setenv PETSC_DIR `pwd`
# ./config/configure.py --with-mpi-dir=/opt/mpich-1.2.6
--with-blas-lapack-dir=/opt/intel/mkl721 --prefix=/opt/petsc-2.3.3
--with-shared=yes
# setenv PETSC_ARCH linux-gnu-intel
# make all
# make test
# sudo mkdir /opt/petsc-2.3.3
# su; make install; exit
{Now add the PETSC_DIR (/opt/petsc-2.3.3) and PETSC_ARCH
(linux-gnu-intel) variables in your .bashrc/.cshrc}
BLACS (needed by
ScaLAPACK)
# tar -xzf mpiblacs.tgz
# cd BLACS
# cp BMAKES/Bmake.MPI-LINUX Bmake.inc
Edit and configure Bmake.inc
# make MPI
# cd LIB
Now create symlinks as follows:
libblacsCinit_MPI-LINUX-0.a -> blacsCinit_MPI-LINUX-0.a
libblacsF77init_MPI-LINUX-0.a -> blacsF77init_MPI-LINUX-0.a
libblacs_MPI-LINUX-0.a -> blacs_MPI-LINUX-0.a
# cd ../../; sudo chown -R root\: BLACS
# sudo mv BLACS /opt
ScaLAPACK
# tar -xzf scalapack-1.8.0.tgz
# mv scalapack-1.8.0 SCALAPACK
# cd SCALAPACK
Edit and configure SLmake.inc
# make lib
# cd ..; sudo chown -R root\: SCALAPACK
# sudo mv SCALAPACK /opt
Note: Now its a good idea to set/update your environment variables.
However the best approach (i.e., the one I use) is to use the
environment-modules
package (installation instructions are available here). It allows you to keep/use multiple
versions of a program without having to worry about environment
variables. For example you can keep two mpich installations, one
configured with Intel compilers and the other with GNU compilers
and switch between the two using a simple command.
Some examples:
MKL
(BLAS/LAPACK)
# ifc svd.f90 -I/opt/intel/mkl721/include
-L/opt/intel/mkl721/lib/32 -lmkl_lapack95 -lmkl_lapack -lmkl_ia32
-lguide -lpthread
# ./a.out
{Multithreaded BLAS 3 will parallelize parts of the above code on a
shared memory machine if you set the OMP_NUM_THREADS variable to 2
or more; Also please note that Fortran 95
wrapper files (modules and libraries) are installed in
/opt/intel/mkl721/include and /opt/intel/mkl/lib/32
respectively}
# ifc matmul.f90
-L/opt/intel/mkl721/lib/32 -lmkl_ia32 -lguide -lpthread
# ./a.out
{Compares the performance of matrix multiplication}
# ifc pardiso.f90
-L/opt/intel/mkl721/lib/32 -lmkl_solver -lmkl_lapack -lmkl_ia32
-lguide -lpthread
# ./a.out
{Tests the functionality of the PARDISO solver}
PETSc
# make -f Makefile_petsc petsc_ex (petsc_ex.f90; Makefile)
# mpirun -np 2 ./a.out
ScaLAPACK
#
mpif90 parallel_svd.f90
-L/opt/SCALAPACK -lscalapack -L/opt/BLACS/LIB -lblacs_MPI-LINUX-0
-lblacsF77init_MPI-LINUX-0 -lblacs_MPI-LINUX-0
-L/opt/intel/mkl721/lib/32 -lmkl_lapack -lmkl_ia32 -lguide
-lpthread
{Note the recursive referencing of libraries}
# mpirun -np 4 ./a.out
Other useful libraries:
MUMPS[b] (a
parallel sparse direct solver based on LU factorization; depends on
ScaLAPACK and (optionally) Metis)
# tar -xzf MUMPS_4.7.3.tar.gz
# cd MUMPS_4.7.3
# cp Make.inc/Makefile.INTEL.PAR Makefile.inc
Edit and configure Makefile.inc
# make double
{use 'make all' for simple, cmplx and cmplx16 versions of the
lib}
# cd ..; sudo chown -R root\: MUMPS_4.7.3
# sudo mv MUMPS_4.7.3 /opt
Example:
# mpif90 -fpp mumps_ex.f90
-I/opt/MUMPS_4.7.3/include -L/opt/MUMPS_4.7.3/lib -ldmumps -lpord
$SCALAPACK -L/opt/parmetis-3.1 -lmetis
# mpirun -np 2 ./a.out < mumps_inp
Some other sparse iterative/direct solvers which Fortran users
might find useful are hypre, Aztec, SuperLU etc.
---
[a] While configuring PETSc it is also advisable to
install hypre (--with-hypre=1 --download-hypre=yes) which allows
the use of the BoomerAMG algebraic multigrid (MG)
preconditioner
[b] PETSc also interfaces with MUMPS so if you want to
use MUMPS from PETSc then simply configure PETSc to
download/install BLACS, ScaLAPACK and MUMPS (and save yourself the
trouble of installing everything separately; the compiled BLACS,
ScaLAPACK and MUMPS libraries can then be symlinked to a more
appropriate location and used as such)