Brief notes for configuring, installing and using PetSc and
ScaLAPACK with Fortran 90/95 (assuming you're running linux
(preferably Debian) and have Intel compilers (7.1) and the MKL
(7.21) already installed
in /opt/intel).
MPICH
# tar -xzf mpich-1.2.6.tar.gz
# cd mpich-1.2.6
# setenv CC icc
# setenv CXX icpc
# setenv FC ifc
# setenv F90 ifc
# ./configure --with-arch=LINUX --prefix=/opt/mpich-1.2.6
--with-device=ch_p4 --enable-debug --without-mpe
--enable-sharedlib
# make
# sudo mkdir /opt/mpich-1.2.6
# sudo make install
# ssh-keygen -t dsa
# cd $HOME/.ssh
# cp id_dsa.pub authorized_keys
{To build mpich2 configure with --with-device=ch3:sock
--enable-fast --enable-debuginfo --without-mpe
--prefix=/opt/mpich2}
PetSc*
# tar -xzf petsc-2.3.3.tar.gz
# cd petsc-2.3.3
# setenv PETSC_DIR `pwd`
# ./config/configure.py --with-mpi-dir=/opt/mpich-1.2.6
--with-blas-lapack-dir=/opt/intel/mkl721 --prefix=/opt/petsc-2.3.3
--with-shared=yes
# make all
# make test
# sudo mkdir /opt/petsc-2.3.3
# su; make install; exit
{Now add the PETSC_DIR (/opt/petsc-2.3.3) and PETSC_ARCH
(linux-gnu-intel) variables in your .bashrc/.cshrc}
BLACS (needed by
ScaLAPACK)
# tar -xzf mpiblacs.tgz
# cd BLACS
# cp BMAKES/Bmake.MPI-LINUX Bmake.inc
Edit and configure Bmake.inc
# make MPI
# cd LIB
Now create symlinks as follows:
libblacsCinit_MPI-LINUX-0.a -> blacsCinit_MPI-LINUX-0.a
libblacsF77init_MPI-LINUX-0.a -> blacsF77init_MPI-LINUX-0.a
libblacs_MPI-LINUX-0.a -> blacs_MPI-LINUX-0.a
# cd ../../; sudo chown -R root\: BLACS
# sudo mv BLACS /opt
ScaLAPACK
# tar -xzf scalapack-1.8.0.tgz
# mv scalapack-1.8.0 SCALAPACK
# cd SCALAPACK
Edit and configure SLmake.inc
# make lib
# cd ..; sudo chown -R root\: SCALAPACK
# sudo mv SCALAPACK /opt
Note: Now its a good idea to set/update your environment variables.
However the best approach (i.e., the one I use) is to use the
environment-modules
package (installation instructions are available here). It allows you to keep/use multiple
versions of a program without having to worry about environment
variables. For example you can keep two mpich installations, one
configured with Intel compilers and the other with GNU compilers
and switch between the two using a simple command.
Some examples:
MKL
(BLAS/LAPACK)
# ifc svd.f90 -I/opt/intel/mkl721/include
-L/opt/intel/mkl721/lib/32 -lmkl_lapack95 -lmkl_lapack -lmkl_ia32
-lguide -lpthread
# ./a.out
{Multithreaded BLAS 3 will parallelize parts of the above code on a
shared memory machine if you set the OMP_NUM_THREADS variable to 2
or more; Also please note that Fortran 95
wrapper files (modules and libraries) are installed in
/opt/intel/mkl721/include and /opt/intel/mkl/lib/32
respectively}
# ifc matmul.f90
-L/opt/intel/mkl721/lib/32 -lmkl_ia32 -lguide -lpthread
# ./a.out
{Compares the performance of matrix multiplication}
# ifc pardiso.f90
-L/opt/intel/mkl721/lib/32 -lmkl_solver -lmkl_lapack -lmkl_ia32 -lguide
-lpthread
# ./a.out
{Tests the functionality of the PARDISO solver}
PetSc
# make -f Makefile_petsc petsc_ex (petsc_ex.f90; Makefile)
# mpirun -np 2 ./petsc_ex
ScaLAPACK
#
mpif90 parallel_svd.f90
-L/opt/SCALAPACK -lscalapack -L/opt/BLACS/LIB -lblacs_MPI-LINUX-0
-lblacsF77init_MPI-LINUX-0 -lblacs_MPI-LINUX-0
-L/opt/intel/mkl721/lib/32 -lmkl_lapack -lmkl_ia32 -lguide
-lpthread
{Note the recursive referencing of libraries}
# mpirun -np 4 ./a.out
Other useful libraries:
SLEPc
Make sure that PETSC_DIR (/opt/petsc-2.3.3) and PETSC_ARCH
(linux-gnu-intel) variables are defined
# tar -xzf slepc-2.3.3
# cd slepc-2.3.3
# setenv SLEPC_DIR `pwd`
# ./config/configure.py --prefix=/opt/slepc-2.3.3
# make all
# make test
# sudo mkdir /opt/slepc-2.3.3
# su; make install; exit
{Now add the SLEPC_DIR (/opt/slepc-2.3.3) variable in your
.bashrc/.cshrc}
Example:
# make slepc_ex (slepc_ex.f90; Makefile)
# mpirun -np 2 ./slepc_ex
MUMPS* (a parallel
sparse direct solver based on LU factorization; depends on
ScaLAPACK and (optionally) Metis)
# tar -xzf MUMPS_4.7.3.tar.gz
# cd MUMPS_4.7.3
# cp Make.inc/Makefile.INTEL.PAR Makefile.inc
Edit and configure Makefile.inc
# make double
{use 'make all' for simple, cmplx and cmplx16 versions of the
lib}
# cd ..; sudo chown -R root\: MUMPS_4.7.3
# sudo mv MUMPS_4.7.3 /opt
Example:
# mpif90 -fpp mumps_ex.f90
-I/opt/MUMPS_4.7.3/include -L/opt/MUMPS_4.7.3/lib -ldmumps -lpord
$SCALAPACK -L/opt/parmetis-3.1 -lmetis
# mpirun -np 2 ./a.out < mumps_inp
---
[*] Other sparse iterative/direct solvers which you might find
useful are hypre, Aztec, SuperLU etc.