Here are some brief notes I made while installing PyLith and GeoFEST on RCAC's recycled x86 linux cluster (radon). Please note that PyLith is in extensive development and installation instructions/requirements keep on changing frequently. For more information follow the relevant documentation.


GeoFEST

GeoFEST v4.5 (Parallel) uses ParMetis for graph partitioning and the Pyramid AMR library (only for domain decomposition and not for AMR). As of now it only works with Intel 7.x compilers (if you want to try 8.x/9.x compilers then you might want to have a look here).

# module purge all
# module load mpich-1.2.6-intel/71 (invokes mpich-1.2.6 with intel-7.1 Compilers)

ParMetis
# tar -xzf ParMetis-3.1.tar.gz
# cd ParMetis-3.1
# make

Pyramid AMR lib
# tar -xzf Pyramid-2.0.tar.gz
# cd Pyramid-2.0
# make -f Makefile.Intel distclean
# cd Pyramid
# ln -s -f ../../ParMetis-3.1 ParMetis
# cd ..
# make -f Makefile.Intel

GeoFEST
# tar -xzf GeoFEST-4.5.tgz
# cd GeoFEST-4.5
# ln -s -f ../Pyramid-2.0 Pyramid
# cd geofest
Make sure that the file i386-linux-Intel.mk (or $MACHTYPE-$OSTYPE-$F90TYPE.mk) contains
    FORTRAN_LIBS=/opt/intel/compiler70/ia32/lib
    fortran_libs=-lCEPCF90 -lPEPCF90 -lintrins -lF90
Switch 'sig_ZZ' with 'sig_XY' in inphase.c:946
# make -f Makefile.Parallel F90TYPE=Intel

Now update $PATH and test the installation

# qsub -I -l nodes=4:ppn=2
# module load mpich-1.2.6-intel/71
# cd my_scratch_dir
# gfmeshparse my_file
(gfmeshparse is a utility that comes with GeoFEST to create a secondary file my_file.jpl for the parallel run; also do not include any velocity bc's in my_file, instead first create the file my_file.jpl and then add the velocity bc's in my_file or else, modify the gfmeshparse source to skip the velocity bc data in my_file)
# mpirun $PBS_NODEFILE -np 8 `which GeoFEST` my_file

Here are the CIG strike-slip benchmark runtimes on an Itanium (dual processor) cluster...

Resolution (m)
Nodes		 CPU's Time (mins) CPU utilization (%)
1000 m
15625		 2
4		 0:02:31
0:01:26		 95
87
500 m
117649		 2
4
8
16
32		 2:06:25
1:06:37
0:33:53
0:22:16
0:09:54		 97
94
88
71
75
250 m
912673		 24
32		 1:04:40
0:49:13
80
78


PyLith

PyLith (v0.8.2) makes use of PetSc (development version) which further needs to be compiled with sieve, chaco (for graph partitioning) and boost. Either GNU or the Intel compilers can be used to build PetSc.

# module purge all
# module load mpich-gcc (invokes mpich-1.2.7p1 with gcc-3.3.5 compilers; or module load mpich-intel)

PetSc
# tar -xzf petsc-dev.tar.gz
# cd petsc-dev
# setenv PETSC_DIR `pwd`
# ./config/configure.py --with-clanguage=C++ --with-mpi-dir=/opt/mpich-1.2.7p1/32/p4-gcc-3.3.5 --with-blas-lapack-dir=/opt/intel/mkl721 --with-shared=0 --with-debugging=0 --with-sieve=1 --with-chaco=1 --download-chaco=1 --with-boost=1 --download-boost=1
# setenv PETSC_ARCH linux-gnu-cxx-opt
# make all

PyLith
# tar -xzf pylith3d-0.8.2.tar.gz
# cd pylith3d-0.8.2
Make sure that $PETSC_DIR and $PETSC_ARCH variables are set
# ./configure --prefix=$HOME/PyLith
# make
# make install

Update $PATH; delete/remove PetSc; create this file in $HOME/.pyre/pylith3d/ and test the installation

# qsub -I -l nodes=4:ppn=2
# module load mpich-gcc
# cd my_scratch_dir
# pylith3dapp.py --fileRoot=my_file --nodes=8 --asciiOutput=echo --ucdOutput=ascii

-Note that PyLith unlike GeoFEST does not use a single input (my_file here is the root file name for my_file.* files). Also make sure that the file pylith3dapp.py contains the proper paths


Misc

If for some reason you are interested in using GeoFEST v4.5 (Parallel) on a x86 desk/laptop then you will have to install old Intel 7.x compilers which are freely available for NON-COMMERCIAL USE.

If you are using Debian then simply follow these steps...

Download the make_deb_7 script (the script and instructions have been shamelessly stolen from here)
# chmod +x make_deb_7
# alien -k intel-ifc7-xxx.i386.rpm
# ./make_deb_7 intel-ifc7_xxx_i386.deb
# sudo dpkg -i intel-ifc7_xxx_i386.deb
Repeat the above for intel-icc7-xxx.i386.rpm and update $PATH (At this stage it is also a good idea to install the Intel MKL)
# source /opt/intel/compiler70/ia32/bin/ifcvars.csh
# source /opt/intel/compiler70/ia32/bin/iccvars.csh

Now you should have working ifc/icc/icpc compilers. However some users might experience problems with newer glibc (on etch and/or sid) and may see "undefined reference to `__ctype_b'" errors during linking. To fix this,

Download ctype.c (stolen from here)
# icc -c ctype.c
# cp ctype.o /opt/intel/compiler70/ia32/lib/ctype.o
# vi /opt/intel/compiler70/ia32/bin/ifc.cfg (and add "-Wl,/opt/intel/compiler70/ia32/lib/ctype.o" to the end of the first line)
Change icc.cfg and icpc.cfg as above

Now install mpich1
# tar -xzf mpich-1.2.6.tar.gz
# cd mpich-1.2.6
# setenv CC  icc
# setenv CXX icpc
# setenv FC  ifc
# setenv F90 ifc
# ./configure --with-arch=LINUX --prefix=/opt/mpich-1.2.6  --with-device=ch_p4 --enable-debug --without-mpe --enable-sharedlib
# make
# sudo mkdir /opt/mpich-1.2.6
# sudo make install
Update $PATH

Now you can build GeoFEST as shown above. If instead you want to use PyLith then build mpich using GNU compilers and build PetSc/PyLith as shown above. Also please note that if you're building PyLith then make sure that the python-dev package is installed.

To build PyLith on vega.rcac.purdue.edu (front end to the old Sun E10K and the newer F6800 SMP clusters) use GNU compilers as Sun's native compilers/MPI tend to give lots of errors while compiling petsc-dev. One easy way is to first to build blas/lapack (sunperf doesnt seem to work with GNU compilers) and mpich with GNU compilers and then build petsc-dev and PyLith (as shown above). To debug any PetSc errors first set the '-v DISPLAY' option in the qsub command and then run pylith with '--petsc.start_in_debugger' and '--petsc.display=vega-03.rcs.purdue.edu:XX.0' options.

Update: Installation instructions for PyLith v1.x are here