Here are some brief notes I made (mostly for my own use) while
installing PyLith and GeoFEST on RCAC's recycled x86 linux cluster
(radon). Please note that PyLith is in extensive development and
installation instructions/requirements keep on changing frequently.
For more information follow the relevant documentation.
GeoFEST
GeoFEST v4.5 (Parallel) uses ParMetis for graph partitioning and
the Pyramid AMR library (only for domain decomposition and not for
AMR). As of now it only works with Intel 7.x compilers (if you want
to try 8.x/9.x compilers then you might want to have a look
here).
# module purge all
# module load mpich-1.2.6-intel/71 (invokes mpich-1.2.6 with
intel-7.1 Compilers)
ParMetis
# tar -xzf ParMetis-3.1.tar.gz
# cd ParMetis-3.1
# make
Pyramid AMR lib
# tar -xzf Pyramid-2.0.tar.gz
# cd Pyramid-2.0
# make -f Makefile.Intel distclean
# cd Pyramid
# ln -s -f ../../ParMetis-3.1 ParMetis
# cd ..
# make -f Makefile.Intel
GeoFEST
# tar -xzf GeoFEST-4.5.tgz
# cd GeoFEST-4.5
# ln -s -f ../Pyramid-2.0 Pyramid
# cd geofest
Make sure that the file i386-linux-Intel.mk (or
$MACHTYPE-$OSTYPE-$F90TYPE.mk) contains
FORTRAN_LIBS=/opt/intel/compiler70/ia32/lib
fortran_libs=-lCEPCF90 -lPEPCF90 -lintrins
-lF90
Switch 'sig_ZZ' with 'sig_XY' in inphase.c:946
# make -f Makefile.Parallel F90TYPE=Intel
Now update $PATH and test the installation
# qsub -I -l nodes=4:ppn=2
# module load mpich-1.2.6-intel/71
# cd my_scratch_dir
# gfmeshparse my_file
(gfmeshparse is a utility that comes with GeoFEST to create a
secondary file my_file.jpl for the parallel run; also do not
include any velocity bc's in my_file, instead first create the file
my_file.jpl and then add the velocity bc's in my_file or else,
modify the gfmeshparse source to skip the velocity bc data in
my_file)
# mpirun $PBS_NODEFILE -np 8 `which GeoFEST` my_file
Here are the CIG strike-slip benchmark runtimes on an Itanium (dual
processor) cluster...
Resolution (m)
Nodes
CPU's
Time (mins)
CPU utilization
(%)
1000 m
15625
2
4
0:02:31
0:01:26
95
87
500 m
117649
2
4
8
16
32
2:06:25
1:06:37
0:33:53
0:22:16
0:09:54
97
94
88
71
75
250 m
912673
24
32
1:04:40
0:49:13
80
78
PyLith
Installation instructions for PyLith v1.x are here
Misc
If you are interested in using GeoFEST v4.5 (Parallel) on a x86
desk/laptop or on a cluster which doesnt have Intel 7.x compilers
then you will have to install the old Intel 7.x compilers.
If you are using Debian then simply follow these steps...
Download the make_deb_7 script (the script
and instructions have been shamelessly stolen from here)
# chmod +x make_deb_7
# alien -k intel-ifc7-xxx.i386.rpm
# ./make_deb_7 intel-ifc7_xxx_i386.deb
# sudo dpkg -i intel-ifc7_xxx_i386.deb
Repeat the above for intel-icc7-xxx.i386.rpm and update $PATH (At
this stage it is also a good idea to install the Intel MKL)
# source /opt/intel/compiler70/ia32/bin/ifcvars.csh
# source /opt/intel/compiler70/ia32/bin/iccvars.csh
Now you should have working ifc/icc/icpc compilers. However some
users might experience problems with newer glibc (on etch and/or
sid) and see "undefined reference to `__ctype_b'" errors
during linking. To fix this,
Download ctype.c (stolen from here)
# icc -c ctype.c
# cp ctype.o /opt/intel/compiler70/ia32/lib/ctype.o
# vi /opt/intel/compiler70/ia32/bin/ifc.cfg (and add
"-Wl,/opt/intel/compiler70/ia32/lib/ctype.o" to the end of the
first line)
Change icc.cfg and icpc.cfg as above
Now install mpich1
# tar -xzf mpich-1.2.6.tar.gz
# cd mpich-1.2.6
# setenv CC icc
# setenv CXX icpc
# setenv FC ifc
# setenv F90 ifc
# ./configure --with-arch=LINUX --prefix=/opt/mpich-1.2.6
--with-device=ch_p4 --enable-debug --without-mpe
--enable-sharedlib
# make
# sudo mkdir /opt/mpich-1.2.6
# sudo make install
Update $PATH