Software Ada / Kal

Modules

At the Ada/Kal system the modules environment is used by the user to specify which application/software he or she wants to use on the cluster. It is a very flexible solution were the user can add, remove or switch application/software by a single command.

The modules package allows dynamical modification of a user's environment (PATH etc.) via the use of modulefiles. A modulefile contains information needed to configure the user's shell for the desired application. Modules can be loaded and unloaded dynamically.

To see the available applications/software installed use:

module avail

To see what modulefiles are loaded at the moment use:

module list

To load/unload a modulefile making a certain application/compiler available or not use:

module load <modulefile>
module add <modulefile>
module unload <modulefile>
module rm <modulefile>

Example: To use the Intel compiler suite version 9.1 use:

module load intel/compilers/9.1

To add/remove a modulefile to your initializations files making the application etc. available or not for every new session use:

module initadd <modulefile>
module initrm <modulefile>

This will add the desired modulefile to your .bashrc/.cshrc file.

Note some applications need to load their environment on all nodes (this is noted in the section describing that program) then you have to load the module in your .bashrc/.cshrc with the initadd command. However, if you use several different modules it helps to keep track of your environment by putting the module load command in your job-script.

For more module commands and further info look at the man pages of modules, "man module".

Software

Compilers

Current compiler support:
Compiler	C/C++	f77/f90	compiler module	MPI module
gcc (3.4)	yes	yes/no	not needed	mpich/1.2.7/gcc/3.4
gcc (4.1)	yes	yes/yes	not needed	mpich/1.2.7/gcc/4.1
intel(9.1)	yes	yes/yes	intel/compilers/9.1	mpich/1.2.7/intel/9.1
intel(10.1)	yes	yes/yes	intel/compilers/10.1	Not yet
pgi(6.2)	yes	yes/yes	pgi/6.2	mpich/1.2.7/pgi/6.2
pgi(7.0)	yes	yes/yes	pgi/7.0	Not yet
pgi(8.0)	yes	yes/yes	pgi/8.0	Not yet

e.g.: to use the Portland group compilers your .bashrc/.cshrc should have the following:

module load pgi/8.0

GNU

With the linux installation comes the GNU compilers These are for 3.4.6

gcc - C/C++
g77 - Fortran77

and for 4.1.0

gcc4 - C/C++
gfortran - Fortran 77/90/95

The GNU compilers are the default ones, you don't have to give a module command

A not very well documented feature of the gprof command is the GMON_OUT_PREFIX environment variable. If this variable is set each thread of the executable will generate a gmon file named $GMON_OUT_PREFIX.pid. Each gmon file can then be analyzed separately or examined as a whole

gprof -s $GMON_OUT_PREFIX.*
gprof your_executable gmon.sum

For more information on the gprof command see the corresponding manual or info pages.

Intel

The intel compilers are

icc/icpc - C/C++
ifort - fortran77/fortran90

Portland Group

The Portland Group Compilers are:

pgf77 - Fortran 77 compiler
pgf90 - Fortran 90 compiler
pgf95 - Fortran 95 compiler
pgcc - C compiler
pgCC - C++ compiler

Use

Ex.: to use the Portland compilers 8.0

module load pgi/8.0

The user manuals are here: 6.2, 7.0 and 8.0.

Libraries

MKL

The Intel Math Kernel Libraries are installed. Currently installed versions are:

10.0
10.1

To use Intel MKL issue, for version 10.0,

module load intel/mkl/10.0

MKL includes optimized math routines such as BLAS/LAPACK, ScaLAPACK, sparse solvers, FFT together with wrappers for FFTW2.x and FFTW3.0, Vector Math Library, Vector Random Number Generators etc.

Manual/documentation can be found here: 10.0 and 10.1

CMKL

NOTE: Since version 10.0 of Intel MKL, CMKL and MKL has been merged into one product.

The intel Cluster Math Kernel Libraries are installed and paths are set by

module load intel/cmkl/9.0

CMKL includes optimized versions of fftw2, fftw3, blacs, lapack, scalapack...

Manual/documentation can be found here: 9.0.

PETSc

There are serial and parallel versions of PETSc available on Kal and Ada. The parallel versions are built with OpenMPI/1.3, and all versions are built with intel compilers and intel MKL versions 11.0 and 10.1, respectively.

There are no module files for PETSc. You will need to provide the path to the version of choice. The available PETSc versions are located relative to /c3se/apps/PETSc/ at

3.0.0-p3/intel11.0-mkl10.1-openmpi1.3
3.0.0-p3/complex-intel11.0-mkl10.1-openmpi1.3 (with fortran-kernels)
3.0.0-p3/debugging-intel11.0-mkl10.1-serial (with debugging enabled, no MPI)

For legacy use, the following versions are also available

2.3.3-p15/intel11.0-mkl10.1-serial
2.3.3-p15/intel11.0-mkl10.1-openmpi1.3

For version 3, the file conf/variables, relative its above listed path, contain various variable declarations to be included in a makefile. For version 2, the corresponding file is named bmake/common/variables.

For information about how to use PETSc, please consult documentation available at the PETSc web-page.

Other software

Abaqus

Abaqus is a commercial finite-element package mainly used in solid mechanics applications. Current installed versions:

6.5-6
6.6-1
6.6-5
6.7-2
6.7-5
6.8-2
6.9-2

use i.e.

module add abaqus/6.6-5

to set up the environment for Abaqus (substitute 6.6-5 for the version you want).

To use user-subroutines, compile your code to an object file (put -c on compile line) and use user=your_object_file.o as argument for Abaqus. Also put

module add abaqus/6.6-5 intel/compilers/10.1

in your job-script (or in your .cshrc/.bashrc, just like above substitute 6.6-5 and/or 10.1 for the versions you want).

Adams

ADAMS is a commercial dynamic system simulation software used for building and testing virtual prototypes of mechanical systems. Current installed version:

2007r1

use e.g.

module add adams/2007r1

to set up the environment for Adams.

Ansys

Current installed versions:

12.1

To use it, load the ansys-module. The Ansys program suite contains e.g.

Fluent
CFX
ICEM

Comsol Multi Physics

Current installed versions:

3.3a
3.5a

To use Comsol Multi Physics load the appropriate modulefile:

module load comsol/3.3a

If you, for example, want to run Comsol Multi Physics together with Matlab and the jobile bigjob.m use:

comsol matlab -ml -nodesktop -ml -nosplash -mlr bigjob

in your jobscript. Here the -ml flag is used to pass flags to Matlab. To use other Matlab version besides the default one use:

comsol matlab "path-to-matlab" -ml -nodesktop -ml -nosplash -mlr bigjob

CP2K

CP2K is available on Ada and Kal.

"CP2K is a freely available (GPL) program, written in Fortran 95, to perform atomistic and molecular simulations of solid state, liquid, molecular and biological systems. It provides a general framework for different methods such as e.g. density functional theory (DFT) using a mixed Gaussian and plane waves approach (GPW), and classical pair and many-body potentials.", cp2k.berlios.de.

The currently installed version of CP2K is 2.0.1, it is built using intel ifort 9.1, and linked against intel MKL 10.1 and OpenMPI 1.3. There is a meta-module (cp2k/working-env) which resolves the necessary dependencies.

Example how to use CP2K in a batch script:

module load cp2k/working-env
cd $PBS_O_WORKDIR
mpirun cp2k <inputfile.inp>

CPMD

CPMD is available on KAL. The CPMD code is a parallellised plane wave/pseudopotential implementation of Density Functional Theory, particularly designed for ab-initio molecular dynamics. CPMD is copyrighted jointly by IBM Corp and by Max Planck Institute, Stuttgart, and is distributed free of charge to non-profit organizations.

Currently installed version

3.11.1

To run CPMD (cpmd.x) in parallel on KAL you need to load the environment on all nodes with

module initadd cpmd/3.11.1

You submit with

nproc=`cat $PBS_NODEFILE|wc -l`
mpirun -m $PBS_NODEFILE -np $nproc cpmd.x cpmd_input_file

in your job script.

For more information on CPMD point your browser towards www.cpmd.org.

Diana

Diana is a commercial finite-element package mainly used in solid mechanics applications.

Current installed versions:

9.2
9.3 manuals here

Fire

Fire is a commercial software package for CFD.

Currently installed version:

v2008.2
v2009.1

To use Fire load the appropriate modulefile

module load fire/2009

Fluent

Fluent is a commercial software package for CFD.

Currently installed version: Note: Ada only!

6.3.26

To use Fluent load the appropriate modulefile

module load fluent/6.3.26

and start with the following arguments: -pethernet -ssh -cnf=$TMPDIR/mpichnodes -g

GAMESS

The General Atomic and Molecular Electronic Structure System (GAMESS) is a general ab initio quantum chemistry package. GAMESS is available under the condition that published work include citation of the program

"General atomic and molecular electronic structure system, M.W.Schmidt,
K.K.Baldridge, J.A.Boatz, S.T.Elbert, M.S.Gordon, J.H.Jensen, S.Koseki,
N.Matsunaga, K.A.Nguyen, S.J.Su, T.L.Windus, M.Dupuis, J.A.Montgomery, Journal
of Computational Chemistry, vol. 14, pages 1347-1363, 1993."

as well as citations to papers describing the implementation in cases when certain features of the code have been used.

Currently installed version

March-24-2007-R3

To run GAMESS (rungms) on ADA you need to load the environment on all nodes with

module initadd gamess/intel-9_1/March-24-2007-R3

You submit with

rungms input_file

in your job script. The number of cpus is taken from the

#PBS -l nodes=2:ppn=4

line in your job script. For more information on GAMESS, e.g. a manual, point your browser towards GAMESS

Gaussian

Gaussian is a electronic structure program. Gaussian is used by chemists, chemical engineers, biochemists, physicists and others for research in established and emerging areas of chemical interest.

Currently installed versions:

G03 Revision D.02
G03 Revision E.01

GaussView version 3.09, which is a graphical user interface (GUI) for Gaussian, is also available.

NOTE: if you want to use Gaussian send an email to our support address.

To use Gaussian and GaussView load the appropriate modulefile, in this case Revision E.01:

module load gaussian/g03-RevE.01

After the module for Gaussian is loaded you can start GaussView by issuing the command:

gv

The installed version of Gaussian on Ada/Kal does not implement LINDA i.e. no mpi. Gaussian can run in parallell on one node over several CPU:s/core:s through SMP. To use SMP over 4 core:s you need to put

%NProcShared=4

in your Gaussian input file just before the Route Section.

The online manual for Gaussian can be found here.

Matlab

Matlab is a commercial package for numerical calculations. Current installed versions:

7.3
7.4
7.5
7.7
7.9

I.e. to use matlab 7.7 load the 7.7 matlab module with

module load matlab/7.7

To submit your bigjob.m script to a queue you put

matlab -nojvm -nodesktop -r bigjob

in your job script, see Submitting jobs for more details on submitting jobs.

PCT (DCT)

The Parallel Computing Toolbox (formerly Distributed Computing Toolbox) is installed on the cluster. This means that you can start testing. Chalmers do not at the moment have a license for the Distributed Computing Server (formerly Distributed Computing Engine) which means that you can't use the full functionality as described for our queuing system torque here. You can however test using the local scheduler as explained here. There is a further limit, the local scheduler will only start up to eigth tasks a time, which means that you are limited to using one node (#PBS -l nodes=1:ppn=4 or #PBS -l nodes=1:ppn=8 depending on the system used). The last part of your submitting script could look like this,

cd $PBS_O_WORKDIR
module load matlab/7.5
matlab -nodisplay -r local_sched

where local_sched.m resides in your submit directory and could e.g. look like

sched = findResource('scheduler','type','local');
set(sched, 'DataLocation', './')
job1 = createJob(sched)
createTask(job1, @rand, 1, {{3,3} {3,3} {3,3} {3,3} {3,3}});
submit(job1)
waitForState(job1).

Compare with the example here.

When the job is finished you can start up matlab in that directory and type

sched = findResource('scheduler','type','local');
set(sched, 'DataLocation', './')
job1=findJob(sched)
results = getAllOutputArguments(job1);
results{1:5}

If you have run several jobs findJob will return an array. As mentioned in the beginning this in not the way Mathworks want you to use the toolbox but it gives you the possibility to evaluate and test the toolbox. Any feedback to support is welcome, this of course includes the question whether there is a great interest in the "Server".

Molcas

Molcas is a quantum chemistry software developed by scientists to be used by scientists. It is not primarily a commercial product and it is not sold in order to produce a fortune for its owner (the Lund University). The authors have tried in MOLCAS to assemble their collected experience and knowledge in computational quantum chemistry. MOLCAS is a research product and it is used as a platform by the Lund quantum chemistry group in their work to develop new and better computational tools in quantum chemistry.

Molcas is available for all C3SE users but it is still a licensed product and may not be used for commercial purposes.

Currently installed versions:

7.4.080 (built with Intel compilers)

To use Molcas load either one of the Molcas modules:

module load molcas/7.4.080

For further info see the Molcas homepage, the online manual can be found here.

OpenFOAM

OpenFOAM is an open-source C++ library for solving partial differential equations. It is mainly used for computational fluid dynamics, for which there are many implemented solvers and different kinds of utilities. For further info see the OpenFOAM homepage, or contact hani(at)chalmers.se.

Currently installed versions:

1.4.1_patch (this is the 1.4.1 version with bug-fixes from the OpenFOAM-Extend svn repository at SourceForge. You can see which files have been patched by opening /c3se/apps/OpenFOAM/OpenFOAM-1.4.1_patch/svnPatch.log, and you can see the latest date of patching by looking at the creation date of that file. If you need a more recent patch, contact hani(at)chalmers.se. Note that this version may be patched any time!)
1.5.x (The present version was downloaded 081008, it is not yet decided how to deal with upgrades)
1.5.x_Debug (For debugging. Same patch as openfoam/1.5.x)
1.5.x_Prof (For profiling. Same patch as openfoam/1.5.x)

Note that the FoamX GUI is not supported at Ada!

It is NOT RECOMMENDED to load the OpenFOAM modules in ~/.bashrc. It is much better to load the modules in the submit script, so that each submit determines its own environment. Use one of the following in your submit script (see example below):

module load openfoam/1.4.1_patch mpich/1.2.7-PIC/gcc/4.1 cmake ; export args='. .'
module load openfoam/1.5.x ; export args='-case .'
module load openfoam/1.5.x_Debug ; export args='-case .'
module load openfoam/1.5.x_Prof ; export args='-case .'
module load openfoam/svea/1.5.x ; export args='-case .'

For new users of OpenFOAM, submit the following script (see Submitting jobs), which will run the interFoam damBreak tutorial in the directory where you submit from (the tutorial will be copied to a local damBreak directory):

Submit-script. Choose which OpenFOAM version to use by uncommenting ONE module line

#PBS -q ada
#For Svea, use '-q svea' and the openfoam/svea/1.5.x module below
#PBS -l walltime=01:00:00
#nodes x ppn must be the same as numberOfSubdomains in system/decomposeParDict:
#PBS -l nodes=1:ppn=4
#PBS -N damBreak
# End of arguments to qsub
# Load the OpenFOAM module for this specific job
#---------------- Uncomment ONE module line -------------------------
#module load openfoam/1.4.1_patch mpich/1.2.7-PIC/gcc/4.1 cmake ; export args='. .'
module load openfoam/1.5.x ; export args='-case .'
#module load openfoam/1.5.x_Debug ; export args='-case .'
#module load openfoam/1.5.x_Prof ; export args='-case .'
#module load openfoam/svea/1.5.x ; export args='-case .'
#--------------------------------------------------------------------
export P4_GLOBMEMSIZE=10240000
cd $PBS_O_WORKDIR
cp -r $FOAM_TUTORIALS/interFoam/damBreak .
cd damBreak
blockMesh `echo $args` >& log_blockMesh #Generates the grid
setFields `echo $args` >& log_setFields #Sets the initial fields
decomposePar `echo $args` >& log_decomposePar #Decomposes for parallel simulation
mpiexec `which interFoam` `echo $args` -parallel >& log #Runs interFoam in parallel
reconstructPar `echo $args` >& log_reconstructPar #Collects the decomposed results
rm -rf processor* #Deletes the decomposed files

View the results using (must be run from inside the damBreak direktory. Pick one of these modules):

cd damBreak
#module load openfoam/1.4.1_patch mpich/1.2.7-PIC/gcc/4.1 cmake ; export args='. .'
module load openfoam/1.5.x ; export args='-case .'
paraFoam `echo $args`

ParaView

ParaView is an open-source, multi-platform application designed to visualize data sets of size varying from small to very large.

Currently installed versions:

2.6.2
3.0.2

To use ParaView load either one of the ParaView modules:

module load paraview/2.6.2

module load paraview/3.0.2

For further info see the ParaView homepage.

Star-CD

Star-CD is a commercial package for CFD calculations. Note: No general license is available. Only users with a valid license can run Star-CD.

Currently installed versions

4.02

To run Star-CD in parallell on Ada with MPI start with the following arguments:

star -dp -mpi=mpich `cat $TMPDIR/mpichnodes`

to run in parallel on KAL use

-mpi=hp

VTune

Intel VTune Performance Analyzer is a tool for application performance tuning. It uses a graphical user interface and no recompiles are required. It is compiler and language independent so it works with C, C++, Fortran etc.

Currently installed version:

9.1

To use VTune load the appropriate modulefile:

module load intel/vtune/9.1

The GUI for VTune is accessed by issuing the command:

vtlec

A command line interface is also available:

vtl

Online documentation for the Intel VTune Performance Analyzer can be found here

Software Ada / Kal

Contents

Modules

Software

Compilers

GNU

Intel

Portland Group

Libraries

MKL

CMKL

PETSc

Other software

Abaqus

Adams

Ansys

Comsol Multi Physics

CP2K

CPMD

Diana

Fire

Fluent

GAMESS

Gaussian

Matlab

PCT (DCT)

Molcas

OpenFOAM

ParaView

Star-CD

VTune

Views

Personal tools

Navigation

Protected Pages

Search

Toolbox