Software Beda

From C3SE
Jump to: navigation, search

Modules

--IMPORTANT-- Do NOT copy your .bashrc or .cshrc script from Ada/Kal.

At the Beda system the modules environment is used by the user to specify which application/software he or she wants to use on the cluster. It is a very flexible solution were the user can add, remove or switch application/software by a single command.

The modules package allows dynamical modification of a user's environment (PATH etc.) via the use of modulefiles. A modulefile contains information needed to configure the user's shell for the desired application. Modules can be loaded and unloaded dynamically.

To see the available applications/software installed use:

module avail

Some software aimed for more advanced users and for testing/debugging can be found in the "extras" module see,

module load extras

To see what modulefiles are loaded at the moment use:

module list

To load/unload a modulefile making a certain application/compiler available or not use:

module load <modulefile>
module add <modulefile>
module unload <modulefile>
module rm <modulefile>

To see what a modulefile does, use

module showme <modulefile>

This will display what environment variables the modulefile sets or modifies.

Installing new software

For any software installation requests, please have a look at getting support.

Development tools

Intel Cluster Studio XE is available and contains the recommended tools for use on Beda. More information on what tools it contains can be found here. (Note that all the tools in Intel Cluster Studio XE not are included in the listings below!)

Compilers

Our default recommendation is to use the Intel compilers. You can add them to your environment by simply issuing the command module load intel-compilers.

The following compilers are available on the Beda cluster:

  • Intel compilers for C (icc), C++ (icpc), and Fortran (ifort) are available via the module intel-compilers.
    • The avalibe versions are, 10.1, 11.1 (default), 12.0 and 13.0
  • GNU Compiler Collection (GCC). There are three available versions installed.
    • Version 4.1.2 is the default system compiler and is the compiler used for compiling the operating system itself. There are C (gcc), C++ (g++), and Fortran (gfortran) compilers. This version is present even if no compiler module have been loaded.
    • Version 4.4.2 is available through the module system and replaces version 4.1.2 if loaded.
    • Version 3.4.6. This is a legacy compiler for backward compatibility with older RedHat systems. C (gcc34), C++ (g++34), and FORTRAN77 (g77) compilers are available. Note that the legacy compilers are always available and never replaced by the module system.

Debuggers

  • Intel (via the intel-compilers module), idb
  • GNU debugger 7.0 ("system debugger", always available), gdb
  • Allinea DDT version 3.2. Allinea DDT is a powerful debugger with MPI capabilities. User guide can be found here

To use Allinea DDT issue

module load allinea/ddt/3.2-24924

The module creates the environment variable ALLINEA_EXAMPLES, pointing to a set of exercises and a walkthrough for the debugger. Do

cp -r $ALLINEA_EXAMPLES <suitable test directory>

and look at the instructions. In order to debug larger programs, you must submit the job through the queuing system. Allinea supports the PBS system used on Beda. In the "Job Submission" section of the MPI options, you can select the beda.qtf or beda-with-account.qtf templates, and let Allinea submit the job. Doublecheck that the "PROCS_PER_NODE_TAG" is set to 8, click OK, and check the other values, including walltime limit. Then click submit, and Allinea should submit the job. Note that you need to keep Allinea running until the job starts. Once it's started, you can use the debugger in the normal way.

For MPI programs the following debuggers/profilers are installed

and found by issuing,

module load extras; module avail

Libraries

MKL

The Intel Math Kernel Libraries are installed. Currently installed versions are:

  • 10.2.1.017
  • 10.2.4.032 (default)
  • 10.3.0.084
  • 11.0.079

To use Intel MKL issue, for version 10.2,

module load mkl/10.2

MKL includes optimized math routines such as BLAS/LAPACK, ScaLAPACK, sparse solvers, FFT together with wrappers for FFTW2.x and FFTW3.0, Vector Math Library, Vector Random Number Generators etc.

Manual/documentation can be found here: 10.2

See also Intel's: "MKL Link line advisor".

Note that the library path has changed from v10.2 to v10.3

  • 10.2, ${MKL_ROOT}/lib/em64t
  • 10.3, ${MKL_ROOT}/lib/intel64

GotoBLAS

GotoBLAS is an open source Basic Linear Algebra Subprograms (BLAS) implementation infamous for its good performance. The library resides at, /beda/sw/apps/GotoBLAS2.

PETSc

There are parallel versions of PETSc available on Beda. The parallel versions are built with OpenMPI/1.4, and all versions are built with Intel compilers and intel MKL versions 11.1 and 10.2, respectively.

There are no module files for PETSc. You will need to provide the path to the version of choice. The available PETSc versions are located relative to /beda/sw/apps/PETSc/ at

  • 3.0.0/p11/intel11.1-mkl10.2-openmpi1.4
  • 3.1/p4/intel11.1-mkl10.2-openmpi1.4
  • 3.2/p3/intel11.1-mkl10.2-openmpi1.4

For version 3, the file conf/variables, relative its above listed path, contain various variable declarations to be included in a makefile. Version 3.2 is compiled with SPOOLES support.

For information about how to use PETSc, please consult documentation available at the PETSc web-page.

SLEPc

SLEPc is the "Scalable Library for Eigenvalue Problem Computations" and relies on PETSc for basic linear algebra routines. The version(s) installed on Beda resides at /beda/sw/apps/SLEPc/ at

  • 3.1/p6/intel11.1-mkl10.2-openmpi1.4/ using PETSc 3.1

For more info on how to use the library see the project homepage.

FFTW

FFTW or, "The Fastest Fourier Transform in the West" is an open source high performance fast Fourier transform implementation. The library resides at, /beda/sw/apps/fftw.

Boost

Boost is a set of open source peer-reviewed portable C++ libraries. Several parts of Boost have been added to the C++ standard in the so called, Library Technical Reports, managed by the C++ standards committee. The library resides at, /beda/sw/apps/boost.

GNU Scientific Library (GSL)

The GNU Scientific Library is an open source library in C and C++ providing a wide range of numerical routines. The library resides at, /beda/sw/apps/gsl

HDF5

HDF5 is an open source library, data model, and file format for storing and managing data. It can be used for high performance, high volume, complex data disk I/O. The library resides at, /beda/sw/appps/hdf5.

NetCDF

NetCDF (Network Common Data Form) is a freely available library, and data formats for machine-independent data storage. The library resides at, /beda/sw/appps/netcdf.

MPI implementations

We recommend using IntelMPI (see above). Mvapich is primarily available as a fallback or for troubleshooting purposes.

  • IntelMPI
  • OpenMPI
  • mvapich2

If compiling and linking Fortran90 programs using the mpi-module (_Fortran_ module, i.e., the code contains use mpi), it is important not to mix compiler implementations. Hence, in such situations one should use the same compiler for the program as was used to build the MPI implementation.

General Purpose

Matlab

Matlab is a commercial package for numerical calculations. Current installed versions:

  • 7.9
  • 7.10
  • 7.12
  • 7.14

I.e. to use matlab 7.9 load the 7.9 matlab module with

module load matlab/7.9

To submit your bigjob.m script to a queue you put

matlab -nojvm -nodesktop -r bigjob

in your job script.

To get Parallel Toolbox working for more than one job at a time on the cluster (with one matlab instance per job) the "Data Location" directory must be individually set for each job. Thus before opening a new matlabpool issue the commands:

[~, tmpdir] = system('echo $TMPDIR');
sched = findResource('scheduler', 'configuration', 'local');
sched.DataLocation = tmpdir(1:end-1);

matlabpool open local

Octave

Octave is an open source software for high level scripting of numerical calculations.

R

The R project is an open source software for statistical computing.

Python

Python is a general purpose (interpreted) programming language. There are several python versions available; in addition to the versions explicitly available via modules, there is also a system version of python which is present by default. The system version of python should normally only be used for system tools and utilities. For each of the python versions provided via modules, there are normally also a set of compatible python tools such as numpy, scipy, mayavi, etc.

E.g., to following example give access to python and numpy (which depends on mkl)

$ module load python mkl numpy
$ module list
Currently Loaded Modulefiles:
  1) python/py26/2.6.4        2) mkl/10.2/4.032           3) numpy/1.5.1rc2-mkl10.2


Continuum Mechanics

Abaqus

Abaqus is a commercial finite-element package mainly used in solid mechanics applications. Current installed versions:

  • 6.9-2
  • 6.10-2
  • 6.11-1

use i.e.

module add abaqus/6.11-1

to set up the environment for Abaqus.

To use user-subroutines, compile your code to an object file (put -c on compile line) and use user=your_object_file.o as argument for Abaqus. Also put

module add abaqus/6.10-2 intel-compilers/11.1

in your job-script (or in your .cshrc/.bashrc, just like above substitute 6.11-1 and/or 11.1 for the versions you want).

Ansys

Current installed versions:

  • 12.1
  • 13.0
  • 14.0
  • 14.5
  • 15.0

To use it, load the ansys-module. The Ansys program suite contains e.g.

  • Fluent
  • CFX
  • ICEM

Suitable arguments for multi-node parallel Fluent:

-pinfiniband -ssh -cnf=$TMPDIR/mpichnodes -g

For multi-node parallel CFX, add the following line to your job script:

nodes="`sed -e :a -e N -e 's/\n/,/' -e ta $TMPDIR/mpichnodes`"

and the following to your cfx-commandline:

-par -par-dist $nodes -start-method "HP MPI Distributed Parallel"

Diana

DIANA is a proprietary FEM program aimed at civil engineering design and assessment projects in concrete, steel, soil, rock and soil-structure interaction. (User license required)

Comsol Multi Physics

Current installed versions:

  • 3.5a
  • 4.0a
  • 4.2

To use Comsol Multi Physics load the appropriate modulefile:

module load comsol/3.5a

If you, for example, want to run Comsol Multi Physics together with Matlab and the jobile bigjob.m use:

comsol matlab -ml -nodesktop -ml -nosplash -mlr bigjob

in your jobscript. Here the -ml flag is used to pass flags to Matlab. To use other Matlab version besides the default one use:

comsol matlab "path-to-matlab" -ml -nodesktop -ml -nosplash -mlr bigjob

To run example 'coupled_vibrations.mph' using Comsol 4.0a in parallel across 2 nodes use:

#PBS -l nodes=2:ppn=8

module load comsol/4.0a

sort $TMPDIR/mpichnodes | uniq > $TMPDIR/comsolenodes
NODES=`wc -l $TMPDIR/comsolenodes | awk '{print $1}'`
CORES=`head -1 $TMPDIR/lamnodes | awk -F'cpu=' '{print $2}'`

comsol -nn $NODES -mpirsh ssh mpd boot -f $TMPDIR/comsolenodes
comsol -nn $NODES -np $CORES batch -inputfile coupled_vibrations.mph
comsol -mpirsh ssh mpd allexit

Fire

Fire is a commercial software package for CFD.

Currently installed version:

  • v2009.1
  • v2010

To use Fire load the appropriate modulefile

module load fire/2009

module load fire/2010

OpenFOAM

OpenFOAM is an open-source C++ library for solving partial differential equations. It is mainly used for computational fluid dynamics, for which there are many implemented solvers and different kinds of utilities. For further info see the OpenFOAM homepage, or contact hani(at)chalmers.se.

Currently installed versions:

  • 1.5-dev (Revision 1629. Copy and modify /beda/sw/etc/modulefiles/openfoam/1.5-dev to make your own installation)
  • 1.6.x (Snap-shot of one revision. Copy and modify /beda/sw/etc/modulefiles/openfoam/1.6.x to make your own installation)
  • 1.7.1 released 26/08/10 (Copy and modify /beda/sw/etc/modulefiles/openfoam/1.7.1 to make your own installation)
  • 2.0.1 released 04/08/11 (Copy and modify /beda/sw/etc/modulefiles/openfoam/2.0.1 to make your own installation)

It is NOT RECOMMENDED to load the OpenFOAM modules in ~/.bashrc. It is much better to load the modules in the submit script, so that each submit determines its own environment. Use one of the following in your submit script (see example below):

  • module load openfoam/1.5-dev
  • module load openmpi/1.4-gcc41 openfoam/1.6.x
  • module load openmpi/1.4-gcc41 openfoam/1.7.1
  • module load openmpi/1.4-gcc41 openfoam/2.0.1

If you need to compile code that links with OpenFoam (1.6.x, 1.7.1 or 2.0.1) the same compiler used to build the software, gcc 4.4, should be used. To get the correct environment load also the gcc/4.4 module.

module load gcc/4.4/4.4.2

For new users of OpenFOAM, submit the following script, which will run the interFoam damBreak tutorial in the directory where you submit from (the tutorial will be copied to a local damBreak directory):

Submit-script. Choose which OpenFOAM version to use by uncommenting ONE module line, and ONE line that copies the damBreak tutorial (default is for OpenFOAM-1.5-dev)

#PBS -q beda
#PBS -l walltime=01:00:00
#nodes x ppn must be the same as numberOfSubdomains in system/decomposeParDict:
#PBS -l nodes=1:ppn=8
#PBS -N damBreak
# End of arguments to qsub
# Load the OpenFOAM module for this specific job
#---------------- Uncomment ONE module line -------------------------
module load openfoam/1.5-dev #OpenFOAM-1.5-dev
#module load openmpi/1.4-gcc41 openfoam/1.6.x #OpenFOAM-1.6.x
#module load openmpi/1.4-gcc41 openfoam/1.7.1 #OpenFOAM-1.7.1
#module load openmpi/1.4-gcc41 openfoam/2.0.1 #OpenFOAM-2.0.1
#--------------------------------------------------------------------
export P4_GLOBMEMSIZE=10240000
cd $PBS_O_WORKDIR
#---------------- Uncomment ONE of the lines ------------------------
cp -r $FOAM_TUTORIALS/interFoam/damBreak . #OpenFOAM-1.5-dev
#cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . #OpenFOAM-1.6.x, OpenFOAM-1.7.1 and OpenFOAM-2.0.1
#--------------------------------------------------------------------
cd damBreak
blockMesh >& log_blockMesh #Generates the grid
setFields >& log_setFields #Sets the initial fields
decomposePar >& log_decomposePar #Decomposes for parallel simulation
mpirun -npernode 4 `which interFoam` -parallel >& log #Runs interFoam in parallel on 4 cores on one node
                                                      #npernode 4 forces mpirun to run only 4 processes
                                                      #instead of 8. When you run later you should fill up
                                                      #the nodes with 8 processes and omit the npernode flag!!!
reconstructPar >& log_reconstructPar #Collects the decomposed results
rm -rf processor* #Deletes the decomposed files

It may be beneficial to run OpenFOAM at Beda with 4 processes per node, and to make sure that there are two processes per socket. With OpenMPI, this is done by changing the npernode flag after mpirun to:

-npersocket 2

This will start four processes per node you allocate (each node has two sockets), and make sure that those processes run two per socket. Then of course decompose your case so that it has four parts per node that you ask for when you submit your script.

How to view the results:

For version 1.5-dev, we did not yet have time to make paraFoam work. Instead you do the following:

cd damBreak
module load openfoam/1.5-dev
module load paraview/3.6.2
foamToVTK
paraview

Use File/Open and browse for the VTK folder to find the results.

For version 1.6.x (and in some cases, like the damBreak case you just run, this also works for 1.5-dev results):

cd damBreak
module load openmpi/1.4-gcc41 openfoam/1.6.x
paraFoam

For version 1.7.1:

cd damBreak
module load openmpi/1.4-gcc41 openfoam/1.7.1
paraFoam

For version 2.0.1:

cd damBreak
module load openmpi/1.4-gcc41 openfoam/2.0.1
paraFoam

PowerFLOW

PowerFLOW is a proprietary computational fluid dynamics program employing the lattice-Boltzman method. (User license required)

Edge

Edge is a proprietary Computational Fluid Dynamics program for solving the compressible Reynolds-Averaged Navier-Stokes (RANS) equations on unstructured grids. (User license required)

Electromagnetism

Efield

Efield is a proprietary program for 3D electromagnetic calculations. (User license required)


Bio-Informatics

BioPerl

BioPerl is an open source software bundle with tools script-able in Perl used in the field of Biology.

BioPython

BioPython is an open source software bundle with tools script-able in Python used in the field of Biology.

Blast

Basic Local Alignment Search Tool (BLAST) is a program for comparison of nucleotide or protein sequences obtained from gene sequencing.

Atomistic Simulation

Abinit

Abinit is an opensource pseudopotential, planewave DFT code. Invoke by

module load intel-compilers mkl openmpi abinit

ASE

ASE or Atomic Simulation Environment, is a Python module for manipulating atoms, analyzing simulations, visualization etc., that can interface numerous programs. For example Abinit, Dacapo, GPAW, Vasp and Siesta.

module load mkl python numpy ase

CP2K

CP2K is an opensource molecular dynamics program with force calculation from, DFT using a mixed plane-wave and gaussian basis, or classical two or many-body potentials. Access and more info is given issuing,

module help cp2k
module load intel-compilers mkl openmpi cp2k

Dacapo

Dacapo is an opensource ultrasoft pseudopotential plane-wave DFT code.

module load mkl openmpi python numpy ScientificPython ase dacapo

Espresso

Espresso is an opensource pseudopotential (ultrasoft and norm-conserving) plane-wave basis DFT program.

 module load mkl openmpi espresso

Gaussian

Gaussian is a proprietary electronic-structure program. (User license required) Two versions are installed: Gaussian 09-B.01 (default) and 03-E.01.

To use Gaussian 09-B.01 issue the command:

 module load gaussian/g09-b.01

To use Gaussian 03-E.01 issue the command:

 module load intel-compilers/11.1/069 mkl/10.2/4.032 gaussian/g03-e.01

GPAW

GPAW is an opensource projector-augumented wave real-space DFT program.

module load mkl python numpy gpaw

GROMACS

GROMACS is an opensource classical molecular dynamics code.

module load openmpi gromacs

Lammps

Lammps is an opensource classical molecular dynamics code.

module load mkl openmpi lammps

Octopus

Octopus is an opensource pseudopotential real-space basis DFT and TDFT code.

module load intel-compilers mkl openmpi octopus

Phonopy

Phonopy is an opensource utility for calculating phonon spectra using DFT results. Currently supports Vasp and Wien2k.

module load mkl python numpy phonopy

Schrodinger

Schrodinger is a proprietary molecular modeling platform. (User license required)

module load schrodinger

Siesta

Siesta is a proprietary linear scaling DFT program using atomic-orbital basis sets and norm-conserving pseudo potentials. (User license required)

module load openmpi siesta

VMD

Visual Molecular Dynamics (VMD) is primary a visualization tool for atomistic simulations that are able to read a multitude of formats.

module load vmd

OTHER SOFTWARE

Information about other software available on Glenn can be found HERE