OpenFOAM

OpenFOAM is an open-source C++ library for solving partial differential equations. It is mainly used for computational fluid dynamics, for which there are many implemented solvers and different kinds of utilities. For further info see the OpenFOAM homepage.

We install OpenFOAM as a complete package, i.e. the module OpenFOAM/4.0 contains the OpenFOAM version and all the packages needed to use this OpenFOAM version rather than installing them separately and creating multiple modules.

Available OpenFOAM versions

To see available OpenFOAM versions write in the terminal:

module spider OpenFOAM

OpenFOAM best practice

When running OpenFOAM it is important to run from your NOBACKUP directory. This is because OpenFOAM generate a tremendous amount of files which will fill up the quota in your BACKUP directory in no time. It is therefore recommended to source OpenFOAM the following way:

ml intel/2017a OpenFOAM/4.1
. $FOAM_BASH WM_PROJECT_USER_DIR=your_openfoam_path/$USER-$WM_PROJECT_VERSION

This will tell OpenFOAM that the user working directory is in the given directory, under version-specific directories.

New users should then (once) create some directory structure, by:

mkdir -p $FOAM_RUN

This will give you a run-directory in the given directory, which you can easily reach by the alias run (as long as you source as described above).

It is recommended to create some aliases for the different versions, at the end of your ~/.bashrc file, as:

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi`

alias OF40='ml intel/2017a OpenFOAM/4.1;. $FOAM_BASH WM_PROJECT_USER_DIR=your_openfoam_path/$USER-$WM_PROJECT_VERSION'

Whenever you open a new terminal window you can then simply activate whichever version you like by each alias (e.g. "OF40"). It is not recommended to switch between versions in the same terminal window! Open a new terminal window for each version.

Note that it is NOT RECOMMENDED to explicitly load the OpenFOAM modules in ~/.bashrc. It is MUCH better to load the modules in the submit script, so that each submit determines its own environment. See the submit script below. In the submit script you should write out the same contents as in the alias, as described above (and do not use the alias itself - it has been problematic).

OpenFOAM on Vera

Hyper-treading(HT) is enabled on Vera and has been evaluated for OpenFOAM. OpenFOAM showed a small benefit from HT and the default settings should be used. To read more about HT on Vera, see Vera.

Submitting OpenFOAM jobs

Submit the following shell script, which will copy and run the interFoam damBreak tutorial (note that you have to change information to match your project and directories):

#!/usr/bin/env bash
#SBATCH -A PROJECT_NUMBER -p hebbe
#SBATCH -n 4
#SBATCH -t 01:00:00
#-----------------------------------------------------------
ml foss/2019b OpenFOAM/v1912
. $FOAM_BASH WM_PROJECT_USER_DIR=your_openfoam_path/$USER-$WM_PROJECT_VERSION
export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php
#------------ Copy and run the damBreak tutorial -----------
mkdir -p $FOAM_RUN/testDamBreak
cd $FOAM_RUN/testDamBreak
cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .
cd damBreak
sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict # Replace ascii with binary output format.
blockMesh                  #Generates the grid
setFields                  #Sets the initial fields
decomposePar               #Decomposes for parallel simulation
mpirun interFoam -parallel #Runs interFoam in parallel
reconstructPar             #Collects the decomposed results
rm -rf processor*          #Deletes the decomposed files

Remember that the results end up in $FOAM_RUN/testDamBreak/damBreak

Tips

Like all jobs in SLURM, you can have dependencies: -d, --dependency=type:jobid defer This can be very useful when a simulation has multiple stages.

For reducing file count in parallell simulations, check out: https://openfoam.org/news/parallel-io/ https://www.openfoam.com/releases/openfoam-v1712/parallel.php

Visualizing with paraFoam

Visualization of results is best done through the graphical login nodes using Thinlinc. and to get best performance, one should make sure to run paraFoam through vglrun which lets Thinlinc make full use of the NVidia GPU:

vglrun paraFoam -builtin -case $FOAM_RUN/testDamBreak/damBreak

The -builtin flag tells paraFoam to use the built-in reader in Paraview. The Reader that comes with the OpenFOAM Installation is not compiled.