OpenFOAM is an open-source C++ library for solving partial differential equations. It is mainly used for computational fluid dynamics, for which there are many implemented solvers and different kinds of utilities. For further info see the OpenFOAM homepage.
We install OpenFOAM as a complete package, i.e. the module
OpenFOAM/4.0 contains the OpenFOAM version and all the packages needed to use this OpenFOAM version rather than installing them separately and creating multiple modules.
Available OpenFOAM versions
To see available OpenFOAM versions write in the terminal:
module spider OpenFOAM
OpenFOAM best practice
When running OpenFOAM it is important to run from your NOBACKUP directory. This is because OpenFOAM generate a tremendous amount of files which will fill up the quota in your BACKUP directory in no time. It is therefore recommended to source OpenFOAM the following way:
ml intel/2017a OpenFOAM/4.1 . $FOAM_BASH WM_PROJECT_USER_DIR=$SNIC_NOBACKUP/OpenFOAM/$USER-$WM_PROJECT_VERSION
This will tell OpenFOAM that the user working directory is in
$SNIC_NOBACKUP, under version-specific directories.
New users should then (once) create some directory structure, by:
mkdir -p $FOAM_RUN
This will give you a run-directory in
$SNIC_NOBACKUP, which you can easily reach by the alias
run (as long as you source as described above).
It is recommended to create some aliases for the different versions, at the end of your ~/.bashrc file, as:
# .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi` alias OF40='ml intel/2017a OpenFOAM/4.1;. $FOAM_BASH WM_PROJECT_USER_DIR=$SNIC_NOBACKUP/OpenFOAM/$USER-$WM_PROJECT_VERSION'
Whenever you open a new terminal window you can then simply activate whichever version you like by each alias (e.g. "OF40"). It is not recommended to switch between versions in the same terminal window! Open a new terminal window for each version.
Note that it is NOT RECOMMENDED to explicitly load the OpenFOAM modules in
It is MUCH better to load the modules in the submit script, so that each submit determines its own environment.
See the submit script below.
In the submit script you should write out the same contents as in the alias, as described above (and do not use the alias itself - it has been problematic).
OpenFOAM on Vera
Hyper-treading(HT) is enabled on Vera and has been evaluated for OpenFOAM. OpenFOAM showed a small benefit from HT and the default settings should be used. To read more about HT on Vera, see Vera.
Submitting OpenFOAM jobs
Submit the following shell script, which will copy and run the interFoam damBreak tutorial (note that you have to change PROJECT_NUMBER to one of your project numbers):
#!/usr/bin/env bash #SBATCH -A PROJECT_NUMBER -p hebbe #SBATCH -n 4 #SBATCH -t 01:00:00 #----------------------------------------------------------- ml foss/2019b OpenFOAM/v1912 . $FOAM_BASH WM_PROJECT_USER_DIR=$SNIC_NOBACKUP/OpenFOAM/$USER-$WM_PROJECT_VERSION export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php #------------ Copy and run the damBreak tutorial ----------- mkdir -p $FOAM_RUN/testDamBreak cd $FOAM_RUN/testDamBreak cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak . cd damBreak sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict # Replace ascii with binary output format. blockMesh #Generates the grid setFields #Sets the initial fields decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel #Runs interFoam in parallel reconstructPar #Collects the decomposed results rm -rf processor* #Deletes the decomposed files
Remember that the results end up in
Like all jobs in SLURM, you can have dependencies:
-d, --dependency=type:jobid defer
This can be very useful when a simulation has multiple stages.
For reducing file count in parallell simulations, check out: https://openfoam.org/news/parallel-io/ https://www.openfoam.com/releases/openfoam-v1712/parallel.php
Visualizing with paraFoam
Visualization of results is best done through the graphical login nodes using Thinlinc.
and to get best performance, one should make sure to run paraFoam through
vglrun which lets Thinlinc make full use of the NVidia GPU:
vglrun paraFoam -builtin -case $FOAM_RUN/testDamBreak/damBreak
-builtin flag tells paraFoam to use the built-in reader in Paraview.
The Reader that comes with the OpenFOAM Installation is not compiled.