A stepwise guide to compile and run OpenFOAM* with Intel® Compiler for Intel® Xeon® Scalable Processor
Objective
Demonstrate the steps involved in compiling OpenFOAM* software with Intel® Compiler for running it on Intel Xeon Scalable Processor. This recipe describes how to get OpenFOAM* and corresponding third party software and build all the code with Intel® Compiler and how to run an example on Intel® Xeon® Scalable Processors.
Introduction
OpenFOAM* (Open source Field Operation And Manipulation) is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of computational fluid dynamics (CFD). It is a unique, and highly extensible, software development kit (“devkit”) for CFD, consisting of 1 million lines of C++ code. OpenFOAM includes hundreds of robust CFD applications built from the devkit, which can be extended and customized, quickly and conveniently. The applications are used to preform CFD simulations by industry, academic, and research and government institutes, across all fields of engineering and science
OpenFOAM is distributed as open source code by OpenFOAM Foundation, under the terms of the GNU Public License. OpenFOAM is developed by multiple contributors led by Henry Weller, creator of OpenFOAM and Director of CFD Direct and The OpenFOAM Foundation. The current version can be downloaded at https://openfoam.org/download/. Links are also included for Ubuntu Linux, Other Linux, Windows* and macOS version of the software.
Compiling OpenFOAM Intel® Compiler
Specifications
OS family | Linux |
OS Name | Ubuntu* |
OS Versions | 16.04 LTS & 16.10 |
Processor | Intel® Xeon® Platinum 8168 CPU @ 2.70GHz |
Software suite | Intel Parallel Studio XE 2018 Cluster Edition (update 1) |
OpenFOAM Version | 4.1.x |
Compiler | Intel C/C++ Compiler 2018 update 1 (gcc version 5.0.0 compatibility) |
MPI library | Intel MPI Library 2018 update 1 |
Code Access
Step 1. Downloading OpenFOAM*
mkdir OpenFOAM
wget -P OpenFOAM "http://download.openfoam.org/source/4-1" -O OpenFOAM-4.1.tgz
tar -xzf OpenFOAM/OpenFOAM-4.1.tgz
cd OpenFOAM
mv OpenFOAM-4.x-version-4.1 OpenFOAM-4.1
Step 2. Downloading Third Party Software :
In order to build OpenFOAM, we need to download Third Party Software which has a basic requirement of having cmake-3.2.1 and above . If you do not have cmake-3.2.1, then please download and install it before compiling any of the Third Party Software. Download the following Third Party Software:
- CGAL-4.8
- boost_1_55_0
- gmp-5.1.2
- mpfr-3.1.2
- metis-5.1.0
wget "http://download.openfoam.org/third-party/4-1" -O ThirdParty-4.1.tgz
tar -xzf OpenFOAM/ThirdParty-4.1.tgz
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1
mv ThirdParty-4.x-version-4.1 ThirdParty-4.1
mkdir download
wget -P download "https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-4.8/CGAL-4.8.tar.xz"
wget -P download "http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2"
wget –P download "https://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2"
wget –P download "http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz"
wget -P download "http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/metis-5.1.0.tar.gz"
tar -xJf download/CGAL-4.8.tar.xz
tar -xjf download/boost_1_55_0.tar.bz2
tar –zxf download/metis-5.1.0.tar.gz
tar -xjvf download/gmp-5.1.2.tar.bz2
tar -xzvf download/mpfr-3.1.2.tar.gz
Build Directions
The most fundamental way to efficiently utilize the resources in modern processors is to write code that can run in vector mode by taking advantage of special hardware like vector registers and SIMD (Single Instruction Multiple Data) instructions. Data parallelism in the algorithm/code is exploited in this stage of the optimization process. The new Intel® Xeon® scalable processor features 512-bit wide vector registers. The new Intel® Advanced Vector Extensions 512 (Intel® AVX-512) instruction set architecture (ISA), offers support for vector-level parallelism, which allows the software to use two vector processing units (each capable of simultaneously processing 16 single precision (32-bit) or 8 double precision (64-bit) floating point numbers) per core. Taking advantage of these hardware and software features is the key to optimal use of the Intel® Xeon® scalable processor, which can be leveraged by compiler option -xCORE-AVX512.
Step 1. Building OpenFOAM* with Intel® Compiler is similar to a normal OpenFOAM build. Need to make a few changes in the config settings and config mpi along with the bashrc. An example build will include the following steps while compiling it for 64 bit architecture:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM/OpenFOAM-4.1
1. vi etc/config.sh/mpi
case "$WM_MPLIB" in - INTELMPI
export MPI_ROOT=/opt/intel/compilers_and_libraries_2018.0.128/linux/mpi/intel64
2. vi etc/config.sh/settings
a. case $WM_ARCH in Linux
case `uname -m` in > x86_64
case $WM_ARCH_OPTION in
i ) 32
ii) 64
> export WM_CC='icc'
> export WM_CXX='icpc'
> export WM_CFLAGS='-xCORE-AVX512 -O2 -no-prec-div -m64 -fPIC'
> export WM_CXXFLAGS='-xCORE-AVX512 -O2 -no-prec-div -m64 -fPIC'
3. vi etc/bashrc
> export FOAM_INST_DIR=/<YOUR_OPENFOAM_LOCATION>/OpenFOAM
> export WM_COMPILER=Icc
> export WM_MPLIB=INTELMPI
Step 2. Create a setup file, which you need to source before any compilation:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
vi setup
> source /opt/intel/compilers_and_libraries_2018.0.128/linux/bin/compilervars.sh intel64
> source /opt/intel/impi/2018.0.128/bin64/mpivars.sh
> source ./OpenFOAM-4.1/etc/bashrc
> cd ThirdParty-4.1
> sed -i -e 's/\(boost_version=\)boost-system/\1boost_1_55_0/' OpenFOAM-4.1/etc/config.sh/CGAL
> sed -i -e 's/\(cgal_version=\)cgal-system/\1CGAL-4.8/' OpenFOAM-4.1/etc/config.sh/CGAL
> cd ..
> source ./OpenFOAM-4.1/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=system FOAMY_HEX_MESH=yes
Step 3. Building Third Party Softwares:
Building gmp with Intel® Compiler:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd ThirdParty-4.1/gmp-5.1.2
./configure --prefix=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2 CC=icc CXX=icpc CFLAGS=-xCORE-AVX512 CXXFLAGS=-xCORE-AVX512
mkdir platforms/linux64/gmp-5.1.2
make
make check
make install
Building mpfr with Intel® Compiler:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd ThirdParty-4.1/mpfr-3.1.2
./configure --prefix=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/mpfr-3.1.2 CC=icc CXX=icpc CFLAGS=-xCORE-AVX512 CXXFLAGS=-xCORE-AVX512 --with-gmp-include=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2/include --with-gmp-lib=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2/lib
mkdir platforms/linux64/mpfr-3.1.2
make
make check
make install
Building CGAL with Intel® Compiler:
In order to build CGAL, we need to build the boost library as well. In the makefile of CGAL, while calling the bootstap.sh, replace "bjam" with "b2" and use intel toolset to build the library (line no.161). Add the Intel C/C++ Compiler and optimization flags as -xCORE-AVX512 (line no 277). Following are the changes:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd ThirdParty-4.1
vi makeCGAL
> --with-toolset=intel-linux \
> && ./b2 -j $WM_NCOMPPROCS install \
> ....
> ....
> -DCMAKE_C_COMPILER=icc \
> -DCMAKE_CXX_COMPILER=icpc \
> -DCMAKE_C_FLAGS=-xCORE-AVX512 \
> -DCMAKE_CXX_FLAGS=-xCORE-AVX512 \
In order to make CGAL with gmp-5.1.2 and mpfr-3.1.2 library, we need to have gmp-5.1.2 and mpfr-3.1.2 libraries, which were created at the location:
cd Thirdparty-4.1.x/platforms/gmp-5.1.2/lib
cd Thirdparty-4.1.x/platforms/mpfr-5.1.2/lib
While compiling CGAL, the library files are searched under lib64 folder, hence we need to rename the lib folder to lib64 as follows and then make the CGAL library:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
mv Thirdparty-4.1.x/platforms/gmp-5.1.2/lib Thirdparty-4.1.x/platforms/gmp-5.1.2/lib64
mv Thirdparty-4.1.x/platforms/mpfr-5.1.2/lib Thirdparty-4.1.x/platforms/mpfr-5.1.2/lib64
vi OpenFOAM-4.1.x/wmake/rules/General/CGAL
> CGAL_INC = -I<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/mpfr-3.1.2/include\
-I<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2/include
> CGAL_LIBS = -L<YOUR_OPENFOAM_LOCATION>OpenFOAM/platforms/linux64/mpfr-3.1.2/lib$(WM_COMPILER_LIB_ARCH) \
-L<YOUR_OPENFOAM_LOCATION>/OpenFOAM/platforms/linux64/gmp-5.1.2/lib$(WM_COMPILER_LIB_ARCH) \
source setup
cd Thirdparty-4.1.x
./makeCGAL CGAL-4.8 boost_1_55_0 gmp-5.1.2 mpfr-3.1.2
Building Scotch library with Intel Compiler
- Remove scotch building steps from OpenFOAM-4.1.x/Allwmake
- Make changes to SCOTCH_Num from int to int64_t in /Thirdparty-4.1.x/scotch_6.0.3/include/scotch.h
- Remove previous simlink to /Thirdparty-4.1.x/scotch_6.0.3/Makefile.inc
- Make changes to /Thirdparty-4.1.x/scotch_6.0.3/Make.inc/Makefile.inc.x86-64_pc_linux2.icc
- CCS = icc
- CCP = mpiicc
- CCD = mpiicc
- #add –xCORE-AVX512 –fPIC flag in CFLAGS
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
cd Thirdparty-4.1.x/scotch_6.0.3/src/
rm Makefile.inc
vi Make.inc/Makefile.inc.x86-64_pc_linux2.icc
> CCS = icc
> CCP = mpiicc
> CCD = mpiicc
> CFALG = -xCORE-AVX512 -fPIC (additional flags)
ln –s Make.inc/Makefile.inc.x86-64_pc_linux2.icc Makefile.inc
make scotch
make prefix=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/scotch_6.0.3 libdir=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/lib install
make ptscotch
make prefix=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/scotch_6.0.3 libdir=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/lib/intel64 includedir=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/scotch_6.0.3/include/intel64 install
Building metis-5.1.0 library with Intel Compiler
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
cd Thirdparty-4.1.x/matis-5.1.0
vi Makefile
> openmp = -qopenmp
> cc = mpiicc
make config prefix=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/metis-5.1.0
make
make install
Step 4. Building OpenFOAM-4.1.x with Intel Compiler:
We have already updated the cOpt and c++Opt with -xCORE-AVX512 in the step no. 2. Now we need to provide correct IntelMPI path in the mplibINTELMPI file. Replace include64 and lib64 with include and lib
- PINC = -isystem $(MPI_ARCH_PATH)/include64
- PLIBS = -L$(MPI_ARCH_PATH)/lib64 –lmpi
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM/OpenFOAM-4.1.x
vi wmake/rules/linux64Icc/mplibINTELMPI
> PINC = -isystem $(MPI_ARCH_PATH)/include
> PLIBS = -L$(MPI_ARCH_PATH)/lib -lmpi
cd ..
source setup
export WM_NCOMPPROCS=8
./Allwmake -j 8 2>&1 | tee output.log
You can check output.log if the compilation is done successful. Once compilation is done, you can run the example and check if it is working fine.
Test with tutorials
You can check your OpenFOAM installation is working fine or not by running all tutorials. This will take a lot of time to run all of them, one by one.
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd OpenFOAM-4.1/tutorials/
./Allrun
However, you can also just run one of the case in the tutorial, e.g.:
cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd OpenFOAM-4.1/tutorials/incompressible/simpleFoam/motorbike/
./Allrun
"