Biswajit Banerjee

Compiling the DEM code ParaEllip3D on a cluster

What to do when few pre-built modules exist


Once every few months I have to port my codes over to a new large parallel computing machine. The configurations of these machines can vary wildly, but recently most machines have started sporting a flavor of Linux - which makes porting easier but not necessarily headache-free. In this article I’ll discuss my recent experience with porting my Discrete Element code paraEllip3D to a RedHat-based Linux cluster. You can download the latest version of the code from GitHub.


Most systems have a two-stage authentication process; typically with a USB keyboard such as Yubikey or with a cell-phone based authenticator such as Duo. On the new system, I had to authenticate using the Duo process below:

1) ssh <>
2) duo:<password>
3) authenticate on mobile phone using the Duo Mobile app

The paraEllip3D code

The paraEllip3D code is written in C++ and uses several C++14 features (and libraries that require C++14 features). It is also primarily written for compilation with clang because of issues I ran into with gcc at some point (I think it was something to do with template specialization). So I had to figure out how to get all the libraries and C++14 features to be built on the new machine without significant changes to the code.

To start the process is usually look at my CMakeLists.txt file to figure out what I need. In this case:

  • boost : for some MPI and serialization code
  • clang : to build the code
  • cmake : to build the makefiles
  • eigen3 : for some matrix operations
  • git: to download the source code
  • googletests : for units testing
  • libxml2 : for XML input/output of Vaango format data
  • openmpi : for MPI code
  • openmp : for OpneMP code
  • qhull : for tetrahedral mesh generation
  • vtk : for XML input/output of VTK format data
  • zenxml : for XML input/output of Ellip3D format data
  • zlib : for compression code

What the cluster already had installed

The first thing I noticed was that the head node and the compile node has different versions of gcc libraries. When I logged into the compile node (ssh scompile) and ran module avail I noted that gcc 6.1.0 was available. I loaded that module with module load and ran tried to run a simple zenxml test case. The code failed to compile with warnings about the non-availability of C++14 headers (note that gcc 6.1 is supposed to come with C++14 compliant standard headers). A little more searching showed that the following tools were installed:

  • boost
  • cmake
  • git
  • zlib

I knew that boost had to be recompiled because of conflicts with clang handling of macros. But the non-availability of the other needed libraries created a sinking feeling (and made me appreciate better why most users still stick to Fortran and flat files in supercomputing). But I decided to go ahead and try to build all the needed libraries.

Building clang

The first step in the process was the build the clang and clang++ compilers and the associated standard libraries and ABI libraries. I also realized (after a few iterations) that I would have to build the associated OpenMP library and headers (I could not find omp.h anywhere on the compile nodes).

The first step in the process was to load the needed modules (and unload the not needed ones)

module load cmake/3.6.2
module load gcc/6.1.0
module load python/2.7.10
module load git
module load subversion
module load zlib
module load perl
module unload openmpi

Then I downloaded the source code (with subversion and later git, because subversion was taking too long and failing)

mkdir Clang
cd Clang
svn co llvm
cd llvm/tools/
git clone
cd ../projects
git clone
git clone
git clone
cd ../..

To build clang and the associated libraries, I did the following

mkdir build
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../../clang -DLLVM_TARGETS_TO_BUILD=X86  ../llvm
make install

and, after much head banging, finally managed to get clang built and installed in /home/<username>/clang along with the libcxx and libomp libraries and headers.

Building vtk

Another major task was to build the vtk libraries. I decided to go with the VTK7.1.1 version because I wasn’t able to get older versions to compile with gcc6.1.0. To download the source code, I used

mkdir VTK-7.1.1
ce VTK-7.1.1
tar xvfz archive.tar.gz

To build the code, I created a build directory and ran

mkdir build
cd build
ccmake ../vtk-v7.1.1-b86da7eef93f75c4a7f524b3644523ae6b651bc4/

I asked for the build script to download all the necessary libraries and just specified that the build be an optimized build and no tests were to be run. I also specified the install prefix to be a local vtk directory.

Finally, I ran the build script:

make -j8
make install

This process was much less painful than the clang build process and I had to iterate only a few times before a clean build was complete.

Building boost

I downloaded a relatively recent version of boost and tried to build it using

tar xvfz boost_1_65_1.tar.gz
cd boost_1_65_1/
./ --prefix=../boost --with-libraries=mpi,serialization --show-libraries
./b2  link=shared install

Building boost appeared to be straightfoward, but after some failed iterations I found that for the mpi libraries to be built, I had to create file ~/user-config.jam containing the single line using mpi ; before I tried to build boost.

I also ran into a problem with the file


and had to replace

#ifndef BOOST_MSVC
    #define DONT_USE_HAS_NEW_OPERATOR (                    \
           BOOST_WORKAROUND(__IBMCPP__, < 1210)            \
        || defined(__SUNPRO_CC) && (__SUNPRO_CC < 0x590)   \


#ifndef BOOST_MSVC
        #if BOOST_WORKAROUND(__IBMCPP__, < 1210)               \
            || defined(__SUNPRO_CC) && (__SUNPRO_CC < 0x590)
                #define DONT_USE_HAS_NEW_OPERATOR 1
                #define DONT_USE_HAS_NEW_OPERATOR 0

Building libxml2

Compared to the previous libraries, build libxml2 was a breeze:

tar xvfz libxml2-2.9.7.tar.gz
module load zlib
module unload python
./configure --prefix=<home-directory>/libxml2 --enable-shared --without-python
make -j4
make install

Building eigen3

The eigen3 build was also straightforward.

tar xvfz 3.3.4.tar.gz
mkdir eigen-build
cd eigen-build
cmake ../eigen-eigen-5a0156e40feb/ -DCMAKE_INSTALL_PREFIX=../eigen3
make install

Building qhull

The qhull source is downloaded during the download of the ParSim source code that contains paraEllip3D as a third-part application.

Building qhull is done in-source:

cd ParSim/Thirdparty/paraEllip3D-DEM-SPH/src/qhull
cd build
make install

Building googletests and zenxml

Googletests are built automatically while building paraEllip3D while zenxml is a header-only file that requires C++14 capabilities in the compiler.

Building paraEllip3D

With all the needed tools built and ready, we can now compile the source code:

module load gcc
module load zlib
module load openmpi
git clone --recursive
cd ParSim/ThirdParty/paraEllip3d_DEM_PD_SPH/
mkdir opt
cd opt
CC=<home_directory>/clang/bin/clang CXX=<home_directory>/clang/bin/clang++ \
      -DBOOST_ROOT=/projects/biba1632/boost \
      -DVTK_DIR=/projects/biba1632/vtk/lib/cmake/vtk-7.1/ -DWITH_VTK7=1 \
      -DEIGEN3_INCLUDE_DIR=/projects/biba1632/eigen3/include/eigen3 \
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/projects/biba1632/clang/lib make -j4

The CMakeLists.txt file in the source root directory needed the following hacks for using the C++14 standrad library provide by clang and for compatibility with the updated API of VTK-7:

  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -stdlib=libc++ -Wno-deprecated -fopenmp")
endif (USE_CLANG)
if (WITH_VTK7)
  find_package(VTK 7.1 REQUIRED)
else (WITH_VTK7)
  find_package(VTK 5.10 REQUIRED)
endif (WITH_VTK7)


It is quite arduous to port modern C++ code and libraries to supercomputing systems. I’d suggest making sure your code compiles with several versions of the GNU compilers.

We will discuss our experience porting the Vaango code to a Cray machine in the next article.

If you have questions/comments/corrections, please contact banerjee at parresianz dot com dot zen (without the dot zen).