Biswajit Banerjee

### Compiling the DEM code ParaEllip3D on a cluster

What to do when few pre-built modules exist

#### Introduction

Once every few months I have to port my codes over to a new large parallel computing machine. The configurations of these machines can vary wildly, but recently most machines have started sporting a flavor of Linux - which makes porting easier but not necessarily headache-free. In this article I’ll discuss my recent experience with porting my Discrete Element code paraEllip3D to a RedHat-based Linux cluster. You can download the latest version of the code from GitHub.

#### Authentication

Most systems have a two-stage authentication process; typically with a USB keyboard such as Yubikey or with a cell-phone based authenticator such as Duo. On the new system, I had to authenticate using the Duo process below:

#### The paraEllip3D code

The paraEllip3D code is written in C++ and uses several C++14 features (and libraries that require C++14 features). It is also primarily written for compilation with clang because of issues I ran into with gcc at some point (I think it was something to do with template specialization). So I had to figure out how to get all the libraries and C++14 features to be built on the new machine without significant changes to the code.

To start the process is usually look at my CMakeLists.txt file to figure out what I need. In this case:

• boost : for some MPI and serialization code
• clang : to build the code
• cmake : to build the makefiles
• eigen3 : for some matrix operations
• git: to download the source code
• googletests : for units testing
• libxml2 : for XML input/output of Vaango format data
• openmpi : for MPI code
• openmp : for OpneMP code
• qhull : for tetrahedral mesh generation
• vtk : for XML input/output of VTK format data
• zenxml : for XML input/output of Ellip3D format data
• zlib : for compression code

#### What the cluster already had installed

The first thing I noticed was that the head node and the compile node has different versions of gcc libraries. When I logged into the compile node (ssh scompile) and ran module avail I noted that gcc 6.1.0 was available. I loaded that module with module load and ran tried to run a simple zenxml test case. The code failed to compile with warnings about the non-availability of C++14 headers (note that gcc 6.1 is supposed to come with C++14 compliant standard headers). A little more searching showed that the following tools were installed:

• boost
• cmake
• git
• zlib

I knew that boost had to be recompiled because of conflicts with clang handling of macros. But the non-availability of the other needed libraries created a sinking feeling (and made me appreciate better why most users still stick to Fortran and flat files in supercomputing). But I decided to go ahead and try to build all the needed libraries.

#### Building clang

The first step in the process was the build the clang and clang++ compilers and the associated standard libraries and ABI libraries. I also realized (after a few iterations) that I would have to build the associated OpenMP library and headers (I could not find omp.h anywhere on the compile nodes).

The first step in the process was to load the needed modules (and unload the not needed ones)

Then I downloaded the source code (with subversion and later git, because subversion was taking too long and failing)

To build clang and the associated libraries, I did the following

and, after much head banging, finally managed to get clang built and installed in /home/<username>/clang along with the libcxx and libomp libraries and headers.

#### Building vtk

Another major task was to build the vtk libraries. I decided to go with the VTK7.1.1 version because I wasn’t able to get older versions to compile with gcc6.1.0. To download the source code, I used

To build the code, I created a build directory and ran

I asked for the build script to download all the necessary libraries and just specified that the build be an optimized build and no tests were to be run. I also specified the install prefix to be a local vtk directory.

Finally, I ran the build script:

This process was much less painful than the clang build process and I had to iterate only a few times before a clean build was complete.

#### Building boost

I downloaded a relatively recent version of boost and tried to build it using

Building boost appeared to be straightfoward, but after some failed iterations I found that for the mpi libraries to be built, I had to create file ~/user-config.jam containing the single line using mpi ; before I tried to build boost.

I also ran into a problem with the file

and had to replace

with

#### Building libxml2

Compared to the previous libraries, build libxml2 was a breeze:

#### Building eigen3

The eigen3 build was also straightforward.

#### Building qhull

The qhull source is downloaded during the download of the ParSim source code that contains paraEllip3D as a third-part application.

Building qhull is done in-source:

#### Building googletests and zenxml

Googletests are built automatically while building paraEllip3D while zenxml is a header-only file that requires C++14 capabilities in the compiler.

#### Building paraEllip3D

With all the needed tools built and ready, we can now compile the source code:

The CMakeLists.txt file in the source root directory needed the following hacks for using the C++14 standrad library provide by clang and for compatibility with the updated API of VTK-7:

#### Remarks

It is quite arduous to port modern C++ code and libraries to supercomputing systems. I’d suggest making sure your code compiles with several versions of the GNU compilers.

We will discuss our experience porting the Vaango code to a Cray machine in the next article.

If you have questions/comments/corrections, please contact banerjee at parresianz dot com dot zen (without the dot zen).