- What the cluster already had installed
Once every few months I have to port my codes over to a new large parallel computing machine. The configurations of these machines can vary wildly, but recently most machines have started sporting a flavor of Linux - which makes porting easier but not necessarily headache-free. In this article I’ll discuss my recent experience with porting my Discrete Element code
paraEllip3D to a RedHat-based Linux cluster. You can download the latest version of the code from GitHub.
Most systems have a two-stage authentication process; typically with a USB keyboard such as Yubikey or with a cell-phone based authenticator such as Duo. On the new system, I had to authenticate using the Duo process below:
paraEllip3D code is written in C++ and uses several C++14 features (and libraries that require C++14 features). It is also primarily written for compilation with
clang because of issues I ran into with
gcc at some point (I think it was something to do with template specialization). So I had to figure out how to get all the libraries and C++14 features to be built on the new machine without significant changes to the code.
To start the process is usually look at my
CMakeLists.txt file to figure out what I need. In this case:
boost: for some MPI and serialization code
clang: to build the code
cmake: to build the makefiles
eigen3: for some matrix operations
git: to download the source code
googletests: for units testing
libxml2: for XML input/output of
openmpi: for MPI code
openmp: for OpneMP code
qhull: for tetrahedral mesh generation
vtk: for XML input/output of
zenxml: for XML input/output of
zlib: for compression code
What the cluster already had installed
The first thing I noticed was that the head node and the compile node has different versions of
gcc libraries. When I logged into the compile node (
ssh scompile) and ran
module avail I noted that
gcc 6.1.0 was available. I loaded that module with
module load and ran tried to run a simple
zenxml test case. The code failed to compile with warnings about the non-availability of C++14 headers (note that gcc 6.1 is supposed to come with C++14 compliant standard headers). A little more searching showed that the following tools were installed:
I knew that
boost had to be recompiled because of conflicts with
clang handling of macros. But the non-availability of the other needed libraries created a sinking feeling (and made me appreciate better why most users still stick to Fortran and flat files in supercomputing). But I decided to go ahead and try to build all the needed libraries.
The first step in the process was the build the
clang++ compilers and the associated standard libraries and ABI libraries. I also realized (after a few iterations) that I would have to build the associated
OpenMP library and headers (I could not find
omp.h anywhere on the compile nodes).
The first step in the process was to load the needed modules (and unload the not needed ones)
Then I downloaded the source code (with
subversion and later
subversion was taking too long and failing)
clang and the associated libraries, I did the following
and, after much head banging, finally managed to get
clang built and installed in
/home/<username>/clang along with the
libomp libraries and headers.
Another major task was to build the
vtk libraries. I decided to go with the
VTK7.1.1 version because I wasn’t able to get older versions to compile with
gcc6.1.0. To download the source code, I used
To build the code, I created a
build directory and ran
I asked for the build script to download all the necessary libraries and just specified that the build be an optimized build and no tests were to be run. I also specified the install prefix to be a local
Finally, I ran the build script:
This process was much less painful than the
clang build process and I had to iterate only a few times before a clean build was complete.
I downloaded a relatively recent version of
boost and tried to build it using
Building boost appeared to be straightfoward, but after some failed iterations I found that for the
mpi libraries to be built, I had to create file
~/user-config.jam containing the single line
using mpi ; before I tried to build
I also ran into a problem with the file
and had to replace
Compared to the previous libraries, build
libxml2 was a breeze:
eigen3 build was also straightforward.
qhull source is downloaded during the download of the
ParSim source code that contains
paraEllip3D as a third-part application.
qhull is done in-source:
Googletests are built automatically while building
zenxml is a header-only file that requires C++14 capabilities in the compiler.
With all the needed tools built and ready, we can now compile the source code:
CMakeLists.txt file in the source root directory needed the following hacks for using the C++14 standrad library provide by
clang and for compatibility with the updated API of VTK-7:
It is quite arduous to port modern C++ code and libraries to supercomputing systems. I’d suggest making sure your code compiles with several versions of the GNU compilers.
We will discuss our experience porting the
Vaango code to a Cray machine in the next article.
If you have questions/comments/corrections, please contact banerjee at parresianz dot com dot zen (without the dot zen).
📅 15.12.2017 📁 DEM