Biswajit Banerjee

### Parallel domain decomposition for particle methods: Part 3

The Plimpton method of communicating ghost regions

#### Introduction

In Part 2 of this series we showed the direct way of communicating ghost particles between patches. That approach requires 26 communication steps per patch in three-dimensions. In this article we discuss the approach suggested by Steve Plimpton (“Fast parallel algorithms for short-range molecular dynamics”, Sandia Report SAND91-1144.UC-405, 1993).

Plimpton’s paper has been cited almost 15,000 times since its publication. Among other things, the paper explains how the number of communication steps can be reduced to six in three dimensions.

#### Plimpton’s scheme for exchanging particles

If you run the animation below, you will notice that the left-right ghost particles are exchanged first. The ghost regions in the up-down directions are then extended to include the left-right ghost regions. The particles in these enlarged regions are then transferred in the up-down directions. Therefore, there are only four communication steps in two-dimensions. The same process can be used in three-dimensions, leading to only six communication steps.

#### MPI implementation

In Part 2 we defined a PatchNeighborComm struct and a Patch struct. We can keep the PatchNeighborComm struct in the same form, with the possible addition of a method of two. However, the Patch struct becomes considerably simplified as show below.

##### Patch struct

The Patch struct now needs only six PatchNeighborComm objects but three waitToFinish and combineReceivedParticles methods.

The new implementation of the Patch struct is shown below. The initialization of the struct is the same as before; as are the setXMinus etc. methods. Also, the sendRecvGhostXMinus and sendRecvGhostXPlus methods are the same as before.

Next we do the communication of the x+ and x- neighbors and combine the received particles with the particles in the current patch.

Now that the patchParticles vector has been updated, we can repeat the process for the y+ and y- directions. Note that the size of the ghost region has been expanded in the negative and positive x-direction.

Finally, we do the third stage of communication in the z-direction. The ghost-regions have now been expanded to contain for the x-, x+ and y-, y+ extensions.

##### The particle exchange function

The Plimpton particle exchange function the main simulation can then be simplified to the following.

In this case the number of communication steps is much smaller. However, there is a wait period at the end of each communication step that may reduce the benefits of the Plimpton approach in some situations where the particles are unevenly distributed.

#### Remarks

Plimpton’s scheme is attractive for its simplicity in communicating ghost particle positions. However, there are two more important communication steps that need to be considered - the computation of interparticle forces and the migration of particles between patches. In the next part of this series, we will discuss the migration of particles when we use the Plimpton scheme.