References:

Barrier (MPI_Barrier):

This is a synchronization function, it holds each process at a certain line of code (where it is called) until all processes have reached that line (all processes grouped by a specific communicator).

Function prototype:

int MPI_Barrier(
	MPI_Comm comm //Communicator
);

It basically receives the communicator.

Example:

A visual example of the difference using this function is as follows:

#include <mpi.h>
#include <iostream>

using namespace std;

int main(int argc, char** argv) {
  int rank, size;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  for (int ind = 0; ind < size; ind++) {
    if (ind == rank) {
      cout << "Hello World from process " <<
			rank << " of " << size << endl;
    }
  }

  MPI_Finalize();
  return 0;
}

Executing this way is not guaranteed that the printout will be in order, the result may be this:

Untitled

We can change this behavior using Barriers, waiting for all processes to reach the same point, so the printout will follow the loop order:

#include <mpi.h>
#include <iostream>

using namespace std;

int main(int argc, char** argv) {
  int rank, size;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  for (int ind = 0; ind < size; ind++) {
    if (ind == rank) {
      cout << "Hello World from process " <<
      rank << " of " << size << endl;
    }

    MPI_Barrier(MPI_COMM_WORLD);
  }

  MPI_Finalize();
  return 0;
}

Now the output will always be:

Untitled

Send (MPI_Send) and Receive (MPI_Recv):