Part A - Introduction

Compiler Support

Review some modern features of the core languages
Introduce the compilers for implementing the parallel programming models

Core Languages | OpenMP | TBB | MPI | Exercises


The compiler support available for implementing parallel programming designs consists of combinations of core languages, language extensions, language libraries and application program interfaces (APIs).  While for the most part compiler support abstracts from hardware implementation entirely, some support exposes access to some underlying hardware mechanisms.  Although this direct access may help improve performance, it does so at the cost of reduced portability. 

This chapter introduces the compiler support tools (OpenMP, Threading Building Blocks (TBB) and Message Passing Interface (MPI)) for parallel programming.  Dedicated chapters list more detailed information pertaining to the options and platform-specific requirements and include coding examples. 


Core Languages

The core languages that are foundational for implementions of parallel programming designs include:

The compilers used in this course are:

  • Microsoft's Visual Studio 2019
  • Intel's Parallel Studio XE 2020
  • GNU's GCC 9.1 - available on matrix at /usr/local/gcc/9.1.0/bin

OpenMP

OpenMP is an API that supports the shared memory programming model.  OpenMP is an active non-profit project owned by the OpenMP Architectural Review Board (ARB).  Its web site is at openmp dot org

The release covered in these notes is 4.5.  Visual Studio 2019 supports OpenMP 2.0 and SIMD instructions.  GCC 9.1 supports OpenMP 4.5 fully and 5.0 partially.  Parallel Studio XE 2020 supports OpenMP 4.5 fully and 5.0 partially.  You can find the up-to-date list of compiler support here.

OpenMP uses pre-processor directives to implement parallelism (# statements).

Compilation

GCC

To enable compiler interpretation of OpenMP directives, the GCC compilers require the -fopenmp option

 g++ -o openmp -fopenmp openmp.cpp

Visual Studio

To enable compiler interpretation of OpenMP directives, the cl compiler requires the /openmp option

 cl /Feopenmp /openmp openmp.cpp

Parallel Studio

To enable compiler interpretation of OpenMP directives, the icl compiler requires the /Qopenmp option

 icl /Feopenmp /Qopenmp openmp.cpp

Example

The following program displays the date of the installed OpenMP release as listed on the right (200203 for the Visual Studio 2019 compiler; 201611 for the Intel 2020 compiler):

 // OpenMP - Hello World
 // openmp.cpp
 #include <iostream>
 #include <omp.h>

 int main() {
     std::cout << "Hello World from OpenMP " 
      << _OPENMP << std::endl;
 }

 Hello World from OpenMP 200203 
 

 Hello World from OpenMP 201611 

 

Threading Building Blocks

Threading Building Blocks (TBB) is an Intel template library that augments the C++ language with capabilities for task parallelism.  TBB is an active open source project under the auspices of Intel.  Its web site is at threadingbuildingblocks dot org.  The release described here is 2018 update 2. 

TBB is available as a stand-alone library or as part of the Parallel Studio XE Suite used in this course.

Compilation

GCC

To enable interpretation of TBB syntax, the GCC compilers require the -ltbb option

 g++ -o tbb -ltbb tbb.cpp

Visual Studio

To access the TBB headers and to link to the associated libraries, the cl compiler requires the /Qtbb option

 cl /Fetbb /Qtbb tbb.cpp

Parallel Studio

To access the TBB headers and to link to the associated libraries, the icl compiler requires the /Qtbb option

 icl /Fetbb /Qtbb tbb.cpp

Example

The following program displays the date of the installed TBB interface as listed on the right:

 // TBB - Hello World
 // tbb.cpp
 #include <iostream>
 #include <tbb/tbb.h>

 int main() {
     std::cout << "Hello World from TBB "
         << TBB_VERSION_MAJOR << "."
         << TBB_VERSION_MINOR << " ("
         << TBB_INTERFACE_VERSION << ")" << std::endl; 
 }








 Hello World from TBB 
 2020.3 (11103)
 

MPI

MPI is a standard for implementing a language-independent communications protocol under the distributed programming model.  The acronym stands for Message Passing Interface.  The web site for the MPI forum is at mpi-forum dot org.  The standard covered in these notes is release 3.1. 

You may need to install MPI manually.  Implementations of the MPI standard include:

  • OpenMPI - MPI-2 Indiana University
  • MPICH2 - MPI-1 and MPI-2 - Argonne National Laboratory
  • MPICH - MPI-1, MPI-2, MPI-3 - IBM, Intel, Microsoft

Installation

GCC

Current release >= OpenMPI 3.1.0

To install OpenMPI at the command line in Ubuntu, enter

 $ sudo apt-get install openmpi*

Visual Studio

To install Microsoft's MPICH implementation for Visual Studio, download the MSMPI Setup and MSMPISDK packages and follow the installation instructions

After installing MSMPI check that the following environment variables are defined.  If they are not defined, define the missing ones and point them to the corresponding SDK directories:

 MSMPI_BIN   path to the bin directory
 MSMPI_INC   path to the include directory
 MSMPI_LIB32 path to the 32-bit library directory
 MSMPI_LIB64 path to the 64-bit library directory 
 Path        includes the path to the bin directory

You will use these in defining your program properties and in running your executables.  Typically, the MPI executable is stored in the Program Files directory and the MPI includes and libraries are stored in the Program Files (x86) \ Microsoft SDKs directory.

Parallel Studio

After installing Parallel Studio check that the following environment variable has been set

 I_MPI_ROOT path to the root of the Intel MPI directory

You will use this variable in defining your program properties and in running your executables.  Typically, the MPI header files and libraries are stored in the Program Files (x86) \ IntelSWTools \ mpi \ 2019.8.254 \ intel64 \ directory, where 2019.8.254 is the version number of the installed Parallel Studio.

Compilation and Execution

GCC

To compile a C program on a Linux platform, you may use the mpicc command

 mpicc -o mpi_1 mpi_1.c

mpicc invokes the native C compiler.

To compile a C++ program on a Linux platform, you may use the mpiCC command or alternatively mpicxx or mpic++

 mpiCC -o mpi_1 mpi_1.cpp

mpiCC invokes the native C++ compiler.

If you use the gcc command, you need to add the -I option pointing to the header files and the -lmpi option to access the MPI library

 gcc -o mpi_1 -I /usr/include/mpi -lmpi mpi_1.c

If you use the g++ command, you need to add the -I option pointing to the header files and the -lmpi option to access the MPI library

 g++ -o mpi_1 -I /usr/include/mpi -lmpi mpi_1.cpp

To execute an MPI program on a single host with 4 processors, enter the following command

 mpirun -np 4 mpi_1

The -np switch specifies the number of processes that you are asking MPI to spawn. 

Visual Studio

To compile an MPI program in Visual Studio 2019, you need to specify access to the MPI directories in the project's property pages

  • Configuration Properties -> C/C++ -> General -> Additional Include Directories: $(MSMPI_INC), $(MSMPI_INC)\x64
  • Configuration Properties -> Linker -> General -> Additional Library Directories: $(MSMPI_LIB32) or $(MSMPI_LIB64)
  • Configuration Properties -> Linker -> Input -> Additional Dependencies: msmpi.lib

If compilation errors appear, confirm that these paths are set for the appropriate configuration and platform.

To execute a compiled MPI program in Visual Studio 2019 from within the IDE, you need to define the command line as an external tool and can then execute that tool:

To execute a compiled MPI program Studio from within the Visual Studio 2019 IDE, you need to issue the mpiexec command and specify the number of processors as a command line argument:

  • Configuration Properties -> Debugging -> Command: $(MSMPI_BIN)\mpiexec.exe
  • Configuration Properties -> Debugging -> Command Arguments: -n $(NUMBER_OF_PROCESSORS) "$(TargetPath)"

Parallel Studio

To compile an MPI program in Parallel Studio using the Visual Studio 2019 IDE, you need to specify access to the Intel MPI directories in the project's property pages

  • Configuration Properties -> C/C++ -> General -> Additional Include Directories: $(I_MPI_ROOT)\intel64\include
  • Configuration Properties -> Linker -> General -> Additional Library Directories: $(I_MPI_ROOT)\intel64\lib\<configuration>, where <configuration> is debug or release
  • Configuration Properties -> Linker -> Input -> Additional Dependencies: impi.lib

To execute a compiled MPI program in Parallel Studio from within the Visual Studio 2019 IDE, you need to issue the mpiexec command and and specify the number of processors as a command line argument:

  • Configuration Properties -> Debugging -> Command: $(I_MPI_ROOT)\intel64\bin\mpiexec.exe
  • Configuration Properties -> Debugging -> Command Arguments: -n $(NUMBER_OF_PROCESSORS) "$(TargetPath)"

Example

Every MPI program includes

  • the MPI header file
  • a function call to initialize MPI
  • a function call to close MPI

MPI executable statements take the form of calls to the MPI API functions.  The identifier-naming convention in C prefixes all MPI names with MPI_, capitalizes the first letter of the name and lower-cases the remaining letters. 

 // An MPI Program - Hello World
 // mpi.cpp

 #include <cstdio>
 #include <mpi.h> // header file

 int main(int argc, char** argv) {
     MPI_Init(&argc, &argv); // initializing MPI
     // MPI statements here ... parallel processing 
     std::printf("Hello from an MPI process\n");
     MPI_Finalize(); // closing MPI
 }




 Hello from an MPI process
 Hello from an MPI process
 Hello from an MPI process
 Hello from an MPI process
 Hello from an MPI process
 Hello from an MPI process
 Hello from an MPI process
 Hello from an MPI process 

MPI_Init initializes the execution environment using the command line arguments. 
MPI_Finalize() closes the MPI environment and releases any resources used by MPI. 

Multiple Machines

To execute an MPI program on multiple machines, we need to provide a hostfile with their IP addresses.  For example, the hostfile may contain

 192.168.0.4
 192.168.0.5
 192.168.0.100

Once we have defined the host file, we can use the following command to spawn processes across the multiple machines. 

 mpirun -np 4 -hostfile hosts mpi_1

MPI distributes the processes over the named hosts in a round-robin fashion. 

If a processor has several cores, we can identify the number of cores that the processor is to use in the hostfile through the slots modifier: 

 192.168.0.4 slots=8
 192.168.0.5
 192.168.0.100

Exercises




Previous Reading  Previous: Parallel Patterns Next: OpenMP - Fundamentals   Next Reading


  Designed by Chris Szalwinski   Copying From This Site   

Creative Commons License