Objectives and Student Learning

Outcomes
This course, Introduction to Parallel Programming is intended to create a fundamental understanding of
how computation is performed so that the benefits of using many computing elements at the same time can
be realized. Many computing elements working on the same problem can deliver large performance gains
over one or two computing elements. This course maps the body of knowledge on parallel programming
found in computer science onto innately parallel hardware like Graphics Processing Units (GPU) and multi-
core CPUs. This is a hands-on project oriented course where you will build a portfolio of programs as a
means of understanding parallel programming. The list of programs can be found at the end of this
orientation topic.

This course is a portfolio based course. You will create a portfolio of programs where each program
introduces you to a specific type of program and in developing the code, you learn the necessary skills. The
portfolio programs are described in a separate page. Reading over this page should help you formulate a
plan to master the framework, fill in any skill gaps, and submit your source code as evidence of completion.
You must submit the portfolio programs in sequence. When you submit the assigned program you will have
access to a page with the exit ticket for that topic. Clicking and downloading on the ticket image will unlock
the next section and begin working on the next portfolio program.

This course emphasizes cohort learning. There is a lot of information to master and not everyone starts with
the same knowledge and skill set. In addition, each person that uses their own computer has a development
environment. The topic forums are a good way to share information about configurations, useful resources,
tips, tutorials. The only thing you should not share is the code you write to implement a portfolio project.
Each person must submit their own work.

Student Learning Outcomes

Upon completion of this course a student will be able to:

A. Describe how a processor executes a single threaded program

B. Use a software model as a tool for investigation, research, and experimentation
C. Describe processes, threads, and the resources used by a running program
D. Write programs that use more than one thread, process or task concurrently to solve a problem
E. Build parallel programs using class inheritance and polymorphism
F. Write programs that use interprocess communication for cooperation and synchronization, and
POSIX standards for portability in parallel programs
G. Test multi-threaded programs
H. Write programs that utilize parallel hardware using standards such as CUDA, OpenCL,
and MPI
I. Write programs for execution on: single system, GPU, cloud or Cluster
Your Portfolio of Projects
CS280 Portfolio of Programs

These programs are assignments that teach you specific aspects of parallel programming. You will be
provided with a recommended way to structure your project, where each portfolio program is encapsulated
within a C++ class.

1. Count3sProcessSerial.cpp - This program opens a binary file containing 32 bit integer numbers (ints).
Your program will read the file into an array and count the number of occurrences of the value 3 in the
array. Your goal is to produce a well understood serial algorithm for solving this problem. This
algorithm will run in a single program execution instance (a process). The algorithm you develop here
will form the basis for parallel versions of the program.
2. Count3sProcessParallel.cpp - This program opens a binary file containing 32 bit signed integer
numbers (ints). Your program will read the file into memory and count the number of occurrences of
the value 3 in this array. This program will divide the work between two processes; multiple instances
of the same program running concurrently. You will develop the skills of dividing the work among
independent units of computation (the map operation), assigning work to each process, and
gathering the result (the reduce operation) into a final count.
3. Count3sThreadParallel.cpp - This program opens a binary file containing 32 bit signed integer
numbers (ints). Your program will read the file into an array and count the number of occurrences of
the value 3 in this array. This program will divide the work between several threads running in a single
program execution instance (process) sharing the same process memory space. You will develop the
skills of coordinating data access and communication using the pthread library and resources such as
semaphores and introduce the concepts of critical sections. The pthread is available across many
platforms and can be used within other parallel technologies to add fine-grained control of
computation.
4. Count3sIPC.cpp - This program opens a binary file containing 32 bit signed integer numbers (ints)
and uses Inter-Process Communication (IPC) capabilities provided by the operating system to share
the data among simultaneous concurrent processes. IPC resources are local to a compute node and
very efficient. They are used to implement the operating system so they are robust and well tested.
You can use IPC as part of your toolkit when doing parallel processing on a single node.
5. RegisterSet.cpp - This program simulates the register, a core component of a Central Processing
Unit (CPU). Registers are a very fast memory store inside the CPU that is used for operations by the
Arithmetic Logic Unit. One of the very first steps to increasing performance is increasing the number
of registers. This program helps you formulate an understanding of registers, operations, and flags;
bits that are set as part of an operation to indicate common results such as: register contents are
zero, register contents are negative. It develops detailed knowledge of data types and their
corresponding memory formats. The resulting insight into address alignment and storage
 requirements of specific data types enables you to avoid bus errors and understand constraints on
GPU registers, operations, and Instruction Level Parallelism (ILP).
6. ALU.cpp, Memory.cpp, IllegalArgumentException.cpp - This program uses the array of data types
created in Registerset in simple operations that correspond to Machine Language; load, store, move,
add, sub, etc. To implement this code you will create the following classes:
1. ALU.cpp - Implement a simple instruction set that invokes register operations and sets flags.
2. Memory.cpp - a class that implements the flat memory space provided to most serial
programs (RAM - the random access model) and simple parallel models (PRAM - Parallel
Random Access Model.)
3. IllegalArgumentException - an exception that is thrown whenever an illegal value is used in an
operation.

Memory.cpp simulates a flat memory space of 512 addresses using an internal array. Each element of
the array address should be able to store any of the following data types: 32 bit signed integer, 64 bit
signed integer, 32 bit float, or 64 bit float. This class should provide two methods:

write(unsigned int address, registertype value) - store the value at the memory address indicated.

registertype read(unsigned int address) - return the value stored at the indicated address.

Any attempt to access an invalid address should throw IllegalArgumentException.

7. Count3sCoprocessor.cpp - This program opens a binary file containing 32 bit floating point numbers
(floats) and distributes the values to a Graphics Processing Unit (GPU) to perform the computation.
Your program will detect and configure a parallel coprocessor, configure it, and use it to perform a
parallel version of your serial Count3sProcessSerial algorithm. You will develop the skills of
communication and data transfer across the PCIe bus visualizing and using a fixed processor &
memory topology, and the best practice of checking coprocessor results against a serial CPU
implementation of the same algorithm.This project also contains Count3s_kernel.cu - This program
is a Cuda kernel that uses a Graphics Processing Unit (GPU) to implement the Count3s algorithm. It is
called from Count3sCoprocessor and the results returned should be checked against the serial CPU
implementation of the same algorithm for correctness.
8. Count3sMPI.cpp - This program opens a binary file containing 32 bit floating point numbers (floats)
and distributes the values to parallel computing resources using the Message Passing Interface (MPI)
library. MPI is a standard created by computing laboratories and academic personnel to use a variety
of different computers in parallel. It is the most common method of writing programs that can scale
from a single node to mulitple clusters incorporating thousands of processors.
9. Count3sOpenCL.cpp - OpenCL is a GPU programming language that is widely available on a broad
range of compute-capable GPUs and CPUs. Your code will discover OpenCL compute devices, select
and configure a device for processing and use this device to count the number of occurrences of the
value 3 in an array of signed ints.
10. Timer.cpp - A class that can provide metrics for measuring performance and speedup in your
programs.
 11. VectorAdd_GPU.cpp - A CUDA program that adds two vectors Matrix_Multiply_GPU.cpp - A CUDA
program that multiplies two matrices
12. ParallelTeam.cpp - This program uses object oriented programming to hide differences in
Application Programming Interfaces (API). The techniques of encapsulation (information hiding) and
polymorphism. Through polymorphism, a method discovers the correct body of code to use at run
time. This allows different APIs to behave in consistent ways according to your design and promotes
re-use of tested code. You will practice design and implementation of a class hierarchy, use of
inheritance and formulate an understanding of abstract and concrete classes.

Weekly Access to the Online Course is

Mandatory
Attendance is mandatory for this course. If the Learning Management System (LMS) indicates that you are
inactive for 2 or more weeks I may drop you from the class with or without notice. Always notify me if your
other obligations call you away for an extended period of time.

What is the Student Header and Why

Must It Appear in My Programs?
What is the Student Header and Why Do I Need It?

The student header ensures that you get credit for your work. Many views in Canvas do not display
information about which student's work I am currently viewing. Having your student header in the file lets
me easily identify the right place to enter grades.

Furthermore, there are often several sections of the same course offered in the same semester, and from
semester to semester. In addition to manual grading there are several tools used to grade programs and
work you submit. The student header is the first comment in your code and is required in order to identify
which student in what section of a course has submitted an assignment. You are graded on the student
header and you can lose up to 10 points of your grade if the student header is incomplete or missing. The
format is:

/*

Name: Student Name

Course: Title of the course

CRN: 012345 -The CRN Number of this section (found online and in the syllabus)
Assignment: Assignment name and/or number

Date: Date of this submission

*/

In cases where you re-submit your work after making changes, be sure to update the Date field so it can be
considered for re-grading.