Analysis of parallel algorithms

This article discusses the analysis of parallel algorithms running on parallel random-access machines (PRAMs): models of parallel computation where an unbounded number of processors have shared access to a random-access memory (RAM).

Overview

Suppose computations are executed on a PRAM machine using $p$ processors. Let $T p$ denote the time that expires between the start of the computation and its end. Analysis of the computation's running time focuses on the following notions:

The work of a computation executed by $p$ processors is the total number of primitive operations that the processors perform.^[1] Ignoring communication overhead from synchronizing the processors, this is equal to the time used to run the computation on a single processor, denoted $T 1$ .
The span is the time $T \infty$ spent computing using an idealized machine with an infinite number of processors.^[2]
The cost of the computation is the quantity $pT p$ . This expresses the total time spent, by all processors, in both computing and waiting.^[1]

Several useful results follow from the definitions of work, span and cost:

Work law. The cost is always at least the work: $pT p \geq T 1$ . This follows from the fact $p$ processors can perform at most $p$ operations in parallel.^[2]^[1]
Span law. A finite number $p$ of processors cannot outperform an infinite number, so that $T p \geq T \infty$ .^[2]

Using these definitions and laws, the following measures of performance can be given:

Speedup is the gain in speed made by parallel execution compared to sequential execution: $S p = T p / T 1$ . When the speedup is $Ω(n)$ for input size $n$ (using big O notation), the speedup is linear, which is optimal in the PRAM model (super-linear speedup can occur in practice due to memory hierarchy effects). The situation $T p / T 1 = p$ is called perfect linear speedup.^[2] An algorithm that exhibits linear speedup is said to be scalable.^[1]
Efficiency is the speedup per processor, $S p / p$ .^[1]

References

^ ^a ^b ^c ^d ^e Casanova, Henri; Legrand, Arnaud; Robert, Yves (2008). Parallel Algorithms. CRC Press. p. 10.
^ ^a ^b ^c ^d Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. Introduction to Algorithms (3rd ed.). MIT Press and McGraw-Hill. ISBN 0-262-03384-4.

[casanova-1] Casanova, Henri; Legrand, Arnaud; Robert, Yves (2008). Parallel Algorithms. CRC Press. p. 10.

[clrs-2] Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. Introduction to Algorithms (3rd ed.). MIT Press and McGraw-Hill. ISBN 0-262-03384-4.

[1]

[2]

v t e Parallel computing
General	Distributed computing Parallel computing Parallel algorithm Massively parallel Cloud computing High-performance computing Multiprocessing Manycore processor GPGPU Computer network Systolic array
Levels	Bit Instruction Thread Task Data Memory Loop Pipeline
Multithreading	Temporal Simultaneous (SMT) Simultaneous and heterogenous Speculative (SpMT) Preemptive Cooperative Clustered multi-thread (CMT) Hardware scout
Theory	PRAM model PEM model Analysis of parallel algorithms Amdahl's law Gustafson's law Cost efficiency Karp–Flatt metric Slowdown Speedup
Elements	Process Thread Fiber Instruction window Array
Coordination	Multiprocessing Memory coherence Cache coherence Cache invalidation Barrier Synchronization Application checkpointing
Programming	Stream processing Dataflow programming Models Implicit parallelism Explicit parallelism Concurrency Non-blocking algorithm
Hardware	Flynn's taxonomy SISD SIMD Array processing (SIMT) Pipelined processing Associative processing MISD MIMD Dataflow architecture Pipelined processor Superscalar processor Vector processor Multiprocessor symmetric asymmetric Memory shared distributed distributed shared UMA NUMA COMA Massively parallel computer Computer cluster Beowulf cluster Grid computer Hardware acceleration
APIs	Ateji PX Boost Chapel HPX Charm++ Cilk Coarray Fortran CUDA Dryad C++ AMP Global Arrays GPUOpen MPI OpenMP OpenCL OpenHMPP OpenACC Parallel Extensions PVM pthreads RaftLib ROCm UPC TBB ZPL
Problems	Automatic parallelization Deadlock Deterministic algorithm Embarrassingly parallel Parallel slowdown Race condition Software lockout Scalability Starvation
Category: Parallel computing