Transcript LEC01 - aiub study guide

CSC2105: Algorithms
Mashiour Rahman
[email protected]
American International University Bangladesh
Literature



Introduction to Algorithms, Second Edition,
Thomas H. Cormen, Charle E. Leiserson,
Ronald L. Rivest, Clifford Stein (CLRS).
Fundamental of Computer Algorithms, Ellis
Horowitz, Sartaj Sahni, Sanguthevar
Rajasekaran (HSR).
Helpful link for Problem Solving :
http://acm.uva.es/problemset/
CSC2105
Mashiour Rahman
2
The Goals of this Course

The main things we will learn in this course:




To think algorithmically and get the spirit of how
algorithms are designed.
To get to know a toolbox of classical algorithms.
To learn a number of algorithm design techniques
(such as divide-and-conquer).
To reason (in a precise and formal way) about the
efficiency and the correctness of algorithms.
CSC2105
Mashiour Rahman
3
General



Algorithms are first solved on paper and later
keyed in on the computer.
The most important thing is to be simple and
precise.
During lectures:



CSC2105
Interaction is welcome; ask questions.
Additional explanations and examples if desired.
Speed up/slow down the progress.
Mashiour Rahman
4
Prerequisites

Introduction to programming






Data types, operations
Conditional statements
Loops
Procedures and functions
C/C++/Java
Computer lab (edit, compile, execute)
CSC2105
Mashiour Rahman
5
History




Name: Persian mathematician Mohammed alKhowarizmi, in Latin became Algorismus.
First algorithm: Euclidean Algorithm, greatest
common divisor, 400-300 B.C.
19th century – Charles Babbage, Ada Lovelace.
20th century – Alan Turing, Alonzo Church,
John von Neumann.
CSC2105
Mashiour Rahman
6
Data Structures and Algorithms

Data structure


Algorithm


Organization of data to solve the problem at hand.
Outline, the essence of a computational procedure,
step-by-step instructions.
Program

CSC2105
Implementation of an algorithm in some
programming language.
Mashiour Rahman
7
Overall Picture

Using a computer to
help solve problems.




CSC2105
Precisely specify the
problem.
Designing programs
 architecture
 algorithms
Writing programs
Verifying (testing)
programs
Data Structure and
Algorithm Design Goals
Correctness
Efficiency
Implementation Goals
Reusability
Robustness
Adaptability
Mashiour Rahman
8
Overall Picture/2

This course is not about:




Programming languages
Computer architecture
Software architecture
Software design and implementation principles


Issues concerning small and large scale programming.
We will only touch upon the theory of
complexity and computability.
CSC2105
Mashiour Rahman
9
Algorithmic problem
Specification
of input

Infinite number of input instances satisfying the
specification. For example:

A sorted, non-decreasing sequence of natural numbers.
The sequence is of non-zero, finite length:


CSC2105
?
Specification
of output as
a function of
input
1, 20, 908, 909, 100000, 1000000000.
3.
Mashiour Rahman
10
Algorithmic Solution
Input instance,
adhering to
the
specification


CSC2105
Algorithm
Output
related to
the input as
required
Algorithm describes actions on the input instance.
There may be many correct algorithms for the
same algorithmic problem.
Mashiour Rahman
11
Definition of an Algorithm


An algorithm is a sequence of unambiguous
instructions for solving a problem, i.e., for
obtaining a required output for any legitimate
input in a finite amount of time.
Properties:



Precision
Determinism
Finiteness



CSC2105
Mashiour Rahman
Efficiency
Correctness
Generality
12
How to Develop an Algorithm


Precisely define the problem. Precisely specify the
input and output. Consider all cases.
Come up with a simple plan to solve the problem at
hand.



The plan is language independent.
The precise problem specification influences the plan.
Turn the plan into an implementation

CSC2105
The problem representation (data structure) influences the
implementation.
Mashiour Rahman
13
Preconditions, Postconditions

It is important to specify the preconditions and
the postconditions of algorithms:


CSC2105
INPUT: precise specifications of what the
algorithm gets as an input.
OUTPUT: precise specifications of what the
algorithm produces as an output, and how this
relates to the input. The handling of special cases
of the input should be described.
Mashiour Rahman
14
Analysis of Algorithms

Efficiency:



Running time
Space used
Efficiency as a function of the input size:


CSC2105
Number of data elements (numbers, points).
The number of bits of an input number .
Mashiour Rahman
15
The RAM model


It is important to choose the level of detail.
The RAM model:

Instructions (each taking constant time), we
usually choose one type of instruction as a
characteristic operation that is counted:





CSC2105
Arithmetic (add, subtract, multiply, etc.)
Data movement (assign)
Control flow (branch, subroutine call, return)
Comparison
Data types – integers, characters, and floats
Mashiour Rahman
16
Analysis of Insertion Sort

Time to compute the running time as a function of
the input size (exact analysis).
for j := 2 to n do
key := A[j]
// Insert A[j] into A[1..j-1]
i := j-1
while i>0 and A[i]>key do
A[i+1]:=A[i]
i-A[i+1]:=key
CSC2105
Mashiour Rahman
cost
c1
c2
0
c3
c4
c5
c6
c7
times
n
n-1
n-1
n-1



n
t
(t  1)
jn = 2 j
(t  1)
j =2 j
j
nj = 2
n-1
17
Analysis of Insertion Sort/2



The running time of an algorithm is the sum of
the running times of each state-ment.
A statement with cost c that is executed n
times contributes c*n to the running time.
The total running time T(n) of insertion sort is

T(n) = c1*n + c2(n-1) + c3(n-1) + c4 j = 2 t j+
n
n
(
t

1)
c5  j =2 j + c6  j =2 (t j  1)+ c7(n-1)
CSC2105
n
Mashiour Rahman
18
Analysis of Insertion Sort/3



Often the performance depends on the details
of the input (not only the length n).
This is modeled by tj.
In the case of insertion sort the time tj depends
on the original sorting of the input array.
CSC2105
Mashiour Rahman
19
Performance Analysis

Often it is sufficient to count the number of iterations
of the core (innermost) part.



No distinction between comparisons, assignments, etc (that
means roughly the same cost for all of them).
Gives precise enough results.
In some cases the cost of selected operations
dominates all other costs.


CSC2105
Disk I/O versus RAM operations.
Database systems.
Mashiour Rahman
20
Best/Worst/Average Case

Analyzing insertion sort’s



CSC2105

n
j =2
(t j  1)
Best case: elements already sorted, tj=1, running
time = n-1, i.e., linear time.
Worst case: elements are sorted in inverse order,
tj=j-1, running time =
(n2-n)/2, i.e., quadratic time.
Average case: tj=j/2, running time =
(n2+n-2)/4, i.e., quadratic time.
Mashiour Rahman
21
Best/Worst/Average Case/3

For inputs of all sizes:
worst-case
average-case
Running time
6n
5n
best-case
4n
3n
2n
1n
1
CSC2105
2
3
4
5
6
7
8
9 10 11 12 …..
InputMashiour
instance
size
Rahman
22
Best/Worst/Average Case/4

Worst case is usually used:





CSC2105
It is an upper-bound.
In certain application domains (e.g., air traffic
control, surgery) knowing the worst-case time
complexity is of crucial importance.
For some algorithms worst case occurs fairly oftenThe average case is often as bad as the worst case.
Finding the average case can be very difficult.
Mashiour Rahman
23
Analysis of Linear Search
INPUT: A[1..n] – a sorted array of integers, q – an integer.
OUTPUT: an index j such that A[j] = q. NIL if "j (1jn): A[j]  q
j := 1
while j  n and A[j]  q do j++
if j  n then return j
else return NIL



Worst case running time: n
Average case running time: n/2
Best case running time: 0
CSC2105
Mashiour Rahman
24
Binary Search


Idea: Have a left and right bound. Elements to
the right of r are bigger than the search element.
Equivalent for l.
In each step reduce the range of the search space
by half.
INPUT: A[1..n] – a sorted array of (increasing) integers, q – an integer.
OUTPUT: an index j such that A[j] = q. NIL, if "j (1jn): A[j]  q
l := 1; r := n
do
m := (l+r)/2
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l <= r
return NIL
CSC2105
Mashiour Rahman
25
Analysis of Binary Search

How many times the loop is executed?

With each execution the difference between l and r
is cut in half.




CSC2105
Initially the difference is n.
The loop stops when the difference becomes 0 (less than
1) .
How many times do you have to cut n in half to get
0?
log n – better than the brute-force approach of
linear search (n).
Mashiour Rahman
26
Linear Search vs Binary Search




Costs of linear search: n
Costs of binary search: log(n)
Should we care?
Phone book with 200’000 entries:


CSC2105
n = 200’000
log n = log 200’000 = 17.6
Mashiour Rahman
27
Asymptotic Analysis

Goal: to simplify the analysis of the running time by
getting rid of details, which are affected by specific
implementation and hardware



“rounding” of numbers: 1,000,001  1,000,000
“rounding” of functions: 3n2  n2
Capturing the essence: how the running time of an
algorithm increases with the size of the input in the
limit.

CSC2105
Asymptotically more efficient algorithms are best for all
but small inputs
Mashiour Rahman
28
Asymptotic Notation
The “big-Oh” O-Notation




asymptotic upper bound
f(n) = O(g(n)), if there exists
constants c>0 and n0>0, s.t.
f(n)  c g(n) for n  n0
f(n) and g(n) are functions
over non-negative integers
Used for worst-case
analysis
CSC2105
Mashiour Rahman
c  g ( n)
f (n )
Running Time

n0
Input Size
29
Asymptotic Notation/2
The “big-Omega” WNotation



asymptotic lower bound
f(n) = W(g(n)) if there exists
constants c>0 and n0>0, s.t.
c g(n)  f(n) for n  n0
Used to describe best-case
running times or lower bounds
of algorithmic problems.

E.g., lower-bound of searching in
an unsorted array is W(n).
CSC2105
Mashiour Rahman
f (n )
c  g ( n)
Running Time

n0
Input Size
30
Asymptotic Notation/3

Simple Rule: Drop lower order terms and
constant factors.




50 n log n is O(n log n)
7n - 3 is O(n)
8n2 log n + 5n2 + n is O(n2 log n)
Note: Although (50 n log n) is O(n5), it is
expected that an approximation is of the
smallest possible order.
CSC2105
Mashiour Rahman
31
Asymptotic Notation/4
The “big-Theta” QNotation




asymptoticly tight bound
f(n) = Q(g(n)) if there exists
constants c1>0, c2>0, and n0>0,
s.t. for n  n0
c1 g(n)  f(n)  c2 g(n)
f(n) = Q(g(n)) if and only if
f(n) = O(g(n)) and
f(n) = W(g(n))
O(f(n)) is often abused instead
of Q(f(n))
CSC2105
Mashiour Rahman
c 2  g (n )
f (n )
Running Time

c 1  g (n )
n0
Input Size
32
A Quick Math Refresher

Arithmetic progression
n
 i = 1  2  3  ...  n =
i =0

n(1  n)
2
Geometric progression

given an integer n0 and a real number 0<a1
n 1
1

a
ai = 1  a  a 2  ...  a n =

1 a
i =0
n

CSC2105
geometric progressions exhibit exponential growth
Mashiour Rahman
33
Miscellaneous

Manipulating logarithms:
 a
log b = log b / log a


log ab = b log a
Manipulating summations:

 ca = c a
 j (aj  bj) =  j aj   j bj
j

CSC2105
j
j
j
Mashiour Rahman
34
Summations

The running time of insertion sort is
determined by a nested loop.
for j := 2 to n
key := A[j]
i := j-1
while i>0 and A[i]>key
A[i+1] := A[i]
i := i-1
A[i+1] := key

Nested loops correspond to summations:
n
 j =2 ( j  1)
CSC2105
Mashiour Rahman
35
Proof by Induction




We want to show that property P is true for all
integers n  n0.
Basis: prove that P is true for n0.
Inductive step: prove that if P is true for all k such
that n0  k  n – 1 then P is also true for n.
n
n(n  1)
Example
S ( n) =  i =
for n  1
2
i =0
1

Basis
S (1) =  i =
i =0
CSC2105
1(1  1)
2
Mashiour Rahman
36
Proof by Induction/2

Inductive Step
k ( k  1)
S (k ) =  i =
for 1  k  n  1
2
i =0
k
n
n 1
i =0
i =0
S (n) =  i = i  n =S (n  1)  n =
(n  1  1)
( n 2  n  2n)
= (n  1)
n=
=
2
2
n( n  1)
=
2
CSC2105
Mashiour Rahman
37
Correctness of Algorithms



An algorithm is correct if for any legal input it
terminates and produces the desired output.
Automatic proof of correctness is not possible.
There are practical techniques and rigorous
formalisms that help to reason about the
correctness of (parts of) algorithms.
CSC2105
Mashiour Rahman
38
Partial and Total Correctness

Partial correctness
IF this point is reached,
Any legal input

Algorithm
THEN this is the desired output
Output
Total correctness
INDEED this point is reached, AND this is the desired output
Any legal input
CSC2105
Algorithm
Mashiour Rahman
Output
39
Assertions

To prove partial correctness we associate a number of
assertions (statements about the state of the
execution) with specific checkpoints in the algorithm.



E.g., A[1], …, A[j] form an increasing sequence
Preconditions – assertions that must be valid before
the execution of an algorithm or a subroutine
(INPUT).
Postconditions – assertions that must be valid after
the execution of an algorithm or a subroutine
(OUTPUT).
CSC2105
Mashiour Rahman
40
Pre/post-conditions

Example:



Write a pseudocode algorithm to find the two
smallest numbers in a sequence of numbers (given
as an array).
INPUT: an array of integers A[1..n], n > 0
OUTPUT: (m1, m2) such that


CSC2105
m1<m2 and for each i[1..n]: m1  A[i] and, if
A[i]  m1, then m2  A[i].
m2 = m1 = A[1] if "j,i[1..n]: A[i]=A[j]
Mashiour Rahman
41
Loop Invariants


Invariants: assertions that are valid any time they are
reached (many times during the execution of an
algorithm, e.g., in loops)
We must show three things about loop invariants:



CSC2105
Initialization: it is true prior to the first iteration.
Maintenance: if it is true before an iteration, then it is true
after the iteration.
Termination: when a loop terminates the invariant gives a
useful property to show the correctness of the algorithm
Mashiour Rahman
42
Example: Binary Search/1
We want to show that q is
not in A if NIL is returned.
Invariant:
"i[1..l-1]: A[i]<q
(Ia)
"i[r+1..n]: A[i]>q (Ib)





l := 1
r := n
do
m := (l+r)/2
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l <= r
return NIL
Initialization: l = 1, r = n the invariant holds because there are
no elements to the left of l or to the right of r.
l=1 yields "j,i [1..0]: A[i]<q
this holds because [1..0] is empty
r=n yields "j,i [n+1..n]: A[i]>q
this holds because [n+1..n] is empty
CSC2105
Mashiour Rahman
43
Example: Binary Search/2
Invariant:
"i[1..l-1]: A[i]<q (Ia)
"i[r+1..n]: A[i]>q (Ib)




l := 1
r := n
do
m := (l+r)/2
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l <= r
return NIL
Maintenance: l, r, m = (l+r)/2
A[m]!=q & A[m]>q, r=m-1, A sorted implies
"k[r+1..n]: A[k]>q (Ib)
A[m]!=q & A[m]<q, l=m+1, A sorted implies
"k[1..l-1]: A[k]<q (Ib)
CSC2105
Mashiour Rahman
44
Example: Binary Search/3



Invariant:
"i[1..l-1]: A[i]<q (Ia)
"i[r+1..n]: A[i]>q (Ib)
Termination: l, r, l<=r
Two cases:



l := 1
r := n
do
m := (l+r)/2
if A[m] = q then return m
else if A[m] > q then r := m-1
else l := m+1
while l <= r
return NIL
l:=m+1 we get (l+r)/2 +1 > l
r:=m-1 we get (l+r)/2 -1 < r
The range gets smaller during each iteration and the loop will
terminate when l<=r no longer holds.
CSC2105
Mashiour Rahman
45
Example: Insertion Sort/1



for j := 2 to n do
key := A[j]
i := j-1
while i>0 and A[i]>key do
A[i+1] := A[i]
i-A[i+1] := key
Invariant:
outside while loop

A[1...j-1] is sorted

A[1...j-1]  Aorig
inside while loop:




A[1...i], key, A[i+1…j-1]
A[1...i] is sorted
A[i+1…j-1] is sorted
A[k] > key, i+1<=k<=j-1
CSC2105
Mashiour Rahman
46
Example: Insertion Sort/2

outside while loop



A[1...j-1] is sorted
A[1...j-1]  Aorig
inside while loop:




A[1...i], key, A[i+1…j-1]
A[1...i] is sorted
A[i+1…j-1] is sorted
A[k] > key, i+1<=k<=j-1
for j := 2 to length(A)do
key := A[j]
i := j-1
while i>0 and A[i]>key do
A[i+1] := A[i]
i-A[i+1] := key
Initialization:
 j=2: the invariant holds, A[1…1] is trivially sorted.
 i=j-1: A[1...j-1], key, A[j…j-1] where key=A[j]
A[j…j-1] is empty (and thus trivially sorted)
A[1…j-1] is sorted (invariant of outer loop)
CSC2105
Mashiour Rahman
47
Example: Insertion Sort/2

outside while loop



A[1…j-1] is sorted
A[1...j-1]  Aorig
inside while loop:




A[1...i], key, A[i+1…j-1]
A[1...i] is sorted
A[i+1…j-1] is sorted
A[k] > key, i+1<=k<=j-1
for j := 2 to length(A)do
key := A[j]
i := j-1
while i>0 and A[i]>key do
A[i+1] := A[i]
i-A[i+1] := key
Maintenance:


(A[1…j-1] sorted) + (insert A[j])
 A[1…j] sorted).
A[1...i-1], key, A[i,i+1…j-1] satisfies conditions because
of condition A[i]>key and A[1...j-1] being sorted.
CSC2105
Mashiour Rahman
48
Example: Insertion Sort/2

outside while loop



A[1…j-1] is sorted
A[1...j-1]  Aorig
inside while loop:




A[1...i], key, A[i+1…j-1]
A[1...i] is sorted
A[i+1…j-1] is sorted
A[k] > key, i+1<=k<=j-1
for j := 2 to length(A)do
key := A[j]
i := j-1
while i>0 and A[i]>key do
A[i+1] := A[i]
i-A[i+1] := key
Termination:
 main loop, j=n+1: A[1…n] sorted.
 A[i]key: (A[1...i], key, A[i+1…j-1]) = A[1…j-1] is sorted
 i=0: (key, A[1…j-1]) = A[1…j-1] is sorted.
CSC2105
Mashiour Rahman
49
Recursion

An object is recursive if



it contains itself as part of it, or
it is defined in terms of itself.
Factorial: n!



CSC2105
How do you compute 10!?
n! = 1 * 2 * 3 *...* n
n! = n * (n-1)!
Mashiour Rahman
50
Factorial Function

Pseudocode of factorial:
fac1
INPUT: n – a natural number.
OUTPUT: n! (factorial of n)
fac1(n)
if n < 2 then return 1
else return n * fac1(n-1)

A recursive procedure includes a


CSC2105
Termination condition (determines when and how
to stop the recursion).
One (or more) recursive calls.
Mashiour Rahman
51
Tracing the Execution
6
fac(3)
3 * fac(2)
2
fac(2)
2 * fac(1)
1
fac(1)
1
CSC2105
Mashiour Rahman
52
Bookkeeping

The computer maintains an activation stack for
active procedure calls (-> compiler
construction). Example for fac(5).
CSC2105
fac(1)
1
fac(2)
2*fac(1)
1
fac(3)
3*fac(2)
2
fac(4)
4*fac(3)
6
fac(5)
5*fac(4)
24
120
Mashiour Rahman
53
Variants of Factorial
fac2
INPUT: n – a natural number.
OUTPUT: n! (factorial of n)
fac2(n)
if n < 2 then return 1
return n * fac2(n-1)
fac3
INPUT: n – a natural number.
OUTPUT: n! (factorial of n)
fac3(n)
return n * fac3(n-1)
if n  2 then return 1
CSC2105
Mashiour Rahman
54
Analysis of the solutions

fac2 is correct


The return statement in the if clause terminates the
function and, thus, the entire recursion.
fac3 is incorrect

CSC2105
Infinite recursion. The termination condition is
never reached.
Mashiour Rahman
55
Fibonacci Numbers

Definition





fib(1) = 1
fib(2) = 1
fib(n) = fib(n-1) + fib(n-2), n>2
Numbers in the series:
1, 1, 2, 3, 5, 8, 13, 21, 34, ...
CSC2105
Mashiour Rahman
56
Fibonacci Implementation
fib
INPUT: n – a natural number larger than 0.
OUTPUT: fib(n), the nth Fibonacci number.
fib(n)
if n  2 then return 1
else return fib(n-1) + fib(n-2)

Multiple recursive calls are possible.
CSC2105
Mashiour Rahman
57
Fibonacci Implementation/2
int fib(int i) {
if (i <= 2) { return 1;}
else { return fib(i-1) + fib(i-2); }
}
int main() {
printf(“Fibonacci of 5 is %d\n", fac(5));
}
CSC2105
Mashiour Rahman
58
Tracing fib(4)
3
fib(4)
fib(3) + fib(2)
1
2
fib(3)
fib(2) + fib(1)
fib(2)
1
1
1
fib(2)
1
CSC2105
fib(1)
1
Mashiour Rahman
59
Bookkeeping

Activation stack for fib(4).
CSC2105
fib(1)
fib(2)
1
fib(2)
fib(3)
1
fib(2)
1
+
fib(1) 1
fib(4)
fib(3) 2
+
fib(2) 1
3
Mashiour Rahman
60
Mutual Recursion


Recursion does not always occur because a
procedure calls itself.
Mutual recursion occurs if two procedures call
each other.
A
B
CSC2105
Mashiour Rahman
61
Mutual Recursion Example

Problem: Determine whether a natural number
is even.

Definition of even:



CSC2105
0 is even
N is odd if N-1 is even
N is even if N-1 is odd
Mashiour Rahman
62
Implementation of even
even
INPUT: n – a natural number.
OUTPUT: true if n is even; false otherwise
odd(n)
if n = 0 then return TRUE
return !even(n-1)
even(n)
if n = 0 then return TRUE
else return !odd(n-1)

Can it be used to determine whether a number
is odd?
CSC2105
Mashiour Rahman
63
Is Recursion Necessary?




Theory: You can always resort to iteration and
explicitly maintain a recursion stack.
Practice: Recursion is elegant and in some cases the
best solution by far.
In the previous examples recursion was never
appropriate since there exist simple iterative
solutions.
Recursion is more expensive than corresponding
iterative solutions since bookkeeping is necessary.
CSC2105
Mashiour Rahman
64
Sorting



Sorting is a classical and important algorithmic
problem.
We look at sorting arrays (in contrast to files, which
restrict random access).
A key constraint is the efficient management of the
space


In-place sorting algorithms
The efficiency comparison is based on the number of
comparisons (C) and the number of movements (M).
CSC2105
Mashiour Rahman
65
Sorting

Simple sorting methods use roughly n * n
comparisons




Insertion sort
Selection sort
Bubble sort
Fast sorting methods use roughly n * log n
comparisons.



CSC2105
Merge sort
Heap sort
Quicksort
Mashiour Rahman
66
Example 2: Sorting
INPUT
OUTPUT
sequence of n numbers
a permutation of the
input sequence of numbers
a1, a2, a3,….,an
2
5
4
10
b1,b2,b3,….,bn
Sort
2
7
4
5
7
10
Correctness (requirements for the output)
For any given input the algorithm halts with the output:
• b1 < b2 < b3 < …. < bn
• b1, b2, b3, …., bn is a permutation of a1, a2, a3,….,an
CSC2105
Mashiour Rahman
67
Insertion Sort
A
3
4
6
1
9
i
Strategy
•
Start with one sorted card.
•
Insert an unsorted card at the
correct position in the sorted
part.
•
Continue until all unsorted
cards are inserted/sorted.
CSC2105
8
7
2
5 1
j
n
A
Mashiour Rahman
44
44
12
12
12
12
06
06
55
55
44
42
42
18
12
12
12
12
55
44
44
42
18
18
42
42
42
55
55
44
42
42
94
94
94
94
94
55
44
44
18
18
18
18
18
94
55
55
06
06
06
06
06
06
94
67
67
67
67
67
67
67
67
94
68
Insertion Sort/2
INPUT: A[1..n] – an array of integers
OUTPUT: a permutation of A such that A[1] A[2] …A[n]
for j := 2 to n do
key := A[j]
j
i := j-1
while i > 0 and A[i] > key do
A[i+1] := A[i]; i-A[j+1] := key
n
j =2

The number of comparisons during the jth
iteration is


CSC2105
at least 1: Cmin =  j =21 = n-1
n
at most j-1: Cmax = j =2 j  1 = (n*n-n-2)/2
n
Mashiour Rahman
69
Insertion Sort/3

The number of comparisons during the jth
iteration is:
j / 2 = (n*n+n–2)/4
 j/2 in average: Cavg = 
The number of movements is Ci+1:
n
j =2

 2 = 2*(n-1),
j / 2  1= (n*n+5n-6)/4
 Mavg =
 Mmax =
 j = (n*n+n-2)/2

n
Mmin =
j =2
n
j =2
n
j =2
CSC2105
Mashiour Rahman
70
Selection Sort
A
1
2
3
4
1
5
7
8
9 6
j
n
i
Strategy
•
Start empty handed.
•
Enlarge the sorted part by
switching the first element of
the unsorted part with the
smallest element of the
unsorted part.
•
Continue until the unsorted part
consists of one element only.
CSC2105
Mashiour Rahman
A 44
06
06
06
06
06
06
06
55
55
12
12
12
12
12
12
12
12
55
18
18
18
18
18
42
42
42
42
42
42
42
42
94
94
94
94
94
44
44
44
18
18
18
55
55
55
55
55
06
44
44
44
44
94
94
67
67
67
67
67
67
67
67
94
71
Selection Sort/2
INPUT: A[1..n] – an array of integers
OUTPUT: a permutation of A such that A[1] A[2] …A[n]
for j := 1 to n-1 do
key := A[j]; ptr := j
for i := j+1 to n do
if A[i] < key then ptr := i; key := A[i];
A[ptr] := A[j]; A[j] := key

The number of comparisons is indepen-dent
of the original ordering (this is less natural
behavior than insertion sort):
j = (n*n-n)/2
 C =
n 1
j =1
CSC2105
Mashiour Rahman
72
Selection Sort/3

The number of movements is:

 Mmax =

Mmin =
n 1
j =1
n 1
j =1
CSC2105
3
= 3*(n-1)
j  3=
(n*n–n)/4 + 3*(n-1)
Mashiour Rahman
73
Bubble Sort
A
1
2
3
1
Strategy
•
Start from the back and
compare pairs of adjacent
elements.
•
Switch the elements if the
larger comes before the
smaller.
•
In each step the smallest
element of the unsorted part
is moved to the beginning of
the unsorted part and the
sorted part grows by one.
CSC2105
4
5
7
9
8 6
j
A 44
06
06
06
06
06
06
06
Mashiour Rahman
i
55
44
12
12
12
12
12
12
n
12
55
44
18
18
18
18
18
42
12
55
44
42
42
42
42
94
42
18
55
44
44
44
44
18
94
42
42
55
55
55
55
06
18
94
67
67
67
67
67
67
67
67
94
94
94
94
94
74
Bubble Sort/2
INPUT: A[1..n] – an array of integers
OUTPUT: a permutation of A such that A[1] A[2] …A[n]
for j := 2 to n do
for i := n to j do
if A[i-1] < A[i] then
key := A[i-1]; A[i-1] := A[i];
A[i]:=key

The number of comparisons is indepen-dent of
the original ordering:

CSC2105
C = j = 2 j  1= (n*n-n)/2
n
Mashiour Rahman
75
Bubble Sort/3

The number of movements is:

Mmin = 0

 Mavg = 

Mmax =
n
j =2
n
CSC2105
j =2
3( j  1)
= 3*n*(n-1)/2
3( j  1) / 2=
Mashiour Rahman
3*n*(n-1)/4
76
CSC2105
Mashiour Rahman
77
CSC2105
Mashiour Rahman
78
CSC2105
Mashiour Rahman
79
CSC2105
Mashiour Rahman
80
The divide- and- conquer design
paradigm
1. Divide the problem (instance) into
subproblems.
2. Conquer the subproblems by solving them
recursively.
3. Combine subproblem solutions.
CSC2105
Mashiour Rahman
81
The divide- and- conquer design
paradigm
Example: merge sort
1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Linear- time merge.
Recurrence for merge sort
T( n) =
2
T( n/ 2)
# subproblems
+
subproblem size
O( n)
work dividing and combining
CSC2105
Mashiour Rahman
82
The divide- and- conquer design
paradigm
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Recurrence for binary search
T(n) =
1
T(n/2)
+
Θ(1)
# subproblems subproblem
work
size
dividing
and
CSC2105
Mashiour Rahman
83
The divide- and- conquer design
paradigm
CSC2105
Mashiour Rahman
84
The divide- and- conquer design
paradigm
Conclusion
• Divide and conquer is just one of several
powerful techniques for algorithm design.
• Divide- and- conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
• Can lead to more efficient algorithms
CSC2105
Mashiour Rahman
85
CSC2105
Mashiour Rahman
86
CSC2105
Mashiour Rahman
87
CSC2105
Mashiour Rahman
88
CSC2105
Mashiour Rahman
89
CSC2105
Mashiour Rahman
90
CSC2105
Mashiour Rahman
91
CSC2105
Mashiour Rahman
92
CSC2105
Mashiour Rahman
93
CSC2105
Mashiour Rahman
94
CSC2105
Mashiour Rahman
95
CSC2105
Mashiour Rahman
96
CSC2105
Mashiour Rahman
97
CSC2105
Mashiour Rahman
98
CSC2105
Mashiour Rahman
99
CSC2105
Mashiour Rahman
100
CSC2105
Mashiour Rahman
101
CSC2105
Mashiour Rahman
102
CSC2105
Mashiour Rahman
103
CSC2105
Mashiour Rahman
104