Transcript Lecture15MRM - School of Computer Science

More LZW / Midterm Review
15-211
Fundamental Data Structures and
Algorithms
Margaret Reid-Miller
1 March 2005
Midterm
Thursday, 12:00 noon, 3 March 2005
WeH 7500
Worth a total of 125 points
Closed book, but you may have one page
of notes.
If you have a question, raise your
hand and stay in your seat
2
Last Time…
Last Time:Lempel & Ziv
4
Reminder: Compressing
We scan a sequence of symbols
A = a1 a2 a3 …. ak
where each prefix is in the dictionary.
We stop when we fall out of the dictionary:
A b
5
Reminder: Compressing
Then send the code for
A = a1 a2 a3 …. ak
This is the classical algorithm.
6
LZW: Compress bad case
Input:
…s…sssb…
^
s
Dictionary:


 - word
(possibly empty)
Output:
….
7
LZW: Compress bad case (time t)
Input:
…s…sssb…
^
s
Dictionary:


Output:
…. …
8
LZW: Uncompress bad case (time t)
Input:
.…
^
s
Dictionary:


Output:
……
9
LZW: Compress bad case (step t+1)
Input:
Dictionary:
…s…sssb…
^
s


s

Output:
…….
10
LZW: Uncompress bad case (time t+1)
Input:
Dictionary:
.…
^
s


Output:
……s
11
LZW: Compress bad case (time t+2)
Input:
Dictionary:
…s…sssb…
^
s


s

b
Output:
…….
 +1
12
LZW: Uncompress bad case (time t+2)
Input:
Dictionary:
…. 
^
What is ??
s


Output:
……s
13
LZW: Uncompress bad case (time t+2)
Input:
Dictionary:
.… 
^
What is ??
It codes for ss!
s


s

Output:
……sss
14
Example
0 0 1 5 3 6 7 9 5  aabbbaabbaaabaababb
s sss
Input
Output
0
add to D
a
0
+
a
3:aa
1
+
b
4:ab
5
-
bb
5:bb
3
+
aa
6:bba
6
+
bba
7:aab
7
+
aab
8:bbaa
9
-
aaba
9:aaba
5
+
bb
s =a
 = ab
10:aabab
15
LZW Correctness
So we know that when this case occurs,
decompression works.
Is this the only bad case? How do we know that
decompression always works? (Note that
compression is not an issue here).
Formally have two maps
comp : texts  int seq.
decomp : int seq.  texts
We need for all texts T:
decomp(comp(T)) = T
16
Getting Personal
Think about
Ann: compresses T, sends int sequence
Bob: decompresses int sequence,
tries to reconstruct T
Question: Can Bob always succeed?
Assuming of course the int sequence is valid
(the map decomp() is not total).
17
How?
How do we prove that Bob can always succeed?
Think of Ann and Bob working in parallel.
Time 0:
both initialize their dictionaries.
Time t:
Ann determines next code number c,
sends it to Bob.
Bob must be able to convert c back into the
corresponding word.
18
Induction
We can use induction on t.
The problem is:
What property should we establish by induction?
It has to be a claim about Bob’s dictionary.
How do the two dictionaries compare over time?
19
The Claim
At time t = 0 both Ann and Bob have the same
dictionary.
But at any time t > 0 we have
Claim: Bob’s dictionary misses exactly the last
entry in Ann’s dictionary after processing the last
code Ann sends.
(Ann can add Wx to the dictionary, but Bob won’t
know x until the next message he receives.)
20
The Easy Case
Suppose at time t Ann enters A b with code
number C and sends c = code(A).
Easy case: c < C-1
By Inductive Hypothesis Bob has codes upto and
including C-2 in his dictionary. That is, c is already in
Bob’s dictionary. So Bob can decode and now knows
A.
But then Bob can update his dictionary: all he needs
is the first letter of A.
21
The Easy Case
Suppose at time t Ann enters A b with code
number C and sends c = code(A).
Easy case: c < C-1
Sent:
c
…
Entered:
A
b…
C-1
C
22
The Hard Case
Now suppose c = C-1.
Recall, at time t Ann had entered A b with code
number C and sent c = code(A).
Sent:
c
…
Entered:
A
b…
C-1
C
23
The Hard Case
Now suppose c = C-1.
Recall, at time t Ann had entered A b with code
number C and sent c = code(A).
Sent:
c
…
Entered:
A’
s’ …
b…
c
C
A = A’ s’
a1 = s’
24
The Hard Case
Now suppose c = C-1.
Recall, at time t Ann had entered A b with code
number C and sent c = code(A).
Sent:
c
… s’ W s’ …
Entered:
b…
c
C
A’ = s’ W
25
The Hard Case
Now suppose c = C-1.
Recall, at time t Ann had entered A b with code
number C and sent c = code(A).
Sent:
c
… s’ W s’ W s’ b …
Entered:
c
C
26
The Hard Case
Now suppose c = C-1.
Recall, at time t Ann had entered A b with code
number C and sent c = code(A).
So we have
Time t-1:
entered c = code(A),
sent code(A’), where A = A’ s’
Time t:
entered C = code(A b),
sent c = code(A), where a1 = s’
But then
A’ = s’ W.
27
The Hard Case
In other words, the text must looked like so
…. s’ W s’ W s’ b ….
But Bob already knows A’ and thus can reconstruct
A.
QED
28
Midterm Review
Basic Data Structures
 List
Persistance
 Tree
Height of tree, Depth of node, Level
Perfect, Complete, Full
Min & Max number of nodes
30
Recurrence Relations





E.g., T(n) = T(n-1) + n/2
Solve by repeated substitution
Solve resulting series
Prove by guessing and substitution
Master Theorem
T(N) = aT(N/b) + f(N)
31
Solving recurrence equations
Repeated substitution:
t(n) = n + t(n-1)
= n + (n-1) + t(n-2)
= n + (n-1) + (n-2) + t(n-3)
and so on…
= n + (n-1) + (n-2) + (n-3) + … + 1
32
Incrementing series
 This is an arithmetic series that comes up over
and over again, because characterizes many
nested loops:
for (i=1; i<n; i++) {
for (j=1; j<i; j++) {
f();
}
}
33
“Big-Oh” notation
T(N) = O(f(N))
“T(N) is order f(N)”
cf(N)
running time
T(N)
n0
N
34
Upper And Lower Bounds
f(n) = O( g(n) )
Big-Oh
f(n) ≤ c g(n) for some constant c and n > n0
f(n) = ( g(n) )
Big-Omega
f(n) ≥ c g(n) for some constant c and n > n0
f(n) = ( g(n) )
Theta
f(n) = O( g(n) ) and f(n) = ( g(n) )
35
Upper And Lower Bounds
f(n) = O( g(n) )
Big-Oh
Can only be used for upper bounds.
f(n) = ( g(n) )
Big-Omega
Can only be used for lower bounds
f(n) = ( g(n) )
Theta
Pins down the running time exactly (up to a
multiplicative constant).
36
Big-O characteristic
 Low-order terms “don’t matter”:
 Suppose T(N) = 20n3 + 10nlog n + 5
 Then T(N) = O(n3)
 Question:
 What constants c and n0 can be used to show
that the above is true?
 Answer: c=35, n0=1
37
Big-O characteristic
 The bigger task always dominates
eventually.
 If T1(N) = O(f(N)) and T2(N) = O(g(N)).
 Then T1(N) + T2(N) = max( O(f(N)), O(g(N) ).
 Also:
T1(N)  T2(N) = O( f(N)  g(N) ).
38
Dictionary

Operations:
 Insert
 Delete
 Find
 Implementations:





Binary Search Tree
AVL Tree
Splay
Trie
Hash
39
Binary search trees
 Simple binary search trees can have bad behavior
for some insertion sequences.
 Average case O(log N), worst case O(N).
 AVL trees maintain a balance invariant to prevent
this bad behavior.
 Accomplished via rotations during insert.
 Splay trees achieve amortized running time of
O(log N).
 Accomplished via rotations during find.
40
AVL trees
 Definition
 Min number of nodes of height H
FH+3 -1, where Fn is nth Fibonacci
number
 Insert - single & double rotations.
How many?
 Delete - lazy. How bad?
41
Single rotation
 For the case of insertion into left
subtree of left child:
Depth
increased by 1
Depth reduced
by 1
Z
Y
X
X
Y
Deepest node of X has depth 2
greater than deepest node of Z.
Z
42
Double rotation
 For the case of insertion into the
right subtree of the left child.
Z
X
X
Y1
Y1
Y2
Z
Y2
43
Splay trees
 Splay trees provide a guarantee that
any sequence of M operations
(starting from an empty tree) will
require O(Mlog N) time.
 Hence, each operation has amortized
cost of O(log N).
 It is possible that a single operation
requires O(N) time.
 But there are no bad sequences of
operations on a splay tree.
44
Splaying, case 3
 Case 3: Zig-zag (left).
Perform an AVL double rotation.
a
b
Z
b
a
X
X
Y1
Y1
Y2
Z
Y2
45
Splaying, case 4
 Case 4: Zig-zig (left).
Special rotation.
a
b
Z
Y
W
X
b
W
a
X
Y
Z
46
Tries
 Good for unequal
length keys or
sequences
 Find O(m), m
sequence length
 But: Few to many
children
4 5
I
4
5
3
like
…
9
…
6
6
8
8
3
love
5
you
9
lovely
47
Hash Tables
 Hash function h: h(key) = index
 Desirable properties:
Approximate random distribution
Easy to calculate
E.g., Division: h(k) = k mod m
 Perfect hashing
When know all input keys in advance
48
Collisions
 Separate chaining
Linked list: ordered vs unordered
 Open addressing
Linear probing - clustering very bad
with high load factor
*Quadratic probing - secondary
clustering, table size must be prime
Double hashing - table size must be
prime, too complex
49
Hash Tables
 Delete?
 Rehash when load factor high double (amortize cost constant)
 Find & insert are near constant
time!
 But: no min, max, next,… operation
 Trade space for time--load factors
<75%
50
Priority Queues
Priority Queues
 Operations:
 Insert
 FindMin
 DeleteMin
 Implementations:
Linked list
Search tree
Heap
52
Possible priority queue implementations
 Linked list
 deleteMin
 insert
O(1)
O(N)
or
O(N)
O(1)
 Search trees
 All operations
O(log N)
 Heaps
 deleteMin
O(log N)
 insert
O(log N)
 buildheap
N inserts
O(N)
53
Heaps

Properties:
1. Complete binary tree in an array
2. Heap order property




Insert: push up
DeleteMin: push down
Heapify: starting at bottom, push
down
Heapsort: BuildHeap + DeleteMin
54
Insert - Push up
 Insert leaf to establish complete tree property.
 Bubble inserted leaf up the tree until the heap order
property is satisfied.
13
21
24
13
16
31 19 68
65 26 32 14
14
24
16
21 19 68
65 26 32 31
55
DeleteMin - Push down
 Move last leaf to root to restore complete tree property.
 Bubble the transplanted leaf value down the tree until the
heap order property is satisfied.
1
2
-14
16
24
21
65 26
32 31
19 68
31
14
24
65 26
16
21
19 68
32
56
Heapify - Push down
 Start at bottom subtrees.
 Bubble subtree root down until the heap order
property is satisfied.
24
32
26
65
23
68
31
21
16
19
14
57
Sorting
Simple sorting algorithms
Several simple, quadratic algorithms (worst case
and average).
- Bubble Sort
- Insertion Sort
- Selection Sort
Only Insertion Sort of practical interest: running
time linear in number of inversion of input
sequence.
Constants small.
Stable?
59
Sorting Review
Asymptotically optimal O(n log n) algorithms
(worst case and average).
- Merge Sort
- Heap Sort
Merge Sort purely sequential and stable.
But requires extra memory: 2n + O(log n).
60
Quick Sort
Overall fastest.
In place.
BUT:
Worst case quadratic. Average case O(n log n).
Not stable.
Implementation details tricky.
61
Radix Sort
Used by old computer-card-sorting machines.
Linear time:
• b passes on b-bit elements
• b/m passes m bits per pass
Each pass must be stable
BUT:
Uses 2n+2m space.
May only beat Quick Sort for very large arrays.
62
Data Compression
Data Compression
 Huffman
Optimal prefix-free codes
Priority queue on “tree” frequency
 LZW
Dictionary of codes for previously seen
patterns
When find pattern increase length by
one  trie
64
Huffman
 Full: every node
 Is a leaf, or
 Has exactly 2 children.
 Build tree bottom up:
 Use priority queue of trees
weight - sum of
frequencies.
 New tree of two lowest
weight trees.
0
0
0
c
1
a
1
d
1
b
a=1, b=001, c=000, d=01
65
Summary of LZW
LZW is an adaptive, dictionary based
compression method.
Incrementally builds the dictionary (trie) as
it encodes the data.
Building the dictionary while decoding is
slightly more complicated, but requires no
special data structures.
66