Transcript Slide 1

Statistical Data Analysis
Daniel Svozil
[email protected]
http://ich.vscht.cz/~svozil
Laboratory of Informatics and Chemistry
Faculty of Chemical Technology
What is it about?
• Though the name does not suggest, the
lecture consists of two parts
1. numerical linear algebra
•
vector and matrix multiplication, matrix inverse,
vector spaces, least squares, eigenproblem,
matrix decompositions, climax: Singular Value
Decomposition
2. statistics
•
probability (including Bayes), confidence
intervals, linear regression, logistic regression,
ANOVA, non-parametric statistics, climax: PCA
Exercises
• The subject will be more like 3/1.
• Numerical linear algebra
– pen and paper
• Statistics
– Excel
Organizační věci
http://ich.vscht.cz/~svozil/teaching.html
zápočet: písemka uprostřed semestru, cokoliv
lepšího než F znamená zápočet
zkouška: písemná zkouška ve zkouškovém období,
známky zapisuji do SISu, ukázkové příklady na
výše uvedené adrese
- 10 otázek, u každé otázky více odpovědí,
zaškrtnout správné (min 1 je správná)
The Geometry of Linear
Equations
Daniel Svozil
based on excelent video lectures by Gilbert Strang, MIT
http://ocw.mit.edu/OcwWeb/Mathematics/18-06Spring-2005/VideoLectures/index.htm
Vectors
• Vector 𝑏 – column vector
0
𝑏 = 𝒃 = −1
4
• If you want write 𝑏 as a row vector, you
must transpose it (swap column for row).
𝑏 𝑇 = 𝒃𝑇 = [0, −1, 4]
Fundamental problem of linear algebra
• solve a system of linear equations
• start with a (normal and nice) case of equal
number of equations and unknowns (n
equations, n unknowns)
• two pictures will be presented:
– row picture (one equation on a line)
– column picture (that’s the important one, and probably
new to you)
• matrix form (the algebra way of looking at the
problem using matrix A)
2 equations, 2 unknowns
2x – y = 0
-x + 2y = 3
• how to write this equation in matrix/vector
form?
0
2 −1 𝑥
⋅ 𝑦 =
3
−1 2
𝐴⋅𝑥 =𝑏
Row picture
• I take one row at a time
2x –
-x + 2y = 3
y = 0
y
x = 1
y = 2
x
2x – y = 0
-x + 2y = 3
Column picture
2x – y = 0
-x + 2y = 3
0
2
−1
𝑥
+𝑦
=
3
−1
2
this equation is asking us to combine 1st and 2nd vectors
in right amounts to get RHS vector
it’s asking us to find a right LINEAR COMBINATION
0
2
−1
𝑥
+𝑦
=
3
−1
2
Let’s take some linear combination.
And why not the correct one to produce RHS
so let x = 1 and y = 2?
y
b
col 2
2 x col 2
col 1
x
• The idea of linear combination is crucial.
• Let’s think about this question: What are
all the combinations?
• i. e. what will be the results if I take any x
and any y?
• Any idea?
All possible linear combinations will
give all possible RHS, so they will fill
the whole plane.
3 equations, 3 unknowns
2x – y
= 0
-x + 2y – z = -1
-3y + 4z = 4
Matrix form? What is 𝐴, what is 𝑏?
2
𝐴 = −1
0
−1 0
2 −1
−3 4
0
𝑏 = −1
4
Row picture
2x – y
= 0
-x + 2y – z = -1
-3y + 4z = 4
• Help me, please!
• Take the second equation.
• However, as you can see, row picture is
getting a little hard to see.
Column picture
2x – y
= 0
-x + 2y – z = -1
-3y + 4z = 4
2
−1
0
0
𝑥 −1 + 𝑦 2 + 𝑧 −1 = −1
0
−3
4
4
• Let’s draw vectors in 3D
• Combine three column vectors with the
right combination to get RHS vector.
• This can be also rather messy. However,
the solution is apparent. Can you see it?
• The system was cleverly chosen by your
lecturer, RHS is equal to Col 3, so the
solution is 0,0,1 𝑇 .
• Of course, I won't always be able to see it
from the column picture, either.
• There is technique to solve the equations,
the Gauss elimination.
• Now change the RHS
2
−1
1
0
𝑥 −1 + 𝑦 2 + 𝑧 −1 = 1
0
−3
−3
4
• This is also special RHS, do you see why?
• It’s a sum of first two columns. The
solution is 1,1,0 𝑇 .
• Now think about all RHS?
2x – y
= 0
-x + 2y – z = -1
-3y + 4z = 4
• Can I solve 𝐴 ⋅ 𝑥 = 𝑏 for every 𝑏?
• In linear combinations words: Do the linear
combinations of the columns fill 3D space?
• For this matrix A the answer is YES.
• There could be other matrices where the
answer can become No?
• When could it go wrong?
– If the three columns all lie in the same plane, then
their combinations will also lie in that plane.
– It may happen if e.g. Col3 = Col1 +Col2
– Such a matrix is called singular, and it will not be
invertible (later lecture).
9-D space
• Can we get every right-hand side 𝑏?
• For certain matrices, YES. But their columns
must be independent (nonsingular, invertible
matrix).
• If they are dependent (e.g. Col9 = Col8), then
they contribute nothing new and there would be
right-hand sides 𝑏 that we couldn't get (8-D
“plane” inside 9-D space).
• And it's those eight dimensional “planes” inside
nine-dimensional space that we have to work
with eventually.
Matrix by a vector multiplication
• For now stay with a nice case.
• One step I was saying in words that I will
write now in letters – matrix by vector
multiplication 𝐴 ⋅ 𝑥.
2
𝐴⋅𝑥 =
1
5 1
⋅
3 2
• How would you solve it using column
picture?
10
2
2
12
2 5 1
5
⋅
=1⋅
+2⋅
=
+
=
1
1
6
7
1 3 2
3
• Here is the point: 𝐴 ⋅ 𝑥 is the combination
of columns of 𝐴.
• Think about 𝐴 ⋅ 𝑥 that way, it’s much
better!
Matrix Multiplication
Daniel Svozil
based on excelent video lectures by Gilbert Strang, MIT
http://ocw.mit.edu/OcwWeb/Mathematics/18-06Spring-2005/VideoLectures/index.htm
Rules for matrix multiplication
• Several ways how to do it, all give the
same answer, and they’re all important.
1st way
column 3
cij23
row 2
⋅
=
𝐴⋅𝐵 =𝐶
We write row number first, then column number.
𝑐34 = row3 of A ⋅ row4 of B
𝑐34 = 𝑎31 𝑏14 + 𝑎32 𝑏24 + ⋯
What are the consequences for the shape of matrices?
𝐴: 𝑚 × 𝑛 𝐵: 𝑛 × 𝑝 𝐶: 𝑚 × 𝑝
2nd way – column way
col1 col2












A
mxn
col1 col2
 
 

 
 
 
B
nxp






C
mxp
Matrix multiplication works so that I can just think of having several columns,
multiplying by A and getting the columns of the answer.
So, what can we say about columns of C? Just a reminder, each column
of C is matrix A times column of B.
Columns of C are linear combinations of columns of what?
of A
3rd way – row way












A
mxn
 
 

 
 
 
B
nxp






C
mxp
Rows of C are linear combinations of rows of what?
of B
• Can you somehow combine 2nd and 3rd
ways to make some conclusion?
CT = (AB)T = BTAT
4th way
So what is left? Column times row
What is the result of column times row? a matrix
 2
31 6
 
4
 2 12
 3 18


 4 24
a big, full-size matrix
What’s the answer? Just follow rules of the 1st way.
What’ the first row of the resulting matrix?
2 12
And next?
This is very special matrix. What are its columns?
Multiples of column vector.
And what are the rows?
Multiples of row vector.
What does it mean geometrically, that all rows
are multiples of the [1 6]?
They all lie on the same line.
And the same is true for cols.
4th way
• Let me write down a 4th way:
AB is a sum of (cols of A x rows of B)
outer product
2 7 
2
7 
3 8  1 6  31 6  8 0 0

 0 0   
 


4 9 
4
9 
outer product expansion
What’s the answer for that matrix multiplication?
Matrix Inverse
Daniel Svozil
based on excelent video lectures by Gilbert Strang, MIT
http://ocw.mit.edu/OcwWeb/Mathematics/18-06Spring-2005/VideoLectures/index.htm
Analogy
• Consider the equation ax = b, where a, x,
b are real numbers. 2x = 4
• We want to find x.
• What is the solution?
• x = b/a = ba-1. What are the conditions?
• a≠0
• Imitate this procedure for Ax = b.
• We need to find a matrix A-1 such that
AA-1 = I
• I is the identity matrix, a matrix equivalent
of number 1.
1 0 0 
I   0  0


0 0 1
• However, we will also require A-1A = I.
• This requirement forces A and A-1 to have
which shape?
• Square.
Square matrices
• Square matrix A may or may not have
inverse, not all matrices have inverses!
• So the question about matrix A is if it is
invertible or not?
• If it’s invertible, then there exists another
matrix A-1 (A inverse).
• Synonyma for invertible is nonsingular.
• If A-1 exists, then A-1A = I = AA-1
• How to identify, if A has inverse?
• Let’s talk about singular case (no inverse).
Singular matrix
• Example
1 3 
2 6


Why is this noninvertible?
det(A) is zero. What is det? We don’t know yet!
OK, another explanation.
So, we’re looking for a matrix K, for which
AK = I. 1st column of I is a linear combination
of columns of A. Can you write down the linear
combination?
 1 3
1 0 
 2 6  K  0 1 




1
3 1
x   y     
 2
6 0
What is the solution of this system
of equations?
Write down the eqs. and try to solve.
This is logical result if you think geometrically. Try to do it !
Geometrically, columns of A lie on the same line, so their linear combination
MUST lie also on that line. But [1 2]T does not line on the [1 0]T line!
Another reason
• Square matrix A does not have an
inverse, if you can find an vector x such
that Ax = 0.
• OK, tell me the x for our case.
1 3   0
2 6    0

   
So, why A is not invertible if Ax = 0?
Left multiply by A-1.
A-1Ax = Ix = 0 means, that x is also 0, but we found non-zero x.
Conclusion
• Singular matrices take some vector x and
turn them into zero.
How to find an inverse?
1 3 a c  1 0
2 7 b d   0 1


 

First column of I is a linear combination of columns of A with coefficients
a and b.
Second column of I is a linear combination of columns of A with coefficients
c and d.
So we need to solve two Ax = b systems.
Gauss-Jordan elimination
solve several equations at once
(this is a difference compared to Gauss elimination)
Some rules
• Without proof.
• 𝐴⋅𝐵
• 𝐴−1
−1
𝑇
= 𝐵−1 ⋅ 𝐴−1
= 𝐴𝑇
−1
– In another words, you can do transposing and
inverting in either order.