Matrix Algebra
Download
Report
Transcript Matrix Algebra
Matrix Algebra
Matrix algebra is a means of expressing large
numbers of calculations made upon ordered
sets of numbers.
Often referred to as Linear Algebra
Many equations would be completely
intractable if scalar mathematics had to be
used. It is also important to note that the
scalar algebra is under there somewhere.
Definitions - scalar
scalar - a number
denoted with regular type as is scalar
algebra
[1] or [a]
Definitions - vector
vector - a single row or column of numbers
denoted with bold small letters
row vector a =
1 2 3 4 5
column vector x =
x1
x
2
x3
x4
x5
Definitions - Matrix
A matrix is a set of rows and columns of
numbers
1 2 3
4 5 6
Denoted with a bold Capital letter
All matrices (and vectors) have an order
- that is the number of rows x the
number of columns.
Thus A = 1 2 3
4 5 6
2 x 3
Matrix Equality
Two matrices are equal iff (if and only
if) all of their elements are identical
Note: your data set is a matrix.
Matrix Operations
Addition and Subtraction
Multiplication
Transposition
Inversion
Addition and Subtraction
Two matrices may be added iff they are
the same order.
Simply add the corresponding elements
a11 a12 b11
a
b
a
22
21
21
a31 a32 b31
b12 c11 c12
b22 c 21 c 22
b32 c31 c32
Addition and Subtraction
(cont.)
Where
a11 b11 c11
a12 b12 c12
a21 b21 c 21
a22 b22 c 22
a31 b31 c31
a32 b32 c32
Hence
1 2 4 6 5 8
3 4 4 6 7 10
5 6 4 6 9 12
Scalar Multiplication
To multiply a scalar times a matrix,
simply multiply each element of the
matrix by the scalar quantity
a11 a12 2a11
2
a21 a22 2a21
2a12
2a22
Matrix Multiplication (cont.)
To multiply a matrix times a matrix, we
write
A times B as AB
This is pre-multiplying
multiplying A by B.
B
by A, or post-
Matrix Multiplication (cont.)
In order to multiply matrices, they must
be conformable (the number of columns
in A must equal the number of rows in
B.)
an (mxn) x (nxp) = (mxp)
an (mxn) x (pxn) = cannot be done
a (1xn) x (nx1) = a scalar (1x1)
Matrix Multiplication (cont.)
The general rule for
matrix multiplication
is:
N
cij aik bkj wherei 1,2,...,M , and j 1,2,...,P
k 1
Matrix multiplication is not
Commutative
AB does not necessarily equal BA
(BA may even be an impossible
operation)
Yet matrix multiplication is
Associative
A(BC) = (AB)C
Special matrices
There are a number of special matrices
Square
Diagonal
Symmetric
Null
Identity
Square matrix
A square matrix is just what it sounds like, an nxn
matrix
a11
a
21
a31
a41
a12
a13
a22
a23
a32
a33
a42
a43
a14
a24
a34
a44
Square matrices are quite useful for describing the
properties or interrelationships among a set of
things – like the correlation matric for your dataset.
Diagonal Matrices
A diagonal matrix is a square matrix that
has values on the diagonal with all offdiagonal entities being zero.
a11 0
0 a
22
0
0
0
0
0
0
a33
0
0
0
0
a44
Symmetric Matrix
All of the elements in the upper right portion
of the matrix are identical to those in the
lower left.
For example, the correlation matrix
Identity Matrix
The identity matrix I is a diagonal
matrix where the diagonal elements all
equal one. It is used in a fashion
analogous to multiplying through by "1"
in scalar math.
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
Null Matrix
A square matrix where all elements equal
zero.
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Not usually ‘used’ so much as sometimes the
result of a calculation.
Analogous to “a+b=0”
The Transpose of a Matrix A'
Taking the transpose is an operation
that creates a new matrix based on an
existing one.
The rows of A = the columns of A'
Hold upper left and lower right corners
and rotate 180 degrees.
Example of a transpose
1 4
1 2 3
A 2 5, A'
4
5
6
3 6
The Transpose of a Matrix A'
If A = A', then A is symmetric (i.e. correlation
matrix)
If AA’ = A then A' is idempotent
The transpose of a sum = sum of transposes
(and A' = A)
Idempotent means applying the same operation
gives invariant results – e.g. 1 x 1 = 1 (see further…)
(A + B + C)’ = A’ + B’ + C’
The transpose of a product = the product of the
transposes in reverse order
(ABC)’ = C’B’A’
An example:
Suppose that you wish to obtain the
sum of squared errors from the vector
e. Simply pre-multiply e by its
transpose e'.
which, in matrices looks like
e' e e e2 ..en
2
1
2
2
An example - cont
Since the matrix product is a scalar
found by summing the elements of the
vector squared.
The Determinant of a Matrix
The determinant of a matrix A is
denoted by |A|.
Determinants exist only for square
matrices.
They are a matrix characteristic, and
they are also difficult to compute
The Determinant for a 2x2 matrix
If A =
Then
a11
a
21
a12
a22
A a11a22 a12a21
That one is easy
The Determinant for a 3x3 matrix
If A =
a11
a
21
a31
a12
a22
a32
a13
a23
a33
Then
A = a11a22 a33 - a11a23a32 + a12 a23a31 - a12 a21a33 + a13a21a32 - a13a22 a31
Determinants
For 4 x 4 and up don't try. For those
interested, expansion by minors and
cofactors is the preferred method.
(However the spaghetti method works
well! Simply duplicate all but the last
column of the matrix next to the
original and sum the products of the
diagonals along the following pattern.)
Spaghetti Method of |A|
A = a11a22 a33 - a11a23a32 + a12 a23a31 - a12 a21a33 + a13a21a32 - a13a22 a31
a11 a12
a
a
21
22
a31 a32
a13 a11 a12
a23 a21 a22
a33 a31 a32
Spaghetti Method of |A|
A = a11a22 a33 - a11a23a32 + a12 a23a31 - a12 a21a33 + a13a21a32 - a13a22 a31
a11 a12
a
a
21
22
a31 a32
a13 a11 a12
a23 a21 a22
a33 a31 a32
Properties of Determinates
Determinants have several mathematical
properties which are useful in matrix
manipulations.
1 |A|=|A'|.
2. If a row of A = 0, then |A|= 0.
3. If every value in a row is multiplied by k,
then |A| = k|A|.
4. If two rows (or columns) are interchanged
the sign, but not value, of |A| changes.
5. If two rows are identical, |A| = 0.
Properties of Determinates
6. |A| remains unchanged if each
element of a row or each element
multiplied by a constant, is added to any
other row.
7. Det of product = product of Det's |AB|
= |A| |B|
8. Det of a diagonal matrix = product of
the diagonal elements
The Inverse of a Matrix (A-1)
For an nxn matrix A, there may be a B such
that AB = I = BA.
(The inverse is analogous to a reciprocal)
A matrix which has an inverse is nonsingular.
A matrix which does not have an inverse is
singular.
An inverse exists only if A 0
Inverse by Row or column
operations
Set up a tableau matrix
A tableau for inversions consists of the
matrix to be inverted post multiplied by
a conformable identity matrix.
Matrix Inversion by Tableau
Method
Rules:
You may interchange rows.
You may multiply a row by a scalar.
You may replace a row with the sum of that row
and another row multiplied by a scalar.
Every operation performed on A must be
performed on I
When you are done; A = I & I = A-1
The Tableau Method of Matrix
Inversion: An Example
Step 1: Set up Tableau
3 1 0 0
1 4
2 5
0 1 0
4
1 3 2 0 0 1
Step 2: Add –2(Row 1)
to Row 2
3 1 0 0
1 4
0 3 2 2 1 0
1 3 2 0 0 1
Matrix Inversion – cont.
3 1 0 0
1 4
0 3 2 2 1 0
1 3 2 0 0 1
Step 3: Add –1(Row
1) to Row 3
3 1 0 0
1 4
0 3 2 2 1 0
0 7 5 1 0 1
Step 4: Multiply Row
2 by –1/3
3 1
0
0
1 4
0 1 2 / 3 2 / 3 1 / 3 0
0 7 5 1
0
1
Matrix Inversion – cont.
3 1
0
0
1 4
0 1 2 / 3 2 / 3 1 / 3 0
0 7 5 1
0
1
Step 5: Add –4 (Row 2) 1
0
to Row 1
1 / 3 5 / 3 4 / 3 0
1 2 / 3 2 / 3 1 / 3 0
0 7 5 1
0
1
Step 6: Add 7(Row 2)
to Row 3
0
1 0 1 / 3 5 / 3 4 / 3 0
0 1 2 / 3 2 / 3
1
/
3
0
0 0 1 / 3 11/ 3 7 / 3 1
Matrix Inversion – cont.
1 0 1 / 3 5 / 3 4 / 3 0
0 1 2 / 3 2 / 3
1
/
3
0
0 0 1 / 3 11/ 3 7 / 3 1
0 2
1 1
2 / 3 2 / 3 1 / 3 0
0 0 1 / 3 11/ 3 7 / 3 1
Step 7: Add Row 3 1 0
0 1
to Row 1
0 2
1 1
1 0
8
Step 9: Add 2(Row 0 1
0
5
2
3) to Row 2
0 0 1 / 3 11/ 3 7 / 3 1
Matrix Inversion – cont.
Step 9: Multiply Row 3
by -3
The original matrix is
now an identity, and
the original identity has
been transformed to the
Inverse
0 2
1 1
1 0
0 1
8
0
5
2
0 0 1 / 3 11/ 3 7 / 3 1
1 1
1 0 0 2
0 1 0 8 5 2
0 0 1 11 7 3
Checking the calculation
Remember AA-1=I
3 2
1 1 1 0 0
1 4
2 5
8 5 2 0 1 0
4
1 3 2 11 7 3 0 0 1
Thus
1 * 2 4 * 8 3 * 11 1
1 * 1 4 * 5 3 * 7 0
etc
Using Stata to do matrix Algebra
Try the following command set
matrix
matrix
matrix
matrix
matrix
matrix
matrix
matrix
matrix
input A= (1, 2 \ 3, 4)
list A
input B= (5, 6 \ 7, 8)
list B
define C = A + B
list C
X = (1, 4, 3 \ 2, 5, 4 \ 1, -3, -2)
Xinv = inv(X)
list Xinv
The Matrix Model
The multiple regression model may be
easily represented in matrix terms.
Y XB e
Where the Y, X, B and e are all
matrices of data, coefficients, or
residuals
The Matrix Model (cont.)
The matrices in
represented by
Y1
Y
2
Y
Yn
X 11 X 12
X X
21
22
X
... ...
X n1 X n 2
Y XB e
... X ik
... X 2 k
... ...
... X nk
B1
B
2
B
Bk
are
e1
e
2
e
en
Note that we postmultiply X by B since this
order makes them conformable.
The Assumptions of the Model
Scalar Version
1. The ei's are normally distributed.
2. E(ei) = 0
3. E(ei2) = 2
4. E(eiej) = 0 (ij)
5. X's are nonstochastic with values fixed in repeated
samples and (Xik-Xbark)2/n is a finite nonzero
number.
6. The number of observations is greater than the
number of coefficients estimated.
7. No exact linear relationship exists between any of
the explanatory variables.
The Assumptions of the
Model: The Matrix Version
These same assumptions expressed in
matrix format are:
1. e N(0,)
2. = 2I
3. The elements of X are fixed in repeated
samples and (1/ n)X'X is nonsingular and
its elements are finite
Derivation of B's in matrix
notation
Given the matrix algebra model we can replicate
the least squares normal equations in matrix
format.
We need to minimize ee’ which is the sum of
squared errors
Setting the derivative equal to 0 we ultimately
get
B (X' X)1 X' Y
Note that X’X is called the sums-of-squares and
cross-products matrix.