Transcript Least Squares and Linear Algebra

Least Squares Approximation: A
Linear Algebra Technique
By Larry Wong and
James Sfregola
Soo……You have a bunch of
Data.
Attendence
Big 4
Derby
League Position
per capita income
adjusted population
39,968
1
0
11
35000
1582564
21,852
1
0
10
36725
137471
24,409
1
0
7
36725
371847
26,770
1
0
19
43600
852013
41,917
1
1
2
43600
1351187
37,162
1
0
6
36725
623160
24,510
1
1
16
43600
724121
43,958
1
1
3
36725
738873
40,699
1
0
14
36725
630043
75,595
1
1
1
36725
1194415

What can we do with this?

Use Least Squares Approximation (LSE) to estimate the
relationship of the dependent variable and the independent
variable(s).
What is Least Squares?
• Approximates solutions for inconsistent over
determined systems
– What does inconsistent mean?
– Over determined, what’s that?
• In other words LSE allows you to describe a
model as being represented by the line of best
fit.
• Where the “best fit” line, curve or polynomial is created from
the approximations.
Least Squares it’s Actually Pretty Useful

Least Squares Approximations are
highly relevant in various fields

Really, how so?

Still don’t believe us?

Fine we’ll give examples…but not now.
So How Does it Estimate a Solution?

Least Squares Approximation estimates
a best fit solution for a system by


Minimizing the magnitude error vector, e.
Estimating the coefficients of the best fit
equation
How Do We Ensure Positive Error?

Least Squares
–
–

Eliminates negative terms
More susceptible to outlying data
Least Absolute Difference
–
Difficult to work with in linear algebra
1
2
3
4
`Petal L.`
5
6
7
0 .5
1 .0
1 .5
` Pe ta l W.`
2 .0
2 .5
The Least Squares Theorem

For an over determined inconsistent
system Ax=b:


This is easily solved using inverse and
transpose multiplication
the resulting vector x can be determined in
the form x=(AT A) -1 AT b

where x will correspond to the coefficients in
your linear or polynomial expression.
How to Approximate the Error Term?
• e = b-Ax , where e is
•
the error term
Where e1= the
distance from our
data point to the
best fit
approximation
given by the
orthogonal
projection
Application to Economics

Econometrics utilizes Ordinary Least
Squares Approximations (OLS)

Similar to LSE except it follows 7 classical
assumptions
Estimated Regression Equation
80000
60000
40000
30000
20000
20000
0
10000
0
-10000
-20000
-30000
50
100
150
Residual
200
250
Actual
300
350
Fitted
Bibliography
• Poole, David Linear Algebra: A Modern
Introduction. Canada: Thomson
Brooks/Cole, 2006.
• Studenmund, A.H. Using Econometrics: A
Practical Guide. New York: Pearson
Education, 2006.