Transcript estimation
Introduction to Signal Estimation Outline 94/10/14 2 Scenario For physical considerations, we know that the voltage is between –V and V volts. The measurement is corrupted by noise which may be modeled as an independent additive zero-mean Gaussian random variable n. The observed variable is r . Thus, r an The probability density governing the observation process is given as 2 r a 2 2 pa (r ) 94/10/14 1 e 2 n n 3 Estimation model H0 Probabilistic Mapping to observation space Observation space Estimation Rule Decision Decision rule H1 Parameter Space Parameter Space: the output of the source is a Probabilistic Mapping from parameter space to observation space :the probability law that governs the effect of parameters on the observation space. Observation space: a finite-dimensional space Estimation rule: A mapping of observation space into an estimate 94/10/14 4 Parameter Estimation Problem In the composite hypothesis-testing problem，a family of distributions on the observation space, indexed by a parameter or set of parameters, a binary decision is wanted to make about the parameter。 In the estimation problem, values of parameters want to be determined as accurately as possible from the observation enbodied。 Estimation design philosophies are different due to o the amount of prior information known about the parameter o the performance criteria applied。 . 94/10/14 5 Basic approaches to the parameter estimation Bayesian estimation: o assuming that parameters under estimation are to be a random quantity related statistically to the observation。 Nonrandom parameter estimation: o the parameters under estimation are assumed to be unknown but without being endowed with any probabilistic structure。 94/10/14 6 Bayesian Parameter Estimation Statement of problem: o : the parameter space and the parameter o Y :random variable observation space o p ; :denoting a distribution on the observation space , and mapping from to ˆ : Finding a function ˆ : s.t ˆ is the best guess of the true value of based on the observation Y=y。 94/10/14 Obser. R.V. y estimate ˆ y ˆ y 7 Bayesian Parameter Estimation Performance evaluation: o the solution of this problem depends on the criterion of goodness by which we measure estimation performance --the Cost function assignment。 C : R C ˆ Y is the cost of estimating a true value as ˆ for in and ˆ in o The conditional risk/cost averaged over Y for each R E C Y C y p y dy o The Bayes risk: if we adopt the interpretation that the actual parameter value is the realization of a random variable , the Bayes risk/average risk is defined as r E R R w d 94/10/14 8 Bayesian Parameter Estimation o where w is the prior for random variable 。the appropriate design goal is to find an estimator minimizing r ˆ ，and the estimator is known as a Bayes estimate of o Actually，the conditional risk can be rewritten as R E C Y E C Y o the Bayes risk can be formulated as r E R E E C Y E C Y o the Bayes risk can also be written as r E R E C Y E E C Y Y E C Y Y y p y dy 94/10/14 9 Bayesian Parameter Estimation o the Bayes risk can also be written as r E R E C Y E E C Y Y E C Y Y y p y dy • The equation suggests that for each y , the Bayes estimate of can be found by minimizing the posterior cost given Y=y E C ˆ y Y y o Assuming that has a conditional density w y given Y=y for each y , then the posterior cost ,given Y=y, is given by E C y Y y C y w y d o Deriving w y • If we know p y ,priori and w w / y p y w p y w p y p y w d o the performance of Bayesian estimation depends on a cost function 94/10/14 C y w y 10 (MMSE) Minimum-Mean-Squared-Error Estimation 2 R and E C , for , R 2 o Measuring the performance of an estimator in terms of the squared of the estimation error ˆ 2 ˆ o The corresponding Bayes risk is E 2 • defined as the mean-squared-error(MMSE) estimator。 o The Bayes estimation is the Minimum-mean-squared-error(MMSE) estimator。 o the posterior cost given Y=y under this condition is given by 2 2 E y Y y E y 2 y 2 Y y 2 E y Y y 2E y Y y E 2 Y y 2 y 2 y E Y y E 2 Y y 94/10/14 11 (MMSE) Minimum-Mean-Squared-Error Estimation o the cost function is a quadratic form of y ,and is a convex function。 o Therefore, it achieves its unique minimum at the point where its derivative y with respective to is zero。 10 H 10 D D10 2 E ˆ y Y y 2 ˆ y 2E Y y 0 ˆ y ˆMMSE y E Y y w y d o the conditional mean of given Y=y 。 The estimator is also sometimes termed the conditional mean estimator-CME。 94/10/14 12 (MMSE) Minimum-Mean-Squared-Error Estimation Another derivation: 2 2 E y Y y y w y d 2 E ˆ y Y y 2 ˆ y w y d 0 ˆ y ˆ y w y d w y d ˆ y w y d E Y y 94/10/14 13 (MMAE) : Minimum-Mean-Absolute-Error Estimation Measuring the performance of an estimator in terms of the absolute value of the estimation error, ˆ R and E C , for ˆ, R 2 The corresponding Bayes risk is E ˆ ，which is defined as the mean-absolute-error (MMAE)。 The Bayes estimation is the Minimum-mean-absoluteerror(MMAE) estimator。 94/10/14 14 (MMAE) : Minimum-Mean-Absolute-Error Estimation From the definition E y Y y P ˆ y x Y y dx 0 P ˆ y x Y y dx P ˆ y x Y y dx 0 0 By change variable t x ˆ y and t x ˆ y for the first and the second integration, respectively, E ˆ y Y y ˆ P t Y y dt 94/10/14 y ˆ y P t Y y dt 15 (MMAE) : Minimum-Mean-Absolute-Error Estimation Actually,the expression represents that it is a differentiable function of ˆ y ， thus, it can be shown that E ˆ y Y y ˆ y P y Y y P ˆ y Y y o The derivative is a non-decreasing function of ˆ y o If ˆ y approaches - ,its value approaches -1 o If ˆ y approaches ,its value approaches 1 o E ˆ y Y y achieves its minimum over ˆ y at the point where its derivative changes sign P t Y y P t Y y t ˆABS y and P t Y y P t Y y t ˆABS y 94/10/14 16 (MMAE) : Minimum-Mean-Absolute-Error Estimation the cost function can be also expressed as ˆ y E y Y y ˆ w y d ˆ w y d y E ˆ y Y y ˆ y ˆ w y d w y d 0 y ˆ y ˆ y w y d ˆ y 1 w y d 2 o the minimum-mean-absolute-error estimator is to estimate the median of the conditional density function of ,Y=y。 o the MMAE is also termed as condition median estimator o For a given density function of , if its mean and median are the same, then,MMSE and MMAE coincide each other,i.e. they have the same performance based on different criterion adopted。 94/10/14 17 MAP Maximum A posterior Probability Estimation Assuming the uniform cost function 0 if ˆ C ˆ, 0 1 if ˆ 1 0 ˆ y ˆ y ˆ y The average posterior cost, given Y=y, to estimate ˆ y is given by E C y , Y y P y Y y 1 P ˆ y Y y 94/10/14 18 MAP Maximum A posterior Probability Estimation Consideration I: o Assuming is a discrete random variable taking values in a finite set 0 M 1and with i j for i j the average posterior cost is given as E C y , Y y 1 P ˆ Y y 1 w ˆ y which suggests that to minimize the average posterior cost, The Bayes estimate in this case is given for each y by any value of which can maximizes the posterior probability w y over ， i.e. the Bayes estimate is the value of that has the maximum a posterior probability of occurring given Y=y 94/10/14 19 MAP Maximum A posterior Probability Estimation Consideration II: o is a continuous random variable with conditional density function given Y=y。Thus, the posterior cost become w y ˆ y E C ˆ y , Y y 1 ˆ w y d y ˆy y w • which suggests that the average posterior cost is minimized over ˆ y by maximizing the area under over the interval y y 。 • Actually, the area can be approximately maximized by choosing ˆ y to be a point of maximum of w y 。 • the value of can be chosen as small as possible and smooth w y , then we can obtain ˆ y ˆ y w y d 2w y ˆ y • where ˆ y is chosen to the value of maximizing w y over 。 94/10/14 20 MAP Maximum A posterior Probability Estimation The MAP estimator can be formulated as 1 E C ˆ y , Y y ˆMAP arg max w / y w y ˆ y ˆ y ˆ y o The uniform cost criterion leads to the procedure for estimating as that value maximizing the a posteriori density w y , which is known as the maximum a posteriori probability (MAP) estimate and is denoted by ˆMAP。 o It approximates the Bayes estimate for uniform cost with small 94/10/14 21 MAP Maximum A posterior Probability Estimation The MAP estimates are often easier to compute than MMSE、 MMAE，or other estimates。 A density achieves its maximum value is termed a mode of the corresponding probability。Therefore, the MMSE、 MMAE、and MAP estimates are the mean、median、and mode of the corresponding distribution, respectively。 94/10/14 22 Remarks -Modeling the estimation problem Given conditions: o Conditional probability density function of Y given p , o Prior distribution for w o Conditional probability density of given Y=y w / y p y w p y w d p y w py o The MMSE estimator ˆMMSE ˆMMSE y E Y y w y d o The MMAE estimator MMAE / ˆABS ˆ y 94/10/14 w y d ˆ y w y d 1 2 23 Remarks -Modeling the estimation problem The MAP estimator ˆMAP ˆMAP arg max w / y o For MAP estimator, it is not necessary to calculate p y because the unconditional probability density of y does not affect the maximization over 。 o ˆMAPcan be found by maximizing p y w over . o For the logarithm is an increasing function, therefore, ˆMAP also maximizes log p y logw over 。 o If is a continuous random variable given Y=y， then for sufficient smooth p y and w ,a necessary condition for MAP is given by MAP equation log p y log w MAP y MAP y 94/10/14 24 Example Probability density of observation given e y if y 0 p y 0 if y 0 The prior probability density of e if 0 w 0 0 if 0 the posterior probability density of given Y=y e y 2 y y e w / y e y d 0 0 if 0 where p y 0 e 94/10/14 y d y if and y 0 2 25 Example The MMSE 0 0 ˆMMSE y w / y d y e y 2 0 2e y 2 y d d 2 y o the Bayes risk is the average of the posterior cost MMSE r ˆMMSE E E E y Y E E ˆMMSE Y Y 2 2 E Var Y • the minimum MSE is the average of the conditional variance of 94/10/14 26 Example • the conditional variance given Y=y is shown as Var y E 2 Y y E 2 Y y Var y 2w y d ˆMMSE y 0 y 2 0 2 y • 2 3w y d 4 y 2 2 MMSE MMSE E Var Y Var y p y dy 0 2 2 2 y y dy 0 2 3 2 94/10/14 27 Example MMAE o From the definition the MMAE estimate is the median of w y o Because is a continuous random variable given Y=y, the MMAE estimate can be obtained 1 ˆABS y w y d 2 2 ˆABS y y e y d 1 2 o By changing the variable x y 94/10/14 y ˆABS y xe xdx 1 2 xe x e x y ˆABS y 1 2 1 y ˆABS y ˆ y ABS y 1 e 2 28 Example o the solution is ˆABS y T y • where T is the solution for 1 T eT 1 2 ¸ and T~1.68. The MAP max w y =max y e y max e e y y 2 y e y ˆMAP 0 1 y o 94/10/14 29 Example multiple observation n p y e y k e n n yk k 1 k 1 w y w p y n 1 y e p y n! n ˆMMSE w y d 0 ˆMAP 94/10/14 n 1 n y k k 1 n y k k 1 1 0 1 n n yk n k 1 n 1 n yk n k 1 n 30 Nonrandom parameter(real) estimation A problem in which we have a parameter indexing the class of observation statistics, that is not modeled as a random variable but nevertheless is unknown. Don’t have enough prior information about the parameter to assign a prior probability distribution to it. Treat the estimation of such parameters in an organized manner. 94/10/14 31 Statement of problem Given the observation Y=y, what is the best estimate of is real and no information about the true value of the only averaging of cost that we can be done is with respect to the distribution of Y given ,the conditional risk R E ˆ y 2 o we can not generally expect to minimize the conditional risk uniformly o For any particular value of , 0 the conditional mean-squared error can be made zero by choosing ˆ y to be identically 0 for all observations o However, it can be poor if the value of 0 is not near the true value of 94/10/14 32 Statement of problem E ˆ y E ˆ y E 2E ˆ y E ˆ y 0 2 2 0 2 2 0 0 o Unless 0 E ˆ y 0 2 E ˆ y 2 o it is not a good estimator due to not with minimum conditional meansquared-error。 the conditional mean E ˆ y ˆ y p y dy o if E ˆ y we say that the estimate is unbiased o in general we have biased estimate b E ˆ y o variance of estimator var E ˆ y E ˆ y 2 o 94/10/14 b 0 33 MVUE var E ˆ y b 2 b 2E b ˆ y E ˆ y b 2b E ˆ y E ˆ y b E ˆ y 2 2 2 bp 0 2 2 2 for the unbias estimator b 0 ,and the variance is the conditional mean-squared error under p var E 94/10/14 ˆ y 2 The best we can hope for is minimum variance unbiased estimator-MVUE 34 Q&A 94/10/14 35