188宝金博页面版

4Lecture 4: Classical Estimation Theory4.1Cramer-Rao InequalityObtaining a point estimator of the parameter of interest is usually the first step in in-ference. Suppose X = (X1, X2, .., Xn) are i.i.d. from f(x, θ), θ ∈ R and we use a statisticTn(X) to estimate θ. If Eθ(Tn) = θ+bn(θ) then the quantity bn(θ) is called bias. Note thatit generally may depend on both θ and the sample size although this dependence maysometimes be suppressed in the notation. We would hope for a zero bias for all θ and n,called unbiasedness. When used repeatedly, an unbiased estimator, in the long run, willestimate the true value on average.Caution: Note, however , that for some families an unbiased estimators may not exist or,even when they exist, may not be very useful. For example, in the case of the geometricdistribution f(x, θ) = θ(1 − θ)x−1, x = 1, 2, . . . an unbiased estimator of θ, say, T(x) mustsatisfy∞estimator satisfying this requirement would be T(1) = 1, T(x) = 0 if x ≥ 2. Having inmind the interpretation of θ( probability of success in a single try), such an estimator isneither very reliable, nor very useful.x=1T(x)θ(1 − θ)x−1= θ for all θ ∈ [0, 1]. By a polynomial expansion, the onlyWhen looking for an estimator of a “good” quality, we are inclined to analyse the meansquared errorMSEθ(Tn) = Eθ(Tn− θ)2= VarθTn+ (bn(θ))2.A small mean squared error as a criterion for choosing a point estimator, is in general moreimportant than unbiasedness. To perform optimally, we would try to find an estimatorthat minimizes the MSE. Unfortunately, in the class of all estimators, an estimator thatminimizes the MSE simultaneously for all θ values, does not exist (the argument for thiswill be given during lectures). Way out of this situation is either to restrict the class ofestimators considered, or to change the evaluation criterion. We shall be dealing with thefirst way out right now (the other way was discussed in the Decision theory chapter: Bayesand minimax estimation).We choose to impose the criterion of unbiasedness. This greatly simplifies the task ofminimizing the mean squared error because then, we only have to minimize the variance.In the (smaller subset of unbiased estimators) one can very often find an estimator withthe smallest MSE(=Var) for all θ values. It is called the uniformly minimum varianceunbiased estimator (UMVUE). Let us first look at a well-known result that will help usin our search of the UMVUE (the Cramer-Rao theorem).Theorem 4.1. Let X = (X1, X2, . . . , Xn) have a distribution that depends on θ andL(X, θ) be the joint density. Let τ(θ) be a smooth (i.e. differentiable) function of θ thathas to be estimated. Consider any unbiased estimator W(X) ofτ(θ), i.e. EθW(X) = τ(θ).Suppose, in addition, that L(X, θ) satisfies:∂∂θ..h(X)L(X, θ)dX1..dXn=..h(X)∂∂θL(X, θ)dX1..dXn(∂∂θτ(θ))2IX(θ)(∗)for any function h(X) with Eθ|h(X)| < ∞. Then: Varθ(W(X)) ≥for all θ holds.Proof: The proof is elegantly simple and is a clever application of the Cauchy- Schwartz28