59 Estimator properties
Definition 59.1 (Bias) The bias of an estimator \(\hat{\theta}\) of a parameter \(\theta\) is defined as: \[\textrm{Bias}[\hat{\theta}]=E(\hat{\theta})-\theta.\] An estimator is biased means its sampling is incorrectly centered. An estimator is unbiased if \[E(\hat{\theta})=\theta.\]
Proposition 59.1 (Sample mean) The sample mean, defined as: \[\bar{X}_{n}= \frac{1}{n} \sum_{i=1}^n X_i,\] is an unbiased estimator of \(\mu=E(x)\).
Proof. \[E(\bar{X}_{n})=E\left(\frac{1}{n}\sum_{i=1}^{n}X_{i}\right)=\frac{1}{n}\sum_{i=1}^{n}E(X_{i})=\frac{1}{n}\sum_{i=1}^{n}\mu=\mu.\]
Proposition 59.2 (Sample variance) The sample variance, defined as: \[s^{2} = \frac{1}{n-1}\sum_{i=1}^{n}(X_{i}-\bar{X_{n}})^{2},\] is an unbiased estimator of \(\sigma^{2}\) (\(\sigma^2<\infty\)).
Proof. First note: \[\sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n (X_i - \mu)^2 - n(\bar{X} - \mu)^2\] Take expectations of both sides: \[E\left[\sum_{i=1}^n (X_i - \bar{X})^2\right] = n\sigma^2 - n\left(\frac{\sigma^2}{n}\right) = (n-1)\sigma^2\] Dividing by \(n-1\) makes \(E(s^2)=\sigma^2\).
Definition 59.2 (Mean absolute error) The mean absolute error (MAE) of an estimator is defined as: \[\textrm{MAE}[\hat{\theta}]=E\left| \hat{\theta}-\theta \right|.\]
Definition 59.3 (Mean square error) The mean square error (MSE) of an estimator is defined as: \[\textrm{MSE}[\hat{\theta}]=E\left[(\hat{\theta}-\theta)^{2}\right].\]
Proposition 59.3 (Bias-variance trade-off) For any estimator with a finite variance, we have \[\textrm{MSE}[\hat{\theta}]=Var[\hat{\theta}]+(\textrm{Bias}[\hat{\theta}])^{2}.\]
Proof. By expanding the MSE we find that \[\begin{aligned} \textrm{MSE}[\hat{\theta}] & =E\left[(\hat{\theta}-\theta)^{2}\right]\\ & =E\left[(\hat{\theta}-E[\hat{\theta}]+E[\hat{\theta}]-\theta)^{2}\right]\\ & =E\left[(\hat{\theta}-E[\hat{\theta}])^{2}\right] + 2E(\hat{\theta}-E[\hat{\theta}])(E[\hat{\theta}]-\theta) + (E[\hat{\theta}]-\theta)^{2}\\ & =Var[\hat{\theta}]+(\textrm{Bias}[\hat{\theta}])^{2}. \end{aligned}\] Thus, the MSE is the variance plus the squared bias. A good estimator balances the variance (precision) and the bias (correctly centered).
Definition 59.4 (Consistency) An estimator is consistent if \(\textrm{MSE}[\hat{\theta}]\to0\) as \(n\to\infty\).
Bias is the property of an estimator for finite samples. Consistency is the property of an estimator when the sample size gets large. A consistent estimator behaves well for large sample size. Whereas an unbiased estimator is correct centered even for small samples.