55 Law of large numbers
Definition 55.1 (Independent and identical RVs) Random variables \(X_{1},X_{2},\dots,X_{n}\) are independently and identically distributed (or i.i.d for short) if they are independently drawn from the same distribution (with the same parameters).
Definition 55.2 (Random sample) Let \(X_{1},X_{2},\dots,X_{n}\) be i.i.d random variables from distribution \(F\). We call the collection \(\{X_{1},X_{2},\dots,X_{n}\}\) a random sample of \(F\). \(F\) is known as the population distribution, or just the population.
The definition above is more general than the concept of population in everyday life (e.g. members of a group). It is an mathematical abstraction of the underlying “truth” we want to learn about.
Definition 55.3 (Converge in probability) A sequence \(Z_1,Z_2,\dots\) of random variables converges to \(b\) in probability if for every number \(\epsilon>0\), \[\lim_{n\to\infty} P(|Z_n - b|<\epsilon) = 1.\] The property is denoted by \(Z_n \to_p b\).
Theorem 55.1 (Law of large numbers) Let \(X_{1},X_{2},\dots,X_{n}\) be a random sample from a distribution for which the mean is \(\mu\) and the variance is finite. Let \(\bar{X}_n\) denote the sample mean, i.e. \(\bar{X}_n=\frac{1}{n}\sum_{i=1}^{n}X_i\). Then \[\bar{X}_n \to_p \mu.\]
Proof. Since \(X_{1},X_{2},\dots,X_{n}\) are i.i.d random variables from the same distribution. Let \(E(X_i)=\mu\) and \(Var(X_i)=\sigma^2\) for \(i=1,2,...,n\). For every number \(\epsilon>0\), by the Chebyshev inequality, \[P( |\bar{X}_n - \mu| < \epsilon ) \geq 1 - \frac{\sigma^2}{n\epsilon^2}. \] Hence, \[\lim_{n\to\infty} P(|\bar{X}_n - \mu|<\epsilon) = 1.\]
Note that sample mean, \(\bar{X}_n=\frac{1}{n}\sum_{i=1}^{n}X_i\), is the weighted sum of random variables. It is therefore itself a random variable. It is easy to see that \[\begin{aligned} E(\bar{X}_n) &= \mu \\ Var(\bar{X}_{n}) &=\frac{\sigma^{2}}{n}\to0,\quad\textrm{as }n\to\infty \end{aligned}\] The random variable \(\bar{X}_{n}\) becomes fixed at \(\mu\) as \(n\) becomes large. Thus, it converges to \(\mu\) in a probabilistic sense.
Theorem 55.2 (Continuous function of RVs) If \(Z_n \to_p b\), and \(g(\cdot)\) is a continuous function, then \(g(Z_n) \to_p g(b)\).
The theorem implies that LLN does not only apply to the sample mean, but also applies to any moments of a random variable, \[\frac{1}{n}\sum_{i=1}^{n}X_i^k \to_p E(X^k).\] This is the foundation of the method of moments estimation that we will discuss later.
Applications of LLN
It might seem that the LLN just states the obvious. But it has wide applications in daily life that you might not even realize. The LLM implies that: the uncertainty at the individual level becomes certain at aggregate level; the risks that are unmanageable at the individual level becomes manageable collectively.
Example 55.1 (Lottery) A lottery company is designing a game with a 6-digit format. Each time someone buys a ticket, they receive a randomly generated 6-digit number. Only one number will win the grand prize of 1 million dollars. What should the company charge per ticket to break even?
Solution. Let \(X_i \sim \textrm{Bern}(p)\) be the random variable that indicates whether the \(i\)-th ticket is a winner, where \(p=1/10^{6}\). For each individual who buys a ticket, the outcome is highly uncertain. But collectively, by the Law of Large Numbers, \[\frac{1}{n}\sum_{i=1}^{n}X_i \to_p p.\] Meaning that when \(n\) is large, the proportion of winners should be very close to \(p\). Therefore, the total number of winners should be very close to \(np\).
If it is estimated that there will be 10 million tickets sold. There would be almost exactly \(10^{7}p=10\) winners. The total cost of the company is therefore 10 million. If the company charge $1 per ticket, it is just enough to cover the cost. Any price above $1 would make the business profitable.
Example 55.2 (Insurance) Insurance is anther great application of the LLN. It is essentially the same as the the lottery game but most people do not realize it. Suppose there is a disease with mortality rate 1 out of a million. The medical expenditure to cure the disease is 1 million dollars. How much the insurance company should charge per customer to cover this disease?
Solution. The solution is essentially the same as above. As long as the number of customers is large enough, by the LLN, the number of claims should be very close to \(np\). Thus the risks that are unmanageable at the individual level becomes manageable when pooled together.
Insurance is a great invention because it not only provides cover for individuals but also improve the capital efficiency of the society as a whole. This about this: Without the insurance, each individual has to set aside 1 million dollars pre-cautiously for the disease (if he is rich enough) or be exposed to the risk completely uncovered. The insurance product enables everyone to get covered at the fraction of the cost and thus frees up capital for more productive uses.