51  Gamma distribution

The Gamma distribution is a continuous distribution on the positive real line; it is a generalization of the Exponential distribution. While an Exponential RV represents the waiting time for the first event to occur, we shall see that a Gamma RV represents the total waiting time for \(n\) events to occur.

Let’s start with a simple case. Suppose we want to find out the total waiting until the 2nd event occurred. Let \(Y=X_{1}+X_{2}\) where \(X_{1},X_{2}\sim \textrm{Exp}(\lambda)\) independently. If \(Y\) is discrete, we have \(P(Y=y)=\sum_{k=0}^{y}P(X_{1}=k,X_{2}=y-k)\). For continuous \(y\), we have \[\begin{aligned} f_{Y}(y) & =\int_{0}^{y}f_{X}(x)f_{X}(y-x)dx=\int_{0}^{y}\lambda e^{-\lambda x}\lambda e^{-\lambda(y-x)}dx\\ & =\int_{0}^{y}\lambda^{2}e^{-\lambda y}dx=\lambda^{2}e^{-\lambda y}y.\end{aligned}\]

If there is a third variable, \[\begin{aligned} f_{Z}(z) & =\int_{0}^{z}f_{X}(x)f_{Y}(z-x)dx=\int_{0}^{z}\lambda e^{-\lambda x}\lambda^{2}e^{-\lambda(z-x)}(z-x)dx\\ & =\lambda^{3}e^{-\lambda z}\int_{0}^{z}(z-x)dx=\lambda^{3}e^{-\lambda z}z^{2}/2.\end{aligned}\]

The general pattern is the Gamma distribution.

Definition 51.1 (Exponential distribution) An random variable X is said to have the Gamma distribution with parameters \(a\) and \(\lambda\), \(a>0\) and \(\lambda>0\), if it has the PDF \[f(x)=\frac{\lambda^{a}}{\Gamma(a)}x^{a-1}e^{-\lambda x},\quad x>0\] We write \(X\sim\textrm{Gamma}(a,\lambda)\).

Verify this is a valid PDF:

\[\int_{0}^{\infty}\frac{1}{\Gamma(a)}(\lambda x)^{a}e^{-\lambda x}\frac{dx}{x}\overset{u=\lambda x}{=}\frac{1}{\Gamma(a)}\int_{0}^{\infty}u^{a}e^{-u}\frac{du}{u}=\frac{\Gamma(a)}{\Gamma(a)}=1.\]

Taking \(a=1\), the \(\textrm{Gamma}(1,\lambda)\) PDF is \(f(x)=\lambda e^{-\lambda x}\), which is the same as \(\textrm{Exp}(\lambda)\). So Exponential distribution is a special case of Gamma distribution.

Let’s find the expectation and variance of the Gamma distribution. Let \(Y\sim\textrm{Gamma}(a,1)\). Recall \(\Gamma\) function has the property \(\Gamma(a+1)=a\Gamma(a)\).

\[E(Y)=\int_{0}^{\infty}y\cdot\frac{1}{\Gamma(a)}y^{a-1}e^{-y}dy=\frac{1}{\Gamma(a)}\int_{0}^{\infty}y^{a}e^{-y}dy=\frac{\Gamma(a+1)}{\Gamma(a)}=a.\]

Apply LOTUS to evaluate the second moment:

\[E(Y^{2})=\int_{0}^{\infty}y^{2}\cdot\frac{1}{\Gamma(a)}y^{a-1}e^{-y}dy=\frac{1}{\Gamma(a)}\int_{0}^{\infty}y^{a+1}e^{-y}dy=\frac{\Gamma(a+2)}{\Gamma(a)}=(a+1)a.\]

Therefore, \[Var(Y)=(a+1)a-a^{2}=a.\]

So for \(Y\sim\textrm{Gamma}(a,1)\), \(E(Y)=Var(Y)=a\). For the general case \(X\sim\textrm{Gamma}(a,\lambda)\), we now show that \(X=\frac{Y}{\lambda}\). Note that \[\begin{aligned} F_{X}(x)=P(X & \leq x)=P(Y\leq x/\lambda)=F_{Y}(x/\lambda)\\ f_{X}(x)=\frac{dF_{X}}{dx} & =\frac{\partial F_{Y}}{\partial y}\frac{dy}{dx}=f_{Y}(y)\lambda\end{aligned}\]

Therefore, \[f_{X}(x)=\frac{1}{\Gamma(a)}y^{a-1}e^{-y}\lambda=\frac{\lambda^{a}}{\Gamma(a)}x^{a-1}e^{-\lambda x}.\]

Hence, we have \(E(X)=\frac{a}{\lambda}\), \(Var(X)=\frac{a}{\lambda^{2}}\).

Theorem 51.1 (Exponential-Gamma connection) Let \(X_{1},\dots,X_{n}\) be independent and identical \(\textrm{Exp}(\lambda)\). Then \[X_{1}+\cdots+X_{n}\sim\textrm{Gamma}(n,\lambda).\]

Proof. Let’s prove by showing the MGFs are equivalent. \[M_{X}(t)=E(e^{tX})=\int_{0}^{\infty}e^{tx}\lambda e^{-\lambda x}dx=\frac{\lambda}{\lambda-t}\quad\textrm{for }t<\lambda\]

Thus, the MGF of \(Y=X_{1}+\cdots+X_{n}\) is \(M_{Y}(t)=\left(M_{X}(t)\right)^{n}=\left(\frac{\lambda}{\lambda-t}\right)^{n}\). We verify this is the MGF of a Gamma distribution. Suppose \(Y\sim\textrm{Gamma}(n,\lambda)\), it has MGF:

\[\begin{aligned} M_{Y}(t) & =E(e^{tY})=\int_{0}^{\infty}e^{ty}\frac{\lambda^{n}}{\Gamma(a)}y^{n-1}e^{-\lambda y}dy\\ & =\frac{\lambda^{n}}{(\lambda-t)^{n}}\int_{0}^{\infty}\frac{1}{\Gamma(a)}((\lambda-t)y)^{n-1}e^{-(\lambda-t)y}(\lambda-t)dy\\ & =\frac{\lambda^{n}}{(\lambda-t)^{n}}\int_{0}^{\infty}\frac{1}{\Gamma(a)}u^{n-1}e^{-u}du\qquad u=(\lambda-t)y\\ & =\left(\frac{\lambda}{\lambda-t}\right)^{n}.\end{aligned}\]

Thus, if \(X_{i}\) represents the i.i.d inter-arrival time. \(Y\) has the interpretation of the arrival time until the \(n\)-th event. \[Y=\sum_{i=1}^{n}X_{i}=\sum_{i=1}^{n}\textrm{(time of the i-th arrival)}\sim\textrm{Gamma}(n,\lambda).\]

Example 51.1 (Service time in a queue) Customer \(i\) must wait time \(X_{i}\) for service once reaching the head of the queue. The average service rate is 1 customer per 10 minutes. Assume the service for each customer is independent. If you are the 5th in the queue. What is the expected waiting to be served?

Solution. \(X_{i}\sim\textrm{Exp}(0.1)\). Then \(E(X_{i})=10\). Let Y be the time until you are served. Then \(Y\sim\textrm{Gamma}(5,0.1)\). Thus, \(E(Y)=\frac{5}{0.1}=50\) minutes. The probabilities of some selected values: \[P(Y\leq t)=\begin{cases} 5\% & t=20\\ 18\% & t=30\\ 71\% & t=60 \end{cases}.\]