What are moment generating functions and how do I use them to find moments?

A **Moment Generating Function (MGF)** is an alternative framework to describe a probability distribution. It packs all the statistical moments ($\mathbb{E}[X], \mathbb{E}[X^2], \mathbb{E}[X^3], \dots$) of a random variable into a single differentiable function. **1. Why is it called "Moment Generating"? How to extract moments?** It is called "moment generating" because it acts as a generating function in combinatorics. If we take the Taylor series expansion of $e^{tX}$ around $t = 0$: $$e^{tX} = 1 + tX + \frac{(tX)^2}{2!} + \frac{(tX)^3}{3!} + \dots = \sum_{n=0}^{\infty} \frac{t^n X^n}{n!}$$ Taking the expected value on both sides yields: $$M_X(t) = \mathbb{E}[e^{tX}] = 1 + t\mathbb{E}[X] + \frac{t^2}{2!}\mathbb{E}[X^2] + \frac{t^3}{3!}\mathbb{E}[X^3] + \dots = \sum_{n=0}^{\infty} \frac{t^n}{n!}\mathbb{E}[X^n]$$ - To extract the $n$-th raw moment ($\mathbb{E}[X^n]$), you differentiate $M_X(t)$ exactly $n$ times with respect to $t$, and then evaluate the derivative at $t = 0$: $$\mathbb{E}[X^n] = \left. \frac{d^n}{dt^n} M_X(t) \right|_{t=0}$$ **2. MGF of $X \sim \text{Exponential}(\lambda)$ and finding $\mathbb{E}[X]$ and $\mathbb{E}[X^2]$** The Probability Density Function (PDF) of an exponential random variable is $f_X(x) = \lambda e^{-\lambda x}$ for $x \ge 0$. **Step-by-Step Computation of the MGF:** $$M_X(t) = \mathbb{E}[e^{tX}] = \int_{0}^{\infty} e^{tx} \left( \lambda e^{-\lambda x} \right) dx$$ $$M_X(t) = \lambda \int_{0}^{\infty} e^{(t - \lambda)x} dx = \lambda \left[ \frac{e^{(t - \lambda)x}}{t - \lambda} \right]_{0}^{\infty}$$ **Finding the Moments:** - **First Derivative for $\mathbb{E}[X]$:** $$M'_X(t) = \frac{d}{dt} \left[ \lambda(\lambda - t)^{-1} \right] = \lambda(-1)(\lambda - t)^{-2}(-1) = \frac{\lambda}{(\lambda - t)^2}$$ $$\mathbb{E}[X] = M'_X(0) = \frac{\lambda}{(\lambda - 0)^2} = \frac{1}{\lambda}$$ - **Second Derivative for $\mathbb{E}[X^2]$:** $$M''_X(t) = \frac{d}{dt} \left[ \lambda(\lambda - t)^{-2} \right] = \lambda(-2)(\lambda - t)^{-3}(-1) = \frac{2\lambda}{(\lambda - t)^3}$$ $$\mathbb{E}[X^2] = M''_X(0) = \frac{2\lambda}{(\lambda - 0)^3} = \frac{2}{\lambda^2}$$ **3. What is the domain of $t$ for which the MGF exists?** For the limit at infinity in the integral $\left[ \frac{e^{(t - \lambda)x}}{t - \lambda} \right]_{0}^{\infty}$ to converge to $0$, the exponent coefficient must be strictly negative: $$t - \lambda < 0 \implies t < \lambda$$ - If $t \ge \lambda$, the integral diverges to infinity. - Therefore, the domain of $t$ is **$(-\infty, \lambda)$**. An MGF is valid only if it exists in an open interval containing $t = 0$. **4. Proving the Sum of Independent Poissons is Poisson** MGFs possess a unique property: the MGF of the sum of independent random variables equals the product of their individual MGFs ($M_{X+Y}(t) = M_X(t) \cdot M_Y(t)$). Let $X \sim \text{Poisson}(\lambda_1)$ and $Y \sim \text{Poisson}(\lambda_2)$ be independent. The MGF of a Poisson variable $X$ is $M_X(t) = e^{\lambda_1(e^t - 1)}$. **Step-by-Step Proof:** $$M_{X+Y}(t) = \mathbb{E}[e^{t(X+Y)}] = \mathbb{E}[e^{tX} \cdot e^{tY}] = \mathbb{E}[e^{tX}] \cdot \mathbb{E}[e^{tY}]$$ $$M_{X+Y}(t) = e^{\lambda_1(e^t - 1)} \cdot e^{\lambda_2(e^t - 1)} = e^{(\lambda_1 + \lambda_2)(e^t - 1)}$$ - Because $e^{(\lambda_1 + \lambda_2)(e^t - 1)}$ matches the structural form of a Poisson MGF, the **Uniqueness Theorem of MGFs** guarantees that $X + Y \sim \text{Poisson}(\lambda_1 + \lambda_2)$. **5. Relationship between MGFs and Characteristic Functions** The main drawback of the MGF is that it can diverge to infinity for heavy-tailed distributions (like the Cauchy distribution). The **Characteristic Function ($\phi_X(t)$)** resolves this issue by introducing the imaginary unit $i = \sqrt{-1}$: $$\phi_X(t) = \mathbb{E}[e^{itX}]$$ - **Relationship:** $\phi_X(t) = M_X(it)$. - Since $|e^{itX}| = 1$, the integral or sum for $\phi_X(t)$ is bounded and absolutely convergent. Therefore, the characteristic function **always exists** for every real number $t$ for any random variable.