Expectations of Random Variables

Expectations

Suppose you play a game whose reward is the random variable $X$. The reward you get on average is called the expectation or mean of the random variable $X$.

Definition 1 (Expectation). Let $X$ be a discrete random variable. We define the expectation of $X$ by $$ \E X = \sum_{x \in \R} x \P(X = x) = \sum_{x \in \R \cap X(\Omega)} x \P(X = x) \,, $$ provided the series $\sum_x \abs{x} \P(X = x)$ converges absolutely.

Remark 2. This definition is only for discrete random variables; In this case the sum above is countable and makes sense. If $X$ is not a discrete random variable, then one needs a little more work to define the expectation.

Remark 3. Think of the expectation as a weighted average. Values $X$ takes on with higher probability have more weight.

Remark 4. We use the convention that expectation has lower precedence than multiplication. That is $\E X^2$ means $\E(X^2)$, and $\E XY$ means $\E(X Y)$.

Proposition 5 (Lazy statistician). Let $g \colon \R \to \R$ be any function, and $X$ be a discrete random variable. Then $g(X)$ is also a discrete random variable, and $$ \E g(X) = \sum_{x \in \R} g(x) \P(X = x)\,, $$ whenever the sum converges absolutely.

Remark 6. If $X$ is not discrete, then you’re not guaranteed $g(X)$ is a random variable unless $g$ is well behaved (e.g. continuous, or Borel measurable).

Proposition 7 (Linearity). If $X, Y$ are two discrete random variables, $\alpha \in \R$ then $\E(X + \alpha Y) = \E X + \alpha \E Y$.

Proposition 8 (Monotonicity). If $X, Y$ are two discrete random variables such that $\P(X \leq Y) = 1$, then $\E X \leq \E Y$ (provided $\E X$ and $\E Y$ are both defined.

Definition 9. We say an event $A$ holds almost surely if $\P(A) = 1$.

Remark 10. Proposition 8 with this terminology says that if $X, Y$ are discrete random variables such that $X \leq Y$ almost surely, then $\E X \leq \E Y$.

Example 11. In the repeated Bernoulli trial example, the random variable $\tau$ is defined almost surely. In other words, you will almost surely get at least one success. (Or, for the infinite monkey theorem, they will almost surely produce the complete works of Shakespear.)

Proposition 12 (Layer cake). Suppose $X \geq 0$ is a nonnegative discrete random variable. Then $$\label{e:LC}\tag{LC} \E X = \int_0^\infty \P(X \geq t ) \, dt \,. $$

Proof sketch. Say $X(\Omega) = \set{x_k \st k \in \N}$, and note \begin{align} \int_0^\infty \P(X \geq t) \, dt &= \lim_{N \to \infty} \frac{1}{N} \sum_{m = 1}^\infty \P\paren[\Big]{X \geq \frac{m}{N} } \\ &= \lim_{N \to \infty} \frac{1}{N} \sum_{m = 1}^\infty \sum_{k = 1}^\infty \one_{\set{x_k \geq m/N}} \P\paren{X = x_k } \\ &= \lim_{N \to \infty} \sum_{k = 1}^\infty \frac{\floor{N x_k}}{N} \P\paren{X = x_k } = \E X \,. \end{align}

Remark 13. The last step of the proof involved interchanging the limit and an infinite sum, which requires proof. This can be done directly in this case, but won’t be done in this course.

Problem 14. Let $\varphi \colon [0, \infty) \to [0, \infty)$ be a continuously differentiable increasing function with $\varphi(0) = 0$, and $X$ be a nonnegative discrete random variable.

  1. Show that $$ \E \varphi(X) = \int_0^\infty \varphi’(t) \P(X \geq t) \, dt \,. $$
  2. Is the above formula still valid if $\varphi(0) \neq 0$?
  3. Is the above formula still valid if $\varphi$ is not necessarily increasing?

Remark 15. So far we defined $\E X$ only for discrete random variables. For random variables that are not discrete the full definition involves the Lebesgue integral, which requires a bit of work to construct. However, an equivalent (but not particularly intuitive) definition would be to define $\E X$ via \eqref{e:LC} for nonnegative random variables, even if they aren’t discrete. If $X$ isn’t nonnegative, you can write $X = X^{+} - X^{-}$, where $X^{+} = \max\set{X, 0}$, and $X^{-} = \max\set{-X, 0}$. Now $X^+$ and $X^{-}$ are nonnegative random variables, so their expectation is defined by \eqref{e:LC}, and we can define $\E X = \E X^{+} - \E X^{-}$ (provided one is finite). While this defines $\E X$, it does pose some challenges – for instance, using this definition it’s not easy to show linearity or monotonicity as we had in Propositions 7 and 8.

Going forward, we will assume that $\E X$ is can be defined for random variables that are not necessarily discrete, in a manner that:

  1. For discrete random variables, $\E X$ is exactly as in Definition 1.
  2. If $\abs{X} \leq Y$ almost surely, and $\E Y$ is defined, then so is $\E X$.
  3. When defined, the expectation satisfies the linearity and monotonicity properties

Variance

The variance of a random variable is one measure of its deviation from the mean.

Definition 16. The standard deviation and variance of a random variable $X$ are defined by $$ \std(X) \defeq \sqrt{ \E( X - \E X)^2 } \quad\text{and}\quad \var(X) \defeq \E( X - \E X)^2 \,, $$ respectively.

Remark 17. A useful formula is $$ \var(X) = \E X^2 - (\E X)^2 \,. $$

Problem 18.

  1. Let $X$ be a discrete random variable such that $\P(X \geq 0) = 1$. Then show that $\E X = 0$ if and only if then $\P(X = 0) = 1$.
  2. If $X$ is a discrete random variable such that $\var(X) = 0$, then show that $X = \E X$ almost surely.

Proposition 19. If $\alpha \in \R$, and $X$ is a random variable then $\var(\alpha X) = \alpha^2 \var(X)$

Proposition 20. If $X, Y$ are two random variables then $\var(X + Y) = \var(X) + \var(Y) + 2 \cov(X, Y)$, where $$ \cov(X, Y) \defeq \E (X - \E X) (Y - \E Y) \defeq \E \paren[\big]{ (X - \E X) (Y - \E Y) } $$ is the covariance between $X$ and $Y$.

Definition 21. Two random variables $X, Y$ are called uncorrelated if $\cov(X, Y) = 0$.

Remark 22. We will see shortly that independent random variables are always uncorrelated; but uncorrelated random variables need not be independent.

Proposition 23.

  1. If $X \sim \Bernoulli(p)$, then $\E X = p$ and $\var(X) = p (1 - p)$.
  2. If $X \sim \Binom(N, p)$, then $\E X = Np$ and $\var(X) = N p (1-p)$.
  3. If $X \sim \Geom(p)$, then $\E X = 1/p$ and $\var(X) = (1 - p) / p^2$.
  4. If $X \sim \Pois(\lambda)$, then $\E X = \lambda$ and $\var(X) = \lambda$.

Problem 24. If $X \sim \Pois(\lambda)$, then show that for every $k \in \set{0, 1, \dots}$, $$ \E \binom{X}{k} = \frac{\lambda^k}{k!} $$

Problem 25 (Coupon collector). There are $N$ different types of coupons. In each cereal box, the maker puts one coupon (whose type is chosen independently, and uniformly at random). Let $T$ be the number of boxes a customer buys until they collect a complete set (i.e. one of each type). Find $\E T$.

Solution sketch. Let $\tau_k$ be the number of boxes a customer buys until they first collect $k$-distinct coupons. (Let $\tau_0 = 0$ for convenience.) Show that $\tau_{k+1} - \tau_k \sim \Geom( (N - k) / N )$. Then write $T = \tau_N = \sum_{k = 0}^{N-1} \tau_{k+1} - \tau_k$, and compute $\E T$.

Problem 26 (Chebychev’s inequality). Let $X$ be a (discrete) random variable. For any $p, \lambda > 0$ we have $$ \P( \abs{X} \geq \lambda ) \leq \E (\abs{X}^p) / \lambda^p \,. $$ (This is sometimes also called Markov’s inequality.)

Problem 27 (Jensen’s inequality). Let $I \subseteq \R$ be an interval, $\varphi \colon I \to \R$ be a convex (differentiable) function, and $X$ be a (discrete) random variable such that $\P(X \in I) = 1$. Show that $\varphi( \E X ) \leq \E \varphi(X)$.

Hint. Convex functions are always above their tangent. That is, for any $a, x \in \R$, we have $\varphi(a) + (x - a)\varphi’(a) \leq \varphi(x)$.

Problem 28 (Young’s inequality). Let $x, y > 0$, $p, q \in (1, \infty)$ be such that $1/p + 1/q = 1$. Show that $$ x y \leq \frac{x^p}{p} + \frac{y^q}{q} \,. $$

Hint. Find a nice random variable $X$ so that $\E X = x^p/p + y^q / q$, and use Jensen’s inequality on $-\ln \E X$.

Problem 29 (Hölder’s inequality). If $p, q \in (1, \infty)$ are such that $1/p + 1/q = 1$, and $X, Y$ are two (discrete) random variables then $$ \abs{\E X Y} \leq \paren[\big]{ \E \abs{X}^p }^{1/p} \paren[\big]{ \E \abs{Y}^q }^{1/q} \,. $$

Hint. Use Young’s inequality and first prove this when $\E \abs{X}^p = \E \abs{Y}^q = 1$.

Problem 30 (Minkowski’s inequality). If $p \in (1, \infty)$, and $X, Y$ are (discrete) random variables then $$ \paren[\big]{ \E \abs{X + Y}^p }^{1/p} \leq \paren[\big]{ \E \abs{X}^p }^{1/p} + \paren[\big]{ \E \abs{Y}^p }^{1/p} $$

Hint. Find a nice random variable $Z$ so that $\E (X + Y) Z = (\E \abs{X + Y}^p)^{1/p}$.

Definition 31 (Pearson’s correlation coefficient). Let $X, Y$ be two random variables which are not constant almost surely. The Pearson’s prodcut-moment correlation coefficient is defined by $$\label{e:r}\tag{r} r \defeq \frac{\cov(X, Y)}{\std(X) \std(Y)} \,. $$

Proposition 32. Let $X, Y$ be two random variables which are not constant almost surely, and let $r$ be the Pearson’s correlation coefficient.

  1. We always have $r \in [-1, 1]$.
  2. $r = 1$ if and only if there exists $\alpha > 0$ such that $Y - \E Y = \alpha (X - \E X)$, almost surely.
  3. $r = -1$ if and only if there exists $\alpha > 0$ such that $Y - \E Y = -\alpha (X - \E X)$, almost surely.

Conditional Expectation

Definition 33. Let $X$ be a discrete random variable and $B \in \mathcal F$ is such that $\P(B) > 0$. Define the conditional expectation of $X$, given $B$ by $$ \E(X \given B) = \sum_{x \in \R} x \P(X = x \given B) = \sum_{k = 1}^\infty x_k \P(X = x_k \given B) \,, $$ where $X(\Omega) = \set{ x_k \st k \in \N}$.

Remark 34. Recall $\mathbf{Q}(A) = \P(A \given B)$ defines a probability measure on $\mathcal F$. Note $\E(X \given B)$ is simply the expectation of the random variable $X$ under the probability measure $Q$. Consequently all theorems we proved about expectations (e.g. linearity, monotonicity, etc) are still true for conditional expectations.

Proposition 35. If $I$ is countable, $\set{B_i \st i \in I}$ partition $\Omega$ and $\P(B_i) > 0$ for every $i$, then $$ \E X = \sum_{i \in I} \E(X \given B_i) \P(B_i) \,, $$ whenever the sum converges absolutely.

Definition 36. A sequence of random variables $\set{X_n \st n \in \N}$ are called independent and identically distributed (abbreviated to i.i.d.) if $\set{X_n}_{n \in \N}$ is independent, and for every $n \in \N$, $\P_{X_n} = \P_{X_1}$. (Recall $\P_X$ is the distribution of the random variable $X$.)

Problem 37. Show that the random variables $\set{X_n}$ defined in this problem are i.i.d. Bernoulli-$p$ random variables. (This is the same as this problem.)

Example 38. If $\tau \sim \Geom(p)$, then $\E \tau = 1/p$.

Proof sketch. This can be done directly (as you probably already did in Problem 23). But here’s a cute way using conditional expectation. Let $\set{X_n}_{n \in \set{1, 2, \dots}}$ be a sequence of i.i.d. $\Bernoulli(p)$ random variables, and $\tau = \min\set{n \st X_n = 1}$. Using Proposition 35 we see $$ \E \tau = \E(\tau \given X_1 = 1) p + \E(\tau \given X_1 = 0) = p + q (1 + \E(\tau)) \,, $$ and hence $\E \tau = 1/p$.

Problem 39. Suppose $\set{X_n \st n \in \N}$ is a sequence of i.i.d. $X_n \sim \Bernoulli(p)$ random variables. Let $Y$ be the longest initial run. That is $Y$ is the smallest $N$ so that $X_1 = X_2 = \dots, X_N$, and $X_N \neq X_{N+1}$. Compute $\E Y$.

Answer. $\E Y = \frac{1}{p(1-p)} - 2$.

Problem 40. Suppose $\set{X_n \st n \in \N}$ is a sequence of i.i.d. $X_n \sim \Bernoulli(p)$ random variables. Let $N \in \N$ and let $\tau$ be the first time $n$ you observe $N$ consecutive occurrences of $X_k = 1$. That is $$ \tau = \min \set{n \in \N \st X_n = X_{n-1} \cdots = X_{n - N + 1} = 1, X_{n - N} = 0 } \,, $$ with the convention $X_0 = 0$. Find $\E \tau$.

Answer. $$ \E \tau = \frac{1 - p^N}{(1 - p) p^N} = \sum_{k=1}^N \frac{1}{p^k} $$