Conditional Probability and Independence

Assume throughout $(\Omega, \mathcal F, \P)$ is a probability space.

Conditional Probability

Most people have notoriously bad intuition about conditional probability. Here’s an example:

Example 1. Say you have a rare disease that affects $0.01\%$ of the population. There’s a test for this disease that’s $90\%$ accurate. Suppose you test positive. What’s the chance you have the disease?

Answer. About $0.0899\%$.

It’s easy to work out the above if you simply restrict your sample space to everyone who tests positive. This is the of conditional probability.

Definition 2 (Conditional Probability). If $A, B \in \mathcal F$ with $\P(B) > 0$, then the (conditional) probability of $A$ given $B$ is defined by $$ \P(A | B ) = \frac{\P(A \cap B )}{\P(B )} \,. $$

Proposition 3. Suppose $\P(B) > 0$. For any $A \in \mathcal F$ define $\mathbf{Q}(A) = \P(A \given B)$. Then $\mathbf Q$ is a probability measure on $(\Omega, \mathcal F)$.

Definition 4. We say $\set{B_i}_{i \in I}$ partition $\Omega$ if:

  1. The index set $I$ is countable, and for every $i \in I$, $B_i \in \mathcal F$.
  2. $\cup_{i \in I} B_i = \Omega$,
  3. and for every $i, j \in I$ with $i \neq j$ we have $B_i \cap B_j = \emptyset$.

(Sometimes $\set{B_i}$ is called a mutually exclusive, exhaustive partition.)

Proposition 5 (Law of total probability). If $\set{B_i | i \in I}$ is partition of $\Omega$ such that $\P(B_i) > 0$ for every $i \in I$, then for every $A \in \mathcal F$ we have $$ \P(A) = \sum_{i \in I} \P( A \given B_i ) \P(B_i) \,. $$

Proposition 6 (Bayes’ theorem). If $\set{B_i | i \in I}$ is partition of $\Omega$ such that $\P(B_i > 0)$ for every $i \in I$, and $A \in \mathcal F$ be such that $\P(A) > 0$. Then for every $j \in \N$, $$ \P(B_j \given A) = \frac{ \P(A \given B_j) \P(B_j)}{\P(A)} = \frac{ \P(A \given B_j) \P(B_j)}{\sum_{i \in I} \P(A \given B_i) \P(B_i) } \,. $$

Remark 7. The proof is trivial (just cross multiply). The reason it gets a name is because it was realized a lot before the modern probabilistic framework was laid. It also addressed a problem or practical significance where intuitive guesses are typically way off (e.g. Example 1).

Problem 8. Solve Example 1 in general. Explicitly, suppose the chance of getting a rare disease is $\epsilon \in (0, 1)$. There’s a test for this disease that’s accurate with probability $p$. What is the probability someone who tests positive has the disease?

Answer. $\epsilon p / (1 - p - \epsilon + 2\epsilon p)$.

Problem 9. In the two envelopes problem, one of the assumptions we made was that “if the envelope you chose has $\dollar x$, then the other is equally likely to have $\dollar x/2$, or $\dollar 2x$”. Let’s now examine this precisely. Suppose $X$ is a random variable representing the amount in the envelope you chose, and $Y$ is a random variable representing the amount in the other envelope. Let’s assume $X, Y$ are discrete random variables. The data given, and assumptions we made translate to the following:

  1. $\P( X > 0 ) = \P(Y > 0) = 1$.
  2. For every $x \in \R$ such that $\P(X = x) > 0$ we have $\P(Y = x/2 | X = x) = \P(Y = 2x | X = x) = 1/2$.
  3. $\P(X = x) = \P(Y = x)$ for every $x \in \R$.

Show that there does not exist a pair of random variables satisfying the above conditions.

Proposition 10 (Memoryless of the geometric distribution). Let $\N = \set{1, 2, \dots}$. Let $\tau$ be a $\N$-valued random variable such that $\P(\tau > k) > 0$ for every $k \in \N$. Then $\tau \sim \Geom(p)$ for some $p \in (0, 1)$ if and only if for every $n, k \in \N$ we have $\P( \tau > n + k \given \tau > n ) = \P(\tau > k )$.

Remark 11. In words, the above says the following. Suppose you run repeated independent Bernoulli-$p$ trials of an experiment. Given that first $n$ fail, the probability of having $k$-more failures does not depend on $n$, and is is exactly the same as the probability of having $k$-failures at the very start.

Independent events

We would like two events to be independent if $$ \P(A \given B) = \P(A \given B^c) = \P(A) \,. $$ For this to make sense, we need $\P(B) \in (0 ,1)$. In this case, it’s easy to see that $\P(A) = \P(A \given B)$ implies $\P(A) = \P(A \given B^c)$, and both of these are equivalent to $$ \P(A \cap B) = \P(A) \P(B) \,. $$ Since this doesn’t have restrictions about $\P(B) \in (0 , 1)$, it’s more convenient to use as a definition.

For more than two events, we would like $A, B, C$ to be independent if each pair of two events is independent. Also, we would like $$ \P(A \given B, C) = \P(A) \quad \P(A \given B, C^c) = \P(A) \quad \P(B \given A^c, C) = \P(B) \quad\text{etc.} $$

It’s easy to see that these 12 criterion (combined with pairwise independence) is equivalent to the four criterion: \begin{align} \P(A \cap B) &= \P(A) \P(B) & \P(B \cap C) &= \P(B) \P(C) \\ \P(A \cap C) &= \P(A) \P(C) & \P(A \cap B \cap C) &= \P(A) \P(B) \P(C) \end{align}

Since this is simpler we use this as the definition.

Definition 12 (Independence). We say a (possibly uncountable) family of events $\set{A_\alpha}_{\alpha \in \mathcal A}$ are independent if for any finite set of distinct indices $\alpha_1$, …, $\alpha_N \in \mathcal A$ we have $$ \P\paren[\big]{ A_{\alpha_1} \cap \cdots \cap A_{\alpha_N}} = \P( A_{\alpha_1} ) \cdots \P( A_{\alpha_N} ) \,. $$

Problem 13. Let $\set{A_1, A_2, A_3}$ be three events. For each of the parts below, provide either a proof, or a counter example.

  1. If every pair of these three events is independent, must all three of them be independent?
  2. If we only knew $\P(A_1 \cap A_2 \cap A_3) = \P(A_1) \P(A_2) \P(A_3)$, then must $\set{A_1, A_2, A_3}$ be independent?
  3. If every pair of these events is independent, must $A_1 \cup A_2$ be independent of $A_3$?
  4. If $\set{A_1, A_2, A_3}$ are independent, then must $A_1 \cup A_2$ be independent of $A_3$?

Problem 14. Suppose $A_1$, …, $A_N$ are independent events such that $\P(A_i) = p \in [0, 1]$ for all $i$. What is the probability that exactly $k$ of them occur?

Problem 15. Suppose $\set{A_\alpha}_{\alpha \in \mathcal A}$ is some family of independent events. For every $\alpha \in \mathcal A$, choose $B_\alpha \in \set{A_\alpha, A_\alpha^c}$. Show that $\set{B_\alpha}_{\alpha \in \mathcal A}$ is independent.

Independent Random Variables

Supopse $X, Y$ are two discrete random variables. We would like two call $X$ and $Y$ independent if all events you can observe from $X$ are independent of all events you can observe from $Y$. Explicitly for any pair of sets $A, B \subseteq \R$, we would like the events $\set{X \in A}$ and $\set{Y \in B}$ to be independent. That is, for every pair of Borel sets $A, B$, we would like $$\label{e:I1}\tag{I1} \P(X \in A, Y \in B) = \P(X \in A) \P(Y \in B) \,. $$ Checking this for all $A, B \subseteq \R$ is cumbersome; it turns out (for discrete random variables) you only have to check it for singleton sets. This is more convenient to use as the definition.

Definition 16. We say two discrete random variables are independent if for every $x, y \in \R$ we have $$\label{e:I2} \P(X = x, Y = y) = \P(X = x) \P(Y = y) \,. $$

Proposition 17. Let $X, Y$ be two independent discrete random variables. Then \eqref{e:I1} holds for every $A, B \subseteq \R$.

The same holds for more than two random variables.

Definition 18. Let $\set{X_\alpha}_{\alpha \in \mathcal A}$ be a family of discrete random variables. We say $\set{X_\alpha}_{\alpha \in \mathcal A}$ are independent if for every finite set of distinct indices $\alpha_1, \dots, \alpha_N \in \mathcal A$, and every $x_1, \dots, x_N \in \R$, we have $$ \P( X_{\alpha_1} = x_1, \dots, X_{\alpha_N} = x_N) = \P( X_{\alpha_1} = x_1) \cdots \P( X_{\alpha_N} = x_N) \,. $$

Proposition 19. If $\set{X_\alpha}_{\alpha \in \mathcal A}$ is a family of independent discrete random variables, then for every family of sets $\set{A_\alpha}_{\alpha \in \mathcal A} \subseteq \mathcal P(\R)$, the events $\set{ \set{X_\alpha \in A_\alpha} \st \alpha \in \mathcal A}$ are independent.

Remark 20. Things are a little trickier for random variables that are not discrete, as the events $\set{X = x}$ may all have probability $0$. We revisit this later.

Proposition 21. Let $\set{X_\alpha}_{\alpha \in \mathcal A}$ be a family of discrete independent random variables. For every $\alpha \in \R$, let $f_\alpha \colon \R \to \R$ be a function. Then $\set{f_\alpha(X_\alpha)}_{\alpha \in \mathcal A}$ are also independent discrete random variables.

Problem 22. Show that the random variables $\set{X_n}$ defined in this problem are independent Bernoulli-$p$ random variables.

Problem 23. Let $X, Y$ be two discrete random variables with probability mass functions $p_X$ and $p_Y$. Note, $X$, $Y$ need not even be defined on the same probability space, and it may not even make sense to talk about their joint distribution or whether or not they are independent.

Show that there exists a probability space $(\tilde \Omega, \tilde{\mathcal F}, \tilde{\P})$ and two random variables $\tilde X, \tilde Y$, both defined on $\tilde \Omega$ such that: $p_{\tilde X} = p_X$, $p_{\tilde Y} = p_Y$, and the pair of random variables $\tilde X, \tilde Y$ are independent.