Probability & Statistics Fundamentals

Cheatsheet Content

1. Basic Concepts of Probability Theory 1.1 Sample Space and Events Sample Space ($\Omega$ or $S$): Set of all possible outcomes of an experiment. Event ($A$): A subset of the sample space. Elementary Event: An event consisting of a single outcome. Certain Event: $\Omega$ (always occurs). Impossible Event: $\emptyset$ (never occurs). 1.2 Set Operations for Events Union ($A \cup B$): $A$ or $B$ or both occur. Intersection ($A \cap B$ or $AB$): Both $A$ and $B$ occur. Complement ($A^c$ or $\bar{A}$): $A$ does not occur. Mutually Exclusive (Disjoint) Events: $A \cap B = \emptyset$. De Morgan's Laws: $(A \cup B)^c = A^c \cap B^c$ $(A \cap B)^c = A^c \cup B^c$ 1.3 Axioms of Probability $P(A) \ge 0$ for any event $A$. $P(\Omega) = 1$. If $A_1, A_2, \dots$ are mutually exclusive events, then $P(A_1 \cup A_2 \cup \dots) = \sum_{i=1}^{\infty} P(A_i)$. 1.4 Properties of Probability $P(\emptyset) = 0$. $P(A^c) = 1 - P(A)$. If $A \subseteq B$, then $P(A) \le P(B)$. $P(A \cup B) = P(A) + P(B) - P(A \cap B)$. For mutually exclusive events: $P(A \cup B) = P(A) + P(B)$. 1.5 Conditional Probability Definition: $P(A|B) = \frac{P(A \cap B)}{P(B)}$, for $P(B) > 0$. Multiplication Rule: $P(A \cap B) = P(A|B)P(B) = P(B|A)P(A)$. Total Probability Theorem: If $B_1, B_2, \dots, B_n$ are a partition of $\Omega$ (mutually exclusive and exhaustive), then $P(A) = \sum_{i=1}^{n} P(A|B_i)P(B_i)$. 1.6 Bayes' Theorem $P(B_k|A) = \frac{P(A|B_k)P(B_k)}{\sum_{i=1}^{n} P(A|B_i)P(B_i)}$. 1.7 Independence of Events Events $A$ and $B$ are independent if $P(A \cap B) = P(A)P(B)$. Equivalent conditions for independence (if $P(A)>0, P(B)>0$): $P(A|B) = P(A)$ $P(B|A) = P(B)$ If $A$ and $B$ are independent, then $A^c$ and $B$, $A$ and $B^c$, $A^c$ and $B^c$ are also independent. 2. Discrete Random Variables 2.1 Definition of a Random Variable (RV) A function that maps outcomes from the sample space to real numbers. Discrete RV: Takes on a finite or countably infinite number of values. 2.2 Probability Mass Function (PMF) For a discrete RV $X$, the PMF is $P_X(x) = P(X=x)$. Properties: $P_X(x) \ge 0$ for all $x$. $\sum_x P_X(x) = 1$. 2.3 Cumulative Distribution Function (CDF) $F_X(x) = P(X \le x) = \sum_{k \le x} P_X(k)$. Properties: $0 \le F_X(x) \le 1$. $F_X(x)$ is non-decreasing. $\lim_{x \to -\infty} F_X(x) = 0$, $\lim_{x \to \infty} F_X(x) = 1$. $P(a $P(X=x) = F_X(x) - F_X(x^-)$ (jump at $x$). 2.4 Expectation of a Discrete RV Expected Value (Mean): $E[X] = \mu_X = \sum_x x P_X(x)$. Expected Value of a Function: $E[g(X)] = \sum_x g(x) P_X(x)$. Properties of Expectation: $E[c] = c$ (for constant $c$). $E[cX] = cE[X]$. $E[X+Y] = E[X] + E[Y]$ (linearity, always true). $E[aX+b] = aE[X]+b$. 2.5 Variance of a Discrete RV Definition: $Var(X) = \sigma_X^2 = E[(X - \mu_X)^2]$. Alternative Formula: $Var(X) = E[X^2] - (E[X])^2$. Standard Deviation: $\sigma_X = \sqrt{Var(X)}$. Properties of Variance: $Var(c) = 0$. $Var(cX) = c^2 Var(X)$. $Var(aX+b) = a^2 Var(X)$. 2.6 Common Discrete Distributions Distribution PMF $P_X(x)$ Mean $E[X]$ Variance $Var(X)$ Bernoulli($p$) $p^x (1-p)^{1-x}$ for $x \in \{0, 1\}$ $p$ $p(1-p)$ Binomial($n, p$) $\binom{n}{x} p^x (1-p)^{n-x}$ for $x \in \{0, \dots, n\}$ $np$ $np(1-p)$ Geometric($p$) $(1-p)^{x-1} p$ for $x \in \{1, 2, \dots\}$ $1/p$ $(1-p)/p^2$ Poisson($\lambda$) $\frac{\lambda^x e^{-\lambda}}{x!}$ for $x \in \{0, 1, 2, \dots\}$ $\lambda$ $\lambda$ 3. One Random Variable (General Concepts) 3.1 General CDF Properties $F_X(x) = P(X \le x)$. $F_X(x)$ is right-continuous: $\lim_{x \to x_0^+} F_X(x) = F_X(x_0)$. $P(X > x) = 1 - F_X(x)$. $P(X = x) = F_X(x) - F_X(x^-)$. (For continuous RVs, $P(X=x)=0$). 3.2 Continuous Random Variables A RV $X$ is continuous if its CDF $F_X(x)$ is continuous everywhere. Probability Density Function (PDF): $f_X(x) = \frac{d}{dx} F_X(x)$. Properties of PDF: $f_X(x) \ge 0$ for all $x$. $\int_{-\infty}^{\infty} f_X(x) dx = 1$. $F_X(x) = \int_{-\infty}^{x} f_X(u) du$. $P(a \le X \le b) = \int_a^b f_X(x) dx$. For continuous RVs, $P(X=x) = 0$. So $P(a \le X \le b) = P(a 3.3 Expectation of a Continuous RV Expected Value (Mean): $E[X] = \mu_X = \int_{-\infty}^{\infty} x f_X(x) dx$. Expected Value of a Function: $E[g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x) dx$. 3.4 Variance of a Continuous RV Definition: $Var(X) = \sigma_X^2 = E[(X - \mu_X)^2] = \int_{-\infty}^{\infty} (x - \mu_X)^2 f_X(x) dx$. Alternative Formula: $Var(X) = E[X^2] - (E[X])^2$. 3.5 Common Continuous Distributions Distribution PDF $f_X(x)$ Mean $E[X]$ Variance $Var(X)$ Range Uniform($a, b$) $\frac{1}{b-a}$ for $a \le x \le b$, 0 otherwise $\frac{a+b}{2}$ $\frac{(b-a)^2}{12}$ $[a, b]$ Exponential($\lambda$) $\lambda e^{-\lambda x}$ for $x \ge 0$, 0 otherwise $1/\lambda$ $1/\lambda^2$ $[0, \infty)$ Gaussian (Normal)($\mu, \sigma^2$) $\frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ $\mu$ $\sigma^2$ $(-\infty, \infty)$ Standard Normal RV ($Z$): $Z \sim N(0, 1)$, $\Phi(z) = P(Z \le z)$. If $X \sim N(\mu, \sigma^2)$, then $Z = \frac{X-\mu}{\sigma} \sim N(0, 1)$. 3.6 Markov and Chebyshev Inequalities Markov's Inequality: For a non-negative RV $X$ ($X \ge 0$), $P(X \ge a) \le \frac{E[X]}{a}$ for $a > 0$. Chebyshev's Inequality: $P(|X - \mu_X| \ge k) \le \frac{Var(X)}{k^2}$ for $k > 0$. Equivalently, $P(|X - \mu_X| 3.7 Moment Generating Function (MGF) Definition: $M_X(s) = E[e^{sX}]$. Discrete: $M_X(s) = \sum_x e^{sx} P_X(x)$. Continuous: $M_X(s) = \int_{-\infty}^{\infty} e^{sx} f_X(x) dx$. Properties: $E[X^n] = M_X^{(n)}(0) = \frac{d^n}{ds^n} M_X(s) \Big|_{s=0}$. $M_{aX+b}(s) = e^{bs} M_X(as)$. If $X$ and $Y$ are independent, $M_{X+Y}(s) = M_X(s) M_Y(s)$.