Gaussian Algebra Toolkit
Two-page extract of UP.A §2 · artifact gaussian-algebra-toolkit
This is the artifact gaussian-algebra-toolkit: the boxed formulas of UP.A §2 and nothing else. No derivations are repeated here. Every result below is proved in the primer, and each row of the last table says where.
The page is laid out to print onto two sides of paper. Use your browser’s print command; the print stylesheet removes the site chrome and squares the boxes. Later sessions — U3.T1, and the detours D1 and D2 — consume this sheet by name.
Throughout: \(x, \mu \in \mathbb{R}^d\), and \(\Sigma\) is symmetric positive definite. \(\mathcal{N}(0, I_d)\) is the source distribution \(p_0\) of the course, and its samples are written \(x_0\).
1 · The density
\[ \mathcal{N}(x; \mu, \Sigma) = \frac{1}{(2\pi)^{d/2} (\det \Sigma)^{1/2}} \exp\!\Big( -\tfrac12 (x-\mu)^\top \Sigma^{-1} (x - \mu) \Big) \]
2 · Standardization
\[ z \sim \mathcal{N}(0, I_d), \quad \Sigma^{1/2}(\Sigma^{1/2})^\top = \Sigma \qquad\Longrightarrow\qquad \mu + \Sigma^{1/2} z \;\sim\; \mathcal{N}(\mu, \Sigma) \]
3 · Linear transforms
\[ x \sim \mathcal{N}(\mu, \Sigma) \qquad\Longrightarrow\qquad Ax + b \;\sim\; \mathcal{N}\big(A\mu + b,\; A\Sigma A^\top\big) \]
Corollary, used constantly: a sub-vector of a jointly Gaussian vector is Gaussian, with mean and covariance read off the corresponding blocks.
4 · Sums of independent Gaussians
\[ \begin{aligned} x \perp y, \quad x \sim \mathcal{N}(\mu_x, \Sigma_x), \quad y \sim \mathcal{N}(\mu_y, \Sigma_y) \\ \Longrightarrow\quad x + y \sim \mathcal{N}(\mu_x + \mu_y,\; \Sigma_x + \Sigma_y) \end{aligned} \]
Independence is required. Without it the cross-covariances survive.
5 · The interpolation form
\[ \begin{aligned} x_0 \sim \mathcal{N}(0, I_d), \quad x_1 \text{ fixed}, \quad \alpha, \sigma \in \mathbb{R},\ \sigma > 0 \\ \Longrightarrow\quad \alpha x_1 + \sigma x_0 \;\sim\; \mathcal{N}\big(\alpha x_1,\; \sigma^2 I_d\big) \end{aligned} \]
If \(x_1 \sim q\) is random and independent of \(x_0\), this is the law conditional on \(x_1\), and the unconditional law is a mixture over \(q\).
6 · Completing the square
\[ \begin{aligned} x^\top A x - 2 b^\top x &= (x - A^{-1}b)^\top A (x - A^{-1}b) - b^\top A^{-1} b \\ &\qquad\qquad (A \text{ symmetric, invertible}) \end{aligned} \]
How to use it. Collect an exponent into \(-\tfrac12(x^\top A x - 2b^\top x)\) and read off: the law is Gaussian in \(x\) with covariance \(A^{-1}\) and mean \(A^{-1}b\). The leftover constant goes into the normalizer and is never computed.
7 · Joint Gaussian: marginals and conditionals
With \(\mu = \begin{bmatrix}\mu_a \\ \mu_b\end{bmatrix}\) and \(\Sigma = \begin{bmatrix}\Sigma_{aa} & \Sigma_{ab} \\ \Sigma_{ba} & \Sigma_{bb}\end{bmatrix}\), \(\Sigma_{ba} = \Sigma_{ab}^\top\):
\[ x_a \sim \mathcal{N}(\mu_a, \Sigma_{aa}) \qquad\text{(marginal: read off the blocks)} \]
\[ x_a \mid x_b \;\sim\; \mathcal{N}\Big( \mu_a + \Sigma_{ab}\Sigma_{bb}^{-1}(x_b - \mu_b), \;\; \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba} \Big) \]
Three properties to remember: the conditional mean is affine in \(x_b\); the conditional covariance does not depend on the value of \(x_b\); and the conditional covariance is never larger than \(\Sigma_{aa}\).
Schur complement / block inverse. With \(S = \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}\) and \(\Lambda = \Sigma^{-1}\):
\[ \Lambda_{aa} = S^{-1}, \qquad \Lambda_{ab} = -S^{-1}\Sigma_{ab}\Sigma_{bb}^{-1}, \qquad \Lambda_{aa}^{-1}\Lambda_{ab} = -\Sigma_{ab}\Sigma_{bb}^{-1} \]
8 · Scalar corollary
\((X, Y)\) jointly Gaussian, correlation \(\rho = \sigma_{XY}/(\sigma_X\sigma_Y)\):
\[ X \mid Y = y \;\sim\; \mathcal{N}\Big( \mu_X + \frac{\sigma_{XY}}{\sigma_Y^2}(y - \mu_Y), \;\; \sigma_X^2(1 - \rho^2) \Big) \]
9 · Noisy scalar observation
\(X \sim \mathcal{N}(\mu_0, \sigma_0^2)\), \(Y = X + \sigma\varepsilon\) with \(\varepsilon \sim \mathcal{N}(0,1)\) independent of \(X\):
\[ \mathbb{E}\big[X \mid Y = y\big] = \frac{\sigma_0^2}{\sigma_0^2 + \sigma^2}\, y + \frac{\sigma^2}{\sigma_0^2 + \sigma^2}\, \mu_0, \qquad \operatorname{Var}\big(X \mid Y\big) = \frac{\sigma_0^2\sigma^2}{\sigma_0^2 + \sigma^2} \]
10 · KL between Gaussians
\[ \begin{aligned} \mathrm{KL}\big(\mathcal{N}(\mu_0, \Sigma_0) \,\big\Vert\, \mathcal{N}(\mu_1, \Sigma_1)\big) = \tfrac12\Big[ &\operatorname{tr}(\Sigma_1^{-1}\Sigma_0) \\ + &(\mu_1-\mu_0)^\top \Sigma_1^{-1}(\mu_1-\mu_0) \\ - &\; d + \log \tfrac{\det\Sigma_1}{\det\Sigma_0} \Big] \end{aligned} \]
\[ \mathrm{KL}\big(\mathcal{N}(\mu_0, \sigma_0^2) \,\big\Vert\, \mathcal{N}(\mu_1, \sigma_1^2)\big) = \log\frac{\sigma_1}{\sigma_0} + \frac{\sigma_0^2 + (\mu_0-\mu_1)^2}{2\sigma_1^2} - \frac12 \]
Where each item is proved, and where it is used
| Item | Proved in | Used in |
|---|---|---|
| 1–2 · density, standardization | UP.A §2.1 | everywhere |
| 3 · linear transforms | UP.A §2.2 | items 4, 5, 7; U3.T1 |
| 4 · sums | UP.A §2.3 | item 9; D1 |
| 5 · interpolation form | UP.A §2.3 | U3.T1 item 4; D1 schedules |
| 6 · completing the square | UP.A §2.4 | items 7, 8 |
| 7–8 · conditionals, marginals | UP.A §2.5 | item 9; U3.T1; D2 |
| 9 · noisy observation | UP.A §2.6 | D2; PS1.2 (UP.T1) |
| 10 · Gaussian KL | UP.A §3.4 | UP.T1; lab comparisons |