Gaussian Algebra Toolkit
Two-page extract of UP.A §2 · artifact gaussian-algebra-toolkit
This is the artifact gaussian-algebra-toolkit: the boxed formulas of UP.A §2 and nothing else. No derivations are repeated here. Every result below is proved in the primer, and each row of the last table says where.
The page is laid out to print onto two sides of paper. Use your browser’s print command; the print stylesheet removes the site chrome and squares the boxes. Later sessions — U3.T1, and the detours D1 and D2 — consume this sheet by name.
Throughout: \(x, \mu \in \mathbb{R}^d\). Law before density. A Gaussian law \(\mathcal{N}(\mu, \Sigma)\) is the law of \(\mu + Lz\) with \(z \sim \mathcal{N}(0, I_k)\) and \(\Sigma = LL^\top \succeq 0\); it may be degenerate. The density formula (item 1), every inverse and every log-determinant below require \(\Sigma\) symmetric positive definite, and that is assumed wherever they appear. A singular \(\Sigma\) gives a law with no density on \(\mathbb{R}^d\) — item 5 at \(\sigma = 0\) is the case that matters. \(\mathcal{N}(0, I_d)\) is the source distribution \(p_0\) of the course, and its samples are written \(x_0\).
1 · The density
\[ \mathcal{N}(x; \mu, \Sigma) = \frac{1}{(2\pi)^{d/2} (\det \Sigma)^{1/2}} \exp\!\left( -\frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x - \mu) \right) \]
2 · Standardization
\[ z \sim \mathcal{N}(0, I_d), \quad \Sigma^{1/2}(\Sigma^{1/2})^\top = \Sigma \qquad\Longrightarrow\qquad \mu + \Sigma^{1/2} z \;\sim\; \mathcal{N}(\mu, \Sigma) \]
3 · Linear transforms
\[ x \sim \mathcal{N}(\mu, \Sigma) \qquad\Longrightarrow\qquad Ax + b \;\sim\; \mathcal{N}\big(A\mu + b,\; A\Sigma A^\top\big) \]
Corollary, used constantly: a sub-vector of a jointly Gaussian vector is Gaussian, with mean and covariance read off the corresponding blocks.
4 · Sums of independent Gaussians
\[ \begin{aligned} x \perp y, \quad x \sim \mathcal{N}(\mu_x, \Sigma_x), \quad y \sim \mathcal{N}(\mu_y, \Sigma_y) \\ \Longrightarrow\quad x + y \sim \mathcal{N}(\mu_x + \mu_y,\; \Sigma_x + \Sigma_y) \end{aligned} \]
Independence is the hypothesis. Without it the cross-covariances survive, and the sum’s covariance is \(\Sigma_x + \Sigma_y + \operatorname{Cov}(x, y) + \operatorname{Cov}(y, x)\).
5 · The interpolation form
\[ \begin{aligned} x_0 \sim \mathcal{N}(0, I_d), \quad x_0 \perp x_1, \quad \alpha, \sigma \in \mathbb{R},\ \sigma \ge 0 \\ \Longrightarrow\quad \alpha x_1 + \sigma x_0 \;\big|\; x_1 \;\sim\; \mathcal{N}\big(\alpha x_1,\; \sigma^2 I_d\big) \\ \sigma = 0 \;\Longrightarrow\; \text{the degenerate law } \delta_{\alpha x_1} \text{ (a point mass, not a density)} \end{aligned} \]
For \(x_1\) fixed this is the unconditional law. For \(x_1 \sim q\) random and independent of \(x_0\) it is the law conditional on \(x_1\), and the unconditional law is a mixture over \(q\).
6 · Completing the square
\[ \begin{aligned} x^\top A x - 2 b^\top x &= (x - A^{-1}b)^\top A (x - A^{-1}b) - b^\top A^{-1} b \\ &\qquad\qquad (A \text{ symmetric, invertible}) \end{aligned} \]
How to use it. Collect an exponent into \(-\frac{1}{2}(x^\top A x - 2b^\top x)\) and read off: the law is Gaussian in \(x\) with covariance \(A^{-1}\) and mean \(A^{-1}b\) — this reading needs \(A \succ 0\). The leftover constant goes into the normalizer and is never computed.
7 · Joint Gaussian: marginals and conditionals
With \(\mu = \begin{bmatrix}\mu_a \\ \mu_b\end{bmatrix}\) and \(\Sigma = \begin{bmatrix}\Sigma_{aa} & \Sigma_{ab} \\ \Sigma_{ba} & \Sigma_{bb}\end{bmatrix}\), \(\Sigma_{ba} = \Sigma_{ab}^\top\), and \(\Sigma \succ 0\) (so \(\Sigma_{bb}\) is invertible; singular conditioning is outside this sheet):
\[ x_a \sim \mathcal{N}(\mu_a, \Sigma_{aa}) \qquad\text{(marginal: read off the blocks)} \]
\[ x_a \mid x_b \;\sim\; \mathcal{N}\Big( \mu_a + \Sigma_{ab}\Sigma_{bb}^{-1}(x_b - \mu_b), \;\; \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba} \Big) \]
Three properties to remember: the conditional mean is affine in \(x_b\); the conditional covariance does not depend on the value of \(x_b\); and the conditional covariance is never larger than \(\Sigma_{aa}\).
Schur complement / block inverse. With \(S = \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}\) and \(\Lambda = \Sigma^{-1}\):
\[ \Lambda_{aa} = S^{-1}, \qquad \Lambda_{ab} = -S^{-1}\Sigma_{ab}\Sigma_{bb}^{-1}, \qquad \Lambda_{aa}^{-1}\Lambda_{ab} = -\Sigma_{ab}\Sigma_{bb}^{-1} \]
8 · Scalar corollary
\((X, Y)\) jointly Gaussian with \(\sigma_X^2 > 0\), \(\sigma_Y^2 > 0\), and correlation \(\rho = \sigma_{XY}/(\sigma_X\sigma_Y)\):
\[ X \mid Y = y \;\sim\; \mathcal{N}\Big( \mu_X + \frac{\sigma_{XY}}{\sigma_Y^2}(y - \mu_Y), \;\; \sigma_X^2(1 - \rho^2) \Big) \]
9 · Noisy scalar observation
\(X \sim \mathcal{N}(\mu_0, \sigma_0^2)\) with \(\sigma_0^2 > 0\), \(Y = X + \sigma\varepsilon\) with \(\sigma > 0\) and \(\varepsilon \sim \mathcal{N}(0,1)\) independent of \(X\):
\[ \mathbb{E}\big[X \mid Y = y\big] = \frac{\sigma_0^2}{\sigma_0^2 + \sigma^2}\, y + \frac{\sigma^2}{\sigma_0^2 + \sigma^2}\, \mu_0, \qquad \operatorname{Var}\big(X \mid Y\big) = \frac{\sigma_0^2\sigma^2}{\sigma_0^2 + \sigma^2} \]
10 · KL between Gaussians
Both \(\Sigma_0, \Sigma_1 \succ 0\); if either law is degenerate, see the extended-value convention of UP.A §3.1.
\[ \begin{aligned} \mathrm{KL}\big(\mathcal{N}(\mu_0, \Sigma_0) \,\big\Vert\, \mathcal{N}(\mu_1, \Sigma_1)\big) = \frac{1}{2}\Big[ &\operatorname{tr}(\Sigma_1^{-1}\Sigma_0) \\ + &(\mu_1-\mu_0)^\top \Sigma_1^{-1}(\mu_1-\mu_0) \\ - &\; d + \log \tfrac{\det\Sigma_1}{\det\Sigma_0} \Big] \end{aligned} \]
\[ \mathrm{KL}\big(\mathcal{N}(\mu_0, \sigma_0^2) \,\big\Vert\, \mathcal{N}(\mu_1, \sigma_1^2)\big) = \log\frac{\sigma_1}{\sigma_0} + \frac{\sigma_0^2 + (\mu_0-\mu_1)^2}{2\sigma_1^2} - \frac{1}{2} \]
Where each item is proved, and where it is used
| Item | Proved in | Used in |
|---|---|---|
| 1–2 · density, standardization | UP.A §2.1 | everywhere |
| 3 · linear transforms | UP.A §2.2 | items 4, 5, 7; U3.T1 |
| 4 · sums | UP.A §2.3 | item 9; D1 |
| 5 · interpolation form | UP.A §2.3 | U3.T1 item 4; D1 schedules |
| 6 · completing the square | UP.A §2.4 | items 7, 8 |
| 7–8 · conditionals, marginals | UP.A §2.5 | item 9; U3.T1; D2 |
| 9 · noisy observation | UP.A §2.6 | D2; PS1.2 (UP.T1) |
| 10 · Gaussian KL | UP.A §3.4 | UP.T1; lab comparisons |