UP.T1 — The Lemma of the Course
Session UP.T1 · date: see calendar-map
How to read this page. This is the first live session of the course. Its central result, the projection Lemma, is proved here in full and will be reused in several later sessions.
These notes are written to be read on their own. The live session proves the same main results on the board; here the derivations are expanded, with the steps and technical assumptions needed later in the course made explicit, and the two ingredients of the proof’s orthogonality calculation are proved before they are used.
The page assumes the async primer, and nothing else. If you have read UP.A, you have everything. In particular you do not need to have attended the bootcamp sessions.
Two books cover this material more slowly, and both are optional: (Bishop and Bishop 2024) chapter 2, and (Murphy 2023) chapters 2 and 5. Each section below names the chapter that matches it.
- The result of Section 2 is cited by name for the rest of the course: the projection Lemma (UP.T1). It is never called “the orthogonality result” or “that thing from the probability session”. One name, one result.
- Every expectation in every loss carries explicit sampling subscripts, including the law of time whenever time is random (\(t \sim \mathrm{Unif}[0,1]\) by default). UP.A §5 drilled this; from here it is enforced.
- Conditional expectations are identified up to \(\mathbb{P}\)-almost-sure equality. If \(\mathbb{E}[X \mid Y] = m(Y)\), the representing regression function \(m\) is identified up to \(p_Y\)-almost-everywhere equality. Whenever we write a pointwise value \(m(y)\), a version of \(m\) has been chosen. This is said once, in Section 1.1, and after that ordinary function notation is used.
§1 — Conditional expectation, done properly
Reference for this section: (Bishop and Bishop 2024) ch. 2, or (Murphy 2023) ch. 2.
The primer stated the law of total expectation and told you not to cite it as proved. This section proves it. It also proves the pull-out property, the other ingredient of the projection Lemma’s orthogonality calculation, and it separates three distinct objects that appear as soon as a conditional mean is written down:
- the deterministic value \(m(y)\) at a fixed \(y\);
- the deterministic regression function \(m \colon y \mapsto m(y)\);
- the random variable \(m(Y) = \mathbb{E}[X \mid Y]\).
The treatment is measure-light. When densities exist, the computations are written with density formulas, and every integral is assumed to converge. The definition, however, is stated in its density-free form, and that form is the authoritative one: UP.A already met a conditional law with no joint Lebesgue density (a label and a feature), and this page must not pretend otherwise. Section 1.4 says what the density calculations are suppressing.
The value, the function, and the random variable
Let \((X, Y)\) have joint law \(p_{XY}\), with marginals \(p_X\) and \(p_Y\). Assume that \(X\) takes values in \(\mathbb{R}^{d}\) and is integrable: \[ \mathbb{E}_{x \sim p_X}\big[\Vert x\Vert\big] \;<\; \infty . \tag{1}\] Throughout this course, \(Y\) takes values in a Euclidean space, a discrete space, or another standard Borel space, so a conditional law \(p(\cdot \mid y)\) and a measurable regression-function representation are available.
When the conditional law of \(X\) given \(Y = y\) admits a density \(p(x \mid y)\), define, for \(p_Y\)-almost every \(y\), \[ \begin{aligned} m(y) &\;:=\; \mathbb{E}\big[X \mid Y = y\big] \\[2pt] &\;=\; \mathbb{E}_{x \sim p(\cdot \mid y)}[x] \\[2pt] &\;=\; \int x\, p(x \mid y)\, \mathrm{d}x \;\in\; \mathbb{R}^{d} . \end{aligned} \tag{2}\]
At a fixed \(y\) for which this version is defined, \(m(y)\) is a deterministic vector. You chose \(y\), so nothing in it is random.
Now let \(y\) range. The map \[ m \colon y \;\longmapsto\; m(y) \tag{3}\] is a function: deterministic, one output vector per input value. It is determined only \(p_Y\)-almost everywhere — its values on a \(p_Y\)-null set may be changed without changing the conditional expectation it represents (house rule 3). It is still not random.
\[ \mathbb{E}\big[X \mid Y\big] \;=\; m(Y) \qquad \mathbb{P}\text{-almost surely}, \tag{4}\] the function of Equation 3 evaluated at the random variable \(Y\).
Equation 2 is a deterministic value at a fixed input. Equation 3 is a deterministic function. Equation 4 is a random variable, because the deterministic function \(m\) is evaluated at the random input \(Y\) — and that is the only reason it is random.
The definitions Equation 2 and Equation 4 used a conditional density. The object exists without one, and the following statement characterizes it in general — the one a measure-theoretic course would start from, and the one in force whenever a density is missing.
Let \(m\) be a measurable function such that \(m(Y)\) is integrable. Then \(m(Y)\) is a version of \(\mathbb{E}[X \mid Y]\) exactly when, for every bounded measurable scalar function \(h\), \[ \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, h(y)\, m(y) \,\big] \;=\; \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, h(y)\, x \,\big] . \tag{5}\] For vector-valued \(X\), read Equation 5 componentwise.
The random variable \(m(Y)\) is unique up to \(\mathbb{P}\)-almost-sure equality, and the representing function \(m\) is unique up to \(p_Y\)-almost-everywhere equality. A version exists whenever \(X\) is integrable; the assumption on the space of \(Y\) is what guarantees a conditional law \(p(\cdot \mid y)\) as well, so that Equation 2 makes sense. When a conditional density exists, Equation 2 provides such a version for \(p_Y\)-almost every \(y\): substitute it into the left side of Equation 5 and integrate. So the density formulas are the special case, and they are the case every computation on this page runs in. What Equation 5 adds is coverage: a discrete \(Y\) with a continuous \(X\), the mixed example of UP.A §1, a \(Y\) with atoms — all have a conditional expectation, and none has a joint Lebesgue density. The tower rule below is Equation 5 with \(h \equiv 1\).
The three panels show the three objects. What they cannot show is the order, and the order is the content: each step changes what is fixed and what is free. The animation below puts that order back. Watch what is random at each moment, because for most of the scene the answer is nothing:
In this construction, \(m\) is deterministic — fixed by the joint law \(p_{XY}\) before anything is drawn — and randomness appears when it is evaluated at the random input \(Y\). The closing frame carries \[ \mathbb{E}_{y \sim \mathrm{Unif}\{1,2,3\}}\big[m(y)\big] \;=\; \frac{14}{3} , \] which is the tower rule of Section 1.2 on this example.
This distinction prevents a common source of confusion later in the course: a pointwise conditional mean, the regression function, and the random variable obtained by evaluating that function at \(Y\) are related but are not the same mathematical object. When you read \(\mathbb{E}[X \mid Y]\) somewhere, ask which panel of Figure 1 it is. If it is averaged under an outer expectation, as in \(\mathbb{E}_{y \sim p_Y}[m(y)]\), the object being averaged is panel 3 — the random variable \(m(Y)\), written in the course’s sampling notation. If it is being differentiated with respect to \(y\), it is panel 2. If it is equal to a specific number, it is panel 1.
The tower rule
Let \((X, Y)\) have joint law \(p_{XY}\) and assume \(\mathbb{E}_{x \sim p_X}\big[\Vert x\Vert\big] < \infty\). If \(m\) is a version of the conditional mean of Section 1.1, then \[ \mathbb{E}_{y \sim p_Y}\big[\, m(y) \,\big] \;=\; \mathbb{E}_{x \sim p_X}\big[\, x \,\big] . \tag{6}\]
Proof (with densities). Assume a joint density \(p_{XY}(x, y)\) and a conditional-density version \(p(x \mid y)\), so that \(p_{XY}(x, y) = p(x \mid y)\, p_Y(y)\) for almost every \((x, y)\). Write the left side as an integral against the marginal \(p_Y\) and substitute Equation 2: \[ \begin{aligned} \mathbb{E}_{y \sim p_Y}\big[m(y)\big] &\;=\; \int \Big( \int x\, p(x \mid y)\, \mathrm{d}x \Big)\, p_Y(y)\, \mathrm{d}y \\[2pt] &\;=\; \int\!\!\int x\, p_{XY}(x, y)\, \mathrm{d}x\, \mathrm{d}y \\[2pt] &\;=\; \int x \Big( \int p_{XY}(x, y)\, \mathrm{d}y \Big)\, \mathrm{d}x \\[2pt] &\;=\; \int x\, p_X(x)\, \mathrm{d}x \;=\; \mathbb{E}_{x \sim p_X}[x] . \end{aligned} \] The second line moves \(p_Y(y)\) inside the inner integral and uses \(p(x \mid y)\, p_Y(y) = p_{XY}(x, y)\) — an identity that holds only almost everywhere, which is all an integral needs. The third line exchanges the order of integration: the \(y\)-integration moves inside. That exchange is Fubini’s theorem, applied componentwise for vector-valued \(X\), and the assumption \(\mathbb{E}_{x \sim p_X}[\Vert x\Vert] < \infty\) supplies the absolute integrability it requires. The fourth line integrates \(y\) out, which leaves the marginal density \(p_X\) under the integral. \(\;\blacksquare\)
The proof is four lines and one exchange of integration order. That exchange is the only place where an assumption is doing work, and Section 1.4 records which one. Without densities the tower rule is shorter still: it is Equation 5 with \(h \equiv 1\).
The pull-out property
This is the second ingredient of the Lemma’s orthogonality calculation, and it is the one that makes a cross term vanish.
Let \(X\) be scalar and integrable, let \(g\) be a measurable function of \(y\) alone, and assume that \(g(Y)\, X\) is also integrable. Then, \(\mathbb{P}\)-almost surely, \[ \mathbb{E}\big[\, g(Y)\, X \mid Y \,\big] \;=\; g(Y)\, \mathbb{E}\big[X \mid Y\big] . \tag{7}\]
Proof (with densities). For \(p_Y\)-almost every \(y\), the factor \(g(y)\) is a constant inside the conditional expectation given \(Y = y\), so it comes out: \[ \begin{aligned} \mathbb{E}\big[g(Y)\, X \mid Y = y\big] &\;=\; \mathbb{E}_{x \sim p(\cdot \mid y)}\big[g(y)\, x\big] \\[2pt] &\;=\; g(y)\, \mathbb{E}_{x \sim p(\cdot \mid y)}[x] \\[2pt] &\;=\; g(y)\, m(y) . \end{aligned} \] That is an identity between two functions of \(y\), valid \(p_Y\)-almost everywhere. Evaluate both at \(Y\): \[ \mathbb{E}\big[g(Y)\, X \mid Y\big] \;=\; g(Y)\, m(Y) \;=\; g(Y)\, \mathbb{E}\big[X \mid Y\big] \] almost surely. \(\;\blacksquare\)
The sentence to remember is: once you condition on \(Y\), a function of \(Y\) behaves like a constant — provided the function is measurable and the products involved are integrable, as in the statement.
The Lemma uses a vector form of Equation 7, and it uses it inside \(L^{2}\), where the integrability comes for free. Let \(a(Y)\) and \(Z\) be square-integrable random vectors of the same dimension. Then the Cauchy–Schwarz inequality guarantees that \(\langle a(Y), Z\rangle\) is integrable, and \[ \mathbb{E}\big[\, \langle a(Y),\, Z \rangle \mid Y \,\big] \;=\; \big\langle a(Y),\, \mathbb{E}[Z \mid Y] \big\rangle \qquad \mathbb{P}\text{-almost surely} . \tag{8}\] Expand the inner product as the sum \(\sum_i a_i(Y)\, Z_i\); each term is integrable by the same inequality, so Equation 7 applies to it; then use linearity. This is exactly the form Section 2 needs.
Conditional expectation is linear in the conditioned variable: for integrable random vectors \(X\) and \(Z\) and constants \(a, b\), the identity \[ \mathbb{E}[aX + bZ \mid Y] \;=\; a\,\mathbb{E}[X \mid Y] + b\,\mathbb{E}[Z \mid Y] \] holds \(\mathbb{P}\)-almost surely. With densities, the proof is the linearity of the integral in Equation 2 at each fixed \(y\); without them, the sum of two versions satisfies Equation 5 for the target \(aX + bZ\). The Lemma uses this without comment.
What the density calculations are suppressing
The computations above are written with densities and read as if every object were defined pointwise. Four things are being suppressed, and each is stated here once.
- Versions and null sets. Conditional expectations are equivalence classes. The random variable \(\mathbb{E}[X \mid Y]\) is defined up to \(\mathbb{P}\)-almost-sure equality, and a representing function \(m(y)\) is defined up to \(p_Y\)-almost-everywhere equality. Density formulas for \(m(y)\) therefore need only hold for \(p_Y\)-almost every \(y\) — in particular, nothing is lost where \(p_Y(y) = 0\).
- Fubini. In the density proof of the tower rule, the exchange of integration order is justified by Fubini’s theorem. The assumption \(\mathbb{E}_{x \sim p_X}[\Vert x\Vert] < \infty\) is sufficient, componentwise, for the calculation used here.
- Density-free conditional expectation. A joint Lebesgue density is not required for conditional expectation. Mixed discrete–continuous laws and empirical distributions are examples in which density-based notation may not apply globally, while Equation 5 still does. Under Equation 5 the tower rule is immediate: take \(h \equiv 1\).
- The general pull-out property. For bounded measurable \(g\), the pull-out property follows directly from the defining characterization: \(g(Y)\, m(Y)\) satisfies Equation 5 for the target \(g(Y)\, X\), because \(h(y)\, g(y)\) is again bounded and measurable. For unbounded \(g\), the same result holds under the integrability assumptions of Equation 7; a fully measure-theoretic proof uses truncation and a limit rather than the pointwise density calculation.
The random elements used in this course live in standard Borel spaces such as \(\mathbb{R}^{d}\) or discrete spaces. When suitable densities exist, we use them for calculations. When they do not, the density-free characterization of conditional expectation, Equation 5, is the one in force.
- Section 2: the tower rule and Equation 8 are the two ingredients of the orthogonality calculation in the proof of the projection Lemma.
- U3.T1 builds a marginal object by averaging conditional ones, which is Equation 6 with the conditioning variable renamed.
- Figure 1 is the reading aid for every conditional expectation in U3.
§2 — The projection Lemma
Here is the question. You observe \(Y\), and you want to guess \(X\). You must commit to a rule: a function \(f\) that turns each observed value into a guess. You will be scored by the mean squared error of your guess. Which \(f\) should you choose?
The answer is short, and it is the reason this session exists. In §1 we called the conditional-mean function \[ m(y) \;=\; \mathbb{E}\big[X \mid Y = y\big] . \] This section shows that \(m\) is the optimal square-loss predictor. Once that optimization role has been established, we will also write \[ f^{*} \;:=\; m , \] the star marking the role. It is not a new object.
The space the question lives in
For a random vector \(Z\) in \(\mathbb{R}^{d}\) with \(\mathbb{E}\Vert Z\Vert^{2} < \infty\), write \[ \Vert Z \Vert_{L^{2}} \;=\; \big(\, \mathbb{E}\Vert Z\Vert^{2} \,\big)^{1/2} . \tag{9}\]
As usual in \(L^{2}\), random variables that agree almost surely are identified. With that convention, Equation 9 defines a norm, and \[ d_{L^{2}}(Z, W) \;=\; \Vert Z - W \Vert_{L^{2}} \] is the corresponding distance. That distance is the point of introducing the norm: it makes the result below a statement about a right triangle, and lets you add lengths the way you learned to in school. Write \[ \mathcal{F}_Y \;=\; \Big\{\, f(Y) \;:\; f \text{ measurable},\; \mathbb{E}\Vert f(Y)\Vert^{2} < \infty \,\Big\} . \tag{10}\] Thus \(\mathcal{F}_Y\) contains the square-integrable prediction rules based only on the observed variable \(Y\). It is a closed linear subspace of the ambient \(L^{2}\) space. Measurability is the regularity condition that ensures \(f(Y)\) is a valid random variable; all prediction functions used in this course satisfy it. The target \(X\) is generally not a member of \(\mathcal{F}_Y\), because \(X\) is not a function of \(Y\) — if it were, you would not be guessing.
The Lemma
Let \((X, Y)\) have joint law \(p_{XY}\), with marginals \(p_X\) and \(p_Y\), and assume \[ \mathbb{E}_{x \sim p_X}\big[\Vert x\Vert^{2}\big] \;<\; \infty . \] Let \(m(y) = \mathbb{E}[X \mid Y = y]\) denote the version fixed in Section 1.1, defined \(p_Y\)-almost everywhere, and set \(f^{*} := m\).
For every \(f(Y) \in \mathcal{F}_Y\), \[ \begin{aligned} &\mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert f(y) - x \Vert^{2} \,\big] \\[2pt] &\qquad\;=\; \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert m(y) - x \Vert^{2} \,\big] \\[2pt] &\qquad\qquad+\; \mathbb{E}_{y \sim p_Y}\big[\, \Vert f(y) - m(y) \Vert^{2} \,\big] . \end{aligned} \tag{11}\]
Consequently, \(m\) — equivalently \(f^{*}\) — minimizes the population squared risk \(\mathbb{E}_{(x, y) \sim p_{XY}}[\Vert f(y) - x\Vert^{2}]\) over \(\mathcal{F}_Y\). The minimizing random variable \(m(Y)\) is unique \(\mathbb{P}\)-almost surely; equivalently, the representing predictor \(m(y)\) is unique \(p_Y\)-almost everywhere.
The minimization consequence reads straight off the identity: the first term on the right of Equation 11 does not involve \(f\), and the second is non-negative and equals zero exactly when \(f(y) = m(y)\) for \(p_Y\)-almost every \(y\) — that is, when \(f(Y) = m(Y)\) almost surely. So the whole content is Equation 11, and Equation 11 is one expansion and one vanishing cross term.
Proof.
Step 0 — the terms are finite. Because \(X \in L^{2}\), conditional Jensen gives \[ \big\Vert \mathbb{E}[X \mid Y] \big\Vert^{2} \;\le\; \mathbb{E}\big[\Vert X\Vert^{2} \mid Y\big] \qquad \mathbb{P}\text{-almost surely} . \] Taking expectations and using the tower rule, \[ \mathbb{E}\Vert m(Y)\Vert^{2} \;\le\; \mathbb{E}\Vert X\Vert^{2} \;<\; \infty . \] Hence \(m(Y) \in \mathcal{F}_Y\). (Componentwise non-negativity of the conditional variance gives the same conclusion.) Since both \(f(Y)\) and \(m(Y)\) are in \(L^{2}\), all quadratic terms below are finite, and the cross term is integrable by the Cauchy–Schwarz inequality.
Step 1 — add and subtract. Let \(a(y) := f(y) - m(y)\). Insert \(m(y)\) inside the norm and expand: \[ \begin{aligned} &\mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert f(y) - x \Vert^{2} \,\big] \\[2pt] &\qquad\;=\; \mathbb{E}_{y \sim p_Y}\big[\, \Vert f(y) - m(y) \Vert^{2} \,\big] \\[2pt] &\qquad\qquad+\; \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert m(y) - x \Vert^{2} \,\big] \;+\; 2\, C , \end{aligned} \] where \[ C \;=\; \mathbb{E}_{(x, y) \sim p_{XY}}\Big[\, \big\langle\, a(y),\; m(y) - x \,\big\rangle \,\Big] . \] The first term on the right depends on \(y\) alone, so its expectation is under the marginal \(p_Y\).
Step 2 — the cross term is zero. Write the joint expectation as a nested one — the tower rule of Section 1.2 — and apply the pull-out property inside: \[ \begin{aligned} C &\;=\; \mathbb{E}_{y \sim p_Y}\Big[\, \mathbb{E}_{x \sim p(\cdot \mid y)}\big[\, \langle a(y),\, m(y) - x \rangle \,\big] \,\Big] \\[2pt] &\;=\; \mathbb{E}_{y \sim p_Y}\Big[\, \big\langle a(y),\; m(y) - \mathbb{E}_{x \sim p(\cdot \mid y)}[x] \big\rangle \,\Big] \\[2pt] &\;=\; \mathbb{E}_{y \sim p_Y}\big[\, \langle a(y),\, 0 \rangle \,\big] \\[2pt] &\;=\; 0 . \end{aligned} \] The second equality is the pull-out property, Equation 8: once \(y\) is fixed, both \(a(y)\) and \(m(y)\) are fixed with respect to the conditional draw \(x \sim p(\cdot \mid y)\), so the inner product passes through the inner expectation. The third equality is the definition \(m(y) = \mathbb{E}_{x \sim p(\cdot \mid y)}[x]\), Equation 2, valid for \(p_Y\)-almost every \(y\). In random-variable notation the same argument reads \[ \begin{aligned} \mathbb{E}\big[\, \langle a(Y),\, m(Y) - X \rangle \mid Y \,\big] &\;=\; \big\langle a(Y),\; m(Y) - \mathbb{E}[X \mid Y] \big\rangle \\[2pt] &\;=\; 0 \end{aligned} \] almost surely, by Equation 8 and Equation 4.
Step 3 — read off the claims. With \(C = 0\), step 1 is exactly Equation 11. The minimization consequence follows as noted above. \(\;\blacksquare\)
Figure 2 draws one competitor \(f(Y)\). The Lemma is a statement about every one of them, and what it asserts is an invariance: as \(f(Y)\) moves anywhere in the plane, the angle at \(f^{*}(Y)\) stays right and the vertical edge does not change length. An invariance under motion is not something a still can carry, so the animation below moves \(f(Y)\) along a closed loop and measures all three edges as it goes:
Two things in it are worth naming. The loop passes through \(f(Y) = f^{*}(Y)\), and there the horizontal edge collapses to zero while the hypotenuse falls to exactly the vertical edge — that is the minimizer, and it is the only place the error is the floor and nothing else.
And the three bars are measured, not drawn. They are \(L^{2}\) norms computed by Monte Carlo on the pair of PS1.2 below (\(X \sim \mathcal{N}(0,1)\), \(Y = X + 0.7\,\varepsilon\), \(n = 400\,000\)), and the scene is only allowed to render if the empirical cross term stays inside four standard errors of zero at every one of the \(241\) loop positions. The maximum standardized cross-term deviation is \(0.71\) standard errors. Under a deliberately incorrect projection, put through the identical test, the same diagnostic reaches about \(65\) standard errors.
Read Figure 2 as linear algebra. After identifying random variables that agree almost surely, the ambient \(L^{2}\) space is a Hilbert space with inner product \[ \langle Z, W \rangle_{L^{2}} \;=\; \mathbb{E}\big[\langle Z, W\rangle\big] , \] and \(\Vert \cdot \Vert_{L^{2}}\) of Equation 9 is its norm. The set \(\mathcal{F}_Y\) is a closed linear subspace: sums and scalar multiples of square-integrable functions of \(Y\) are again square-integrable functions of \(Y\). Step 2 shows that \(X - m(Y)\) is orthogonal to every element of that subspace — for an arbitrary \(g(Y) \in \mathcal{F}_Y\), choose \(f(Y) = m(Y) + g(Y)\) in the cross-term calculation, so that \(a(Y) = g(Y)\). Therefore \(m(Y)\) is the orthogonal projection of \(X\) onto \(\mathcal{F}_Y\), and Pythagoras is what orthogonal projections always give you. (The drawing is schematic: \(\mathcal{F}_Y\) is in general infinite-dimensional, not a two-dimensional plane.)
The corollary that gets used
The Lemma is the theorem. The corollary below is the form later sessions actually apply, and it is worth stating separately for that reason alone.
Fix a parametric family \(\{f_\theta\}\) — in practice a neural network with parameters \(\theta\) — with \(f_\theta(Y) \in \mathcal{F}_Y\) for every \(\theta\). Everything below is about population risks: expectations under the joint law, not averages over a dataset. The distinction is the subject of the caution after the proof.
(a) Constant offset. Define the two population objectives \[ \begin{aligned} \mathcal{L}_{\mathrm{target}}(\theta) &\;=\; \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert f_\theta(y) - x \Vert^{2} \,\big] , \\[2pt] \mathcal{L}_{\mathrm{proj}}(\theta) &\;=\; \mathbb{E}_{y \sim p_Y}\big[\, \Vert f_\theta(y) - m(y) \Vert^{2} \,\big] , \end{aligned} \] and the irreducible constant \[ C_X \;:=\; \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert x - m(y) \Vert^{2} \,\big] . \] Then, exactly, \[ \mathcal{L}_{\mathrm{target}}(\theta) \;=\; \mathcal{L}_{\mathrm{proj}}(\theta) \;+\; C_X . \tag{12}\] Therefore the two objectives have the same minimizer set over the chosen parametric family, and \[ \nabla_\theta \mathcal{L}_{\mathrm{target}}(\theta) \;=\; \nabla_\theta \mathcal{L}_{\mathrm{proj}}(\theta) \] wherever these gradients exist.
(b) Surrogate target. Let \((X, Y, Z)\) be jointly distributed, with \(X\) and \(Z\) square-integrable. Write \(p_{XY}\) and \(p_{ZY}\) for the pairwise marginal laws and \(p_Y\) for the common marginal law of \(Y\). Assume that, \(\mathbb{P}\)-almost surely, \[ \mathbb{E}\big[Z \mid Y\big] \;=\; \mathbb{E}\big[X \mid Y\big] \;=\; m(Y) ; \tag{13}\] equivalently, for the chosen versions, \(\mathbb{E}[Z \mid Y = y] = \mathbb{E}[X \mid Y = y] = m(y)\) for \(p_Y\)-almost every \(y\). Define \[ \begin{aligned} \mathcal{L}_{\mathrm{sur}}(\theta) &\;=\; \mathbb{E}_{(z, y) \sim p_{ZY}}\big[\, \Vert f_\theta(y) - z \Vert^{2} \,\big] , \\[2pt] C_Z &\;:=\; \mathbb{E}_{(z, y) \sim p_{ZY}}\big[\, \Vert z - m(y) \Vert^{2} \,\big] . \end{aligned} \] Then \(\mathcal{L}_{\mathrm{sur}}(\theta) = \mathcal{L}_{\mathrm{proj}}(\theta) + C_Z\), and therefore \[ \mathcal{L}_{\mathrm{sur}}(\theta) \;=\; \mathcal{L}_{\mathrm{target}}(\theta) \;+\; \big( C_Z - C_X \big) . \tag{14}\] The target and surrogate population risks therefore have exactly the same minimizer set over the chosen model family and identical parameter gradients wherever those gradients exist.
Proof.
(a) is Equation 11 read with \(f = f_\theta\): \(\mathcal{L}_{\mathrm{target}}(\theta) = C_X + \mathcal{L}_{\mathrm{proj}}(\theta)\), and \(C_X\) contains no \(\theta\).
(b) Apply the projection Lemma separately to the pairs \((X, Y)\) and \((Z, Y)\): \[ \begin{aligned} \mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \Vert f_\theta(y) - x \Vert^{2} \,\big] &\;=\; C_X + \mathcal{L}_{\mathrm{proj}}(\theta) , \\[2pt] \mathbb{E}_{(z, y) \sim p_{ZY}}\big[\, \Vert f_\theta(y) - z \Vert^{2} \,\big] &\;=\; C_Z + \mathcal{L}_{\mathrm{proj}}(\theta) . \end{aligned} \] The two decompositions share the same \(\mathcal{L}_{\mathrm{proj}}\) because Equation 13 gives the same conditional-mean function \(m\) for both targets. Subtract: the \(\theta\)-dependent term cancels, and what is left is the constant \(C_Z - C_X\). \(\;\blacksquare\)
Same conditional mean.
Part (b) says you may throw away the regression target you cannot compute, and regress against any surrogate that has the same conditional mean. The two population risks take different values. Their minimizer sets are the same, and their gradients agree wherever the population-risk gradients exist. Three sentences of proof, and it is the engine of half this course.
The corollary is an identity between population risks. On a finite dataset, the two empirical losses do not in general differ by a constant: their difference depends on which samples were drawn, so their empirical minimizers, and their minibatch gradients, need not coincide.
Under suitable statistical consistency and optimization assumptions, empirical minimizers may approach the common population minimizer set of the two objectives within the chosen model family. If the model class is realizable, so that it contains \(m = f^{*}\), that common population optimum is the conditional-mean predictor. Without realizability, the common optimum is the best approximation available inside the model class, not necessarily \(f^{*}\) itself.
The constants \[ C_X \;=\; \mathbb{E}\big[\Vert X - m(Y)\Vert^{2}\big] , \qquad C_Z \;=\; \mathbb{E}\big[\Vert Z - m(Y)\Vert^{2}\big] \] measure the average residual target variability around the shared conditional mean; for vector targets they equal the expected trace of the corresponding conditional covariance matrices. A larger \(C_Z\) means that the surrogate target has greater average conditional noise. This can increase finite-sample estimation variance, although the exact effect depends on the model class and the estimator. PS1.2(c) studies this effect in the specific setup used there.
What \(X\), \(Y\) and \(Z\) will be
Nothing yet. That is deliberate.
The projection Lemma concerns a square-integrable target \(X\) and an observed random element \(Y\). The surrogate-target corollary introduces a second square-integrable target \(Z\). Their concrete interpretations will change later in the course. Learn the result first in this abstract form. Later sessions will instantiate \(X\), \(Y\), and \(Z\) differently while using exactly the same argument.
The next section is a slide, and the slide answers exactly this question.
§3 — The announcement
This section is one slide. It is reproduced in the notes because it is shown again, twice, in later sessions.
This Lemma will be used exactly three times.
- U3.T1 — after the continuity-equation argument identifies the marginal velocity field as a conditional expectation of conditional velocities, to recognize that field as the \(L^{2}\)-optimal regression target given the current state.
- U3.T2 — to prove the CFM theorem: swapping the unknown marginal target for a conditional surrogate changes the population objective only by a \(\theta\)-independent constant and therefore leaves the \(\theta\)-gradients unchanged wherever those gradients exist (corollary (b)).
- U3.T2, detour D2 — to derive denoising score matching: same projection argument, different random variables.
You are not supposed to understand items 1 to 3 yet. Every technical term in them belongs to a session that has not happened. You are supposed to recognize them when they arrive, and to know, when they do, that the argument is one you already have.
The point of announcing these applications now is simply to make their common structure recognizable when it reappears. The technical meaning of each application will be developed in the corresponding session.
§4 — Pushforwards, and what a generative model is
Reference for this section: (Murphy 2023) ch. 5, which covers the divergence and states the equivalence of Section 4.3 in its own notation; and (Lipman et al. 2024) §3.3, which is where the symbol \(\psi_{\#}\) of Section 4.1 comes from. Read that section with one caveat: the Guide states the push-forward for diffeomorphisms, so that it can also give the density formula. Equation 16 below states it for any measurable map, which is weaker and is all this session needs.
The rest of the session builds the second thing you need, which is the vocabulary to say precisely what the course is trying to do.
Pushforward
Let \(p\) be a probability law on \(\mathbb{R}^{d}\) and let \(\psi \colon \mathbb{R}^{d} \to \mathbb{R}^{d}\) be measurable. The pushforward \(\psi_{\#}\, p\) is the probability law given, for every measurable set \(A\), by \[ (\psi_{\#}\, p)(A) \;=\; p\big(\psi^{-1}(A)\big) , \tag{15}\] that is, the law of \(\psi(x)\) when \(x \sim p\). Equivalently, it is the unique law for which \[ \mathbb{E}_{y \sim \psi_{\#} p}\big[\, g(y) \,\big] \;=\; \mathbb{E}_{x \sim p}\big[\, g(\psi(x)) \,\big] \tag{16}\] holds for every bounded measurable \(g\).
Two remarks on what kind of object this is. A pushforward is a probability law, always — and it need not have a density, even when \(p\) does: push \(\mathcal{N}(0, I_2)\) through a projection onto a line and the result lives on that line, with no density on \(\mathbb{R}^{2}\) (UP.A §2.1). And \(\psi\) is required to be measurable and nothing more; (Lipman et al. 2024) §3.3 states the same operation for diffeomorphisms, because it wants the density formula alongside, which is a stronger setting than this page needs.
Equation 16 also gives the operational sampling rule: draw \(x \sim p\) and return \(\psi(x)\). The resulting random variable has law \(\psi_{\#}\, p\). The notation \(\psi_{\#}\) is fixed for the course.
You have already built one. UP.A §4.2 wrote \(x = \mu + \sigma\varepsilon\) with \(\varepsilon \sim \mathcal{N}(0,1)\), and observed that sampling is pushing a fixed noise source through a deterministic map. In the notation of Equation 16 that reparameterization is \(\mathcal{N}(\mu, \sigma^{2}) = \psi_{\#}\, \mathcal{N}(0, 1)\) with \(\psi(\varepsilon) = \mu + \sigma\varepsilon\). It was the first pushforward of the course, written before the word existed.
A generative model
For the transport-based generative models studied in this course, the model law is obtained by pushing a fixed source distribution through a parameterized transport.
\[ p_\theta \;=\; \big(\psi^{\theta}\big)_{\#}\, p_0 , \qquad p_0 = \mathcal{N}(0, I_d) . \tag{17}\]
Once that scope is stated, this page calls Equation 17 “a generative model”. Four elements, a few sentences each.
- \(p_0 = \mathcal{N}(0, I_d)\) is the fixed source distribution. It is not learned, and it is cheap to sample from.
- \(\psi^{\theta}\) is the parameterized transport map defining the model law. Its parameters \(\theta\) are what a training run changes. Different units parameterize this transport differently: U1 constructs it directly from learned invertible layers, while later units may obtain the flow map indirectly from a learned dynamical field.
- \(p_\theta\) is the model law induced by the pushforward. Equation 17 defines that law without requiring a closed-form density; you sample from it by Equation 16. Whether the law admits a density, and whether that density can be evaluated tractably, depends on the structure of the transport. One caution on the symbol: a pushforward is a law, and the lowercase \(p_\theta\) of the course standard earns its reading as a density only once the model law is known to admit one — which for the maps of U1 and U2 it is, and which is precisely what those units establish.
- \(q\) is the data distribution. In the usual learning setting its density is not available analytically; what we have is a dataset of samples \(x_1 \sim q\).
\(t\) runs over \([0, 1]\). \(t = 0\) is noise and \(t = 1\) is data. So \(p_0\) is the source, \(x_1 \sim q\) is a data point, and a map that generates runs from \(0\) to \(1\).
The literature on diffusion models runs the arrow the other way. This course translates all of it, and the translation lives in exactly one document, the Rosetta Stone. Outside that document the arrow never reverses.
In this session \(\psi^{\theta}\) is one map, and how it is built is not the subject. U1 builds it as a composition of invertible layers. U2 builds it as the time-1 value of a differential equation. U3 changes what is learned: the learned primitive is a field, and the flow map is induced by it. The definition Equation 17 survives all three.
Maximum likelihood is a KL
Given data \[ x_1^{(1)}, \ldots, x_1^{(n)} \;\sim\; q , \] maximum likelihood maximizes the empirical average \[ \widehat{\mathcal{J}}_n(\theta) \;=\; \frac{1}{n} \sum_{i=1}^{n} \log p_\theta\big(x_1^{(i)}\big) . \] Its population counterpart is \[ \mathcal{J}(\theta) \;=\; \mathbb{E}_{x_1 \sim q}\big[\, \log p_\theta(x_1) \,\big] . \] The KL identity below concerns the population objective \(\mathcal{J}(\theta)\); the empirical maximum-likelihood objective \(\widehat{\mathcal{J}}_n\) is its sample-average estimator. The identity says that maximum likelihood is also a divergence criterion, and being able to read it both ways is what makes the rest of the course possible.
Assume that the data law and each model law under consideration admit densities, denoted \(q(x)\) and \(p_\theta(x)\), with respect to the same reference measure. Assume \(q \ll p_\theta\) for every \(\theta\) under consideration — equivalently, \(p_\theta(x) > 0\) for \(q\)-almost every \(x\) at which \(q(x) > 0\). This support condition is necessary for a finite KL but does not by itself guarantee finiteness (UP.A §3.1). For the calculation below, also assume that \(\mathbb{E}_{x_1 \sim q}[\log q(x_1)]\) and \(\mathbb{E}_{x_1 \sim q}[\log p_\theta(x_1)]\) are finite real numbers, and write \[ C_q \;:=\; \mathbb{E}_{x_1 \sim q}\big[\, \log q(x_1) \,\big] . \] Then \[ \mathrm{KL}\big(q \,\Vert\, p_\theta\big) \;=\; C_q \;-\; \mathcal{J}(\theta) , \tag{18}\] and consequently \[ \operatorname*{arg\,max}_{\theta}\; \mathcal{J}(\theta) \;=\; \operatorname*{arg\,min}_{\theta}\; \mathrm{KL}\big(q \,\Vert\, p_\theta\big) . \tag{19}\]
Proof.
Expand the divergence with the definition from UP.A §3.1 and split the logarithm: \[ \begin{aligned} \mathrm{KL}\big(q \,\Vert\, p_\theta\big) &= \mathbb{E}_{x_1 \sim q}\Big[\, \log \frac{q(x_1)}{p_\theta(x_1)} \,\Big] \\[2pt] &= \mathbb{E}_{x_1 \sim q}\big[\log q(x_1)\big] \;-\; \mathbb{E}_{x_1 \sim q}\big[\log p_\theta(x_1)\big] \\[2pt] &= C_q \;-\; \mathcal{J}(\theta) . \end{aligned} \] The split is legitimate because both expectations are finite by assumption. \(C_q\) is a property of the data alone — it has no \(\theta\) in it. (Under that finiteness assumption, the negative of \(C_q\) is the differential entropy of \(q\); the name is not used again.) So the two population objectives differ by a constant, which is Equation 18, and minimizing one is maximizing the other, which is Equation 19. \(\;\blacksquare\)
This is another constant-offset equivalence: the two population objectives differ only by a term independent of \(\theta\), as in the corollary of Section 2.3. It is not, however, an \(L^{2}\) projection statement, and the projection Lemma is not used here.
The data sits in the first argument of the divergence. UP.A §3.3 showed what that costs for a model family that cannot match the data. When the model family cannot represent \(q\) accurately, the mass-covering pressure of this KL direction can favor compromises that place model density between modes. As emphasized in UP.A, this is a useful heuristic for restricted approximation families, not a universal theorem about forward KL. Notice it now; it returns whenever the course compares training criteria.
The problem this creates
Equation 17 makes sampling immediate, but it does not by itself provide a tractable density evaluation procedure. The population maximum-likelihood objective \(\mathcal{J}(\theta)\) requires the quantity \(\log p_\theta(x_1)\), the log-density of the model at a data point. At this stage, the pushforward definition alone does not give us a tractable way to evaluate that density. Three things are distinct here: that the density exists, that it has an analytic expression, and that it can be evaluated tractably. The result above assumed the first; the objective needs the third.
For the same-dimensional invertible transports developed in U1 and U2, density evaluation requires relating the data point to its source point and accounting for the change in volume produced by the transport. In a discrete invertible flow this leads to inverse maps and Jacobian determinants. In a continuous-time flow the corresponding density change is tracked through the differential equation. Making these operations tractable is the technical subject of the next two units.
That sentence is the bridge out of this session.
§6 — Problem set PS1.2
This is one of the two graded parts of PS1. The other, PS1.1, is the nine exercises embedded in the primer.
The setting upgrades the worked example of UP.A §2.6 from one dimension to \(d\), and then asks you to see the Lemma happen on a screen.
Let \[ \begin{gathered} X \sim \mathcal{N}(\mu_0, \sigma_0^{2} I_d) , \qquad Y = X + \sigma \varepsilon , \\[2pt] \varepsilon \sim \mathcal{N}(0, I_d), \qquad \varepsilon \perp X . \end{gathered} \]
(a) Closed form. Compute the conditional mean and the conditional covariance of \(X\) given \(Y = y\), in terms of \(\mu_0\), \(\sigma_0\), \(\sigma\) and \(y\). Use the gaussian-algebra-toolkit sheet; the scalar case is item 9 on it. Show that \[
\begin{aligned}
m(y) \;:=\; \mathbb{E}\big[X \mid Y = y\big]
&\;=\;
\mu_0 + \frac{\sigma_0^{2}}{\sigma_0^{2} + \sigma^{2}}\,(y - \mu_0) \\[4pt]
&\;=\;
\frac{\sigma_0^{2}}{\sigma_0^{2} + \sigma^{2}}\, y
+ \frac{\sigma^{2}}{\sigma_0^{2} + \sigma^{2}}\, \mu_0
\end{aligned}
\] and \[
\operatorname{Cov}\big(X \mid Y = y\big)
\;=\;
\frac{\sigma_0^{2}\, \sigma^{2}}{\sigma_0^{2} + \sigma^{2}}\, I_d .
\] Explain in one sentence why the scalar toolkit result extends to \(d\) dimensions: every covariance matrix in the problem is a scalar multiple of \(I_d\), so the coordinates decouple and the joint-Gaussian conditional formula applies componentwise. Then read off the two limits, one line each. As \(\sigma \to 0\), \(m(y) \to y\) and \(\operatorname{Cov}(X \mid Y = y) \to 0\), because the observation becomes essentially noiseless. As \(\sigma \to \infty\), \(m(y) \to \mu_0\) and \(\operatorname{Cov}(X \mid Y = y) \to \sigma_0^{2} I_d\), because the observation becomes asymptotically uninformative.
(b) Numerical verification, \(d = 1\). Fix \(\mu_0 = 0\), \(\sigma_0 = 1\), \(\sigma = 0.5\). Part (a) gives \(m(y) = 0.8\, y\) and \(\operatorname{Var}(X \mid Y) = 0.2\). The population risk of a predictor \(f_\theta\) is \[
\mathcal{L}_X(\theta)
\;=\;
\mathbb{E}_{(x, y) \sim p_{XY}}\big[\, \lvert f_\theta(y) - x \rvert^{2} \,\big] ,
\] and the projection Lemma characterizes its minimizer. The starter trains on a finite dataset, so what it minimizes is the empirical sample-average approximation \[
\widehat{\mathcal{L}}_{X, n}(\theta)
\;=\;
\frac{1}{n} \sum_{i=1}^{n} \big\lvert f_\theta(y_i) - x_i \big\rvert^{2} .
\] Draw \(n = 20{,}000\) pairs \((y_i, x_i)\) and train the small multilayer perceptron \(f_\theta \colon \mathbb{R} \to \mathbb{R}\) that the starter fixes. The closed form \(m(y) = 0.8\, y\) is the unrestricted population-optimal regression function. The MLP should approximate it well on the region of \(y\) carrying substantial probability mass; exact global realizability is not assumed — a finite Tanh network is bounded as \(\lvert y\rvert \to \infty\), and \(0.8\, y\) is not. Evaluate on a dense grid between the empirical \(0.5\%\) and \(99.5\%\) quantiles of the sampled \(y_i\) (the starter’s eval_grid, \(601\) points), plot the trained \(f_\theta\) against \(m\) on that grid, and report \[
\max_{y \text{ on the evaluation grid}} \big\lvert f_\theta(y) - m(y) \big\rvert .
\]
(c) Population versus empirical: the surrogate. Keep everything from (b), and replace the regression target \(X\) by \[ Z = X + \eta , \qquad \eta \sim \mathcal{N}(0, \tau^{2}), \quad \eta \perp (X, Y), \qquad \tau = 1 . \]
On paper, before any training. First verify in one line that \(\mathbb{E}[Z \mid Y] = \mathbb{E}[X \mid Y] = m(Y)\), which is Equation 13. Then define the surrogate population risk \[ \mathcal{L}_Z(\theta) \;=\; \mathbb{E}_{(z, y) \sim p_{ZY}}\big[\, \lvert f_\theta(y) - z \rvert^{2} \,\big] \;=\; \mathbb{E}\big[\, \lvert f_\theta(Y) - Z \rvert^{2} \,\big] \] and show that, for every fixed \(\theta\), \[ \begin{aligned} \mathcal{L}_Z(\theta) &\;=\; \mathbb{E}\big[\, \lvert f_\theta(Y) - X - \eta \rvert^{2} \,\big] \\[2pt] &\;=\; \mathcal{L}_X(\theta) + \tau^{2} , \end{aligned} \] because the cross term vanishes: \(\eta\) is independent of \((X, Y)\) and has mean zero. With \(\tau = 1\), \[ \mathcal{L}_Z(\theta) \;=\; \mathcal{L}_X(\theta) + 1 \qquad \text{for every } \theta . \tag{20}\] This is corollary (b) of Section 2.3 on this example: the entire population risk is shifted by a parameter-independent constant. With the numbers of (b), predict the two constants of the corollary, \(C_X = \operatorname{Var}(X \mid Y) = 0.2\) and \(C_Z = \operatorname{Var}(Z \mid Y) = 0.2 + 1 = 1.2\), so that \(C_Z - C_X = 1\). That is the quantity to predict before running. It is a statement about population risks: it does not say that the final empirical training losses of two separately optimized networks differ by exactly \(1\).
Why the empirical objectives are different. For paired samples \(z_i = x_i + \eta_i\), define \[ \widehat{\mathcal{L}}_{Z, n}(\theta) \;=\; \frac{1}{n} \sum_{i=1}^{n} \big\lvert f_\theta(y_i) - z_i \big\rvert^{2} . \] Expand the difference and obtain \[ \begin{aligned} &\widehat{\mathcal{L}}_{Z, n}(\theta) - \widehat{\mathcal{L}}_{X, n}(\theta) \\[2pt] &\qquad\;=\; \frac{1}{n} \sum_{i=1}^{n} \eta_i^{2} \;-\; \frac{2}{n} \sum_{i=1}^{n} \big( f_\theta(y_i) - x_i \big)\, \eta_i . \end{aligned} \tag{21}\] Which term prevents the finite-sample difference from being a parameter-independent constant? Note that, conditional on the sampled \((x_i, y_i)\), the expectation over the \(\eta_i\) of the difference in Equation 21 is exactly \(\tau^{2}\) — but a realized finite dataset does not satisfy this equality exactly. This calculation is the centerpiece of part (c): it is the finite-sample caution of Section 2.3, written out.
The numerical experiment. Train the same architecture against \(X\) and against \(Z\), for at least three random seeds and at two sample sizes, \(n = 2{,}000\) and \(n = 20{,}000\). For each paired \(X\)/\(Z\) experiment, use the same \((x_i, y_i)\) sample and the same network initialization; the \(Z\) dataset adds the independent \(\eta_i\) (the starter does this for you within one seed). Report:
- the two fitted curves together with the theoretical regression function \(m(y) = 0.8\, y\), one figure per sample size;
- the maximum \(X\)-fit versus \(Z\)-fit discrepancy on the same predefined evaluation grid;
- the maximum deviation of each fitted curve from \(m\) on that grid;
- the empirical training losses;
- the theoretically predicted population risk offset, \(\tau^{2} = 1\);
- how the observed empirical loss difference compares with that population prediction.
The theoretical prediction is about population objectives. The numerical experiment asks whether the expected statistical pattern is visible: the noisier \(Z\) target should generally produce greater finite-sample variability, and increasing \(n\) should generally stabilize both fits. These are statistical trends, not deterministic pass/fail conditions for each seed. Summarize the behavior across seeds rather than confirming a prescribed ordering: compare the distribution, mean, or median of the curve-discrepancy metric across seeds at the two sample sizes. Does the larger sample size show the expected stabilization? If one individual seed does not follow the trend, report it rather than treating it as an error.
The surrogate has larger conditional residual variance, \[ \begin{gathered} \operatorname{Var}(Z \mid Y) \;=\; \operatorname{Var}(X \mid Y) + \tau^{2} , \\[2pt] \operatorname{Var}(X \mid Y) = 0.2, \qquad \operatorname{Var}(Z \mid Y) = 1.2 . \end{gathered} \] This extra conditional noise can increase finite-sample estimation variability, but it does not imply a deterministic ordering of the learned-curve errors for every seed.
And the sentence below, in your own write-up, verbatim:
The projection Lemma gives population objectives with the same regression target and differing only by a parameter-independent constant; it does not say that two realized finite-sample losses or their minibatch gradients are identical.
This exercise has a secret identity, to be revealed in U3.
You do not need any earlier session to do this, and you do not need to write the training loop: the assignment ships one. labs/ps1-2-starter.py is self-contained — torch, numpy, matplotlib, nothing from the bootcamp — and its architecture and optimizer are fixed: three Linear layers with Tanh between them and 64 hidden units, Adam at \(10^{-3}\), \(1{,}000\) full-batch steps. The starter is intentionally lightweight and is designed to run on a laptop CPU; --sweep runs the whole protocol of part (c). Change the settings only where a part of the exercise asks you to; the point is the theorem, not the framework.
Report your seeds. One seed fixes both the data draw and the initialization: the \(X\)-run and the \(Z\)-run of one seed share the same \((x_i, y_i)\) and the same initial network, and the \(Z\)-run adds the \(\eta_i\). Two runs with different seeds change both the data and the initialization, so their gap is a baseline for ordinary run-to-run variability, not a control for sampling noise alone. The evaluation grid is eval_grid: \(601\) points between the empirical \(0.5\%\) and \(99.5\%\) quantiles of the sampled \(y_i\).
Corollary (b) is an exact population statement. Here, \(\mathcal{L}_Z(\theta) = \mathcal{L}_X(\theta) + 1\) for every \(\theta\), Equation 20.
The corresponding empirical losses do not differ by a constant, because the realized cross term involving the \(\eta_i\) in Equation 21 is generally nonzero. Therefore separately trained \(X\)- and \(Z\)-fits need not coincide on a finite dataset.
The \(Z\) target also has greater conditional residual variance: \(1.2\) instead of \(0.2\). Across repeated samples, one therefore expects greater finite-sample variability from the surrogate regression, while increasing the sample size should generally stabilize both procedures. The exercise asks you to investigate that statistical tendency, not to force every individual seed to exhibit a predetermined ordering.
Because the supplied Tanh MLP does not exactly contain the unbounded function \(m(y) = 0.8\, y\) on all of \(\mathbb{R}\), the closed form should be interpreted as the unrestricted Bayes predictor and as the reference curve on the evaluation range, not as a globally realizable network target.
Where this session reappears
| From this session | Reappears in |
|---|---|
| The projection Lemma, Equation 11 | U3.T1 (to define an object); U3.T2 (to prove a theorem) |
| Corollary (b), “same conditional mean” | U3.T2, twice — the main line and detour D2 |
| The tower rule, Equation 6 | inside every application of the Lemma |
| Pushforward notation \(\psi_{\#}\) | U1.T1; U2.T1; U3.T1 |
| \(p_\theta = (\psi^{\theta})_{\#} p_0\), Equation 17 | U1.T1 — \(\psi^{\theta}\) built from invertible layers; U2 — \(\psi^{\theta}\) is the time-1 map of a Neural ODE; U3 — the transport flow is induced by the learned Flow Matching velocity field |
| Maximum likelihood is a KL, Equation 18 | U1.T1 (the training objective); U2.T2 |
| The time arrow \(t = 0\) noise, \(t = 1\) data | everything from U3 on, without exception |
Next session
The next session opens U1 and takes the first of the three strategies of Section 5: make \(\psi^{\theta}\) invertible by construction, and read its volume change off the architecture. The result that makes it work is the change-of-variables theorem, and proving it in \(d\) dimensions is part of PS1.
Before then: PS1.1 is the primer’s nine exercises, and PS1.2 is Section 6. Both are due together.