NL-3 — HOPE: assembling it all

The question: how do the pieces combine into one continual-learning architecture?

Final module of the Nested Learning track, and the last of the course. Prerequisites: foundations M1–M7, NL-1, NL-2. Runs on CPU in seconds; PyTorch throughout.

This is the capstone. Everything before it was a part; HOPE is the machine you get when you snap them together. By the end you should be able to:

State HOPE’s two halves — a self-modifying (“self-referential”) Titans memory followed by a Continuum Memory System — and say which module each piece comes from.
Explain what makes a memory self-modifying rather than merely adaptive: it generates its own write targets, not just its own contents.
Say why the self-referential loop doesn’t collapse to a trivial solution (the inner self-targets are wrapped in an outer loop that sees real labels).
Read HOPE as NL’s existence proof: “add nested levels” yields one uniform stack of feed-forward memories that continually learns — dissolving the attention-vs-MLP split.

Grounding: Nested Learning (Behrouz, Razaviyayn, Zhong & Mirrokni, Dec 2025) §8 (the HOPE architecture) and §9. HOPE is that paper’s own architecture — the thing the paradigm was written to produce.

Why this module exists — what HOPE is built to fix

Through the NL lens, a Transformer is only a two-level system (NL-2): attention does in-context learning at frequency \(\infty\), while the projections \(W_k,W_v,W_q\) and the MLP are frozen at frequency \(0\) after pre-training. §8 opens by naming the two limitations that follow:

Frozen contextualization. Because \(W_k,W_v,W_q\) are fixed after pre-training, “the Transformer’s ability to contextualize and map tokens is bounded by the knowledge stored in these blocks” — it cannot modify itself in-context.
Limited computational depth. Two levels only (NL-2 §5), which §8 ties to the state-tracking literature.

HOPE removes both. Its goal: let every component perform in-context learning and modify itself, and replace the single frozen MLP with a spectrum of memories. It does this by combining two pieces — and every piece is something we already built.

1. The assembly — every module in its place

HOPE = self-modifying Titans → Continuum Memory System (§8.3). Each part is a course module:

HOPE component	what it does	from module
in-context memory \(\mathcal M_t\), written per token	the fast associative store	M1 (Hebbian write), M3 (linear attn = fast weights)
projections \(\mathbf k,\mathbf v\) — produced by their own in-context memories	contextualization that moves with the context	M2 (FWP), M5 (meta-learned init)
in-context \(\eta_t,\alpha_t\) — the write rule’s own knobs, per token	write hard when surprised, gently once learned	M4 (learned rate \(\beta_t\)), NL-1 (Titans’ surprise gate)
delta / gated write	erase-then-write, bounded norm	M4 (DeltaNet), NL-1 (Titans)
self-modifying update: each memory generates its own target \(\hat{\mathbf v}_{\square,t}\)	learns what to write, not just what it was handed	M5 (SRWM self-reference)
inner optimizer = DGD + weight decay	correlated-token-aware write	M6 (DGD)
CMS after the memory: a chain of MLP blocks at \(k\) frequencies	large, persistent, multi-timescale storage	NL-2 (levels + CMS)

One asymmetry worth catching, because it is easy to misread: \(\mathbf k\) and \(\mathbf v\) get their own memories, but \(\mathbf q\) does not — §8.1 states plainly that “\(\mathbf q_t=\mathbf x_tW_q\) is the only non-adaptive projection”. (The paper is not quite self-consistent here — its Eq. 85 still ranges over \(\mathbf q\) — but the prose is unambiguous.)

Two halves, complementary by design: self-modifying Titans = small capacity, expressive learning rule; CMS = large capacity, simple rule, persistent storage (§8.3). The rest of this module zooms into the one genuinely new idea — self-modification — then snaps the whole thing together.

2. Self-modification — learning how to update, not just what to store

NL-1 already let a memory write itself at test time. HOPE pushes further along M5’s self-reference axis — and §8.1 climbs there in three rungs, which are worth separating, because the rung that earns the name is not the one you would guess:

Plain (Eq. 76). \(\mathbf k,\mathbf v,\mathbf q,\eta_t,\alpha_t\) all come from static projections: \(\eta_t=\mathbf x_tW_\eta\), and so on. The knobs already depend on the token — but the maps that produce them never move.
Adaptive (Eq. 79). Each of those is produced instead by its own in-context memory: \(\eta_t=\mathcal M_{\eta,t-1}(\mathbf x_t)\), \(\alpha_t=\mathcal M_{\alpha,t-1}(\mathbf x_t)\), likewise \(\mathbf k,\mathbf v,\mathbf q\). The rule’s knobs are now set by something that itself learns while the sequence runs. The paper calls this “a fully adaptive memory, where all the components can adapt themselves in-context.”
Self-modifying (Eq. 83–85). Adaptive is still not self-modification — the paper says so outright: that design “still lacks self-modification”. The step that earns the name is that each memory generates its own target, \(\hat{\mathbf v}_{\square,t}=\mathcal M_{\square,t-1}(\mathbf v_t)\) for \(\square\in\{\mathbf k,\mathbf v,\eta,\alpha,\text{memory}\}\) — “Generating its own values for each memory” — instead of being handed one, and stops sharing a single key/value pair across all of them.

Rung 2 is the tangible one, and it is what the demo shows: the update rule’s own knobs are produced from the input, so the memory writes aggressively when a context is surprising and conservatively once it’s learned — a learned learning algorithm rather than a fixed schedule. The demo ties \(\eta_t,\alpha_t\) to the running surprise (Titans-style) and presents the same token repeatedly: watch \(\eta_t\) ease off as the memory learns it.

Rung 3 — the self-generated target — is what §3 takes up, because it is also where the obvious objection lives.

import numpy as np
import torch

torch.manual_seed(3)
d = 16
gg = torch.Generator().manual_seed(5)
Wt = torch.randn(d, d, generator=gg) / np.sqrt(d)
A = torch.randn(d, generator=gg)
A /= A.norm()
vA = Wt @ A  # one token, presented repeatedly

M = torch.zeros(d, d)
print("the SAME token A, written repeatedly; η_t and α_t are GENERATED from the current surprise:")
print(f"{'step':>4} {'surprise':>9} {'η_t (write)':>12} {'α_t (retain)':>13}")
for t in range(8):
    surprise = (M @ A - vA).norm()
    eta_t = torch.sigmoid(2.0 * surprise - 1.0)  # in-context learning rate
    alpha_t = 1.0 - 0.5 * torch.sigmoid(surprise)  # in-context forget gate
    print(f"{t:>4} {surprise.item():>9.3f} {eta_t.item():>12.3f} {alpha_t.item():>13.3f}")
    M = alpha_t * M - eta_t * torch.outer(M @ A - vA, A)  # gated delta write with in-context η,α
print("\nstep 0: high surprise → η=0.77 (write hard).  once learned: surprise falls → η eases to ~0.38–0.47,")
print("and α (retention) rises 0.63→0.70.  The memory modulates its OWN update rule by context.")

the SAME token A, written repeatedly; η_t and α_t are GENERATED from the current surprise:
step  surprise  η_t (write)  α_t (retain)
   0     1.103        0.770         0.625
   1     0.254        0.380         0.718
   2     0.397        0.449         0.701
   3     0.430        0.465         0.697
   4     0.434        0.467         0.697
   5     0.434        0.467         0.697
   6     0.434        0.467         0.697
   7     0.434        0.467         0.697

step 0: high surprise → η=0.77 (write hard).  once learned: surprise falls → η eases to ~0.38–0.47,
and α (retention) rises 0.63→0.70.  The memory modulates its OWN update rule by context.

How is this different from backprop, which also generates its own values (M6 §1.1)? The paper never draws this comparison directly, but it supplies both halves — §4.1 reads backpropagation as a self-referential associative memory (“the update rule in backpropagation is a self-referential process … where the values of the associative memory is generated by itself”), and §8.1 builds HOPE’s self-generated targets — so it is worth drawing. Two differences:

Backprop’s self-value carries the truth every step. §4.5 writes it as \(\mathbf v_t=-\nabla_{y_t}\mathcal L(W_t;\mathbf x_t)\) — an error \(\approx(\text{predicted}-\text{true})\), so it references the external target on every update. HOPE’s inner self-target \(\hat{\mathbf v}_\square=\mathcal M_\square(\mathbf v)\) is a pure learned transform of the input — no target subtracted; truth enters only via the outer loss.
Backprop’s value-generating rule is fixed (differentiation, forever); HOPE’s is a memory — meta-learned, and updated in-context as the sequence runs. Backprop is self-referential in state; HOPE is self-referential in state and rule.

That second point is the whole upgrade, and it is exactly the optimizer→architecture lift from M6: §4.5’s Generalized Gradient Descent (Def. 5) frees the value-generating function \(\mathbf f_{W_t}(\cdot)\) to be something other than the gradient; HOPE puts a learned one inside the architecture.

3. Why the self-referential loop doesn’t collapse

A memory trained toward its own outputs has an obvious failure mode: map everything to a constant — the inner loss is zero, and the memory is useless. So why doesn’t HOPE collapse?

Because the self-generated targets are an inner mechanism wrapped inside an outer loop that does have ground truth. The demo shows both sides: train a memory whose target is self-generated (\(\hat{\mathbf v}=\mathcal M_v(\mathbf v)\), both learned), once with the inner objective only, once with an outer next-token loss added.

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
d, T = 8, 200
g = torch.Generator().manual_seed(1)
Wtrue = torch.randn(d, d, generator=g) / np.sqrt(d)
X = torch.randn(T, d, generator=g)
X = X / X.norm(dim=1, keepdim=True)
Y = X @ Wtrue.T  # the real (outer) targets


def train(use_outer, steps=600, lr=0.05, wd=1e-2):
    M = nn.Linear(d, d)
    Mv = nn.Linear(d, d)  # Mv generates the inner target
    opt = torch.optim.Adam(list(M.parameters()) + list(Mv.parameters()), lr=lr, weight_decay=wd)
    for _ in range(steps):
        L = ((M(X) - Mv(X)) ** 2).mean()  # inner: chase own self-generated target
        if use_outer:
            L = L + ((M(X) - Y) ** 2).mean()  # outer: predict the REAL next target
        opt.zero_grad()
        L.backward()
        opt.step()
    with torch.no_grad():
        out = M(X)
        return out.var().item(), (out - Y).norm(dim=1).mean().item()


chance = Y.norm(dim=1).mean().item()
print(f"chance error (predicting zero) = {chance:.3f}\n")
for use_outer in [False, True]:
    var, err = train(use_outer)
    tag = "inner + OUTER anchor" if use_outer else "inner-ONLY (no anchor)"
    print(f"{tag:24s}: output variance = {var:.4f}   prediction error vs truth = {err:.3f}")
print("\ninner-only → output COLLAPSES to a constant (var≈0) and predicts truth at chance: useless.")
print("outer anchor → memory stays expressive (var>0) and predicts far better than chance.")

chance error (predicting zero) = 0.984

inner-ONLY (no anchor)  : output variance = 0.0000   prediction error vs truth = 0.984
inner + OUTER anchor    : output variance = 0.0509   prediction error vs truth = 0.358

inner-only → output COLLAPSES to a constant (var≈0) and predicts truth at chance: useless.
outer anchor → memory stays expressive (var>0) and predicts far better than chance.

So self-modification is safe for the same reason MAML and TTT (M5, NL-1) are: the inner adaptation has no labels, but it is a learned procedure validated by an outer loop that does. Four things hold it in place: (1) the output still predicts real tokens through the LM head; (2) the value-generating functions and initial states are meta-learned against that real loss; (3) inputs are real data, so each self-target transforms real signal; (4) the update is bounded — one gated step per token from the meta-learned init.

Only the second is the paper’s own argument, and it is emphatic: §8.1 calls meta-learning the initial states “essential for both fast-adaption, training stability, robustness to noise in the data.” The other three are our reading of the design — NL never raises collapse as a failure mode, so it names no defence against it. Nor is there a formal guarantee. But note what the paper’s own sentence concedes: the stability of this thing is engineered, not free.

4. The two pieces are complementary — and DGD lives inside

Why bolt a CMS onto the self-modifying memory at all? Because they cover for each other’s weakness (§8.3):

Self-modifying Titans — small capacity, but an expressive, learned write rule. Great at fast, clever, in-context adaptation; too small to be the model’s knowledge store.
CMS (NL-2) — large capacity, a simple rule, persistent multi-frequency storage. Great at holding knowledge across time; not adaptive on its own.

Fast-and-clever in front, big-and-persistent behind. And inside the memory, the inner optimizer is DGD + weight decay (M6), not plain GD — and §8.1 gives the reason in one line: gradient descent’s write “is solely based on the input and does not incorporate the previous data samples”, but “when performing optimization in the token space … we know tokens are highly correlated.” So the delta-rule write manages a correlation that a dot-product/GD write would smear into interference. Every “why this and not the obvious thing” in HOPE is a limitation we already hit and fixed earlier in the course.

5. The whole block, assembled

Here is a minimal HOPE block — self-modifying memory → CMS — in one forward pass. It is a toy (forward-pass only, untrained) whose point is legibility: every line is a piece from an earlier module, and they compose into one object that runs end to end.

Want to see it learn? The companion aside — training a HOPE block trains this block end-to-end on associative recall (CPU: seconds with OMP_NUM_THREADS=1, minutes without): it reaches ~100% recall, generalizes to longer contexts, and shows the outer loss anchoring the self-generated target (§3 above) in practice.

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
d, T = 16, 32


class HopeBlock(nn.Module):
    def __init__(self, d, cms_periods=(1, 8)):
        super().__init__()
        self.Wk, self.Wv, self.Wq = (nn.Linear(d, d, bias=False) for _ in range(3))  # projections   (M2)
        self.g_eta, self.g_alpha = nn.Linear(d, 1), nn.Linear(d, 1)  # in-context η,α (M4/NL-1)
        self.Mv = nn.Linear(d, d, bias=False)  # self-target   (NL-3)
        self.cms = nn.ModuleList(nn.Linear(d, d) for _ in cms_periods)  # CMS levels    (NL-2)
        self.periods = cms_periods
        self.cms_updates = [0] * len(cms_periods)

    def forward(self, seq):
        M = torch.zeros(d, d)  # fast in-context memory          (M1)
        outs = []
        for t in range(seq.shape[0]):
            x = seq[t]
            k, v, q = self.Wk(x), self.Wv(x), self.Wq(x)
            vhat = self.Mv(v)  # self-generated target           (NL-3)
            eta = torch.sigmoid(self.g_eta(x)).squeeze()  # in-context learning rate         (NL-1)
            alpha = torch.sigmoid(self.g_alpha(x)).squeeze()  # in-context forget gate           (NL-1)
            M = alpha * M @ (torch.eye(d) - eta * torch.outer(k, k)) - eta * torch.outer( M @ k - vhat, k)  # DGD-style gated delta write  (M4+M6)
            o = M @ q  # read                             (M1)
            for l, Cl in enumerate(self.periods):  # CMS frequency chain              (NL-2)
                o = torch.relu(self.cms[l](o))
                if (t + 1) % Cl == 0:
                    self.cms_updates[l] += 1
            outs.append(o)
        return torch.stack(outs)


seq = torch.randn(T, d)
seq /= seq.norm(dim=1, keepdim=True)
block = HopeBlock(d)
out = block(seq)
print(f"HOPE block forward pass OK:  input {tuple(seq.shape)}  →  output {tuple(out.shape)}")
print(f"CMS level update counts over {T} steps (periods {block.periods}): {block.cms_updates}")
print(f"output all finite: {torch.isfinite(out).all().item()}   output std: {out.std().item():.3f}")
print(
    "\nEvery line traces to a module — M1 store, M2 projections, M4+M6 write, M4/NL-1 in-context η,α, NL-3 self-target, NL-2 CMS."
)

HOPE block forward pass OK:  input (32, 16)  →  output (32, 16)
CMS level update counts over 32 steps (periods (1, 8)): [32, 4]
output all finite: True   output std: 0.097

Every line traces to a module — M1 store, M2 projections, M4+M6 write, M4/NL-1 in-context η,α, NL-3 self-target, NL-2 CMS.

6. What HOPE buys — and the honest caveats

The paper’s experiments (§9) back the assembly with results, and the ablations are what make them interpretable:

Continual learning. On class-incremental tasks HOPE beats ICL, EWC, and even the external-learner InCA; under a continual setup where plain ICL catastrophically forgets a new language, HOPE-3 almost recovers ICL’s own non-continual score (the CMS multi-frequency retention, NL-2).
Long context. Trained from scratch, HOPE leads attention-free models on needle-in-a-haystack recall, and on BABILong it holds to ~10M tokens where large Transformers fail by ~128K–256K and even Titans/ARMT drop after 1M — “mainly due to its CMS design.”
Language modeling. Best average across common-sense benchmarks at 760M and 1.3B params.
The ablations localize the credit: gains over ICL come from multiple levels / CMS; gains over Titans come from self-reference + CMS; and Hope-Attention > Transformer — HOPE with the self-modifying memory swapped out for plain softmax attention — isolates CMS as helpful on its own.

Caveats worth keeping honest. HOPE is a single instantiation, so the paradigm’s contribution is hard to fully separate from this architecture’s engineering (the ablations are the mitigation). For language modeling, scale tops out at 1.3B params / 100B tokens — which matters because §8 opens by granting the very thing that scale would test: parametric memories “are not expected to outperform softmax attention when the model size and data scales.” HOPE is built to dodge that premise by not being a plain parametric memory; whether it does at frontier scale is not settled here. And §8.1’s own “essential for … training stability” (§3 above) says the bilevel training is delicate enough to need securing.

7. The course in one breath

Start with one primitive: a memory you write with an outer product and read with a matrix–vector product (M1). Make the writes programmable and you get fast-weight programmers (M2), which are linear attention (M3). Swap the additive write for an error-correcting one and you get the delta rule / DeltaNet (M4). Let the memory supply its own keys, values and write rate — ask who writes the writer — and you get meta-learning and self-referential weights (M5). Then notice the optimizer was an associative memory over the gradient stream all along, and that M4’s fix applies to it too (M6). Collapse all six into one recurrence with three dials (M7). Now turn the write dial until the write rule is a full learner trained at inference, and you get TTT → Titans → Miras → Atlas (NL-1); order every memory by how often it updates and you get levels and a continuum of timescales (NL-2); stack the self-modifying memory onto that continuum, optimized with the delta-rule optimizer, anchored by an outer loss, and you get HOPE (NL-3): one uniform stack of feed-forward memories that continually learns. Then ask what every memory in that stack was actually optimizing — and find that the answer, shared by all of them and argued for by none, is itself a dial (NL-4).

The dense paper now reads as inevitable: each idea is the obvious fix to the limitation of one that came before. That was the whole point.

Where to go from here. The open threads are the honest caveats turned into questions. Does the paradigm hold at frontier scale — the thing §8’s own premise says should be hard? Can the bilevel training be stabilized without leaning on meta-learned initial states? And one the paper does not raise: NL-2 §5 separated two axes of computational depth — NL buys depth with nested levels, extra computation inside a single forward pass, while reasoning models buy it in token space, by generating more and thinking longer. Whether those are the same quantity bought in two places, or genuinely different things, is open — and this framing is unusually well placed to ask it, since it is the one that made “how many levels?” a countable question in the first place. (That last thread is ours, not the paper’s: NL never discusses chain-of-thought or test-time compute scaling.) These are live, paper-independent questions; this course gives you the machinery to take them on. A fourth is not left open at all: NL-4 takes up the objective every memory here descends on, and asks — from outside this group, using this group’s own construction — whether it was ever needed.

Code walkthrough — where this lives

hope/block.py — the full HOPE block. HOPEBlock composes the Titans memory, the CMS, the self-modifier and a per-level optimizer manager; its forward pass is the faithful counterpart of §5’s toy. It is also where the CMS’s chunked per-level updates (NL-2’s Eq. 71) actually run — not in cms.py, where you would look first.
hope/self_mod.py — the SelfModifier, forty lines: an MLP over [key, value, error_signal] whose output is subtracted from the teach signal to form the memory’s target. The “generate your own target” mechanism of §2, in the flesh.
titan/self_modifying.py — the self-referential Titans memory: a residual-MLP memory that computes its own gradients (via torch.func.grad/vmap) and applies its own chunked updates with meta-learned rates.
cms.py — the continuum. optim/m3.py — the paper’s M3 optimizer (Multi-scale Momentum Muon, §7.2, Algorithm 1), which applies the CMS idea to momentum instead of memory: a fast momentum every step, a slow one every hundred, both orthogonalized Newton–Schulz-style. Its name collides with this course’s module M3 (linear attention); the collision is the paper’s and they are unrelated.
train_hope.py + generate.py — a small from-scratch HOPE (~86M params, attention-free) you can read end to end and sample from on a laptop. Training it is a real GPU job rather than a laptop one: its self-modifying layer is a Python loop over 512 timesteps per forward pass.

The two repos are independent community reimplementations, not the authors’ code.

Exit check

1. What are HOPE’s two halves, and what is each good at? A self-modifying (“self-referential”) Titans memory followed by a Continuum Memory System (§8.3). The self-modifying memory has small capacity but an expressive, learned write rule — fast, clever, in-context adaptation; the CMS has large capacity, a simple rule, and persistent multi-frequency storage (NL-2) — the knowledge store. Fast-and-clever in front, big-and-persistent behind; they cover each other’s weakness.

2. What makes a memory “self-modifying” rather than just “adaptive”? Not the in-context \(\eta_t,\alpha_t\) — those are already there in the adaptive design (Eq. 79), and in weaker form even in the plain one (Eq. 76). What earns the name is that each memory generates its own write target, \(\hat{\mathbf v}_{\square,t}=\mathcal M_{\square,t-1}(\mathbf v_t)\), rather than being handed one (Eq. 83–85) — the paper is explicit that the merely adaptive design “still lacks self-modification” (§2). It is self-referential in state and rule, where backprop is self-referential only in state.

3. Why doesn’t the self-referential loop collapse to a trivial solution? Because the self-generated targets are an inner mechanism wrapped in an outer loop that has ground truth (§3). Inner-only training collapses (the demo: output variance → 0, prediction at chance), but the real LM next-token loss, meta-learned initial states, real-data inputs, and a bounded gated update together anchor it — the same MAML/TTT trick (M5, NL-1): a label-free inner procedure validated by a labeled outer loop. Note this is our reading, not the paper’s: it never raises collapse, and only the meta-learned initial states are its own stated requirement.

4. In what sense is HOPE the payoff of the whole course? It is NL’s existence proof that “add nested levels” yields one uniform stack of feed-forward memories that continually learns — dissolving the attention-vs-MLP split into a spectrum of frequencies (§7). Every piece is an earlier module: the store (M1), programmable/linear-attn writes (M2–M3), delta correction (M4), self-reference (M5), the delta-rule optimizer inside (M6), the test-time write (NL-1), and the continuum it sits in (NL-2). The dense paper becomes the inevitable assembly of pieces you built yourself.

Next → NL-4. You have now built every memory this track names, from a single outer-product store up to HOPE, each idea the repair of an earlier one. They have one thing in common that none of them argues for: every one learns by predicting a token’s value from its key. NL-4 is the paper that names that objective from outside — calls it KV Binding, files this whole track under it — and asks whether the binding was ever the point. Take §6 and §7’s open questions with you; the next module adds one they do not raise.