The question: M1 through M6 built six mechanisms. How many ideas is that, really?
Answer: one, with three dials. This module collapses the spine into a single recurrence, then shows every module you have read as a setting of it. It is also the hub the rest of the course attaches to — each track states its dial setting and departs from here.
Runs on CPU in seconds.
Objective
After this module you should be able to:
Write the master recurrence\(\mathbf{S}_t=\alpha_t\mathbf{S}_{t-1}+\beta_t(\text{write}_t)\phi(\mathbf{k}_t)^\top\) and derive it as the generalization of M3’s linear-attention state update.
Name the three dials — write rule (what goes in), gate\(\alpha_t\) (what stays), optimizer (how the write is applied) — and say which module of the spine turns each.
State any of M1–M6 as a setting of those dials, and read the table backwards: given a dial setting, say which mechanism you are looking at.
Reproduce M1/M3 and M4 from one implementation by flipping a single argument, and predict when the two settings coincide and when they diverge.
State a new method’s dial setting — the contract every track in this course binds to.
Why it exists (the role it plays)
M7 is not a fix. Every module before it repaired an earlier one: M2 gave M1’s memory its keys, M3 named what M2 had built, M4 replaced its blind write, M5 asked who writes the writer, M6 reached back past M5 to find the same memory hiding inside the optimizer. Six mechanisms, each motivated by an earlier one’s failure.
Read that way the spine is a chain, and a chain is the wrong shape for what comes next. Every paper worth adding to this course would have to be bolted onto the end of it, inheriting five prerequisites it does not need.
So the spine ends by collapsing. One recurrence, three dials, and the six mechanisms fall out as settings — not as a sequence of repairs but as points in a small space you can now name coordinates in. That is what makes the rest of the course attachable: a track is a paper (or a family) that picks a dial setting, states it against the table in §3, and departs from there. Tracks never depend on each other, and adding one renumbers nothing.
Reading
Nested Learning §5 (Existing Architectures as Neural Learning Modules) — the grounding source. Eq. 64 is the master recurrence with the Hebbian write; the paragraph under it is the dial claim in the paper’s own words. Eq. 65 (delta rule), Eq. 66 (Oja’s rule), Eq. 68 (Omega rule) are three more settings. §3.1 (Def. 1) defines the associative memory the whole framing rests on.
Everything else you need is behind you: M3 §3 (the recurrence this generalizes), M4 §3–§4 (the write dial), M6 §5 (the optimizer dial).
1. The master recurrence
Start where M3 left the memory. Deleting the softmax collapsed “attend to every past token” into a fixed-size matrix updated once per token (M3 §3):
Now read it as an assertion rather than a definition, and two free choices show up.
Nothing forces the added term to be \(\mathbf{v}_t\). The recurrence needs some vector to pair with \(\phi(\mathbf{k}_t)\) in the outer product. M3 uses the raw value because its objective is dot-product similarity; M4 writes the residual \(\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\) because its objective is \(L_2\) regression (M4 §3). Both are legal. Call whatever goes in \(\text{write}_t\in\mathbb{R}^{d_v}\), and give it a scalar strength \(\beta_t\) — M4’s learned per-token write rate.
Nothing forces the old state to survive intact. M3 keeps \(\mathbf{S}_{t-1}\) whole, which is a choice — the choice to never forget. Scale it by \(\alpha_t\) and the memory can fade.
The symbols: \(\mathbf{S}_t\in\mathbb{R}^{d_v\times d_k}\) is the memory (M1’s matrix, M2’s fast weights, M3’s state — three names, one object); \(\phi\) is the feature map on the key; \(\alpha_t\) is the gate, \(\beta_t\) the write strength, \(\text{write}_t\) the write rule. M3’s recurrence is the special case \(\alpha_t=1\), \(\beta_t=1\), \(\text{write}_t=\mathbf{v}_t\) — never forget, unit strength, store the value raw.
This is not a re-derivation for its own sake. Nested Learning §5 prints the same equation (Eq. 64, for the Hebbian write) and then makes the claim this module exists to cash out:
“Given different settings for \(\alpha_t\) (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also \(\phi(\cdot)\) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule.”
2. The three dials
The master recurrence has exactly three places a designer can make a decision. Every model in this course — and, the claim goes, most of the modern literature — is a choice at each.
Dial 1 — the write rule: what goes in
\(\text{write}_t\) is whatever vector gets paired with \(\phi(\mathbf{k}_t)\). It is set by the memory’s inner objective — what the write is trying to achieve — because the write is one gradient step on that objective:
Hebbian (M1): \(\text{write}_t=\mathbf{v}_t\), one ascent step on the dot-product objective \(\langle\mathcal{M}\mathbf{k},\mathbf{v}\rangle\) (M1 §5’s callout). Store the value as handed in; never consult the memory. Cheap, and state-blind.
Delta (M4): \(\text{write}_t=\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\), one descent step on \(\tfrac12\lVert\mathcal{M}\phi(\mathbf{k})-\mathbf{v}\rVert^2\) (M4 §3). Read the memory at this key first and write only what it is getting wrong — so a revisited key is edited, not piled onto.
Optimizer-shaped (M6): once the write is a gradient step, the write rule is whatever an optimizer does — many steps, a deeper inner objective, a preconditioned step.
The other module in this dial’s history is M2, which changed not what rule writes but who supplies the arguments: \(\mathbf{k}_t,\mathbf{v}_t,\mathbf{q}_t=W_k\mathbf{x}_t,W_v\mathbf{x}_t,W_q\mathbf{x}_t\). M5 closed that loop — the memory supplies its own.
Dial 2 — the gate \(\alpha_t\): what stays
\(\alpha_t\) multiplies the whole state, so it is the only dial that touches what is already stored:
\(\alpha_t=1\) — never forget. M1, M2, M3, M4 and M5 all sit here; the memory is append-or-edit only, and its failure mode is saturation (M2’s capacity law).
Constant\(\alpha<1\) — a fixed exponential fade of everything, oldest first.
Learnable / channel-wise / input-dependent — the memory decides per token what to drop, which is what the paper’s own list (§5, quoted above) enumerates.
The distinction worth holding, from M4 §4: the gate is a global fade — it decays every stored association at once, including the ones you were not writing to. The delta rule’s erase factor \((\mathbf{I}-\beta\mathbf{k}\mathbf{k}^\top)\) looks like forgetting but is not this dial: it erases only along \(\mathbf{k}\). Targeted edit (write dial) and global forget (gate dial) are independent, and you can turn both.
Dial 3 — the optimizer: how the write is applied
The write rule fixes the direction; this dial fixes what is actually done with it. M1–M5 all take a single, unpreconditioned step — the only variation is the rate \(\beta_t\) (fixed at 1 in M1–M3; learned per token in M4, \(\beta_t=\sigma(W_\beta\mathbf{x}_t)\); self-generated in M5). Turn it further and you get: many steps instead of one; momentum, which carries a memory of past writes; preconditioning (Adam), which reshapes the step’s coordinates; orthogonalization (Muon).
M6 is the module that turns this dial, and its result is the reason the hub is shaped like this: those optimizers are themselves memories over a second stream — gradients instead of tokens — with their own write rule and their own gate (M6 §5). Turn the optimizer dial far enough and you are setting the three dials again, one level down. That nesting is where the tracks begin.
3. Every foundations module as a dial setting
Read across a row and you get the module. Read down a column and you get its history.
\(1\) on the weights; \(\alpha\) on the momentum buffer
this dial is the subject — momentum, Adam, Muon
Four things to notice, because they are the module’s whole argument:
The gate column is a constant. Five of six rows are \(1\). Nothing in the foundations ever forgets on purpose — a whole dial the spine never turned, and the reason §5’s zoo has rows this table does not.
M1, M2 and M3 are one row. Their write rules differ only in who computes \(\mathbf{v}_t\) and what \(\phi\) does to the key. The recurrence is identical. Three modules, one setting — which is exactly the claim in the title.
M4 turns exactly one dial, and §4 below is that dial flip, run.
M6’s row is a different stream. Its keys are layer inputs and its values are output surprises (M6 §1); the memory being written is \(W\), not a token store. Same three dials, one level down — which is why its optimizer cell reads “this dial is the subject.”
NoteQ: Isn’t \(\phi\) a fourth dial? It sits right there in the recurrence.
It is a real choice — NL §5 lists it alongside \(\alpha_t\) in the sentence quoted in §1 (“identity, polynomial kernels, etc.”), and M3 §6 spends a section on what it buys. So why three dials and not four?
Because \(\phi\) is not an independent axis: it is part of the write rule’s objective. The write rule is one gradient step on a similarity, and \(\phi\)is that similarity’s kernel — \(\mathrm{sim}(\mathbf{q},\mathbf{k})=\phi(\mathbf{q})^\top\phi(\mathbf{k})\) (M3 §3). Choosing \(\phi\) is choosing what “this key matches that key” means; you cannot set it without having set the objective.
The sharper reason is that \(\phi\)does not change what the recurrence does — only how well it does it. Swap \(\phi=\text{identity}\) for \(\phi=\mathrm{elu}+1\) and M1 becomes M3: same write rule, same gate, same optimizer, same failure mode, more capacity (M2’s capacity law; M3 §6). Contrast the write dial, where the same swap turns a memory that cannot update an association into one that can (§4). One changes the quantity; the others change the kind.
So: \(\phi\) is a knob on the write dial rather than a dial of its own. Keep it in the recurrence — it is load-bearing, and §4’s demo passes it as an argument like the rest — just don’t expect it to buy you a new mechanism. (This is the same three-dial count M1 §4’s callout gives, and it is the count the tracks bind to.)
4. The dial board
The argument so far is a table, and a table is a promise. Here is the promise kept: one implementation of the master recurrence, dials as keyword arguments, reproducing three earlier modules by flipping them.
The two asserts are the test. Under orthogonal keys the Hebbian and delta settings must produce the identical matrix — not close, identical — because with orthogonal keys the memory has nothing stored at \(\mathbf{k}_t\) to correct, so the residual \(\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\)is\(\mathbf{v}_t\). Revisit a key and they must come apart, with delta recovering the second value exactly.
import torchdef dial_board(keys, vals, *, write="hebb", alpha=1.0, phi=lambda k: k, beta=1.0):"""One recurrence. Three dials. Returns the final memory S.""" d_k, d_v = phi(keys[0]).shape[0], vals[0].shape[0] S = torch.zeros(d_v, d_k)for k, v inzip(keys, vals): pk = phi(k) w = v if write =="hebb"else v - S @ pk # "delta" S = alpha * S + beta * torch.outer(w, pk)return Storch.manual_seed(0)keys = torch.eye(4) # orthogonal keys: both rules must agreevals = torch.randn(4, 3)S_hebb = dial_board(keys, vals, write="hebb")S_delta = dial_board(keys, vals, write="delta")assert torch.allclose(S_hebb, S_delta, atol=1e-6), "orthogonal keys: the dials coincide"print("orthogonal keys -> max |S_hebb - S_delta| =", (S_hebb - S_delta).abs().max().item())keys2 = torch.stack([keys[0], keys[0]]) # same key twice: they must divergevals2 = torch.stack([vals[0], vals[1]])S_h = dial_board(keys2, vals2, write="hebb")S_d = dial_board(keys2, vals2, write="delta")print("hebb recall of k0:", (S_h @ keys[0])) # piles both values onprint("delta recall of k0:", (S_d @ keys[0])) # overwrites -- recovers vals[1]print("the two values written to k0:", vals2[0], vals2[1])assert torch.allclose(S_d @ keys[0], vals2[1], atol=1e-6)
orthogonal keys -> max |S_hebb - S_delta| = 0.0
hebb recall of k0: tensor([ 2.1094, -1.3780, -3.5774])
delta recall of k0: tensor([ 0.5684, -1.0845, -1.3986])
the two values written to k0: tensor([ 1.5410, -0.2934, -2.1788]) tensor([ 0.5684, -1.0845, -1.3986])
Read the two prints together. On orthogonal keys the settings are not merely similar — the difference is exactly zero, because the delta rule’s correction term is zero by construction: writing to \(\mathbf{e}_i\) touches only column \(i\), so when the write to \(\mathbf{e}_j\) arrives, \(\mathbf{S}_{t-1}\mathbf{e}_j\) is still the zero vector and the residual collapses to \(\mathbf{v}_t\). A memory that never revisits a key has nothing to correct, and the fancier rule does nothing at all.
Revisit one key and the same two settings tell opposite stories: hebb returns \(\mathbf{v}_0+\mathbf{v}_1\) (it can only accumulate), delta returns \(\mathbf{v}_1\) (it read \(\mathbf{v}_0\), wrote the difference, and the sum telescoped). That is M4’s entire argument — the failure in its §1, the fix in its §2, the payoff in its §5 — reproduced here by changing one string in one function call.
Which is the point of the module. M4 is not a new model. It is write="delta".
The same board reaches M1 and M3 the same way — this time on the \(\phi\) knob and the gate.
import torch.nn.functional as Ftorch.manual_seed(1)K, V = torch.randn(6, 8), torch.randn(6, 5)# M1: phi = identity, alpha = 1, hebbian write -> S = sum_j v_j k_j^TS_m1 = dial_board(K, V, write="hebb")assert torch.allclose(S_m1, V.t() @ K, atol=1e-5)# M3: the ONLY change is the feature map on the key (M3 section 3's collapsed state)m3_phi =lambda k: F.elu(k) +1.0S_m3 = dial_board(K, V, write="hebb", phi=m3_phi)assert torch.allclose(S_m3, V.t() @ m3_phi(K), atol=1e-5) # M3's parallel form, same numbersprint("M1 -> M3 is one argument: same write rule, same gate; only the key's feature map moved")# The gate dial: the one the whole spine left at 1.# Orthogonal keys, so recall is exact and the fade is the only thing moving.E, Ve = torch.eye(5), torch.randn(5, 3)for a in (1.0, 0.5): S = dial_board(E, Ve, write="hebb", alpha=a)assert torch.allclose(S @ E[0], a**4* Ve[0], atol=1e-6) # oldest: 4 writes have passed over itassert torch.allclose(S @ E[4], Ve[4], atol=1e-6) # newest: written last, untouchedprint(f"alpha={a}: oldest key recalls {a**4:.4f} x v_0 | newest key recalls 1.0000 x v_4")print("the gate fades every association at once, by AGE -- a global forget, not a targeted edit")
M1 -> M3 is one argument: same write rule, same gate; only the key's feature map moved
alpha=1.0: oldest key recalls 1.0000 x v_0 | newest key recalls 1.0000 x v_4
alpha=0.5: oldest key recalls 0.0625 x v_0 | newest key recalls 1.0000 x v_4
the gate fades every association at once, by AGE -- a global forget, not a targeted edit
The gate lines are the honest version of the table’s flattest column. The fade is exactly \(\alpha^{\text{age}}\): at \(\alpha=1\) being old costs an association nothing, and at \(\alpha=0.5\) the first thing written comes back at \(1/16\) of its size while the last comes back untouched — nothing about which key it was, only how long ago. That is the difference from M4’s erase factor in one line: the gate is indexed by age, the delta rule by key.
Nothing in M1–M6 needed this dial, so the spine never turned it. But it is in the recurrence, and §5 is where the rest of the literature turns it.
5. The write-rule zoo
Most rows are the master recurrence with a different \(\text{write}_t\); the last three bend its shape, each in a way worth knowing — a frame is only useful if you know where it strains. The last column is where the course covers it — a dash means it is named here and nowhere else, which makes this table the course’s roadmap as well as its map.
Write rule
\(\text{write}_t\)
Gate \(\alpha_t\)
Effect
Named models
Covered in
Hebbian / sum
\(\mathbf{v}_t\)
\(1\)
append only
linear attention (Katharopoulos et al., 2020), fast-weight programmers (Schmidhuber, 1992)
M1, M2, M3
gated Hebbian
\(\mathbf{v}_t\)
constant, or input-dependent
append + global forget
RetNet (Sun et al., 2023; \(\gamma\) fixed per head), Mamba-2 (Dao & Gu, 2024; \(a_t\) input-dependent)
The first five rows are the master recurrence exactly. Here is how each of the last three departs from it, because each departure says something:
Oja’s rule (NL §5, Eq. 66) subtracts its correction from \(\phi(\mathbf{k}_t)\), not from \(\mathbf{v}_t\). The outer product is still there, but the corrected factor is the key — so it is not a setting of \(\text{write}_t\) at all. NL derives it as one gradient step on a Hebbian objective plus a \(\lVert\mathcal{M}^\top\mathbf{v}_t\rVert^2\) term (Eq. 67), which is where the norm control comes from; the same source reports it underperforming delta-rule models in practice.
The Omega rule (Atlas §3; NL §5, Eq. 68) writes a sum of outer products per step, one per token in a window — so one token is no longer one rank-one write. At \(c=1\) it collapses back to the delta row.
Optimizer-shaped writes are the row that stops being a formula. Once \(\text{write}_t\) is “run an optimizer on an inner objective,” the memory can be a whole network and the write can take many steps — the frame survives, but its content moves into the optimizer dial. That is the Nested Learning track’s departure point.
The three dashes are the roadmap: the gate dial is where this course has the most left to say.
The names and dates carry a second reading of the same table. Nothing in it arrives from nowhere: the delta row is Widrow and Hoff’s 1960 rule carried into a fast-weight memory sixty years later; the gated-delta row is that row plus the gate the RetNet/Mamba-2 line had already built, by authors who could name both parents; Oja’s 1982 rule waited forty years for a sequence model to want it. Every row is earlier rows composed — the three-dial claim told through people instead of dials.
6. How tracks attach
The hub is now a contract, and it is short.
A track is a self-contained spoke: one paper or one family, hanging off this module. To attach, a track states its dial setting relative to §3’s table — which of the three dials it turns, from what to what — and departs from there. That is the whole interface. A track may name foundations modules as prerequisites (its landing page says which); it may never depend on another track, because tracks are siblings, not a sequence.
Numbering follows from that. Foundations are M1–M7 permanently; a track numbers its own modules in its own namespace. Adding a track renumbers nothing and invalidates no link, which is the property the chain M1→M6 could not have offered.
Two consequences worth stating plainly. First, a paper that turns no dial in this table does not belong in this course — the table is the scope. Second, a paper that turns one is already half-explained by the time its landing page names it, because you have run that dial’s neighbours by hand.
Exit check
Self-test — question, then the answer we’d give.
1. State any model in this course as a setting of the three dials. Pick the recurrence \(\mathbf{S}_t=\alpha_t\mathbf{S}_{t-1}+\beta_t(\text{write}_t)\phi(\mathbf{k}_t)^\top\) and fill three slots. DeltaNet:\(\text{write}_t=\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\), \(\alpha_t=1\), one plain step at learned rate \(\beta_t\). Linear attention:\(\text{write}_t=\mathbf{v}_t\), \(\alpha_t=1\), one plain step at rate \(1\) — and M1 is the same setting with \(\phi=\text{identity}\), M2 the same setting with \(\mathbf{v}_t=W_v\mathbf{x}_t\). SRWM: the DeltaNet setting, with \(\mathbf{k},\mathbf{q},\beta\) and both values read out of the memory itself. If you can say the three slots, you can say the model.
2. Which dial does M4 turn? The write rule, and only it: \(\mathbf{v}_t\to\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\). The gate stays at \(1\) and the optimizer stays a single plain step. §4 ran the flip: identical to Hebbian on orthogonal keys, and the only thing that can recover the second value on a revisited one.
3. Which dial does M6 turn? The optimizer — one plain step \(\to\) momentum, DGD, Adam, Muon. But M6’s content is what turning it means: an optimizer is itself a memory over a second stream (gradients, keyed by layer inputs, valued by output surprises), with its own write rule and its own gate. Turning the optimizer dial is setting the three dials again, one level down. That is the nesting the tracks build on.
Next → the tracks. The foundations are done; the spine has no seventh mechanism waiting. From here the course goes wide instead of long: each track picks up the recurrence at this table, names its dial setting, and follows one line of work as far as it goes. The first is the Nested Learning track — which turns all three dials, and the write dial so far that the memory stops being a matrix and becomes a whole model, trained at inference time.
