Cheatsheet

Every formula in the course, one line each.

Compact reference for the course. One line per formula, in module order, then the unifying view.

Symbols. \(\mathbf{x}_t\) input token · \(\mathbf{k},\mathbf{v},\mathbf{q}\) key/value/query · \(\mathcal{M},\mathbf{S}\) memory (fast weights) · \(\phi\) feature map · \(\beta\) write strength · \(\alpha\) forget gate · \(\bar{\mathbf{v}}\) value currently stored at a key. Slow weights \(W_\bullet\) (trained, frozen in the forward pass); fast weights \(\mathcal{M}\) (rewritten per token, reset per sequence).

M1 — Associative memory & the Hebbian write

\[\text{write (Hebbian): } \mathcal{M}\mathrel{+}=\mathbf{v}\mathbf{k}^\top \qquad \text{read: } \mathbf{y}=\mathcal{M}\mathbf{q}\]

One Hebbian write = one gradient step on the dot-product objective \(-\langle\mathcal{M}\mathbf{k},\mathbf{v}\rangle\) (since \(\nabla_{\mathcal{M}}\langle\mathcal{M}\mathbf{k},\mathbf{v}\rangle=\mathbf{v}\mathbf{k}^\top\)) — ascent on the similarity.
Recall a stored key: \(\mathcal{M}\mathbf{q}=\sum_i\mathbf{v}_i(\mathbf{k}_i^\top\mathbf{q})\) → clean iff keys orthonormal; else crosstalk.
Capacity \(\approx\) #near-orthogonal keys \(\approx\) effective rank of \(\{\mathbf{k}_i\}\).
Backprop is the same write: a linear layer’s gradient is \(\partial\mathcal{L}/\partial W=\boldsymbol{\delta}\mathbf{x}^\top\) — key = input \(\mathbf{x}\), value = \(-\eta\boldsymbol{\delta}\) (negative output surprise, \(\boldsymbol{\delta}=\partial\mathcal{L}/\partial\mathbf{y}\)). NL §3.1, Eq. 8 names \(\boldsymbol\delta\) the Local Surprise Signal → the M6 thread.

M2 — Fast Weight Programmers (FWP)

\[\mathbf{k}_t,\mathbf{v}_t,\mathbf{q}_t=W_k\mathbf{x}_t,\,W_v\mathbf{x}_t,\,W_q\mathbf{x}_t \qquad \mathcal{M}_t=\mathcal{M}_{t-1}+\mathbf{v}_t\mathbf{k}_t^\top \qquad \mathbf{y}_t=\mathcal{M}_t\mathbf{q}_t\]

Slow net programs fast weights via outer products of self-invented \(\mathbf{k},\mathbf{v}\). Nesting: one net’s outputs = another’s weights.
Capacity law: \(\text{capacity}\approx d_{\text{key}}\) (refined: effective rank of the key set).
Loss reaches slow weights through \(\mathcal{M}\) (inner loop = autograd through the memory; outer loop = GD on \(W_\bullet\)).

M3 — Linear attention is fast weights

\[\text{softmax: } \mathbf{o}_t=\frac{\sum_{j\le t}\exp(\mathbf{q}_t^\top\mathbf{k}_j/\sqrt d)\,\mathbf{v}_j}{\sum_{j\le t}\exp(\mathbf{q}_t^\top\mathbf{k}_j/\sqrt d)} \qquad\xrightarrow[\text{drop softmax}]{\text{general kernel}}\qquad \mathbf{o}_t=\frac{\sum_{j\le t}\mathrm{sim}(\mathbf{q}_t,\mathbf{k}_j)\,\mathbf{v}_j}{\sum_{j\le t}\mathrm{sim}(\mathbf{q}_t,\mathbf{k}_j)}\]

Same shape both sides — the only change is \(\exp(\cdot)\to\mathrm{sim}(\cdot)\).

Factorize \(\mathrm{sim}(\mathbf{q},\mathbf{k})=\phi(\mathbf{q})^\top\phi(\mathbf{k})\) → pull \(\phi(\mathbf{q}_t)\) out of the sum → collapse:

\[\mathbf{S}_t=\sum_{j\le t}\mathbf{v}_j\phi(\mathbf{k}_j)^\top,\quad \mathbf{z}_t=\sum_{j\le t}\phi(\mathbf{k}_j) \qquad\Rightarrow\qquad \mathbf{o}_t=\frac{\mathbf{S}_t\,\phi(\mathbf{q}_t)}{\mathbf{z}_t^\top\phi(\mathbf{q}_t)}\]

\[\boxed{\ \mathbf{S}_t=\mathbf{S}_{t-1}+\mathbf{v}_t\,\phi(\mathbf{k}_t)^\top\ }\quad\text{(= M2's write with }\mathbf{k}\mapsto\phi(\mathbf{k})\text{)}\]

	form	cost	use
parallel	\(\mathbf{O}=\mathrm{tril}(\Phi_Q\Phi_K^\top)\,V\)	\(O(N^2)\)	train
recurrent	carry \(\mathbf{S}_t=\mathbf{S}_{t-1}+\mathbf{v}_t\phi(\mathbf{k}_t)^\top\)	\(O(1)\)/token	infer

Identical because reassociation + state-Jacobian \(\partial\mathbf{S}_t/\partial\mathbf{S}_{t-1}=I\) (Mamba-2 SSD duality).
\(\phi=\mathrm{elu}+1>0\): (1) keeps the normalizer positive, (2) raises \(\dim\phi\) → capacity.
Cost of linearizing: lossy superposition. Softmax = \(\exp\) kernel = infinite-dim feature map → exact recall, uncollapsible. Finite \(\phi\) = finite-rank approx → collapsible, blurry.

M4 — Delta rule / DeltaNet

\[\bar{\mathbf{v}}_t=\mathcal{M}_{t-1}\phi(\mathbf{k}_t)\quad(\text{read first})\qquad \boxed{\ \mathcal{M}_t=\mathcal{M}_{t-1}+\beta_t(\mathbf{v}_t-\bar{\mathbf{v}}_t)\,\phi(\mathbf{k}_t)^\top\ }\qquad \beta_t=\sigma(W_\beta\mathbf{x}_t)\]

Write new − remove old (FWP Eq. 23): \(\mathcal{M}_t=\mathcal{M}_{t-1}+\mathbf{v}^{\text{new}}_t\phi(\mathbf{k}_t)^\top-\bar{\mathbf{v}}_t\phi(\mathbf{k}_t)^\top\), with \(\mathbf{v}^{\text{new}}_t=\beta_t\mathbf{v}_t+(1-\beta_t)\bar{\mathbf{v}}_t\).
= one SGD step on \(\tfrac12\lVert\mathcal{M}\phi(\mathbf{k})-\mathbf{v}\rVert^2\), since \(\nabla_{\mathcal{M}}=(\bar{\mathbf{v}}-\mathbf{v})\phi(\mathbf{k})^\top\); step size \(=\beta_t\).
Erase-then-write (matrix form): \(\mathcal{M}_t=\mathcal{M}_{t-1}\big(I-\beta_t\phi(\mathbf{k}_t)\phi(\mathbf{k}_t)^\top\big)+\beta_t\mathbf{v}_t\phi(\mathbf{k}_t)^\top\) — a targeted erase along \(\mathbf{k}_t\), vs. the global decay gate \(\mathcal{M}_t=(1-\beta)\mathcal{M}_{t-1}+\beta\mathbf{v}_t\mathbf{k}_t^\top\).
DeltaNet = linear Transformer with this write; wins in overcapacity / revisited-key regimes.
Decoupled erase/write (GDN2, M4 §6): \(\mathcal{M}_t=\mathcal{M}_{t-1}\big(I-(\mathbf{b}_t\odot\mathbf{k}_t)\mathbf{k}_t^\top\big)+(\mathbf{w}_t\odot\mathbf{v}_t)\mathbf{k}_t^\top\) — channel-wise erase gate \(\mathbf{b}_t\) (key side) and write gate \(\mathbf{w}_t\) (value side) replace the single \(\beta_t\); tie \(\mathbf{b}_t{=}\mathbf{w}_t{=}\beta_t\mathbf{1}\) ⇒ the coupled form above. Gated DeltaNet-2 → KDA (gates tied, decay channel-wise) → gated delta (decay scalar).

M5 — Meta-learning & self-reference

\[\text{MAML inner: } \theta'_i=\theta-\alpha\nabla_\theta\mathcal{L}^{\text{sup}}_{T_i}(\theta) \qquad \text{outer: } \min_\theta\sum_i\mathcal{L}^{\text{qry}}_{T_i}(\theta'_i),\quad \theta\leftarrow\theta-\eta\nabla_\theta\textstyle\sum_i\mathcal{L}^{\text{qry}}_{T_i}(\theta'_i)\]

Outer step differentiates through the inner step → second-order (Hessian) term; first-order MAML drops it. \(\theta\) = launchpad, not a task solution.
SRWM (one matrix modifies itself): \([\mathbf{y}_t,\mathbf{k}_t,\mathbf{q}_t,\beta_t]=W_{t-1}\varphi(\mathbf{x}_t)\), \(\ \bar{\mathbf{v}}_t=W_{t-1}\varphi(\mathbf{k}_t)\), \(\ \mathbf{v}_t=W_{t-1}\varphi(\mathbf{q}_t)\),

\[\boxed{\ W_t=W_{t-1}+\sigma(\beta_t)(\mathbf{v}_t-\bar{\mathbf{v}}_t)\otimes\varphi(\mathbf{k}_t)\ }\]

= M4’s delta write with the producer tied to \(W\) itself (programmer = program). Untie it → DeltaNet.
Levels = update frequency. Two routes to learning-to-learn: adapt weights (MAML/SRWM) vs adapt activations (RL²/L2RL, in-context, no inner gradient).

M6 — Optimizers as associative memories

Backprop = a Hebbian write (NL Eq. 29): for \(\mathbf y=W\hat{\mathbf x}\),

\[\frac{\partial\mathcal L}{\partial W}=\boldsymbol\delta\,\hat{\mathbf x}^\top \quad\Rightarrow\quad W_{t+1}=W_t-\eta\,\boldsymbol\delta_t\hat{\mathbf x}_t^\top=W_t+\eta\,\mathbf u_t\hat{\mathbf x}_t^\top,\qquad \mathbf u_t=-\boldsymbol\delta_t=-\tfrac{\partial\mathcal L}{\partial\mathbf y_t}\]

key \(=\hat{\mathbf x}\) (layer input), value \(=-\boldsymbol\delta\) (output surprise; zero at a perfect prediction). Training = the matrix accumulating (input → surprise) outer products.
Self-referential (≠ linear attention): the value \(\mathbf u_t\) is a function of the current \(W_t\) — the memory generates its own write targets. This is what licenses a delta-rule optimizer (§4.5).

Momentum = a memory over the gradient stream (Eq. 32–34): \(\ \mathbf m_{t+1}=\alpha_t\mathbf m_t-\eta_t\nabla_{W_t}\mathcal L,\quad W_{t+1}=W_t+\mathbf m_{t+1}\).

\(\alpha{=}1\) ⇒ GD on \(\langle\mathbf m\hat{\mathbf x},\boldsymbol\delta\rangle\) (a value-less AM compressing past gradients); \(\alpha{\neq}1\) ⇒ + an \(\ell_2\) forget term. Two-level: inner learns \(\mathbf m\), outer steps \(W\).
Capacity limit (§4.3): the EMA is a low-pass filter — the \(i\)-th-back gradient carries weight \(\beta^i(1-\beta)\), so the \(n\) most recent carry \(1-\beta^n\). At \(\beta{=}0.9\) the last 7 carry 50%, the last 44 carry 99% → optimizer-level catastrophic forgetting on orthogonal tasks. (§4.3 prints 6 and 43; its cumulative sum runs over \(t+1\) terms, so it quotes the index, not the count — M6 Q1 derives it.)

The delta-rule fix, one level up:

\[\text{DGD (§4.5, Eq. 57): }\ \boxed{\,W_{t+1}=W_t(\mathbf I-\eta'_t\hat{\mathbf x}_t\hat{\mathbf x}_t^\top)-\eta'_t\nabla_{W_t}\mathcal L\,}\quad \eta'_t=\tfrac{\eta_t}{1+\eta_t}\]

= DeltaNet’s matrix form (M4) applied to weights: erase what \(\hat{\mathbf x}\) predicts, then write. Bounded/interference-resistant under correlated inputs (SGD diverges at aggressive lr, DGD stays bounded).
Delta Momentum (§4.4, Eq. 48–49): the delta rule on the buffer — \(\mathbf m_{i+1}=\mathbf m_i(\alpha_{i+1}-\nabla\mathcal L^\top\nabla\mathcal L)-\eta_t\mathbf P_i\nabla\mathcal L\) — M4’s erase factor with \(\mathbf k=\nabla\mathcal L\), collapsed to a scalar; the buffer compresses the gradient→preconditioner mapping (internal objective \(\lVert\mathbf m\nabla\mathcal L^\top-\mathbf P_i\rVert_2^2\)), so the write is preconditioned by \(\mathbf P_i\) (next bullet; \(\mathbf P=\mathbf I\) ⇒ the plain buffer). Actively forgets stale gradients (wins on time-varying curvature).

Preconditioning & deeper buffers: \(\ W_{t+1}=W_t-\eta P^{-1}\mathbf g\) (Eq. 38) = an AM mapping gradients into a better coordinate system (Eq. 39–40). Adam = the optimal element-wise \(\ell_2\) memory of the gradient variance; Muon = momentum buffer + a Newton–Schulz non-linearity on its output (\(\sigma=\mathrm{NS}_k\), Eq. 52); DMGD = MLP-valued buffer.

Payoff: the optimizer is an AM over a second stream (gradients, keyed by layer inputs, valued by surprises) — same problems (interference, finite capacity), same fixes (delta, forget gate, depth). Every write rule from M1–M5 is now available to it.

NL-1 — Test-time learning (TTT · Titans · Miras · Atlas)

TTT — the hidden state is a model \(f_W\); the write = a GD step on a self-supervised loss:

\[\ell(W;x_t)=\lVert f_W(\mathbf{k}_t)-\mathbf{v}_t\rVert^2,\qquad W_t=W_{t-1}-\eta\nabla_W\ell(W_{t-1};x_t),\qquad \mathbf{y}_t=f_{W_t}(\mathbf{q}_t)\]

Linear \(f_W\), one step, \(\eta=\tfrac12\) ⇒ M4’s delta rule. MLP \(f_W\) ⇒ deep memory (more capacity, nonlinear maps).

Titans — the write becomes GD + momentum + weight decay:

\[\boxed{\,M_t=(1-\alpha_t)M_{t-1}+S_t,\quad S_t=\eta_t S_{t-1}-\theta_t\nabla\ell(M_{t-1};x_t)\,}\]

\(\nabla\ell\) = momentary surprise · \(\eta_t S_{t-1}\) = momentum / past surprise · \((1-\alpha_t)\) = adaptive forgetting (bounds \(\lVert M\rVert\), recency).

Miras — the design space, four axes: memory architecture · attentional bias (the inner objective) · retention gate · optimizer. Every model above, plus linear attention / DeltaNet / softmax, is a point in it. The bias need not be \(\ell_2\) (Huber / \(\ell_p\) → robust to outliers). The write rule is an optimizer at test time — M7’s three dials with the write pulled apart into architecture and objective.

Atlas — takes Miras’ axes and moves two at once: the Omega rule (memorize a window, §3.2 Eq. 9) + a Muon-style step (§5, Eqs. 32–33):

\[\min_{\mathcal M}\!\!\sum_{i=t-c+1}^{t}\!\!\gamma_i^{(t)}\lVert \mathcal M(\mathbf{k}_i)-\mathbf{v}_i\rVert_2^2,\qquad \mathcal M_t=\alpha_t \mathcal M_{t-1}-\eta_t\,\mathrm{NS}_k(\mathcal S_t)\]

\(c{=}1\) ⇒ the delta rule; larger \(c\) ⇒ context memorization. \(\mathrm{NS}_k\) = Newton–Schulz orthogonalization (whitens the update → approximate second-order information, not a second-order method). Atlas’ own model OmegaNet adds polynomial \(\phi\) on keys/queries (§3.1) — at the update rule (Eq. 10), not in the objective above.

NL-2 — Levels & the Continuum Memory System

Level = update frequency (NL Def. 2): the unit of time is one update over one data point; \(f_A\) = updates per unit. Ordering:

\[A\succ B\iff f_A>f_B\ \ \textbf{or}\ \ (f_A=f_B\ \text{and}\ B\ \text{is computed from}\ A)\quad(\text{dependency tiebreak}).\]

Same \(f\) ⇒ same level; higher level ⇒ lower \(f\). Equal \(f\) + independent ⇒ same level, parallel.
Level counts: MLP+SGD = 1 · +momentum = 2 (the optimizer adds a level) · linear attention = 2 · linear attention + momentum = 3.
Transformer = the two frequency extremes: attention re-solved every token ⇒ \(f=\infty\); the MLP frozen in-sequence ⇒ \(f=0\); the middle empty.

Stability–plasticity tradeoff: a single update period \(C\) is either plastic (small \(C\) → learns new, forgets old) or stable (large \(C\) → retains, can’t adapt) — never both ⇒ motivates a spectrum.

Continuum Memory System (CMS, Eq. 70–71): a chain of MLP memories at \(k\) frequencies \(f_1,\dots,f_k\),

\[\mathbf y_t=\text{MLP}^{(f_k)}\!\big(\dots\text{MLP}^{(f_1)}(\mathbf x_t)\big),\qquad \theta^{(f_\ell)}\ \text{updates every}\ C^{(\ell)}:=\tfrac{\max_i C^{(i)}}{f_\ell}\ \text{steps (chunked grad)}.\]

Fast levels = plastic / short-term; slow levels = persistent. Resists forgetting: knowledge forgotten by a fast block survives in slower blocks and circles back via backprop (§7.1). A Transformer is the degenerate \(k{=}1,\ f{=}0\) case.
Written \(f_1,\dots,f_k\) without an asserted order: §3.2 makes level 1 the fastest, but §7.1’s own \(C^{(\ell)}\) and Eq. 73 make \(f_1\) the slowest. The paper never reconciles them, and nothing here turns on it (NL-2 §4).
Computational depth = add levels, not layers (latent within-pass depth; distinct from token-space CoT, which NL does not discuss).

NL-3 — HOPE (assembling it all)

HOPE = self-modifying Titans → CMS (NL §8.3). Two complementary halves: self-modifying Titans = small capacity, expressive learned rule; CMS = large capacity, simple rule, persistent.

Self-modifying memory (vs merely adaptive) — §8.1’s three rungs:

plain (Eq. 76): \(\mathbf k,\mathbf v,\mathbf q,\eta_t,\alpha_t\) from static projections.
adaptive (Eq. 79): each produced by its own in-context memory — the paper is explicit that this “still lacks self-modification”.
self-modifying (Eq. 83–85): each memory generates its own target \(\hat{\mathbf v}_{\square,t}=\mathcal M_{\square,t-1}(\mathbf v_t)\) for \(\square\in\{\mathbf k,\mathbf v,\eta,\alpha,\text{memory}\}\) → it learns how to update. Self-referential in state and rule; backprop is self-referential in state only.
Inner objective \(\lVert\mathcal M(\mathbf k)-\hat{\mathbf v}\rVert^2\); inner optimizer = DGD + weight decay (M6 — tokens are correlated); every memory is a 2-layer residual MLP (Eq. 89): \(\mathcal M_\square(\cdot)=(\cdot)+W_{\square,1}\sigma(W_{\square,2}(\cdot))\). \(\mathbf q_t=\mathbf x_tW_q\) is the only non-adaptive projection (§8.1).

No collapse — the inner self-target is wrapped in an outer loss with ground truth. Four anchors: (1) the output predicts real tokens via the LM head; (2) the value-generators and initial states are meta-learned; (3) the inputs are real data; (4) the update is a bounded gated step from the init. Same MAML/TTT trick (M5, NL-1): a label-free inner procedure validated by a labeled outer loop. Only (2) is the paper’s own argument — it never raises collapse as a failure mode (NL-3 §3).

Each piece ↔︎ module: store (M1) · projections/FWP (M2) · linear-attention write (M3) · delta correction (M4) · self-reference/meta-init (M5) · DGD inside (M6) · test-time write + gate (NL-1) · CMS continuum (NL-2). HOPE = NL’s existence proof that “add nested levels” yields one uniform feed-forward memory stack that continually learns.

NL-4 — End-to-end TTT (TTT-E2E)

The objective every memory above shares — KV Binding, layer-local (TTT-E2E Eq. 7):

\[\ell_t^{(l)}\big(W_{t-1}^{(l)}\big)=\big\lVert g\big(\theta_K^{(l)}\mathbf{x}_t^{(l)};W_{t-1}^{(l)}\big)-\theta_V^{(l)}\mathbf{x}_t^{(l)}\big\rVert^2\]

§2.4.1 fn. 1 (crediting TTT): linear \(g\) ⇒ DeltaNet (M4); gradient taken at \(W_0^{(l)}\) instead of \(W_{t-1}^{(l)}\) ⇒ linear attention (M3); \(\ell_t^{(l)}\) by non-parametric kernel estimator ⇒ self-attention. One construction, three knobs — M7’s thesis reached from outside the course.

E2E at test time — delete the proxy, descend on the real loss (Eqs. 1–2, generalized to mini-batches in Eq. 5):

\[\ell_t(W)=\texttt{CE}\big(f(x_{t-1};W),x_t\big),\qquad W_i=W_{i-1}-\eta\,\frac1b\!\!\sum_{t=(i-1)b+1}^{ib}\!\!\nabla\ell_t(W_{i-1})\]

No \(\theta_K\), no \(\theta_V\), no key, no value, no per-layer target — one loss at the end of the network. Memory = the MLP weights of the last \(\tfrac14\) of blocks; reading it = the forward pass. Gate \(\alpha_t=1\) (Eq. 5 has no decay term); inner optimizer = plain SGD, one step per mini-batch.
The KVB write is blind to \(x_t\) (Eq. 7 is built from \(\mathbf{x}_t^{(l)}\) alone); the E2E write is not. That is the whole of “E2E at test time”.

E2E at training time — meta-learn \(W_0\) through the inner steps (M5’s MAML), not around them:

\[\mathcal{L}(W_0;X)=\frac1T\sum_{i=1}^{T/b}\ \sum_{t=(i-1)b+1}^{ib}\ell_t(W_{i-1})\qquad\text{vs.}\qquad \mathcal{L}_{\texttt{naive}}(W_0;X)=\frac1T\sum_{t=1}^{T}\ell_t(W_0)\]

\(W_{i-1}\) vs \(W_0\) is the entire difference: Eq. 6 scores the model after TTT ⇒ gradients of gradients; Eq. 4 pretends the inner steps won’t happen (= dynamic evaluation, Krause et al.). Cost: 3.4× slower training at 8K (FlashAttention has no grad-of-grad), 1.2× faster at 128K.
Division of labour (§5): updated weights = long-term memory (lossy, compressed); sliding window \(k\) = short-term memory (exact). \(k\ge b\) required, else the model is “a bigram again within each batch” (§2.3). Forgetting lives in the architecture, not the gate.
Defaults \(k{=}8\)K, \(b{=}1\)K, last \(1/4\) of blocks (§2.3, §3.2). Scales with context like full attention at 3B/164B tokens, 2.7× faster at 128K — but loses badly on NIAH (§3.5): compression ≠ recall. Both halves predate it in Clark et al., 2022; what is new is linear complexity and the scaling evidence (§4.3).

SSM-1 — The linear state-space model

Continuous system \(h'(t)=Ah(t)+Bx(t)\), \(y(t)=Ch(t)\); zero-order hold discretizes it (S4D §3.1):

\[\bar A=\exp(\Delta A),\qquad \bar B=(\Delta A)^{-1}\big(\exp(\Delta A)-I\big)\,\Delta B\]

\[\boxed{\ h_t=\bar A\,h_{t-1}+\bar B\,x_t,\quad y_t=C\,h_t\ }\qquad\xrightarrow{\text{unroll}}\qquad y=\bar K * x,\quad \bar K_\ell=C\,\bar A^{\,\ell}\,\bar B\]

Gate \(=\bar A\). M3 multiplied the old state by \(1\); this multiplies by \(\bar A=\exp(\Delta A)\), a fixed, structured per-channel decay — M7’s gate \(\alpha_t\) off its \(1\) pin (SSM-1 §2).
Two schedules, one function (S4 Eq. 5): train as the convolution \(y=\bar K*x\) (\(O(L\log L)\) by FFT), infer by the recurrence (\(O(1)\)/token) — M3’s parallel/recurrent duality again.
Why a structured \(A\) remembers (HiPPO): a random diagonal forgets; S4D-Lin \(a_n=-\tfrac12+i\pi n\) tiles the frequency axis so a fixed-size state reconstructs the far past. Diagonal kernel \(\bar K_\ell=\sum_n C_n\bar A_n^{\,\ell}\bar B_n\) (S4D Eq. 7).

SSM-2 — Selectivity (Mamba / S6)

Make the SSM parameters input-dependent (Mamba §3.2, Alg. 2), with \(A\) left a learned parameter:

\[\Delta_t=\operatorname{softplus}\big(\text{param}+s_\Delta(x_t)\big),\quad B_t=s_B(x_t),\quad C_t=s_C(x_t),\qquad \bar A_t=\exp(\Delta_t A)\]

At a single mode \(N{=}1,\,A{=}-1\) the recurrence is exactly a gate (Mamba Thm 1):

\[\boxed{\ h_t=(1-g_t)\,h_{t-1}+g_t\,x_t,\qquad g_t=1-e^{-\Delta_t}=\sigma\big(\text{param}+s_\Delta(x_t)\big)\ }\]

\(\Delta_t\) is the gate and step size (Mamba §3.5.2): large \(\Delta_t\Rightarrow\bar A_t\to0\), write the token and reset; small \(\Delta_t\Rightarrow\bar A_t\to1\), hold the state and ignore it. Same gate M4, the LSTM and GRU use.
Dial move: the gate \(\alpha_t=\bar A_t\) goes from fixed-structured (SSM-1) to input-dependent, channel-wise (diagonal \(A\)). The write stays additive.
Cost: input-dependence makes the influence matrix \(W_{t,j}=C_t\big(\prod_{k=j+1}^{t}\bar A_k\big)B_j\) non-Toeplitz — no single convolution, so a parallel scan replaces the FFT.

SSM-3 — Mamba-2 and the duality (SSD)

Restrict the state matrix to a scalar, \(A_t=a_tI\) (Mamba-2 §5.1). The whole sequence-to-sequence map collapses to one matrix:

\[\boxed{\ M=L\circ(CB^\top),\qquad L_{ij}=\prod_{k=j+1}^{i}a_k\ \ (j\le i)\ }\qquad y_i=\sum_{j\le i}L_{ij}\,(C_i^\top B_j)\,x_j\]

= masked linear attention: query \(q_i=C_i\), key \(k_j=B_j\), value \(v_j=x_j\), decay mask \(L\) — M3’s \((L\circ(QK^\top))V\) with the causal ones replaced by running decays (Mamba-2 Cor. 5.1). Set every \(a_t=1\) ⇒ M3 exactly.
Structured state-space duality: one \(M\), a linear \(O(T)\) recurrence and a quadratic \(O(T^2)\) attention form; the chunked block algorithm interpolates (diagonal blocks = intra-chunk attention, off-diagonal = recurrence over chunk states) — why Mamba-2 runs 2–8× over Mamba’s scan (§6).
Dial: additive (Hebbian) write, selective scalar gate \(\alpha_t=a_t\) — M7 §5’s gated-Hebbian row, beside RetNet. “Transformers are SSMs” = scalar-identity SSM \(\equiv\) 1-semiseparable masked linear attention (not softmax).

SSM-4 — The gate, from none to selective

The master recurrence with the gate \(\alpha_t\) swept across its range, and the write axis (additive vs. delta) crossing it — M7 §5’s zoo read down the gate column.

\[\mathbf{S}_t=\alpha_t\,\mathbf{S}_{t-1}+\beta_t\,(\text{write}_t)\,\phi(\mathbf{k}_t)^\top,\qquad \mathbf{o}_t\propto\mathbf{S}_t\,\phi(\mathbf{q}_t)\]

Two independent bits set the gate: input-dependent or not, scalar or channel-wise. Crossed with the write:

gate \(\alpha_t\)	additive write \(\mathbf{v}_t\)	delta write \(\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\)
none, \(\alpha_t=1\)	linear attention	DeltaNet
fixed scalar \(\gamma\)	RetNet	—
fixed diagonal \(\bar A\)	S4 (SSM-1)	—
input-dep. scalar \(a_t\)	Mamba-2 (SSM-3)	Gated DeltaNet
input-dep. channel-wise \(\mathrm{diag}(\boldsymbol\alpha_t)\)	Mamba / S6 (SSM-2), GLA	KDA

Endpoints are exact identities: gate \(1\) = M3 linear attention; a scalar gate = SSM-3’s masked attention. A channel-wise gate is a memory neither can express (fade one channel, hold another).
Two of M7’s three dials (write × gate). The optimizer dial is the one this track never turns — every cell here is a single plain step; turn it and the memory becomes a network trained at test time (another track’s departure).

The unifying view

Master recurrence — the load-bearing equation across the modern literature:

\[\mathbf{S}_t=\alpha_t\,\mathbf{S}_{t-1}+\beta_t\,(\text{write}_t)\,\phi(\mathbf{k}_t)^\top,\qquad \mathbf{o}_t\propto\mathbf{S}_t\,\phi(\mathbf{q}_t)\]

Three dials — write rule (what goes in) · gate \(\alpha_t\) (what stays) · optimizer (how the write is applied). Almost every named model is a setting of them.

Write-rule zoo (\(\text{write}_t\) in the master recurrence):

rule	update	effect
sum (linear attention)	\(\mathbf{S}\mathrel{+}=\mathbf{v}\phi(\mathbf{k})^\top\)	append only
gated	\(\mathbf{S}=\alpha\mathbf{S}+\beta\,\mathbf{v}\phi(\mathbf{k})^\top\)	append + global forget
delta (DeltaNet)	\(\mathbf{S}\mathrel{+}=\beta(\mathbf{v}-\mathbf{S}\phi(\mathbf{k}))\phi(\mathbf{k})^\top\)	targeted edit
gated delta (GDN)	\(\mathbf{S}=\alpha\mathbf{S}+\beta(\mathbf{v}-\mathbf{S}\phi(\mathbf{k}))\phi(\mathbf{k})^\top\)	edit + forget

M7 derives the recurrence from M3, states every foundations module as a dial setting, runs the dial board, and extends the zoo to models the course does not otherwise cover. This section is the one-line summary; M7 is where it is built.