Skip to main content

Cheatsheet

Every formula in the course, one line each.

Compact reference for the course. One line per formula, in module order, then the unifying view.

Symbols. \(\mathbf{x}_t\) input token · \(\mathbf{k},\mathbf{v},\mathbf{q}\) key/value/query · \(\mathcal{M},\mathbf{S}\) memory (fast weights) · \(\phi\) feature map · \(\beta\) write strength · \(\alpha\) forget gate · \(\bar{\mathbf{v}}\) value currently stored at a key. Slow weights \(W_\bullet\) (trained, frozen in the forward pass); fast weights \(\mathcal{M}\) (rewritten per token, reset per sequence).


M1 — Associative memory & the Hebbian write

\[\text{write (Hebbian): } \mathcal{M}\mathrel{+}=\mathbf{v}\mathbf{k}^\top \qquad \text{read: } \mathbf{y}=\mathcal{M}\mathbf{q}\]

  • One Hebbian write = one gradient step on the dot-product objective \(-\langle\mathcal{M}\mathbf{k},\mathbf{v}\rangle\) (since \(\nabla_{\mathcal{M}}\langle\mathcal{M}\mathbf{k},\mathbf{v}\rangle=\mathbf{v}\mathbf{k}^\top\)) — ascent on the similarity.
  • Recall a stored key: \(\mathcal{M}\mathbf{q}=\sum_i\mathbf{v}_i(\mathbf{k}_i^\top\mathbf{q})\) → clean iff keys orthonormal; else crosstalk.
  • Capacity \(\approx\) #near-orthogonal keys \(\approx\) effective rank of \(\{\mathbf{k}_i\}\).
  • Backprop is the same write: a linear layer’s gradient is \(\partial\mathcal{L}/\partial W=\boldsymbol{\delta}\mathbf{x}^\top\) — key = input \(\mathbf{x}\), value = \(-\eta\boldsymbol{\delta}\) (negative output surprise, \(\boldsymbol{\delta}=\partial\mathcal{L}/\partial\mathbf{y}\)). NL §3.1, Eq. 8 names \(\boldsymbol\delta\) the Local Surprise Signal → the M6 thread.

M2 — Fast Weight Programmers (FWP)

\[\mathbf{k}_t,\mathbf{v}_t,\mathbf{q}_t=W_k\mathbf{x}_t,\,W_v\mathbf{x}_t,\,W_q\mathbf{x}_t \qquad \mathcal{M}_t=\mathcal{M}_{t-1}+\mathbf{v}_t\mathbf{k}_t^\top \qquad \mathbf{y}_t=\mathcal{M}_t\mathbf{q}_t\]

  • Slow net programs fast weights via outer products of self-invented \(\mathbf{k},\mathbf{v}\). Nesting: one net’s outputs = another’s weights.
  • Capacity law: \(\text{capacity}\approx d_{\text{key}}\) (refined: effective rank of the key set).
  • Loss reaches slow weights through \(\mathcal{M}\) (inner loop = autograd through the memory; outer loop = GD on \(W_\bullet\)).

M3 — Linear attention is fast weights

\[\text{softmax: } \mathbf{o}_t=\frac{\sum_{j\le t}\exp(\mathbf{q}_t^\top\mathbf{k}_j/\sqrt d)\,\mathbf{v}_j}{\sum_{j\le t}\exp(\mathbf{q}_t^\top\mathbf{k}_j/\sqrt d)} \qquad\xrightarrow[\text{drop softmax}]{\text{general kernel}}\qquad \mathbf{o}_t=\frac{\sum_{j\le t}\mathrm{sim}(\mathbf{q}_t,\mathbf{k}_j)\,\mathbf{v}_j}{\sum_{j\le t}\mathrm{sim}(\mathbf{q}_t,\mathbf{k}_j)}\]

  • Same shape both sides — the only change is \(\exp(\cdot)\to\mathrm{sim}(\cdot)\).

Factorize \(\mathrm{sim}(\mathbf{q},\mathbf{k})=\phi(\mathbf{q})^\top\phi(\mathbf{k})\) → pull \(\phi(\mathbf{q}_t)\) out of the sum → collapse:

\[\mathbf{S}_t=\sum_{j\le t}\mathbf{v}_j\phi(\mathbf{k}_j)^\top,\quad \mathbf{z}_t=\sum_{j\le t}\phi(\mathbf{k}_j) \qquad\Rightarrow\qquad \mathbf{o}_t=\frac{\mathbf{S}_t\,\phi(\mathbf{q}_t)}{\mathbf{z}_t^\top\phi(\mathbf{q}_t)}\]

\[\boxed{\ \mathbf{S}_t=\mathbf{S}_{t-1}+\mathbf{v}_t\,\phi(\mathbf{k}_t)^\top\ }\quad\text{(= M2's write with }\mathbf{k}\mapsto\phi(\mathbf{k})\text{)}\]

form cost use
parallel \(\mathbf{O}=\mathrm{tril}(\Phi_Q\Phi_K^\top)\,V\) \(O(N^2)\) train
recurrent carry \(\mathbf{S}_t=\mathbf{S}_{t-1}+\mathbf{v}_t\phi(\mathbf{k}_t)^\top\) \(O(1)\)/token infer
  • Identical because reassociation + state-Jacobian \(\partial\mathbf{S}_t/\partial\mathbf{S}_{t-1}=I\) (Mamba-2 SSD duality).
  • \(\phi=\mathrm{elu}+1>0\): (1) keeps the normalizer positive, (2) raises \(\dim\phi\) → capacity.
  • Cost of linearizing: lossy superposition. Softmax = \(\exp\) kernel = infinite-dim feature map → exact recall, uncollapsible. Finite \(\phi\) = finite-rank approx → collapsible, blurry.

M4 — Delta rule / DeltaNet

\[\bar{\mathbf{v}}_t=\mathcal{M}_{t-1}\phi(\mathbf{k}_t)\quad(\text{read first})\qquad \boxed{\ \mathcal{M}_t=\mathcal{M}_{t-1}+\beta_t(\mathbf{v}_t-\bar{\mathbf{v}}_t)\,\phi(\mathbf{k}_t)^\top\ }\qquad \beta_t=\sigma(W_\beta\mathbf{x}_t)\]

  • Write new − remove old (FWP Eq. 23): \(\mathcal{M}_t=\mathcal{M}_{t-1}+\mathbf{v}^{\text{new}}_t\phi(\mathbf{k}_t)^\top-\bar{\mathbf{v}}_t\phi(\mathbf{k}_t)^\top\), with \(\mathbf{v}^{\text{new}}_t=\beta_t\mathbf{v}_t+(1-\beta_t)\bar{\mathbf{v}}_t\).
  • = one SGD step on \(\tfrac12\lVert\mathcal{M}\phi(\mathbf{k})-\mathbf{v}\rVert^2\), since \(\nabla_{\mathcal{M}}=(\bar{\mathbf{v}}-\mathbf{v})\phi(\mathbf{k})^\top\); step size \(=\beta_t\).
  • Erase-then-write (matrix form): \(\mathcal{M}_t=\mathcal{M}_{t-1}\big(I-\beta_t\phi(\mathbf{k}_t)\phi(\mathbf{k}_t)^\top\big)+\beta_t\mathbf{v}_t\phi(\mathbf{k}_t)^\top\) — a targeted erase along \(\mathbf{k}_t\), vs. the global decay gate \(\mathcal{M}_t=(1-\beta)\mathcal{M}_{t-1}+\beta\mathbf{v}_t\mathbf{k}_t^\top\).
  • DeltaNet = linear Transformer with this write; wins in overcapacity / revisited-key regimes.
  • Decoupled erase/write (GDN2, M4 §6): \(\mathcal{M}_t=\mathcal{M}_{t-1}\big(I-(\mathbf{b}_t\odot\mathbf{k}_t)\mathbf{k}_t^\top\big)+(\mathbf{w}_t\odot\mathbf{v}_t)\mathbf{k}_t^\top\) — channel-wise erase gate \(\mathbf{b}_t\) (key side) and write gate \(\mathbf{w}_t\) (value side) replace the single \(\beta_t\); tie \(\mathbf{b}_t{=}\mathbf{w}_t{=}\beta_t\mathbf{1}\) ⇒ the coupled form above. Gated DeltaNet-2 → KDA (gates tied, decay channel-wise) → gated delta (decay scalar).

M5 — Meta-learning & self-reference

\[\text{MAML inner: } \theta'_i=\theta-\alpha\nabla_\theta\mathcal{L}^{\text{sup}}_{T_i}(\theta) \qquad \text{outer: } \min_\theta\sum_i\mathcal{L}^{\text{qry}}_{T_i}(\theta'_i),\quad \theta\leftarrow\theta-\eta\nabla_\theta\textstyle\sum_i\mathcal{L}^{\text{qry}}_{T_i}(\theta'_i)\]

  • Outer step differentiates through the inner step → second-order (Hessian) term; first-order MAML drops it. \(\theta\) = launchpad, not a task solution.
  • SRWM (one matrix modifies itself): \([\mathbf{y}_t,\mathbf{k}_t,\mathbf{q}_t,\beta_t]=W_{t-1}\varphi(\mathbf{x}_t)\), \(\ \bar{\mathbf{v}}_t=W_{t-1}\varphi(\mathbf{k}_t)\), \(\ \mathbf{v}_t=W_{t-1}\varphi(\mathbf{q}_t)\),

\[\boxed{\ W_t=W_{t-1}+\sigma(\beta_t)(\mathbf{v}_t-\bar{\mathbf{v}}_t)\otimes\varphi(\mathbf{k}_t)\ }\]

  • = M4’s delta write with the producer tied to \(W\) itself (programmer = program). Untie it → DeltaNet.
  • Levels = update frequency. Two routes to learning-to-learn: adapt weights (MAML/SRWM) vs adapt activations (RL²/L2RL, in-context, no inner gradient).

M6 — Optimizers as associative memories

Backprop = a Hebbian write (NL Eq. 29): for \(\mathbf y=W\hat{\mathbf x}\),

\[\frac{\partial\mathcal L}{\partial W}=\boldsymbol\delta\,\hat{\mathbf x}^\top \quad\Rightarrow\quad W_{t+1}=W_t-\eta\,\boldsymbol\delta_t\hat{\mathbf x}_t^\top=W_t+\eta\,\mathbf u_t\hat{\mathbf x}_t^\top,\qquad \mathbf u_t=-\boldsymbol\delta_t=-\tfrac{\partial\mathcal L}{\partial\mathbf y_t}\]

  • key \(=\hat{\mathbf x}\) (layer input), value \(=-\boldsymbol\delta\) (output surprise; zero at a perfect prediction). Training = the matrix accumulating (input → surprise) outer products.
  • Self-referential (≠ linear attention): the value \(\mathbf u_t\) is a function of the current \(W_t\) — the memory generates its own write targets. This is what licenses a delta-rule optimizer (§4.5).

Momentum = a memory over the gradient stream (Eq. 32–34): \(\ \mathbf m_{t+1}=\alpha_t\mathbf m_t-\eta_t\nabla_{W_t}\mathcal L,\quad W_{t+1}=W_t+\mathbf m_{t+1}\).

  • \(\alpha{=}1\) ⇒ GD on \(\langle\mathbf m\hat{\mathbf x},\boldsymbol\delta\rangle\) (a value-less AM compressing past gradients); \(\alpha{\neq}1\) ⇒ + an \(\ell_2\) forget term. Two-level: inner learns \(\mathbf m\), outer steps \(W\).
  • Capacity limit (§4.3): the EMA is a low-pass filter — the \(i\)-th-back gradient carries weight \(\beta^i(1-\beta)\), so the \(n\) most recent carry \(1-\beta^n\). At \(\beta{=}0.9\) the last 7 carry 50%, the last 44 carry 99% → optimizer-level catastrophic forgetting on orthogonal tasks. (§4.3 prints 6 and 43; its cumulative sum runs over \(t+1\) terms, so it quotes the index, not the count — M6 Q1 derives it.)

The delta-rule fix, one level up:

\[\text{DGD (§4.5, Eq. 57): }\ \boxed{\,W_{t+1}=W_t(\mathbf I-\eta'_t\hat{\mathbf x}_t\hat{\mathbf x}_t^\top)-\eta'_t\nabla_{W_t}\mathcal L\,}\quad \eta'_t=\tfrac{\eta_t}{1+\eta_t}\]

  • = DeltaNet’s matrix form (M4) applied to weights: erase what \(\hat{\mathbf x}\) predicts, then write. Bounded/interference-resistant under correlated inputs (SGD diverges at aggressive lr, DGD stays bounded).
  • Delta Momentum (§4.4, Eq. 48–49): the delta rule on the buffer — \(\mathbf m_{i+1}=\mathbf m_i(\alpha_{i+1}-\nabla\mathcal L^\top\nabla\mathcal L)-\eta_t\mathbf P_i\nabla\mathcal L\) — M4’s erase factor with \(\mathbf k=\nabla\mathcal L\), collapsed to a scalar; the buffer compresses the gradient→preconditioner mapping (internal objective \(\lVert\mathbf m\nabla\mathcal L^\top-\mathbf P_i\rVert_2^2\)), so the write is preconditioned by \(\mathbf P_i\) (next bullet; \(\mathbf P=\mathbf I\) ⇒ the plain buffer). Actively forgets stale gradients (wins on time-varying curvature).

Preconditioning & deeper buffers: \(\ W_{t+1}=W_t-\eta P^{-1}\mathbf g\) (Eq. 38) = an AM mapping gradients into a better coordinate system (Eq. 39–40). Adam = the optimal element-wise \(\ell_2\) memory of the gradient variance; Muon = momentum buffer + a Newton–Schulz non-linearity on its output (\(\sigma=\mathrm{NS}_k\), Eq. 52); DMGD = MLP-valued buffer.

Payoff: the optimizer is an AM over a second stream (gradients, keyed by layer inputs, valued by surprises) — same problems (interference, finite capacity), same fixes (delta, forget gate, depth). Every write rule from M1–M5 is now available to it.

NL-1 — Test-time learning (TTT · Titans · Miras · Atlas)

TTT — the hidden state is a model \(f_W\); the write = a GD step on a self-supervised loss:

\[\ell(W;x_t)=\lVert f_W(\mathbf{k}_t)-\mathbf{v}_t\rVert^2,\qquad W_t=W_{t-1}-\eta\nabla_W\ell(W_{t-1};x_t),\qquad \mathbf{y}_t=f_{W_t}(\mathbf{q}_t)\]

  • Linear \(f_W\), one step, \(\eta=\tfrac12\) ⇒ M4’s delta rule. MLP \(f_W\) ⇒ deep memory (more capacity, nonlinear maps).

Titans — the write becomes GD + momentum + weight decay:

\[\boxed{\,M_t=(1-\alpha_t)M_{t-1}+S_t,\quad S_t=\eta_t S_{t-1}-\theta_t\nabla\ell(M_{t-1};x_t)\,}\]

  • \(\nabla\ell\) = momentary surprise · \(\eta_t S_{t-1}\) = momentum / past surprise · \((1-\alpha_t)\) = adaptive forgetting (bounds \(\lVert M\rVert\), recency).

Miras — the design space, four axes: memory architecture · attentional bias (the inner objective) · retention gate · optimizer. Every model above, plus linear attention / DeltaNet / softmax, is a point in it. The bias need not be \(\ell_2\) (Huber / \(\ell_p\) → robust to outliers). The write rule is an optimizer at test time — M7’s three dials with the write pulled apart into architecture and objective.

Atlas — takes Miras’ axes and moves two at once: the Omega rule (memorize a window, §3.2 Eq. 9) + a Muon-style step (§5, Eqs. 32–33):

\[\min_{\mathcal M}\!\!\sum_{i=t-c+1}^{t}\!\!\gamma_i^{(t)}\lVert \mathcal M(\mathbf{k}_i)-\mathbf{v}_i\rVert_2^2,\qquad \mathcal M_t=\alpha_t \mathcal M_{t-1}-\eta_t\,\mathrm{NS}_k(\mathcal S_t)\]

  • \(c{=}1\) ⇒ the delta rule; larger \(c\) ⇒ context memorization. \(\mathrm{NS}_k\) = Newton–Schulz orthogonalization (whitens the update → approximate second-order information, not a second-order method). Atlas’ own model OmegaNet adds polynomial \(\phi\) on keys/queries (§3.1) — at the update rule (Eq. 10), not in the objective above.

NL-2 — Levels & the Continuum Memory System

Level = update frequency (NL Def. 2): the unit of time is one update over one data point; \(f_A\) = updates per unit. Ordering:

\[A\succ B\iff f_A>f_B\ \ \textbf{or}\ \ (f_A=f_B\ \text{and}\ B\ \text{is computed from}\ A)\quad(\text{dependency tiebreak}).\]

  • Same \(f\) ⇒ same level; higher level ⇒ lower \(f\). Equal \(f\) + independent ⇒ same level, parallel.
  • Level counts: MLP+SGD = 1 · +momentum = 2 (the optimizer adds a level) · linear attention = 2 · linear attention + momentum = 3.
  • Transformer = the two frequency extremes: attention re-solved every token ⇒ \(f=\infty\); the MLP frozen in-sequence ⇒ \(f=0\); the middle empty.

Stability–plasticity tradeoff: a single update period \(C\) is either plastic (small \(C\) → learns new, forgets old) or stable (large \(C\) → retains, can’t adapt) — never both ⇒ motivates a spectrum.

Continuum Memory System (CMS, Eq. 70–71): a chain of MLP memories at \(k\) frequencies \(f_1,\dots,f_k\),

\[\mathbf y_t=\text{MLP}^{(f_k)}\!\big(\dots\text{MLP}^{(f_1)}(\mathbf x_t)\big),\qquad \theta^{(f_\ell)}\ \text{updates every}\ C^{(\ell)}:=\tfrac{\max_i C^{(i)}}{f_\ell}\ \text{steps (chunked grad)}.\]

  • Fast levels = plastic / short-term; slow levels = persistent. Resists forgetting: knowledge forgotten by a fast block survives in slower blocks and circles back via backprop (§7.1). A Transformer is the degenerate \(k{=}1,\ f{=}0\) case.
  • Written \(f_1,\dots,f_k\) without an asserted order: §3.2 makes level 1 the fastest, but §7.1’s own \(C^{(\ell)}\) and Eq. 73 make \(f_1\) the slowest. The paper never reconciles them, and nothing here turns on it (NL-2 §4).
  • Computational depth = add levels, not layers (latent within-pass depth; distinct from token-space CoT, which NL does not discuss).

NL-3 — HOPE (assembling it all)

HOPE = self-modifying Titans → CMS (NL §8.3). Two complementary halves: self-modifying Titans = small capacity, expressive learned rule; CMS = large capacity, simple rule, persistent.

Self-modifying memory (vs merely adaptive) — §8.1’s three rungs:

  • plain (Eq. 76): \(\mathbf k,\mathbf v,\mathbf q,\eta_t,\alpha_t\) from static projections.
  • adaptive (Eq. 79): each produced by its own in-context memory — the paper is explicit that this “still lacks self-modification”.
  • self-modifying (Eq. 83–85): each memory generates its own target \(\hat{\mathbf v}_{\square,t}=\mathcal M_{\square,t-1}(\mathbf v_t)\) for \(\square\in\{\mathbf k,\mathbf v,\eta,\alpha,\text{memory}\}\) → it learns how to update. Self-referential in state and rule; backprop is self-referential in state only.
  • Inner objective \(\lVert\mathcal M(\mathbf k)-\hat{\mathbf v}\rVert^2\); inner optimizer = DGD + weight decay (M6 — tokens are correlated); every memory is a 2-layer residual MLP (Eq. 89): \(\mathcal M_\square(\cdot)=(\cdot)+W_{\square,1}\sigma(W_{\square,2}(\cdot))\). \(\mathbf q_t=\mathbf x_tW_q\) is the only non-adaptive projection (§8.1).

No collapse — the inner self-target is wrapped in an outer loss with ground truth. Four anchors: (1) the output predicts real tokens via the LM head; (2) the value-generators and initial states are meta-learned; (3) the inputs are real data; (4) the update is a bounded gated step from the init. Same MAML/TTT trick (M5, NL-1): a label-free inner procedure validated by a labeled outer loop. Only (2) is the paper’s own argument — it never raises collapse as a failure mode (NL-3 §3).

Each piece ↔︎ module: store (M1) · projections/FWP (M2) · linear-attention write (M3) · delta correction (M4) · self-reference/meta-init (M5) · DGD inside (M6) · test-time write + gate (NL-1) · CMS continuum (NL-2). HOPE = NL’s existence proof that “add nested levels” yields one uniform feed-forward memory stack that continually learns.

NL-4 — End-to-end TTT (TTT-E2E)

The objective every memory above shares — KV Binding, layer-local (TTT-E2E Eq. 7):

\[\ell_t^{(l)}\big(W_{t-1}^{(l)}\big)=\big\lVert g\big(\theta_K^{(l)}\mathbf{x}_t^{(l)};W_{t-1}^{(l)}\big)-\theta_V^{(l)}\mathbf{x}_t^{(l)}\big\rVert^2\]

  • §2.4.1 fn. 1 (crediting TTT): linear \(g\) ⇒ DeltaNet (M4); gradient taken at \(W_0^{(l)}\) instead of \(W_{t-1}^{(l)}\) ⇒ linear attention (M3); \(\ell_t^{(l)}\) by non-parametric kernel estimator ⇒ self-attention. One construction, three knobs — M7’s thesis reached from outside the course.

E2E at test time — delete the proxy, descend on the real loss (Eqs. 1–2, generalized to mini-batches in Eq. 5):

\[\ell_t(W)=\texttt{CE}\big(f(x_{t-1};W),x_t\big),\qquad W_i=W_{i-1}-\eta\,\frac1b\!\!\sum_{t=(i-1)b+1}^{ib}\!\!\nabla\ell_t(W_{i-1})\]

  • No \(\theta_K\), no \(\theta_V\), no key, no value, no per-layer target — one loss at the end of the network. Memory = the MLP weights of the last \(\tfrac14\) of blocks; reading it = the forward pass. Gate \(\alpha_t=1\) (Eq. 5 has no decay term); inner optimizer = plain SGD, one step per mini-batch.
  • The KVB write is blind to \(x_t\) (Eq. 7 is built from \(\mathbf{x}_t^{(l)}\) alone); the E2E write is not. That is the whole of “E2E at test time”.

E2E at training time — meta-learn \(W_0\) through the inner steps (M5’s MAML), not around them:

\[\mathcal{L}(W_0;X)=\frac1T\sum_{i=1}^{T/b}\ \sum_{t=(i-1)b+1}^{ib}\ell_t(W_{i-1})\qquad\text{vs.}\qquad \mathcal{L}_{\texttt{naive}}(W_0;X)=\frac1T\sum_{t=1}^{T}\ell_t(W_0)\]

  • \(W_{i-1}\) vs \(W_0\) is the entire difference: Eq. 6 scores the model after TTT ⇒ gradients of gradients; Eq. 4 pretends the inner steps won’t happen (= dynamic evaluation, Krause et al.). Cost: 3.4× slower training at 8K (FlashAttention has no grad-of-grad), 1.2× faster at 128K.
  • Division of labour (§5): updated weights = long-term memory (lossy, compressed); sliding window \(k\) = short-term memory (exact). \(k\ge b\) required, else the model is “a bigram again within each batch” (§2.3). Forgetting lives in the architecture, not the gate.
  • Defaults \(k{=}8\)K, \(b{=}1\)K, last \(1/4\) of blocks (§2.3, §3.2). Scales with context like full attention at 3B/164B tokens, 2.7× faster at 128K — but loses badly on NIAH (§3.5): compression ≠ recall. Both halves predate it in Clark et al., 2022; what is new is linear complexity and the scaling evidence (§4.3).

SSM-1 — The linear state-space model

Continuous system \(h'(t)=Ah(t)+Bx(t)\), \(y(t)=Ch(t)\); zero-order hold discretizes it (S4D §3.1):

\[\bar A=\exp(\Delta A),\qquad \bar B=(\Delta A)^{-1}\big(\exp(\Delta A)-I\big)\,\Delta B\]

\[\boxed{\ h_t=\bar A\,h_{t-1}+\bar B\,x_t,\quad y_t=C\,h_t\ }\qquad\xrightarrow{\text{unroll}}\qquad y=\bar K * x,\quad \bar K_\ell=C\,\bar A^{\,\ell}\,\bar B\]

  • Gate \(=\bar A\). M3 multiplied the old state by \(1\); this multiplies by \(\bar A=\exp(\Delta A)\), a fixed, structured per-channel decay — M7’s gate \(\alpha_t\) off its \(1\) pin (SSM-1 §2).
  • Two schedules, one function (S4 Eq. 5): train as the convolution \(y=\bar K*x\) (\(O(L\log L)\) by FFT), infer by the recurrence (\(O(1)\)/token) — M3’s parallel/recurrent duality again.
  • Why a structured \(A\) remembers (HiPPO): a random diagonal forgets; S4D-Lin \(a_n=-\tfrac12+i\pi n\) tiles the frequency axis so a fixed-size state reconstructs the far past. Diagonal kernel \(\bar K_\ell=\sum_n C_n\bar A_n^{\,\ell}\bar B_n\) (S4D Eq. 7).

SSM-2 — Selectivity (Mamba / S6)

Make the SSM parameters input-dependent (Mamba §3.2, Alg. 2), with \(A\) left a learned parameter:

\[\Delta_t=\operatorname{softplus}\big(\text{param}+s_\Delta(x_t)\big),\quad B_t=s_B(x_t),\quad C_t=s_C(x_t),\qquad \bar A_t=\exp(\Delta_t A)\]

At a single mode \(N{=}1,\,A{=}-1\) the recurrence is exactly a gate (Mamba Thm 1):

\[\boxed{\ h_t=(1-g_t)\,h_{t-1}+g_t\,x_t,\qquad g_t=1-e^{-\Delta_t}=\sigma\big(\text{param}+s_\Delta(x_t)\big)\ }\]

  • \(\Delta_t\) is the gate and step size (Mamba §3.5.2): large \(\Delta_t\Rightarrow\bar A_t\to0\), write the token and reset; small \(\Delta_t\Rightarrow\bar A_t\to1\), hold the state and ignore it. Same gate M4, the LSTM and GRU use.
  • Dial move: the gate \(\alpha_t=\bar A_t\) goes from fixed-structured (SSM-1) to input-dependent, channel-wise (diagonal \(A\)). The write stays additive.
  • Cost: input-dependence makes the influence matrix \(W_{t,j}=C_t\big(\prod_{k=j+1}^{t}\bar A_k\big)B_j\) non-Toeplitz — no single convolution, so a parallel scan replaces the FFT.

SSM-3 — Mamba-2 and the duality (SSD)

Restrict the state matrix to a scalar, \(A_t=a_tI\) (Mamba-2 §5.1). The whole sequence-to-sequence map collapses to one matrix:

\[\boxed{\ M=L\circ(CB^\top),\qquad L_{ij}=\prod_{k=j+1}^{i}a_k\ \ (j\le i)\ }\qquad y_i=\sum_{j\le i}L_{ij}\,(C_i^\top B_j)\,x_j\]

  • = masked linear attention: query \(q_i=C_i\), key \(k_j=B_j\), value \(v_j=x_j\), decay mask \(L\) — M3’s \((L\circ(QK^\top))V\) with the causal ones replaced by running decays (Mamba-2 Cor. 5.1). Set every \(a_t=1\) ⇒ M3 exactly.
  • Structured state-space duality: one \(M\), a linear \(O(T)\) recurrence and a quadratic \(O(T^2)\) attention form; the chunked block algorithm interpolates (diagonal blocks = intra-chunk attention, off-diagonal = recurrence over chunk states) — why Mamba-2 runs 2–8× over Mamba’s scan (§6).
  • Dial: additive (Hebbian) write, selective scalar gate \(\alpha_t=a_t\) — M7 §5’s gated-Hebbian row, beside RetNet. “Transformers are SSMs” = scalar-identity SSM \(\equiv\) 1-semiseparable masked linear attention (not softmax).

SSM-4 — The gate, from none to selective

The master recurrence with the gate \(\alpha_t\) swept across its range, and the write axis (additive vs. delta) crossing it — M7 §5’s zoo read down the gate column.

\[\mathbf{S}_t=\alpha_t\,\mathbf{S}_{t-1}+\beta_t\,(\text{write}_t)\,\phi(\mathbf{k}_t)^\top,\qquad \mathbf{o}_t\propto\mathbf{S}_t\,\phi(\mathbf{q}_t)\]

Two independent bits set the gate: input-dependent or not, scalar or channel-wise. Crossed with the write:

gate \(\alpha_t\) additive write \(\mathbf{v}_t\) delta write \(\mathbf{v}_t-\mathbf{S}_{t-1}\phi(\mathbf{k}_t)\)
none, \(\alpha_t=1\) linear attention DeltaNet
fixed scalar \(\gamma\) RetNet —
fixed diagonal \(\bar A\) S4 (SSM-1) —
input-dep. scalar \(a_t\) Mamba-2 (SSM-3) Gated DeltaNet
input-dep. channel-wise \(\mathrm{diag}(\boldsymbol\alpha_t)\) Mamba / S6 (SSM-2), GLA KDA
  • Endpoints are exact identities: gate \(1\) = M3 linear attention; a scalar gate = SSM-3’s masked attention. A channel-wise gate is a memory neither can express (fade one channel, hold another).
  • Two of M7’s three dials (write × gate). The optimizer dial is the one this track never turns — every cell here is a single plain step; turn it and the memory becomes a network trained at test time (another track’s departure).

The unifying view

Master recurrence — the load-bearing equation across the modern literature:

\[\mathbf{S}_t=\alpha_t\,\mathbf{S}_{t-1}+\beta_t\,(\text{write}_t)\,\phi(\mathbf{k}_t)^\top,\qquad \mathbf{o}_t\propto\mathbf{S}_t\,\phi(\mathbf{q}_t)\]

Three dials — write rule (what goes in) · gate \(\alpha_t\) (what stays) · optimizer (how the write is applied). Almost every named model is a setting of them.

Write-rule zoo (\(\text{write}_t\) in the master recurrence):

rule update effect
sum (linear attention) \(\mathbf{S}\mathrel{+}=\mathbf{v}\phi(\mathbf{k})^\top\) append only
gated \(\mathbf{S}=\alpha\mathbf{S}+\beta\,\mathbf{v}\phi(\mathbf{k})^\top\) append + global forget
delta (DeltaNet) \(\mathbf{S}\mathrel{+}=\beta(\mathbf{v}-\mathbf{S}\phi(\mathbf{k}))\phi(\mathbf{k})^\top\) targeted edit
gated delta (GDN) \(\mathbf{S}=\alpha\mathbf{S}+\beta(\mathbf{v}-\mathbf{S}\phi(\mathbf{k}))\phi(\mathbf{k})^\top\) edit + forget

M7 derives the recurrence from M3, states every foundations module as a dial setting, runs the dial board, and extends the zoo to models the course does not otherwise cover. This section is the one-line summary; M7 is where it is built.