Continual Learning
Memory, fast weights, and learning at many timescales — the mechanisms, rebuilt by hand.
I made this course because I could not follow the Nested Learning paper as fully as I wanted to.
It is a good paper and a dense one — a unifying theory of memory, and a new architecture built from it — resting on decades of work it can only compress into notation. Reading it straight through means reconstructing that work as you go.
So I rebuilt the pieces, one at a time, in notebooks you can run and change. They proved worth separating from the paper that sent me looking for them: the foundations stand on their own, and Nested Learning is one track on top. Each module starts from a limitation of an earlier one, introduces a single idea in response, and cites the paper that first published it. The order is conceptual, not chronological — the delta rule comes fourth but dates from 1960.
It teaches the mechanism family behind continual learning: associative memory, fast weights, test-time learning, and updates at many timescales, each built from scratch, one at a time.
It is not a survey of the continual-learning literature. There is no EWC, no replay, no class-incremental benchmark here. What you get instead is the small set of mechanisms that let a model keep changing after training.
Who it’s for
Anyone who has read — or bounced off — a paper like DeltaNet, Titans, or Nested Learning and wants the machinery underneath rather than a summary of it. You should be able to read PyTorch and follow linear algebra and backprop; where a module needs more, it points you at a primer that builds the piece from scratch.
By the end, you should be able to:
- state a model as a dial setting — linear attention, DeltaNet, SRWM, Titans and HOPE are one recurrence with three dials set differently, and you can say which;
- come to a new paper in this area with its parts already built, and see which dial it turns;
- read the notation back to a mechanism, and to the failure that motivated it.
The modules
Start at M1 — Associative memory & the Hebbian write and run it. Every notebook carries an Open in Colab badge, so there is nothing to install, and nothing in the course needs a GPU.
Primers — 2 notebooks, optional, off the spine
Read one when a module points you at it, not before.
| Primer | What it covers |
|---|---|
| Linear algebra | Orthogonality, eigen/SVD/condition number, effective rank — the recurring math, each with a runnable demo. |
| Backpropagation | How a gradient is actually computed: the chain rule, a scalar autograd engine from scratch, torch.autograd, and higher-order/meta-gradients. |
Foundations — 7 notebooks, in order
Each of M2–M6 repairs a failure of an earlier module, so the order is the argument; M7 collapses all six into one object.
| # | Module | The question it answers |
|---|---|---|
| M1 | Associative memory & the Hebbian write | What is the single primitive everything else is built from? |
| M2 | Fast Weight Programmers | Where do the keys, values, and queries come from? |
| M3 | Linear attention is fast weights | What happens to attention if you delete the softmax? |
| M4 | The delta rule / DeltaNet | How does a memory edit an association instead of piling onto it? |
| M5 | Meta-learning & self-reference | Can a network learn a good place to start its learning steps — or run them on itself? |
| M6 | Optimizers as associative memories | Why are momentum, Adam, and Muon also memories? |
| M7 | The three dials | M1 through M6 built six mechanisms. How many ideas is that, really? |
Tracks — self-contained spokes off M7
Each track names the dial it turns and follows one line of work as far as it goes. Tracks are independent of each other — read one, all, or none.
| Track | The dial it turns | The question it answers |
|---|---|---|
| Nested Learning 3 modules + 4 asides |
All three — the write rule until it stops being a formula and becomes a learner | What if every memory in the model is a learner, and the only thing that separates them is how often they update? |
Adding a track renumbers nothing — the contract is in M7 §6 and spelled out in the README.
The spine in one breath
Start with one primitive: a memory you write with an outer product and read with a matrix–vector product (M1). Make the writes programmable and you get fast-weight programmers (M2), which are linear attention (M3). Swap the additive write for an error-correcting one and you get the delta rule / DeltaNet (M4). Let the memory supply its own keys, values and write rate — ask who writes the writer — and you get meta-learning and self-referential weights (M5). Then notice the optimizer was an associative memory over the gradient stream all along, and that M4’s fix applies to it too (M6). Collapse all six into one recurrence with three dials (M7) — and the tracks take it from there, where the argument and the calendar finally coincide: the Nested Learning track follows TTT → Titans → Miras → Atlas in the order they were published, one team’s twelve months, each paper repairing the gap the last left open.
Every formula in the course, one line each, is on the cheatsheet.
Running it
Zero install. Click any notebook’s Open in Colab badge and it runs; torch, numpy, and matplotlib are already there. Nothing in the course needs a GPU.
Or run it locally. Clone the repo and uv sync in courses/continual-learning/. The dependencies are exactly torch, numpy, matplotlib, and jupyter.
Runtimes. Most notebooks finish in seconds. Three take minutes: M5 and the Nested Learning track’s image-TTT and HOPE-training asides. M5 and image-TTT each fetch a ~10 MB dataset once (Omniglot, MNIST) and cache it locally; everything else is self-contained.
If a notebook feels absurdly slow, cap the thread count. Several demos use tensors small enough that torch’s intra-op parallelism costs far more than it saves, and the penalty grows with core count. On a 14-core machine the HOPE-training aside took 9 minutes with default threads and 23 seconds under OMP_NUM_THREADS=1 — same output, 24× apart. Worth doing for the whole course:
OMP_NUM_THREADS=1 uv run jupyter labSources
The papers the modules are built around, by public URL.
| Paper | Where it lands |
|---|---|
| Nested Learning | M1, M2, M6, M7, and the whole Nested Learning track |
| FWP | M2; DeltaNet in M4 |
| Linear Transformers | M1, M3, M7 |
| MAML | M5; the backpropagation primer |
| SRWM | M5 |
| RL² | M5 |
| L2RL | M5 |
| TTT | NL-1 |
| Titans | NL-1 |
| Miras | NL-1 |
| Atlas | NL-1 |
Where each mechanism came from. These are the papers the course rebuilds; they are not the papers that invented the pieces. Each module names and links the originating work where the mechanism is introduced — Hebb and the 1972 trio for the outer-product write (M1), Schmidhuber’s fast-weight programmers (M2), Widrow–Hoff 1960 for the delta rule and Yang et al. for its gated descendant (M4), Polyak 1964 through AdaGrad, Adam, Shampoo and Muon (M6), and the write-rule zoo in M7, where RetNet, Mamba-2, Gated DeltaNet and OjaNet each link their own paper as dial settings the course does not otherwise cover. Where priority is genuinely contested — and in this lineage it often is — the module says so rather than picking a winner.
Licence
Prose, figures, and course text: CC BY 4.0. Code — every code cell in every notebook: MIT. Full text in LICENSE.