NL-4 — End-to-end TTT: was the proxy ever necessary?
The question: every memory in this track learns to predict a value from a key. What if it trained on the real task instead?
Fourth module of the Nested Learning track. Prerequisites: foundations M1–M7 and NL-1. Runs on CPU in seconds; PyTorch throughout.
NL-1 through NL-3 turned the write dial until the memory became a learner: TTT trains \(f_W\) at inference, Titans gives that training an optimizer, Miras makes its objective an explicit axis, HOPE stacks the result on a continuum of timescales.
They also share something none of them argues for, because from inside the lineage it is not a choice — it is the air. Every memory you have built in this track learns the same thing: predict this token’s value from its key, \(\lVert f_W(\mathbf{k}_t)-\mathbf{v}_t\rVert^2\). The keys and values are minted by projections the outer loop trains, the loss is local to a layer, and it looks inevitable because it was reverse-engineered from self-attention — which stores exactly those pairs.
TTT-E2E (Tandon et al., Dec 2025) names that assumption from outside, calls it KV Binding, files this whole track under it, and then argues the binding was never the point. Its inner loop trains on the loss the model is actually graded on: the next token. This module is that argument, run.
Objective
After this module you should be able to:
Name the objective every memory in NL-1–NL-3 shares — KV Binding, \(\ell_t^{(l)}(W)=\lVert g(\theta_K^{(l)}\mathbf{x}_t^{(l)};W)-\theta_V^{(l)}\mathbf{x}_t^{(l)}\rVert^2\) — and say why it is a proxy rather than the task.
State the two senses of “end-to-end” TTT-E2E claims, and which loop each one fixes: the inner loop on the real next-token loss (test time), the outer loop on the loss after TTT (training time).
Show that a KVB write and an E2E write are different writes — the proxy write is provably blind to the next token — and say what that costs and buys.
Explain why the naive training objective \(\mathcal{L}_{\texttt{naive}}\) is a mismatch, and demonstrate that an initialization meta-learned through the inner steps beats one that never took any.
Say where this leaves the track’s dial setting: the write rule’s objective was a dial all along, and Miras’ axis was pointing at it.
Why it exists (the assumption it removes)
Every module in this track fixed a mechanism. This one questions an objective, so it is worth being precise about what is and is not under attack.
Nothing in NL-1–NL-3 is wrong. TTT-MLP does store more than a matrix; Titans’ momentum does help; HOPE does assemble. The question is narrower and sharper: all of them optimize a loss the model is not graded on. The user does not care whether \(f_W(\mathbf{k}_t)\) lands on \(\mathbf{v}_t\). The user cares about the next token. KV Binding is a stand-in — chosen because self-attention’s cache is a key–value store, so a memory that replaces attention was assumed to need attention’s bookkeeping.
TTT-E2E’s claim is that the assumption was inherited rather than earned, and that once you drop it the design gets simpler: no \(\theta_K\), no \(\theta_V\), no per-layer loss, no multi-head split, no LoRA on the state. What remains is a Transformer with sliding-window attention that keeps training on its own context — “our method is derived purely under the formulation of a continual learning problem, with minimal changes to the architecture” (§4.2).
Reading
TTT-E2E — §2.1–2.2 (the method and the meta-learning), §2.3 (mini-batch + window), §2.4 (the derivation from TTT-KVB, which is the one to read if you have finished NL-1).
Behind you: NL-1 §1 (the objective this module names), M5 §2 (MAML — meta-learning an initialization through a gradient step), M6 (test-time GD), M7 §3 (the dial table this lands on).
1. TTT-KVB — the bucket, named from outside
TTT-E2E’s §2.4.1 gives the family a name and a formula. Given input embeddings \(\mathbf{x}_t^{(l)}\) at layer \(l\), a TTT-KVB layer takes a gradient step on its Eq. 7:
where \(g\) is usually an MLP and \(\theta_K,\theta_V\) are outer-loop parameters. That is NL-1 §1’s \(\lVert f_W(\mathbf{k}_t)-\mathbf{v}_t\rVert^2\) with the projections written out. The paper’s own list of what sits in this bucket — “This learning objective, later known as KV Binding, has been the core component in many popular variants of TTT” — runs MesaNet, Titans, and Nested Learning (§2.4.1), and elsewhere it notes that Titans and Nested Learning follow the same construction (§3.1). This track has been inside one bucket for three modules.
Before pulling on it, notice what the bucket contains. TTT-E2E’s footnote 1 collects, crediting TTT (Sun et al., 2024) for the result:
“when \(g\) is a linear model, TTT-KVB recovers DeltaNet; when the gradient in Equation 7 is taken w.r.t. \(W_0^{(l)}\) instead of \(W_{t-1}^{(l)}\), TTT-KVB recovers linear attention; and when \(\ell_t^{(l)}\) is learned using a non-parametric kernel estimator instead of a parametric model, TTT-KVB recovers self-attention.”
Read that against M7 §3’s table. One construction, three knobs — how expressive is \(g\), which weights the gradient is taken at, parametric or not — and out fall M4, M3, and the softmax attention of M3 §1. M7 argued the modern literature is one recurrence with dials; here a paper from outside the Nested Learning group reaches for the same move to situate itself. It is the frame’s best outside evidence — and also the setup for the punchline, because the very next thing TTT-E2E does is leave the construction that generated all three.
2. The key step — E2E at test time
“The key step is to replace their layer-wise reconstruction loss with the standard next-token prediction loss” (§2.4).
The replacement is its Eqs. 1–2, and it has nothing in it:
No key. No value. No \(\theta_K\), no \(\theta_V\), no per-layer target — “Our inner loop directly optimizes the next-token prediction loss at the end of the network” (§1). One loss, at the end of the whole network, on the token that actually comes next. Deleting the projections is not a saving they went looking for; it falls out, because without a layer-local reconstruction there is nothing for \(\theta_K\) and \(\theta_V\) to be for.
The demo below is the difference in its smallest form. Same memory, same weights, same context token — two objectives, and the question is only whether the resulting writes differ. The discriminating test is the one the word “end-to-end” actually names: change the next token and see which write notices.
import mathimport torchimport torch.nn.functional as FV, D, H, T =16, 32, 64, 64# vocab, embedding dim, memory hidden dim, context lengthETA =1.0# inner-loop step sizedef suffix_mlp(W, e):"""The memory: an MLP over token embeddings, its weights batched one-per-sequence. This is the paper's hidden state -- 'our hidden state takes the form of regular MLP layers' (Sec 3.7). Reading it is just the forward pass; there is no key-value interface. """ W1, b1, W2, b2 = Wreturn torch.relu(torch.bmm(e, W1) + b1[:, None]) @ W2 + b2[:, None]def per_sequence(W0, n):"""One private copy of the memory per sequence -- TTT is per test instance."""return [w.unsqueeze(0).expand(n, *w.shape) for w in W0]def init_memory(gen):return [(torch.randn(D, H, generator=gen) / D **0.5).requires_grad_(), torch.zeros(H, requires_grad=True), (torch.randn(H, V, generator=gen) / H **0.5).requires_grad_(), torch.zeros(V, requires_grad=True)]gen = torch.Generator().manual_seed(1)emb = torch.randn(V, D, generator=gen) *0.5W0 = init_memory(gen)theta_K = torch.randn(D, D, generator=gen) / D **0.5# KVB's key projection (Eq. 7)theta_V = torch.randn(D, V, generator=gen) / D **0.5# KVB's value projection (Eq. 7)e_prev = emb[torch.tensor([3])][:, None] # x_{t-1}: the token we condition ondef e2e_write(next_token):"""TTT-E2E (Eq. 1): one step on the real next-token loss at the END of the network.""" W = per_sequence(W0, 1) loss = F.cross_entropy(suffix_mlp(W, e_prev).reshape(1, V), torch.tensor([next_token]))return torch.autograd.grad(loss, W)def kvb_write(next_token):"""TTT-KVB (Eq. 7): one step on a LAYER-LOCAL reconstruction loss. Note the signature takes next_token -- and the body never uses it. That is not an oversight in the demo; it is the objective. """ W = per_sequence(W0, 1) k, v = e_prev @ theta_K, e_prev @ theta_V loss = ((suffix_mlp(W, k) - v) **2).sum()return torch.autograd.grad(loss, W)flat =lambda g: torch.cat([t.reshape(-1) for t in g])a, b =5, 11# two candidate next tokenskvb_a, kvb_b = kvb_write(a), kvb_write(b)e2e_a, e2e_b = e2e_write(a), e2e_write(b)# "End-to-end at test time", as two asserts.assert torch.equal(flat(kvb_a), flat(kvb_b)), "the proxy write must be blind to the next token"assertnot torch.allclose(flat(e2e_a), flat(e2e_b)), "the E2E write must depend on it"print(f"next token = {a} vs next token = {b}:")print(f" KVB proxy write moves by {(flat(kvb_a) - flat(kvb_b)).norm():.2e} <- bit-identical")print(f" TTT-E2E write moves by {(flat(e2e_a) - flat(e2e_b)).norm():.4f}")print(f"\ncos(E2E write, KVB write) = "f"{F.cosine_similarity(flat(e2e_a), flat(kvb_a), dim=0).item():+.4f}"" -- near-orthogonal: two different writes into the same weights")
next token = 5 vs next token = 11:
KVB proxy write moves by 0.00e+00 <- bit-identical
TTT-E2E write moves by 6.3737
cos(E2E write, KVB write) = +0.0076 -- near-orthogonal: two different writes into the same weights
The first number is the module in one line. The KVB write is bit-identical whichever token comes next, because \(x_t\) does not appear anywhere in Eq. 7 — the objective is built entirely from \(\mathbf{x}_t^{(l)}\), the current token’s own embedding, projected two ways. A KVB memory writes what a token is. An E2E memory writes what a token predicts.
The cosine says the same thing structurally: the two gradients are near-orthogonal, so this is not a matter of degree. They are unrelated directions in weight space.
Now, the honest half — because “blind to the next token” is not by itself an indictment. Self-supervision is supposed to be label-free; that is the trick M5 and NL-1 both turn, and a KVB write is not useless. It is useful exactly to the extent the outer loop has arranged for it to be: \(\theta_K\) and \(\theta_V\) are trained, so the outer loop can shape the proxy until stepping on it happens to help the real loss. That indirection is the whole design. TTT-E2E’s bet is that a proxy the outer loop must be taught to make useful is worse than an objective that was the real one to begin with — and, per its Table 1, replacing the loss “significantly improves performance in language modeling” (§2.4.3).
3. E2E at training time — the initialization is the other half
The inner loop is now honest, and it is still not enough. Ask where \(W_0\) — the weights TTT starts from — comes from.
The obvious answer is “pre-training”, and it is the wrong one. Pre-training optimizes the loss of the model as it stands, but the model will not stand still; it will be trained on the context before it is asked anything. So there are two candidate training objectives — its Eqs. 3 and 4 — and they differ in exactly one place:
\(W_{t-1}\) versus \(W_0\). The first scores the model after TTT, so it must differentiate through the inner gradient steps — M5’s MAML move, create_graph=True, gradients of gradients. The second pretends the inner steps will not happen. TTT-E2E’s argument against it is a lack of guarantee rather than a proof of failure: “we can provide little guarantee that a minimizer of \(\mathcal{L}_{\texttt{naive}}\) will also produce low test loss” (§2.2), and it notes this has been the mainstream approach in dynamic evaluation (Krause et al.) — train normally, then adapt at test time and hope.
The task below is built to isolate the claim, the way the paper’s own toy example strips the attention layers out to “understand the effect of TTT in isolation” (§2.1). Each sequence walks its own random permutation, \(x_t=\pi(x_{t-1})\), with \(\pi\) redrawn per sequence. So every fact worth knowing is in the context and none of it is in the weights: no static model can beat chance, and the only way to score is to compress the context into \(W\) as you read. That is a harsher world than language — where \(W_0\) carries most of the load — and it is harsh on purpose, so that what TTT contributes is the entire signal.
def make_batch(n, gen):"""Each sequence walks its OWN random permutation: x_t = pi(x_{t-1}). Nothing is shared, so pi is knowable only from the context -- and knowable perfectly.""" pi = torch.argsort(torch.rand(n, V, generator=gen), dim=1) x = torch.zeros(n, T +1, dtype=torch.long) x[:, 0] = torch.randint(V, (n,), generator=gen)for t inrange(T): x[:, t +1] = pi.gather(1, x[:, t:t +1]).squeeze(1)return xdef token_losses(emb, W0, x, b, create_graph=False):"""Per-token NTP loss under mini-batched TTT: one SGD step per b tokens (Eq. 5).""" n = x.shape[0] e, y = emb[x[:, :-1]], x[:, 1:] W, out = per_sequence(W0, n), []for i inrange(0, T, b): logits = suffix_mlp(W, e[:, i:i + b]) l = F.cross_entropy(logits.reshape(-1, V), y[:, i:i + b].reshape(-1), reduction="none").reshape(n, -1) out.append(l)if i + b < T:# .mean(1) is Eq. 5's (1/b) sum over the mini-batch. .sum() over sequences keeps# each sequence's write its own: d(sum_s L_s)/dW[s] = dL_s/dW[s], since sequences# share no weights after per_sequence(). TTT is per test instance or it is nothing. g = torch.autograd.grad(l.mean(1).sum(), W, create_graph=create_graph) W = [w - ETA * gi for w, gi inzip(W, g)]return torch.cat(out, 1)def train_init(objective, b=4, steps=300, seed=0):"""Learn (emb, W0). 'e2e' backprops THROUGH the inner steps; 'naive' never takes one.""" gen = torch.Generator().manual_seed(seed) emb = (torch.randn(V, D, generator=gen) *0.5).requires_grad_() W0 = init_memory(gen) opt = torch.optim.Adam([emb] + W0, lr=3e-3)for _ inrange(steps): x = make_batch(64, gen)if objective =="e2e": loss = token_losses(emb, W0, x, b, create_graph=True).mean() # Eq. 6else: loss = token_losses(emb, W0, x, T).mean() # Eq. 4: no inner step opt.zero_grad(); loss.backward(); opt.step()return emb.detach(), [w.detach() for w in W0]def evaluate(emb, W0, b, n=256, seed=999): x = make_batch(n, torch.Generator().manual_seed(seed)) W0 = [w.clone().requires_grad_() for w in W0]return token_losses(emb, W0, x, b).detach().mean(0)emb_naive, W_naive = train_init("naive")emb_e2e, W_e2e = train_init("e2e")chance = math.log(V)static = evaluate(emb_e2e, W_e2e, T).mean().item() # b = T: one batch, zero inner stepsnaive = evaluate(emb_naive, W_naive, 4)e2e = evaluate(emb_e2e, W_e2e, 4)print(f"chance (ln {V}) {chance:.3f}")print(f"no TTT {static:.3f} <- no static W can do this task")print(f"TTT from the naive init (Eq. 4) {naive.mean():.3f}")print(f"TTT from the E2E init (Eq. 6) {e2e.mean():.3f}")print(f"\nloss by token index first 8 last 8")print(f" TTT, naive init {naive[:8].mean():7.3f}{naive[-8:].mean():7.3f}")print(f" TTT, E2E init {e2e[:8].mean():7.3f}{e2e[-8:].mean():7.3f}")assertabs(static - chance) <0.02, "without TTT this task is exactly chance"assert e2e.mean() <0.6* naive.mean(), "the meta-learned init must beat the naive one"assert e2e[-8:].mean() <0.05, "by the end of the context, pi is in the weights"assert e2e[:8].mean() >4* e2e[-8:].mean(), "the loss must FALL with context, not sit flat"
chance (ln 16) 2.773
no TTT 2.778 <- no static W can do this task
TTT from the naive init (Eq. 4) 0.888
TTT from the E2E init (Eq. 6) 0.374
loss by token index first 8 last 8
TTT, naive init 2.566 0.104
TTT, E2E init 2.267 0.000
Three readings, in order of what they cost you to believe.
The task is what it says it is. Without TTT the loss is \(\ln 16\) to three decimals — not approximately chance, chance. The memory has no way to know \(\pi\) and does not pretend to.
TTT works, from either init. Both curves fall with token index: the permutation goes into the weights as the context is read, and by the last eight tokens the E2E model has it exactly (loss \(\approx 0\)). This is the track’s thesis with the KVB objective removed, and it still holds — which is the point. The write did not need the binding.
The init is not a detail. Same architecture, same inner loop, same \(\eta\), same \(b\) — only the outer objective differs, and TTT from the naive init costs you more than twice the loss. And the naive init is not lazy or under-trained: it converged, on an objective that is blind to a fact about its own future. It is \(\mathcal{L}_{\texttt{naive}}\)’s minimizer. That is what “mismatch” means.
A footnote worth having: this notebook cannot be written the naive way and still be E2E. Drop create_graph=True and the outer backward does not merely get worse — it raises, because the graph it needs was never built. Gradients of gradients are not a refinement of the meta-learned init; they are the mechanism (M5 §2). The paper pays for this in wall-clock: “our training latency is 1.2× faster than full attention at 128K context length, but 3.4× slower at 8K” (§3.7), since FlashAttention’s kernel has no gradient-of-gradient support.
4. Mini-batching, and where short-term memory went
Eq. 2 updates \(W\) every token, which is both slow (no parallelism across a sequence) and fragile — “each gradient step in the inner loop depends on only a single token, which can easily lead to gradient explosion by chance” (§2.2). The fix is the standard one: take a step per mini-batch of \(b\) tokens (Eq. 5), with the outer loss generalized to match (Eq. 6).
That buys stability and costs freshness, and the sweep below shows both walls. Read it as a statement about the procedure, not just a hyper-parameter: the init here was meta-learned at \(b=4\), and the outer loop shaped \(W_0\) for that inner loop. Move \(b\) at test time and you are running someone else’s launchpad.
print("test-time b mean loss")for b in (1, 2, 4, 8, 16, 64): L = evaluate(emb_e2e, W_e2e, b).mean().item() note = {1: " <- single-token steps explode (Sec 2.2's stability problem)",4: " <- the b this init was meta-learned at",64: " <- b = T: one batch, no TTT at all"}.get(b, "")print(f"{b:>10}{L:10.3f}{note}")L = {b: evaluate(emb_e2e, W_e2e, b).mean().item() for b in (1, 4, 16, 64)}assert L[4] < L[16] < L[64], "staleness: a bigger b means a longer-stale memory, so worse loss"assert L[1] >10, "b=1 is unstable here -- the instability Sec 2.2 names"
test-time b mean loss
1 67234.594 <- single-token steps explode (Sec 2.2's stability problem)
2 0.372
4 0.374 <- the b this init was meta-learned at
8 0.455
16 0.781
64 2.778 <- b = T: one batch, no TTT at all
Both failure modes are real, and they pin \(b\) from opposite sides. At \(b=1\) the loss is not merely worse, it is five digits: a single-token gradient, taken with a step size the outer loop tuned for the average of four, diverges — §2.2’s “gradient explosion by chance”, reproduced. At the other end, \(b=64=T\) is one batch and zero inner steps, so the model is back to chance. In between, the cost of a larger \(b\) is staleness: predictions in a mini-batch are all made with weights that stopped learning at the batch boundary. The paper reads the same trade-off off its own ablation and lands at \(b=1\)K (§3.2).
Which leaves the hole this module has been ignoring. If \(W\) only updates every \(b\) tokens, the model is “a bigram again within each batch” (§2.3) — inside a mini-batch it has no memory of the tokens it just read. The paper’s answer is not a better write rule. It is to stop asking the write rule to do this job at all:
“It is important to set \(k\geq b\) so our model can remember the context within each mini-batch before TTT has a chance to update its weights” (§2.3)
\(k\) is the sliding-window attention size. The architecture is a Transformer whose attention is windowed to \(k=8\)K, with TTT running on the MLPs of the last quarter of the blocks at \(b=1\)K (§2.3, §3.2). So the labour is split, and the conclusion says so plainly: “the weights updated at test time can be interpreted as long-term memory and the sliding window as short-term memory” (§5). Recent tokens are recalled exactly, by attention, within the window; older context is compressed into weights, lossily, by gradient descent.
That is worth holding against M7 §3’s flattest column. The gate \(\alpha_t\) is still \(1\) here — nothing decays \(W\), and Eq. 5 has no decay term. TTT-E2E does not forget by turning the gate; it forgets by letting tokens fall out of the window. Titans put forgetting in the write rule (NL-1 §2); this design puts it in the architecture and leaves the rule alone.
5. What it buys, and what it costs
The results are about scaling, not about winning a benchmark, and the honest summary has a loss in it.
What it buys. For 3B models trained on 164B tokens, TTT-E2E “scales with context length in the same way as Transformer with full attention” while Mamba 2 and Gated DeltaNet do not (abstract, §3.4) — and it keeps an RNN’s constant per-token cost, “making it 2.7× faster than full attention for 128K context” (abstract). That is the trade the whole track has been chasing: attention’s context scaling at an RNN’s inference cost.
What it costs. On Needle-in-a-Haystack, “Transformer with full attention dramatically outperforms the other methods, including ours, especially in long context” (§3.5). This is not a footnote, it is the mechanism: compression discards detail, and NIAH is built to demand exactly the detail a compressor throws away. The paper’s own framing is that recall and compression are different goods, and it is buying one.
Two things it is honest about, which are worth copying.
The architecture is not doing the work. Set \(b=8\)K at the 8K pre-training length and TTT never fires; the modified architecture then scores 2.825 against full attention’s 2.827 — “architecture design plays a minor, supporting role in our method” (§3.2). The gain is the training procedure, not the block diagram. Contrast NL-1–NL-3, where the block diagram is the contribution.
It is not the first.Clark et al., 2022 already added an MLP as fast weights, meta-learned its initialization as slow weights, and “update[d] the fast weights by taking a gradient step on the next-token prediction loss computed over each chunk (mini-batch) of tokens” — TTT-E2E calls it “the most relevant to ours in methodology” and “a valuable inspiration” (§4.3). Both halves of “end-to-end” are there in 2022. What was missing was linear complexity — Clark et al. “does not improve efficiency, since their combined architecture does not have linear complexity” — and interleaving the fast weights through the stack rather than bolting them on the end. The 2025 contribution is the sliding window, the scale, and the evidence.
6. Synthesis — what this does to the track
The write dial had something inside it, and Miras was pointing at it the whole time.
NL-1 §3 recorded Miras’ four axes — memory architecture · attentional bias (the inner objective) · retention gate · optimizer — and its second axis is precisely this module’s subject. But Miras varies the shape of the binding loss: \(\ell_2\), Huber, \(\ell_p\), robust to outliers. It never asks whether the thing being bound should be a key and a value at all. The axis was drawn; the far end of it was not visited. TTT-E2E goes there, and finds the objective the axis was a proxy for.
So the track’s dial setting holds and gains a floor. Write rule, gate, optimizer — and inside the write rule, the objective, which four papers treated as fixed and a fifth treated as the variable. Read the two designs side by side:
NL-1 → NL-3 (TTT-KVB)
NL-4 (TTT-E2E)
inner objective
\(\lVert g(\theta_K\mathbf{x}_t)-\theta_V\mathbf{x}_t\rVert^2\), per layer
\(\texttt{CE}(f(x_{t-1};W),x_t)\), once, at the end
what the memory stores
key → value associations
whatever lowers next-token loss
the memory
a state \(\mathcal{M}\) replacing an attention layer
the MLP weights already in the block
reading it
a query \(\mathbf{q}_t\) against the store
the forward pass
inner optimizer
momentum, DGD, Muon (Titans, HOPE)
plain SGD, one step per mini-batch
forgetting
the gate \(\alpha_t\), in the write rule
the sliding window, in the architecture
short-term memory
the fast level of the hierarchy
attention, over the last \(k\) tokens
The right column is simpler in every row, which is the uncomfortable part. This track’s arc was toward richer write rules — momentum, then orthogonalization, then self-modification. TTT-E2E turns the optimizer dial back to M1’s setting, one plain step, and spends the complexity budget somewhere the recurrence in M7 §3 never had a column for: \(W_0\), the value the memory starts at. Every foundations demo starts its memory at zeros. Here that starting point is the meta-learned artifact, and the inner loop is deliberately dumb.
Both directions are live. HOPE’s bet is that the write rule should be expressive and learned; TTT-E2E’s is that the write rule should be trivial and the initialization should carry everything. Nothing here settles it — the two are not even evaluated against each other, and the papers are a month apart. What this module settles is narrower: the KV binding was a choice, not a fact, and it is now on the list of things you are allowed to turn.
Exit check
Self-test — question, then the answer we’d give.
1. What is KV Binding, and what makes it a proxy? The inner objective \(\ell_t^{(l)}(W)=\lVert g(\theta_K^{(l)}\mathbf{x}_t^{(l)};W)-\theta_V^{(l)}\mathbf{x}_t^{(l)}\rVert^2\) (TTT-E2E Eq. 7) — learn to predict each token’s value from its key — shared by TTT, Titans, Miras, Atlas and HOPE, and inherited from self-attention’s key–value cache. It is a proxy because it is not the loss the model is graded on: it is built from the current token alone and never sees the next one. §2’s demo is that fact as an assert — the KVB write is bit-identical whichever token comes next, while the E2E write is not.
2. In what two senses is TTT-E2E “end-to-end”, and which is which?At test time, the inner loop optimizes the real next-token loss at the end of the network (Eq. 1) rather than a layer-local reconstruction — that is what §2.4’s “key step” replaces. At training time, the outer loop optimizes the loss after TTT, \(\mathcal{L}(W_0;X)=\frac1T\sum_t\ell_t(W_{t-1})\) (Eq. 3), rather than \(\mathcal{L}_{\texttt{naive}}\)’s loss of the un-adapted model (Eq. 4) — differentiating through the inner steps, M5’s MAML. Drop either and the method is one of its own baselines: drop the first and you are TTT-KVB, drop the second and you are dynamic evaluation.
3. If the gate is still 1 and the optimizer is plain SGD, where did forgetting and short-term memory go? Into the architecture. Sliding-window attention holds the last \(k=8\)K tokens exactly and drops them by construction — “the weights updated at test time can be interpreted as long-term memory and the sliding window as short-term memory” (§5) — and \(k\geq b\) is required so the model can see within a mini-batch before TTT updates on it (§2.3). Titans put forgetting in the write rule; this puts it in the block diagram and leaves the rule alone.
4. What is the strongest case against reading this as the track’s refutation? Three things. It does not beat full attention on recall (§3.5) — it is buying context scaling, not memory. Its own ablation says “architecture design plays a minor, supporting role” (§3.2), so it is not a comparison of block designs and HOPE is; they are answering different questions. And both halves of its idea are in Clark et al., 2022, which it credits as “the most relevant to ours in methodology” (§4.3) — what is new is linear complexity and the scaling evidence, not the objective. This module removes an assumption from the track; it does not close it.
Track complete. Four modules: the write rule became a learner (NL-1), every memory became a level (NL-2), the pieces assembled into HOPE (NL-3), and the objective they all shared turned out to be a dial too (NL-4). M7 holds the map, and the open questions are the ones the last two modules disagree about — expressive write rules versus meta-learned initializations, and whether a compressor should ever have been asked to do recall.
