Experience Is More Than Reward

In Welcome to the Era of Experience, David Silver and Richard Sutton argue that artificial intelligence is entering a new era. Today’s most capable systems are trained mostly on human-generated data: books, code, demonstrations, preferences, and other records of what people already know how to do. This has produced general-purpose models. Yet a system trained only to reproduce human knowledge cannot reliably discover what lies beyond it. The authors instead propose agents that live in a continuing stream of experience, act in the world, observe the results, and improve throughout their lifetimes.

I agree with this direction. Static datasets describe the past. Agents that pursue scientific questions, adapt to individual users, or operate in a changing world will need to learn from events that occur after training. They will also need signals based on what actually happened, rather than only judgments made before the results were known. An experiment’s measurement, a program’s execution, and a treatment’s effect can reveal errors that imitation cannot resolve. Without contact with the world, even a highly capable system may only repeat the knowledge on which it was trained.

But experience is broader than reward, and learning is broader than reward maximisation. An account built mainly around reinforcement learning leaves out one of our main ways of learning: learning from others through language.

Experience is not one-dimensional

Silver and Sutton place grounded rewards at the centre of their proposal. Unlike human prejudgments, these rewards reflect the results of an action in its environment. A health agent could observe whether a user’s sleep improved; a scientific agent could measure whether a material became stronger; a coding agent could run a program and see whether its tests passed. These signals can correct assumptions learned from human data and reward strategies that human experts did not expect.

Yet a result and an explanation of that result carry different information. A failed test says that something is wrong. An error message may identify where it went wrong. A colleague may explain why the approach is mistaken, which cases are affected, and what rule should replace it. All three come from the same event, but each provides a different learning signal.

A scalar reward compresses the event into a single number. It can rank outcomes or guide optimisation, but it cannot preserve all the information in the evidence that produced it. This does not make reward unnecessary. Reward is one signal within a richer stream of observations, explanations, corrections, demonstrations, and questions. Human feedback is also experience when the human is part of the agent’s environment. It does not have to be a preference label. A conversation in which a person diagnoses an error is an experience too.

The same point applies to other sources. Environments provide structured information through compiler diagnostics, scientific measurements, database records, documentation, proofs, and reports generated by other agents. Reducing these signals to rewards before the learner can interpret them may discard useful information.

Intelligence is also cumulative

The paper often compares experiential agents with humans and other animals, which learn from long streams of interaction. But humans do not learn only by seeking rewards or rediscovering everything for themselves. We also learn socially. We ask questions, receive explanations, imitate demonstrations, challenge claims, and use concepts developed over generations.

Scientific progress does not come from individual experience alone. An experiment may produce the observation that overturns a theory, but the hypothesis, equipment, mathematical language, and interpretation all depend on the work of others. Each researcher starts with knowledge accumulated by many people. Experimental results lead to progress through communication, criticism, and revision.

This would remain true for an agent more intelligent than any human. Greater intelligence would not remove the need to communicate. Such an agent could still learn about events it did not observe, use evidence collected elsewhere, and exchange discoveries with people and other agents. Language is not only a record of old human answers. It also lets us share learnings and build on it.

Moving beyond existing human knowledge therefore does not require moving beyond language. The important distinction is between imitating a fixed dataset and learning through ongoing interaction. Language can serve both roles. It can be static training data, or it can carry new observations, instructions, disagreements, and explanations during the agent’s experience.

The learner must make sense of the signal

Silver and Sutton describe a bi-level process in which a high-level objective represents human goals and a lower-level reward function selects and combines grounded signals. In principle, a small amount of human guidance could direct a much larger amount of autonomous experience. But the paper does not explain how this process would work.

Which observations should become rewards? How should their meaning change with context? When two signals conflict, which one should dominate? How should later evidence revise what was learned earlier? Saying that the reward function will adapt does not explain how the agent will answer these questions without moving away from the user’s intent or exploiting the reward.

Difficult learning may also require more than a single update based on the size of an error. An agent may need to inspect its own reasoning, request clarification, search for missing evidence, retry an action, compare hypotheses, and test whether a correction generalises. These are not simply extra samples for an optimiser. They are actions the learner takes to understand its experience.

Large language models already use their context as a workspace, carrying out multi-step procedures with language and tools. They should be able to use the same flexibility for learning. An experiential agent should not only receive a signal. It should be able to investigate the signal and decide how to learn from it. Reinforcement learning may help train this ability, but the system should not convert every experience into reward through a fixed procedure.

Adaptation cuts both ways

The paper also argues that continual adaptation may improve safety. A fixed system can become poorly suited to a changing environment, whereas an experiential agent can notice new conditions, recognise human concern, and correct its behaviour. This is possible, but adaptation can also create new risks.

The same awareness could help an agent recognise when it is being monitored, learn which actions cause intervention, or hide behaviour that people would reject. An adaptable reward function may correct a poorly specified objective, but misleading feedback may also push it further off course. Continual learning helps a system respond to evidence; it does not ensure that the system learns what we intended.

The safety question is therefore not only whether an agent adapts, but whether people can guide, inspect, and correct its learning over a lifetime. Adaptation can help a system learn from its mistakes. It can also make the system more capable at pursuing a flawed objective.

A broader era of experience

The paper’s main prediction is right: future agents will learn from long streams of interaction, and grounded experience may eventually provide more learning data than static human datasets. Since the preprint is published, RLVR already unlocked agents working for long trajectories, supporting this direction. But we should not reduce experience to reward or learning to trial and error by an individual agent.

Agents will learn by acting and observing, but also by asking questions, receiving instruction, interpreting explanations, and sharing what they discover. Grounded signals will connect their learning to real outcomes. Language and social interaction will help them interpret those signals and share what they learn. Reward can provide incentives, but the learner must still work out what its experience means.

The era of experience should therefore be an era of both autonomous action and autonomous learning. Agents should learn from the world, from other agents, and from people.