Reinforcement Learning: An Introduction: Summary & Key Insights

It always starts with the formulation. Every learning problem in reinforcement learning must be grounded in the interplay between the agent and the environment. This is captured neatly through the Markov Decision Process (MDP)—a mathematical model that binds states, actions, transitions, rewards, and policies in a seamless dynamic loop.

In an MDP, at each time step, the agent observes a state, selects an action, receives a reward, and transitions to a new state. The reward signals the immediate consequence of the agent’s action, while the transition function captures the environment’s stochastic nature. Crucially, the agent aims not for short-term gratification but for cumulative reward maximization, represented through expected returns.

Policies lie at the heart of MDPs. A policy defines the agent’s behavior—a mapping from states to probabilities of selecting actions. Determining or improving this policy is the essence of RL. When the model of the environment is known, classic methods in dynamic programming emerge; when it is unknown, learning becomes the centerpiece.

This formalism offers more than notation—it provides a lens into autonomy. It connects control problems, game strategies, and adaptive behaviors under a single theoretical umbrella. The elegance of the MDP formulation lies in its generality: whether a robot exploring terrain or software adjusting pricing strategies, each operates through this loop of observation, choice, and reward.

Dynamic Programming (DP) is where reinforcement learning finds its computational heartbeat for problems with known models. When transitions and rewards are fully specified, we can exploit mathematical precision to compute optimal policies through iterative refinement.

Policy evaluation calculates the value of a policy—the expected return from each state if that policy is followed. Policy improvement leverages those values to produce a better policy. These two steps, alternated repeatedly, yield what we call policy iteration. Alternatively, value iteration compresses this process into a unified update mechanism that drives values directly to optimality.

Beyond the equations, dynamic programming establishes a fundamental intuition: learning is iteration. Each evaluation, each improvement reflects a gradual honing of performance, approaching optimal behavior through recursive self-consistency. Even though DP assumes full knowledge, it prepares the conceptual ground for later methods that operate under uncertainty.

This technique connects generations of research—from Richard Bellman’s principle of optimality to today’s neural approximators. It reminds us that every reinforcement learner, human or machine, relies on dynamic programming principles at its core: refining beliefs, adjusting actions, and converging upon greater efficiency through cycles of feedback.