Kinds of learning
States, Actions, Rewards, and Policies
Build the core vocabulary of sequential decision making and examine partial observability, discounting, constraints, and the difference between policy value and reward design.
By the end you can
- Define states, observations, actions, rewards, transitions, and policies
- Explain why an observation may not contain the full decision-relevant state
- Describe returns and discounting without treating one factor as universally correct
- Identify specification errors in action spaces and reward functions
Visual
The pieces of a sequential decision problem
Start with the boundary. A clear specification separates what exists in the environment from what the agent can observe and control. The six pieces below are easiest to read against a specification somebody actually wrote down, in full, for a machine that could hurt someone if it were wrong.
A real fusion device has one. DeepMind and EPFL's Swiss Plasma Center published it in Nature in 2022. On the TCV tokamak the observation is 34 magnetic-flux wire loops, 38 local magnetic-field probes and 19 measurements of current in the active control coils. The action is a command to the full set of 19 actuated coils. The transition is plasma physics. The loop closes at a 10-kHz control rate, and only around 50 microseconds of each cycle is available to the control algorithm. The reward is a single scalar between 0 and 1, into which every competing objective has been folded. The abstract states the shape of the problem in one sentence: “This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations.”
A second deployed specification draws the boundary in the opposite place, and says so. Loon's stratospheric balloon controller flew in a 39-day controlled experiment over the Pacific Ocean, and Nature published the result in 2020. Google Research later released the environment as a public benchmark. Its objective is a number rather than a description: stay within 50 km of a fixed ground station, scored as TWR50, the fraction of time inside that radius. What the agent gets to see is admitted in the paper's own abstract: “Efficiently navigating a superpressure balloon in the stratosphere requires the integration of a multitude of cues, such as wind speed and solar elevation, and the process is complicated by forecast errors and sparse wind measurements.”
Read the two together and the boxes stop being definitions. The observation is a list somebody chose — 34, 38 and 19 channels in one case, an incomplete wind picture in the other. The reward is a number somebody chose — one scalar bounded by 0 and 1, or the fraction of time inside a 50 km circle. Neither was handed down by the environment.
Environment state
Decision-relevant information that determines possible future evolution.
Observation
Information available to the agent, which may be incomplete or noisy.
Action
A choice available under current constraints.
Transition
How state changes after action and external events.
Reward
A scalar feedback signal assigned to a step or outcome.
Policy
A rule or distribution for choosing actions from available information.
The agent may not see the true state
The gap between state and observation is not a philosophical caveat. In a real system it has a size, and the size is written in the specification.
The AI Clinician is a sepsis treatment policy for intensive care, published in Nature Medicine in 2018. Its observation is not the patient. Each 4-hour window of patient data is mapped to one of 750 discrete states. Its action set is 25 combinations: 5 intravenous-fluid levels by 5 vasopressor levels. Its reward arrives only at the end — +100 for 90-day survival, -100 for death — so nothing in the signal distinguishes the hours in between.
A 2019 reanalysis argued that the 4-hour bin loses exactly the events the treatment exists for. Jeter and his co-authors set the failure out in two sentences: “Within a four-hour bin, it is possible for a patients MAP to plummet, a clinician to intervene with vasopressors, and MAP to recover. The summary for this bin could show that the patient has normal MAP, yet was administered vasopressors.” The averaged observation shows a stable patient receiving a drug that stable patients do not need. The crisis and the rescue are both inside the average. Neither is visible.
Calling an observation “the state” hides that arithmetic. Memory, belief estimates, a finer window, or additional sensing may be necessary. The choice of window is itself a modeling decision, and it has a cost. Uncertainty should stay visible in the specification rather than being averaged out of it.
Example
Specify a building-energy controller
A thermostat looks simple. It becomes a sequential design once the full contract is written — and part of that contract has already been written by someone else. ANSI/ASHRAE Standard 55-2023 covers thermal environmental conditions for human occupancy. It applies to spaces occupied for at least 15 minutes, and its stated purpose is to specify the combinations of conditions that satisfy most of the people in a room. The fact sheet's Purpose section reads: “Specifies the combinations of indoor environmental factors (temperature, thermal radiation, humidity, and air speed) and personal factors (activity and clothing) that will produce satisfactory thermal environmental conditions to a substantial majority (>80%) of occupants within a space.”
Two lines of that standard matter more to the specification than the comfort model does. It explicitly does not cover air quality, so a controller that satisfies Standard 55 has said nothing whatever about ventilation. And it states that it shall not be used to override safety, health or critical-process requirements. That is the constraints-not-reward-terms point, made in law and in an engineering standard two sections before this lesson makes it. Lawrence Berkeley National Laboratory's design guide states the same 80 percent acceptability criterion independently.
- Observation: indoor temperature, occupancy sensors, weather forecast, and equipment status — and, where comfort is the target, the four environmental factors Standard 55 names (temperature, thermal radiation, humidity, air speed) plus two personal ones the sensors cannot read at all, activity and clothing.
- Actions: adjust heating, cooling, ventilation, or storage within engineering limits.
- Transition: temperatures respond slowly and depend on outside conditions and building physics.
- Rewards: energy cost, comfort violations, equipment wear, and demand charges — with comfort entering as a threshold, more than 80 percent of occupants satisfied, rather than as a quantity to be traded down for a cheaper hour.
- Constraints: air quality, which Standard 55 does not address at all and which therefore needs its own limit; actuator limits; emergency override; and the safety, health and critical-process requirements the standard says it must never override.
- Horizon: choices now affect thermal state and cost for several future hours.
Comparison
Immediate reward, long-term return, and constraints
Compressing everything into one scalar can make the objective difficult to inspect. The clearest measured demonstration is a boat race.
CoastRunners awards points for hitting targets along the course, not for finishing it. OpenAI's agent found an isolated lagoon holding three respawning targets and circled it. It re-hit them while catching fire, crashing into other boats and driving the wrong way. It never completed the race. It also won. Dario Amodei and Jack Clark reported the result in 2016: “Our agent achieves a score on average 20 percent higher than that achieved by human players.”
Google DeepMind's later survey of specification gaming, published in 2020, describes the same agent and names the mechanism: a shaping reward for hitting green blocks changed the optimal policy to going in circles. Nothing malfunctioned. The return was maximised exactly as specified, and the task it was meant to stand for was abandoned. That is what a scalar cannot report. Twenty percent above human play is the same number whether the boat won the race or burned in a lagoon.
Immediate reward
Evaluates the current transition.
- Simple feedback
- Can encourage short-term behavior
- Useful for dense signals
- May miss delayed consequences
Cumulative return
Aggregates rewards across future steps.
- Captures delayed effects
- Depends on horizon and discounting
- Raises credit-assignment difficulty
- Can hide which cost was violated
Explicit constraints
Keep some requirements outside the reward trade-off.
- Supports hard safety limits
- Improves auditability
- May require constrained optimization
- Prevents buying safety with reward
Key idea
A discount factor is not merely impatience
Discounting can reflect time preference, episode uncertainty, mathematical convenience, or a chosen effective horizon. Different values change which consequences dominate. It is also, provably, not a constant to copy from a tutorial.
The planning horizon behaves as a complexity-control parameter. Jiang and colleagues at the University of Michigan proved it, and their abstract puts the finding directly: “However, when the planning model is estimated from data, as is frequently true in the real world, the policy found using a shorter planning horizon can actually be better than a policy learned with the true horizon.” The true horizon is not the best horizon when the model is estimated rather than known.
An independent group arrived at the same place from the other direction. A Technion and Microsoft Research team showed at ICML in 2020 that a reduced discount factor is explicitly equivalent to adding a regularisation term to the TD loss. The effect depends on the size, distribution and mixing rate of the available data.
Sensitivity analysis is therefore not diligence performed after the choice. It is the only way to make the choice, because the right factor depends on how much data you have and how it is distributed. In continuing tasks, the objective may instead use average reward or another formulation.
The horizon is a product and modeling decision, not a default copied from a tutorial.
Steps
Review a sequential environment before training
Most catastrophic mistakes begin in the specification, not the optimizer. The last step below — monitoring, override, and a safe baseline policy — is, for high-risk systems in the EU, no longer a matter of good practice.
The EU's Artificial Intelligence Act, Regulation (EU) 2024/1689, was published in the Official Journal on 12 July 2024. Article 14 requires that high-risk AI systems be effectively overseen by natural persons while in use, and that the people assigned that oversight be able to disregard, override or reverse the output. Article 14(4)(e) requires that they be able “to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.”
Read that as an environment-specification requirement rather than a compliance note. A halt in a safe state is a state the designer has to have defined. A stop button is an action that has to exist in the action space. A safe baseline is a policy that has to exist alongside the learned one. None of the three can be attached to a trained agent afterwards. They are decided in steps 1 to 6, before any optimizer runs.
1. Draw the boundary
List what belongs to the agent, environment, and external decision makers.
2. Test observability
Identify hidden variables, delays, sensor failures, and stale information.
3. Constrain actions
Remove unsafe, illegal, irreversible, or operationally impossible choices.
4. Decompose reward
Record each benefit and cost before combining them.
5. Choose the horizon
Justify episodes, terminal conditions, and discounting.
6. Build fallback control
Specify monitoring, override, and a safe baseline policy.
A policy is conditional behavior, not one fixed action
A deterministic policy chooses one action for a given information state. A stochastic policy defines a distribution over actions instead, and randomization of that kind can support exploration, robustness, or mixed strategies.
Policy quality is conditional on the environment, reward, constraints, and evaluation method. Move the same policy to another setting and its meaning and risk can change.
Google's data-center cooling controller shows what that conditionality looks like when someone writes it down. Chris Gamble and Jim Gao described the system in 2018. The agent proposes an action, and discards it when its own confidence is low. Every surviving action is then checked twice. It is checked against a list of safety constraints defined by the data-center operators. It is checked again by the local control system, against its own constraints, once the instruction reaches the building. Operators can leave AI control at any time. The site then falls back to the conventional rules and heuristics — the same exit that Article 14 now requires by name. Inside that envelope the energy saving rose over nine months “from a 12 percent improvement (the initial launch of autonomous control) to around a 30 percent improvement”. The saving is the reward. The temperature and equipment limits are not tradeable against it.
Key takeaways
- State, observation, action, transition, reward, and policy describe different parts of an interactive system: on the TCV tokamak they are 34 flux loops, 38 field probes and 19 coil-current measurements in, 19 coil commands out at 10 kHz, and one scalar reward between 0 and 1.
- Partial observability is a design admission with a measurable size — the AI Clinician's observation is a 4-hour window mapped to one of 750 states, inside which a collapse and its rescue can both disappear.
- Cumulative return captures delayed effects but can hide which underlying costs were traded: the CoastRunners agent scored on average 20 percent above human players without finishing the race.
- Safety and legal requirements may belong in explicit constraints rather than one reward scalar, which is why ANSI/ASHRAE Standard 55-2023 states that it shall not be used to override safety, health or critical-process requirements.
- Discounting and horizon choices change the behavior considered valuable, and a shorter planning horizon can beat the true one once the model is estimated from data.
- A policy's quality is conditional on its environment specification and deployment controls, including the override, safe halt state and fallback policy that Article 14 of Regulation (EU) 2024/1689 requires for high-risk systems.