Action-value function
How good one action is in one state, assuming you behave normally afterwards. Unlike the state value it names a choice, so a greedy policy can be read straight off it without a model of the environment.
- The action-value function under policy pi.
- The action taken in state s.
Advantage function
How much better one action is than the state's average. Subtracting the state value removes everything the choice had no say in, which cuts gradient variance without biasing the estimate.
- The advantage of action a in state s.
- The value of that specific action.
- The average value of the state.
Bellman expectation equation
Value defined in terms of itself one step later: a reward now, plus the discounted value of wherever you land. Every value-based algorithm is some way of solving this fixed point.
- The next state.
- The immediate reward.
- The discount factor.
Bellman optimality equation
The same recursion with a maximum where the policy used to be. It describes the best achievable value without reference to any particular policy, and the greedy choice over it is an optimal policy.
- The optimal value function.
- Taking the best available action rather than the usual one.
Discounted return
The total future reward from a point onwards, with later rewards worth less. The discount both encodes impatience and keeps the sum finite on tasks that never end.
- The return from time t onwards.
- The discount factor, between zero and one.
- The reward collected at a single step.
PPO clipped objective
Limits how far one update may move the policy by clipping the probability ratio and taking the pessimistic branch. That single trick is why PPO trains stably enough to be the workhorse of RLHF.
- The ratio of new to old action probability.
- The advantage estimate for that step.
- The clip width, typically 0.2.
Q-learning update
Nudges the value of the action you took toward the reward plus the best value available next. The max is over actions you did not necessarily take, which is what makes it off-policy and lets it learn the optimal policy while exploring.
- The learning rate.
- The best value available in the next state.
- The reward just received.
REINFORCE policy gradient
Raises the log-probability of actions that led to high return and lowers it for the rest. It optimises the policy directly with no value model required, and its variance is high enough that a baseline is essentially mandatory.
- The expected return being maximised.
- The policy, parameterised by theta.
- The return that followed the action.
State-value function
How good a state is: the return you expect from it if you keep following your current policy. It judges the situation, not the choice — which is why it cannot on its own tell you what to do.
- The value function under policy pi.
- The state being evaluated.
- The policy whose behaviour is being assumed.
Temporal-difference error
The gap between what you predicted and what the next step plus the next prediction say. It is the learning signal for every TD method, and it lets you learn from a single transition without waiting for the episode to end.
- The error at step t.
- The current prediction.
- The one-step bootstrapped target.