Value-Based Training

Use DQN or DDQN when your environment has a discrete action space and you want the agent to learn an action value for every possible action.

This chapter explains the pieces shared by DQNAgent and DDQNAgent. Read DQN for a complete CartPole program. Read Double DQN when you want DDQN’s target calculation instead.

Choose DQN or DDQN

Both agents train an online Q-network and periodically copy its parameters to a target Q-network. They differ only in how they form the next-state training target:

Start with DQN when you need the standard DQN update. Choose DDQN when you want to reduce overestimation of action values. In DQN, the target network both chooses the largest next-state value and uses that value in the training target. Taking a maximum tends to favor values that are accidentally too high. DDQN uses the online network to choose the action and the target network to evaluate it, which reduces that optimistic bias. The builder fields and training loop are otherwise the same.

Terms

A Q-network maps one observation to one value per discrete action. Its output width must equal the number of actions. An online Q-network is the network the optimizer updates. A target Q-network has the same architecture and variable names, but the agent only refreshes its parameters by copying the online network at a fixed interval.

Epsilon-greedy exploration chooses a random valid action with probability epsilon and otherwise chooses the online network’s highest-valued action. epsilon_schedule controls epsilon over the configured training horizon.

Use a Dueling Q-Network

DuelingMLP separates its final representation into a scalar state value and one advantage per action. It mean-centers the advantages and combines the two streams into the same [batch, action_count] output expected from any Q-network:

#![allow(unused)]
fn main() {
let online_q_network = DuelingMLP::builder()
    .input_size(observation_space.shape()[0])
    .output_size(2)
    .vb(online_vb)
    .hidden_layer_sizes(vec![64, 64])
    .value_hidden_layer_sizes(vec![64])
    .advantage_hidden_layer_sizes(vec![64])
    .build()?;
}

hidden_layer_sizes configures the shared trunk. The value and advantage hidden-layer fields configure the two independent streams after that trunk. Leave either stream’s list empty when you want its output head to connect directly to the shared features.

Pass identically configured online and target DuelingMLP instances to either DQNAgent or DDQNAgent. Dueling changes the network architecture; DDQN independently changes the next-state target calculation, so the two techniques can be used together.

Where to Go Next

Build the complete DQN CartPole program. It shows the two Q-networks, replay configuration, and training call in one place. Then use the small, documented change in Double DQN to change its target calculation.