Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DDPG

Deep Deterministic Policy Gradient (DDPG) trains one deterministic actor and one Q-value critic from replayed transitions. This page shows how the complete MuJoCo example connects those networks to DDPGAgent.

You should already understand the actor, critic, target-network, and replay contracts in Deterministic Actor-Critic Training.

Run the DDPG Example

The example supports Ant, HalfCheetah, Hopper, and Walker2d:

cargo run --release -p examples --example ddpg_mujoco --features ant
cargo run --release -p examples --example ddpg_mujoco --features half-cheetah
cargo run --release -p examples --example ddpg_mujoco --features hopper
cargo run --release -p examples --example ddpg_mujoco --features walker2d

Normally enable one environment feature; add rendering to it to open a viewer. Additive builds with several environment features select ant, half-cheetah, hopper, then walker2d. The program uses the CPU by default. Enable cuda or metal to select an accelerator. It trains for one million collected transitions, then displays terminal graphs for optimization and episode metrics.

The complete source is crates/examples/examples/ddpg_mujoco.rs. It contains four parts:

  1. An owned actor model.
  2. One ActionValueCritic with an owned online network and optimizer.
  3. A bounded continuous action space and replay-device strategy.
  4. A DDPGAgent that owns collection and optimization.

Build the Actor and Critic

The actor implements Module and Forward<O, A>. Observation rank O and action rank A include the batch axis. The agent creates a detached target actor automatically. ActionValueCritic creates its own detached target network.

use modurl::agents::action_value_critic::ScalarStateActionCritic;

let online_actor = MLP::builder()
    .input_size(observation_size)
    .output_size(action_size)
    .hidden_layer_sizes(vec![64, 64])
    .activation(Relu)
    .output_activation(Tanh)
    .options((&device, DType::F32))
    .build()?;
let critic = ActionValueCritic::builder()
    .online_network(ScalarStateActionCritic::new(online_critic))
    .optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
    .learning_rate(1e-3)
    .build()?;

The critic receives observations [batch_size, ...observation_shape] and actions [batch_size, ...action_shape]. It returns Q values [batch_size, 1]. ScalarStateActionCritic flattens event axes and joins Float observations and actions. Its inputs must share dtype and device. Custom critics can accept other native observation kinds.

After each replay update, targets move toward online parameters by tau.

The actor learns to maximize the online critic’s mean Q estimate. Equivalently, the optimizer minimizes the negative mean Q value.

Assemble the Agent

The builder accepts owned models and optimizers:

let mut agent = DDPGAgent::builder()
    .online_actor(online_actor)
    .actor_optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
    .actor_learning_rate(1e-4)
    .critic(critic)
    .action_space(action_space)
    .observation_space(observation_space)
    .gamma(0.99)
    .tau(0.005)
    .exploration_noise(0.1)
    .replay_capacity(1_000_000)
    .batch_size(256)
    .training_start(10_000)
    .training_horizon(TOTAL_TIMESTEPS)
    .logger(&mut grapher)
    .build()?;

agent.learn(&mut env, TOTAL_TIMESTEPS)?;

The action_space implements ActionMap<A, A> with Float actions. BoxSpace clamps actor outputs to the environment bounds. Actor outputs must match the policy event shape. Environment actions must match the action event shape. The included MuJoCo environments use one-dimensional action vectors bounded by -1.0..=1.0; their vector length depends on the selected environment.

Observations keep their native kind and dtype in inputs and replay. Models must perform any required conversion explicitly. The dtype setting controls actions, rewards, and Q values, including F64 where supported. Actions and Q values must share the configured dtype; mismatches return an error. Replay rewards are converted to that dtype. Model parameters must use the optimization device. Replay stores detached transitions and uses terminal observations for truncation targets.

Important defaults and constraints are:

SettingDefaultConstraint or effect
actor_learning_rateRequiredFinite and positive
gamma0.99Finite and between zero and one
tau0.005Target-network update coefficient from zero to one
exploration_noise0.1Non-negative Gaussian standard deviation
replay_capacity1_000_000At least batch_size; at first learn, larger than and divisible by env.num_envs()
batch_size256Nonzero
update_frequency1Nonzero transition interval
training_start1_000Random-action transitions before optimization
training_horizonFirst learn call’s num_timestepsNonzero global transition horizon

The example overrides training_start to collect 10,000 random transitions. Those transitions remain in replay. After warm-up, the agent adds exploration noise to online-actor actions and updates the actor, critic, and target networks at every selected replay update.

Evaluate Without Exploration Noise

Agent::act includes Gaussian exploration noise. Use act_deterministic for evaluation:

let actions = agent.act_deterministic(&observations)?;

The observations must be shaped [batch, ...observation_shape]. The returned actions are shaped [batch, ...action_shape] and are clamped to the configured BoxSpace.

Read Understand a Deterministic Actor-Critic Training Run to add a logger and interpret replay-update metrics. Read TD3 to add twin critics, target-policy smoothing, and delayed actor updates.