DDPG

Deep Deterministic Policy Gradient (DDPG) trains one deterministic actor and one Q-value critic from replayed transitions. This page shows how the complete MuJoCo example connects those networks to DDPGAgent.

You should already understand the actor, critic, target-network, and replay contracts in Deterministic Actor-Critic Training.

Run the DDPG Example

The example supports Ant, HalfCheetah, Hopper, and Walker2d:

cargo run --release -p examples --example ddpg_mujoco --features ant
cargo run --release -p examples --example ddpg_mujoco --features half-cheetah
cargo run --release -p examples --example ddpg_mujoco --features hopper
cargo run --release -p examples --example ddpg_mujoco --features walker2d

Normally enable one environment feature; add rendering to it to open a viewer. Additive builds with several environment features select ant, half-cheetah, hopper, then walker2d. The program selects CUDA when it is available and otherwise uses the CPU. It trains for one million collected transitions, then displays terminal graphs for optimization and episode metrics.

The complete source is crates/examples/examples/ddpg_mujoco.rs. It contains four parts:

1. Separate online and target actor networks.
2. One online/target critic pair.
3. A bounded continuous action space and replay-device strategy.
4. A DDPGAgent that owns collection and optimization.

actor and critic are private helper functions defined in that example, not functions provided by ModuRL. actor builds an actor MLP. critic builds the online and target critic networks and packages them as a DeterministicCritic.

Build the Online and Target Actors

Both actor networks must have the same architecture and parameter names. Only the online actor has an optimizer:

let online_actor_variables = VarMap::new();
let mut target_actor_variables = VarMap::new();

let online_actor = actor(
    &online_actor_variables,
    observation_size,
    action_size,
    &device,
)?;
let target_actor = actor(
    &target_actor_variables,
    observation_size,
    action_size,
    &device,
)?;
let actor_optimizer = AdamW::new(
    online_actor_variables.all_vars(),
    actor_optimizer_parameters,
)?;

DDPGAgent copies the online actor parameters to the target actor during construction. Later, it moves each target parameter toward the corresponding online parameter by tau after every replay update.

Build One Critic

DDPG requires exactly one DeterministicCritic. Its online network learns from replay. Its target network supplies the next-state Q estimate:

let critic = DeterministicCritic::builder()
    .online_network(ScalarStateActionCritic::new(
        online_critic,
    ))
    .target_network(ScalarStateActionCritic::new(
        target_critic,
    ))
    .online_vars(&online_critic_variables)
    .target_vars(&mut target_critic_variables)
    .optimizer(critic_optimizer)
    .build()?;

The actor learns to maximize the online critic’s mean Q estimate. Equivalently, the optimizer minimizes the negative mean Q value.

Assemble the Agent

The example passes all owned modules and borrowed parameter maps to the builder:

let mut agent = DDPGAgent::builder()
    .online_actor(online_actor)
    .target_actor(target_actor)
    .online_actor_vars(&online_actor_variables)
    .target_actor_vars(&mut target_actor_variables)
    .actor_optimizer(actor_optimizer)
    .critic(critic)
    .action_space(action_space)
    .observation_space(observation_space)
    .replay_storage_config(ReplayStorageConfig::new(
        ReplayDeviceStrategy::OneDevice(device),
    ))
    .gamma(0.99)
    .tau(0.005)
    .exploration_noise(0.1)
    .replay_capacity(1_000_000)
    .batch_size(256)
    .training_start(10_000)
    .training_horizon(TOTAL_TIMESTEPS)
    .logger(&mut grapher)
    .build()?;

agent.learn(&mut env, TOTAL_TIMESTEPS)?;

The action_space must be a BoxSpace. Its shape must match the actor output, and its bounds must match the environment’s accepted actions. The included MuJoCo environments use one-dimensional action vectors bounded by -1.0..=1.0; their vector length depends on the selected environment.

Important defaults and constraints are:

The example overrides training_start to collect 10,000 random transitions. Those transitions remain in replay. After warm-up, the agent adds exploration noise to online-actor actions and updates the actor, critic, and target networks at every selected replay update.

Evaluate Without Exploration Noise

Agent::act includes Gaussian exploration noise. Use act_deterministic for evaluation:

let actions = agent.act_deterministic(&observations)?;

The observations must be shaped [batch, ...observation_shape]. The returned actions are shaped [batch, ...action_shape] and are clamped to the configured BoxSpace.

Read Understand a Deterministic Actor-Critic Training Run to add a logger and interpret replay-update metrics. Read TD3 to add twin critics, target-policy smoothing, and delayed actor updates.