Deterministic Actor-Critic Training

Use DDPG or TD3 when the environment has a continuous action space and you want a deterministic policy trained from replay. Both algorithms use an actor that returns one action directly and a critic that estimates the value of a state-action pair.

This chapter explains the pieces shared by DDPGAgent and TD3Agent. Read DDPG for the smaller, single-critic configuration. Read TD3 when you want twin critics, target-policy smoothing, and delayed actor updates.

Choose DDPG or TD3

DDPG and TD3 share the same collection and replay loop. TD3 changes how the agent calculates targets and when it updates the actor:

DDPG is the direct deterministic actor-critic baseline. TD3 adds safeguards against overestimated Q values and an actor that exploits narrow errors in the critic. Start with TD3 when those safeguards are appropriate. Choose DDPG when you need the single-critic algorithm or a simpler baseline for comparison.

ModuRL also lets TD3Agent use any nonempty critic ensemble and explicit aggregation modes. Those settings are experimental variants, not canonical TD3.

Follow the Shared Training Loop

Both agents collect and train in the same order:

1. Before training_start, sample actions uniformly from the BoxSpace.
2. After training_start, run the online actor and add Gaussian exploration noise.
3. Clamp the resulting actions to the action-space bounds and store each transition in replay.
4. On collection timesteps selected by update_frequency, sample a replay batch and update every online critic.
5. Update the online actor at the algorithm’s actor-update interval.
6. Polyak-update the target actors and critics whenever the actor is updated.

Collection timesteps count transitions, not calls to the vectorized environment. A step with N environments adds N transitions. Each transition whose global index is a multiple of update_frequency causes one replay update after the warm-up threshold.

training_horizon records the intended number of collected transitions and tracks progress across calls to learn. DDPG and TD3 do not currently expose scheduled hyperparameters, but the shared progress counter still determines the global collection timestep used by training and logging.

Match the Actor Contract

The online and target actors are ordinary Candle Modules. For observations shaped [batch, ...observation_shape], each actor must return values shaped [batch, ...action_shape].

The supplied BoxSpace clamps those values to its bounds. The MuJoCo examples use tanh as the actor’s output activation and bounds of -1.0..=1.0:

let actor = MLP::builder()
    .input_size(observation_size)
    .output_size(action_size)
    .vb(actor_vb)
    .hidden_layer_sizes(vec![64, 64])
    .activation(Tensor::relu)
    .output_activation(Tensor::tanh)
    .name("actor".to_owned())
    .build()?;

Build the online and target actors with separate VarMaps and identical parameter names. Agent construction copies the online parameters into the target map. Construction returns DeterministicActorCriticError::ActorParameterMapMismatch if the names differ.

The exploration-noise standard deviation is measured in the actor’s output units. ModuRL adds that noise before BoxSpace clamps the action. Use act_deterministic when evaluating a trained policy without exploration noise.

Match the Critic Contract

DeterministicCritic is the deterministic-agent name for SACCritic. Each critic owns:

• an online state-action network
• a target state-action network
• separate online and target VarMaps
• an optimizer for the online parameters

A scalar critic receives a state and action and returns one Q value. Wrap a module whose input width is observation_size + action_size with ScalarStateActionCritic:

let critic = DeterministicCritic::builder()
    .online_network(ScalarStateActionCritic::new(
        online_network,
    ))
    .target_network(ScalarStateActionCritic::new(
        target_network,
    ))
    .online_vars(&online_vars)
    .target_vars(&mut target_vars)
    .optimizer(critic_optimizer)
    .build()?;

As with the actors, the online and target critic parameter names must match. The constructor initializes the target parameters from the online parameters. Each TD3 critic needs its own networks, parameter maps, and optimizer.

Keep Replay and Models on the Intended Devices

Wrap ReplayDeviceStrategy::OneDevice(device) in ReplayStorageConfig to store replay and run optimization on the same device. A configuration using ReplayDeviceStrategy::Hybrid can keep detached replay transitions on one device, such as the CPU, and move sampled batches to the optimization device.

The strategy does not move models or environments. Build the environment, actors, critics, target networks, and optimizers on the optimization device. Read Run on CUDA or Metal before splitting replay storage from optimization.

The DDPG chapter uses these pieces to build the single-critic baseline. The TD3 chapter shows how to use two critics, target-policy noise, and delayed actor updates.