Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

TD3

Twin Delayed Deep Deterministic Policy Gradient (TD3) keeps DDPG’s deterministic actor and replay loop, then adds three safeguards:

  1. Two critics reduce optimistic target estimates.
  2. Noise on target actions smooths the critic target.
  3. Delayed actor and target-network updates give the critics more update steps.

This page starts from the network and replay contracts in Deterministic Actor-Critic Training and focuses on the TD3-specific choices.

Run the TD3 Example

The example supports Ant, HalfCheetah, Hopper, and Walker2d:

cargo run --release -p examples --example td3_mujoco --features ant
cargo run --release -p examples --example td3_mujoco --features half-cheetah
cargo run --release -p examples --example td3_mujoco --features hopper
cargo run --release -p examples --example td3_mujoco --features walker2d

Normally enable one environment feature; add rendering to it to open a viewer. Additive builds with several environment features select ant, half-cheetah, hopper, then walker2d. The program uses the CPU by default. Enable cuda or metal to select an accelerator. It trains for one million collected transitions, then displays terminal graphs for optimization and episode metrics.

The complete source is crates/examples/examples/td3_mujoco.rs. Its actor construction is the same as the DDPG example. The difference is the critic ensemble and the TD3 builder settings.

Build Two Independent Critics

Canonical TD3 uses two critics. Each critic owns its network and optimizer. Target networks are created automatically:

use modurl::agents::action_value_critic::ScalarStateActionCritic;

let mut critics = Vec::new();
for _ in 0..2 {
    let online_critic = MLP::builder()
        .input_size(observation_size + action_size)
        .output_size(1)
        .hidden_layer_sizes(vec![64, 64])
        .activation(Relu)
        .options((&device, DTYPE))
        .build()?;
    critics.push(ActionValueCritic::builder()
        .online_network(ScalarStateActionCritic::new(online_critic))
        .optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
        .learning_rate(1e-3)
        .build()?);
}

Do not share model parameters or an optimizer between the critics. Their independent errors make the smaller target estimate useful.

Configure the Three TD3 Safeguards

Pass both critics and the TD3-specific update settings:

let mut agent = TD3Agent::builder()
    .online_actor(online_actor)
    .actor_optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
    .actor_learning_rate(1e-3)
    .critics(critics)
    .action_space(action_space)
    .observation_space(observation_space)
    .gamma(0.99)
    .tau(0.005)
    .exploration_noise(0.1)
    .target_policy_noise(0.2)
    .target_noise_clip(0.5)
    .actor_update_interval(2)
    .replay_capacity(1_000_000)
    .batch_size(256)
    .training_start(10_000)
    .training_horizon(TOTAL_TIMESTEPS)
    .logger(&mut grapher)
    .build()?;

agent.learn(&mut env, TOTAL_TIMESTEPS)?;

target_policy_noise is the standard deviation of Gaussian noise added to the target actor’s output. target_noise_clip limits that noise component by component. The BoxSpace then clamps the noisy target action to the environment bounds.

By default, target_aggregation_mode is ActionValueCriticAggregationMode::Min. The smaller target Q estimate becomes the bootstrap value shared by both critic losses.

actor_update_interval defaults to 2. The agent updates every critic on each replay optimization, but updates the actor and all target networks only on every second optimization. On skipped actor updates, logger fields such as actor_loss are None.

Important defaults and constraints are:

SettingDefaultConstraint or effect
gamma0.99Finite and between zero and one
tau0.005Target-network update coefficient from zero to one
exploration_noise0.1Non-negative collection-noise deviation
target_policy_noise0.2Non-negative target-noise deviation
target_noise_clip0.5Non-negative target-noise bound
actor_update_interval2Nonzero replay-update interval
replay_capacity1_000_000At least batch_size; at first learn, larger than and divisible by env.num_envs()
batch_size256Nonzero
update_frequency1Nonzero transition interval
training_start1_000Random-action transitions before optimization
training_horizonFirst learn call’s num_timestepsNonzero global transition horizon

Collection exploration noise and target-policy noise solve different problems. exploration_noise changes actions sent to the environment after warm-up. target_policy_noise changes only next actions used to calculate replay targets.

Keep Canonical Actor and Target Aggregation

When actor_aggregation_mode is omitted, the actor maximizes the first online critic’s Q estimate. This is canonical TD3 behavior.

ModuRL supports nonempty ensembles of other sizes and explicit aggregation for experiments:

.target_aggregation_mode(ActionValueCriticAggregationMode::Mean)
.actor_aggregation_mode(ActionValueCriticAggregationMode::Median)

Min, Mean, Median, and Max use the same elementwise aggregation implemented for SAC critic ensembles. Setting actor_aggregation_mode makes the actor optimize the selected aggregate instead of the first critic.

These choices define algorithm variants. Omit them when reproducing canonical TD3 with two critics.

Evaluate the Deterministic Actor

As with DDPG, Agent::act adds collection exploration noise. Evaluate the online actor with:

let actions = agent.act_deterministic(&observations)?;

Read Understand a Deterministic Actor-Critic Training Run to distinguish critic updates from delayed actor updates in logs.