Understand a Deterministic Actor-Critic Training Run

The DDPG and TD3 examples confirm that their actors, critics, target networks, and replay buffers train together. Add a DDPGLogger or TD3Logger when you need to compare runs or diagnose one of those components.

Each logger receives two streams. log receives one entry per replay optimization. log_collection receives rewards and completed episodes from one vectorized environment step. An update describes a sampled replay batch; collection describes the policy’s newest behavior.

Add a Logger

Both logger traits use the shared deterministic actor-critic entry types. One logger can therefore support both algorithms:

use modurl::prelude::*;

struct ConsoleLogger;

fn log_update(entry: &DeterministicActorCriticLogEntry) {
    if entry.update_index % 1_000 != 0 {
        return;
    }

    let critic_loss = entry.critic_losses[0]
        .to_scalar::<f32>()
        .expect("critic loss must be scalar");
    let actor_loss = entry.actor_loss
        .as_ref()
        .map(|loss| loss.to_scalar::<f32>().expect("actor loss must be scalar"));

    println!(
        "step={} update={} critic_loss={critic_loss:.4} \
         actor_updated={} actor_loss={actor_loss:?}",
        entry.collection_timestep,
        entry.update_index,
        entry.actor_updated,
    );
}

fn log_collection<I>(entry: &DeterministicActorCriticCollectionLogEntry<I>) {
    for episode in &entry.completed_episodes {
        println!(
            "step={} env={} return={} length={} terminated={} truncated={}",
            episode.collection_timestep,
            episode.environment_index,
            episode.episode_return,
            episode.episode_length,
            episode.terminated,
            episode.truncated,
        );
    }
}

impl<I> DDPGLogger<I> for ConsoleLogger {
    fn log(&mut self, entry: &DeterministicActorCriticLogEntry) {
        log_update(entry);
    }

    fn log_collection(
        &mut self,
        entry: &DeterministicActorCriticCollectionLogEntry<I>,
    ) {
        log_collection(entry);
    }
}

impl<I> TD3Logger<I> for ConsoleLogger {
    fn log(&mut self, entry: &DeterministicActorCriticLogEntry) {
        log_update(entry);
    }

    fn log_collection(
        &mut self,
        entry: &DeterministicActorCriticCollectionLogEntry<I>,
    ) {
        log_collection(entry);
    }
}

Pass a mutable reference while building either agent:

let mut logger = ConsoleLogger;

let mut agent = TD3Agent::builder()
    // Keep the remaining TD3 configuration unchanged.
    .logger(&mut logger)
    .build()?;

The agent borrows the logger. Drop the agent before reading or displaying values held by the concrete logger, as the terminal graph examples do.

Read Replay-Update Metrics

DeterministicActorCriticLogEntry exposes these values:

DDPG sets actor_updated on every replay update. TD3 sets it according to actor_update_interval. When it is false, actor_loss, policy_q_values, and policy_actions are all None; critic metrics remain present.

Compare critic_q_values with bellman_targets when a critic loss changes unexpectedly. Compare policy_actions with replay_actions to distinguish the current actor from behavior stored earlier in replay.

In canonical TD3, policy_q_values come from the first online critic. If actor_aggregation_mode is configured, they contain the aggregate used by the actor objective.

Read Collection Metrics

DeterministicActorCriticCollectionLogEntry describes the newest vectorized environment step:

Each completed episode records its environment index, return, length, ending condition, and global collection timestep. Partial episodes carry across vectorized steps until the environment terminates or truncates.

During the initial random-action phase, collection entries arrive but replay update entries do not. After training_start, update entries occur only at timesteps selected by update_frequency.

You can now separate current policy behavior from replay optimization and, for TD3, delayed actor updates from critic-only updates. The repository’s DeterministicActorCriticGrapher applies the same split to terminal plots in both MuJoCo examples.