Understand an SAC Training Run

The SAC examples confirm that the actor, critics, replay buffer, and entropy configuration can train together. Add a SACLogger when you need to compare runs or diagnose one of those components.

SACLogger::log_update receives metrics from replay optimization. SACLogger::log_collection receives rewards and completed episodes from the current environment step. Keep these streams separate: an update describes a sampled replay batch, while collection describes the policy’s newest behavior.

Add a Logger

This logger prints occasional update values and every completed episode:

use modurl::prelude::*;

struct ConsoleLogger;

impl<I> SACLogger<I> for ConsoleLogger {
    fn log_update(&mut self, entry: &SACLogEntry) {
        if entry.update_index % 1_000 != 0 {
            return;
        }

        let actor_loss = entry
            .actor_loss
            .to_scalar::<f32>()
            .expect("actor loss must be scalar");
        let alpha = entry
            .alpha
            .to_scalar::<f32>()
            .expect("alpha must be scalar");

        println!(
            "step={} update={} actor_loss={actor_loss:.4} alpha={alpha:.4}",
            entry.collection_timestep,
            entry.update_index,
        );
    }

    fn log_collection(&mut self, entry: &SACCollectionLogEntry<I>) {
        for episode in &entry.completed_episodes {
            println!(
                "step={} env={} return={} length={} terminated={} truncated={}",
                episode.collection_timestep,
                episode.environment_index,
                episode.episode_return,
                episode.episode_length,
                episode.terminated,
                episode.truncated,
            );
        }
    }
}

Pass a mutable reference while building the agent:

let mut logger = ConsoleLogger;

let mut agent = SACAgent::builder()
    // Keep the remaining SAC configuration unchanged.
    .logger(&mut logger)
    .build()?;

agent.learn(&mut env, total_timesteps)?;

The repository’s sac_mujoco example uses SACGrapher to aggregate these callbacks into terminal graphs.

Start With Episode Performance

SACCollectionLogEntry::completed_episodes contains one SACEpisodeLogEntry for each environment that ended during the latest vectorized step. Each entry reports:

• environment_index: which inner environment completed
• episode_return: the sum of that episode’s rewards
• episode_length: the number of environment steps
• terminated: whether the environment reached a terminal state
• truncated: whether the episode was truncated
• collection_timestep: the total collected-transition count at completion

Episode return is the clearest measure of task performance. Compare it across several episodes rather than treating one episode as a trend. Episode length is useful when termination timing is meaningful for the environment.

SACCollectionLogEntry also contains:

• collection_rewards: one reward per inner environment from the newest step
• infos: the corresponding typed environment metadata
• collection_timestep: the total number of collected transitions
• replay_len: the number of entries currently held in replay

log_collection runs once per vectorized environment step. With N inner environments, one callback normally advances collection_timestep by N.

Read Update Metrics as a Group

After training_start, SAC performs an optimization step for each collected transition. Each call to log_update describes one sampled replay batch. update_index is the zero-based optimization count, while collection_timestep identifies the transition that triggered the update.

Let B be replay batch size and K the number of policy candidates. For a categorical policy, K is the action count. For a sampled continuous policy, K is the configured samples value.

Losses do not have a universal target value. Reward scale, model architecture, and entropy configuration all change their magnitude. Compare them with episode return and with earlier runs using the same environment and reward handling.

Critic loss measures disagreement with the soft Bellman targets. A persistent increase alongside unstable Q-values can indicate an aggressive learning rate or reward scale. A small critic loss alone does not prove that the policy is good.

Actor loss combines expected Q-values and the entropy term. It can be negative, and its raw value is not a task score. Use it to spot abrupt changes rather than to rank policies.

Alpha controls the strength of the entropy term. A larger alpha places more weight on uncertain actions. With automatic tuning, compare alpha, policy entropy, and target entropy together.

The expected policy entropy for one update is:

-mean(sum(policy_weights * policy_log_probabilities, candidate_axis))

Entropy commonly falls when a policy becomes more certain. Whether that is healthy depends on the target entropy and episode performance. A rapid collapse with poor returns suggests that exploration ended too soon.

entropy_change_loss appears only when the stabilization configuration enables the replay-to-current entropy penalty. alpha_loss and target_entropy appear only with automatic entropy tuning.

Compare Collection and Replay Values

collection_rewards show the newest environment behavior. replay_rewards come from a randomly sampled historical batch. They need not move together at each callback.

The same distinction applies to time:

• collection_timestep counts transitions gathered from environments.
• update_index counts replay optimization steps.
• replay_len grows until it reaches replay capacity.

Graph completed episode returns against collection_timestep. Graph update losses, alpha, expected policy entropy, and mean Bellman targets at a lower frequency if logging every optimization step is too expensive.

Return to Soft Actor-Critic to change entropy, critic aggregation, or stabilization. Read Run on CUDA or Metal to separate replay storage from optimization.