Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Soft Actor-Critic

The SAC agent uses Burn and is available through modurl::agents::sac and the prelude. Its critics use the generic action-value API in modurl::agents::action_value_critic.

SACModelGroup groups the policy, critic, and execution tensor types. The builder infers this group from the supplied models and spaces. Explicit agent annotations use SACAgent<'a, G, OS, AS, GE, I, A>, where G is the model group. The observation and candidate ranks are inferred; A is the environment action batch rank.

Soft Actor-Critic (SAC) trains one stochastic actor and two Q-value critics from replayed transitions. The algorithm traditionally known as SAC almost always uses exactly two critics. ModuRL accepts any nonempty critic ensemble, but use two unless you are intentionally studying another aggregation scheme.

SAC is primarily a continuous-control algorithm. This chapter follows the continuous MuJoCo example and explains the actor, critic, entropy, replay, and device choices you need to adapt it. You should already be familiar with MultiGym, native Burn models, and probabilistic policies from Core Concepts.

Run Continuous SAC

The example supports Ant, HalfCheetah, Hopper, and Walker2d:

cargo run --release -p examples --example sac_mujoco --features ant
cargo run --release -p examples --example sac_mujoco --features half-cheetah
cargo run --release -p examples --example sac_mujoco --features hopper
cargo run --release -p examples --example sac_mujoco --features walker2d

Normally enable one environment feature; add rendering to it to open a viewer. Additive builds with several environment features select ant, half-cheetah, hopper, then walker2d. The program uses CPU by default. Enable cuda or metal to select an accelerator. It trains for one million collected transitions, then displays terminal graphs for episode performance and SAC diagnostics.

The complete source is crates/examples/examples/sac_mujoco.rs. It contains four parts:

  1. A Gaussian parameter module and squashed distribution for bounded actions.
  2. Two independently optimized online/target critic pairs.
  3. Automatic entropy tuning.
  4. An SACAgent that owns replay and runs collection and optimization.

The agent stores every collected transition in replay, including transitions from the initial random-action phase. Once collection_timestep reaches training_start, each new transition triggers one replay optimization step. This gives the implementation an update-to-data ratio of one. A vectorized step with N environments stores N transitions and, after the threshold, triggers N updates.

Build the Continuous Policy

GaussianModule returns distribution parameters rather than an environment action. ProbabilisticPolicyModel combines the module with a distribution to form the actor’s policy. For an action with A components, GaussianDistribution expects A means followed by A log standard deviations.

The module uses observation-dependent means and one trainable log standard deviation per action component:

#[derive(Module, Debug)]
struct GaussianModule {
    mean: MLP,
    log_std: Param<Tensor<2>>,
}

impl Forward<2, 2> for GaussianModule {
    type InputKind = Float;
    type OutputKind = Float;
    type Error = ModelError;

    fn forward(&self, observations: Tensor<2>) -> Result<Tensor<2>, Self::Error> {
        let mean = self.mean.forward(observations)?;
        let log_std = self.log_std.val().expand(mean.dims()).clamp(-20.0, 2.0);
        Ok(Tensor::cat(vec![mean, log_std], 1))
    }
}

Build the distribution with the environment’s action shape:

let action_size = action_space.shape()[0];
let mean = sac_mlp(observation_size, action_size, 0.01, &device)?;
let log_std = Param::from_tensor(Tensor::zeros([1, action_size], (&device, DType::F32)));
let distribution = TransformedDistribution::new(
    GaussianDistribution::<2>::new([action_size])?,
    TanhTransform,
);
let policy = ProbabilisticPolicyModel::new(
    GaussianModule { mean, log_std },
    distribution,
);

TanhTransform keeps policy candidates between -1 and 1, matching the included MuJoCo environments’ action bounds. The transform also includes its Jacobian correction in action log probabilities.

Build Two Critics

Each ActionValueCritic owns an online Burn model, a detached target snapshot, and an optimizer. The constructor clones the online model to initialize the target. Each critic has its own explicit learning rate.

A continuous critic receives observations and actions and returns one Q-value per batch item. ScalarStateActionCritic concatenates flattened observation/action pairs before calling the model. The model implements Burn’s Module and ModuRL’s Forward<2, 2>. It accepts Float [batch_size, observation_size + action_size] and returns [batch_size, 1]. Observations and actions must share dtype and device; the adapter does not cast them.

let critic = ActionValueCritic::builder()
    .online_network(ScalarStateActionCritic::new(online_network))
    .optimizer(critic_optimizer)
    .learning_rate(3e-4)
    .build()?;

Create each critic with its own online model and optimizer. Separate target models and parameter maps are unnecessary. backward_step(&loss) optimizes the online model from a Float loss [1] and leaves its target unchanged. hard_update() copies the online model into a detached target snapshot. polyak_update(tau) blends each floating-point target parameter with its matching online parameter. Both target-evaluation methods detach their results. Actor evaluation retains action gradients while excluding critic-parameter gradients.

ActionValueCriticNetwork<O, A, C> describes native observation and action kinds. Ranks O and A include batch and event axes; rank C adds a candidate axis to rank A. The methods retain borrowed tensor inputs. For a scalar critic, use online_replay_values::<2, 2, 3>(&observations, &actions) for rank-2 batches.

DiscreteVectorHeadCritic accepts native observation kinds and a Float model output [batch_size, action_count]. It requires Int replay actions [batch_size, 1] or Int candidates [batch_size, candidate_count, 1]. The adapter checks index bounds before gathering values, reading one Boolean result to the host. The model must perform any required observation conversion explicitly.

The default aggregation mode is ActionValueCriticAggregationMode::Min. This is the traditional SAC choice: using the lower estimate helps reduce optimistic Q-value errors.

ModuRL also exposes Mean, Median, and Max for experiments with different critic ensembles:

ChoiceUse
MinTraditional SAC with two critics
MeanArithmetic mean of the critic ensemble
MedianMiddle estimate for an ensemble with several critics
MaxOptimistic experimental estimate

Configure Entropy

The entropy coefficient, usually written as alpha, balances expected return against policy entropy. Larger values put more weight on keeping the policy’s action distribution stochastic. Use automatic entropy tuning for the standard SAC workflow:

let log_alpha = Param::from_tensor(Tensor::zeros([1], (&device, DType::F32)));
let alpha_optimizer = AdamWConfig::new().with_weight_decay(0.0).init();
let entropy = SACEntropyConfiguration::automatic(log_alpha, alpha_optimizer, 3e-4)?;

automatic asks the policy for its default target entropy. The built-in Gaussian policy uses the negative action-component count. An affine distribution transform adjusts that default for its scale.

Pass a ParameterSchedule when the desired entropy should change during the run:

let target_entropy = LinearSchedule::new(-2.0, -1.0);
let entropy = SACEntropyConfiguration::automatic_with_target_schedule(
    log_alpha,
    alpha_optimizer,
    3e-4,
    target_entropy,
)?;

LinearSchedule moves the target entropy from -2.0 to -1.0. Choose values appropriate for the environment’s action dimension rather than copying these illustrative values.

Use a fixed coefficient when you need a known constant or want to reproduce a configuration without an entropy optimizer:

let entropy = SACEntropyConfiguration::fixed(0.2)?;

Entropy configuration constructors return a typed error for invalid parameters. A fixed coefficient must be finite and nonnegative. Automatic tuning requires an explicit learning rate and Float log alpha [1].

Assemble the Agent

The continuous example assembles the pieces as follows:

let mut agent = SACAgent::builder()
    .policy(policy)
    .actor_optimizer(actor_optimizer)
    .actor_learning_rate(3e-4)
    .critics(vec![critic_1, critic_2])
    .entropy_configuration(entropy)
    .action_space(action_space)
    .observation_space(observation_space)
    .aggregation_mode(ActionValueCriticAggregationMode::Min)
    .training_horizon(total_timesteps)
    .build()?;

agent.learn(&mut env, total_timesteps)?;

training_horizon defines how many collected transitions parameter schedules take to reach their final values. Schedule progress continues across learn calls and stops at the end of the horizon.

Important defaults and constraints are:

SettingDefaultConstraint or effect
gamma0.99Finite and between zero and one
tau0.005Target-network update coefficient from zero to one
replay_capacity1_000_000At least batch_size; at first learn, larger than and divisible by env.num_envs()
batch_size256Nonzero
training_start1_000Random-action collection before optimization
samples1Continuous expectation candidates per observation
training_horizonFirst learn call’s num_timestepsNonzero schedule horizon in transitions
actor_learning_rateRequiredFinite and positive
dtypeF32Floating-point training statistics; does not cast observations

Models and automatic log alpha must use the optimization device. Critic outputs must use the configured compute dtype. Replay storage is allocated for the first collected transition and retained across learn calls. Replay preserves native observation kind and first-batch dtype; later observation dtype changes return an error. User models or explicit preprocessing perform input conversions. Rewards and policy statistics are converted to compute dtype for optimization. The agent owns concrete native policy and critic models; the builder returns SACConfigurationError for invalid construction. Runtime failures retain policy, critic, environment, space, tensor-read, and replay causes in SACError.

A categorical CPU example is also available:

cargo run --release --locked -p examples --example sac_cartpole

samples controls the Monte Carlo estimate used by continuous distributions. Larger values cost more critic evaluations but use more candidates to estimate the actor and target expectations.

Q-value clipping is an optional experimental stabilization:

.q_value_clip(0.5)

The value is a positive bound on a critic’s update around its target-network estimate. Its useful scale depends on environment rewards, so it remains opt-in.

Implement a Custom Critic Adapter

Use ScalarStateActionCritic unless your critic has a different input or output layout. A custom ActionValueCriticNetwork<O, A, C> must implement Burn’s Module and follow these contracts:

OperationInputsOutput
replay_valuesobservations [B, ...observation_shape], actions [B, ...action][B, 1]
policy_valuesobservations [B, ...observation_shape], candidates [B, K, ...action][B, K, 1]
actor_valuesobservations [B, ...observation_shape], candidates [B, K, ...action][B, K, 1]

actor_values must preserve gradients through differentiable candidate actions while excluding gradients to critic parameters.

Discrete SAC Is an API Variant

SACAgent can also work with categorical policies and DiscreteVectorHeadCritic. Experimental discrete-SAC configurations can select mean critic aggregation with .aggregation_mode(ActionValueCriticAggregationMode::Mean) and add an entropy-drift penalty with .entropy_change_penalty(0.5). It is not the canonical example because traditional SAC is a continuous-control algorithm.

Read Understand an SAC Training Run to add a logger and interpret its metrics. Read Run on CUDA or Metal when replay storage should remain on the CPU while optimization runs on an accelerator.