ModuRL Guide
This guide is for Rust developers who want to build reinforcement learning programs with ModuRL. It starts with PPO on CartPole, then explains the library types behind that example, stochastic and deterministic actor-critic algorithms, and the value-based DQN and DDQN paths.
The guide assumes basic Rust and Cargo knowledge. It does not teach reinforcement learning or neural networks from first principles.
ModuRL is early, and API stability is not guaranteed. Start with Getting Started to run and assemble a PPO CartPole program. For off-policy stochastic actor training, read Soft Actor-Critic. For deterministic continuous-control training, read Deterministic Actor-Critic Training before choosing DDPG or TD3. For discrete-action value-based training, read Value-Based Training.
The README provides project status and the shortest repository-based commands. Rustdoc provides the precise contracts for public traits, structs, and builders.
Getting Started
This page builds a small PPO training program in a new Cargo binary crate.
What You Will Build
The program trains PPO on several CartPole environments at once. It uses an
MLP to produce policy logits, a probabilistic policy model to sample actions,
and another MLP to estimate state values.
Prerequisites
Install a Rust toolchain that supports edition 2024.
Create a Small Program
Create a new binary crate:
cargo new modurl-hello
cd modurl-hello
Start by adding the libraries used by this example:
[package]
name = "modurl-hello"
version = "0.1.0"
edition = "2024"
[dependencies]
modurl = "0.1"
modurl_gym = "0.1"
burn = { version = "=0.22.0", default-features = false, features = ["std", "optim", "flex", "autodiff"] }
The snippets below are consecutive pieces of src/main.rs. Add each one in
order; the file compiles after the final Train section.
Imports
First, bring the tensor types, optimizer, ModuRL traits, and CartPole environment into scope:
use burn::{nn::Tanh, optim::AdamWConfig, tensor::{DType, Device}};
use modurl::prelude::*;
use modurl_gym::classic_control::cartpole::CartPoleV1;
modurl::prelude::* brings the common ModuRL traits and training types into
scope. Agent and MultiGym are traits that make .learn(),
.observation_space(), and .action_space() available.
A Few Terms
Before we build the program, it is worth separating two similar names; agent and actor.
In ModuRL, an Agent is the object that can act in an environment and learn
from it. For this example, the agent is a PPOAgent. It contains the PPO
rollout logic, loss calculations, optimizers, policy distribution, and neural
networks.
The actor network is the source of the policy’s action scores. It produces
logits, which are raw scores for each action.
ProbabilisticPolicyModel interprets those scores as
a categorical policy: it can sample an action from them and later compute the
log probability of that action. PPO needs both operations during training.
You will also see MLP below. MLP builds a dense feed-forward neural network:
linear layers with an activation between them. We use one MLP for the actor
network and one MLP for the critic network.
Choose a Device
Start main by choosing where tensors and models will live:
fn main() {
let device = Device::flex().autodiff();
The CPU device is the simplest first run. See Run on CUDA or Metal when the CPU version works.
Create Environments
PPO learns from batches of environment steps, so create several CartPole environments and wrap them as one vectorized environment:
let envs = (0..4)
.map(|index| CartPoleV1::builder().seed(42 + index).build().unwrap())
.collect::<Vec<_>>();
let mut env = DeviceMultiGymWrapper::new(
VectorizedGymWrapper::new(envs).unwrap(), Device::flex(), device.clone(),
);
Each inner CartPoleV1 is a single environment. VectorizedGymWrapper stacks
them so PPO can collect multiple transitions per step.
Read the Spaces
Ask the environment for its observation and action spaces before building the networks:
let observation_space = env.observation_space();
let action_space = env.action_space();
The observation space determines the input size for both networks. The action space determines how many action logits the actor network must produce.
Build the Actor Network
The actor network maps observations to action logits:
let actor_network = MLP::builder()
.input_size(observation_space.shape()[0])
.output_size(action_space.get_possible_values())
.options((&device, DType::F32))
.activation(Tanh)
.hidden_layer_sizes(vec![64, 64])
.build()
.expect("failed to build actor network");
The MLP here has two hidden dense layers of width 64. Its output size is the
number of action logits. For CartPole, that means two logits: one for pushing
left and one for pushing right. Later, CategoricalDistribution will interpret
those logits as a discrete policy.
Build the Critic Network
The critic maps observations to one value estimate:
let critic_network = MLP::builder()
.input_size(observation_space.shape()[0])
.output_size(1)
.options((&device, DType::F32))
.activation(Tanh)
.hidden_layer_sizes(vec![64, 64])
.build()
.expect("failed to build critic network");
This is another MLP with the same hidden-layer shape. PPO uses the critic to
estimate how good each observed state is, so the critic network has the same
input size as the actor network but only one output.
Create Optimizers
Give the actor network and critic network separate AdamW optimizers:
let optimizer_config = AdamWConfig::new();
let actor_optimizer = optimizer_config.init();
let critic_optimizer = optimizer_config.init();
Each network owns its parameters. Set learning rates in the network builder.
Assemble PPO
Wrap the actor network in ProbabilisticPolicyModel so PPOAgent can sample
actions and evaluate their log probabilities:
let policy =
ProbabilisticPolicyModel::new(actor_network, CategoricalDistribution);
let network_info = SeparatePPONetwork::builder()
.actor_network(policy)
.critic_network(critic_network)
.actor_optimizer(actor_optimizer)
.critic_optimizer(critic_optimizer)
.actor_learning_rate(3e-4)
.critic_learning_rate(3e-4)
.build().unwrap();
policy is the probabilistic policy PPO trains. In this configuration, the
policy model and value model are independent neural networks with independent
optimizers.
Train
Finally, build the PPOAgent and run learning:
let mut agent = PPOAgent::builder()
.observation_space(observation_space)
.action_space(action_space)
.network_info(network_info)
.batch_size(1024)
.mini_batch_size(64)
.clip_range(ConstantSchedule::new(0.2))
.training_horizon(10_000)
.build().unwrap();
agent.learn(&mut env, 10_000).expect("PPO learning failed");
println!("Training complete.");
}
batch_size controls how many transitions PPO collects before an update.
mini_batch_size controls how those transitions are split during optimization.
The clip range is the PPO policy-update bound.
training_horizon defines the total number of environment transitions over
which parameter schedules progress. This way, the schedules can span multiple learn calls.
Run the program with:
cargo run
When the training loop finishes, the program prints Training complete.. The
default run collects 10,000 environment steps, so it does more than compile the
program but is smaller than a longer experiment.
Complete File
After applying the pieces above, src/main.rs should look like this:
use burn::{
nn::Tanh,
optim::AdamWConfig,
tensor::{DType, Device},
};
use modurl::prelude::*;
use modurl_gym::classic_control::cartpole::CartPoleV1;
fn main() {
let device = Device::flex().autodiff();
let envs = (0..4)
.map(|index| CartPoleV1::builder().seed(42 + index).build().unwrap())
.collect::<Vec<_>>();
let mut env = DeviceMultiGymWrapper::new(
VectorizedGymWrapper::new(envs).unwrap(),
Device::flex(),
device.clone(),
);
let observation_space = env.observation_space();
let action_space = env.action_space();
let actor_network = MLP::builder()
.input_size(observation_space.shape()[0])
.output_size(action_space.get_possible_values())
.options((&device, DType::F32))
.activation(Tanh)
.hidden_layer_sizes(vec![64, 64])
.build()
.expect("failed to build actor network");
let critic_network = MLP::builder()
.input_size(observation_space.shape()[0])
.output_size(1)
.options((&device, DType::F32))
.activation(Tanh)
.hidden_layer_sizes(vec![64, 64])
.build()
.expect("failed to build critic network");
let optimizer_config = AdamWConfig::new();
let actor_optimizer = optimizer_config.init();
let critic_optimizer = optimizer_config.init();
let policy = ProbabilisticPolicyModel::new(actor_network, CategoricalDistribution);
let network_info = SeparatePPONetwork::builder()
.actor_network(policy)
.critic_network(critic_network)
.actor_optimizer(actor_optimizer)
.critic_optimizer(critic_optimizer)
.actor_learning_rate(3e-4)
.critic_learning_rate(3e-4)
.build()
.unwrap();
let mut agent = PPOAgent::builder()
.observation_space(observation_space)
.action_space(action_space)
.network_info(network_info)
.batch_size(1024)
.mini_batch_size(64)
.clip_range(ConstantSchedule::new(0.2))
.training_horizon(10_000)
.build()
.unwrap();
agent.learn(&mut env, 10_000).expect("PPO learning failed");
println!("Training complete.");
}
Where to Go Next
You now have a PPO training program with vectorized CartPole environments, a stochastic categorical policy, and a value model. Read Understand a PPO Training Run to learn what the example’s metrics show, or Use Vectorized Environments to work directly with batched environments.
Core Concepts
This chapter explains the names you will see in ModuRL examples. The goal is not to explain all of reinforcement learning at once. The goal is to make the types and builder arguments in the code feel less mysterious.
Agent
An Agent is the top-level object that interacts with an environment.
In ModuRL, an agent can do two things:
actfrom observationslearnfrom a vectorized environment
ModuRL provides agent types named after the algorithms they implement.
PPOAgent implements PPO, SACAgent implements Soft Actor-Critic,
DDPGAgent implements DDPG, TD3Agent implements TD3, DQNAgent implements
DQN, and DDQNAgent implements Double DQN.
An agent coordinates the pieces needed for training: model modules, optimizers, schedules, buffers, distributions, and update logic.
Observation and action spaces share TensorBatchTypes from modurl::tensor_rank.
This contract declares Kind, Item, and Batch. Item omits the batch axis;
Batch adds one leading batch axis. The types must agree with NextRank and
PrevRank. ObservationSpace<O> and ActionSpace<A> bind the batch type to a
native Burn tensor with the declared rank and kind.
For BoxSpace<2>, Item is Tensor<1> and Batch is Tensor<2>.
For Discrete, those types are Tensor<1, Int> and Tensor<2, Int>.
Gym::step uses the action space’s Item. Batched environment interaction and
agent inputs use each space’s Batch. Dtype and device requirements still
follow the environment and model contracts.
Custom spaces implement TensorBatchTypes once, then implement the observation
or action space methods. Space trait objects must specify Kind, Item, and
Batch. Original policy samples retain their own layout; ActionMap converts
them into environment action batches.
Environments and Spaces
An environment is the world the agent interacts with.
A single environment implements the Gym interface. It can reset, accept an
action, and return the next observation and reward.
Training usually needs more than one transition at a time, so ModuRL also uses
vectorized environments. A MultiGym behaves like a batch of environments.
When the agent sends one batch of actions, the vectorized environment steps each
inner environment and returns a batch of results.
VectorizedGymWrapper is the simple wrapper used in the getting-started
example. It turns several single environments into one vectorized environment.
With the multithreading feature enabled,
MultithreadedVectorizedGymWrapper can step the inner environments on worker
threads. Applications can also implement the public MultiGym trait when
they need a different way to manage or step a batch of environments.
Vectorized environments auto-reset each inner environment when it returns
done or truncated. The states field in MultiGymStepInfo contains the
next state to continue training from. If an inner environment ended, that next
state is already the reset state for the next episode.
A Space describes what observations or actions look like.
For observations, the space tells you the input shape your networks need. For actions, the space tells you what kind of actions the environment accepts.
CartPole has a discrete action space. That is why the PPO example uses
CategoricalDistribution: the policy chooses from a fixed set of actions.
Spaces also help convert policy outputs into environment actions. In PPO, the policy produces action information as tensors. The action space defines how those tensors map to valid actions for the environment.
Models and Policies
Models are neural network modules.
MLP builds a dense feed-forward network. Dense means each linear layer connects
all of its inputs to all of its outputs. In the getting-started example, both
networks are MLPs with two hidden layers.
MLP does not know about PPO by itself. It maps input tensors to output
tensors. The agent and policy model decide what those outputs mean.
A policy is the rule used to choose actions. A stochastic policy samples from
a distribution. PPO wraps its actor module in
ProbabilisticPolicyModel, whose distribution defines how to interpret the actor’s
output. D can be a distribution supplied by ModuRL or a user-defined type
that implements the public Distribution trait. This gives PPO the operations
it needs:
- sample an action representation
- compute the log probability of an action
- compute the policy entropy
For DQN and DDQN, an MLP is a Q-network. Its output has one value for each
discrete action. The agent selects an action from those values with
epsilon-greedy exploration instead of a probability distribution.
For DDPG and TD3, an actor MLP returns a continuous action directly. These
algorithms add Gaussian noise while collecting experience, but the underlying
policy is deterministic. Their scalar state-action critics receive an
observation and action together and return one Q value.
Read Models, Policies, and Distributions for how tensors move between actors, distributions, and action spaces. Read Value-Based Training for discrete-action Q-networks, or Deterministic Actor-Critic Training for continuous actors and state-action critics.
Tensors and Devices
ModuRL uses Burn tensors.
A tensor has a shape, data type, and device. The shape must match what the model or environment expects. The device says where the tensor lives, such as CPU, CUDA, or Metal.
Most examples start with:
let device = Device::flex();
The important rule is that tensors and models used together should live on the same device. If a model is on the CPU, the observations passed to it should also be on the CPU.
Training Configuration
An optimizer updates model parameters.
In PPO with separate networks, the two model modules can have separate optimizers. Each model owns the parameters that its optimizer updates.
A schedule is a value that changes during training. For example,
ConstantSchedule keeps the PPO clip range fixed. LinearSchedule can move a
value from one number to another over training progress.
In the CartPole Example
The CartPole PPO example configures a probabilistic policy that wraps a model
which produces policy scores. It also configures a separate model that estimates
values. PPOAgent uses the policy and value model while it collects experience
and updates their parameters.
These are roles in the PPO configuration, not separate library architecture
types. The public types in the example are MLP,
ProbabilisticPolicyModel, and PPOAgent.
Read Understand a PPO Training Run to inspect the results of the example, or Use Vectorized Environments to work with batched environment steps directly.
Models, Policies, and Distributions
A module returns a tensor. An environment expects an action. The agent decides what the tensor means and how it becomes that action.
Two possible paths are:
direct: model -> action representation -> action space -> environment
probabilistic:
probabilistic policy -> action representation -> action space -> environment
The algorithm determines whether the action path uses a distribution. A model can return an action representation that an action space understands directly. DDPG and TD3 use this direct path for their deterministic actors, then add Gaussian noise inside the agent during collection. An agent can also insert other selection logic. DQN and DDQN, for example, choose from Q-values with epsilon-greedy selection.
On the probabilistic path, the policy uses a model and a distribution together. The distribution is part of the policy rather than a standalone stage.
Models Produce Tensors
A model transforms an input tensor into an output tensor. MLP supplies dense
layers and activation functions, but it does not assign meaning to its output.
Let B be the batch size and I the number of input features. A dense MLP
receives [B, I]. The features may be observations, the output of an
earlier module, or other values prepared by an agent. Other architectures may
use different input shapes.
The component that receives a model’s output determines its shape and meaning.
A critic, for example, returns [B, 1] state-value estimates. A Q-network
returns action values. A probabilistic actor returns parameters for a
distribution.
Probabilistic Policies Use a Distribution
ProbabilisticPolicyModel<T> owns the native model and distribution directly.
T: PolicyTypes groups their types and the native tensor signature. The trait contains associated types and no methods.
The tuple implementation groups the types as:
(Model, Distribution, (ObservationTensor, ParameterTensor, ActionTensor))
The tensor types contain their ranks and kinds. The group describes types; it does not store tensors or provide tensor operations.
Policy implementations use native Tensor<O, K>, Tensor<P>, and Tensor<A> signatures.
Policy models implement Burn’s Module and ModuRL’s Forward<O, P>.
Each model declares InputKind and OutputKind as associated types. Policy parameter outputs must use Float.
Custom models can accept integer or boolean observations and perform explicit conversions.
Construct a policy without specifying the type group:
let policy = ProbabilisticPolicyModel::new(
actor,
CategoricalDistribution,
);
Forward::forward takes Tensor<O, Self::InputKind> and returns Result<Tensor<P, Self::OutputKind>, Self::Error>.
Both tensors start with a batch axis, and the model must preserve batch size.
The model performs any observation dtype or device conversion explicitly.
All execution uses the same owned model that Burn visits for optimizer updates, records, and device transfers.
policy.module() returns that native model directly.
A component that only owns the policy needs one type parameter:
struct PolicyOwner<T: PolicyTypes> {
policy: ProbabilisticPolicyModel<T>,
}
Code that performs tensor calculations still constrains the required native tensor types or policy ranks. The group does not remove the underlying type information. Fully explicit policy types contain the tuple shown above.
The policy implements ProbabilisticPolicy<O, A>. Its sample and mode
operations pass model outputs directly to the distribution. Its
log_prob_and_entropy operation calls D::dist_eval and returns two [batch_size, 1] tensors.
Distribution is a public trait. ModuRL currently supplies
CategoricalDistribution and GaussianDistribution, but applications can add
their own implementations:
let policy = ProbabilisticPolicyModel::new(actor, MyDistribution::default());
A custom Distribution<P, A> implementation provides sample, mode,
dist_eval, and an associated Error type. It documents its parameter and action layouts.
Ranks are checked at compile time. Dynamic dimensions, dtypes, and devices need runtime validation.
Its action representation must match the chosen ActionMap input.
Spaces Produce Environment Actions
ActionMap::tensor_from_neurons converts an action representation into the tensor
passed to the environment.
Discrete selects the index of the largest component. BoxSpace clamps each
component to its lower and upper bounds. PPO retains the original sample for
log-probability calculations while sending the converted action to the
environment.
Built-In Distributions
Categorical Distribution
Let C be the number of discrete choices. CategoricalDistribution expects
one logit for each choice:
| Value | Shape |
|---|---|
| Model output | [B, C] |
| Sampled representation | [B, C] |
Action after Discrete conversion | [B, 1] |
| Log probability | [B, 1] |
| Entropy | [B, 1] |
The logits are unnormalized scores. Sampling adds an independent random
perturbation called Gumbel noise to each score. Taking the largest perturbed
score samples choices according to the logits. Discrete then selects that
score’s index.
CartPole has two actions, so the getting-started actor returns two logits per observation.
Gaussian Distribution
Let A be the number of components in a one-dimensional continuous action
space with shape [A]. GaussianDistribution expects two values for each
component. Along dimension 1, all A means come first, followed by all A log
standard deviations:
[mean_0, ..., mean_(A-1), log_std_0, ..., log_std_(A-1)]
| Value | Shape |
|---|---|
| Model output | [B, 2 * A] |
| Means | [B, A] |
| Log standard deviations | [B, A] |
| Sampled representation | [B, A] |
Action after BoxSpace conversion | [B, A] |
| Log probability | [B, 1] |
| Entropy | [B, 1] |
GaussianDistribution applies exp to the log standard deviations before
sampling. Neither half of the model output contains log probabilities. Each
action component uses an independent Gaussian, and dist_eval sums the
component log probabilities and entropies into one value per batch row.
The actor can return the complete [B, 2 * A] tensor itself. It can also combine
state-dependent means with a separate trainable log_std, as the MuJoCo PPO
example does.
Read Getting Started for a categorical policy,
Soft Actor-Critic for policies that expose exact or sampled action
expectations, Deterministic Actor-Critic
Training for direct continuous actors,
Value-Based Training for action selection without a
distribution, and crates/examples/examples/ppo_mujoco.rs for a Gaussian
policy.
PPO
PPOAgent is a ModuRL agent that implements Proximal Policy Optimization
(PPO). It collects transitions from a MultiGym, then updates a stochastic
policy and a value model from that experience.
The separate-network PPO configuration owns the policy and value models, such
as Burn MLP models. Each model has its own ModuleOptimizer and explicit
learning rate. SharedPPONetwork is also
available when a configuration needs one shared model followed by separate
policy and value heads.
SeparatePPONetwork::builder() requires actor_learning_rate and
critic_learning_rate. SharedPPONetwork::builder() requires learning_rate.
Both builders return Result. PPO’s builder also requires observation_space
and returns Result. The constructor names remain new.
Pass a SeparatePPONetwork or SharedPPONetwork directly to .network_info(network).
PPO estimates advantages with GAE (lambda 0.95) by default. Pass any
AdvantageEstimator to .advantage_estimator(..), for example
.advantage_estimator(GAE::new(0.9)?). An estimator returns an
AdvantageEstimate with the advantages and the critic’s value targets; GAE
uses values + advantages as those targets.
The built-in NStep::new(n)? estimator is an alternative for n-step returns.
Use GAE::new(0.95)?.with_value_lambda(1.0)? to use a separate lambda for value targets.
The built-in UPGO::new(GAE::new(0.95)?) estimator is also available for upgoing policy updates.
The agent infers its model group from PPONetwork::ModelGroup.
Custom PPONetwork<O, H, L> implementations must set ModelGroup to
PPOModelGroup<Self, Tensor<O, Self::InputKind>, Tensor<H, Self::LatentKind>, Tensor<L>>.
The network must implement Module. Its module operations must traverse model parameters and training flags.
PPO calls Module::valid() and removes autodiff from inference observations without changing their kind or dtype.
It reuses one validation network for collection and fixed value estimates, then releases it before optimizer updates.
The built-in networks supply these implementations directly. Their learning-rate schedules must implement Send + Sync.
Schedules are shared through Arc when a validation network is created.
Observations keep [batch_size, ...observation_shape], their native kind, and their dtype.
Models perform any required conversions. Separate models accept observations directly;
shared trunks produce Float features, and shared heads accept Float inputs. The compiler rejects incompatible kind connections.
PPO has no dtype setting: critic outputs must match the environment reward dtype.
Training rejects mismatches without casting rewards or critic outputs. Returns and advantages share the reward dtype.
Values, log probabilities, and advantages use [batch_size, 1].
Losses use [1]. Raw reward logs preserve the environment’s reward dtype.
reset_current_observations() clears observations retained between training calls.
Start with Getting Started for the complete CartPole program. Then read Understand a PPO Training Run before changing its configuration.
The continuous-control example supports ant, half-cheetah, hopper, and
walker2d. Normally enable one environment feature; rendering may be added
to that feature to open a viewer. Cargo features are additive, so builds that
enable several environment features use the priority ant, half-cheetah,
hopper, then walker2d. PPO writes TensorBoard events below
runs/ppo_mujoco/ and also displays terminal graphs.
PPO can use any compatible Distribution implementation. ModuRL currently
supplies categorical and Gaussian distributions, and applications can define
their own. Models, Policies, and
Distributions explains the extension
point, the built-in tensor layouts, and how sampled representations become
environment actions.
For value-based agents in discrete action spaces, read Value-Based Training.
Atari training saves Burn checkpoints with the .bpk extension.
ppo_atari_play loads the shared encoder, actor, and critic from that checkpoint.
Candle safetensors checkpoints require conversion before use.
agent.network().models() returns a snapshot of the current owned models for saving.
For shared networks, the snapshot contains the encoder, actor head, and critic head.
For separate networks, it contains the actor and critic.
Understand a PPO Training Run
The Getting Started program confirms that training completed, but it does not
record metrics. Add a PPOLogger when you need to compare runs or diagnose a
configuration. PPOLogger::log receives update metrics, while
PPOLogger::log_collection receives the newest environment rewards and any
episodes completed during that vectorized step.
Each PPOLogEntry contains tensors for one optimization minibatch. Aggregate
them before graphing or comparing them across a run. The graphs are useful for
comparing or diagnosing runs.
Start With Episode Performance
CartPole gives a reward of one for each step that keeps the pole balanced. The example graphs completed episode returns and lengths, so longer episodes are the clearest sign that the policy is improving.
PPOCollectionLogEntry::completed_episodes contains one PPOEpisodeLogEntry
for each environment that terminated or truncated on that vectorized step.
Each episode entry reports its environment index, return, length, ending flags,
and collection timestep. Partial episodes carry across PPO rollout boundaries
and repeated learn calls.
PPOCollectionLogEntry::infos contains the typed step metadata from each
vectorized environment in the same environment order as collection_rewards.
Wrappers can add logging-only values there without changing the reward used for
training. For example, RecordRawRewardGym records rewards before outer reward
wrappers normalize or clip them.
Read the Other Metrics Together
The logger exposes the following values through PPOLogEntry:
actor_loss: the policy objective used for the updatecritic_loss: the value-model objective used for the updateentropy: how spread out the policy’s action choices arekl_divergence: how far the updated policy moved from the policy that collected the rolloutexplained_variance: how well the value model accounts for return variationrewards: rewards from the collected transitions
Loss values do not have one universally good target. They are most useful when compared with episode length and with earlier runs using the same configuration.
Entropy commonly falls as the policy becomes more certain. A rapid collapse can mean the policy stopped exploring too early. KL divergence shows the size of the policy change. Large, erratic changes suggest that the update settings may be too aggressive.
Explained variance shows whether the value model predicts returns better than a constant average prediction: values nearer one are better, but early updates can be noisy.
Change One Setting at a Time
The CartPole example sets these PPO builder values:
.batch_size(2048)
.mini_batch_size(64)
.ent_coef(0.005)
.clip_range(ConstantSchedule::new(0.2))
.num_epochs(10)
.training_horizon(120_000)
Use one changed value per experiment. Keep the device, environment count, model shape, and remaining PPO settings fixed while comparing the graph. That makes a change in episode length or KL divergence easier to attribute to its cause.
Next, read Use Vectorized Environments to see how PPO receives batches of transitions.
A2C
A2CAgent implements synchronous Advantage Actor-Critic by configuring an
inner PPO agent. The wrapper disables policy and value clipping, performs one
optimization epoch, and uses the complete rollout as one minibatch. These
algorithm-defining settings cannot be changed through the A2C builder.
The remaining defaults follow Stable-Baselines3 A2C: gamma is 0.99, the
advantage estimator is GAE with lambda 1.0, advantages and returns are not
normalized, vf_coef is 0.5, ent_coef is 0.0, and gradient clipping is
0.5. The application supplies Burn models, optimizers, and explicit learning
rates as part of its network configuration.
Use SharedA2CNetwork when the actor and critic share a model, or
SeparateA2CNetwork when they have independent models and optimizers. Wrap the
successful build result with A2CNetworkInfo::shared or A2CNetworkInfo::separate before
building the agent. A2C also provides typed errors, log entries, and the
A2CLogger trait. Pass an A2C logger directly to the agent’s logging_info
builder field.
Both network builders return Result and own concrete Burn models.
The separate builder requires actor_learning_rate and critic_learning_rate;
the shared builder requires learning_rate.
Learning-rate schedules must implement Send + Sync.
The agent builder requires both observation_space and action_space.
A2C has no dtype setting. Models preserve native observations and must produce
critic values with the environment reward dtype.
reset_current_observations() clears observations retained between training calls.
Scalar actions and statistics use [batch_size, 1].
batch_size is the total number of transitions in one rollout, across every
vectorized environment. Choose a value divisible by the environment count so
the agent performs exactly one full-batch update.
The MuJoCo example uses a separate Gaussian actor and critic. Run it with one of the supported environment features:
cargo run -p examples --example a2c_mujoco --features ant
cargo run -p examples --example a2c_mujoco --features half-cheetah
cargo run -p examples --example a2c_mujoco --features hopper
cargo run -p examples --example a2c_mujoco --features walker2d
Choose one command for the environment you want to train. Read PPO for the underlying on-policy training model.
Add rendering alongside the environment feature to open a viewer, for
example --features ant,rendering. A2C writes TensorBoard events below
runs/a2c_mujoco/ and also displays terminal graphs. If several environment
features are enabled by an additive feature build, the example uses ant,
half-cheetah, hopper, then walker2d as its selection priority.
Soft Actor-Critic
The SAC agent uses Burn and is available through modurl::agents::sac and the prelude. Its critics use the generic action-value API in modurl::agents::action_value_critic.
SACModelGroup groups the policy, critic, and execution tensor types.
The builder infers this group from the supplied models and spaces.
Explicit agent annotations use SACAgent<'a, G, OS, AS, GE, I, A>, where G is the model group.
The observation and candidate ranks are inferred; A is the environment action batch rank.
Soft Actor-Critic (SAC) trains one stochastic actor and two Q-value critics from replayed transitions. The algorithm traditionally known as SAC almost always uses exactly two critics. ModuRL accepts any nonempty critic ensemble, but use two unless you are intentionally studying another aggregation scheme.
SAC is primarily a continuous-control algorithm. This chapter follows the
continuous MuJoCo example and explains the actor, critic, entropy, replay, and
device choices you need to adapt it. You should already be familiar with
MultiGym, native Burn models, and probabilistic policies from Core
Concepts.
Run Continuous SAC
The example supports Ant, HalfCheetah, Hopper, and Walker2d:
cargo run --release -p examples --example sac_mujoco --features ant
cargo run --release -p examples --example sac_mujoco --features half-cheetah
cargo run --release -p examples --example sac_mujoco --features hopper
cargo run --release -p examples --example sac_mujoco --features walker2d
Normally enable one environment feature; add rendering to it to open a
viewer. Additive builds with several environment features select ant,
half-cheetah, hopper, then walker2d. The program uses CPU by default. Enable cuda or metal to select an accelerator. It trains for one million collected
transitions, then displays terminal graphs for episode performance and SAC
diagnostics.
The complete source is crates/examples/examples/sac_mujoco.rs. It contains
four parts:
- A Gaussian parameter module and squashed distribution for bounded actions.
- Two independently optimized online/target critic pairs.
- Automatic entropy tuning.
- An
SACAgentthat owns replay and runs collection and optimization.
The agent stores every collected transition in replay, including transitions
from the initial random-action phase. Once collection_timestep reaches
training_start, each new transition triggers one replay optimization step.
This gives the implementation an update-to-data ratio of one. A vectorized step
with N environments stores N transitions and, after the threshold, triggers
N updates.
Build the Continuous Policy
GaussianModule returns distribution parameters rather than an environment
action. ProbabilisticPolicyModel combines the module with a distribution to
form the actor’s policy. For an action with A components,
GaussianDistribution expects A means followed by A log standard
deviations.
The module uses observation-dependent means and one trainable log standard deviation per action component:
#[derive(Module, Debug)]
struct GaussianModule {
mean: MLP,
log_std: Param<Tensor<2>>,
}
impl Forward<2, 2> for GaussianModule {
type InputKind = Float;
type OutputKind = Float;
type Error = ModelError;
fn forward(&self, observations: Tensor<2>) -> Result<Tensor<2>, Self::Error> {
let mean = self.mean.forward(observations)?;
let log_std = self.log_std.val().expand(mean.dims()).clamp(-20.0, 2.0);
Ok(Tensor::cat(vec![mean, log_std], 1))
}
}
Build the distribution with the environment’s action shape:
let action_size = action_space.shape()[0];
let mean = sac_mlp(observation_size, action_size, 0.01, &device)?;
let log_std = Param::from_tensor(Tensor::zeros([1, action_size], (&device, DType::F32)));
let distribution = TransformedDistribution::new(
GaussianDistribution::<2>::new([action_size])?,
TanhTransform,
);
let policy = ProbabilisticPolicyModel::new(
GaussianModule { mean, log_std },
distribution,
);
TanhTransform keeps policy candidates between -1 and 1, matching the
included MuJoCo environments’ action bounds. The transform also includes its
Jacobian correction in action log probabilities.
Build Two Critics
Each ActionValueCritic owns an online Burn model, a detached target snapshot, and an optimizer.
The constructor clones the online model to initialize the target. Each critic has its own explicit learning rate.
A continuous critic receives observations and actions and returns one Q-value per batch item.
ScalarStateActionCritic concatenates flattened observation/action pairs before calling the model.
The model implements Burn’s Module and ModuRL’s Forward<2, 2>.
It accepts Float [batch_size, observation_size + action_size] and returns [batch_size, 1].
Observations and actions must share dtype and device; the adapter does not cast them.
let critic = ActionValueCritic::builder()
.online_network(ScalarStateActionCritic::new(online_network))
.optimizer(critic_optimizer)
.learning_rate(3e-4)
.build()?;
Create each critic with its own online model and optimizer. Separate target models and parameter maps are unnecessary.
backward_step(&loss) optimizes the online model from a Float loss [1] and leaves its target unchanged.
hard_update() copies the online model into a detached target snapshot.
polyak_update(tau) blends each floating-point target parameter with its matching online parameter.
Both target-evaluation methods detach their results. Actor evaluation retains action gradients while excluding critic-parameter gradients.
ActionValueCriticNetwork<O, A, C> describes native observation and action kinds.
Ranks O and A include batch and event axes; rank C adds a candidate axis to rank A.
The methods retain borrowed tensor inputs. For a scalar critic, use online_replay_values::<2, 2, 3>(&observations, &actions) for rank-2 batches.
DiscreteVectorHeadCritic accepts native observation kinds and a Float model output [batch_size, action_count].
It requires Int replay actions [batch_size, 1] or Int candidates [batch_size, candidate_count, 1].
The adapter checks index bounds before gathering values, reading one Boolean result to the host.
The model must perform any required observation conversion explicitly.
The default aggregation mode is ActionValueCriticAggregationMode::Min. This is the
traditional SAC choice: using the lower estimate helps reduce optimistic
Q-value errors.
ModuRL also exposes Mean, Median, and Max for experiments with different
critic ensembles:
| Choice | Use |
|---|---|
Min | Traditional SAC with two critics |
Mean | Arithmetic mean of the critic ensemble |
Median | Middle estimate for an ensemble with several critics |
Max | Optimistic experimental estimate |
Configure Entropy
The entropy coefficient, usually written as alpha, balances expected return against policy entropy. Larger values put more weight on keeping the policy’s action distribution stochastic. Use automatic entropy tuning for the standard SAC workflow:
let log_alpha = Param::from_tensor(Tensor::zeros([1], (&device, DType::F32)));
let alpha_optimizer = AdamWConfig::new().with_weight_decay(0.0).init();
let entropy = SACEntropyConfiguration::automatic(log_alpha, alpha_optimizer, 3e-4)?;
automatic asks the policy for its default target entropy. The built-in
Gaussian policy uses the negative action-component count. An affine
distribution transform adjusts that default for its scale.
Pass a ParameterSchedule when the desired entropy should change during the
run:
let target_entropy = LinearSchedule::new(-2.0, -1.0);
let entropy = SACEntropyConfiguration::automatic_with_target_schedule(
log_alpha,
alpha_optimizer,
3e-4,
target_entropy,
)?;
LinearSchedule moves the target entropy from -2.0 to -1.0. Choose values
appropriate for the environment’s action dimension rather than copying these
illustrative values.
Use a fixed coefficient when you need a known constant or want to reproduce a configuration without an entropy optimizer:
let entropy = SACEntropyConfiguration::fixed(0.2)?;
Entropy configuration constructors return a typed error for invalid parameters.
A fixed coefficient must be finite and nonnegative. Automatic tuning requires an explicit learning rate and Float log alpha [1].
Assemble the Agent
The continuous example assembles the pieces as follows:
let mut agent = SACAgent::builder()
.policy(policy)
.actor_optimizer(actor_optimizer)
.actor_learning_rate(3e-4)
.critics(vec![critic_1, critic_2])
.entropy_configuration(entropy)
.action_space(action_space)
.observation_space(observation_space)
.aggregation_mode(ActionValueCriticAggregationMode::Min)
.training_horizon(total_timesteps)
.build()?;
agent.learn(&mut env, total_timesteps)?;
training_horizon defines how many collected transitions parameter schedules
take to reach their final values. Schedule progress continues across learn
calls and stops at the end of the horizon.
Important defaults and constraints are:
| Setting | Default | Constraint or effect |
|---|---|---|
gamma | 0.99 | Finite and between zero and one |
tau | 0.005 | Target-network update coefficient from zero to one |
replay_capacity | 1_000_000 | At least batch_size; at first learn, larger than and divisible by env.num_envs() |
batch_size | 256 | Nonzero |
training_start | 1_000 | Random-action collection before optimization |
samples | 1 | Continuous expectation candidates per observation |
training_horizon | First learn call’s num_timesteps | Nonzero schedule horizon in transitions |
actor_learning_rate | Required | Finite and positive |
dtype | F32 | Floating-point training statistics; does not cast observations |
Models and automatic log alpha must use the optimization device. Critic outputs must use the configured compute dtype.
Replay storage is allocated for the first collected transition and retained across learn calls.
Replay preserves native observation kind and first-batch dtype; later observation dtype changes return an error.
User models or explicit preprocessing perform input conversions. Rewards and policy statistics are converted to compute dtype for optimization.
The agent owns concrete native policy and critic models; the builder returns SACConfigurationError for invalid construction.
Runtime failures retain policy, critic, environment, space, tensor-read, and replay causes in SACError.
A categorical CPU example is also available:
cargo run --release --locked -p examples --example sac_cartpole
samples controls the Monte Carlo estimate used by continuous distributions.
Larger values cost more critic evaluations but use more candidates to estimate
the actor and target expectations.
Q-value clipping is an optional experimental stabilization:
.q_value_clip(0.5)
The value is a positive bound on a critic’s update around its target-network estimate. Its useful scale depends on environment rewards, so it remains opt-in.
Implement a Custom Critic Adapter
Use ScalarStateActionCritic unless your critic has a different input or
output layout. A custom ActionValueCriticNetwork<O, A, C> must implement Burn’s Module and follow these contracts:
| Operation | Inputs | Output |
|---|---|---|
replay_values | observations [B, ...observation_shape], actions [B, ...action] | [B, 1] |
policy_values | observations [B, ...observation_shape], candidates [B, K, ...action] | [B, K, 1] |
actor_values | observations [B, ...observation_shape], candidates [B, K, ...action] | [B, K, 1] |
actor_values must preserve gradients through differentiable candidate
actions while excluding gradients to critic parameters.
Discrete SAC Is an API Variant
SACAgent can also work with categorical policies and
DiscreteVectorHeadCritic. Experimental discrete-SAC configurations can select
mean critic aggregation with .aggregation_mode(ActionValueCriticAggregationMode::Mean)
and add an entropy-drift penalty with .entropy_change_penalty(0.5). It is not
the canonical example because traditional SAC is a continuous-control algorithm.
Read Understand an SAC Training Run to add a logger and interpret its metrics. Read Run on CUDA or Metal when replay storage should remain on the CPU while optimization runs on an accelerator.
Understand an SAC Training Run
The SAC examples confirm that the actor, critics, replay buffer, and entropy
configuration can train together. Add a SACLogger when you need to compare
runs or diagnose one of those components.
SACLogger::log_update receives metrics from replay optimization.
SACLogger::log_collection receives rewards and completed episodes from the
current environment step. Keep these streams separate: an update describes a
sampled replay batch, while collection describes the policy’s newest behavior.
Add a Logger
This logger prints occasional update values and every completed episode:
use modurl::prelude::*;
struct ConsoleLogger;
impl<I> SACLogger<I> for ConsoleLogger {
fn log_update(&mut self, entry: &SACLogEntry) {
if entry.update_index % 1_000 != 0 {
return;
}
let actor_loss = entry
.actor_loss
.clone()
.try_into_scalar::<f32>()
.expect("actor loss must be scalar");
let alpha = entry
.alpha
.clone()
.try_into_scalar::<f32>()
.expect("alpha must be scalar");
println!(
"step={} update={} actor_loss={actor_loss:.4} alpha={alpha:.4}",
entry.collection_timestep,
entry.update_index,
);
}
fn log_collection(&mut self, entry: &SACCollectionLogEntry<I>) {
for episode in &entry.completed_episodes {
println!(
"step={} env={} return={} length={} terminated={} truncated={}",
episode.collection_timestep,
episode.environment_index,
episode.episode_return,
episode.episode_length,
episode.terminated,
episode.truncated,
);
}
}
}
Pass a mutable reference while building the agent:
let mut logger = ConsoleLogger;
let mut agent = SACAgent::builder()
// Keep the remaining SAC configuration unchanged.
.logger(&mut logger)
.build()?;
agent.learn(&mut env, total_timesteps)?;
The repository’s sac_mujoco example uses SACGrapher to aggregate these
callbacks into terminal graphs.
Start With Episode Performance
SACCollectionLogEntry::completed_episodes contains one
SACEpisodeLogEntry for each environment that ended during the latest
vectorized step. Each entry reports:
environment_index: which inner environment completedepisode_return: the sum of that episode’s rewardsepisode_length: the number of environment stepsterminated: whether the environment reached a terminal statetruncated: whether the episode was truncatedcollection_timestep: the total collected-transition count at completion
Episode return is the clearest measure of task performance. Compare it across several episodes rather than treating one episode as a trend. Episode length is useful when termination timing is meaningful for the environment.
SACCollectionLogEntry also contains:
collection_rewards: one reward per inner environment from the newest stepinfos: the corresponding typed environment metadatacollection_timestep: the total number of collected transitionsreplay_len: the number of entries currently held in replay
log_collection runs once per vectorized environment step. With N inner
environments, one callback normally advances collection_timestep by N.
Read Update Metrics as a Group
After training_start, SAC performs an optimization step for each collected
transition. Each call to log_update describes one sampled replay batch.
update_index is the zero-based optimization count, while
collection_timestep identifies the transition that triggered the update.
Let B be replay batch size and K the number of policy candidates. For a
categorical policy, K is the action count. For a sampled continuous policy,
K is the configured samples value.
| Field | Shape | Meaning |
|---|---|---|
critic_losses | One [1] tensor per critic | Critic objectives for the replay batch |
actor_loss | [1] | Objective used for the actor update |
alpha_loss | Optional [1] | Automatic entropy-coefficient objective |
entropy_change_loss | Optional [1] | Discrete stabilization penalty |
target_entropy | Optional f64 | Current automatic target entropy |
alpha | [1] | Current entropy coefficient |
bellman_targets | [B, 1] | Detached soft targets shared by all critics |
policy_log_probabilities | [B, K, 1] | Candidate log probabilities |
policy_weights | [B, K, 1] | Candidate expectation weights |
policy_q_values | [B, K, 1] | Aggregated Q-values used by the actor |
replay_rewards | [B, 1] | Rewards from the sampled replay entries |
Losses do not have a universal target value. Reward scale, model architecture, and entropy configuration all change their magnitude. Compare them with episode return and with earlier runs using the same environment and reward handling.
Critic loss measures disagreement with the soft Bellman targets. A persistent increase alongside unstable Q-values can indicate an aggressive learning rate or reward scale. A small critic loss alone does not prove that the policy is good.
Actor loss combines expected Q-values and the entropy term. It can be negative, and its raw value is not a task score. Use it to spot abrupt changes rather than to rank policies.
Alpha controls the strength of the entropy term. A larger alpha places more weight on uncertain actions. With automatic tuning, compare alpha, policy entropy, and target entropy together.
The expected policy entropy for one update is:
-mean(sum(policy_weights * policy_log_probabilities, candidate_axis))
Entropy commonly falls when a policy becomes more certain. Whether that is healthy depends on the target entropy and episode performance. A rapid collapse with poor returns suggests that exploration ended too soon.
entropy_change_loss appears only when the stabilization configuration enables
the replay-to-current entropy penalty. alpha_loss and target_entropy appear
only with automatic entropy tuning.
Compare Collection and Replay Values
collection_rewards show the newest environment behavior.
replay_rewards come from a randomly sampled historical batch. They need not
move together at each callback.
The same distinction applies to time:
collection_timestepcounts transitions gathered from environments.update_indexcounts replay optimization steps.replay_lengrows until it reaches replay capacity.
Graph completed episode returns against collection_timestep. Graph update
losses, alpha, expected policy entropy, and mean Bellman targets at a lower
frequency if logging every optimization step is too expensive.
Return to Soft Actor-Critic to change entropy, critic aggregation, or stabilization. Read Run on CUDA or Metal to separate replay storage from optimization.
Deterministic Actor-Critic Training
Use DDPG or TD3 when the environment has a continuous action space and you want a deterministic policy trained from replay. Both algorithms use an actor that returns one action directly and a critic that estimates the value of a state-action pair.
This chapter explains the pieces shared by DDPGAgent and TD3Agent. Read
DDPG for the smaller, single-critic configuration. Read
TD3 when you want twin critics, target-policy smoothing, and delayed
actor updates.
Choose DDPG or TD3
DDPG and TD3 share the same collection and replay loop. TD3 changes how the agent calculates targets and when it updates the actor:
| Behavior | DDPGAgent | Canonical TD3Agent |
|---|---|---|
| Critics | Exactly one | Two |
| Target Q estimate | The sole target critic | The smaller target-critic estimate |
| Target action | Target actor action | Target actor action plus clipped noise |
| Actor update | Every replay update | Every second replay update |
| Target-network update | Every replay update | With each delayed actor update |
| Actor objective | The sole online critic | The first online critic |
DDPG is the direct deterministic actor-critic baseline. TD3 adds safeguards against overestimated Q values and an actor that exploits narrow errors in the critic. Start with TD3 when those safeguards are appropriate. Choose DDPG when you need the single-critic algorithm or a simpler baseline for comparison.
ModuRL also lets TD3Agent use any nonempty critic ensemble and explicit
aggregation modes. Those settings are experimental variants, not canonical
TD3.
Follow the Shared Training Loop
Both agents collect and train in the same order:
- Before
training_start, sample actions uniformly from theBoxSpace. - After
training_start, run the online actor and add Gaussian exploration noise. - Clamp the resulting actions to the action-space bounds and store each transition in replay.
- On collection timesteps selected by
update_frequency, sample a replay batch and update every online critic. - Update the online actor at the algorithm’s actor-update interval.
- Polyak-update the target actors and critics whenever the actor is updated.
Collection timesteps count transitions, not calls to the vectorized
environment. A step with N environments adds N transitions. Each transition
whose global index is a multiple of update_frequency causes one replay update
after the warm-up threshold.
training_horizon records the intended number of collected transitions and
tracks progress across calls to learn. DDPG and TD3 do not currently expose
scheduled hyperparameters, but the shared progress counter still determines
the global collection timestep used by training and logging.
Match the Actor Contract
DDPG and TD3 use owned models. Each actor implements Module and Forward<O, A>.
Observations use [batch_size, ...observation_shape]; actions use [batch_size, ...action_shape].
The agent creates a detached target actor during construction.
The DDPG chapter shows the complete builder configuration.
The supplied BoxSpace clamps policy outputs to its bounds.
The exploration-noise standard deviation is measured in policy output units.
The agent adds noise before mapping policy outputs to environment actions.
Use act_deterministic for evaluation without exploration noise.
Match the Critic Contract
DDPG requires one ActionValueCritic with an owned online network and optimizer.
TD3 accepts a nonempty ensemble of these critics.
The critic creates a detached target network automatically.
A scalar critic maps observations and actions to Q values [batch_size, 1].
Wrap an MLP with input width observation_size + action_size in ScalarStateActionCritic.
The DDPG chapter shows this construction.
Keep Replay and Models on the Intended Devices
Optimization runs on the device of the actor and critic parameters, which must
all share one device. Replay is stored there too unless .replay_device(..)
keeps detached transitions on another device, such as the CPU, and moves
sampled batches to the optimization device.
The agent does not move models or environments. Build actors and critics on the optimization device.
Target networks inherit their online network device.
Keep CPU environments on CPU and use DeviceMultiGymWrapper for transfers.
Read Run on CUDA or Metal before splitting replay
storage from optimization.
The DDPG chapter uses these pieces to build the single-critic baseline. The TD3 chapter shows how to use two critics, target-policy noise, and delayed actor updates.
DDPG
Deep Deterministic Policy Gradient (DDPG) trains one deterministic actor and
one Q-value critic from replayed transitions. This page shows how the complete
MuJoCo example connects those networks to DDPGAgent.
You should already understand the actor, critic, target-network, and replay contracts in Deterministic Actor-Critic Training.
Run the DDPG Example
The example supports Ant, HalfCheetah, Hopper, and Walker2d:
cargo run --release -p examples --example ddpg_mujoco --features ant
cargo run --release -p examples --example ddpg_mujoco --features half-cheetah
cargo run --release -p examples --example ddpg_mujoco --features hopper
cargo run --release -p examples --example ddpg_mujoco --features walker2d
Normally enable one environment feature; add rendering to it to open a
viewer. Additive builds with several environment features select ant,
half-cheetah, hopper, then walker2d. The program uses the CPU by default. Enable cuda or metal to select an accelerator. It trains for one million collected
transitions, then displays terminal graphs for optimization and episode
metrics.
The complete source is crates/examples/examples/ddpg_mujoco.rs. It contains
four parts:
- An owned actor model.
- One
ActionValueCriticwith an owned online network and optimizer. - A bounded continuous action space and replay-device strategy.
- A
DDPGAgentthat owns collection and optimization.
Build the Actor and Critic
The actor implements Module and Forward<O, A>.
Observation rank O and action rank A include the batch axis.
The agent creates a detached target actor automatically.
ActionValueCritic creates its own detached target network.
use modurl::agents::action_value_critic::ScalarStateActionCritic;
let online_actor = MLP::builder()
.input_size(observation_size)
.output_size(action_size)
.hidden_layer_sizes(vec![64, 64])
.activation(Relu)
.output_activation(Tanh)
.options((&device, DType::F32))
.build()?;
let critic = ActionValueCritic::builder()
.online_network(ScalarStateActionCritic::new(online_critic))
.optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
.learning_rate(1e-3)
.build()?;
The critic receives observations [batch_size, ...observation_shape] and actions [batch_size, ...action_shape].
It returns Q values [batch_size, 1].
ScalarStateActionCritic flattens event axes and joins Float observations and actions.
Its inputs must share dtype and device. Custom critics can accept other native observation kinds.
After each replay update, targets move toward online parameters by tau.
The actor learns to maximize the online critic’s mean Q estimate. Equivalently, the optimizer minimizes the negative mean Q value.
Assemble the Agent
The builder accepts owned models and optimizers:
let mut agent = DDPGAgent::builder()
.online_actor(online_actor)
.actor_optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
.actor_learning_rate(1e-4)
.critic(critic)
.action_space(action_space)
.observation_space(observation_space)
.gamma(0.99)
.tau(0.005)
.exploration_noise(0.1)
.replay_capacity(1_000_000)
.batch_size(256)
.training_start(10_000)
.training_horizon(TOTAL_TIMESTEPS)
.logger(&mut grapher)
.build()?;
agent.learn(&mut env, TOTAL_TIMESTEPS)?;
The action_space implements ActionMap<A, A> with Float actions.
BoxSpace clamps actor outputs to the environment bounds.
Actor outputs must match the policy event shape. Environment actions must match the action event shape. The included
MuJoCo environments use one-dimensional action vectors bounded by
-1.0..=1.0; their vector length depends on the selected environment.
Observations keep their native kind and dtype in inputs and replay.
Models must perform any required conversion explicitly.
The dtype setting controls actions, rewards, and Q values, including F64 where supported.
Actions and Q values must share the configured dtype; mismatches return an error.
Replay rewards are converted to that dtype.
Model parameters must use the optimization device.
Replay stores detached transitions and uses terminal observations for truncation targets.
Important defaults and constraints are:
| Setting | Default | Constraint or effect |
|---|---|---|
actor_learning_rate | Required | Finite and positive |
gamma | 0.99 | Finite and between zero and one |
tau | 0.005 | Target-network update coefficient from zero to one |
exploration_noise | 0.1 | Non-negative Gaussian standard deviation |
replay_capacity | 1_000_000 | At least batch_size; at first learn, larger than and divisible by env.num_envs() |
batch_size | 256 | Nonzero |
update_frequency | 1 | Nonzero transition interval |
training_start | 1_000 | Random-action transitions before optimization |
training_horizon | First learn call’s num_timesteps | Nonzero global transition horizon |
The example overrides training_start to collect 10,000 random transitions.
Those transitions remain in replay. After warm-up, the agent adds exploration
noise to online-actor actions and updates the actor, critic, and target
networks at every selected replay update.
Evaluate Without Exploration Noise
Agent::act includes Gaussian exploration noise. Use act_deterministic for
evaluation:
let actions = agent.act_deterministic(&observations)?;
The observations must be shaped [batch, ...observation_shape]. The returned
actions are shaped [batch, ...action_shape] and are clamped to the configured
BoxSpace.
Read Understand a Deterministic Actor-Critic Training Run to add a logger and interpret replay-update metrics. Read TD3 to add twin critics, target-policy smoothing, and delayed actor updates.
TD3
Twin Delayed Deep Deterministic Policy Gradient (TD3) keeps DDPG’s deterministic actor and replay loop, then adds three safeguards:
- Two critics reduce optimistic target estimates.
- Noise on target actions smooths the critic target.
- Delayed actor and target-network updates give the critics more update steps.
This page starts from the network and replay contracts in Deterministic Actor-Critic Training and focuses on the TD3-specific choices.
Run the TD3 Example
The example supports Ant, HalfCheetah, Hopper, and Walker2d:
cargo run --release -p examples --example td3_mujoco --features ant
cargo run --release -p examples --example td3_mujoco --features half-cheetah
cargo run --release -p examples --example td3_mujoco --features hopper
cargo run --release -p examples --example td3_mujoco --features walker2d
Normally enable one environment feature; add rendering to it to open a
viewer. Additive builds with several environment features select ant,
half-cheetah, hopper, then walker2d. The program uses the CPU by default. Enable cuda or metal to select an accelerator. It trains for one million collected
transitions, then displays terminal graphs for optimization and episode
metrics.
The complete source is crates/examples/examples/td3_mujoco.rs. Its actor
construction is the same as the DDPG example. The difference is the critic
ensemble and the TD3 builder settings.
Build Two Independent Critics
Canonical TD3 uses two critics. Each critic owns its network and optimizer. Target networks are created automatically:
use modurl::agents::action_value_critic::ScalarStateActionCritic;
let mut critics = Vec::new();
for _ in 0..2 {
let online_critic = MLP::builder()
.input_size(observation_size + action_size)
.output_size(1)
.hidden_layer_sizes(vec![64, 64])
.activation(Relu)
.options((&device, DTYPE))
.build()?;
critics.push(ActionValueCritic::builder()
.online_network(ScalarStateActionCritic::new(online_critic))
.optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
.learning_rate(1e-3)
.build()?);
}
Do not share model parameters or an optimizer between the critics. Their independent errors make the smaller target estimate useful.
Configure the Three TD3 Safeguards
Pass both critics and the TD3-specific update settings:
let mut agent = TD3Agent::builder()
.online_actor(online_actor)
.actor_optimizer(AdamWConfig::new().with_weight_decay(0.0).init())
.actor_learning_rate(1e-3)
.critics(critics)
.action_space(action_space)
.observation_space(observation_space)
.gamma(0.99)
.tau(0.005)
.exploration_noise(0.1)
.target_policy_noise(0.2)
.target_noise_clip(0.5)
.actor_update_interval(2)
.replay_capacity(1_000_000)
.batch_size(256)
.training_start(10_000)
.training_horizon(TOTAL_TIMESTEPS)
.logger(&mut grapher)
.build()?;
agent.learn(&mut env, TOTAL_TIMESTEPS)?;
target_policy_noise is the standard deviation of Gaussian noise added to the
target actor’s output. target_noise_clip limits that noise component by
component. The BoxSpace then clamps the noisy target action to the environment
bounds.
By default, target_aggregation_mode is
ActionValueCriticAggregationMode::Min. The smaller target Q estimate becomes the
bootstrap value shared by both critic losses.
actor_update_interval defaults to 2. The agent updates every critic on each
replay optimization, but updates the actor and all target networks only on
every second optimization. On skipped actor updates, logger fields such as
actor_loss are None.
Important defaults and constraints are:
| Setting | Default | Constraint or effect |
|---|---|---|
gamma | 0.99 | Finite and between zero and one |
tau | 0.005 | Target-network update coefficient from zero to one |
exploration_noise | 0.1 | Non-negative collection-noise deviation |
target_policy_noise | 0.2 | Non-negative target-noise deviation |
target_noise_clip | 0.5 | Non-negative target-noise bound |
actor_update_interval | 2 | Nonzero replay-update interval |
replay_capacity | 1_000_000 | At least batch_size; at first learn, larger than and divisible by env.num_envs() |
batch_size | 256 | Nonzero |
update_frequency | 1 | Nonzero transition interval |
training_start | 1_000 | Random-action transitions before optimization |
training_horizon | First learn call’s num_timesteps | Nonzero global transition horizon |
Collection exploration noise and target-policy noise solve different problems.
exploration_noise changes actions sent to the environment after warm-up.
target_policy_noise changes only next actions used to calculate replay
targets.
Keep Canonical Actor and Target Aggregation
When actor_aggregation_mode is omitted, the actor maximizes the first online
critic’s Q estimate. This is canonical TD3 behavior.
ModuRL supports nonempty ensembles of other sizes and explicit aggregation for experiments:
.target_aggregation_mode(ActionValueCriticAggregationMode::Mean)
.actor_aggregation_mode(ActionValueCriticAggregationMode::Median)
Min, Mean, Median, and Max use the same elementwise aggregation
implemented for SAC critic ensembles. Setting actor_aggregation_mode makes
the actor optimize the selected aggregate instead of the first critic.
These choices define algorithm variants. Omit them when reproducing canonical TD3 with two critics.
Evaluate the Deterministic Actor
As with DDPG, Agent::act adds collection exploration noise. Evaluate the
online actor with:
let actions = agent.act_deterministic(&observations)?;
Read Understand a Deterministic Actor-Critic Training Run to distinguish critic updates from delayed actor updates in logs.
Understand a Deterministic Actor-Critic Training Run
The DDPG and TD3 examples confirm that their actors, critics, target networks,
and replay buffers train together. Add a DDPGLogger or TD3Logger when you
need to compare runs or diagnose one of those components.
Each logger receives two streams. log receives one entry per replay
optimization. log_collection receives rewards and completed episodes from
one vectorized environment step. An update describes a sampled replay batch;
collection describes the policy’s newest behavior.
Add a Logger
Both logger traits use the shared deterministic actor-critic entry types. One logger can therefore support both algorithms:
use modurl::prelude::*;
pub struct ConsoleLogger;
fn log_update(entry: &DeterministicActorCriticLogEntry) {
if !entry.update_index.is_multiple_of(1_000) {
return;
}
let critic_loss = entry.critic_losses[0]
.clone()
.cast(burn::tensor::DType::F32)
.into_scalar::<f32>();
let actor_loss = entry.actor_loss.as_ref().map(|loss| {
loss.clone()
.cast(burn::tensor::DType::F32)
.into_scalar::<f32>()
});
println!(
"step={} update={} critic_loss={critic_loss:.4} \
actor_updated={} actor_loss={actor_loss:?}",
entry.collection_timestep, entry.update_index, entry.actor_updated,
);
}
fn log_collection<I>(entry: &DeterministicActorCriticCollectionLogEntry<I>) {
for episode in &entry.completed_episodes {
println!(
"step={} env={} return={} length={} terminated={} truncated={}",
episode.collection_timestep,
episode.environment_index,
episode.episode_return,
episode.episode_length,
episode.terminated,
episode.truncated,
);
}
}
impl<I> DDPGLogger<I> for ConsoleLogger {
fn log(&mut self, entry: &DeterministicActorCriticLogEntry) {
log_update(entry);
}
fn log_collection(&mut self, entry: &DeterministicActorCriticCollectionLogEntry<I>) {
log_collection(entry);
}
}
impl<I> TD3Logger<I> for ConsoleLogger {
fn log(&mut self, entry: &DeterministicActorCriticLogEntry) {
log_update(entry);
}
fn log_collection(&mut self, entry: &DeterministicActorCriticCollectionLogEntry<I>) {
log_collection(entry);
}
}
Pass a mutable reference while building either agent:
let mut logger = ConsoleLogger;
let mut agent = TD3Agent::builder()
// Keep the remaining TD3 configuration unchanged.
.logger(&mut logger)
.build()?;
The agent borrows the logger. Drop the agent before reading or displaying values held by the concrete logger, as the terminal graph examples do.
Read Replay-Update Metrics
DeterministicActorCriticLogEntry exposes these values:
| Field | Meaning and shape |
|---|---|
critic_losses | One scalar mean-squared Bellman loss per critic |
critic_q_values | One [batch_size, 1] replay-action Q tensor per critic |
actor_loss | Scalar negative mean policy Q, or None on a delayed update |
policy_q_values | [batch_size, 1] actor-objective Q values, or None |
policy_actions | [batch_size, ...action_shape], or None |
replay_actions | Sampled replay actions [batch_size, ...action_shape] |
bellman_targets | Detached target Q values [batch_size, 1] |
replay_rewards | Sampled rewards [batch_size, 1] |
actor_learning_rate | Current actor optimizer learning rate |
critic_learning_rates | Current learning rate for each critic optimizer |
exploration_noise_standard_deviation | Collection-noise setting |
actor_updated | Whether this update changed actor and target networks |
update_index | Zero-based replay-update index |
collection_timestep | Global transition count that triggered the update |
DDPG sets actor_updated on every replay update. TD3 sets it according to
actor_update_interval. When it is false, actor_loss, policy_q_values, and
policy_actions are all None; critic metrics remain present.
Compare critic_q_values with bellman_targets when a critic loss changes
unexpectedly. Compare policy_actions with replay_actions to distinguish
the current actor from behavior stored earlier in replay.
In canonical TD3, policy_q_values come from the first online critic. If
actor_aggregation_mode is configured, they contain the aggregate used by the
actor objective.
Read Collection Metrics
DeterministicActorCriticCollectionLogEntry describes the newest vectorized
environment step:
| Field | Meaning |
|---|---|
collection_rewards | One newest reward per environment |
infos | Typed metadata returned by each environment |
collection_timestep | Global transition count after this vectorized step |
completed_episodes | Episodes that terminated or truncated on this step |
replay_len | Number of transitions currently retained in replay |
Each completed episode records its environment index, return, length, ending condition, and global collection timestep. Partial episodes carry across vectorized steps until the environment terminates or truncates.
During the initial random-action phase, collection entries arrive but replay
update entries do not. After training_start, update entries occur only at
timesteps selected by update_frequency.
You can now separate current policy behavior from replay optimization and, for
TD3, delayed actor updates from critic-only updates. The repository’s
DeterministicActorCriticGrapher applies the same split to terminal plots in
both MuJoCo examples.
Value-Based Training
Use DQN or DDQN when your environment has a discrete action space and you want the agent to learn an action value for every possible action.
This chapter explains the pieces shared by DQNAgent and DDQNAgent. Read
DQN for a complete CartPole program. Read Double DQN
when you want DDQN’s target calculation instead.
Choose DQN or DDQN
Both agents train an online Q-network and periodically copy its parameters to a target Q-network. They differ only in how they form the next-state training target:
| Agent | Chooses the next action | Evaluates that action |
|---|---|---|
DQNAgent | The target Q-network’s largest value | The same target Q-network |
DDQNAgent | The online Q-network’s largest value | The target Q-network |
Start with DQN when you need the standard DQN update. Choose DDQN when you want to reduce overestimation of action values. In DQN, the target network both chooses the largest next-state value and uses that value in the training target. Taking a maximum tends to favor values that are accidentally too high. DDQN uses the online network to choose the action and the target network to evaluate it, which reduces that optimistic bias. The builder fields and training loop are otherwise the same.
Terms
A Q-network maps one observation to one value per discrete action. Its output width must equal the number of actions. An online Q-network is the network the optimizer updates. A target Q-network is a detached copy of the online model. The agent refreshes that copy at a fixed interval.
Epsilon-greedy exploration chooses a random valid action with probability
epsilon and otherwise chooses the online network’s highest-valued action.
epsilon_schedule controls epsilon over the configured training horizon.
Use a Dueling Q-Network
DuelingMLP separates its final representation into a scalar state value and
one advantage per action. It mean-centers the advantages and combines the two
streams into the same [batch, action_count] output expected from any
Q-network:
#![allow(unused)]
fn main() {
let online_q_network = DuelingMLP::builder()
.input_size(observation_space.shape()[0])
.output_size(2)
.options(&device)
.hidden_layer_sizes(vec![64, 64])
.value_hidden_layer_sizes(vec![64])
.advantage_hidden_layer_sizes(vec![64])
.build()?;
}
hidden_layer_sizes configures the shared trunk. The value and advantage
hidden-layer fields configure the two independent streams after that trunk.
Leave either stream’s list empty when you want its output head to connect
directly to the shared features.
Pass the owned online DuelingMLP to either
DQNAgent or DDQNAgent. Dueling changes the network architecture; DDQN
independently changes the next-state target calculation, so the two techniques
can be used together.
Where to Go Next
Build the complete DQN CartPole program. It shows the owned Q-network, replay configuration, and training call in one place. Then use the small, documented change in Double DQN to change its target calculation.
DQN
This page builds a DQN agent for CartPole. It uses one vectorized CartPole environment, an owned Q-network, an epsilon schedule, and an experience replay buffer. You need Rust, Cargo, and the dependencies from Getting Started.
The program trains for 500,000 environment transitions, so the first complete
run takes longer than the PPO quick-start. Lower training_horizon and the
argument to learn together when you only want to check that the program runs.
The Q-Networks
Create an online Q-network. The agent creates its detached target copy. CartPole has four observation values and two discrete actions, so the network reads the environment’s observation shape and produces two Q-values.
DQNAgent needs a Discrete action space. CartPoleV1 supplies that concrete type through env.action_space().
Complete Program
Enable Burn 0.22.0 features std, optim, flex, and autodiff in the application.
use burn::{optim::AdamWConfig, tensor::{DType, Device}};
use modurl::prelude::*;
use modurl_gym::classic_control::cartpole::CartPoleV1;
fn main() {
let device = Device::flex().autodiff();
let environment_device = Device::flex();
let envs = vec![CartPoleV1::builder().seed(42).build().unwrap()];
let env = VectorizedGymWrapper::new(envs).unwrap();
let mut env = DeviceMultiGymWrapper::new(env, environment_device, device.clone());
let observation_space = env.observation_space();
let online_q_network = MLP::builder()
.input_size(observation_space.shape()[0])
.output_size(env.action_space().get_possible_values())
.options((&device, DType::F32))
.hidden_layer_sizes(vec![64, 64])
.build()
.expect("failed to build the online Q-network");
let optimizer = AdamWConfig::new().init();
let mut agent = DQNAgent::builder()
.dtype(DType::F32)
.action_space(env.action_space())
.observation_space(observation_space)
.online_q_network(online_q_network)
.optimizer(optimizer)
.learning_rate(2.5e-4)
.replay_capacity(10_000)
.batch_size(128)
.training_start(10_000)
.update_frequency(10)
.target_update_interval(500)
.training_horizon(500_000)
.epsilon_schedule(|progress: f64| {
let exploration_progress = (progress / 0.5).min(1.0);
1.0 + (0.05 - 1.0) * exploration_progress
})
.build()
.expect("DQN configuration should be valid");
agent.learn(&mut env, 500_000).expect("DQN learning failed");
println!("Training complete.");
}
The Burn optimizer updates the owned online model. The target model has no gradient graph. The agent copies the online model at construction and every 500 transitions.
The epsilon schedule decreases exploration from 1.0 to 0.05 during the
first half of the 500,000-transition horizon. The agent collects 10,000
transitions before its first update, then samples 128 replay entries every 10
transitions.
Run the program with:
cargo run
When training finishes, it prints Training complete.. Read Value-Based
Training for the DQN and DDQN distinction, or Double
DQN to use the alternative target calculation. To record and
interpret training metrics, read Understand a Q-Learning Training
Run.
Graph a Training Run
Run the repository example when you want terminal graphs instead of the minimal program above:
cargo run --example dqn_cartpole
The example prints each completed CartPole episode’s collection step, length, and return during training. After its 500,000-transition run, it plots DQN loss, exploration epsilon, mean selected Q-values, episode returns, and episode lengths. The update metrics come from replay batches; the episode graphs come from the current collection stream. Read Understand a Q-Learning Training Run for the distinction.
Double DQN
Double DQN, or DDQN, uses the same setup, builder fields, and training loop as DQN. Its difference is the next-observation target: the online Q-network chooses the next action, and the target Q-network evaluates that selected action. This separation avoids using the same values for both decisions.
Start with the complete DQN CartPole program. Keep its environment,
owned online Q-network, Burn optimizer, replay configuration, epsilon schedule,
and learn call unchanged. Then replace the agent construction with this
version:
let mut agent = DDQNAgent::builder()
.action_space(Discrete::new(2).unwrap())
.observation_space(observation_space)
.online_q_network(online_q_network)
.optimizer(optimizer)
.learning_rate(2.5e-4)
.replay_capacity(10_000)
.batch_size(128)
.training_start(10_000)
.update_frequency(10)
.target_update_interval(500)
.training_horizon(500_000)
.epsilon_schedule(|progress: f64| {
let exploration_progress = (progress / 0.5).min(1.0);
1.0 + (0.05 - 1.0) * exploration_progress
})
.build()
.expect("DDQN configuration should be valid");
agent.learn(&mut env, 500_000).expect("DDQN learning failed");
DDQNAgent has the same configuration validation as DQNAgent: the replay
capacity must be at least the batch size; the replay capacity, batch size,
update frequency, target-update interval, and training horizon must be nonzero;
and gamma and epsilon values must stay in the inclusive 0.0..=1.0 range.
This is a change in the agent’s learning target, not a change in the Q-network shape or device setup. Use Value-Based Training for the shared configuration rules. To record and interpret training metrics, read Understand a Q-Learning Training Run.
Understand a Q-Learning Training Run
The DQN and DDQN programs confirm that training completed, but they do not record metrics. Add a logger when you need to compare runs or diagnose a configuration. Each logger receives an update entry when the agent optimizes from replay and a collection entry after every vectorized environment step.
The repository’s dqn_cartpole example uses both entry types. It
prints completed CartPole episodes during training and draws loss, exploration,
Q-value, return, and length graphs when training ends. Run it with:
cargo run --example dqn_cartpole
Implement the logger trait that matches the agent. Both traits receive the same entry types:
struct TrainingLogger;
impl DQNLogger for TrainingLogger {
fn log(&mut self, entry: &QLogEntry) {
println!("update {}: loss = {:?}", entry.update_index, entry.loss);
}
fn log_collection(&mut self, entry: &QCollectionLogEntry) {
for episode in &entry.completed_episodes {
println!(
"environment {}: return = {}, length = {}",
episode.environment_index,
episode.episode_return,
episode.episode_length,
);
}
}
}
let mut logger = TrainingLogger;
let mut agent = DQNAgent::builder()
// The remaining DQN configuration is unchanged.
.logger(&mut logger)
.build()
.expect("DQN configuration should be valid");
For DDQN, implement DDQNLogger and pass the logger to DDQNAgent::builder()
in the same way. log_collection has a default no-op implementation, so a
logger can ignore collection metrics when it only needs update metrics. The
agent borrows the logger, so keep it alive for as long as the agent is in use.
Read Update Metrics
QLogEntry describes one optimization update from a sampled replay batch. The
agent does not emit it for every environment transition. It starts after replay
warm-up and then follows the update frequency. The entry reaches the logger
before the optimizer applies that update.
The logger exposes these values through QLogEntry:
loss: the mean-squared error between the online network’s value for each sampled action and its Q-learning targetepsilon: the exploration probability used while collecting experiencelearning_rate: the optimizer’s current learning rateq_values: the online network’s value for the selected action in each sampled replay transitionreplay_rewards: the one-step rewards in that sampled replay batchupdate_index: the zero-based index of this optimization updatecollection_timestep: the total number of environment transitions collected when the agent formed this update
q_values and replay_rewards are tensors for a replay batch, not single
summary numbers. Aggregate them, such as with a mean, before graphing or
comparing runs. A replay batch can contain transitions from old episodes and
from earlier versions of the policy.
Read Collection Metrics
QCollectionLogEntry describes the newest environment interaction, not a
sample from replay. The agent emits it after every call to MultiGym::step,
including during replay warm-up.
collection_rewards: one reward for each inner environment from the latest vectorized stepepsilon: the exploration probability that selected the actions for that stepcollection_timestep: the total number of transitions collected after that stepcompleted_episodes: episodes that ended in this vectorized step
completed_episodes is empty when every inner environment continues its current
episode. It contains one QEpisodeLogEntry for each inner environment that
ended in the latest vectorized step, so one collection entry can report several
completed episodes.
Each QEpisodeLogEntry contains the following summary for one finished episode:
environment_index: which inner environment produced the episodeepisode_return: the sum of every reward collected since that environment’s last resetepisode_length: the number of environment steps collected since that resetterminatedandtruncated: the environment’s ending flagscollection_timestep: the total number of collected transitions when that environment finished
The agent resets its running return and length for that environment after recording the entry. An entry never represents an in-progress episode, and it never combines rewards from several episodes. Episode returns and lengths are current collection metrics, so they are not mixed with older replay data.
Read the Metrics Together
Loss has no universal target. Compare it with earlier runs that use the same environment, reward scale, model shape, and replay settings. A noisy loss is normal because every update samples a different replay batch. A persistently growing loss or non-finite values is a signal to inspect the configuration.
The Q-values are estimates, not a direct success score. Their magnitude depends
on the reward scale, gamma, and the remaining rewards the agent expects. Use
them to spot abrupt changes or divergence, and use episode returns and lengths
to decide whether the policy is improving.
Epsilon should follow the schedule you configured. In the DQN example it falls
from 1.0 to 0.05; if it remains high, the agent continues to choose random
actions often. If it falls too quickly, the collection stream may contain too
little exploration when the agent begins learning.
Change One Setting at a Time
Keep the environment, model shape, replay capacity, and epsilon schedule fixed while comparing a change to the learning rate, target-update interval, or update frequency. That makes a change in loss or Q-values easier to attribute to one configuration decision.
Return to DQN or Double DQN to change the training program.
Environments
An environment defines the interaction loop an agent learns from: reset to an
initial observation, apply an action, and return the next observation and
reward. ModuRL represents one environment with Gym and a batch of environments
with MultiGym.
The getting-started example uses CartPoleV1 for individual environments and
VectorizedGymWrapper to train from several of them at once.
modurl_gym includes these Gymnasium-compatible environments:
| Module | Environment | Action space | Observation shape |
|---|---|---|---|
classic_control::acrobot | AcrobotV1 | Discrete, 3 actions | [6] |
classic_control::cartpole | CartPoleV1 | Discrete, 2 actions | [4] |
classic_control::mountain_car | MountainCarV0 | Discrete, 3 actions | [2] |
classic_control::pendulum | PendulumV1 | Continuous [1] | [3] |
box_2d::bipedal_walker | BipedalWalkerV3 | Continuous [4] | [24] |
box_2d::lunar_lander | LunarLanderV3 | Discrete, 4 actions | [8] |
Discrete actions have shape [1] and kind Int;
continuous actions retain the component shapes above. Vectorized batches add one leading axis.
Reset and step results expose observation. Observations and space bounds stay on CPU.
Each environment owns a host RNG. Use the builder’s .seed(u64) option for
reproducible behavior; omitted seeds use OS entropy and are independent of
Device::seed.
BipedalWalkerV3 implements the standard environment with uneven grass terrain;
the hardcore obstacle variant is not included.
These structs expose Gymnasium’s unwrapped dynamics. Registry time limits are
applied explicitly with TimeLimitGym: use 500 steps for AcrobotV1, 200 for
PendulumV1, and the registry horizon appropriate to the other environment.
Keeping the limit in a wrapper makes truncation visible and composable.
modurl_mujoco provides the Gymnasium v5 AntV5, HalfCheetahV5, HopperV5,
HumanoidV5, and Walker2dV5 environments. See that crate’s README for model,
installation, metadata, and parity details.
Read Use Vectorized Environments before writing manual training or evaluation loops. Read Build a Custom Gym Environment when you need a new environment type.
Environment Wrappers
Wrappers change an environment’s observations, rewards, metadata, or episode
boundaries while preserving the Gym interface. When wrappers are nested, the
innermost wrapper processes a transition first.
Core Wrappers
These wrappers are available from modurl::wrappers and work with any
compatible Gym.
| Wrapper | What it does |
|---|---|
TimeLimitGym | Sets truncated after a fixed number of steps. |
RecordEpisodeStatisticsGym | Adds the completed episode’s return and length to EpisodeStatisticsInfo. |
RecordRawRewardGym | Copies each reward into RawRewardInfo before outer wrappers can change it. |
NormalizeObservationGym | Normalizes each observation component with a running mean and variance, with optional clipping. |
NormalizeRewardGym | Scales rewards by the running standard deviation of discounted rewards, with optional clipping. |
FrameStackGym | Stacks recent observations along a new leading dimension. |
MaxAndSkipGym | Repeats an action, sums its rewards, and max-pools the final two observations. |
ClipRewardGym | Maps each reward to -1, 0, or 1 according to its sign. |
Wrapper order determines which values a wrapper sees. For example, placing
RecordEpisodeStatisticsGym inside ClipRewardGym records the underlying return
while the agent receives clipped rewards.
FrameStackGym<G, BF, S> uses native Burn tensors. BF is the stacked batch
rank. Two PrevRank mappings determine the inner observation’s single rank. The defaults
stack vectors [features] into [stack_size, features]. Scalar observations use
FrameStackGym::<_, 3>::new(...) and produce [stack_size, 1].
The supplied space S must use the inner observation kind. For BoxSpace,
bounds include a size-one batch axis. Invalid rank relationships fail during compilation.
InputMapMultiGymWrapper and OutputMapMultiGymWrapper callbacks return
Result with a caller-defined error type. Callback errors use
TensorMapMultiGymError::Mapping; environment errors use Gym. Callbacks
must preserve the documented tensor layouts and kinds. Observation normalization
reports host-read failures through NormalizeObservationGymError::TensorRead.
Output callbacks use the observation space’s Batch type for batched observations
and Item type for terminal observations. These types follow the underlying space.
Atari Wrappers
These wrappers are available from modurl_ale::wrappers and implement the
standard Atari preprocessing steps.
| Wrapper | What it does |
|---|---|
NoopResetGym | Takes a random number of action 0 no-ops after reset; the default range is 1 through 30. |
EpisodicLifeGym | Reports a lost life as done while continuing the same game on reset. |
FireResetGym | Takes actions 1 and 2 after reset to start games that require FIRE. |
WarpGym | Converts Atari observations to grayscale and resizes them to 84 by 84 pixels. |
EpisodicLifeGym requires metadata that implements AtariLives. AtariInfo
and EpisodeStatisticsInfo<I> provide that implementation when their inner
metadata exposes Atari lives.
Batching adapters such as VectorizedGymWrapper and StackedMultiGym are
covered in Use Vectorized Environments.
Use Vectorized Environments
MultiGym steps several environments with one batch of actions. PPO uses
this interface so one rollout step can collect several transitions.
VectorizedGymWrapper turns a Vec<G> of ordinary Gym values into a
vectorized environment:
let envs = (0..4)
.map(|index| CartPoleV1::builder().seed(42 + index).build().unwrap())
.collect::<Vec<_>>();
let mut env = DeviceMultiGymWrapper::new(
VectorizedGymWrapper::new(envs)?, Device::flex(), device.clone(),
);
With the multithreading feature enabled,
MultithreadedVectorizedGymWrapper runs each inner Gym on a persistent
worker thread. Pass constructors so every inner environment is created on the
thread that owns it, together with representative observation and action
spaces:
let constructors = (0..4)
.map(|index| move || CartPoleV1::builder().seed(42 + index).build().unwrap())
.collect();
let env = MultithreadedVectorizedGymWrapper::new(
constructors,
observation_space,
action_space,
)?;
let mut env = DeviceMultiGymWrapper::new(env, Device::flex(), device.clone());
Each inner environment remains one unit of work. The wrapper preserves the
batching and auto-reset behavior of VectorizedGymWrapper. If an inner
environment returns an error, reset the complete batch before stepping it
again.
Reset Once, Then Step
Before a manual loop, call reset once to receive one initial observation per
inner environment. Pass that batch to Agent::act, then pass the returned batch
of actions to MultiGym::step.
Scalar observations and actions have shape [1] in a single Gym and
[num_envs, 1] in a MultiGym. Batching always adds one leading axis.
Rewards retain shape [num_envs, 1]; terminal observations retain their unbatched shape.
let mut observations = env.reset()?;
loop {
let actions = agent.act(&observations, &mut ())?;
let step = env.step(actions)?;
observations = step.observations;
}
observations has one next observation for every inner environment, so it is ready
for the next call to act.
Understand Auto-Reset
When an inner environment returns done or truncated, ModuRL resets that one
environment immediately. The observations field then contains the first observation
of its next episode. This lets the next batched step continue without a special
reset branch.
The terminal observation is still available. terminal_observations contains an
entry for each inner environment: Some(observation) when that environment ended and
None when it continued.
If code needs the observation before reset for each transition, call
transition_next_observations:
let step = env.step(actions)?;
let transition_observations = step.transition_next_observations();
let observations_for_the_loop = step.observations;
transition_next_observations uses a terminal observation where one exists and the normal
next observation otherwise. The second value, step.observations, remains the right input
for the following action-selection step.
PPOAgent::learn handles this distinction while it collects experience. You
only need it when you write a loop that consumes transitions yourself.
Use One Shared World for Several Players
A custom MultiGym can use batch rows for coupled players instead of
independent simulations. The environment must consume every player’s action
before it advances the shared world and must keep termination and reset
behavior consistent across those rows.
For example, a custom two-player game can expose players as the leading tensor dimension:
let mut env = CoupledGame::new()?;
let observations = env.reset()?; // [players, ...observation_shape]
let actions = agent.act(&observations, &mut ())?; // [players, ...action_shape]
let step = env.step(actions)?; // advances the shared game once
For a shared-policy agent, acting on the whole observation batch applies the same policy to every player and provides self-play without creating duplicate physics simulations. A coupled implementation should end and reset every player row together whenever the shared episode ends.
Stack Several Batched Environments
StackedMultiGym combines several homogeneous MultiGym values into one flat
batch. For example, four two-player games become eight batch rows:
let games = (0..4)
.map(|seed| {
let mut game = CoupledGame::new()?;
game.seed(seed);
Ok(game)
})
.collect::<Result<Vec<_>, GameError>>()?;
let mut env = StackedMultiGym::new(games)?;
let observations = env.reset()?; // [8, ...observation_shape]
let actions = agent.act(&observations, &mut ())?; // [8, ...action_shape]
let step = env.step(actions)?; // steps each shared game once
Rows are ordered first by inner gym and then by that gym’s own row order.
group_offsets() maps the flattened rows back to their inner gyms. All inner
gyms must expose the same observation and action shapes, and each inner gym
keeps responsibility for its own auto-reset behavior. The stack checks action
batch size before stepping and validates each group’s step and reset observation
shapes, reward shape, and metadata lengths before concatenating. Dtypes and
devices must still match the spaces and tensors used by each inner gym.
With the multithreading feature enabled,
MultithreadedStackedMultiGym runs each inner MultiGym on a persistent
worker thread. Pass constructors so every inner gym is created on the thread
that owns it, together with representative observation and action spaces:
let constructors = (0..4)
.map(|seed| move || {
let mut game = CoupledGame::new().unwrap();
game.seed(seed);
game
})
.collect();
let mut env = MultithreadedStackedMultiGym::new(
constructors,
observation_space,
action_space,
)?;
The whole inner gym remains one unit of work, so coupled player rows are never split across threads. If an inner gym returns an error, reset the complete stack before stepping it again.
Next, read Build a Custom Gym Environment to provide your own single-environment implementation.
Build a Custom Gym Environment
Implement Gym for one environment. Then place one or more instances in a
VectorizedGymWrapper when an agent needs batched interaction.
This small environment has one floating-point observation. Action 0 moves its
state left and action 1 moves it right. An episode ends when the state reaches
either bound.
Define the Environment
The following code belongs in src/counter_env.rs:
use burn::tensor::{Device, Int, Tensor, TensorReadError};
use modurl::prelude::*;
#[derive(Debug, thiserror::Error)]
pub enum CounterEnvError {
#[error("action {0} is outside the action space")]
InvalidAction(i32),
#[error("reading the action failed: {0}")]
Read(#[from] TensorReadError),
}
pub struct CounterEnv {
state: i32,
device: Device,
}
impl CounterEnv {
pub fn new(device: Device) -> Self {
Self { state: 0, device }
}
/// Returns the position as an unbatched observation `[1]` with one feature.
fn observation(&self) -> Tensor<1> {
Tensor::from_data([self.state as f32], &self.device)
}
}
impl Gym for CounterEnv {
type Error = CounterEnvError;
type ObservationSpace = BoxSpace<2>;
type ActionSpace = Discrete;
/// Returns the initial position as unbatched `[1]` on the environment device.
fn reset(&mut self) -> Result<ResetInfo, Self::Error> {
self.state = 0;
Ok(ResetInfo {
observation: self.observation(),
info: (),
})
}
/// Consumes one scalar integer action `[1]` and returns an unbatched position `[1]`.
/// Actions must be readable on the host; observations use the environment device.
fn step(&mut self, action: Tensor<1, Int>) -> Result<StepInfo, Self::Error> {
match action.try_into_scalar::<i32>()? {
0 => self.state -= 1,
1 => self.state += 1,
action => return Err(CounterEnvError::InvalidAction(action)),
}
let done = self.state.abs() >= 4;
Ok(StepInfo {
observation: self.observation(),
reward: 1.0,
done,
truncated: false,
info: (),
})
}
fn observation_space(&self) -> Self::ObservationSpace {
BoxSpace::new_with_universal_bounds(
[1, 1],
-4.0,
4.0,
&self.device,
)
}
fn action_space(&self) -> Self::ActionSpace {
Discrete::new(2).unwrap()
}
}
Save and Restore State
Implement PerfectStateGym when the environment can save and restore all its
mutable state. For this counter, that state is the current position:
impl PerfectStateGym for CounterEnv {
type State = i32;
fn save_state(&self) -> Result<Self::State, Self::Error> {
Ok(self.state)
}
fn set_state(&mut self, state: &Self::State) -> Result<(), Self::Error> {
self.state = *state;
Ok(())
}
}
Implement LegalActionsGym to expose the currently legal action indices:
impl LegalActionsGym for CounterEnv {
fn legal_actions(&self) -> Result<Vec<usize>, Self::Error> {
Ok(if self.state.abs() >= 4 { vec![] } else { vec![0, 1] })
}
}
Include the random-number generator state too when an environment uses randomness.
Wrapping PerfectStateGym environments in VectorizedGymWrapper provides
PerfectStateMultiGym, which saves and restores individual slots by index.
reset returns the initial observation and step consumes one action and
returns the observation that follows it, its reward, and the episode flags.
The default Gym information type is (), so ordinary environments use
ResetInfo and StepInfo without an explicit type parameter. Environments
with additional typed metadata can instead implement Gym<MyInfo>.
The full signature is Gym<I, O, A>. Observation and action batch ranks
O and A default to 2. Both spaces inherit TensorBatchTypes, which declares
Kind, Item, and Batch. Item omits the batch axis; Batch adds one leading
batch axis with the same tensor kind. BoxSpace<2> supplies Tensor<1> items
and Tensor<2> batches. Discrete supplies Tensor<1, Int> items and
Tensor<2, Int> batches. Gym, MultiGym, and Agent use these shared types.
Custom spaces implement TensorBatchTypes once before implementing the space
methods. Access an item type as <S as TensorBatchTypes>::Item.
Single-environment tensors omit the batch axis. Scalar values use
[1] individually and [batch_size, 1] in batches. Space operations use
batched tensors. The associated spaces determine the native tensor kinds:
BoxSpace<2> uses Float, and Discrete uses Int.
For explicit observation types, use StepInfo<MyInfo, Tensor<1, Int>> and
ResetInfo<MyInfo, Tensor<1, Int>>. Both store native unbatched tensors.
In src/main.rs, declare the module and bring the environment into scope:
mod counter_env;
use counter_env::CounterEnv;
The observation and action spaces are part of the contract. The observation space must match
the tensors returned by reset and step. The action space must accept the
actions that step understands.
Vectorize the Environment
Build several instances, then wrap them as a batch:
let envs = (0..4)
.map(|_| CounterEnv::new(device.clone()))
.collect::<Vec<_>>();
let env = VectorizedGymWrapper::new(envs)?;
VectorizedGymWrapper handles the batched action split and auto-reset behavior.
The individual environment only needs to implement the single-environment
Gym contract.
For four instances, observations have shape [4, 1] and actions have shape
[4, 1]. Construction fails when the environment list is empty.
How-to Guides
These pages solve focused tasks after you understand a CartPole training example. They assume that you already know the surrounding concepts and link back to the tutorial when a complete program is more useful.
Start with Run on CUDA or Metal after the CPU version of the program works.
Run on CUDA or Metal
First run the CPU version of your program. Then enable one backend feature and construct a device for that backend.
CUDA
Enable CUDA on your direct burn dependency in Cargo.toml:
burn = { version = "=0.22.0", default-features = false, features = ["std", "optim", "autodiff", "cuda", "flex"] }
In a program, replace Device::flex() with:
let device = Device::cuda(0).autodiff();
0 selects the first CUDA device. The CUDA runtime and a build with CUDA
support must be available on the machine.
Metal
Enable Metal on your direct burn dependency in Cargo.toml:
burn = { version = "=0.22.0", default-features = false, features = ["std", "optim", "autodiff", "metal", "flex"] }
In a program, replace Device::flex() with:
let device = Device::metal(burn::tensor::DeviceKind::DefaultDevice).autodiff();
The default device selects the system Metal device. Metal builds require a supported Apple platform.
Transfer at the Environment Batch Boundary
CPU environments always return CPU observations. Classic-control and Box2D
environments own host RNGs; set .seed(u64) for reproducibility. Omitted
seeds use OS entropy and are independent of Device::seed. MuJoCo uses its
existing seeded CPU RNG and has no device setting. Wrap vectorized environments
to transfer actions and observations once per batch, and build model parameters
on the agent device.
let env = CartPoleV1::builder().seed(42).build().unwrap();
let mut envs = DeviceMultiGymWrapper::new(
VectorizedGymWrapper::from(env), Device::flex(), device.clone(),
);
let model = MLP::builder().input_size(4).output_size(2)
.hidden_layer_sizes(vec![64, 64]).options((&device, burn::tensor::DType::F32)).build()?;
The selected backend requires its runtime and drivers.
If initialization fails, fix the backend installation or return to Device::flex().
Split Replay Storage From Optimization
Replay-based agents can keep a large replay buffer on the CPU while running models and optimization on an accelerator:
use burn::tensor::Device;
use modurl::prelude::*;
let optimization_device = Device::cuda(0).autodiff();
let storage_device = Device::flex();
let env = CartPoleV1::builder().seed(42).build()?;
let mut envs = DeviceMultiGymWrapper::new(
VectorizedGymWrapper::from(env), storage_device.clone(), optimization_device.clone(),
);
let mut agent = SACAgent::builder()
.policy(policy)
.actor_optimizer(actor_optimizer)
.actor_learning_rate(3e-4)
.critics(critics)
.entropy_configuration(entropy)
.action_space(envs.action_space())
.observation_space(envs.observation_space())
.training_horizon(total_timesteps)
.replay_device(storage_device)
.build()?;
.replay_device(..) selects the replay storage device; optimization runs on the device of the model parameters. SAC stores the native observation kind and dtype from the first batch. It does not
move an environment or model parameters for you.
Build SAC’s actor, critics, target critics, optimizers, and automatic entropy
variable on optimization_device. Use DeviceMultiGymWrapper to connect CPU
environments to that device. For
DDPG or TD3, this includes both the online and target actors as well as every
critic pair. You do not create any of these components on storage_device.
Internally, the agent transfers detached transitions to storage_device when
it adds them to replay. It transfers sampled replay batches back to
optimization_device before each update.
This strategy trades transfer time for accelerator memory. Start with
the default (replay on the model device) and measure the run. Set .replay_device(..) when
replay memory is the limiting resource and the transfer cost is acceptable.