Use Vectorized Environments
MultiGym steps several environments with one batch of actions. PPO uses
this interface so one rollout step can collect several transitions.
VectorizedGymWrapper turns a Vec<G> of ordinary Gym values into a
vectorized environment:
let envs = (0..4)
.map(|index| CartPoleV1::builder().seed(42 + index).build().unwrap())
.collect::<Vec<_>>();
let mut env = DeviceMultiGymWrapper::new(
VectorizedGymWrapper::new(envs)?, Device::flex(), device.clone(),
);
With the multithreading feature enabled,
MultithreadedVectorizedGymWrapper runs each inner Gym on a persistent
worker thread. Pass constructors so every inner environment is created on the
thread that owns it, together with representative observation and action
spaces:
let constructors = (0..4)
.map(|index| move || CartPoleV1::builder().seed(42 + index).build().unwrap())
.collect();
let env = MultithreadedVectorizedGymWrapper::new(
constructors,
observation_space,
action_space,
)?;
let mut env = DeviceMultiGymWrapper::new(env, Device::flex(), device.clone());
Each inner environment remains one unit of work. The wrapper preserves the
batching and auto-reset behavior of VectorizedGymWrapper. If an inner
environment returns an error, reset the complete batch before stepping it
again.
Reset Once, Then Step
Before a manual loop, call reset once to receive one initial observation per
inner environment. Pass that batch to Agent::act, then pass the returned batch
of actions to MultiGym::step.
Scalar observations and actions have shape [1] in a single Gym and
[num_envs, 1] in a MultiGym. Batching always adds one leading axis.
Rewards retain shape [num_envs, 1]; terminal observations retain their unbatched shape.
let mut observations = env.reset()?;
loop {
let actions = agent.act(&observations, &mut ())?;
let step = env.step(actions)?;
observations = step.observations;
}
observations has one next observation for every inner environment, so it is ready
for the next call to act.
Understand Auto-Reset
When an inner environment returns done or truncated, ModuRL resets that one
environment immediately. The observations field then contains the first observation
of its next episode. This lets the next batched step continue without a special
reset branch.
The terminal observation is still available. terminal_observations contains an
entry for each inner environment: Some(observation) when that environment ended and
None when it continued.
If code needs the observation before reset for each transition, call
transition_next_observations:
let step = env.step(actions)?;
let transition_observations = step.transition_next_observations();
let observations_for_the_loop = step.observations;
transition_next_observations uses a terminal observation where one exists and the normal
next observation otherwise. The second value, step.observations, remains the right input
for the following action-selection step.
PPOAgent::learn handles this distinction while it collects experience. You
only need it when you write a loop that consumes transitions yourself.
Use One Shared World for Several Players
A custom MultiGym can use batch rows for coupled players instead of
independent simulations. The environment must consume every player’s action
before it advances the shared world and must keep termination and reset
behavior consistent across those rows.
For example, a custom two-player game can expose players as the leading tensor dimension:
let mut env = CoupledGame::new()?;
let observations = env.reset()?; // [players, ...observation_shape]
let actions = agent.act(&observations, &mut ())?; // [players, ...action_shape]
let step = env.step(actions)?; // advances the shared game once
For a shared-policy agent, acting on the whole observation batch applies the same policy to every player and provides self-play without creating duplicate physics simulations. A coupled implementation should end and reset every player row together whenever the shared episode ends.
Stack Several Batched Environments
StackedMultiGym combines several homogeneous MultiGym values into one flat
batch. For example, four two-player games become eight batch rows:
let games = (0..4)
.map(|seed| {
let mut game = CoupledGame::new()?;
game.seed(seed);
Ok(game)
})
.collect::<Result<Vec<_>, GameError>>()?;
let mut env = StackedMultiGym::new(games)?;
let observations = env.reset()?; // [8, ...observation_shape]
let actions = agent.act(&observations, &mut ())?; // [8, ...action_shape]
let step = env.step(actions)?; // steps each shared game once
Rows are ordered first by inner gym and then by that gym’s own row order.
group_offsets() maps the flattened rows back to their inner gyms. All inner
gyms must expose the same observation and action shapes, and each inner gym
keeps responsibility for its own auto-reset behavior. The stack checks action
batch size before stepping and validates each group’s step and reset observation
shapes, reward shape, and metadata lengths before concatenating. Dtypes and
devices must still match the spaces and tensors used by each inner gym.
With the multithreading feature enabled,
MultithreadedStackedMultiGym runs each inner MultiGym on a persistent
worker thread. Pass constructors so every inner gym is created on the thread
that owns it, together with representative observation and action spaces:
let constructors = (0..4)
.map(|seed| move || {
let mut game = CoupledGame::new().unwrap();
game.seed(seed);
game
})
.collect();
let mut env = MultithreadedStackedMultiGym::new(
constructors,
observation_space,
action_space,
)?;
The whole inner gym remains one unit of work, so coupled player rows are never split across threads. If an inner gym returns an error, reset the complete stack before stepping it again.
Next, read Build a Custom Gym Environment to provide your own single-environment implementation.