PPO

PPOAgent is a ModuRL agent that implements Proximal Policy Optimization (PPO). It collects transitions from a MultiGym, then updates a stochastic policy and a value model from that experience.

The getting-started program uses the separate-network PPO configuration. It gives the policy model and the value model their own MLP, VarMap, and Adam optimizer. PPONetworkInfo::Shared is also available when a configuration needs one shared model followed by separate policy and value heads.

Start with Getting Started for the complete CartPole program. Then read Understand a PPO Training Run before changing its configuration.

The continuous-control example supports ant, half-cheetah, hopper, and walker2d. Normally enable one environment feature; rendering may be added to that feature to open a viewer. Cargo features are additive, so builds that enable several environment features use the priority ant, half-cheetah, hopper, then walker2d. PPO writes TensorBoard events below runs/ppo_mujoco/ and also displays terminal graphs.

PPO can use any compatible Distribution implementation. ModuRL currently supplies categorical and Gaussian distributions, and applications can define their own. Models, Policies, and Distributions explains the extension point, the built-in tensor layouts, and how sampled representations become environment actions.

For value-based agents in discrete action spaces, read Value-Based Training.