Proximal Policy Optimization (PPO)

It’s interesting that large language models (LLMs) such as ChatGPT have been revered as quintessential AI models, when they are more dissimilar from human learning than other kinds of algorithms. Using contextual information from word parts, known as tokens, the LLMs act as a huge lookup model. It feels more reminiscent of a google search, but instead of returning web articles, it spits out information directly to the user. When it comes to completing physical goals and tasks, Reinforcement Learning is the preferred method of learning. Modeled after animal behavioral training, reinforcement learning uses rewards to guide the agent through an environment to complete a goal. Learning with a specific objective in mind lends itself well to games, which is why it is implemented in robots or computer players in online games.

Under the big umbrella of reinforcement learning, there are many types of algorithms. We have many because not all problems are the same. Sometimes we have more or less information, time, and memory space for different tasks. For example, the approach you take to solving a maze would be very different if you know where the exit is compared to if you didn’t. One of these algorithms, Proximal Policy Optimization (PPO), has become a leading algorithm in RL since its presentation by OpenAI in 2017. It is actually used to guide ChatGPT’s responses based on human feedback. This article will break down some of the mechanisms that PPO employs to simulate human-like learning at a high level.

First, we will break down the name Proximal Policy Optimization. A policy in RL is a set of rules that make up a behavior. For example, a simple self-driving car policy may look like keeping the car to the right of the centerline and stopping at red lights. The policy is what the agent is learning and improving over time because it does not know what the best behavior is until it interacts with the environment. In our example, the self-driving car does not know “red means stop” and will initially drive through the red light, resulting in a crash. The agent driving the car will receive a penalty that tells it: “Uh oh! That behavior was bad, let me try to do something different next time.”, resulting in a policy update.

The proximal part of the name is the major key to this algorithm’s success. In other algorithms, when the agent decides to make changes to its behavior (the policy), it is unconstrained. Therefore, if a self-driving car runs a red light and receives a penalty, the agent may decide to drive faster and make a left turn at the red light next time. These huge swings in behavior don’t help the agent learn what the right policy is quickly. Instead, PPO tells the agent: “Only make small changes to your behavior at one time.” Under PPO, the agent may decide to go slower during the next iteration, receiving a slightly less harsh penalty, and then learn to stop. It learns to creep toward the right answer rather than randomly guessing under extreme changes. 

Optimization is a word that is so frequently that it can be overlooked, but it is important to revisit. In this context, optimization means to find either the maximum or minimum return based on behavior. The return is aligned with the goal of the agent. If the agent’s goal is to drive safely, the return will be maximized if the car drives the speed limit and stops at red lights. Optimization is often computed using gradient updates. The gradients tell the agent how much to change its policy and in what direction. For example, if the car got -5 for driving 10 mph over the speed limit, but last time received +1 for driving the speed limit, the gradient tells the agent  it needs to reduce its speed variable and by how much. It does this indirectly through the neural network node weights, but at a high level, this is what the gradients control. They are knobs that the agent turns to dial into the correct behavior.

Now that we understand the basics, we can go into the specific enhancements of PPO. We start off with pretty random policies, since the agent has no idea what qualifies as “good” or “bad”  behavior yet. Much like a teenager learning to drive for the first time, the first runs are choppy and dangerous, and it receives rewards and penalties in turn. You can think of these like your dad telling you did well or screaming at you to brake before you hit the mailbox. The agent does this hundreds of times under a behavior pattern and saves this information into a buffer. The buffer is meant to act as a memory system that the agent can draw from as it updates its policy. 

One of the cool things the agent does with these stored experiences in the buffer is Generalized Advantage Estimation (GAE). Formally, GAE is a sum of temporal difference residuals. These are fancy words that basically mean: “I’m going to assign a value based on how much reward I got for taking an action compared to other actions.” For example, if the agent got a big reward for stopping at a red light, GAE says “Stop at red lights more! That is a good action!” This helps the agent learn faster, as it learns what behavior to prioritize.

Coming back to the proximal part of the algorithm, the algorithm prevents the behavior from changing too much between runs through objective function clipping. For example, if the algorithm received the highest reward when stopping at the red light, the agent may decide the best behavior is to stay stationary at all times. Clipping prevents extreme values from overpowering behavior, perhaps settling to stop only when coming to a traffic light instead. 

Finally, all of this information is used to update the policy and the cycle repeats, where the new policy is used to gain experiences and store it in the buffer, for which each experience is assigned a value through GAE, and policy updates are clipped before gradient descent. This pattern continues hundreds and thousands of times until the policy is hardly changing and rarely receiving penalties. This means that the agent has converged to an optimal policy (a.k.a decided on the best behavior to complete a task).  

PPO contains a lot of new features to fix issues in previous algorithms, making it a robust learning algorithm. When considering all the moving parts, isn’t it interesting how much thought goes into…thinking? Things we learn when we are young that now come instantaneously are really hard to break down mathematically, leading to complex algorithms to do tasks that we can do nearly instinctually. Pretty neat!