in the next part of today's lecture i'm going to give you a kind of a whirlwind tour through different types of reinforcement learning algorithms we'll talk about each of these types in much more detail in the next few lectures but for now we'll just discuss what these types are so they don't come as a surprise later so the rl algorithms we'll cover will generally be optimizing the rl objective that i defined before policy graded algorithms attempt to directly calculate a derivative of this objective with respect to theta and then perform a grading descent procedure using
that derivative value-based methods estimate value functions or q functions for the optimal policy and then use those value functions or q functions which are typically themselves represented by a function approximately like a neural network to improve the policy oftentimes pure value-based functions don't even represent the policy directly but rather represented implicitly as something like an arg max of a queue function actor critic methods are a kind of hybrid between the two actor critic methods learn a q function or value function and then use it to improve the policy typically by using them to calculate a
better policy gradient and then model based reinforcement learning algorithms will estimate a transition model they'll estimate some model of the transition probabilities t and then they will either use the transition model for planning directly without any explicit policy or use the transition model to improve the policy and there are actually many variants in model based rl for how the transition model can be used all right let's start our conversation with model-based star algorithms so for model-based startup algorithms the green box will typically consist of learning some model for p of s t plus one given
st comma a t so this could be a neural net that takes in st comma 80 and either outputs a probability distribution over t plus one or if it's a deterministic model just attempts to predict st plus one directly and then the blue box has a number of different options so let's focus in on that blue box since model based rl algorithms will differ greatly in terms of how they implement this part so one option for model based rl algorithms is to simply use the learned model directly to plan so you could for example learn
how the rules of a chess game work and then use your favorite discrete planning algorithm like monte carlo tree search to play chess or you can learn the physics of a continuous environment for a robot and then use some optimal control or trajectory optimization procedure through that learned physics model to control the robot another option is to use the learned model to compute derivatives of the reward function with respect to the policy essentially through back propagation this is a very simple idea but it actually requires quite a few tricks to make it work well typically
in order to account for numerical stability so for example second order methods tend to work a lot better than first order methods for back propagating the policy another common use of a model is to use the model to actually learn a separate value function or q function and then use that value function or q function to improve the policy so the value function or q function would be learned using some type of dynamic programming method and it's also fairly common to kind of extend number three to essentially use a model to generate additional data for
a model free reinforcement learning algorithm and that can often work very well all right value function based algorithms so for value function based algorithms the green box involves fitting some estimate of v of s or q of s comma a usually using a neural network to represent e of s or q of s comma a where the network takes in s or s com a as input and outputs a real valued number and then the blue box if it's a pure value based method would simply choose the policy to be the argmax of qsa so
in a pure value based method we wouldn't actually represent the policy explicitly as a neural net we would just represent it implicitly as an arg max over a neural net representing qsa direct policy gradient methods uh would implement the blue box simply by taking a gradient step or grading a sense step on theta using the gradient of the expected value of the reward we'll talk about how this grading can be estimated in the next lecture but the green box for policy gradient algorithms is very very simple it just involves computing the total reward along each
trajectory simply by adding up the rewards that were obtained during the rollout by the way when i use the term rollout that simply means sample of your policy it means run your policy one step at a time and the reason we call it a rollout is because you're unrolling your policy one step at a time actor critic algorithms are a kind of hybrid between value-based methods and policy gradient methods ectocritic algorithms also fit a value function or a q function in the green box just like value-based methods but then in the blue box they actually
take a gradient ascend step on the policy just like policy gradient methods utilizing the value function or q function to obtain a better estimate of the gradient a more accurate gradient
