all right next let's get started discussing some reinforcement learning algorithms there are quite a few reinforcement learning algorithms that we'll cover in this course but at a high level all of these algorithms would share more or less the same high high-level anatomy they will consist of three basic parts the first part which i will always draw in orange is to generate samples so reinforcement learning is about learning through trial and error in some sense and trial the trial part of trial and error means actually attempting to run your policy in your environment actually have it
interact with your markov decision process and collect samples what are samples well samples are trajectories so you will interact with your mtp and typically what you will be doing is you will be sampling trajectories from the trajectory distribution typically the one induced by your policy although when we talk about exploration later sometimes we might actually choose to sample from a slightly different trajectory distribution than the one that our policy would define but for now let's just assume that generating samples means sampling trajectories from the structure distribution defined by your policy which simply means running your
policy in the environment then we will have the green box and the green box corresponds to learning some kind of model this could be literally a model of the dynamics in a model based rl algorithm or it can be some sort of more implicit model such as a value function and this this green box basically corresponds to estimating something about your current policy something about uh how your policy is doing how well it's performing what kind of rewards it's attaining and then we'll have the blue box which is where you actually change your policy to
make it better and then you repeat this process pretty much all of the algorithms that we will cover will have these three parts in some cases one of these parts might be very very simple in some cases some of them might be very complex but all three of them are generally going to be present so here are some simple examples let's say that we're going to run our policy and we'll generate sample trajectories denoted by these black lines and we'll evaluate them to see if they're good or bad so evaluate means just sum up their
rewards summing up their rewards is what happens in the green box and then when we improve the policy we might try to make the good trajectories kind of the green check marks be more likely and the bad trajectories the red ones be less likely so that's the improvement step now what i'm describing here is the basic high level scheme for a policy gradient algorithm which we'll learn about in detail in the next lecture so in a policy grading algorithm the green box the box where we estimate something about our policy is very very simple it
simply consists of summing up the rewards along the trajectories that we sampled and that tells us how good our policy is so the green box is just a summation the blue box might involve calculating the gradient of the reward of your policy and we'll talk about how to do that in the next lecture and applying that gradient to the policy parameters theta so that's a very simple trial and error style reinforcement learning algorithm run your policy get some trajectories measure how good those trajectories are and then modify the policy to make the better trajectories have
a higher probability you could also imagine doing a model based uh rl procedure you can kind of think of this as rl by back prompt so maybe in the green box you learn a model you learn some some other neural network f phi such that s t plus one is approximately equal to f5 stat and you train f5 on data generated in the orange box with supervised learning so maybe you have a whole other neural network that goes from st80 to sd plus one now this green box is much more complex than the one we
had on the previous slide on the previous slide we would just sum over the rewards in our trajectories here we're actually fitting a whole other neural network the summation might take a millisecond this might take minutes to train or maybe even hours if it's using images and then in the blue box we might backprop through f and and r to train the policy pi theta so if we want to calculate the reward of the policy we can basically compose the policy with f actually use our automatic differentiation software to calculate the reward and back prop
through all of them to optimize the policy we're going to cover methods that do some variant of this when we talk about model-based reinforcement learning much later in the course if the details of this don't currently make sense don't worry about it the only point of this slide is to explain to you the different incarnations of the green box and the blue box that we might see in some very different reinforcement learning algorithms all right now which parts in this whole process are expensive and which parts might be cheap well the orange box it's a
cost in terms of time and computation depends a great deal on what kind of problem you're solving if you're collecting samples by running a real-world system like a real robot a real car a real power grid a real chemical plant whatever the orange box can be potentially extremely expensive right because you have to collect data in real time at least until we invent time travel and if you need like thousands of samples each iteration of your rl algorithm this can be very very costly on the other hand if you're collecting samples in the mujoko simulator
that all of you are using for homework one then you know the pedro simulator can run up to 10 000 x real time so the cost of the orange box might actually be trivial so depending on which of these regimes you're in or where you are on the spectrum you might care more or less about how many samples you need in the orange box which will influence your choice of reinforcement learning algorithm so this can range from prohibitively expensive to trivially cheap depending on how you're learning the green box also could range from extremely cheap
to extremely expensive so if you're just estimating the return on your policy by summing up the rewards that you obtained this is very very cheap it's just a summation operator if you are learning an entire model by training a whole other neural net this might be very expensive it might require you know a big supervised learning run in the inner loop of your rl algorithm similarly in the blue box if you're just taking one gradient step uh this might be fairly cheap if you have to back prop through your model and your policy like i
discussed in the model based slide this might be very expensive and there will be algorithms that fall at different points of the spectrum four different boxes for instance a q learning algorithm which we'll cover a couple weeks from now basically spends all of its effort in the green box and the blue box is just an art max
