CS 285: Lecture 2, Imitation Learning. Part 4

all right in part four of the lecture I'll talk about a few algorithmic approaches that can also make behavioral cloning work pretty well the first one I'll discuss is to use multitask learning so this might seem a little paradoxical at first but it turns out that sometimes learning many tasks at the same time can actually make imitation learning easier so let's say that you would like to train your agent let's say your vehicle to drive to a particular location let's call it location P1 and you might have lots of demonstrations of driving to location P1

and then you'll train a policy a given s so that's a pretty straightforward thing to do but as we discussed before if you want to train a policyclist to be robust so that it doesn't suffer too much from compounding errors maybe you would really want to get the expert to make some intentional mistakes put it into some states where it can recover from those mistakes and teach the policy that way well what if you instead don't have such optimal expert data instead you have data at the expert attempt to drop lots of different locations what

you can do is you can actually train a policy that is that receives the desired location as input and the way that you get the desired locations by just looking at the last time step that the human expert landed into and then you'll train a policy for reaching NEP one of the nice things about this is that of course the expert will visit many more different states if they're trying to go to many different locations so by conditioning the policy on the location you can still get a policy for location P1 that you wanted but

you're getting a lot more training data and perhaps more importantly you're getting data from lots of different states that the expert might not have visited if they were just trying to reach P1 and if they were behaving optimally so what you can do is something called goal condition cable coin at training time you might receive a collection of trajectories where you're not even told what the extra is trying to do they're just sequence of states and actions and you assume that whatever the expert was doing was a good example for the state that they actually

reached so you say that well demo one is a good demo for reaching the state S capital T demo 2 is a good State demo for reducing whatever state that reached and then you just feed in the last state as an additional input into the policy and train the policy to take the action that the expert took whenever the expert was trying to reach that state and that gives you access to a lot more training states that provide much better coverage and hopefully give you many more of those instances where you might learn Corrections so

in this case while you still suffer from distributional shift you might still make mistakes and find yourself in states that are unfamiliar for the particular goal you're commanding that state might be more familiar for some other goal and the other really nice thing about this is that you can actually leverage data that is more sub-optimal because maybe the expert failed at reaching the position P1 but they succeeded at reaching some other position and you can still learn from that so for each demo you maximize the log probability of the action in the demo given the

state and given the last state of the demo that's basically the entirety of the method and this is goal condition behavioral point so you just feed in two states instead of one now one thing I will note here is that while in practice this often makes things work better in theory this methodology is actually a little bit problematic because now we actually see distributional shift in two places we see distributional shift as before in the sense that our state distribution is different our P train is different from sorry RP data is different from D Pi

Theta but we also see distributional shift in another place uh when we do a labeling like this and I'll leave that part as an exercise to the reader uh to and something we could discuss in class so as a hint we see distributional shift actually in two places I'm going to train this way and you could think about where that what that second source of distributional shift is so in theory this is actually potentially worse but in practice it's often better um so let me show you a few examples of works that have done this

um the goal condition behavioral cloning method was arguably popularized by uh these two papers learning latent plans for play and unsupervised visual motor control through distributional planning networks and I'll talk about lowering the laden plans from playlo but first so the concept there was to collect data with humans that were not actually told to do any tasks in particular but they were just told to play around with objects in their environment so in this data set the people are using a VR controller to control the simulated robot and they're kind of Performing random stuff so

they're not moving the arm Randall either but they're performing random tasks and that of course covers lots of different states so a policy trade on this kind of data would have a really hard time trying to find a state that is out of distribution because almost uh all the reasonable states have actually been seen here somewhere or at least states are very much like them but of course it's not clear what task is being performed into the trajectures so by taking this data and Performing this goal relabeling where every structure is labeled with the state

that was actually reached later on in that trajectory and using a latent variable policy that can express multimodality uh the authors of this work are actually able to get a pretty effective policy for reaching a wide variety of goals so this uses that latent variable model trip from before and it uses the gold labeling and putting them together you can get a policy where you can give it a goal like a state where the door is closed or a state where the drawer is open and the robot arm will actually autonomously go and do that

so you can see that it actually does a pretty significant variety of behaviors all in a single policy one of the interesting things you could do with these goal condition key over cloning methods is you can actually use them as online self-improvement methods very similar in spirit to RL so these are not I guess true RL methods but there are alike in that they can improve through experience so the idea is you can start with a random policy collect data by commanding the policy to go to random goals treat this data as demonstrations for the

state that was actually reached so relabel them but let's say that these random uh trajectory is reached use that to improve the policy and then run it again and the idea is that initially the policy does mostly random things but then it learns about the actions that led to the states that actually reached and then it can be more deliberate on the next iteration so the method simply applies this goal relabeling Invitational learning approach iteratively running relabeling imitation then more data collection then more relabeling and then more annotation and that can actually be a pretty

decent and simple way to improve a policy the other nice thing about these goal condition behavioral cloning methods is they are quite scalable so you can apply them at a huge scale this next case study I'm going to tell you about this was a paper led by Drew Shawn and Ajay sridar where what they did is they developed a policy for driving ground robots not autonomous cars yet but smaller scale ground robots that could actually generalize across many different kinds of robots so it's a gold condition limitation learning method that takes in the current observation

and the goal image and actually takes in a history to deal with uh that non-markovianist problem and then it outputs the action and it's trained on data collected for many different kinds of robots ranging from small-scale RC cars to full large-scale ATVs and the cool thing about this policy is that it can then reach goals even for new types of robots that it was not trained in like for example this drone in the top left corner of the video the policy was never trained on drones but it actually control drones into your shot by generalizing

to them from being trained on lots of different vehicles and you can see that it's using some of the ideas we discussed it's of course using this gold relabeling trick and it's using the history that is written in this case by just concatenating the frames although in later work it's also read in with a sequence model Transformer the last thing I want to mention here is a paper called hindsight experience replay which introduced a very similar principle but in the context of all policy reinforcement learning algorithms we'll talk about off policy reinforcement learning much more

later I didn't describe what this is yet but I just wanted to mention this paper because it is something that often comes up in the context of this work it is not doing all condition behavioral cloning but it is applying a hindsight relabeling method to off policy RL and actor critic methods so we'll talk about crl and we'll talk about active critic methods later but I want to mention this because it is an idea that's also very widely used in current methods foreign