Reinforcement Learning for Trading Tutorial | $GME RL Python Trading

what's happening guys my name is nicholas renate and in this video we're going to be using reinforcement learning for trading and hopefully we can try to make a couple of bucks of gme in the process let's take a deeper look as to what we'll be going to so this video is really going to be focused on using reinforcement learning to help us out with trading now i mentioned jme but this could just as easily be used with different stocks forex whole bunch of different equities and other types of securities as well so in terms of what we're going to be covering we're going to be using the open ai gym any trading environment so this is a custom trading environment that you can use to trade a whole bunch of different types of securities what we're then going to do is we're going to build a custom training bot using reinforcement learning and specifically we're going to do that using the stable baselines package which is just a really great package to get started with reinforcement learning really really quickly then what we're going to do is load in gme trading data into our trading environment so hopefully our reinforcement learning agent is able to learn how to trade that particular stock let's take a look as to how this is all going to fit together so first up as i said we're going to be using the any trading open ai gym environment so this just makes it easier to build a custom trading environment we'll then load in our gme price data that we've got from marketwatch and this will be available in the github description below as will all the code and then we're going to train a reinforcement learning bot to try to trade against that particular environment ideally trying to make additional profit on top of our portfolio ready to do it let's get to it alrighty so in order to get started with reinforcement learning for trading there's going to be five key things that we need to do so first up what we need to do is install and import our dependencies and really we're going to be relying on the jim any trading environment there as well as stable baselines then we're going to want to bring in some market watch gme data so specifically gamestop data and again in this case we're going to be using daily data but if you could get minute data or even second data that's probably going to give you a much more sophisticated model ideally the data that you use should mimic the frequency that you want to be trading at so once we've done that we're then going to build up our environment and this is specifically going to give us a trading environment that allows us to work with open ai gym now in this case the environment that we're going to be using is called gym dash any trading and it allows you to pass through a whole bunch of different types of data so say for example you didn't want to do it on gme you wanted to do it on amc or if you wanted to do it on bitcoin or ethereum or a different cryptocurrency you could actually do that simply by passing through different reference data to any trading environment which is actually pretty cool once we've got our environment built we're then going to train our reinforcement learning model and specifically for that we're going to be using stable baseline so this is a great reinforcement learning package which i've recently discovered that allows you to do a whole bunch of sophisticated stuff reasonably easily and then we're going to test it out so we'll train on for example 50 to 100 days of data and then we'll test it out on the next 10 days now again all the code here is shown for demonstration only so if you are going to deploy this please do your own research and it's not meant to be financial advice all righty first things first let's go on ahead and install and import our dependencies alrighty before we go on ahead and run that let's take a look at the line that we've written so in order to install stuff from within our jupyter notebook we're just going to pass through an exclamation mark then pif install and then followed by all the different packages that we want to install so the first one that we want to install is tensorflow gpu and specifically we want to install version 1. 15 because stable bass lines only works with tensorflow versions below version two as at this point in time so we're going to be leveraging those versions of tensorflow so tensorflow dash gpu equals equals 1. 15.

0 will allow us to do that then we've got tensorflow equals 1. 15. 0 so this is also giving us the non-gpu version of tens flow to make sure that we've got both bases covered then the next dependency that we're installing is stable dash baseline so this is going to give us our different reinforcement learning algorithms then jim dash any trading so this is going to give us our trading environment and then last but not least we're installing jim as well so this is going to give us open ai gym which is the base framework that gym dash any trading is actually built on so the entire line is exclamation mark pip install tensorflow dash gpu equals equals 1.

15. 0 tensorflow equals equals 1. 15.

0 stable dash baselines gym dash any trading and gym so if you go on ahead and run that that's going to install all of your dependencies that you're going to need to be able to get this up and started and you can see it's already installed and this has gone reasonably quickly because i've already got a large number of these dependencies installed now the next thing that we want to do is actually go ahead and import our dependency so there's quite a few here so let's go ahead write them out and then we'll take a step back and take a look at them alrighty so those are our dependencies now imported so we've gone and ridden seven lines there so if you add in the comments that's ten lines so we've broken this up into three different sections so first up all the open ai gym stuff so first up we've been import gym and this is going to be the same dependency that we installed up here and this is really open ai gym so it gives us our base framework for reinforcement learning and building reinforcement learning environments then we've imported gym dash any trading or gym underscore any trading and this is going to give us a specific trading environment so to do that with written import gym underscore any trading so these two environment components are actually where our trading bot and specifically our trading reinforcement learning model are actually going to learn to trade then we've imported some dependencies from stable baselines and this is really where we get all of our rl stuff so any of our reinforcement learning algorithms and specifically our helpers like our wvec environment are imported from stable baselines so we've imported two dependencies there so from stable underscore baselines dot common dot vec import dummy vec env so this is a vectorized wrapper that basically we wrap our trading environment in in order to pass it to stable baseline so think of this as a wrapper then we've imported the actual algorithm that we're going to be using so in this case it's a2c so to do that we've written from stable underscore baselines import a2c now you could try out a whole bunch of other different reinforcement learning algorithms so if you actually take a look in stable bass lines there's actually a whole bunch that you can try out here so there's acer ppo2 sac the all-famous dqn so there's a bunch that you can actually try out for this then the next set of so that's really our reinforcement learning stuff imported so really our dummy vec envy and our algorithm then we brought in some processing libraries so we brought in numpy so import numpy as np so we're going to use this a little bit later on when we go to evaluate our model import pandas as pd so we've brought in pandas to be able to read in our gme trading data and you'll see that when we actually go to step one and then last but not least we brought in matplotlib so this is going to give us some visualization functions when we go to plot out our trades that we've actually gone and passed through and that is really our dependencies import it's really our gym environments our stable baseline algorithm stuff and our processing libraries those three components or those three sets of components are going to be the core dependencies that we're going to need for this so that's really step zero now done install and import our dependencies now we're up to step one so bring in our market watch gma data and in order to do this we're really going to be relying on pandas here so again if you wanted to you could actually change this data so i actually had a bitcoin data set that i got in from kaggle but i've also got a gme data set given how prominent it is at the moment i figured it might be useful to take a look at that as well now again this is for demonstration purposes so if you were going to go try this out please be mindful that this isn't financial advice it is really to show what's possible with the technology so in this case we're going to work with our gme data but you could just as easily use the bitcoin data set as well it just will require a little bit of additional pre-processing so you can see here that i've got a data set called gme data and it's in a csv format so we're going to be leveraging that so let's jump back into our notebook so we're now going to bring it in so in order to bring this in we're actually going to be using pandas so let's go on ahead and do that and take a look at what our data set looks like okay so that's our data now brought in so in order to bring it in we've just used the pretty much standard pandas read csv functionality so we've written df equals pd dot read underscore csv and then we pass through the path to our data set so in this case it's going to be data forward slash gme data dot csv if you had data sitting in a different folder you just need to specify the full path to that particular data set if we wanted to bring in our bitcoin data for example i could just type in bitcoin data and that would bring that in instead of our gme data but in this case we're going to stick with our gme data and we're going to take a look at that so if we type in df. head this is going to show us what types of data we've actually got in there so we've got date open high low close and the volume as well now again this is just a straight download from marketwatch i selected a custom range which you can see over here and then basically all i did is i hit download data and this allows you to download a data set from market watch but again if you had more rich data sets or a data set with a increased level of granularity you might choose to use that as well and also if you want to see how to build custom indicators and how to use different indicators please let me know in the description below i'd be more than happy to take a look at building a video on that so in this case we're going to stick with us pretty straightforward data set and start setting this up for trading so in order to actually do that what we need to do is first up convert our date column to a date time type so if i actually take a look at id types at the moment you can see that our date is just an object now in order to get this to work with the gym any trading environment we actually need our date to be a date time so we can do that reasonably easy using the pandas. truedatetime function so let's go ahead and perform that conversion first up all right so that's our conversion now done so in order to do that we've written two lines and really we only need to write the one line the second line is to do the display so we've written d f and then we'll pass through an index which is date and this is basically resetting or overwriting our date column with our new datetime function so to convert the date column we've used our pd.

truedatetime function so in order to do that we're two date in pd. twodaytime then we've passed through our date column so this is effectively taking our existing day column passing it to our pd. true date time function which then converts it to a date time type we then go and overwrite the existing date column with that new date time type and then if we type in df.

d types you can see that we have in fact now converted this to a datetime 64 type so this is going to now be able to work with the jim any trading environment but in order to pass it through to that we need to make one more change and specifically we need to set this column as the index so this is just the way the gym any trading environment expects to receive the data so let's go on ahead and do that first up alrighty so that's our column now set as an index so in order to do that written df dot set index and then we'll specify the column that we want to have as our index so now we can actually refer to this by date so to do that we're in df dot set index we've passed through the date column as being the column that we wanted that index and then in order to do it in place we've just said in place equals true so this means that we don't need to go and make a copy of our data frame we're just updating our present data frame in place and then we've gone and shown the five top columns so df. head and this just shows us our first five columns within our data set but again the core changes that we've made is we've brought in our data set we've converted this column to a date time type and then we've gone and set it as our index and that's really all you need to do in order to be able to get started in terms of passing this to any trading environment another thing to note is that it does expect to have certain columns i believe it needs open high low close and volume but again i'll clarify that in the comments below just so you've got an idea of what types of data it's actually expecting that's really it in terms of the different types of data transformations that you need to perform to load up a custom data set into our any trading environment but again a key thing to note that you could sub this out with a bunch of different types of data if you wanted to okay so now what we're going to do is we're actually going to pass this into our any trading environment that we had up here so let's go ahead and do that all right so that's our environment now created now again uh think about environment as where our reinforcement learning bot or our reinforcement learning agent is going to learn how to trade so if we actually take a look at our environment you can see that we've got prices i believe so this is actually taking all of the prices from our data frame and displaying it out there we can also take a look at the different signal features that it's going to use as part of our trading and i believe it takes price and the change in price from the previous time step so it's actually doing a bit of a difference which is why this first feature here doesn't actually have a difference because it's not it doesn't have any previous data so in order to create this environment which you'll see in action in a second we've gone and written one key line so we've written in env equals gym dot make and this is from our open ai gym that we brought in up here and then because we've imported and installed jim any trading we're able to set up a template environment in this case we've set up the stocks environment so to do that we're in stocks dash v0 then the next parameter that we've passed through is the data that we actually want to work on which in this case is our gme stock data that we just went and pre-processed so to set that we've set df equals to df so keyword argument and then we've gone and set two other keyword arguments that you can play around with so the first one is we've set what portion of data that we actually want to be included within our environment so in this case we're including the first 100 time steps and we're starting from the 10th time step and i'll explain this why this is actually needed so we actually need this to be at minimum number five and so the reason that we need it to be a minimum of number five is because when we set our window size we need this first parameter equal at least this or greater so what actually happens when you set this window size parameter is you're actually specifying how many previous time steps our trading bot is going to have as reference data for when it goes and makes its next trade so in this case we're specifying our window size equals to five so this means when our trading bot deter is trying to determine whether or not to place a trade it's actually got the five previous sets of price and price difference so effectively these five sets of signal data or five sets of signal features are what our trading bot is going to have access to so in this case you actually need to start at five so that it's got five sets of previous data and then this second parameter you can change and tune that to whatever you really that you want it to be in this case we could set it to 100 we could set it to 150 but you can't obviously go more than the maximum amount of data that you've got so if i summarize that entire line we've specified enb equals jim dot make we've specified that we want the stocks environment so stocks dash v0 we've passed through our data which in this case is our gme data by specifying df equals df specified how much data we want to pass through to our environment by setting frame underscore bound equals 5 comma 100 so we're going to start at the fifth time step and go all the way to the hundredth time step and then we've specified how big we want our window to be so in this case it's window underscore size equals five but again you could pass through more data you could pass through less data as well and then you can take a look at the environment by specifying env dots prices so in this case if we type in prices we can get our prices we can also take a look at our signal features and i think it'd be really cool to extend this out and actually build some custom signal features so say for example we actually went and calculated some specific financial formulas might be interesting to bring those in but again let me know in the comments below if you'd like to see a video on how to do that so for now we've actually got our environment set up now the next thing that you're probably wondering is okay that's great how do i actually see this liven in action well let's go on ahead and test out our environment so rather than actually trading properly we're just going to take a bunch of random actions and see how this actually performs and if you watched any of my other reinforcement learning videos so this is akin to creating a test environment and really what we're trying to do here is establish how it actually works and take a look at some of the features available within so let's go ahead and do that and we'll take a look alrighty so that's our environment tested now we've written quite a fair bit there and specifically we've written one two three four five six seven eight nine ten eleven lines of code and basically what we've actually gone and done is two things so we've gone and run through a bunch of random steps within our trading environment and then we've actually gone and visualized it out so we can actually break these two components out separately so this entire code block here is really similar to what we've written if you've ever watched any of my reinforcement learning tutorials before pretty much the exact same as what you might do in any other open ai environment in this case we're just testing it out and specifically we're just taking a bunch of random steps using env. actionspace.

sample so if we take a look at the whole bit of code so we've written state equals env. reset and this basically allows us to get our initial state and ideally what we're going to get back out of this is our signal features from our environment and specifically our five windows or the five components within our window then we're basically looping through each step and we're taking a random action so within any within most open ai gym environments you're able to access the action space so if we actually take a look at that to env. action space you've really got two different actions that you can take within the any trading environment buy and sell so they don't actually have a hold environment i believe it's only buy and sell and if you actually scroll inside of here you can actually take a look that it really only has long and short positions i don't believe it's got a hold position really yeah so it's only got sell and buy which you can see down here which are these two actions that you can see here now in order to access those actions we can actually access our action space so enb dot action space dot sample and this is just going to take a random action so it's akin to just randomly buying and selling gme shares and then we're actually going to take that action and store it inside of a variable called action and then we're going to apply that action to our environment which you can see here so basically we're calling emv dot step and we're passing through our buy or sell action to our trading environment then out of that we're going to get our state again so we're going to get what our environment looks like our reward so whether or not we made any profit whether or not we're done and ideally our done is going to trigger once we've gone through all of the different steps within our trading environment so really the different time or the different days that we've got within that data in this case remember we're using 100-5 data so 95 days worth of data for this particular environment now if we're done then we're going to print out our info and break the loop so you can see the info has been printed here in this case we've got our total reward of 64 which doesn't mean too much but we're specifically interested in this total profit figure so we've actually managed to lose money so our total profit is 0.

78 so rather than so this is our profit plus that balance so if we actually made a profit we'd expect it to be one dot something in this case taking a bunch of random steps has actually lost us money which is pretty much as you'd expect to happen so everything up to about this line or everything that you can see visualize in this line is really this code here so it's just taking random actions then the cool thing about the any trading environment is that you can actually visualize the environment as well itself and in order to do this we can actually use matplotlib so remember we imported this over here in this line so in this case what we're doing is we're calling or we're resizing our plot and setting it up so dot figure fig size equals 15 comma six so 15 by six we're calling plot. cla and then this is the most important function so env. render all allows us to render all of the different trades that are happening or that have happened as part of our trading within that particular environment so you can see the red dots are shorts the green dots are going long in this case here you can see that it's trying to short up here it's shorting here which is probably an absolute terrible trade shorting up here which isn't so bad buying down here or going long maybe not so good as as well and then we're calling plot.

show so this actually renders the plot so all up you can see that we've taken a bunch of random steps within our environment using these two lines and we're effectively visualizing it out but you can see now you can sort of get an idea of how we can actually play around with this environment now this is pretty much step number two building our environment now done so we've created a test environment and we've tried it out now it's actually time to actually start building our reinforcement learning model to try to actually trade profitably in this particular environment now again you can sub out different types of data if you wanted to you can get more granular or higher level i believe this data is daily so if you actually take a look yep so it looks like some day so 11th march 12th of march 10th march 9th march 8th the march so on so what we're now going to do is build up our proper environment and actually start training it using stable bass lines so first up what we actually need to do is wrap our environment inside of our dummy vectorized environment wrapper that we brought in up here so we're going to create our environment wrap it in this and then we're going to create an algorithm using the a2c algorithm and start training it so let's go ahead and kick off that first bit of code which is where we need to wrap our environment inside of dummy vec environment so let's do it okay so that's our dummy vectorized environment now created so in order to do that we've written two lines of code so env underscore makeup equals lambda and then we've included a colon and then we specified jim dot make pretty much his exact same line that we had up here so when we actually created our dummy environment so we've specified so jim dot make stocks dash v0 another cool thing to point out is there's a generic environment there's also a forex environment as well so if you wanted to do something more like commodities trading you might choose to use that it actually has different i believe bid and us spreads as well as different fees applied so then again in this particular line we've again specified we want to pass through our gma data so to do that we've written df equals df specified our frame bound so in this case we're going to be using 95 days worth of windowed data again and then we specified our window size equals 5 again so this entire component here is exactly the same as what we wrote just above the only difference is that we've now appended a lambda function to the front of it so this is basically going to allow us to loop through and create multiple environments if we wanted to in this case we're not we're just going to be using the one environment so when we're calling dummy vec env we're passing through a square array and we're passing through our e and v make up variable which is what we had here and we're storing that inside of e and v so if i summarize we're creating an environment maker function we're then taking that environment maker and we're putting it inside of our dummy vec e and v wrapper and we're storing that inside of a variable called env so basically when we now go and start building our training model we're actually going to be using this e and v variable here and that brings us to our next step so we're now up to actually setting up our algorithm and actually kicking off our training so let's go ahead write that code and before we start running it i'm going to tell you what to look for and actually how to train this so let's go ahead and do it alrighty so those are our next two lines of code done and dusted and really in these two lines of code we're going to start training our reinforcement learning bot to start being able to learn how to trade gme so in order to do that we've written model equals a2c and a2c is the actual algorithm that we're going to be using inside of this particular run so again you can try using some of the other algorithms that we've got here so acer is one that's quite popular so is ppo2 and again if you wanted to use your own custom algorithm you could do that as well so we've written model equals a2c and then we've passed through the policy that we want to use in this case we're using a multi-layer perceptron lstm policy what this basically means is we're going to be using a deep neural network which is why we need a tensorflow right at the start which has a lstm layer and this is really really important when it comes to trading because our lstm layer allows our neural network to keep context and learn about previous history within its neurons so it's actually able to take into account what's happened in the previous windows to work out what the next best trade outcome is then the next parameter that we've passed through is our environment which we had up here and then we're specifying a verbose equals one so passing through verbose equals one just allows us to get a bunch of information as our model is training then the next so that so this particular line here is really giving us our algorithm so this is our algorithm now step set up now the next line that we've set up is really where we start to train our algorithm so we've got model dot learn and then to that will pass through total underscore time steps equals a hundred thousand so if you watch the video on training reinforcement learning models with stable bass lines again these will be pretty familiar because effectively we're just using an algorithm and then we're just going on and kicking off and learning i'll link to that video somewhere up above check it out so now what we're going to do is we're actually going to kick this off so ideally what you want to be paying attention to as your stud or as you start training is you want to take a look at the explained variance value so you want this to be as high as possible because that means that your model has begun to learn about all the different variances within that particular data set and is now starting to understand some of the patterns so what we're going to do is we're going to kick this off and if we get to a point at where it's significantly higher so ideally you're going to be looking for a number between 0 and 1 the closer to 1 the better so if we get to a high level we'll stop that model and then we'll give it a test so let's go on ahead and test this out and take a look at how this is going to run actually before that as well you can actually configure something called a callback within this learning function as well which allows you to automatically stop the model when it hits a certain level of accuracy as well so if you'd like to see a video on that by all means mention it in the comments below and i'll take a look in this case let's go ahead and kick it off so you can see our model started training and you can start to see these explained variants the fps the number of updates our policy entropy how many time steps it's taken as well as our value loss so ideally you want your explained variance to be as high as possible so ideally the closer to the number one you get the better and you want your value loss to be as low as possible as well so if we keep scrolling down ideally you can see it's sort of bumping around and this is pretty common but if you start to see a value which is close to 1 then ideally you might want to stop it there and test it out but again there's a more structured way to do this if you want to see how to do that by all means drop a mention in the comments below so we're going to wait and see if we get to the elusive 0. 99 explained variance then we'll stop there and we'll see how our model is actually trading alrighty so we managed to stop our model training with an explained variance of 0. 129 we had a value loss of 0.

0669 and a policy entropy of 0. 204 again if you'd like a deeper description into this again hit me up in the comments below more than happy to explain this in greater detail but ideally what you want is as high a possible explained variance as possible this is going to give you a better model you can see that as we're getting towards the end of the training so if i scroll a little bit further down we did have an explained variance of 0. 903 so only a couple of steps before so ideally this should give us a model which performs reasonably well but again with all things algorithmic trading and reinforcement learning finding the perfect solution isn't always super straightforward so in this case we managed to get a reasonably well performing model so again we had 0.

903 up here and it bounced to the negatives and then hung around still in the positives before we manage to stop at training so now what we want to do is actually test this out and see how it actually performs so we're going to test it out over 10 days and see how it actually performs trading in that particular space so let's go ahead and write this out this is going to be really similar to what we've written up here so we might even reuse some of this code or we'll probably write it from scratch and see how we go so let's go ahead and do it so we're going to take this model and we're going to test it out and see how well it actually performs alrighty so that's our evaluation code now written now before we go ahead and test this let's take a look at the code that we've written so first up what we're doing is we're creating a new environment and we're passing through again stocks our data frame our frame now in this case we've specified that we want to go from day 90 to day 110 so there is a little bit of overlap with the trading environment or the training environment that we actually had so again the more data that you've got and the more significant your trading factors are the better this is likely to perform so again having that overlap is cheating in some senses but again this is really just for demonstration just to show you what's possible really with reinforcement learning and then the last thing that we pass through to that environment is we've set our window size equal to five so really similar to what we had up here but we're not wrapping it inside of this dummy vect e and b then we're specifying a really similar code to what we had inside of our test environment here the core change however is that rather than taking random actions we're now actually using our model and predicting which action it should take so whether or not it should buy sell or hold so in order to do that we've written obs equals obs so this is taking our observation and we're basically just reshaping it using numpy dot new access to be able to work with a non-vectorized environment we're then passing through that observation to the model. predict function so this is effectively taking what our current environment looks like passing it to our predict function and generating a prediction out of that we're getting the action that we should take as well as our states so this action is really one of the most important components because we're then going to apply that action so buy or sell to our trading environment to take a look at how that's going to perform so this should not have an equals and then we're going to get back from that our rewards whether or not we're done and any info so this info is going to tell us what our profit looks like at a point in time so all things holding equal this should run and we should see how our trading environment performs and again we'll test this in a few different scenarios just so we can see what it looks like so in this case again we've got the overlap we'll test it without overlap so we can take a look at how the reinforcement learning bot is actually performing okay so let's take a look at what we've done there oh we need to have obs here all right so that's our reinforcement learning model operating on day 90 to 110 and it looks like our total profit in this case is about three percent if i zoom in on that you can see that we've got a multiple of one so we've got 1. 03 so this ideally is about 3.

1 percent return but again this has our overlap so we might not actually be performing that well in the real world so if we take away the overlap now and say if we went from day 110 to day 140 oh it looks like our profit drops to about 60 so now we're losing cash so if we went earlier for example say we went completely within the trading cycle so what would that be day 70 to day 90 again 58 so it looks like 90 to 110 it was trading pretty well but again outside of that maybe not so well so that was five percent so again 5.