"Optimizing Trading Strategies without Overfitting" by Dr. Ernest Chan - QuantCon 2018

thanks for us you know Hong Kong is my favorite Khan training conference but you know I have a problem of keeping up because every year the level of the sophistication of the talks get higher and I feel that you know I need to you know keep doing deeper and deeper research just to keep up you know with the with the crowd and this is you know one of those complicated talks that that we have prepared and going to talk about today actually is some work that we have done in collaboration with our quantitative researcher in the firm dr. Wang and it's about optimizing training strategies using a method that is quite complicated which most people have not would not have used ordinarily so let's take a look at our typical backtest workflow you know we have some pricey use that historical price news and we have a trading model that might be defined by a number of parameters which we use to apply to the where we we can have these prices feed into this model and generate buy and sell trading signals you know for every part and from this training signal obviously created long and short positions again you can imagine you know each part it defines whether you should be long or short a particular asset and then this position generate pls based on real market prices okay so that's you know a typical you know method how we can practice trading strategy and by optimizing trading strategy what we usually mean is that you know for example we might optimize the sum of the P&L by treating the parameters in the trading strategy all right the problem with this approach is that typically the number of trading signals is far fewer than the number of prizes available unless you are doing - the trade which has its own problem but for you know the majority of trading strategies you don't get to make a decision every bar and you know and put on a new position every bar because of this inequality it is very easy to cherry-pick trading signals to optimize based on your historical time series that's a well known problem it's people called overfitting or data snooping or data snooping buyers and so forth and this method of optimizing a trading strategy result in a trading model that has no predictive power on unseen or out-of-sample data right because you you are just cherry picking your parameters that's happen to be best for a small number of these trading signals in the past so what are the ways to overcome this very common problem you know one way the most naive way of course is to give it more data right so with longer historical practice period you get more trading signals and with Motoring signal hopefully the you know the number of trading signals relative to the number of parameters in your trading model is getting larger and larger and so the statistical significance is increases as well but there are some problem with this approach namely that sometimes the older historical prices are not very well into your trading signal for your trading strategy for example you know pea trading used to be very profitable prior to let's say 2002 or something that reason for that is because in the old days the bid-ask spread is much wider due to the fact that the the minimum take is 1/16 or 1/8 sometimes after decimalisation of the u. s.

saw prices all of a sudden the the pit a spread collapse typically nowadays is 1 cent or even less if you're trading on certain dark pools so you know if you are using all the equity days prior to lessor 2002 to backtest an equity model you know you would actually get a tremendous return in old days but that's hardly welcome to to market microstructure today so you know where Jim changes market microstructure changes macroeconomic changes render some you know older data quite irrelevant actually for optimization it might actually create a quite sub optimal model going forward and now last year in my talk I discussed one method of overcoming that and which is using a machine learning technique or bootstrapping some people would call it over sampling with replacement essentially you sample your rotator random number of times to generate more noise in the old data to prevent your trading model from fitting to well to one particular realization of the historical path now that's certainly you know Fallot method to deal with it but it is less easy to implement that for time series data because obviously time series data come with a embedded autocorrelation structure you cannot just take you know one particular taste prices and replicate it five times they stick it randomly sprinkle it throughout your historical data you would destroy the autocorrelation structure so that particular method was more suitable when your data has you know very little autocorrelation and each data point can be treated as independent so you can sample them you know repeatedly without without trouble so so that's so much of that so now this year we are confronted with a lot of training models where it does have to confront this autocorrelation theory autocorrelation head-on okay so there's another method of dealing with this overfitting problem and that is to create a mathematical model of historical price and find an analytical op trading signal now that's if you can do that you effectively have an infinite back test period right because it's a mathematical model as there's no data involved actually the topic of this approach besides that it is quite challenging you know mathematically is that the price model tend to be very simple you know it's not you know I will show you a couple of analytically solvable model later on but you know it is not going to be very realistic in general and you know but when when not only the price model has to be simple in order to be analytically tractable but also the trading model has to be very simple essentially the trading model has to be linear to the prices it cannot have all these bells and whistles that we all like to put into our trading strategy in order to make it work in real life right so linearity of trading signals and then finally even the performance objective that you are going to optimize has to be simple and linear you cannot say oh I want to optimize the return subtract to a maximum drawdown constrained and so forth you know it has to be the performance objective basically had to be as simple as okay what's the P&L per trade right which is linear linear related to to your trading signals and to the underlying prices so and but I will talk about you know a simple mathematical model that people have developed in solving that problem and use that as a contrast to what we can do with a simulation approach which is the main thrust of today's talk in a simulation approach instead of creating a mathematical model of a historical price series we simulate them using a discrete model right I by a discrete model of course it's in some sense still mathematical but at least it's easy to program it in a computer and generate one price after another with without limit like it's not a continue is not as nice as a continuous model of the prices but at least you can programmatically generate as many prizes as possible okay so the only constraint is computational resource rather than you know the availability of various mathematical techniques and furthermore because of this you know a program can be as complicated as you like as we most of us know we are all you know have some programming experience here I think and you know unlike mathematics you can make a program as complicated as you like and as close to reality as you like so you can capture many of the quirks of actual historical prizes for example you can certainly capture see you correlation you can capture philately the clustering you can capture chums and tale events whatever you like whatever you read about in papers you can just throw it in into your time series model in your computer okay and then finally with a with a simulation approach there is no limit on what kind of training model you can impose on this time series on this simulator time series it doesn't have to be linear anymore you know it can be as complicated as what more most of us do when we actually trade life it can be an certainly a nonlinear training strategy and furthermore T the performance objective can also be non linearly related to this training signal again we can measure a common ratio we can measure Sharpe ratio we can optimize anything subject to whatever constraint that we see so please know no longer constrained by what is analytically tractable okay so that's that's alcohol in this talk we want to get there but before we get there let's see what mathematician have done so far with regard to the analytical optimization of a trading strategy so the simplest prize series that you know mathematician can handle and that is meaningful is a mean reverting PI series of course the simplest pie series would be a geometric random walk but you know we both we all know that you can hardly develop a trading strategy based on a geometric random walk so the next its simplest PI series that you can actually trade on and earn a consistent profit instead of a mean reverting lock PI series and that is it's well known that is described by continuous equation called a Austin woolum back equation right so this is a equation where DX X is a lot price let's imagine it's always nice nicer to work in low price paste and price base because then you can you know the the the noise is simply a Gaussian rather than some complicated log normal distribution so DX is the change in the log price in the next time period and the last term wonder if I can use a pointer actually maybe not so easy okay well it's hard to point but anyway the the last term here is a simple random walk right you have a DW which is a Gaussian noise multiplied by volatility factor Sigma we call that the conditional volatility for a reason that we might become clear later and then that's that part is just a simple random walk and then you have a the first term is what's important is a mean reversion term fader is the mean mu is the mean level of the lock price and what this term says is that whenever your price is higher than theta the mean price level there is a force that brings you down word back to failure and whenever the price is below the mean level okay if X is less than fader there is a force that caused the next move to be upwards it brings you back up to the bin to the mean price level so that's a very intuitively understandable equation except that it is written in a continuous price space and but because it is written in continuous spice price space it's actually analytically solvable there are you know you can write down an analytical solution of what the the law price is going to behave going forward okay so but that's just a base price level what the model for a for the price what about a model for trading strategy okay so we first have to decide what kind of trading strategy we want to impose on this simple being governing price series the simplest thing that one can think of is almost like a bowl in Japan method right we want to find out what is the optimal entry level you know how far away from the mean the price has to be before we get into a long or short position and how far away from the mean we should we should get out we should exit due to a profit okay so by optimal entry and exit levels you what is the objective what are you trying to optimize well as I said you can optimize anything too complicated with with this kind of analytical model so we people have only found the simplest objective which is just the round-trip profit okay 2 1 1 3 profit can't do anything more than that if you want to be analytically solvable so ok we'll stick with that but with a caveat that you know there is a certain discount time discount factor so for example if your trade lasts one year and you only make a taller at the end of that year is not as good as if you trade last one minute and you make a dollar at the end of one minute so there is at this time discount factor that you do want to impose on this expected profit but still it is an extremely simplistic objective and that's the only thing that people have been able to solve so far you thinking with this example objective so the solution of this you know what is the optimal entry and exit level in other words think of it as the optimal Bollinger Band form in one time series the solution to that is given in a you know in a book by three author's professor Carty and so and at our publishing 215s called - situating assume in that book they have utilized some quite advanced mathematics from sarcastic control theory hamilton-jacobi bellman equation and so forth to transform this stochastic problem into a deterministic deterministic problem you know transform a stochastic differential equation into a partial differential equation right that's a big step because a partial differential equation so it's hard to solve has you can solve it numerically with any precision do you like so that you know I'm not certainly not going to display all the complicated map but just to get to that step and say it's a mess if you read that book and down and still and and still we are still stuck with a very simple model like that you know even with all this advanced mathematics but there's something very interesting if you look at the solution for example they find that the entry and exit life over a symmetric with respect to the mean if you have a non sealed time discount factor I will show you graphically in the next slide all these interesting features of this and as a good solution right but let me just go through them it worse - worse first the the entry level had turned out that the optimal entry level is closer to the mean than the exit level okay both of them however both exit and entry levels the distance to the mean increases with a decreasing kappa cappa measured the grade of mini version so the weaker the mini version the farther away from the mean you should enter or exit that's eminently intuitive right because if the you know in the case of in the limit of no mini version you should never enter you should wait forever before you should enter and so that's eminently reasonable if and on the other hand if there is very high rate of mini version you better enter into into this asset very quick very soon right after deviate a little bit from the mean because very soon it will mean before right if the mean reverse rate of mean version hi you need to be in a hurry to get in okay and then also the distance of the entry and exit level to the business of that to the mean also increases with increasing conditional forests ility that's also reasonable if the series is very volatile you might as well wait until it by chance get very far away from being before you enter right so you no reason to be in too much of a hurry if by random fluctuation it might hit a very large deviation so these are you know these features are all very intuitive you know these features from the analytical solution you know the last two point in fact can be combined because if you calculate the unconditional agility of a awesome Olympic equation is just is the conditional Felicity divided by the square root of true kappa so the two factors can be combined and that's also apparent if you look at the individual solution but there's the last three points are a little bit interesting if they find for example that the long exit is the same point at the short entry so in other words and and vice-versa that means at no point in this optimal solution are you flat right when we trade me in for in strategy based on polling to pen oftentimes they say and is sort of no no go zone right around the mean where we are neither long or short that turns out not to be optimal based on an exact solution the exact solution that you should either be always long or short there's no point where you should be flat that's quite interesting but also that whether you are long and short is not only a function of your current price so you cannot know by looking at the current price level compared to mean whether you should be long or short that long and short position is path dependent so this this stochastic event equation solution is quite intricate even with a simple model it gives you a path to handin solution for your precision so let's summarize all this by simple picture from this analytical model so you have a mean level here denoted by this horizontal line failure the long entry is level is denoted by this blue dot is very close to me and once the price drop to or below this long entry level we get into a long position we buy and then we will keep this long position even as the price goes up right until the long exit level denoted by the Queen thought and that long exit level as I said is going to be farther from the mean than the long entry level and then once you get out of the long exit position you will immediately enter into a short precision and you will be shorting and shorting until the price drop to the original long entry level whereupon you switch to a long position so at no point in this at no price level are you flat ok as I said and furthermore the distance between the long exit and and mean and as long entry been with the mean scale with the square root of the Sigma square divided by 2 Kappa as I said so this is summary of what we said so it is you know I I think it's quite interesting you know bone Japan trading it's kind of the first thing is again so contrary to learn about by reading is it's the easy thing but it turns out that you know you can create a mathematical model to it and find the exact solution of what constitute the best poor in Japan model and this is it according to these authors given you know with the caveat that the objective that you're optimizing is not the average return but the return of one single trade very big simplification so it this sound nice you know it's very elegant and and you know exact but there are so many problem so many caveats and shortcomings in this model what if for example the underlying price is not a simple minivan pie series as I said what if there is you know for TV casting that has to be modeled by you know a GARCH model of what not what if our objective function is not just the profit of the single trade you know maybe a simple thing like a total PNM already this model cannot describe it model where we're interested in the analyze return we have eminently reasonable criteria everybody are interested in the annualized return well that's not possible to to you know not easy at least to analytically determine based on this model so oh you know the smallest changes to this model make the equation very difficult to transform the stochastic the verification intuitive shamanistic PDE and that's why we need you know for practical purposes we need to use simulation not mathematics even though it's so elegant and so exact it's not really useful for most practical situation okay so that brings us to the main thrust of the talk which is that you know how do we use a numerical assimilation to accomplish a lot of what a mathematician wanted to do but are unable to do okay so let's say we stick with a simplest time series model for registration it could be as complicated as you want as I said but let's start with a simple one such as an auto regressive model in fact we can start with an AR 1 model you know which means a a model that tells you that the future price is linearly related to the current price and plus some noise term ok I will show you the exact equation of that but this is as simple as you can get it's very similar to the awesome ruling by equation except that the prices at the time is now discrete rather than continuous now and also we want to use the objective of averaging the the objective we want to optimize east ed of the Sharpe ratio not rather than a single period return or a single trade P&L right that turns out to be not useful for most people so we are interested in maximizing a Sharpe ratio now but even saying that you want to maximize the Sharpe ratio take take and have two meanings one is that you know let's say we simulate many of these prices based on an AR 1 model okay we have let's say 10,000 replicas of this time series over however long three years five years or one on one way we can formulate our objective is that I want to maximize the average Sharpe ratio of course all these 10,000 parts that would be in spirits very similar to the analytical solution right that analytic solution is also maximizing the average of something across all the paths and we can do the same in simulation no problem but alternatively you can also say that we want to find the trading parameters that most often maximizes the Sharpe ratio of a simulated prices in a manner very similar to a maximum likelihood estimation in that second scenario we are only looking at a bunch of path not averaging across all the paths but only looking at those paths that has the highest Sharpe ratio we will compare the post and calls up to two approaches but just to to have a spoiler the first approach is actually more accurate okay so just to illustrate this the workflow of the simulation we have first of all start with we have to start with historical price use just like any other practice that's at the top box and then the next step is that we will have to fit a discrete time series model to it in our case let's stick with the simplest autoregressive model usually fit by a maximum likelihood estimation no problem with that any you know finance financial times you is the textbook will we have a function for doing that any any software package we have a function for doing that okay so after you fit a Al model to the historical prices now you say well here we are you know aren't we over fitting because you know we are still we we still need to use one particular historical time series to fit this model yes but we are not fitting a trading model we are fitting at times this model there are many prices many more prizes than trading signals right so it's okay to fit a time series model to historical prices it's just not okay to fit a trading model to historical prices there's a big distinction here okay but let's assume that we can fit this prime series model very easily and without overfitting we can then run this AR model and generate as many simulated price users we like and for however long as we like let's say we generate 10,000 of these time series and each one might be ten years long okay then we can run our training strategy no matter how complicated with a particular parameter set on each of these time series and find out the Sharpe ratio and as I said one approaches then we can adjust the numerous parameters of this training strategy so that the X the average Sharpe ratio across all these paths 10,000 maybe 1 billion paths if you have a big computer and so that you can maximize it so that is no problem in the simulation because you don't you have not subtracted scarcity of data you can generate as many companies as you like there is it is almost as good as an analytical optimization because there's no limit to how many times you is you can generate and therefore as little overfitting as is possible so now you found the optimal parameters of your trading strategy based on the simulator time series you can use that training strategy which are now completely specified all the parameters are specified and then do a real back cast on either the original try series or on some out-of-sample data to see how well your fitted model really does either in sample out-of-sample so that is is the workflow of a simulation approach to to optimizing training strategies so let's do a quick and simple example let's say we want to optimize a bollinger you know a mini voting strategy not meaningful it optimize a simple trading strategy on a Forex time series a UDC ad it turns out that I picked this time series because it is quite mean voting on a hourly time bar basis if you run an ADF test testing for stationarity it is better than 1% the p-value so it's a highly stationary type our times use if it isn't we will have a problem because if it isn't it's most likely gonna be very close to a random walk and you know no matter how how much how good a training model you forward it it might not make any sense well you have to have something that is not undergoing random walk in order to to to have any sensible trading strategy so this is a good candidate because of the stationarity but instead of applying a oh you model a continuous time model we are applying a discrete model a r1 autoregressive with lack one very simple the future price or the future log price is linearly related to the current stock price plus random noise and in fact that random noise is given by Gaussian and I have you know there's there's no implication that this simple model is a good model for a UDC ad it's just a first step most likely you will find that it is not a good model how will you be able to find out that it's not a good model well if you try and more complicate model chances are you will find a higher expected Sharpe ratio in a simulation it's easy to tell so you you know you did there's no constraint on how complicated a model you can impose on your on this simulation approach but of course we should start with the simplest which is what we're doing now so this model has only three parameters for the time series model not for the training models try and see is model has only three parameters which you can easily fit with any standard software okay now now that you have computed the parameters for this time series model you can generate as many Pisces you like 10,000 if you like each one might be 3. 7 that's what I we did you know we generate the three point seven years long time series and the reason we picked that is that it is about ten times the half-life of mini-version right you cannot generate time series that is shorter than half-life because the trading strategy can hardly have time to make any money and already you say okay game is over let's liquidate but that doesn't make any sense what kind of Chinese argues that so you need to let it at least has some chance to generate profit so let's just borrow poverty let's make make it 10 times 1/2 line which turns out to be about three point seven years so okay so for each time series we will practice at first what is a symbol strategy well simply by if the expected lock return is greater than some multiple times the unconditional at the times the conditional volatility and sell if it is expected lock return it's less and a negative multiple of that same condition of volatility otherwise neither by ourselves just leave it flat so this is even simpler than the sort of the Bollinger Band model right this is this is each period we get make a decision we don't care about history there's no history involved it's not path dependent at each point you look at that most recent period return and you decide whether the weather is a large number positive number or very negative number otherwise you do nothing extremely simple now again this is just for illustration nobody said that this strategy is the path you you you know what this approach again you can impose as complicated as charges you like right but this simple strategy has only one parameter you can hold strategy that have 10 parameters which in this case doesn't have the data snooping problem because you have infinite amount of data to fit your 10 parameters unlike the traditional approach of practice if you have a 10 parameter of training model and you apply to a pack test time series most likely you can over fit it but here we don't have that problem because you can have 1 million of this price used ok so and we can of course also imposes transaction cost to our trading strategy so no worries so as I said there are two ways to find the optimal parameters one is to maximize the every sub ratio across all this 10,000 paths and if you take that approach which is mathematically summarized here you you find the Sharpe ratio one path given this particular parameter K and then you take the average of this shop across all different paths and then you find the arc net of this expected Sharpe ratio over all the parameter K okay so that's a summary of what we are trying to do in our computer and it turns out that the optimal parameter is 0. 08 with some air path how do i compute the air path where we compared it by giving this image and different random seeds and see how different the optimal K comes out right so you do get some sense of you know how how much randomness tears in this optimal and it turns out that you know is we can estimate it to be second decimal place and as I said there is another way to do this which is instead of looking at the air for Sharpe ratio across all paths we take we look at the Sharpe ratio of a path with a given parameter and we find out what are the the maximum Sharpe ratio that we can get by tuning the parameter okay so for that particular path maybe a certain K has give you the Sharpe ratio and then you can put a histogram of all this path and see what particular parameter give you most often give you the most pass with the highest Sharpe ratio here is a picture of that so you can plot for each optimum okay you you know you can how many of the paths has to have you know as high Sharpe ratio and then you've just picked the mode of this distribution to locate the K that gives you most often to give you the K that gives you the best Sharpe ratio for most realization okay so that's the second method and with that you get a different slightly different K you know point on one is that 0.

08 a but you notice also that the arrow pi is much larger because in this case you are determining the optimal K using a small subset of the path those that give you the highest operation instead of averaging of course order parts so the second method is a little bit less precise but it may have a better intuitive meaning because you can see the distribution of Sharpe ratio as a function of the optimal K and you can see the Oh for that K most part you know today you are most likely in some sense to get a higher Sharpe ratio right okay but anyway you get not too different result within errorbar with either of the method so now it's the time to see how where it does in an out-of-sample test and here it is doesn't seem too impressive right the Sharpe ratio of this path of this trading strategy with a given K I give you this cumulative return out-of-sample out of sample means that we didn't use any of this data to fit the time series model or certainly not the trading model right so yes it's positive yes it's a positive Sharpe ratio but it's hardly something that everybody is going to start investing millions in now why is that we will see why that is the case why is the result not that impressive let's do something to test out our optimal solution let's just randomly pick some other parameter koc oh that's it instead of K evil is 0. 08 let's try what is the back test out-of-sample Pap test look like we've a supposedly sub optimal parameter K equals zero and it look turns out that it has a better shot ratio that the cumulative return looks better with just this you know not randomly picked parameter and you say well how can that be you know we did so much work generating thousands and tens of thousand of this time series do this and that with all the sophistication and you ended up with a training model that is doesn't look at school that's just randomly picking a parameter what's going on well there are many things that can go that is going on actually why is a so cost sub optimal parameter can give you better out-of-sample results or optimal parameters well one reason possibly is that the times this model we have fit using a fix in sample set certainly if you are doing real-life trading you would we fit this AR 1 model continuously with new data either with a moving window or with an ever expanding window right so nobody says that the time series model that you fit using data prior to let's say 2003 is still going to be going to work today so you might as well continuously fit your time series model that's one reason but more fundamentally this methodology does not promise that a particular realized path that our optimal K will be optimal for a particular realized path that is the crucial thing that we can actually get out of this in other words what we have optimized is the average Sharpe ratio over 10,000 paths but if you just pick one of them randomly clearly we know from this chart that it could well be suboptimal right because for example if you pick a point O 5 at the far right there are still some paths where that parameter give you the maximum Sharpe ratio or if you can pick a GU 0 at the far left again maybe 450 of those paths have the maximum Sharpe ratio with ku0 so you know you have a sports spectrum of possible outcomes and your actual out-of-sample path is one of them it's almost like falling attack in this distribution where the thought will end up in this you know area under the curve in you know is represent a particular realized path and for that particular path the optimal K doesn't have to be equal to the one that we found we have done the best we can we cannot you know force the future path to be one or the other right so that that is a you know one of the major problem in in quantitative trading so you have reached the limit of the possibility of populistic model because there is randomness in any kind of you know path going forward you have no control over what it is going to be and therefore your optimal that you painstakingly find in a path may not be the optimal for that particular realization going forward and that's a very important point to realize in any kind of trading but what is there only despair therefore in in you know optimizing trading strategy no because this problem is felt bad only if you are trading just one time Sears right so with one time series of course you can have a bad luck and the particular realized path for way out of the typical path you know or the average path but of course if you are applying this model to ten thousands might one thousand stocks or you know 30 different Forex series of 50 different Forex series then of course with the law of averages it is unlikely that all of them you will be less likely at least that all of them will be so far away from the optimal that this approach will finally work so you have to again take an ensemble approach to make this you know to to realize the the power of this method just using it to try one time series one instrument is not going to you know realize the benefit of it immediately okay so I should mention that all this work actually we have done and then after you know reading it the literature we find that actually dr.