(IS30) Sum-of-Squares Analysis

[Music] hello everyone and welcome back to this series on intermediate statistics today we're going to be continuing our exploration of variable analysis And discussing method known as the sum of squared analysis method sometimes also called the Anova sum of squares that can be used in order to assess the particular contribution of predictors so let's assume that we have this multiple linear regression model beta 0 plus beta 1 X1 plus beta 2x2 all the way down to Beta pxp where p is the number of predictors inside of your model as we already know uh there are several different ways of assessing the appropriateness or the suitability or the fit of these particular parameters for these response variables y um but Nova uh sum of squares in particular SST is equal to SS plus SSM and building F test statistics assumming the normality and Independence assumptions are met is a very common way to do so so in case you aren't familiar with these particular metrics the SST uh can be calculated by doing the sum of your particular y values Associated to their means uh the ssse can be found by doing the sum from K is equal to 1 to n your particular y values minus their predicted values squar and again that sum also ranges from 1 to n and then we have another uh sum of squares the SSM which can be found by doing the sum from K isal 1 to n of our predicted values y High Kus Y Bar the^ squ and one can show that these sums do add up to one another i. SS is equal to SS plus SSM so what exactly do these particular components mean well the SSM can be viewed as the uh portion of the variation that is captured by these predictors so we can say that this is the variation captured by our predictors which is pretty much our model and then we have ssse which is the variation not captured by our model and then our SST is our total variation right so total variation is partitioned one could say uh into the amount of variation that is not captured by our model and the variation that is captured by our particular model obviously we would like our SS to be minimized that's how these betas are constructed via the least squares approach um but we don't want it to be extremely close to zero um else we we have you know evidence that our model has been overfitted and if there is some variation that our data does not encapsulate um our model will not be able to pick up on that um in that particular case so that's you know all fine and well um but there is one natural question uh that one could ask because all of our predictors are contributing to this particular SSM so one could ask is what portion of our SSM in our model is uh contributed contributed from each of our predictors x k let's say where K ranges from 1 to P so uh this particular uh question what portion of the SSM um is sometimes referred to sum of squared analysis sum of of square analysis analysis um some people also call it an Nova sum of squares um but generally speaking I prefer to call it the sum of square analysis because we're in particular focused on the uh SSM and the things that are contributing to it usually Nova Su squares is just focus on your ssse SSM MSE MSM and the quotient of the two building F test statistic right so sum score analysis is is going to be focused on our SSM metric so we're going to see if we can partition that SSM into for example SSM for variable one SSM for variable 2 SSM uh for all the way down to variable P all right so for our discussion what I'm going to do is I'm just going to be using a very pedagogical model um let's assume Y is going to be representing happiness and let's assume that I want to use uh X1 to be age to predict happiness uh let's assume that we want to use another predictor X2 that's going to correspond to height and let's assume that we have a third predictor let's just keep things very um basic let's assume that that's your average average sleep per night let's assume that um you've already calculated a data set of size n um that has these three predictors and Y response variables so we have p uh responses p equal to 3 and one y response for our multiple linear regression data set um and let's assume that we've already ran our least squares regression and let's assume for the safe of this example so let's assume that we obtained an SST which came out to 12 60. 6 let's also assume that the ssse for that particular model came out to 12 32.

9 um and obviously if SSD and SS are these values then one can find um that the SSM is equal to 27. 7 so one can see um that the SSM is uh not that big um which means that when we do the MSM over the msse um that is going to not be so large right because usually when ssse is close to zero that's going to make your fat extremely large um forcing your P value to be extremely closer to zero causing you to reject the claim that these Co I are uh close to zero right so obviously our SSM is not as large as we would hope uh in order to prove that our model is uh working for predicting these particular y values but of course it doesn't necessarily mean that you can't perform a sum of squared analysis on those coefficients even if those coefficients appear to be potentially useless right so let's just work with these numbers and sort of see uh what comes from it right so let's restate the question that we seek to ask ask so what portion of our SSM is associated to our X1 X2 and our X3 and let's sort of see a few ways of how we're going to answer that question so the method that I want to introduce first that's actually quite easy to implement sequentially adds each variable one by one and looks at the percentage for which the SSM increases on each of these iterations so let's actually walk through this example and so to see how that would look like so the first thing that we want to do is first off let's take our model Y and let's regress it against X1 only so let's pretend uh that for this particular data set we see that the SSM for this particular model came out to 17 let's also pretend that the SS for this particular model came out to to 12436 then what we're going to do is we're going to take X1 and then Union X2 inside of that model so now Y is are gred against X1 and X2 so we get went from simple linear regression to multiple linear regression now keep in mind every time you do add a variable your SS will always be less than it was before and your SSM will always be more than was before so SSM is going to guarantee um to increase as long as the coefficient for our new variable is non Zer so in this particular case let's assume that our SSM is going to be equal to 23. 2 right so obviously it increased and let's pretend that our ssse also decreases in a similar manner in particular this would go to 12 37.

4 keep in mind SSM and ssse increase and decrease by the same magnitude because SS this T stays fixed and lastly let's assume we regress y against all three variables X1 X2 and X3 and this is going to be the same as we had before so SSM is going to be equal to 27. 6 and our ssse again for that particular full model uh was 12 32. 9 right so as we clearly see uh 17 increases to 23.

2 and 23. 2 increases to 27. 6 right so uh let's also assume um that for the sake of our next table that we want to say um if Y is regressed against no variables against zero variables we will say that the SSM for that particular case would be zero okay so your SS would be equal to your SST all right so let's look at our table so we can organize all our results so we have our SSM uh which we're modeling um and then we're going to have the number of x's in our particular model and then we're going to look at the change in the SSM over our uh addition of these variables right so we're going to start off with zero variables then we're going to have one variable and then we're going to two variables and then we're going to have our final three variable value um so uh for the number of x's obviously since we have none then our SSM will be equal to zero and there's no Delta to calculate since there's nothing before zero and then for one in particular variable X1 uh then we saw that the SSM uh went up to 17 um so the change from 0 to 17 was equal to 17 and then for two we had 23.

2 and again that's for X1 and X2 and the difference from 17 to 23. 2 is just 6. 2 and then from from two variables to three variables which was X1 X2 and X3 um our SSM of course increased up to 27.

6 and the difference between 23. 2 and 27. 6 is only 4.

4 all right so we have these numbers 7 6. 2 and 4. 4 um so obviously uh our models SSM increase the most um for X1 sometimes looking at these uh Delta ssms in their original form can be a little bit tricky uh so sometimes we would look at the percent change so the percent change in SSM okay so what we're going to do is we're just going to take our uh ssms that we get in terms of deltas and then divide by that by the total amount of SSM right because keep in mind our total SSM uh one goes from 0 to 27.

6 so we're going to be using 27. 6 as our comparison so for example again that's not going to have any value or contribution so we're going to have 17 all over 27. 6 we're going to have 6.

2 all over 27. 6 and then we're going to have 4. 4 all over 27.

6 and keep in mind those ratios will always sum up to one so if that is the case then what we're going to have here for percentages is approximately 69. 1% and then we're going to have 22. 5% for the next and then 15.

9% for the next and everything ranges from 0 to 1 in this case so you can start to make some arguments about which one is contributing the most so from this table we clearly can see that introducing X one uh into our model first uh gives us the largest change in our SSM so the next question that I want to pose to you so is X1 contributing the most so I want you to take a moment maybe even pause the video and think about can we interpret this 69. 1% since it's bigger than the other two um as justifying that X1 is really contributing the most to our multiple linear regression model so before I reveal the answer to this very potentially tempting to answer question um I want to mention one thing that we may have taken for granted in particular when we defined our X1 X2 X3 we had three different variables uh for example age height and sleep per night um but we could have assigned them to any variable in any order for example we chose age for X1 height for X2 and then sleep for X3 but we could have assigned age to X2 height for X3 and then sleep for X1 if that is the case when we introduced X1 into our model first in order to calculate our percentage change in our SSM sleep could have been you know the first variable to enter the model so the natural question is would that change still be 15. 9 % um if we were to replace it first and would our next percentage change be our 69.

1 and would our last change be 22. 5% well naturally it may seem okay yeah maybe that is the case um but let's actually experimentally you know just check it out and see if that is actually um the case so before we actually experiment with that you know pretty much what we're sort of looking at is does order matter for this sequential summon scores analysis if it does matter then that means we would have to technically consider all possible hierarchies of our variables because our first hierarchy was X1 X2 X3 where we're going to keep consistent with X1 age X2 height and X3 sleep but you of course you know that's not the only hierarchy you could consider in order you could do X1 X3 and X2 you could start with X2 you could also start with X3 uh giving you X1 X3 or X3 X1 following and of course you could do X2 X1 and start with X3 X1 X2 so in this particular case um since you're not allowed to repeat variables there would be six possible hierarchies you could assign to your predictor values for your model in particular there's going to be three for the first two for the second and one for the third by default and hence there's going to be three factorial different types of hierarchies for your three predictors so if that is the case let's try and answer does the order matter and even if we do get different percentages in ssms will we still get the same ranking in terms of contribution for these variables so what I want to do is I want to just create another table uh and instead we're going to be introducing X2 first X2 being height and then we're going to follow up with X3 and then we're going to end with X1 so X2 X3 and X1 in that particular order right so these are going to be our models of interest and then we're going to look at our SSM obviously we're going to be coming from an SSM of zero if we have zero predictors okay so SSM for this let's assume uh is going to be 1. 6 11.

3 and 27. 6 respectively keep in mind 27. 6 will be the same as long as you have all three predictors introduced from the same exact data set at the very end and of course there would be Associated sses um for each of these which you should be able to calculate as for example 1259 12493 and 12329 and again that ssse will be the same as the full model regardless of the order for which you introduced them all right so if that is the case what is the percentage change in the SSM so the percentage change in our SSM because this is how we're going to establish our ranking for contribution so if we look at these percentage changes we're going from 0 to 1.

6 that's going to be 59. 1% and then we're going to go from 1. 6 to 11.

3 and that's going to be given to be actually no we're going backwards right so actually let's do Delta first let's do Delta first all right so Delta SSM so Delta SSM 0 to 1. 6 that's obviously just going to be 1. 6 1.

6 to 11. 3 that's obviously going to be 9. 7 and uh 9 uh 11.

3 to 27. 6 that's going to be 16. 3 right and then we're going to take each of those and divide by our total SSM so we're going to divide by 27.

6 uh divide by 27. 6 and divide by 27. 6 to get our percentage changes right so our percentage changes let's assume that we're just focusing again on X1 X2 and X3 the only difference is X1 is going to be coming from our last cell that's where our 59.

1% is going to come from our X2 when we introduce that well that's obviously going to be first and you can find that to be 5. 8% and then when we introduce X3 next that's going to be our second introduction so that's 9. 7 over 27.

6 and that's going to be 35. 1% right and that's if we do an X2 X3 X1 hierarchy for our particular models and we can look at our hierarchy of X1 X2 X3 to sort of uh recollect whether they were the same exact variables and if you remember this was 69. 1% this was 22.

5% and this was 15 9% so do they still have the same exact ranking in terms of contribution obviously no but one thing is uh very interesting is X1 appears to be the largest across these two hierarchies um but X2 and X3 do swap positions in terms of ranking right so you cannot trust sequential sum of squares directly because it obviously depends on the order for which you introduce them into the model so obviously we do have some information coming from sequential sum of squares um but when order matters then comes a couple natural questions or a couple points uh that we need to keep in the back of our mind in case this is the method that you want to discuss so overall this method as I've already called it is called sequential sum of squares and this is a sum of squares analysis and some people also call this a type one uh sum of squares analysis so uh what have we observed so far about sequential sum of squares obviously depends on the order of introduction uh which then begs the question which one to trust and one could say okay maybe I could calculate all six factorial different types of hierarchies look at the percentage changes and then average all those percentage changes and then I can get from there but again that could be potentially misleading so another approach which I will introduce is sometimes or can be referred to as the full reduced model comparisons full reduced model comparisons and if you've ever learned about partial F test then this should uh sound a bit familiar to you and we're going to also refer this as type two all right so what exactly is our full model so our full model is going to be the model containing all of our predictors so that's going to be y regressed against X1 X2 and X3 all at once so obviously it doesn't really matter the order and as we've already observe the SSM um for this pedagogical data set uh came out to 27. 6 all right so what we're going to do is we're going to remove these variables one and then one and then one so for example we're going to look at X1 removed and then we're going to look at X2 removed and then we're going to look at X3 removed so we're going to look at our reduced models so reduced models um and see what our SSM turns into when you remove these particular variables all right so in particular uh if you remember for example from uh AV plots or partial F tests uh the model that we're going to be interested in is why regressed against X2 and X X3 this is going to be y regressed against X1 X3 and this is going to be y regressed against X1 and X2 so obviously you can easily calculate the ssms for each of those particular models and one can find that this is going to be equal to 11. 3 20.

2 and 23. 3 let's say and then we can calculate the change in SSM and one can find this is going to be equal to 16. 3 7.

4 and 4. 4 respectively right and keep in mind when we're looking at the percentage change we're looking at what are ssms were with respect to our full model what our SSM is with respect to the full and what our SSM is again with respect to to the full so we can then look at our percentage change so our percentage change in SSM um by looking at our particular values divided by our total SSM of 27. 6 and that will tell us similar information as was before so in this particular case one can find that this is about 59.

1% and then we're going to have 26. 8% and then 15. 9% right um as you can clearly see uh X1 still is having the larger one X2 is now the second largest and third is the third largest or the small smallest in this case but before we proceed I just want to mention that the sum for these is not equal to 100% right so why would these percentages not add up to 100% well we're not considering that you know when we remove X1 we're also removing a bit of influence of X2 or some of x1's influence is still remaining inside of X2 and X3 so when you see this you know sum not equal uh 100% you can ask the question do they overlap right and obviously looking at the individual predictors we should already know um is not necessarily trustworthy because they could consider or could contain interactions or dependencies within our data even if they is by chance right and that's why you shouldn't just look at the beta coefficients and their magnitude from zero even if it is in the standardized realm right but nonetheless it does give us some um notion that we don't have to focus on the order for which we remov compared to sequential sum of squares right so even in this case we see okay maybe X1 is really contributing um to our particular data but it does not consider this overlapping issue right so even if it does not add up to 1% it still captures it does capture the percentage contribution percentage contribution roughly if they were not interacting right so it doesn't matter the order but they could be interacting in this particular model does not necessarily con consider that right so what do you think a type three sum of sores analysis method will look like you may want to pause the video instead of think of how that would play out so the next sum of squares analysis method can be sometimes referred to as the full reduced model comparisons method with polinomial and interaction terms included and I'm going to be referring to this method as type 3 uh sum of squares analysis so what exactly will the full model be in this scenario so again we're going to be considering X1 X2 and X3 inside of our model and then we're going to be including some polinomial terms for example X1 2 X2 2 X3 2 you could also go up to cubic or Cordic if you wish but let's just stop at quadratic and then we have X1 and X2 multiplied X1 X3 multiplied X2 X3 multiplied let's also do our X1 X2 and X3 all multiplied together right so these are our base variables or predictors and then we have our polom polinomial terms and then we have our interaction terms interaction terms so our X1 X2 X1 X3 X2 X3 that's obviously going to be our first order interaction terms and since we only have three variables that means we are only going to have one second order interaction term because it's one variable with respect to two all right so this particular model is what we're going to be referring to as our full model right and what we're going to do is we're only going to remove our base variables so we will only remove our X1 X2 and X3 for the reduced quote unquote Models All right so let's assume that once you actually uh use the same exact data as we were doing before for our full model let's assume that the SSM for this particular case is 3184 right and of course you could look at the ssse for this particular model and let's pretend that that's 9422 right and obviously since we're including several different variables we have 3 + 3 + 3 + 1 um obviously that's going to have our ssse 10 to zero so let's also assume that we are not overfitting that is you've performed all of your explanatory variable analysis and your model analysis to assume that we're actually not overfitting that this model is indeed appropriate for your data so once you have your full SSM now we can get our partial uh ssms and then analyze from there right so for our model we're going to take our full model and take away X1 and then we're going to have our full model and take away X2 and then our full model and then we're going to take away X3 in each of them and then we're going to be calculating our SSM as usual our change in SSM and then we can get our uh percentage contribution contribution to our model as well right so let's see for this data set what it actually comes out to and see if we get any uh spooky math so for SSM uh for when you remove just X1 you're going to have 278.

50 obviously it has decreased and then 281. 50 and 38. 5 right so then we're going to calculate the percentage of SSM that is associated to each of these Delta ssms so 39.

9 over 38. 4 36. 6 over 3 38.

4 and so on and that's going to give you 12. 5% so 12. 5% 11.

5% and 12. 1% and if you add up these total these little percentages then what you're actually going to get is a total of 36. 1% of your SSM is contributing to X1 X2 X3 now uh where is the rest of this uh SSM located well obviously it's going to be located to our polinomial terms and our interaction terms um so since this is substantially less than 50% let's say then we have evidence to believe that maybe our model is not entirely linear or our variables are interacting with each other right so these percentages are not necessarily meaningful nonetheless we can say that um X1 is potentially contributing a little bit more than uh X3 but usually since it doesn't add up to 100% it's usually hard to interpret these in their raw form so what we instead would do so this is obviously our total contribution with respect to everyone usually we would only like to look at the relative contribution with respect to each other so you take each of these objects and we divide them by our 36.

1% so you get a bunch of percentages that indeed always add up to 100% when you do that you're going to have 34. 6% you're going to have 31. 9% and then you're going to have 33.