hi in this video we're going to take a a look at regression analysis so this is going to be a review I'm going to be covering a lot of regression very quickly very briefly and a very shallow manner because the goal of this video is to help prepare us for logistic regression all right so the theory of regression has been built up by far the most for linear regression where we have a continuous Target variable now in logistic regression we're going to have two possible levels like success failure pass fail broken not broken true false sick not sick something like that two different levels and in logistic regression the math is a little bit more complicated than you know be just the nature of building it up and what we're going to do is we're going to first take a look at reg regular old regression analysis to help get us familiar with the concepts that are important in logistic regression all right so let's get into it all right so for regression analysis our equation is going to be a linear regression model of this form all right so something to take a look at is that I have predictor variables which are our independent variables here I've got P of them P could be 1 two 3 four five any you know positive integer value and then I have coefficients multiplied by our predictor variable now something that's important to realize is that all of these predictor variables have to be numeric if I have categorical data what I'm going to end up doing is converting to zeros and ones with dummy variables and then we're going to have an intercept term it's our Y intercept and then we're going to have this little error term over here so Y is the dependent variable it's continuous when we're doing logistic regression beta0 is The Intercept so if I plug in zero and for all my predictor variables I'm going to get beta 0 plus the error term then all of the betas are the coefficients of my predictor variables which are the x's and then Epsilon is the error term so all other aspects of the situation all other aspects of the data that's not accounted for with my model is actually housed inside that single Epsilon there okay so let's go ahead and let's build a simple linear regression model just to start out so I'm going to use the Mt cars data set and we're going to say hey I want to know the relationship between miles per gallon and the weight of a vehicle and we're only going to consider weight and we're not going to consider all the other features of a car such as cylinders and displacement so what we're trying to do we're trying to say on average overall what is the relationship between miles per gallon and weight regardless of what the other features of the car is okay and so I I fit a linear regression model and so what I see is a negative 5. 3 for the coefficient of weight and then I see an intercept of 37. 3 all right so what this tells me is that if I had a a car of zero weight zero weight I would expect to be getting 37.
3 miles per gallon all right that's kind of a weird thing to have happened so here we can see that they The Intercept interpretation doesn't make sense on this particular model because zero weight doesn't make sense now what's more important to us what we want to focus in on more is this 5. 3 what this is saying is that if I increase the weight of the C now I didn't take a look to see what the units of measurement are for weight shame on me uh but I'm filming and I'm going to keep rolling uh if I increase the weight of the car by one unit I would expect on average for the miles per gallon to decrease by 5. 3 huh okay and so we can we could use this to kind of get an idea of if I if I see a car I have its weight I have its miles per gallon I can say hey is it overall fuel efficient for its weight compared to other comparable cars right um also this can tell me if I'm if I'm developing a car and maybe I'm designing one I can check and see well you know if I increase you know the weight to have you know like maybe an extra spare tire or maybe an extra seat we increase the weight to add a feature well what will that do to miles per gallon you know definitely an important question during the fuel crisis all right and so we also have some hypothesis testing going on using the T distribution here's our T statistics and here we see we have very small P values so we reject the null hypothesis that the uh coefficient is equal to zero now here we get an R SAR estimate and we also have an F statistic or an F hypothesis test coming off of this which you know if you're familiar with that you know you can use that pretty nicely all right so now let's take a look at the plot that we get when we do this all right so there's a bunch of default plots that R gives us so there are four default plots if I want to see particular ones I enter a vector what I want there are six of them six possible the default is a selected four of them I take my linear model put it into the plot function and if I leave this alone if I don't have the witch parameter filled out out I'll get the default four if I want some more in there I can just control it and I can pick and choose so the first one is the fitted values on the horizontal axis with the residuals on the vertical axis now when I look at these plots I want to not see a pattern that's why I'm primarily interested in I don't want to see a pattern now you might say Well it kind of looks like a happy face or here's a good rule of thumb if you were to move one data point and that or add a point or delete a point and that would change your interpretation you don't see a pattern that's a good good rule of thumb and so here if I was to add a point here it would kind of like disrupt this U shape that I'm seeing and so I I you know there's so such little data here I can't make a conclusion about the relationship between the fitted and residuals now next thing we're going to look at is check to see if the residuals are normally distributed um all right so when I so what we want to see ideally the points would all be close to this line This is the normal QQ plot and when I see points away that's a sign that I don't have normality now here I think probably not normally distributed probably has got some heavier Tails than than we would like to see but this isn't horribly off I don't have that much data when we have less data you know there's a we expect things to adhere less to hypothesized distribution also in these plots by default the largest three extreme cases are always labeled so when I come back here you notice I have prry in Imperial the Fiat 128 and Toyota Corolla these three are plotted and then I have the Chrysler Imperial Toyota Corolla and Fiat 12 now this is the the same three points came up but this was a different calculation so here we're looking at the largest absolute value in residual here we're looking at the largest deviation from normal on these three those are two different calculations but we see the same points are popping up if the points deviate severely from a straight line then that a violation of the normality assumption and I can't trust that f I can't trust the T statistics from the previous hypothesis test that we were looking at previously now here we have scale location so here we have fitted values and here this is the square root of the absolute value of the standardized residuals similar calculation to the first one but this is done in a slightly different manner first of all it's a standardized residual absolute value square roots and we're seeing the same three come up again so I'm starting to to think that these three might be outliers compared to the rest of the data sets now Cook's distance this gives the cook's distance for each observation and if I have a lot of data you can see that this would be a very crowded plot now when I look at this notice that the same three are popping up once again this is very consistent with what we'll see with real data in real life and so this is basically setting up a case that I might want to take another look at the at these three maybe omit them maybe impute their values to make them less extreme in real life if I had something like this going on I want to double check to make sure were these measurements actually taken correctly is there something wrong with the data and now here we have leverage we have standardized residuals and then you know we're going to have a Cooks distance Contour plot placed on top of it and basically by the cooks distance Contour plot is where we'll be getting things flagged the most now here we have leverage on the horizontal axis this comes from the Hat Matrix we have Cooks distance on the vertical axis and so here these lines tell us the ratio of cooks distance versus leverage uh you know the H II / 1us hi II and so you know we can see you know which ones are coming up so the ones that have the largest ratio are the ones that we're going to be most interested in once again it's the same three all right so now let's talk about simple linear regression so my example I gave you was actually simple linear regression but I want to be a little bit more specific first I gave General multiple linear regression now we're going to go to a more uh simple case with just one predictor variable the compute example I gave before that well it's actually the same model um that's simple in regression I only have one predictor variable and so in this situation I have one predictor variable notice that I didn't bother with the subscript on it here I have the coefficient I have the intercept and then I have that error term accounting for all aspects of what's Happening that's not accounted for by the data we typically make the assumption that this is normally distributed with mean zero and constant variance now with multiple linear regression we have more than one predictor variables we have multiple predictors all right it's the same equation I showed at the beginning and so here I'm going to let's say that I'm interested in weight QC and automatic or manual transmission okay so now let's go ahead and build a model so now it's going to be a little bit tougher to interpret what's going on because when I have multiple variables I have to assume when make an interpretation that I've got constant on the others for this all right so we have miles per gallon depends on weight QC and automatic or manual transmission and here we can see that weight has gone down in magnitude so now it's let's just say -4 instead of -5 all right so this is saying when I take a more precise picture of of of the of the vehicle of all the features of the vehicle weight as weight increases the miles per gallon goes down when I'm accounting for other variables So It Goes Down by less now QC has a coefficient of 1.
2 and automatic or manual transmission has a a uh coefficient of three now automatic or manual transmission that automatic transmission is a one manual transmission is a zero so this is saying that we're going to get three more miles per gallon on average when it's an automatic transmission H kind of interesting here we see we have a stronger multiple R squar but here's a little secret that will always go up or stay constant if I include more pred variables and so I can't really trust this at by itself to make an evaluation of my model this evaluation adjusted R squ this takes into account the predictor variables I have if I see a big gap between these two and there there's a little bit of Gap but not a lot adjusted R squ will always be less than multiple R squ if I see a big gap that's a sign of overfitting and once again we get our F statistic now here the assumptions of linear regression so first of all linearity this means that I've got the right equation whatever equation I have I picked it now do I have the right equation we'll do model Diagnostics to check to see do I have the right equation does it look right does it smell right does it make sense to my stakeholders when I talk to them my subject matter experts but ultimately we never really know independent so we're going to make the assumption that the errors in the model the residuals are independent of each other we're going to assume homel cacity that the residuals have constant variance at every level of the independent variable so we have constant variables with respect to constant variance of residuals with respect to our predictor variables and then we're going to make the Assumption of normality and here we're going to assume that the residuals are normally distributed with mean zero and a constant variance with respect to X now these three are the ones we really need to have actually it's linearity and Independence are the ones that we need to fit a model otherwise without these this just doesn't even make sense homo scity like we need that to be a be able to get through some more analysis and the normality is what we need for hypothesis testing I can actually go without the normality assumption and still get a model still interpret the coefficients but I can't trust in of the hypothesis testing if the normality of residuals is violated all right so now to do with the Assumption check basically we run through Diagnostics on our model we're going to go through and just check to see does this make sense does this look right to us now something I like to do when I work on a real project is that I if there is a predictive variable that's very important preferably categorical I'll go through and I'll color by that predictor variable or if it's not categorical I'll go through and bend the continuous values and I could do this for the Target variable or I could do this for a predictor variable either way I find this helped me find patterns I would have missed otherwise all right so now here is a a residual plot now this is the raw residuals and you know I kind of see a u- shape but realize if I was to move this point over here then I would really not see a pattern if you can move one point delete a point or add a point and disrupt a pattern you think you see you don't see a pattern so uh use that trick quite a bit now you'll notice that the automatic transmissions are the ones with the better miles per gallon so on they're on the right for the fitted values that's consistent with that coefficient we saw earlier now for our residuals for hypothesis testing we want our residual to be normally distributed so here I'm just doing a histogram of the residuals to me that doesn't look normally distributed but I don't have enough data to really say yes or no it's light enough on data that I'll go ahead and accept it as being normally distributed for the purpose of proceeding through my project and then we'll go through and do our plots once again I'm coloring by the automatic transmission just in case I might find something and you know so here it's the same thing that we looked at before but now I have a little bit more information with the automatic transmissions included and instead of automatic transmissions I could have Bend uh you know the target variable I could bend a you know a predictor variable now when I have a categorical predictor what's going to end up happening here I have two levels I have manual transmission which is a zero and I have a automatic transmission which is a one in the AM column all right so what's getting up happening is that R is going to fit where the the dummy variable that zeros and ones is equal to zero and then what it's going to do it's going to look at like a shift is going to ship the values by a certain amount for the cases where the automatic transmission column is equal to one or it's automatic transmiss so we can see that it decreases by 0.
