[Music] now certain algorithms perform better when the geometric fields are standardized so that the field mean equals zero and the field standard deviation equals one as illustrated in the slide positive z values may be interpreted as representing the number of standard deviations above the mean the data value lies while negative z values represent the number of standard deviations below the mean some analysts standardize all the numeric fields as a matter of course next i'll show you how to standardize numeric fields in phyton from scipy or scipy import stats we will standardize the age variable and save it as a new variable h underscore z you may refer the command in the slide the z-score function calculates the z-value of the given variable in this case age written in the code as bank underscore train open bracket h close bracket as the z-score command is part of the stat package we write the command as stats that z-score open close parenthesis we save it as a new variable in the data set h underscore z once the numeric field are standardized one may use the z values to identify outliers which are records with extreme values along a particular dimension or dimensions for example consider the field number of contacts which represents the number of customer contacts made over the course of marketing campaign the mean number of contacts per customer is 2. 6 with a standard deviation of 2. 7 which we allow for rounding so we obtain the standardized field based on the first equation a rough rule of the thumb is that a data value is an outlier if its z value is either greater than 3 or less than negative 3.
for instance a customer who had been contacted 10 times which seems a lot would have standardized value based on the second equation thus 10 contacts while a lot is not identified as an outlier using this method since 2. 7 is less than 3. data scientists should consult with a client regarding what he or she would like to do with any outlier outliers should not be automatically removed nor should they be automatically changed their unusual values may bring to light important aspect of data that should be discussed with the client or with the database administrator for this example we will continue using the age underscore z variable that we created in the previous section we will find outliers by using the query function which identifies rows that meet a particular condition we will be using this command the condition we want all returned records to meet is given as age underscore z greater than 3.
then age underscores z less than negative three so in other words the condition requires each record to either have an age underscore z value greater than three or an age underscore z value less than negative three the or is specified by the character pipe between the two conditions all records which meet the specified condition are returned in our example there are 220 records that have age underscore z values greater than 3 or less than negative 3. we can use these records to create a new data set which is made up only of these values in this command by giving the output of the query command a name we create a new data set of only outliers which we have called bam underscore train underscore outliers let us sort the data set bank underscore train outliers by its age underscore z variable here we use this command the sort value command will sort the records in the data set based on the specified variable the sort can be ascending or descending in this example we want the largest age underscore z values at the top so we sort in descending order by specifying ascending equals false the head command will give the top records stopping after n records if n is given or after 5 records if no n value is given in our case we specify n equals to 15.
