[Music] hello everyone and welcome to this lecture on dimensionality reduction dimensional reduction refers to the idea that whenever we collect some data and it had lots of variables in it lots of columns in it if you will in the form of an structure table then it has been identified that some of them might not be useful for further prediction task that can be carried out using machine learning algorithm so let's say if you have 20 independent variables out of them it might be the case that only five are useful 10 are useful and so on
so dimensionality reduction refers to the idea that we somehow reduce the dimensionality or the number of features which are there in the data by using some technique apart from that we also can merge the information of different uh features together maybe in two three features so what it does it it reduces the complexity of the model the execution of the algorithm becomes faster and these benefits ultimately help us in order to make better predictions let's discuss about this concept of dimensional reduction and related Concepts in detail in this lecture so then in the previous lectures
we we are discussing the pre-processing techniques and within that we have discussed a few uh techniques like uh handling the missing values normalization standardization Etc we also discuss image processing tasks for uh using different pre-processing techniques that can be applied on images such as we can uh do the image rotation we can do the cropping clipping and and and many more similarly there are a few few pre-processing steps that we can apply on text also to make it more suitable to be then fed into the machine learning model today in the context of dimensionality reduction
we will discuss these three related ideas they are feature ranking feature selection and feature extraction they look similar but uh there is there are subtle differences between all these three terms by the way all of them are used in the context of dimensionality reduction so let's start with the importance or significance of the features when we collect data for certain tasks what we do that at the initial stage we collect all the data related to all the features to whatever we have access to okay at that point we do not decide much that this feature
is going to be useful later on for some prediction task or some analysis Tas we do not make that assumption earlier we just consider all the features and we just uh start collecting the data later on we might identify that some of them are completely irrelevant they do not affect much the reasoning or analysis of that data let's say if we have some data then often there is a column let's say a serial number which is 1 2 three or ID maybe and so on obviously it is just there to uh give numbering and ordered
numbering to the data but it does not affect the decision making so this attribute here is an irrelevant attribute it might be the case that some features are less significant or important than some other features so identification of this relative importance of the features might help us in in in better decision making that's why the analysis of individual features is important in many data science related tasks and of course during this kind of analysis and coming up with the relative significance of features can helps us reducing the dimensionality overall dimensionality to uh of the data
so obviously if we have let's say 20 features and we somehow reduce it to the five features only then it will reduce the overall complexity of the algorithm let's take one example to understand uh this concept in a better way let's say we have these two independent variables X1 and X2 and this is some y that we want to predict on the basis of that so this is an example of classification problem we are given the uh input uh independent variables we want to predict what is the output is going to be in terms of
yes and no that's why is a classification task now if you anal analyze this data then you realize that in this specific case X2 that is breakes applied is an irrelevant attribute now obviously why this is the case because you can see that whenever the speed of the vehicle is 80 then the corresponding answer is yes and and whenever the speed is 40 then the corresponding answer is no so as you can see that in this specific case only on the basis of speed we can predict the value of the accident either is going to
be yes or it is going to be no and this column here this feature here breaks applied does not even count in making a better prediction so breaks applied here is an example of irrelevant attribute that can easily be discarded uh from this from this data set without affecting the performance of the algorithm or of the model so this does two things it helps us identify the relative importance of the features as well as we have now reduced the dimensionality of this data also making the overall execution fast now let's talk about these three terms
that I mentioned uh to you earlier feature ranking feature selection and feature extraction now the the name might give you some idea let's start all of them uh start discussing them uh one by one so feature ranking is a process where features are ranked based on certain criteria such as uh their scores in some statistical test or their importance derived from machine learning algorithms there could be various ways but the goal is to identify which features are the more important or relevant to the prediction task that does not necessarily reduce the dimensionality of the data
as it simply orders the features by their importance so maybe if you have let's say five features F1 FS2 F3 F4 and F5 and we apply some uh measure statistical measure of relevance on it we identify that out of them F4 is the most important one then comes F3 then comes F5 then comes fub1 and maybe last F2 so this is the ranking of the feature this stands at rank one rank two rank three rank four and rank five so knowing that relative importance of the features and arranging them accordingly in in in in according
to their ranks is called feature ranking it once again it does not reduce the uh the the dimensionality of the data on the other hand we have the concept of feature selection which refers to the idea that we are now looking for a subset of relevant features out of the total number of features that can be used in machine learning task what it does is it reduces the dimensionality of the data by removing irrelevant or less important features which can help in improving even the model performance so there are many techniques for the same but
famously in feature selection we have broader categories like filter methods wrapper methods embedded methods and so on so let's say in feature selection you have total five features and you want to reduce the number of features to three so one of the approach that we can do is we can apply some stat statistical measure to know the relevance of all five Mees as we did in the case of feature ranking and let's say we get this ordering like this since we want to keep only three features so what we can do is we can discard
fub1 and FS2 because they come l in the relevance of the features accordingly we will keep only F4 F3 and F5 whenever we calculate the scores related to feature ranking or even feature selection they are they usually give you some numeric value in order to tell its importance so if the values are usually greater as we'll see later on then they are considered better and the values are smaller then it means they are not so useful so if the few last few features have very little relevance which can be identified in terms of the score
that they have got we can simply remove them and discard them that's why the name feature selection that out of in this example out of five we are finally selecting only three features finally feature extraction feature extraction is a process of transforming the original data into A reduced set of features this transformation is conducted in such a way that the new features capture the most significant information of the original features this often involves the creation of new features by combining the original features techniques for feature extraction include famously principal component analysis or PCA is a
famous technique we also have this linear discriminant analysis LDA and there are some others Sol so let's say what do we mean by feature extraction with the help of an example let's say once again I have sorry I have five features I want to let's say reduce it to two so feature extraction could be that I design a new feature NF1 which is the combination of original feature 1 3 and five and maybe I design a new feature which is the combination of let's say F2 F4 and fub1 so if you focus on this you
will identify that now out of five features I have only two features new features NF1 and nf2 but they carry the information of all the features because we have included F1 F3 and F5 F5 to design the First new feature and F2 F4 and F1 to design the uh the new the second new feature so in this way this process is by the way called feature extraction where we do not simply discard a few features we take the information from them all the features and we construct new features out of the original features so let's
go them one by one feature ranking means assigning ranking to the features feature selection means selecting a few uh most relevant features out of the total number of features and feature extraction refers to the idea that you have been given lots of features you combine their information and derive new features now let's discuss each one of them one by one with the help of example for feature ranking there are uh many algorithms one famous is calculating the cor correlation between the feature and the tarbit variable let's say my data or or some part of the
data looks like this where I have five features and finally a Target is there which I want to predict what I will do is I will calculate the correlation uh to be specific pearon correlation between feature one and Target feature two and Target feature three and Target feature four and Target and feature five with my target higher correlation will tell that there is a strong relationship between the Target and that fature so depending upon the correlation value I will keep that feature for example if in this specific case I calculate the correlation with Target for
each feature it turns out that feature one has the highest level of correlation that's why I'm assigning it the value or rank rank one feature five is a second uh highest correlation with the target that's why rank two two feature four third and fifth like this so on the basis of the correlation with the target it which suggests that a specific feature has a stronger or lesser correlation or dependence with the target we can assign values or ranks to these features in the case of feature selection let's once again we calculate the correlation it turns
out that this is the correlation so feature one is rank one 2 3 4 and five and I want to keep only the three most relevant features then I will consider feature 1 feature 5 feature 4 as the most relevant features and we'll consider only these three for further prediction or further analysis uh in in the machine learning task so here we are since reducing or selecting only a few features that's why we call it feature selection then we have uh for feature extraction this very famous algorithm called principal component analysis uh what it does
is it it it takes uh your data uh with features in it and it generates the new unrelated uh combinations of the existing features we call them principal components so it involves mathematical operations to be applied on your data to calculate these principal components the number of principal components is usually less than or equal to the number of original features that are there in our data and there is ordering within the principal components as well so the first principal component has the largest possible variance which means that it contains the maximum information that was there
in the original features and the second principal component is is second in that order and that come then comes third and fourth and like so on we call them principal components and they are it is important to note that they are orthogonal to each other that is in in a high dimensional space they are they are at 90° to each other now calculating or applying PCA is is a mathematically uh uh extensive task is it requires first applying standardization on the data then Computing The co-variance Matrix then calculating the igen values and igen vectors igen
vectors are by the way the principal component and uh principal components and then we sort them and select the relevant principal components but implementing PCA with the help of Inu libraries is very straightforward so maybe if you supply this kind of data to PCA uh with five features it can give us only two uh features which are made up of these original five features so pc1 which represent the principal component analysis one has some has some portion of feature one feature two feature three and so on similarly PC2 also has some combination of all these
features combined so reducing the dimensionality help us in many ways it might improve the performance and it at least reduces the running time of the algorithm or the machine learning task and it as I told you affects the performance and this kind of dimensional deduction using uh PCA using some feature selection is very common in machine learning task now we will see how we can Implement some of them using python libraries let's try to implement feature ranking feature selection and feature extraction using inbuilt python libraries as you can see on your screen uh we will
first try to implement feature ranking for that first I have imported the necessary libraries which are pandas and napai let me create some random data using the random function available in the numpy array here I am creating six columns six columns with the name feature 1 2 3 4 five and finally the target variable and I'm creating 100 such instances and storing that into my data frame DF if you look at the data frame it contains six columns five features and one Target and it has overall 100 rows now what we'll do is we will
calculate the correlation among them using thecr function what this does is it gives you the correlation Matrix where the correlation between each feature with every other feature is mentioned out of them we will keep only that those values which are related with the correlation of each feature with the Target only when we do that we get the feature correlation variable we can finally arrange that into the form of a table uh by applying the Sorting operation and finally when we print the table you will see that feature two has the highest correlation with the target
with a value 0.08 feature 3 has 0.06 and so on so on the basis of that we can say that feature 2 has the high highest ranking feature three is is next uh after feature two then comes feature four then comes feature five and so on you might also notice that the the value of correlation is not very high because data we have generated using a random function if there is some relation between the target variable and this feature we might get a better value of correlation which might give a better uh understanding of the
relative difference uh between these features but nonetheless this is how we can do the feature ranking now if you want to apply feature selection to it and we are interested only in keeping let's say the top three uh features that of course in this specific case that is going to be feature two feature three and feature 4 however there are more sophisticated uh feature selection algorithms especially in the subcategory of rapper methods which extensively try to find the best feature subset that can give you the best performance for a predictive task now let's move on
to the feature extraction where we will try to implement PCA or principal component analysis here also I'm generating some data randomly and there are total 10 rowes and number of features are five for PCA often we first standardize the data with mean zero and variance one and then we apply PCA as you can see that as the argument we are uh we are forcing to take only the top two principal components which are higher in ranking we can also mention some other number of principal components as well so using PCA we will fit our data
the scale data which has we which we uh which we have derived after implementing the standardization process and apply PC name now for better representation we are assigning them the column name pc1 and PC2 and finally we can display those so out of the five features having uh different values we have extracted their information and arranged them only into the form of two columns pc1 and PC2 so this is how we can apply the feature ranking feature selection and feature extraction for Dimension reduction so this is all about this lecture thank you [Music]