hi everyone welcome to this Channel today we will be talking about the other explainable AI organism namely grid Cam and also some other variants like Greek hand plus plus and scorecam in order to double it down we're going to cover the intuition mathematics as well as the code implementation of this technique and please stay tuned to the end of this video I also want to discuss the purpose of explainable AI and how it can help us all right let's start with a quick recap of our previous topic which is Cam [Music] we're gonna take Luigi
as an example to explain can and this is how it works with gg16 is composed of 13 count layers 5 Max playing followed by three FC layers which stands for fully connected layers therefore the number of layers having trainable parameter is 16 13 count layers and the three FC layers so this is the reason why the model name is vg16 and ug has the this kind of block structure as you can see in this figure each block consists of a sequence of count layers followed by Max pulling and after each pulling layer the size of
the feature map that flows into the block will be reduced by half and the number of channels will be doubled so after one two three or five block five count blocks we get the last official map with the shape of seven by seven by 512. then the three-dimensional feature map the the feature map Matrix will be flattened to a uh 25 000 and 88 dimensional vector after that the vector goes through the three uh fully connected layers until we get we have the final layer which is the classification result so fundamentally this is the overall
architecture of the vgg16 to make it work we have to train the model by fitting one they sent to the model and start the training Loop like forward pass loss calculation back propagation parameters update according to the gradients under a given optimized strategy and so on and so forth right and we repeat the loop over and over again until we get a converged model with let's say maybe 90 of currency so training is done and the model is ready to go uh now I suppose that we want to make the model explainable this is where
it can comes in first thing we need to do is to modify the model's architecture to make it work with scam considering that the very restrictive characteristic of Cam starting by remove the hand from the model and hook up a new Gap plus one FC fully connecting layers so this modification forces us to retrain the model and usually we only retrain the ways here that mapping the Gap and the final classification result and freeze all those weights where parameters of the backbone and after a few Apple golf training once the loss converged we get the
new model ready which is explainable next up we start to explain the model by fitting an input image which has a tabby cat photo in this case I I believe it's a tabby cat I'm not quite sure because I just randomly picked from the internet so please correct me if I'm wrong okay so we fit this particular image into this model and it goes over the count blocks then stops here where it becomes the feature map that we are interested in and uh next since we want to explain the model by figuring out how each
pixel in the input image contributes towards the final result now given that the highly semantic characteristic of this visual map layer I think we can expect it as a a representative of the original image so once we figure out which areas in this feature map contributes the most to the final result by producing a uh a sort of contribution distribution heat map then we can map that heat map back to the original image to to reflect the logically equivalent right and so the question here is when it comes to the 512 layers of feature map
how do we get the ways representing the importance of each layer here we simply take out the 281st group of the ways which becomes a vector with a total of 512 scalars that each corresponding to the contribution to off each layer in this feature map towards the particular prediction score of the class of in this case copycat right oh by the way the index of tabby cat is 281 in imagenet to be specific here wkc refers to W1 281 similarly W2 281 and 2 W 512 281 then we can wait to average the feature Maps
against this tabby cat specific ways to generate the final cam mathematically the class score for an output class C in the cam is given by the equation that YC equals summation over kwkc one over t uh summation where I summation or AIG where wkc is the weight of the uh the the case feature map for Class C and AIG denotes the pixel at location of IJ in the case feature map and IG represent the The Wiz on the head of the feature map respectively and actually here they both they are both seven and Z is
the total number of pixels in uh in one single layer of the feature map which is essentially multiplication by I and J so once we get the cam once we get the heat map we need to resize the heat map by using some sort of bilinear interpolation to the original size then overlay it onto the top of the original image so there we go now we get this visual explanation of the model on the very specific tabby cat photo so yeah this is the overall idea of Cam to recap cam is very helpful in explaining
the predictions for a given image however it involves some disadvantages given that it requires a particular convolutional neural network architecture the first one is the need of structural modification that ends up with a linear structure and also the overhead of learn the ways that help with the final feature map projection and this can be a very limiting factor right the second one is the requirement for The Gap layer after the convolutional layers because it surprised the expressiveness of the model due to the fact that that Gap will automatically shrink the entire feature map to one
single scalar which results in a a a serious information loss right and maybe too restrictive for other complex tasks like for example vqa image captioning object detection semantic segmentation Etc other than the simple classification and uh last but not least is that due to this very limited structure we can only explain the last official map which followed by the required Gap plus FC and nothing else okay now let's see how we can address the disadvantages which was done in a follow-up work called grid Cam that published in ictv of 2017. gray cam stands for gradient
weighted cam it's uh very smart on array proposing Camp using existing gradients in ACN and the motivation of great time was to was like come up with a can-like method that does not restrict the network architectures let's see how how that works here's the overall idea and the the architecture diagram the intuition behind great cam is actually quite similar as can we want to Leverage The spatial information that preserved through the the ZN and to uh to understand which parts of the input image were important to a classification prediction result and so similar to cam
quickham uses the feature Maps produced by the last convolutional layer of ACN here's how it works there's an input image we we have the convolutional part of the network now we take the input and push it through the canvas and we are going to get a convolutional feature Maps here so at this point for cam we would need to do a gap right Global algebra pulling followed by AFC fully connected layer to get the weights but for cam or for sorry for grid cam we can do anything actually for example uh multiple FC that are
commonly being used in classification tasks or any other task a specific hand structures the only requirement is that the layers take Homer after the feature map denoted by a here have to be mathematically differentiable so that we can get the gradient the reason we usually choose the last account layer as the target efficient map is that we assume in this very last feature map there are more crucial information and high level semantic information towards that final classification score also there is a consensus that the deeper the future map goes the Richer the semantic information it
becomes since it's convolution operations are all the way to here the spatial information is also preserved so we can expect the launch to count layers to have the best compromise between high level semantics and detailed spatial information now we get the probability score for each of the target for say a corgi Papi Papi cat tiger cat right Etc similarly gray cam heat map is also a weighted combination of feature Maps against these ways or scores for each of the interested feature map layers now we have to somehow calculate the scores that awaiting each of the
feature map according to the influence that they have in the final output the difference between Cam and grid cam is in how the feature maps are weighted to make the final heat map remember in cam we combine feature Maps using those weights taken from the last fully connected layers of the network well great time doesn't require a particular architecture to obtain the class discriminative localization map for example pick up a target class say YC or ycat which is the prediction of class probability and the great count computes the gradients denoted by a prime here through
the back propagation and in deep learning when we talk about gradients it's basically in the context of optimization process of the loss function and it specifically refers to the partial derivative of loss function with respect to two ways that helps with minimizing the final loss however within the context of great cam it's different here gradient refers to partial derivative of YC which is the square of Class C with respect to the feature Maps so please be aware of that and now we are going to back propagate and enough with a matrix actually a tensor that
has the same size as the target visual maps and that one is being plotted in colors here now what we can do is a global average pooling on each of this back propagated gradient layers we end up with a bunch of scalar now we use this as scores to weight all those feature Maps by multiplying them to the Target efficient map the same way as can to force all the feature maps and project to a single 2D map here gradients information are used to create those weights is based on the assumption that the stronger the
partial derivative the more important on the influence the case channel of the final feature map has on YC however some recent research has shown that it doesn't really work well when locating multiple objects and suffers from the vanishing gradient problem which we will discuss later in this video in a nutshell the activation channels are weighted with the corresponding gradients that representing the importance the importance ways AK Super C is given by AKC equals 1 by Z summation over I is summation over j dou y c by dou a i j k don't worry about the
terms in this equation we will divide into the mathematical intuition later anyway we would expect a feature map with a higher score to have more positive gradients summed up the expression another deviation from cam is the output followed by a relu which is used to cut off any non-positive values the intuition behind using value is that an active values more likely contributes to other classes which are not supposed to be present in the final 70 map the final output of great cam is a hidden map where the hot part corresponds to a region that has
stronger influence towards a particular Class C and the size of the map is same as the size of the convolution feature Maps used to compute this gradient then again the heat map have to be resized or upsumbled to Mac back on the original image to visualize which region was relevant in predicting the class C if there are 1000 possible output classes for any given image where you can announce with 1000 different grid cam here Maps one hit map for each class so fundamentally great cam is a generalization of Cam it's a method that doesn't subtract
to any particular architecture this paper also proposed a variant of graycon namely guided grid cam is a combination of gray Cam's map and guided back propagation attribution to compute the guided curriculum value we are going to perform the hardware product of the attribution from GBP with a map from grid cam combining GBP on grid cam allows us to generate sharp attributions however it's worth imagining that guided back propagation is currently controversial Insanity check someone found this semester has nothing to do with the model parameters and suggests the stick to the original gray cam as it
appears to rely on the model wins and also the training examples now let's derive the equations to mathematically prove why grid can becomes a generalization of Cam previously from cam we have learned that YC which is the class Square in the last layer is given by summation over K wkc one over Z summation over I summation over J AIG is first grade k where summation K is representing the K feature Maps where we assume now it has K visual Maps and then we have the Waze WK superscript c as we saw in chem and we
know that c corresponds to the weight for each class and K correspond to the weight for each activation map of the feed of the target feature map and one Pi Z summation over I summation over j a i j super squared K is going to be the global average pulling of each of the K feature Maps all right this is what we learned through the linear models we implemented in cam now here we go let's assume that F superscript K is given by the turn 1 by Z summation over I summation over J a sub
i j superscript k substituting this capital F K in the above equation done we have YC is going to be summation over k wkc into f k now took the gradient of YC with respect to FK we have dou y z by dou F K is equal to Del YC with respect to dou a i j k divided by dou F K with respect to dou a i j k if we look at this closely the YC by dou FK we can find from about equation again which YC is equal to summation wck into FK
according to product rule for derivatives don't YC by dou F K will be wkc for a particular K whichever f k the global average scalar remember from a particular feature map now let's take a look at the denominator on the right side of the equation but the definition of FK it will turn out to be 1 by Z since 1 by Z is in the denominator to write the new equation more clearly we got an into Z on the numerator here if we sum the terms on both sides over I and J which are all
the pixel locations in each feature map we got the new equation here now the summation over height and weight is denoted by I and J for wck it's just going to be Z which is the total number of pixels in each feature map and similarly the constant Z comes in front here followed by summation over I summation over j dou y c by dou a i j k now we can say wck is given by summation over I summation over j dou y c by dou a i j k this equation tells us something fantastic
isn't it the wck we learned in the cad model are simply the summation of the gradients of each of the class square with respect to every pixel in the feature map and adding them all up which means the class feature ways here are the gradients themselves and in fact we don't need the global average pooling and fully connection layers in the end of the model anymore and we don't need the retraining as the way we did in cam the weights can be simply obtained as gradients of the last class score with respect to any feature
map where we want to compute the resiliency Maps now we are going to have a normalization factor 1 by Z because we've summed the overall pixel locations previously and now we want to divide it across all of the pixels so this becomes the new weight and according to the general practice we denote the partial derivative of YC with regard to aijk as a prime eventually we got to the final silicon map which given by summation over k wkc AK where AK is the different choosing of case activation map because we won't retain only the positive
correlations to be shown on the final silency map so we apply a relu on the weighted combination of the feature Maps essentially where it can just uses the global average of the gradient flowing into the feature maps of the last conclusional layer now I will give you a demo and in this demo we're gonna take Ouija g19 as example to explain how to implement curricum in pi torch so after importing all the required lives let's start with some helper functions initially we load image classes from internet into a list namely classes because we are going
to use pre-trained imagenet weights then let's define three image processing helpers that help with loading pre-processing and transform as well as adding a new batch access by unsqueeze the tensor let's try to load in an image using the helper that's it later on we will push this African elephant photo into the great Comics planer here this is the overall architecture diagram of the OGG Network we want to find the result of the last convolutional layer as the target feature map that we are interested in to generate the cam so here's the layer we will load
the model from torch wish and model Zoo directly as well as the pre-trained weights which are trained on imagenet 1000 classification data set we can easily observe the ogg-19 layers by printing the model in this particular implementation of vgg-19 the model is divided into three major parts namely features average pooling and classifier compared with the architecture diagram above apparently the output of the 35th's relu layer is exactly the one that we are interested in next up let's modify the model a little bit to make it more suitable with great camera once we localize the target
feature map we're gonna break the model apart from this crack that's how the light gets in by this line of code we inject an inspection Point here now we have the capability to pull the gradients and also the activation Maps out of the back propagation flow all the layers priority this crack are defined as features conf and all the subsequent layers are continually finished as Max pooling Plus classifier and one more thing to notice here is that this adaptive average Boolean layer in the original vgg implementation is supposed to allow the system to be used
with any arbitrary input sizes it has no effect in our case because input images have the size of 224 by 224 and consequently the output of this count layer followed by a value is a tensor with the shape of 7x7 so it won't change anything right to reduce the unnecessary computation let's just remove it from the network we know that we can compute the gradients in pi torch by calling the backward method on one tensor however pytorch only catches the gradients of the leaf nodes in the computational graph such as weights bias and other parameters
the gradients of the output with respect to the activations intermediate values will be discarded as soon as the gradient propagates through them on the way back then how to reserve the gradients and pull them out here comes the tricky part there is a callback instrument in pi torch namely hooks hooks can be used in different scenarios this link have another Noble card prepared previously will tell how it works in all the different scenarios I will leave the link Down Below in the description you can check it out later if interested essentially the point is we
need to attach a hook to this intermediate activation map to pull the corresponding ingredients out of the model before they are discarded as well as the activation map by this line in the forward function this is exactly how it works here we register the backward hook to the Target activation map layer with a callback function namely activation hook and in this callback we can preserve the gradients into a cell variable namely gradients then we can access the variable with across the class which is to say we can get gradients and activation Maps through these two
functions defined below now after the model structure modification we Define a inverse function that print out top three classes being predicted by the model now we come to the great cam acquisition function where we input the gradient and get the output of the final cam let's check the code against the figure side by side attacked each line in the code with a number and marked it in the figure accordingly so some lines of code corresponds to seven steps firstly we call the backward function on a particular class Square predicated by the model which is in
this case the one referring to the African elephant class to trigger the back propagate and since we have hooked up a callback to preserve the gradients with respect to this activation map denoted by a now we can fetch gradients and activation Maps respectively by step 20 step 3. in the next step we pull the gradients map simply by a mean function then come to the weighted average part where each layer of this activation map will be multiplied by the gradients scalar that representing the importance corresponding to each of the layers followed by a main function
that reduce the dimension and projected the result to 2D heat map in the second step we simply perform a Max function to achieve an equivalent of value then we normalize the heat map with a magnitude of 1 by dividing the max value among the entire map next up we Define a visualization function called overlay cam which uses opencv for resizing scaling and also color mapping and then overallize the heat map onto the top of the original image finally we come to the explain section starting by instantiate the modified widget model then we load the image
that had been shown about and targeted into the model to perform the forward inference let's take a look at the top three prediction results out of the model this pre-trained model is trained out of the imagenet 1000 classification design so it only covers the 1000 categories that we frequently see in our daily life it's not a dedicated elephant species classification model and from the top three prediction result we found that it can not only distinguish this is an elephant but also distinguish different subspeces I found three pictures online depicting the three substitutions in the prediction
Result One By One African elephant tusker and Indian elephant let's take a closer look at how the model does it from a camp perspective apparently the model pays more attention onto the ears face skin and upper part of front leg now let's inspect another image with multiple African elephants same as the previous one we start with the forward inference followed by cam acquisition and overlay similarly for this elephant on the right side the model mainly focuses on the head and the ears for the other one on the left the wrinkled skin also attracts some attention
from the model in order to verify whether the focus of the model is in the right place I did some research online and found this article about difference between union and the African elephant however the top few criteria doesn't help because we can't really tell habitat diet and the weight from the photos right neither the model then came to the next few judgment factors like ears head shape and skin actually the skin weakness which are highly consistent with the focus of the model it seems that the explanation heat map is making sense however when we
look closely to this sample image I found this approach has some shortcomings as it fails to properly localize other elephants other than the three in the front unfortunately it has been proven that it does have some limitations when there were multiple instances of objects or when there were all closures in the image so here is another image with multiple docks and you can see that gray cam gives such an output where it doesn't seems to capture all the docks in the image how can we address this issue this was done in a work called grid
cam plus plus the main motivation of great time plus plus is observing that great cam took the gradients of the YC with respect to each of the pixels in the activation Maps then I reach all of them to get the final weight as evaluating each pixel equally by taking the average when it gets the score scalar well great time plus plus states that some pixel contribute more towards a class should get more weight than have equal distribution while Computing this weight wkc before we go deeper let's try to understand the intuition of grid cam plus
plus visually so here you notice that if we had an image with three different objects say Docs one on the left occupying a large spatial space another one on the top right occupying a mid-level space and another one on the bottom right occupying a smaller space let's assume that different feature Maps capture different docks so for each feature map let's assume captures each of these objects and you can see here that in Grid cam when the silencer map gradients are computed we can see that the area with the largest space ends up getting most of
the gradient while the gradient towards rest of the pixels are smaller because they have fewer pixels and hence can be built less towards the output well great time plus plus tries to overcome this by doing the pixel wise waiting mathematically grid cam plus plus suggests that while Computing the final weight wkc we are going to give each pixel in each activation map a certain weight as to how they contribute to that 70 map denoted by Alpha sub IJ superscript KC so sub IJ corresponds to each IJ location of a feature map K corresponds to the
case feature map C corresponds to the class which we want to look toward great campus also adds a Reload here to ensure that only positive gradients are considered in the computation of this weight how do we get those Alpha ijkc at each pixel level this learns the equation is actually how we get a ijkc what happens is by reorganizing the gradients and using some arithmetic around the Expressions quickham plus plus shows that Alpha ijkc can actually be obtained as a closed form expression for the second and third derivatives of the gradients that we already have
if you are interested please go ahead and look at the paper for grid cam plus plus to understand how this derivation is done once this derivation is done the rest of it stays very similar to Great cam in Grid cam plot plus we still do a relu at the end of the summation over K wck AK where wsk is given by summation over I summation over J of ijkc relu of dot YC by dou a i j k how does this help see that given the same image of marble dogs quick hand didn't localize all
of them effectively while grid cam plus plus seems to get a better silencer map around the dogs and it works better when there are three dogs and also it captures the the instances when there are structures such as speak relax under occlusion as compared to grid cam here are more examples of great cam plus plus for multiple occurrences of objects improve the performance over great cam although graycon plus improves some grid cam deficiencies I still cannot avoid some problems introduced by the gradient itself firstly we know that the gradient has always been very unreliable and
has a lot of noise in addition the feature map Channel with a larger gradient doesn't necessarily contribute bigger to the class prediction score during the forward pass the activation value also need to be considered lastly there are some non-der available operators in many Network have structures where we cannot calculate those gradients through the back propagation here I want to share another great free method which overcomes these gradient problems named as scorecam in contrast to Great Cam and grid cam plus plus which use the gradient information flowing through the last layer of the convolutional neural network
to represent the importance of each activation map squirkham used the weights of the score obtained for a specific class hence scorecam can get rid of the dependency on the gradient and works as a more General framework as it only requires access to the Class Activation map and output score of the model the pipeline of square time is shown in this figure and it shows all the steps involved to reach the final facility map in order to obtain the class discriminative sentencing map using Square count the process is divided into the following steps the first step
involved is passing images to a CNN model and performing a forward pass after the forward pass the activations are extracted from last convolutional layer in the network each activation obtained from the last layer having shape 1 by m by n is up sampled to the same size as the input image after up sampling the activation Maps they are normalized with each pixel within the range of 0 to 1 to maintain the relative intensities between the pixels the normalization is activated using this formula after the normalization of the activation Maps is completed the highlighted areas of
the activation maps are projected on the input Space by multiplying each normalized activation map with the original input image to obtain a masked image M with shape 3 by W by h to mask the image M does obtained are then passed to a convolutional neural network with some of the max output after getting the scores for each class we extract the score of the target class to represent the importance of the case activation map then we compute the sum across all the activation maps for the linear combination between the target class score and each activation
map this results in a single activation map having the same size as that of the input class finally we apply pixel wise value to the final activation map we apply value as usual because we are interested only in the features that have a positive influence on the class of Interest so fundamentally this is how the Square cam works all right that's all for the great cam another cam variant algorithms we talked to today and at the end of this video I want to talk about myself on explainable AI and how it can help us first
of all apparently we can help the AI models to learn better when we peek into the black box of AI and obtain a certain explanation we can inspect whether the decision making process of the model is biased and is it fair enough logical as cool enough in this way we can start to understand the model improve it trust it and even hand over the decision making authority to AI to achieve the next level automation the second one I want to talk about is machine teaching we've seen a major shift coming in the machine learning industry
with the transition from machine learning that learn by examples to machine teaching that teach by explanations because the explainable AI brings out not only the simple result but also brings its decision making process which makes it more possibly that some professional experience in certain Fields could be effectively and widely shared to more people Let me Give an example which is also a very interesting word that is teaching human knowledge to identify birth species through AI expert model basically the investigation scenario where a trained deep learning model functions as a knowledge provider in helping average people
like you and me to become better domain experts like have some capable in distinguishing between different bird species briefly the authors propose a method that obtains useful Knowledge from a fine-grained visual classification model that can help to improve noises ability the procedure is like first try a model based on the data set labeled by domain experts then we can visualize the features learned by the model in the form of attention map to get people to truly learn the knowledge the author claims that this work could apply to a wide range of end user applications that
require fine-grained recognition outputs the last one I want to share is about exploring many years ago we had realized that AI has the capability to recognize some new patterns and knowledge that humans have never noticed we've heard a lot of fancy stories about using AI to conquer tough problems for example AI protein folding algorithms self-structures faster than ever and offer Goal Zero took just three days to master the Chinese board game of Go purely learned by playing itself millions of times over and over again with no human help it turns out that alphago discovers better
moves that surpass human Intelligence on this specific game however back then the AI was considered as narrow because it can only perform one single task at once such as translating languages or recalled as animals play the vision of General AI becomes more promising in the context of the rapid rise of cutting-edge AI Technologies such as large-scale pre-training multi-modality unsupervised learning and also the fascinating aigc General AI could potentially help human to explore those unknown worlds and many different tasks in the next decade I believe those models like Arthur go will help alongside human as scientists
and medical experts and if we bring the models enough explainability it will help us better at once the frontiers of science and medicine such as discovering new drugs and materials and solving the mysteries of particle physics so these are some of my personal thoughts and Reflections hopefully it helps to give a new perspective on the topic of explainable AI in the next video we will discuss more about great free algorithms which make it easier to explain more different models alright stay tuned and see you soon
