recording um okay it's 7 10 it's probably a good time to start so today we will be talking about something called diffusion models and this is going to be the last lecture in the series of image generation uh lectures that we have seen in the last few weeks and hopefully this will cap out everything that you've learned so far in a nice way uh this is a quick agenda today um I just want to give a disclaimer that this lecture is more mathematically involved than the previous lectures I just want to make sure that these
details you know you don't need to know any of them I'm gonna also mention those because I feel like if I don't I'm going to be doing a big disservice to the fusion models but yeah if you get lost in the weeds that's fine this is understandable this math is not easy to understand I would I'll say that myself yeah in that case I guess we can get started so there are many different kinds of generative models out there and we have seen two of these classes uh in this class so far we have seen
variational Auto encoders or vaes and we spent the last two lectures on Gans there are other models too like Auto aggressive models energy-based models we won't talk about those in this course but if you want to learn more about them I think you should check cs294158 it's taught by Dr Peter reveal they'll that course will go over this um these kind of models what we will do however is we are going to talk about this new class of models called dinizing diffusion models so I'm going to start off with some theory behind diffusion and some
of the math that goes into it before I delve into that let's talk about how uh density let's talk about how you synthesize data in the first place um in machine learning we sort of make this assumption that your data comes from a distribution can be any kind of data can be image data can be text Data audio anything else we assume that it it has like an underlying distribution and your data set is basically samples from the distribution the goal of generating models is to basically learn the distribution so you could sample from it
and basically get another data point that looks like it came from the training data set does that sort of intuition make sense yeah and if you if you look at if you look at the image to the uh to the right on the top we have this like print data set and we can kind of see that it comes from a distribution that if you follow along my cursor kind of looks something like this and what the model does is then it learns to like sample from this distribution and it generates this like test set
and you can kind of see that it looks very similar to the Affinity computer cleaning data set which means that it must have learned what the distribution actually is now this distribution is not known to us which is why we have to learn it from the actual like data samples themselves and that is what like all of the models that you've seen so far have been doing in a sense now how do you learn distributions this is kind of a complicated question to ask um there are two main methods to do it you your distribution
is is going to be a probabilistic nature right so you could Define something called a likelihood for the distribution so you can say that okay if I have a sample if this likelihood value for the sample is going to be high then there is a good chance that that sample came from the distribution but if it's going to be low this means that that sample probably did not come from the distribution what we can do is that we can try to increase or maximize the likelihood of the of of the of samples that this model
generates that's one way to do it the other way to do it is to minimize some sort of Divergence metric between your data distribution and the distribution that the model learns what this diversion says there are many ways to there are many Divergence metrics um I'll get to some in a few in a few minutes but that's sort of how generative modeling works and in a sense we learned about these like two kinds of models we learned about vaes we learned about Gans and even though it might not be obvious those clusters or models are
exactly doing what I just described so the vae loss looks something like this complicated expression up there but what really is happening is that this x this loss is basically a Bound for the likelihood so if you minimize the loss you're in a sense increasing the likelihood and that's why vaes work similarly for against we have this kind of adversarial loss function which looks like a complicated mess but what's really happening is that when you're trying to optimize again but when you're trying to optimize the subjective you're essentially Trying to minimize something called the DJs
between the data distribution for the of the training set and the distribution that that generator learns this DJs is called the Jensen Shannon diversions it's basically a Divergence metric that says that okay if this value is low then the distributions that we are looking at are fairly similar but this value is high then they're very dissimilar any questions about this so far okay so once you have learned these distributions that sort of mimic the data distribution how do you generate a new sample one way that pretty much all pretty much all models do it is
you look at a sample from random noise and you convert that sample to something that looks like it might have come from the data distribution if you remember again you should you took a look at some like latent noise Vector you pass it through the generator and and I'll pop an image right that's how guns work and it's exactly it's exactly how vais work as well and in in both of these cases what's really happening is that you pass this noise into this like model and you get an image out so you're taking a single
step from your noise Vector to the actual image the single step might not be easy to analyze it's it's like you have a whole like neural network in between you don't really know what's happening in there so what diffusion models do is instead of taking one big step they take multiple Tiny Steps because these are actually easier to analyze and but what's really happening is that these tiny steps are basically a Markov chain have you heard the term Markov chain before yeah so yeah this is basically the motivation for why diffusion models work you don't
need to sort of notice the the main idea is that these models were inspired from non-equilibrium stat Mac or statistical mechanics yeah the slide I just wanted to like put the inspiration out there if you don't even know what that is now a diffusion model has two parts there is something called the forward process which is you take an image and you add noise to it and you can you take that noised image and you add some more noise to it and you keep repeating this process so your image can be some data point say
x0 you add noise to it you get X1 you add noise level you get x2 all the way up to some XC you can you can repeat this process for three time steps and the noise that you add can be parameterized by the this gaussian distribution given over here you define these values called beta for each time step the sort of like the set of values is called a noise schedule because it controls how much noise you're adding at each time step it follows a particular schedule and well what happens is that if you keep
adding noise to that image you can actually kind of show mathematically that what you end up with at the very end is basically Pure Noise um this is a mathematical trick that was proposed in the original diffusion paper it shows that okay um I know that my trying stuff I know that the image that time step T condition on the image at times of P minus one is going to be a noisy version of T minus one and you can sort of Express that in like this particular wave again don't worry if you don't know
what that means but what this trick is basically showing is that you can express X of T in terms of X of Q minus one but you could recursively repeat this process and you can actually directly Define X of K in terms of x sub zero so you can directly go from the initial image to a noisy version at some given time sub key and this is a very cool track that will simplify some of the mathematical um I guess like results that you will see later so uh I said that there are two parts
here the fusion model this is the second part it's called the reverse process so we just trigger an image and basically made it into Pure Noise that isn't really helpful right we want to generate a new image and in the vae in game lecture we saw that we could generate an image out of noise and this is what the reverse process is trying to do so what it's saying is that okay if I look at my sequence of like noising Steps in the forward process what if I can find a way to reverse that so
I I start with noise if I move to a slightly less noisy version of the image and I keep repeating that until I get back to the original image the goal of a diffusion model is to basically learn how to do this process in reverse any questions about the two main steps of the fusion model okay so when we were moving so if you look at the whole process which went from x0 to x 1 to X of 2 to X of T minus 1 to X of T what we were saying is that we
know that the conditional distribution of X of T condition on X of T minus 1 is the gaussian right but we don't know what the reverse distribution is like we don't know what the distribution of X of T minus 1 conditional on X of T is and that's going to be very hard to find because it turns out that if you apply something called base rule to this like set of distributions you need like these set of terms to Define this like reverse distribution and these set of terms are very hard to compute because you
need to know what the exact like distribution looks like and that's not possible to like it's like it's intractable to compute you can't really compute it in like polynomial time for example you have to evaluate this like very nice integral which is not possible so what you do is you try to approximate what the distribution looks like okay any questions so far yeah yeah the Q function is going to be the distribution of X of T conditioned on X of T minus one so it's the distribution that defines what what x of T looks like
yeah yeah and key of X of T minus 1 given X of K is basically the reverse distribution we don't know that so we try to approximate it using this P function which is basically what the neural network does which because it's parameterized by this Theta right so and before we Define our neural network we need to sort of Define what the distribution should even look like because if you don't know what the distribution is then we can fill it to 4 in a neural network um and a I guess like proper manner so turns
out that okay if I were to look at my forward process and I look at say some step X5 if I look at the distribution of say X4 given X5 or maybe X3 given X5 or maybe X2 given X5 it's like this is what the this is what the distributions look like um turns out that these distributions are very hard to sort of compute because they look very weird like I don't know what this distribution is in the red right this is some like random distribution that I can't really parameterize similarly it's the same if
you were to find the distribution of X1 condition on X5 all the way up to MX3 but wait if I look at the distribution of X4 which is right before X5 if I explain to my eyes enough I can see that this kinda looks like a gaussian right does that pretty agree that this looks like a gaussian Down Below so what you can do is we can say that okay I can just make a guess and say that okay my reverse distribution should also be a gaussian uh there actually it turns out that our guest
is indeed correct and the reason behind that is it goes into this like very deep theory behind stochastic differential equations I don't want to get into that because I haven't taken a question like myself and I don't completely understand it myself either but uh but there is like a justification for why we can make that assumption as to why the reverse distribution is also a gaussian now a gaussian is parameterized by two terms you need to have a mean and a variance what diffusion does is it tries to learn this mean and variance through a
neural network any questions so far cool now how do we train this we need to have a loss function right we need something that we can minimize or maximize um so it turns out that we can use the same trick that you might have seen in the VA lecture is that okay I can't really what we're trying to do is we're trying to maximize the likelihood that an image that we generate looks like it comes from the original data distribution but that likelihood cannot be calculated but in the VA lecture we can see that okay
we can bound the likelihood using something called the variation variational lower bound we basically apply the same trick to diffusion the the uh apply this bound to the actual likelihood and it turns out that you end up getting a loss function out of this which sort of looks like a sum of like multiple smaller loss functions like L of T all the way up to your power of zero and those loss functions are going to be given by these terms down below now typically we ignore the term L of T because it's almost always zero
doesn't really contribute to the loss it turns out that the main terms that do contribute to the loss are L of zero all the way up to all of T minus one in this lecture we are going to sort of ignore what lf0 does because it's a bit complicated and I don't want to get into the reads what we will do instead is we will analyze what these like middle terms look like especially this L of little T which is defined from 1 to T minus one any questions yeah again the little T is going
to be a time step and the uppercase t is going to be how many steps we're on the diffusion process four it's going to be say a horizon of that term sounds uh if that if you have heard of that term before so if you look at this term over here this is what we are trying to sort of analyze we are looking at the kale diversions okay you don't even know what the kill diversion says it's basically another Divergence metric that says that okay if this term is low then the distributions here of x
t condition on X of t plus 1 and X of zero is going to be very similar to the distribution that are neural network lines but if this term is high then those are going to be distributed our goal is to sort of drive this term as low as possible we are looking at the kill diversions between this distribution over here and this P Theta over here it turns out that this distribution Q of X of T minus 1 condition on both X of T and X naught has a very explicit form which I have
written um on the top over here I'm not going to go into the derivation behind why this form why this distribution has this form I have linked some papers at the very end if you can check this out if you want to if you're curious about that and what ends up happening is it turns out that if you try to look at the KL diversions between this distribution over here and the distribution that your neural network is trying to predict you end up getting this term down below what what really ends up happening is you're
trying to minimize the L2 loss between the mean of your learned distribution and the mean of the conditional distribution that is given by this expression over here by utility of mu turns out that you can do some more algebra and you can re-express this like whole term where my cursor is pointing at um utility of mu t as this expression down below in in the red box those two expressions are the exact same again the derivation is going to be in some some papers that I linked at the very end so what this objective is
saying is that we want our learned mean to predict what's in the red box because we're trying to minimize this L2 loss right so what the authors of the diffusion papers say is okay what if we What If instead of trying to predict everything inside the red box we try to predict a single term instead because this whole expression looks very complicated right so let's see that okay we know that our totality of mu has this form over here what if we take inspiration from that and parameterize mu of theta which is the learn mu
in a very similar form but what we will do is we will make this very um just like noise term at the end parameters instead so what what this equation at the very top is saying that okay I can write my learn mu as this equation and the learn part comes from this Epsilon if you make those substitutions into the loss function and you do some more algebra what really ends up happening is you're trying to predict the noise that was added at time step 0 to get to time step key you're trying to predict
the noise that was added and what this and what this means is that if you if you know what the noise is you can take a noisy image subtract that noise and get the original image back but if you were to say start with Pure Noise and you subtract some other noise from it it's possible that you might get a completely new image back instead so this is the main intuition behind diffusion models any questions that before I move on I just want to make sure that this part is kind of important so make sure
the intuition is clear again doesn't really matter if you understand the math or not the math is very complex um don't get lost in the Beats I just want to make sure that you get the high level details so this is the objective that we are trying to minimize over here and in the at the very last line and it has like this complicated mess of terms at the very beginning turns out the authors say that okay just so you can drop all of that it's kind of like extra baggage we don't need that what
you can do is instead minimize this very simplified expression instead so instead of looking at all of those like different weights you just focus on minimizing the noise between the noise that was added to the uh to an image during the forward process versus the noise that you're trying to predict so you could denoise that image now this does mean that you don't really have that same lower bound that you started with anymore because it's going to be a weighted version of that doorbound but it turns out to be fine and it actually actually it
actually increases the quality of images that you see um later on so this is what the training process looks like you start with some image in your data set you add noise to it and using the trick that we defined earlier you can instead of like adding iteratively during the forward process to go from x0 to some X of T you can go to x of T in one step directly you pass that noise image into your model you predict the noise that was added and you try to minimize that with the actual and you
try to minimize self the distance between that and the actual noise and he just repeat this process up to the model has sort of converged this is like a pretty standard training Loop that you might have seen in quite Rush before and once the model has trained the way you can synthesize new images is you can start with random noise you could pass just like completely noisy image into your model protect some noise from that and you can subtract this predicted noise from the actual noisy image in hopes of getting a denoised image back instead
but and and if you recall all the way back here we were going to do this process in Reverse right we're going to start with Pure Noise find a slightly less uh noisy version of that and keep repeating that until we get something that looks like an actual image and that is now possible that our model that can predict this noise which is what this algorithm is doing any questions about that yay okay so yeah that is it for the theory of diffusion um again there are some derivations that I did not go over but
those aren't going to be included in some blog posts and papers that I linked at the very end I would encourage you guys to check it out again just like a final call any questions about how diffusion Works before I move on to how it's actually performed practically boom so what we have discussed so far is how diffusion Works in general um and turns out that this x sub T we were we were pretending that this x of T was going to be some image it doesn't have to be an image it can be anything
else as well what we will see now is what happens is that X of T is indeed an image and this is sort of what the diffusion process looks like you in the top half you see an image that is being um noised up in the forward process you keep adding noise to it until you get like pure stochastic ones at the very end and in the bottom half you see sort of the reverse process you start with Pure Noise But as time goes on you get an actual image out of that hopefully these two
sort of um videos give you some more intuition as to what's happening in diffusion any questions about this oh yeah I'm curiously with each of the uh lines oh right so this is so the image is going to be like some very high dimensional um object right because this can like 256 by 56 by three the plot on the right is basically showing what's happening if you were to look at say a single dimensional object for example so what what's what this is what this is really showing is that um on the right on the
left hand side we have the actual like distribution of the data and we take some sample from it which is like going to be some starting point on the very left and we keep adding noise to this until we get something that looks like a pure gaussian which is what sort of this like gradient of colors look like at the very end and the picture at the bottom is basically doing the opposite it's saying that okay if I were to start with a gaussian if I learned some like diffusion reverse process I couldn't map back
to something that looks like my original data distribution because you can kind of see that the left-hand points of the left endpoint of the top image is very similar to the right endpoint of the bottom image which kind of shows that this diffusion process was successfully able to learn the original data distribution okay so we know that we are going to predicting noise and this noise has to be added to the image which means that the noise must have the same dimensions as the image what is one model that has the same dimensions um as
the input and the output and you saw this during the segmentation lecture right you know that if you pass an image into a unit you get some segmentation mask out insert you get some segmentation mask out which you can use to sort of like label different pixels into like different classes but the authors say that okay I can also use this model to like predict noise instead because this is just a model the output purely depends on what objective we are trying to well what what objective we are minimizing the model with respect to right
so yeah it turns out that with some with the objective that we showed earlier which was a noise predictive objective you can use this unit model to like output noise and this is not really the same Unit Model that you saw earlier it turns out that they added a whole bunch of stuff to it they replaced um your normal unit blocks with resnet blocks convex blocks they had like attention modules okay don't worry if you don't know what attention is I think that's going to be covered next week they replaced um batch Norm with something
called grip Norm they replace values with other activations called swish Engineers these are basically activations that look like radio but have mice of properties they basically took like all the modern CV tricks that you might see in different papers and they threw it all into this like one model and it actually works very well so I guess they were right um that's a good question it actually so okay the original paper use a thousand time stops turns out that using more transps is better so there was an improved paper that repeated this process for four
thousand five substance said and they were able to get like better quality and there have been improvements ever since to sort of reduce the number of time steps you need I think modern diffusion models take Maybe 250. something around that um there are other tricks to make it even faster there is a model called ddim I'm not going to cover that in this lecture I think you can that makes it so that you only need to run this block like say 25 frames instead yeah um why okay I wanted to escape a presentation yeah so
I actually have what this model looks like pulled up um here so this is the normal unit but they actually throw in like this whole bunch of stuff to it so they add like these residual blocks which you might have seen on the resnet homework they add like these attention blocks again don't worry if you don't know what attention is that will be covered next week um yeah um I I have linked the slide in the at the very end if you want to like take a look if you want to have a deeper look
at the architecture that uh and how it's implemented inside torch okay so that is how image generation and using diffusion models work uh and it actually gives like pretty good results and there have since been some tricks to improve the results even further so remember those beta parameters that we defined at the very beginning those controls how much noise you add in the forward process and and in a sense it also sort of affects the reverse process because the reverse process is trying to reverse for the process right so um researchers said that okay if
I were to say use a linear schedule which means I increase these like beta terms um linearly I get a forward process that looks like this top row over here and it converts my image to noise really quickly which makes it harder for my reverse process to sort of go the other way so what the instead hypothesized is that okay what if I use a different function maybe something like a cosine which is um I guess it looks something like this and at the very top it has a very practical change it doesn't change that
quickly right so saying that okay what if I change slowly at the beginning um a sort of like ramp up later on and the idea is that you convert image images to like noise slowly you your destroying information at a slower rate and this should help your model learn the reverse process better and it actually does improve uh sort of the results uh I just want to emphasize that this choice of cosine was like completely arbitrary you could also use some other function it's just something that happened to workflow yeah this is all right um
it's they do some weird tricks to make this thing work so it turns out that you apply the the linear schedule you require schedule to like these like beta turns the apply the cosine schedule to this like malfunction that I sort of um glossed over at the very beginning so they Define Alphas like when minus beta which is why you sort of go ahead and something like that okay oh I just realized that this bar was floating over this entire drawing okay so when we defined our distribution for the reverse process we said that okay
it's going to be some gaussian with some mean and covariance what we we kind of neglected the covariance part until now so the original diffusion paper used to fix covariance just so that okay I'm going to say my covariance is equal to some Sigma Square Times the other the identity Matrix where Sigma squared is either the beta term or the modified version of that and the idea behind that was um my covariance Matrix is going to be very small it's not going to have a big impact on my model anyways and the mean is going
to contribute more significantly uh it's going to be it's going to contribute more significantly than the covariance so it should be fine to go ahead with something like that and that actually works kind of well but there was a second paper that came after the original DDP and paper that said that okay you can that's a completely clear statement to make but if you try to learn the covariance it's possible that you might end up improving your images even more and what they found is that it doesn't really affect the image quality but it can
help you improve your log likelihood now my understanding is that people in this field use log likelihood as a measure of how much of the initial distribution you were able to learn so if you have a lower if you have a higher log likelihood this means that you are learning more of the initial data distribution which means that you could um sample more and more images from it so in a sense this log likelier term can be viewed as a measure of diversity right so uh the authors of the second paper said that okay if
I parameterize my learned covariance Matrix um in this way where V is a term that I'm going to predict with a neural network I could get better likelihoods and that's one Improvement they made any questions about the the noise schedule or The covariance Matrix right uh yeah not really uh well the covariance Matrix of some like random Vector is going to be given by uh or rather uh it's gonna be like it's it's it's it's a big Matrix but it turns out to be fine because there are some tricks you can do with covariance matrices
and like numerical linear algebra it's sort of like non-deep learning related but yeah weird magic basically yeah so yeah it's not that big of videos they also propose some more architectural improvements um uh I put those up there I don't think that these are that important because architecture of this unit model like changes cut off every week if different papers use they all use some unit but like different variations of it so these are some changes that they found helpful in like this one particular paper so that okay make the model bigger which is kind
of obvious um they were using something called attention modules which I sort of preferenced earlier this is that okay add more number of heads again if you don't know what that means you will learn about attention next week um this is that okay I'm using these like researchable blocks and stuff in my unit and I know that there's this other Gan model called Big gang which works really well what if I steal some parts of the architecture from there and that also improves the quality even more they also something called adaptive to primalization my understanding
is that your covert adaptive incidence normalization last week this is basically A variation of that in a sense but applied kicked up normulation group normalization instead um again don't Focus too much on these improvements because like I said the unit model kind of changes from paper to paper like I I don't even think like these changes are used in like something like imagine or Dolly tube something like that oh and this is a big one so recall when you covered Gans you said that okay I can condition on a class label and generate an image
that sort of pertains to that label because earlier games were just like spreading out stuff right there was like no control over what you sort of generated but you say that okay if I condition on some label instead I could guide the Gan to sort of generate a very particular kind of image foreign you can sort of apply that same idea to diffusion so they call it classified guidance and the way it works is very very Jank they train a classifier to predict the class label from the images in your data set but instead of
just going with the original images they also feed in the noisy version of the images and try to predict a label from that and once this glassware has been trained what they do is uh you think the gradient of that of that classifier and they added to the result of your diffusion model and they do some like weird sampling thing with that gradient it's kind of highlighted in the step down below they take the gradient of the classifier they sort of multiplied with some other terms and they added to the um mean that you would
get from the neural network and as and the reason as to why this works is going to be highlighted in another blog post that I've linked at the very end it is the the justification for this is kind of mathematically heavy uh I don't want to get into that in this lecture but it turns out that this improved results by a lot so uh so when you have like generative models you need a way to measure how good a model is relative to other models and there are some metrics that you can use um one
metric that is very popular among folks in this field is something called the FID or the um fresher Inception distance I think it's called yeah and there's also like a version of that called the sfid which is just spatial FID um basically what this what these metrics measure is how good damage quality is and turns out that the lower your metric is the better your model is uh this sort of metric was designed to sort of imitate how humans would sort of Judge images and it's kind of widely accepted by the community so yeah people
people trust us a lot um there are other metrics called the Precision and recall what Precision measures is the image Fidelity it tries to measure how close the your general generated images are to something from the data distribution and recall measures the image diversities like are you sample are you generating a very diverse group of images are you trying to like generate like one thing every time and you can see that with these improvements that I mentioned earlier along with classifier guidance diffusion was able to sort of beat Gans and pretty much like all across
the board diffusion is getting better FID than Gan models or Transformer models for generating images and I think this these two images will sort of like Drive the point home so this is from the paper called diffusion models Beats beat Gans at image modeling so on the very left you have images from a big gang which was sort of the state of the art until this point in the middle you have images from the diffusion model and on the left you have the training set and okay these two images sort of like these two classes
of images are pretty good you don't see like an noticeable difference between these two yet but something that you might notice is that the images from the Gan look very similar if you look at all like the flamingos down here it looks like sort of a varied version of the same image whereas you see more diversity in the diffusion models so this is sort of showing that diffusion models can get like can capture better Fidelity and better diversity of your data of your data distribution now there are some more assistive images again this is sort
of the same issue with diversity and philology as before but you can kind of see if you look at all of this like images of people holding fishes the begin model is kind of struggling that say Ryan got faces you can see like just like demonic phase over here but like in all of in pretty much all of these images whereas diffusion is much uh is able to capture sort of like some of this like finer detail uh in a much better way so this was basically the point when diffusion models took over against this
was I think this paper came out 2021 last year and yeah ever since the diffusion models have sort of been dominant in this field so earlier I sort of highlighted this method called plus bar guidance which was able to sort of steer diffusion models to sort of like this dominant path and the way that word was taking was by looking at the gradients of a pre-trained classified model or rather not a pre-trained but a model that you trained yourself and I was looking at the gradients of those however that is a very Jank process because
you don't know what's sort of like going on under the hood there it's hard to sort of interpret why this sort of technique worked well and you can't really use a pre-trained classifier because this classifier must also work on noisy images so you must train your own and to sort of like avoid these sort of dilemmas researchers came up with a better way to sort of guide your division model and they call that classifier free guidance what how this model works is you train something called a conditional diffusion model so like the way you had
traditional gains you could feed in the class directly into the diffusion process itself instead of using like an external classifier and while they were using this model to sort of understand the behavior of how guidance work they accidentally ended up improving the results even more so uh these are sort of like some hyper parameters that they tried and they were able to notice that okay for a certain type of parameters the results were even better than older diffusion models and like other Gan models and this is sort of the way different models work right now
people use classify free guidance all the time I just wanted you to be aware that this thing sort of exists but one thing about guidance is that it can sort of trade diversity for increased quality because when you condition your model on some class label you're inherently telling the model to predict certain kinds of images which means that you're kind of automatically decreasing the kinds of images that you can produce you're decreasing the diversity of all the images that you can produce and this is sort of more evident in this picture so in the left
column is you have a diffusion model that is predicting that is sort of like generating images without any guidance but on the right column you have a guided process now you can see that the guided images are slightly like there are definitely like better quality but there's also like less diversity in them now the question of diversity versus quality is sort of my open question like some people might say that okay more diversity is better some people might say that better quality at the expense of diversity is better it's kind of up to the uh
the human protection human practitioner to sort of decide on what they consider to be more important okay now there's another class of diffusion models called latent diffusion models so the idea is that you were training this diffusion model on images until now and turns out that these images are like very high dimensional which is going to be a problem because this is going to make the whole process like very confusionally expensive turns out that training a huge Fusion model can take can easily take uh days to maybe even months on like gpus which is like
not acceptable right in certain conditions another thing about diffusion models or I guess like images is that if you were to say store an image and if you would when you're like storing an image you can convert that image into say binary and what research has discovered is that more bits were being used for sort of capturing the pixel level details and less bits were being used to capture some of the semantic details in an image so like if you had an image of a dog there were less there's dedicated to sort of describing what
the picture contained and more bits just describing something like I don't know like the pause or the four or something like that but it turns out that when you're trying to run a generative model and this is not exactly just for diffusion I think this sort of goes for all kinds of generative models like even vaes or Gans or any kind of model that you may see is that you want to run a a generative model on the semantic part of the image instead because that is what you're really trying to sort of generate right
so the researchers came up with a way to sort of make that happen they said that okay instead of trying to predict instead of trying to run this model on an image directly what I can do is I could compress this model to a latent space run diffusion on the latent space instead of like up sample that latent back up to an image so the way this works is they they try to learn latent space first um and they sort of use this like very complex loss up there what this is really doing is this
looks like again because it's using a combination of adversarial losses which sort of promote um reconstruction quality and it's also trying to use like some other kinds of losses and other terms this sort of learns like some meaningful latent space again um I would recommend reading the LDN paper to sort of get an intuition as to why this loss is defined this way but it turns it turned out to be sort of the best one that they could use and once you have just like latent space learn you can convert from an image to the
latent and back to the image using an encoder and decoder model which is what like this D and E represent and once this space has been learned they run diffusion on the latency themselves and this is sort of what that process looks like you take an image you run it through the encoder you get some latent you run the forward process on it and now you run the reverse process on the latent itself and once that is done you can pass it through a decoder to get an image output back um I wanted to throw
this model out here because this is how stable diffusion Works have you guys heard of stable diffusion yeah this is exactly how the model works you can also like add other things to it like you can try to feed in say you can try to condition on things like semantic labels text other things breaks etc etc and this model can sort of like handle all of that okay so that is what I had for today's like diffusion lecture we can spend the rest of the time looking at pretty images so there have been a lot
of progress and diffusion in the past two years and I want to highlight some of the models that people have that Labs have come up with so uh this one is called Dolly chew this was of the for this was one of the first like texture image models that sort of um highlighted how powerful diffusion can be at uh the sort of uh modeling so you can see that okay if I were to like feed in some like random text I could get very like high quality images back um in fact um I have sort
of linked I'm gonna go through like multiple models and I have linked the websites for all those models in the slideshow so you can like check those out on your own time as well so in response to Dolly Q which was developed by open AI Google came up with their own model called imagine which I think it's a pretty clever name because you have image gen imagine Right image generation I think that's kind of cute so um they were able to get like better results than Dolly too because Dolly wasn't exactly like a fully diffusion
model I think it has some other things thrown around in it as well but imagine was a pure diffusion model it was only diffusion um okay so after this model came out people were like why stop at images what if you try to do video generation using diffusion instead that also works this was proposed by Google this paper called video diffusion what they do is they feed in this prompt firecrackers and they were able to get this sort of result out in response to Google releasing this Facebook made their own model called make a video
uh so on this slide I have some examples of videos that might that are like somewhat realistic like a horse running out of a pond or an artist like paint brush on a piece of canvas they can also like generate other kinds of things it's like you can fit in the prompt a grizzly bear in a Calculus class and it throws out this image to the left or you could have like a panda I think let's mix that and this model is able to like uh generate images for like all of these kinds of prompts
in response to Facebook Google again came up with their with another model the extended the Imagine model that I had that I showed it shown you earlier two videos now again these are just some didn't list the prompts for this um on their website which uh that would be kind of curious to see what sort of prominent like these results out and if I stop there you can also turn you can also like generate 3D models using text so uh I think this paper actually came out last week so this is like very very new
um I have linked the actual website in the slideshow so you can look at other models too see um how this sort of uh model works finally you can also apply the vision to RL I don't know if there's like any robotics enthusiasts In This Crowd but that's also completely fair game you can apply diffusion to so trajectory planning and even something like offline RL this was a paper that came out earlier this year called diffusion ql and it achieves set of the art results in offline RL yeah again uh just a summary you've entered
over a lot of stuff today we eventually over how the generative modeling works and how you can view image synthesis as sampling from a data distribution we talked about some theory of diffusion again I know this was very math heavy if you're confused that's completely fine I've actually linked all the papers that I used in the in the slide over here I've also linked some other helpful papers and other resources like blog posts or maybe um uh like say videos or that that others have made I would recommend checking all of these out in your
own time if you want to like learn more about this because like I mentioned earlier diffusion is sort of the state of the art in general modeling right now that's all I have for today
