CS 182: Lecture 1, Part 3: Introduction

all right so you know if you're in this class you probably heard something about deep learning before you probably heard that you know it's quite a good idea you probably heard that it works pretty well uh maybe you've heard that it is a multi-billion dollar a year industry maybe you've heard that it's something that can allow machines to solve uh cognitively and computationally demanding tasks that previously were not possible to solve what is it that makes deep learning actually work well let's start with a very crude uh uh very simplistic timeline of developments that are in one way or another highly relevant to deep learning we could start in the 1950s alan turing first described how learning could be a path to machine intelligence in his seminal paper uh in fact some of the methods that he described for doing this looked an awful lot like what we might recognize as artificial neural networks today in 1957 rosenblatt's perceptron was proposed as a practical learning method the perceptron basically does this it does the the linear separation of two classes and for a while people were really excited about this because here was an actual learning algorithm that had um you know some elements related to those neural networks it seemed to work it seemed to separate classes at least on 2d planes uh seemed like a really good idea in 1969 uh minsky and pepper publish a book regarding learning algorithms like the perceptron and one of the things this book describes is some fundamental limitations of neural networks we're actually pretty familiar with those limitations today like for example the fact that it's very difficult to find a globally optimal neural network basically the best possible neural network for a given problem uh there are many other limitations but at the time this caused a lot of concern because people didn't realize prior to this book just how many limitations there were to this basic idea in fact after the publication of this book while there was you know a fair bit of work on neural networks and things that were today called deep learning a lot of mainstream research focused on what i call shadow learning previously simpler learning methods uh built on top of hand-designed features but nonetheless there was a fair bit of progress in 1986 back propagation was proposed as a practical method for training deep neural networks uh in 1989 there was work uh you know at the labs developing neural networks for handwritten digital recognition getting very good results recognizing zip codes written envelopes and things like that but this was in some ways kind of in the background in the foreground 90s to early 2000s there was a huge wave of interest in the ml community in probabilistic methods convex optimization but mostly shallow models models where the learning process could be very well understood and was comparably simple and led to you know lots of good theoretical analysis and then some somehow around 2006 deep neural networks started gaining a lot more attention uh in fact they started gaining attention much the way that a speeding freight train gets your attention before you know it it's right upon you and in 2012 alex krashewski uh published the alexnet paper which all of a sudden beat all other methods on the imagenet benchmark the hardest image recognition task in computer vision at the time so what's going on what the heck happened here in the middle it seemed like 1969 things kind of went a little downhill there's some early murmurings in 2006 and all thousands of thousands in 2012 there's a paper coming out that beats everybody else at one of the hardest uh computer perception problems uh currently being studied that's a little surprising well so what is it that makes deep learning work and perhaps if we understand these factors this timeline will make a little bit more sense to us one of the things that's really important for deep learning to work is big models with many many layers this is important because a big part of the reason for the effectiveness of deep learning is its ability to learn representations but good representations are high level representations they are abstract they are invariant nuisances they capture meaningful higher level features objects concepts semantic meaning things like that acquiring such higher level concepts requires many layers of processing each layer of processing is very simple remember so in order to get something complex you need to stack up many layers you also need large data sets and those large data sets need to have many examples and this is very important because when you have so many layers you're going to have a great many parameters and they'll train that many parameters you need large datasets but that's not the only reason you need large datasets the other reason that you need large datasets is because the things that you're learning are just really complex imagine how many different things a human child sees in the first three years of their life while our visual systems are extremely sophisticated and extremely capable at generalizing and avoiding mistakes we actually learn from a pretty considerable amount of data so it should come as no surprise to us that effective machine learning systems should also need large datasets and lastly in order to train models of many many layers on very large data sets you need enough compute to handle all of this you need enough compute to be able to train your model on those large data sets and train all of those parameters uh you know before you die of old age essentially right you can't wait for years and years for your model to train the world is going to move on so you need powerful enough computers to train these giant models on large data sets fast enough to be useful now the all three of these things are you know not really very controversial today they're pretty widely known but i would posit that each of those three things have also been the major you know provide a big part of the explanation for that gap for what the heck happened here so let's talk about the number of layers let's talk about the size of the model so this is a diagram of lynette this was the network for handwritten digit recognition in 1989. now this network has seven layers but they're fairly small layers so we'll learn about what these words mean we'll learn about convolutions and fully connected layers very soon in a couple of lectures but it's a network essentially with seven layers of processing seven layers means there's seven operations each one transforming the previous representation into another more abstract one um alex strzewski's model for imagenet which was published in 2012 had eight players which is only a little bit more but they were much larger layers so uh you can tell just to give you some idea uh this might not mean that much to you yet because we haven't really learned about how neural networks work but you see that the f6 layer here for london has 84 units uh the last fully connected layer in alex net uh it actually has 4096 units it's like two blocks of 2048 um so it's quite a bit bigger these days and this is already dated this is from 2015 but let's say these days within the last five years state-of-the-art models for image recognition have 152 layers this is resin at 152. there are larger models there are models with over a thousand layers at this point and they work a lot better so more layers really helps here's a plot of how much better they work so each bar here represents the error on ils vrc the imagenet large scale visual recognition challenge this is a large scale problem for recognizing objects in images and you can see here alexnet that's the third bar error of 16.

4 eight layers uh fast forward to 2014 vgg very classic network error rate 7. 3 percent 19 layers fast forward even further 2015 resonant 152 layers error rate 3. 57 so more layers helps now those of you that are watching this with a more critical eye might be saying you're getting some diminishing returns there so we'll come back to that data set size one of the most popular data sets historically in machine learning especially in deep learning is the mnist handwritten characters data set widely in use from 1990 today it's a data set of handwritten numbers zero one two three four five six seven eight and nine it has sixty thousand uh training images you might say sixty thousand is a lot caltech 101 around 2013 if you think back to that timeline that predates the deep learning era uh 9000 images see far 10 2009 now we're getting into kind of the the early stages of the deep learning uh transformation 60 000 images okay kind of like ms getting a little bit better ils vrc which is the training set for what is often referred to as image now imagine itself is actually much larger but ios vrc is a bit smaller developed in 2009 1.

5 million images these days uh data sets that are used commercially by large companies like facebook google microsoft and so on are larger yet larger by one or two orders of magnitude and these are the models that people use when they uh classify objects on the internet when you upload your photo to facebook facebook recognizes that with a model that is trained probably on one to two orders of magnitude more than interest than this so this this is the amount of data that you need for such complex situations for deep learning to work so of course if in the 1990s everybody was testing their networks with models strain of 60 000 images maybe should be no surprise that you know things didn't transform until much later what about compute well this is a plot it'll take a little while to digest this plot shows many different uh neural networks in this case for computer vision the vertical axis is the accuracy on an internet so before we're looking at error now it's accuracy so higher is better the horizontal axis is the number of operations in gigaflops gigafloating point operations per second to run the neural network so this is roughly a measure of how much compute it takes to evaluate the network not to train it to evaluate it and the size of each circle represents the number of parameters so generally you'd expect that as as you go to the right the size of the circle should get bigger but it's not actually a one-to-one correspondence because the number of operations might be a lot larger than the number of parameters because of how convolutions work and we'll learn about convolutions later so the number of operations is really the measure of the computational cost of a neural network whereas the number of parameters is its size so if we're talking about compute it's really the operations the horizontal axis that we care about um so alexnet is over there in the lower left hand corner it's sitting at uh you know somewhere in the ballpark of 0.