Lecture 1 | Introduction to Convolutional Neural Networks for Visual Recognition

- So welcome everyone to CS231n. I'm super excited to offer this class again for the third time. It seems that every time we offer this class it's growing exponentially unlike most things in the world. This is the third time we're teaching this class. The first time we had 150 students. Last year, we had 350 students, so it doubled. This year we've doubled again to about 730 students when I checked this morning. So anyone who was not able to fit into the lecture hall I apologize. But, the videos will be up on the SCPD website within

about two hours. So if you weren't able to come today, then you can still check it out within a couple hours. So this class CS231n is really about computer vision. And, what is computer vision? Computer vision is really the study of visual data. Since there's so many people enrolled in this class, I think I probably don't need to convince you that this is an important problem, but I'm still going to try to do that anyway. The amount of visual data in our world has really exploded to a ridiculous degree in the last couple of years.

And, this is largely a result of the large number of sensors in the world. Probably most of us in this room are carrying around smartphones, and each smartphone has one, two, or maybe even three cameras on it. So I think on average there's even more cameras in the world than there are people. And, as a result of all of these sensors, there's just a crazy large, massive amount of visual data being produced out there in the world each day. So one statistic that I really like to kind of put this in perspective is a 2015

study from CISCO that estimated that by 2017 which is where we are now that roughly 80% of all traffic on the internet would be video. This is not even counting all the images and other types of visual data on the web. But, just from a pure number of bits perspective, the majority of bits flying around the internet are actually visual data. So it's really critical that we develop algorithms that can utilize and understand this data. However, there's a problem with visual data, and that's that it's really hard to understand. Sometimes we call visual data the

dark matter of the internet in analogy with dark matter in physics. So for those of you who have heard of this in physics before, dark matter accounts for some astonishingly large fraction of the mass in the universe, and we know about it due to the existence of gravitational pulls on various celestial bodies and what not, but we can't directly observe it. And, visual data on the internet is much the same where it comprises the majority of bits flying around the internet, but it's very difficult for algorithms to actually go in and understand and see what

exactly is comprising all the visual data on the web. Another statistic that I like is that of Youtube. So roughly every second of clock time that happens in the world, there's something like five hours of video being uploaded to Youtube. So if we just sit here and count, one, two, three, now there's 15 more hours of video on Youtube. Google has a lot of employees, but there's no way that they could ever have an employee sit down and watch and understand and annotate every video. So if they want to catalog and serve you relevant videos

and maybe monetize by putting ads on those videos, it's really crucial that we develop technologies that can dive in and automatically understand the content of visual data. So this field of computer vision is truly an interdisciplinary field, and it touches on many different areas of science and engineering and technology. So obviously, computer vision's the center of the universe, but sort of as a constellation of fields around computer vision, we touch on areas like physics because we need to understand optics and image formation and how images are actually physically formed. We need to understand biology and

psychology to understand how animal brains physically see and process visual information. We of course draw a lot on computer science, mathematics, and engineering as we actually strive to build computer systems that implement our computer vision algorithms. So a little bit more about where I'm coming from and about where the teaching staff of this course is coming from. Me and my co-instructor Serena are both PHD students in the Stanford Vision Lab which is headed by professor Fei-Fei Li, and our lab really focuses on machine learning and the computer science side of things. I work a little

bit more on language and vision. I've done some projects in that. And, other folks in our group have worked a little bit on the neuroscience and cognitive science side of things. So as a bit of introduction, you might be curious about how this course relates to other courses at Stanford. So we kind of assume a basic introductory understanding of computer vision. So if you're kind of an undergrad, and you've never seen computer vision before, maybe you should've taken CS131 which was offered earlier this year by Fei-Fei and Juan Carlos Niebles. There was a course taught

last quarter by Professor Chris Manning and Richard Socher about the intersection of deep learning and natural language processing. And, I imagine a number of you may have taken that course last quarter. There'll be some overlap between this course and that, but we're really focusing on the computer vision side of thing, and really focusing all of our motivation in computer vision. Also concurrently taught this quarter is CS231a taught by Professor Silvio Savarese. And, CS231a really focuses is a more all encompassing computer vision course. It's focusing on things like 3D reconstruction, on matching and robotic vision, and

it's a bit more all encompassing with regards to vision than our course. And, this course, CS231n, really focuses on a particular class of algorithms revolving around neural networks and especially convolutional neural networks and their applications to various visual recognition tasks. Of course, there's also a number of seminar courses that are taught, and you'll have to check the syllabus and course schedule for more details on those 'cause they vary a bit each year. So this lecture is normally given by Professor Fei-Fei Li. Unfortunately, she wasn't able to be here today, so instead for the majority of

the lecture we're going to tag team a little bit. She actually recorded a bit of pre-recorded audio describing to you the history of computer vision because this class is a computer vision course, and it's very critical and important that you understand the history and the context of all the existing work that led us to these developments of convolutional neural networks as we know them today. I'll let virtual Fei-Fei take over [laughing] and give you a brief introduction to the history of computer vision. Okay let's start with today's agenda. So we have two topics to cover

one is a brief history of computer vision and the other one is the overview of our course CS 231 so we'll start with a very brief history of where vision comes from when did computer vision start and where we are today. The history the history of vision can go back many many years ago in fact about 543 million years ago. What was life like during that time? Well the earth was mostly water there were a few species of animals floating around in the ocean and life was very chill. Animals didn't move around much there they

don't have eyes or anything when food swims by they grab them if the food didn't swim by they just float around but something really remarkable happened around 540 million years ago. From fossil studies zoologists found out within a very short period of time — ten million years — the number of animal species just exploded. It went from a few of them to hundreds of thousands and that was strange — what caused this? There were many theories but for many years it was a mystery evolutionary biologists call this evolution's Big Bang. A few years ago an

Australian zoologist called Andrew Parker proposed one of the most convincing theory from the studies of fossils he discovered around 540 million years ago the first animals developed eyes and the onset of vision started this explosive speciation phase. Animals can suddenly see; once you can see life becomes much more proactive. Some predators went after prey and prey have to escape from predators so the evolution or onset of vision started a evolutionary arms race and animals had to evolve quickly in order to survive as a species so that was the beginning of vision in animals after 540

million years vision has developed into the biggest sensory system of almost all animals especially intelligent animals in humans we have almost 50% of the neurons in our cortex involved in visual processing it is the biggest sensory system that enables us to survive, work, move around, manipulate things, communicate, entertain, and many things. The vision is really important for animals and especially intelligent animals. So that was a quick story of biological vision. What about humans, the history of humans making mechanical vision or cameras? Well one of the early cameras that we know today is from the 1600s,

the Renaissance period of time, camera obscura and this is a camera based on pinhole camera theories. It's very similar to, it's very similar to the to the early eyes that animals developed with a hole that collects lights and then a plane in the back of the camera that collects the information and project the imagery. So as cameras evolved, today we have cameras everywhere this is one of the most popular sensors people use from smartphones to to other sensors. In the mean time biologists started studying the mechanism of vision. One of the most influential work in

both human vision where animal vision as well as that inspired computer vision is the work done by Hubel and Wiesel in the 50s and 60s using electrophysiology. What they were asking, the question is "what was the visual processing mechanism like in primates, in mammals" so they chose to study cat brain which is more or less similar to human brain from a visual processing point of view. What they did is to stick some electrodes in the back of the cat brain which is where the primary visual cortex area is and then look at what stimuli makes

the neurons in the in the back in the primary visual cortex of cat brain respond excitedly what they learned is that there are many types of cells in the, in the primary visual cortex part of the the cat brain but one of the most important cell is the simple cells they respond to oriented edges when they move in certain directions. Of course there are also more complex cells but by and large what they discovered is visual processing starts with simple structure of the visual world, oriented edges and as information moves along the visual processing pathway

the brain builds up the complexity of the visual information until it can recognize the complex visual world. So the history of computer vision also starts around early 60s. Block World is a set of work published by Larry Roberts which is widely known as one of the first, probably the first PhD thesis of computer vision where the visual world was simplified into simple geometric shapes and the goal is to be able to recognize them and reconstruct what these shapes are. In 1966 there was a now famous MIT summer project called "The Summer Vision Project." The goal

of this Summer Vision Project, I read: "is an attempt to use our summer workers effectively in a construction of a significant part of a visual system." So the goal is in one summer we're gonna work out the bulk of the visual system. That was an ambitious goal. Fifty years have passed; the field of computer vision has blossomed from one summer project into a field of thousands of researchers worldwide still working on some of the most fundamental problems of vision. We still have not yet solved vision but it has grown into one of the most important

and fastest growing areas of artificial intelligence. Another person that we should pay tribute to is David Marr. David Marr was a MIT vision scientist and he has written an influential book in the late 70s about what he thinks vision is and how we should go about computer vision and developing algorithms that can enable computers to recognize the visual world. The thought process in his, in David Mars book is that in order to take an image and arrive at a final holistic full 3d representation of the visual world we have to go through several process. The

first process is what he calls "primal sketch;" this is where mostly the edges, the bars, the ends, the virtual lines, the curves, the boundaries, are represented and this is very much inspired by what neuroscientists have seen: Hubel and Wiesel told us the early stage of visual processing has a lot to do with simple structures like edges. Then the next step after the edges and the curves is what David Marr calls "two-and-a-half d sketch;" this is where we start to piece together the surfaces, the depth information, the layers, or the discontinuities of the visual scene, and

then eventually we put everything together and have a 3d model hierarchically organized in terms of surface and volumetric primitives and so on. So that was a very idealized thought process of what vision is and this way of thinking actually has dominated computer vision for several decades and is also a very intuitive way for students to enter the field of vision and think about how we can deconstruct the visual information. Another very important seminal group of work happened in the 70s where people began to ask the question "how can we move beyond the simple block world

and start recognizing or representing real world objects?" Think about the 70s, it's the time that there's very little data available; computers are extremely slow, PCs are not even around, but computer scientists are starting to think about how we can recognize and represent objects. So in Palo Alto both at Stanford as well as SRI, two groups of scientists that propose similar ideas: one is called "generalized cylinder," one is called "pictorial structure." The basic idea is that every object is composed of simple geometric primitives; for example a person can be pieced together by generalized cylindrical shapes or

a person can be pieced together by critical part in their elastic distance between these parts so either representation is a way to reduce the complex structure of the object into a collection of simpler shapes and their geometric configuration. These work have been influential for quite a few, quite a few years and then in the 80s David Lowe, here is another example of thinking how to reconstruct or recognize the visual world from simple world structures, this work is by David Lowe which he tries to recognize razors by constructing lines and edges and and mostly straight lines

and their combination. So there was a lot of effort in trying to think what what is the tasks in computer vision in the 60s 70s and 80s and frankly it was very hard to solve the problem of object recognition; everything I've shown you so far are very audacious ambitious attempts but they remain at the level of toy examples or just a few examples. Not a lot of progress have been made in terms of delivering something that can work in real world. So as people think about what are the problems to solving vision one important question

came around is: if object recognition is too hard, maybe we should first do object segmentation, that is the task of taking an image and group the pixels into meaningful areas. We might not know the pixels that group together is called a person, but we can extract out all the pixels that belong to the person from its background; that is called image segmentation. So here's one very early seminal work by Jitendra Malik and his student Jianbo Shi from Berkeley from using a graph theory algorithm for the problem of image segmentation. Here's another problem that made some

headway ahead of many other problems in computer vision, which is face detection. Faces one of the most important objects to humans, probably the most important objects to humans, around the time of 1999 to 2000 machine learning techniques, especially statistical machine learning techniques start to gain momentum. These are techniques such as support vector machines, boosting, graphical models, including the first wave of neural networks. One particular work that made a lot of contribution was using AdaBoost algorithm to do real-time face detection by Paul Viola and Michael Jones and there's a lot to admire in this work. It

was done in 2001 when computer chips are still very very slow but they're able to do face detection in images in near-real-time and after the publication of this paper in five years time, 2006, Fujifilm rolled out the first digital camera that has a real-time face detector in the in the camera so it was a very rapid transfer from basic science research to real world application. So as a field we continue to explore how we can do object recognition better so one of the very influential way of thinking in the late 90s til the first 10

years of 2000 is feature based object recognition and here is a seminal work by David Lowe called SIFT feature. The idea is that to match and the entire object for example here is a stop sign to another stop sight is very difficult because there might be all kinds of changes due to camera angles, occlusion, viewpoint, lighting, and just the intrinsic variation of the object itself but it's inspired to observe that there are some parts of the object, some features, that tend to remain diagnostic and invariant to changes so the task of object recognition began with

identifying these critical features on the object and then match the features to a similar object, that's a easier task than pattern matching the entire object. So here is a figure from his paper where it shows that a handful, several dozen SIFT features from one stop sign are identified and matched to the SIFT features of another stop sign. Using the same building block which is features, diagnostic features in images, we have as a field has made another step forward and start to recognizing holistic scenes. Here is an example algorithm called Spatial Pyramid Matching; the idea is

that there are features in the images that can give us clues about which type of scene it is, whether it's a landscape or a kitchen or a highway and so on and this particular work takes these features from different parts of the image and in different resolutions and put them together in a feature descriptor and then we do support vector machine algorithm on top of that. Similarly a very similar work has gained momentum in human recognition so putting together these features well we have a number of work that looks at how we can compose human

bodies in more realistic images and recognize them. So one work is called the "histogram of gradients," another work is called "deformable part models," so as you can see as we move from the 60s 70s 80s towards the first decade of the 21st century one thing is changing and that's the quality of the pictures were no longer, with the Internet the the the growth of the Internet the digital cameras were having better and better data to study computer vision. So one of the outcome in the early 2000s is that the field of computer vision has defined

a very important building block problem to solve. It's not the only problem to solve but in terms of recognition this is a very important problem to solve which is object recognition. I talked about object recognition all along but in the early 2000s we began to have benchmark data set that can enable us to measure the progress of object recognition. One of the most influential benchmark data set is called PASCAL Visual Object Challenge, and it's a data set composed of 20 object classes, three of them are shown here: train, airplane, person; I think it also has

cows, bottles, cats, and so on; and the data set is composed of several thousand to ten thousand images per category and then the field different groups develop algorithm to test against the testing set and see how we have made progress. So here is a figure that shows from year 2007 to year 2012. The performance on detecting objects the 20 object in this image in a in a benchmark data set has steadily increased. So there was a lot of progress made. Around that time a group of us from Princeton to Stanford also began to ask a

harder question to ourselves as well as our field which is: are we ready to recognize every object or most of the object in the world. It's also motivated by an observation that is rooted in machine learning which is that most of the machine learning algorithms it doesn't matter if it's graphical model, or support vector machine, or AdaBoost, is very likely to overfit in the training process and part of the problem is visual data is very complex because it's complex our models tend to have a high dimension a high dimension of input and have to have

a lot of parameters to fit and when we don't have enough training data overfitting happens very fast and then we cannot generalize very well. So motivated by this dual reason, one is just want to recognize the world of all the objects, the other one is to come back the machine learning overcome the the machine learning bottleneck of overfitting, we began this project called ImageNet. We wanted to put together the largest possible dataset of all the pictures we can find, the world of objects, and use that for training as well as for benchmarking. So it was

a project that took us about three years, lots of hard work; it basically began with downloading billions of images from the internet organized by the dictionary we called WordNet which is tens of thousands of object classes and then we have to use some clever crowd engineering trick a method using Amazon Mechanical Turk platform to sort, clean, label each of the images. The end result is a ImageNet of almost 15 million or 40 million plus images organized in twenty-two thousand categories of objects and scenes and this is the gigantic, probably the biggest dataset produced in the

field of AI at that time and it began to push forward the algorithm development of object recognition into another phase. Especially important is how to benchmark the progress so starting 2009 the ImageNet team rolled out an international challenge called ImageNet Large-Scale Visual Recognition Challenge and for this challenge we put together a more stringent test set of 1.4 million objects across 1,000 object classes and this is to test the image classification recognition results for the computer vision algorithms. So here's the example picture and if an algorithm can output 5 labels and and top five labels includes

the correct object in this picture then we call this a success. So here is a result summary of the ImageNet Challenge, of the image classification result from 2010 to 2015 so on x axis you see the years and the y axis you see the error rate. So the good news is the error rate is steadily decreasing to the point by 2012 the error rate is so low is on par with what humans can do and here a human I mean a single Stanford PhD student who spend weeks doing this task as if he were a

computer participating in the ImageNet Challenge. So that's a lot of progress made even though we have not solved all the problems of object recognition which you'll learn about in this class but to go from an error rate that's unacceptable for real-world application all the way to on par being on par with humans in ImageNet challenge, the field took only a few years. And one particular moment you should notice on this graph is the the year 2012. In the first two years our error rate hovered around 25 percent but in 2012 the error rate was dropped

more almost 10 percent to 16 percent even though now it's better but that drop was very significant and the winning algorithm of that year is a convolutional neural network model that beat all other algorithms around that time to win the ImageNet challenge and this is the focus of our whole course this quarter is to look at to have a deep dive into what convolutional neural network models are and another name for this is deep learning by by popular popular name now it's called deep learning and to look at what these models are what are the

principles what are the good practices what are the recent progress of this model, but here is where the history was made is that we, around 2012 convolutional neural network model or deep learning models showed the tremendous capacity and ability in making a good progress in the field of computer vision along with several other sister fields like natural language processing and speech recognition. So without further ado I'm going to hand the rest of the lecture to to Justin to talk about the overview of CS 231n. Alright, thanks so much Fei-Fei. I'll take it over from here.

So now I want to shift gears a little bit and talk a little bit more about this class CS231n. So this class focuses on one of these most, so the primary focus of this class is this image classification problem which we previewed a little bit in the contex of the ImageNet Challenge. So in image classification, again, the setup is that your algorithm looks at an image and then picks from among some fixed set of categories to classify that image. And, this might seem like somewhat of a restrictive or artificial setup, but it's actual quite general.

And, this problem can be applied in many different settings both in industry and academia and many different places. So for example, you could apply this to recognizing food or recognizing calories in food or recognizing different artworks, different product out in the world. So this relatively basic tool of image classification is super useful on its own and could be applied all over the place for many different applications. But, in this course, we're also going to talk about several other visual recognition problems that build upon many of the tools that we develop for the purpose of image

classification. We'll talk about other problems such as object detection or image captioning. So the setup in object detection is a little bit different. Rather than classifying an entire image as a cat or a dog or a horse or whatnot, instead we want to go in and draw bounding boxes and say that there is a dog here, and a cat here, and a car over in the background, and draw these boxes describing where objects are in the image. We'll also talk about image captioning where given an image the system now needs to produce a natural language

sentence describing the image. It sounds like a really hard, complicated, and different problem, but we'll see that many of the tools that we develop in service of image classification will be reused in these other problems as well. So we mentioned this before in the context of the ImageNet Challenge, but one of the things that's really driven the progress of the field in recent years has been this adoption of convolutional neural networks or CNNs or sometimes called convnets. So if we look at the algorithms that have won the ImageNet Challenge for the last several years, in

2011 we see this method from Lin et al which is still hierarchical. It consists of multiple layers. So first we compute some features, next we compute some local invariances, some pooling, and go through several layers of processing, and then finally feed this resulting descriptor to a linear SVN. What you'll notice here is that this is still hierarchical. We're still detecting edges. We're still having notions of invariance. And, many of these intuitions will carry over into convnets. But, the breakthrough moment was really in 2012 when Jeff Hinton's group in Toronto together with Alex Krizhevsky and Ilya

Sutskever who were his PHD student at that time created this seven layer convolutional neural network now known as AlexNet, then called Supervision which just did very, very well in the ImageNet competition in 2012. And, since then every year the winner of ImageNet has been a neural network. And, the trend has been that these networks are getting deeper and deeper each year. So AlexNet was a seven or eight layer neural network depending on how exactly you count things. In 2015 we had these much deeper networks. GoogleNet from Google and VGG, the VGG network from Oxford which

was about 19 layers at that time. And, then in 2015 it got really crazy and this paper came out from Microsoft Research Asia called Residual Networks which were 152 layers at that time. And, since then it turns out you can get a little bit better if you go up to 200, but you run our of memory on your GPUs. We'll get into all of that later, but the main takeaway here is that convolutional neural networks really had this breakthrough moment in 2012, and since then there's been a lot of effort focused in tuning and tweaking

these algorithms to make them perform better and better on this problem of image classification. And, throughout the rest of the quarter, we're going to really dive in deep, and you'll understand exactly how these different models work. But, one point that's really important, it's true that the breakthrough moment for convolutional neural networks was in 2012 when these networks performed very well on the ImageNet Challenge, but they certainly weren't invented in 2012. These algorithms had actually been around for quite a long time before that. So one of the sort of foundational works in this area of convolutional

neural networks was actually in the '90s from Jan LeCun and collaborators who at that time were at Bell Labs. So in 1998 they build this convolutional neural network for recognizing digits. They wanted to deploy this and wanted to be able to automatically recognize handwritten checks or addresses for the post office. And, they built this convolutional neural network which could take in the pixels of an image and then classify either what digit it was or what letter it was or whatnot. And, the structure of this network actually look pretty similar to the AlexNet architecture that was

used in 2012. Here we see that, you know, we're taking in these raw pixels. We have many layers of convolution and sub-sampling, together with the so called fully connected layers. All of which will be explained in much more detail later in the course. But, if you just kind of look at these two pictures, they look pretty similar. And, this architecture in 2012 has a lot of these architectural similarities that are shared with this network going back to the '90s. So then the question you might ask is if these algorithms were around since the '90s, why

have they only suddenly become popular in the last couple of years? And, there's a couple really key innovations that happened that have changed since the '90s. One is computation. Thanks to Moore's law, we've gotten faster and faster computers every year. And, this is kind of a coarse measure, but if you just look at the number of transistors that are on chips, then that has grown by several orders of magnitude between the '90s and today. We've also had this advent of graphics processing units or GPUs which are super parallelizable and ended up being a perfect tool

for really crunching these computationally intensive convolutional neural network models. So just by having more compute available, it allowed researchers to explore with larger architectures and larger models, and in some cases, just increasing the model size, but still using these kind of classical approaches and classical algorithms tends to work quite well. So this idea of increasing computation is super important in the history of deep learning. I think the second key innovation that changed between now and the '90s was data. So these algorithms are very hungry for data. You need to feed them a lot of labeled

images and labeled pixels for them to eventually work quite well. And, in the '90s there just wasn't that much labeled data available. This was, again, before tools like Mechanical Turk, before the internet was super, super widely used. And, it was very difficult to collect large, varied datasets. But, now in the 2010s with datasets like PASCAL and ImageNet, there existed these relatively large, high quality labeled datasets that were, again, orders and orders magnitude bigger than the dataset available in the '90s. And, these much large datasets, again, allowed us to work with higher capacity models and train

these models to actually work quite well on real world problems. But, the critical takeaway here is that convolutional neural networks although they seem like this sort of fancy, new thing that's only popped up in the last couple of years, that's really not the case. And, these class of algorithms have existed for quite a long time in their own right as well. Another thing I'd like to point out in computer vision we're in the business of trying to build machines that can see like people. And, people can actually do a lot of amazing things with their

visual systems. When you go around the world, you do a lot more than just drawing boxes around the objects and classifying things as cats or dogs. Your visual system is much more powerful than that. And, as we move forward in the field, I think there's still a ton of open challenges and open problems that we need to address. And, we need to continue to develop our algorithms to do even better and tackle even more ambitious problems. Some examples of this are going back to these older ideas in fact. Things like semantic segmentation or perceptual grouping

where rather than labeling the entire image, we want to understand for every pixel in the image what is it doing, what does it mean. And, we'll revisit that idea a little bit later in the course. There's definitely work going back to this idea of 3D understanding, of reconstructing the entire world, and that's still an unsolved problem I think. There're just tons and tons of other tasks that you can imagine. For example activity recognition, if I'm given a video of some person doing some activity, what's the best way to recognize that activity? That's quite a challenging

problem as well. And, then as we move forward with things like augmented reality and virtual reality, and as new technologies and new types of sensors become available, I think we'll come up with a lot of new, interesting hard and challenging problems to tackle as a field. So this is an example from some of my own work in the vision lab on this dataset called Visual Genome. So here the idea is that we're trying to capture some of these intricacies in the real world. Rather than maybe describing just boxes, maybe we should be describing images as

these whole large graphs of semantically related concepts that encompass not just object identities but also object relationships, object attributes, actions that are occurring in the scene, and this type of representation might allow us to capture some of this richness of the visual world that's left on the table when we're using simple classification. This is by no means a standard approach at this point, but just kind of giving you this sense that there's so much more that your visual system can do that is maybe not captured in this vanilla image classification setup. I think another really

interesting work that kind of points in this direction actually comes from Fei-Fei's grad school days when she was doing her PHD at Cal Tech with her advisors there. In this setup, they had people, they stuck people, and they showed people this image for just half a second. So they flashed this image in front of them for just a very short period of time, and even in this very, very rapid exposure to an image, people were able to write these long descriptive paragraphs giving a whole story of the image. And, this is quite remarkable if you

think about it that after just half a second of looking at this image, a person was able to say that this is some kind of a game or fight, two groups of men. The man on the left is throwing something. Outdoors because it seem like I have an impression of grass, and so on and so on. And, you can imagine that if a person were to look even longer at this image, they could write probably a whole novel about who these people are, and why are they in this field playing this game. They could go

on and on and on roping in things from their external knowledge and their prior experience. This is in some sense the holy grail of computer vision. To sort of understand the story of an image in a very rich and deep way. And, I think that despite the massive progress in the field that we've had over the past several years, we're still quite a long way from achieving this holy grail. Another image that I think really exemplifies this idea actually comes, again, from Andrej Karpathy's blog is this amazing image. Many of you smiled, many of you

laughed. I think this is a pretty funny image. But, why is it a funny image? Well we've got a man standing on a scale, and we know that people are kind of self conscious about their weight sometimes, and scales measure weight. Then we've got this other guy behind him pushing his foot down on the scale, and we know that because of the way scales work that will cause him to have an inflated reading on the scale. But, there's more. We know that this person is not just any person. This is actually Barack Obama who was

at the time President of the United States, and we know that Presidents of the United States are supposed to be respectable politicians that are [laughing] probably not supposed to be playing jokes on their compatriots in this way. We know that there's these people in the background that are laughing and smiling, and we know that that means that they're understanding something about the scene. We have some understanding that they know that President Obama is this respectable guy who's looking at this other guy. Like, this is crazy. There's so much going on in this image. And, our

computer vision algorithms today are actually a long way I think from this true, deep understanding of images. So I think that sort of despite the massive progress in the field, we really have a long way to go. To me, that's really exciting as a researcher 'cause I think that we'll have just a lot of really exciting, cool problems to tackle moving forward. So I hope at this point I've done a relatively good job to convince you that computer vision is really interesting. It's really exciting. It can be very useful. It can go out and make

the world a better place in various ways. Computer vision could be applied in places like medical diagnosis and self-driving cars and robotics and all these different places. In addition to sort of tying back to sort of this core idea of understanding human intelligence. So to me, I think that computer vision is this fantastically amazing, interesting field, and I'm really glad that over the course of the quarter, we'll get to really dive in and dig into all these different details about how these algorithms are working these days. That's sort of my pitch about computer vision and

about the history of computer vision. I don't know if there's any questions about this at this time. Okay. So then I want to talk a little bit more about the logistics of this class for the rest of the quarter. So you might ask who are we? So this class is taught by Fei-Fei Li who is a professor of computer science here at Standford who's my advisor and director of the Stanford Vision Lab and also the Stanford AI Lab. The other two instructors are me, Justin Johnson, and Serena Yeung who is up here in the front.

We're both PHD students working under Fei-Fei on various computer vision problems. We have an amazing teaching staff this year of 18 TAs so far. Many of whom are sitting over here in the front. These guys are really the unsung heroes behind the scenes making the course run smoothly, making sure everything happens well. So be nice to them. [laughing] I think I also should mention this is the third time we've taught this course, and it's the first time that Andrej Karpathy has not been an instructor in this course. He was a very close friend of mine.

He's still alive. He's okay, don't worry. [laughing] But, he graduated, so he's actually here I think hanging around in the lecture hall. A lot of the development and the history of this course is really due to him working on it with me over the last couple of years. So I think you should be aware of that. Also about logistics, probably the best way for keeping in touch with the course staff is through Piazza. You should all go and signup right now. Piazza is really our preferred method of communication with the class with the teaching staff.

If you have questions that you're afraid of being embarrassed about asking in front of your classmates, go ahead and ask anonymously even post private questions directly to the teaching staff. So basically anything that you need should ideally go through Piazza. We also have a staff mailing list, but we ask that this is mostly for sort of personal, confidential things that you don't want going on Piazza, or if you have something that's super confidential, super personal, then feel free to directly email me or Fei-Fei or Serena about that. But, for the most part, most of your

communication with the staff should be through Piazza. We also have an optional textbook this year. This is by no means required. You can go through the course totally fine without it. Everything will be self contained. This is sort of exciting because it's maybe the first textbook about deep learning that got published earlier this year by E.N. Goodfellow, Yoshua Bengio, and Aaron Courville. I put the Amazon link here in the slides. You can get it if you want to, but also the whole content of the book is free online, so you don't even have to buy

it if you don't want to. So again, this is totally optional, but we'll probably be posting some readings throughout the quarter that give you an additional perspective on some of the material. So our philosophy about this class is that you should really understand the deep mechanics of all of these algorithms. You should understand at a very deep level exactly how these algorithms are working like what exactly is going on when you're stitching together these neural networks, how do these architectural decisions influence how the network is trained and tested and whatnot and all that. And, throughout

the course through the assignments, you'll be implementing your own convolutional neural networks from scratch in Python. You'll be implementing the full forward and backward passes through these things, and by the end, you'll have implemented a whole convolutional neural network totally on your own. I think that's really cool. But, we also kind of practical, and we know that in most cases people are not writing these things from scratch, so we also want to give you a good introduction to some of the state of the art software tools that are used in practice for these things. So

we're going to talk about some of the state of the art software packages like Tensor Flow, Torch, [Py]Torch, all these other things. And, I think you'll get some exposure to those on the homeworks and definitely through the course project as well. Another note about this course is that it's very state of the art. I think it's super exciting. This is a very fast moving field. As you saw, even these plots in the imaging challenge basically there's been a ton of progress since 2012, and like while I've been in grad school, the whole field is sort

of transforming ever year. And, that's super exciting and super encouraging. But, what that means is that there's probably content that we'll cover this year that did not exist the last time that this course was taught last year. I think that's super exciting, and that's one of my favorite parts about teaching this course is just roping in all these new scientific, hot off the presses stuff and being able to present it to you guys. We're also sort of about fun. So we're going to talk about some interesting maybe not so serious topics as well this quarter

including image captioning is pretty fun where we can write descriptions about images. But, we'll also cover some of these more artistic things like DeepDream here on the left where we can use neural networks to hallucinate these crazy, psychedelic images. And, by the end of the course, you'll know how that works. Or on the right, this idea of style transfer where we can take an image and render it in the style of famous artists like Picasso or Van Gogh or what not. And again, by the end of the quarter, you'll see how this stuff works. So

the way the course works is we're going to have three problem sets. The first problem set will hopefully be out by the end of the week. We'll have an in class, written midterm exam. And, a large portion of your grade will be the final course project where you'll work in teams of one to three and produce some amazing project that will blow everyone's minds. We have a late policy, so you have seven late days that you're free to allocate among your different homeworks. These are meant to cover things like minor illnesses or traveling or conferences

or anything like that. If you come to us at the end of the quarter and say that, "I suddenly have to give a presentation "at this conference." That's not going to be okay. That's what your late days are for. That being said, if you have some very extenuating circumstances, then do feel free to email the course staff if you have some extreme circumstances about that. Finally, I want to make a note about the collaboration policy. As Stanford students, you should all be aware of the honor code that governs the way that you should be collaborating

and working together, and we take this very seriously. We encourage you to think very carefully about how you're collaborating and making sure it's within the bounds of the honor code. So in terms of prerequisites, I think the most important is probably a deep familiarity with Python because all of the programming assignments will be in Python. Some familiarity with C or C++ would be useful. You will probably not be writing any C or C++ in this course, but as you're browsing through the source code of these various software packages, being able to read C++ code at

least is very useful for understanding how these packages work. We also assume that you know what calculus is, you know how to take derivatives all that sort of stuff. We assume some linear algebra. That you know what matrices are and how to multiply them and stuff like that. We can't be teaching you how to take like derivatives and stuff. We also assume a little bit of knowledge coming in of computer vision maybe at the level of CS131 or 231a. If you have taken those courses before, you'll be fine. If you haven't, I think you'll be

okay in this class, but you might have a tiny bit of catching up to do. But, I think you'll probably be okay. Those are not super strict prerequisites. We also assume a little bit of background knowledge about machine learning maybe at the level of CS229. But again, I think really important, key fundamental machine learning concepts we'll reintroduce as they come up and become important. But, that being said, a familiarity with these things will be helpful going forward. So we have a course website. Go check it out. There's a lot of information and links and syllabus

and all that. I think that's all that I really want to cover today. And, then later this week on Thursday, we'll really dive into our first learning algorithm and start diving into the details of these things.