What we see and what we value: AI with a human perspective—Fei-Fei Li (Stanford University)

PRESENTER: Welcome, everybody. I'm very excited to welcome Fe-Fei Li today. And of course, judging by how packed this room is, Fei-Fei doesn't really need an introduction.

And of course, if I actually were to introduce her by reading her bio, it would take a majority of today's time. So I'll keep this brief. Fei-Fei is a professor at Stanford, where she was also my PhD advisor.

She's the director of the Human Centered AI Institute at Stanford. And during the years of 2017 and 2018, she was also the Vice President at Google, as well as the Chief Scientist of AI and Machine Learning at Google Cloud. She, of course, has published hundreds of papers.

And perhaps one of the ones that a lot of people know her for is ImageNet, which as you all know, has ushered in the deep learning AI revolution that we're all in today. She also serves as a special advisor to the Secretary General of the United Nations, and is also a member of the National AI Resource Task Force for the White House Office of Science and Technology. And recently, she also has published her book titled, The World I See-- Curiosity, Exploration, and Discovery at the Dawn of AI.

And I'm sure she'll be talking about parts of that book today. So with that, Fei-Fei, welcome to the University of Washington. FEI-FEI LI: Thank you.

[APPLAUSE] Thank you. Thank you. Well, it's quite an honor to be here.

Actually it's as a professor one of the greatest joys and honor is to work with the best students, and see how their career has grown. And so being invited by Ranjay and his colleagues is really very special. And I'm just loving all the energy I've seen throughout the day today.

So OK, I want to share with you a talk that is a little bit meant at the high level and an overview of what I have done over the years, through the lens of computer vision and the development of AI. So the title is "What We See & What We Value-- AI with a Human Perspective. " I'm going to take you back to history a little bit.

And when I say history, I meant 540 million years ago. So 540 million years ago, what was that? Well, the Earth is a primordial soup.

And it's all living things live in the water. And there aren't that many of them. They are just simple animals floating around.

But something really strange happened in geologically a very short period of time, about 10 million years, is from fossil studies scientists have found there is an explosion of the number of animal species around that time. So much that that period is called the Cambrian Explosion or some people call it the Big Bang of evolution. And so what happened?

Why suddenly when life was so chill and simple, not too many animals, why life went from that picture to an explosive number of animal species? Well, there are many theories, from climate change to chemical composition of the water, to many things. But one of the leading theories of that Cambrian Explosion is by Andrew Parker, a zoologist from Australia.

He conjectured that this speciation explosion is triggered by the sudden evolution of vision, which sets off an evolutionary arms race where animals either evolved or died. Basically, he's saying as soon as you see the first light, you see the world in fundamentally different ways. You can see food.

You can see shelter. You can become someone's food. And they would actively prey on you.

And you have to actively interact and engage with the world in order to survive and reproduce, and so on. So from that point on, 540 million years to today, vision, visual intelligence has become a cornerstone of the development and the evolution of nervous system of animal intelligence. All the way to, of course, the most incredible visual machine we know in the universe which is the human vision.

And whether we're talking about people and many animals, we use vision to navigate the world, to live life, to communicate, to entertain ourselves, to socialize, to do so many things. Well, that was a very brief history of nature's vision. What about computer vision?

The history of computer vision is a little shorter than evolution. Urban legend goes around 60 years ago, 1966 I think, that there was one ambitious MIT professor who said, well, AI as a field has been born. And it looks like it's going well.

I think we can just solve vision in a summer. In fact, we'll solve vision by using our summer workers, undergrads, and we'll just spend this one summer to create or construct a significant part of visual system. This is not a frivolous conjecture.

I actually sympathize with him. Because for humans when you open your eyes, it feels so effortless to see. It feels that as soon as you open your eyes, the whole world's information is in front of you.

So it might be turned out to be an underestimation of how hard it is to construct the visual system. But it was a heroic effort. They didn't solve vision in a summer, not even a tiny bit of vision.

But 60 years later, vision today has become a very thriving field, both academically as well as in our technology world. I'm just showing you a couple of examples of where we are. Right?

We have visual applications everywhere. We're dreaming of self-driving cars, which hopefully will happen in our lifetime. We are using image classification or image recognition and so many image technologies for many things from, health care to just daily lives.

And generative AI has brought a whole new wave of visual applications and breakthroughs. So the rest of the talk is organized to answer this question. Where have we come from and where are we heading to?

And I want to share with you three major theses of the work that I have been doing in my career in recent few years, and just to share with you what I think. Let's begin with building AI to see what humans see. Why do we do that?

Because humans are really good at seeing. This is a 1970s cognitive science experiment to show you how good humans are. Every frame is refreshed at 10 Hertz, 100 milliseconds of presentation.

If I ask you as audience, I assume given how young you are-- you're not even born then-- you've never seen this video. Nod your head when you see one frame that has a person in it. You will see it.

Yeah, OK. You've never seen this video. I didn't tell you what the person looked like.

I didn't tell you which frame it will appear. You have no idea-- the gesture, the clothes, everything about this. Yet, you're so good at detecting this person.

Around the turn of the century, a group of French researchers have put a time on this effortlessness. It turned out seeing complex objects or complex categories for humans is not only effortless and accurate, it's fast. 150 milliseconds after the onset of a complex photo, either containing animals or not containing animals, humans you can measure brain signal that shows that differential signal of pictures, of scene pictures with animals and scene pictures without animals.

It means that it takes about 150 milliseconds in our wetware, right here, from the photons landing on your retina to the decision that you can make accurately. I know this sounds slow for silicons. But for our brain, for those of you who come from a little bit of neuroscience background, this is actually super fast.

It takes about 10 stages of spikes from passing from neuron to neuron to get here. So it's a very interesting measurement. At around the same time, neurophysiologists, so we've had psychologists telling us humans are really good at seeing objects.

We've got neuroscientists telling us not only we're good at it, we're fast. Now, this last set of study, also neurophysiologists use MRI study to tell us, because evolution has optimized recognition so much that we have dedicated neural correlates in the brain, areas that specializes in visual recognition. For example, the fusiform face area, or the parahippocampal place area-- these are areas that we see objects and scenes.

So what all this has told us, this research from the '70s, '80s, and '90s have told us, is that objects are really important for visual intelligence. It's a building block for people. And it's become a North Star for what vision needs to do.

It's not it's not all the North Stars, but it's one important North Star. And that has guided the early phase of my own research as well as the field of computer vision. As a field, we identified that object recognition, object categorization, is an important problem.

And it's a mathematically really challenging problem. It's effortless for us. But to recognize, say, a cute animal wombat, you actually have mathematically infinite way of rendering this animal wombat from 3D to the 2D pixels, whether it's lighting and texture variations, or background clutter and occlusion variations, or viewing angle camera angle occlusions, and so on.

So it's mathematically a really hard problem. So what did we do as a field? I summarized the progress of object recognition in three phases.

The first phase was concurrent. It's a very early phase, concurrent with this cognitive studies is what I call the hand-designed features of models. This is where very smart researchers use their own sheer power of their brain to design the kind of building blocks of objects, as well as the model, the parameters, and so on.

So we see Geon theory. We see generalized cylinder. We see parts and springs models.

And these are in the '70s, '80s, or early '90s. They're beautiful theory. They're mathematically beautiful models.

But the thing is, they don't work. They're theoretically beautiful. Then there's a second phase, which I think is the most important phase actually, leading up to deep learning, which is machine learning.

It's when we have introduced machine learning as a statistical modeling technique, but the input of these models are hand-designed features like patches, and parts of objects that are meant to carry a lot of semantic information. And the idea is that in order to recognize something like a human body, or a face, or whatever, a chair-- it's important to get these patches that contains ears and eyes and whatever. And then you use machine learning models to learn the parameters that stitch them together.

And this is when the whole field has experimented with many different kinds of statistical models from Bayes Net, support vector machine, boosting, conditional random field, random forest, and neural network. But this is the first phase of that. Something also important happened concurrently with this phase is actually the recognition of data.

In the early years of the 21st century, the field of computer vision recognized it's important to have benchmarking data sets, data sets like the PASCAL VOC data set, the Caltech 101 data set. That is meant to measure the progress of the field. And it turned out they can also become some level of training data.

But they're very small. They're in the order of hundreds and thousands of pictures, and a handful of object categories. Personally for me, this was around the time I stumbled upon a very incredible number.

I call it, if you read my book, I call it the Biederman number. Professor Biederman who sadly just passed away a year ago, is a cognitive psychologist studying vision and thinking about the scale and scope of human visual intelligence. And back of envelope, he put a guesstimate of humans can recognize 30 to 100,000 object categories in their lives.

And it's not a verified number. It's very hard to verify. This is a conjecture in one of his papers.

And he also went on to say that by age 6, you actually learn pretty much all the visual categories that a grown-up has learned. This is an incredible speed of learning, a dozen a day or so. So this number bugged me a lot because it just doesn't compare to all the data sets we've seen at that point.

And that was the reason, the inception of ImageNet, that we recognized, my students, Jordan, and collaborators, and I recognize that there's a new way of thinking about visual intelligence. It's deeply, deeply data driven. And it's not just the size of the data.

It's the diversity of data. And this is really history. You all know what ImageNet is.

And it also brought back the most important family of algorithm that is high capacity, and needs to be data driven, which is convolutional or neural network algorithm. And in the case of vision, we started with convolutional neural network. For those of you who are very young students, you probably don't remember this.

But even when I was a graduate student at the turn of the century, convolutional neural network was considered a classic algorithm, meaning it was pretty old. And it didn't work. But we still studied it when I was a graduate student.

It was incredible to see how data and the new techniques revitalized this whole family of algorithms. And for this audience, I'm going to skip. This is really too trivial.

But what happened is that this brought us the third phase of object recognition. And I would say more or less, quite a triumphant phase of object recognition, where using big data as training and convolutional neural network, we're able to recognize objects in the wild in a way that the first two phases couldn't. And these are just examples.

And of course, the most incredible moment, even for myself who was behind ImageNet, was 2012 when Professor Geoff Hinton and his students, very famous students, have written this defining paper as the beginning of the deep learning revolution. And ever since then, vision as a field and ImageNet as a data set has really been driven a lot of the algorithm advances in the pre-transformer era of deep learning. And very proudly as a field, even work like RESNET, were the precursors of many of the attention is all you need paper.

So vision as a field has contributed a lot to deep learning evolution. OK, so let me fast forward. As researchers, after ImageNet, we were thinking about what is beyond object recognition.

And this is really Ranjay's thesis work, is that the world is not just defined by object identities. If it were, these two pictures both contain a person and a llama, would mean the same thing. But they don't.

I'd rather be the person on the left than the person on the right. Actually, I'd rather be the llama on the left than the llama on the right as well. So objects are important, but relationships, context, and structure and compositionality of the scene are all part of the richness of visual intelligence.

And the image, that was not enough to push forward this kind of research. So again, heroically Ranjay was really the key student who was pushing a new way of thinking about images and visual representation, mostly focusing on visual relationships. So the way Ranjay and we put together the next wave of work was through scene graph representation.

We recognize the entities of the scene in the unit of objects, but also their own attributes as well as the inter-entity relationships. And we made a data set-- it was a lot of work-- called Visual Genome. That consisted of hundreds of thousands of images, but millions of relationships, attributes, objects, and even natural language descriptions of the images as a way to capture the richness of the visual world.

There are many works that came out of Visual Genome, and a lot of them were written by Ranjay. But one of my favorite works is this one-shot learning of visual relationships that Ranjay did where you use the compositionality of the objects to learn relationships like people riding horse, people wearing hats. But what comes out of it with the compositionality is almost for free, is the capability of recognizing long-tail relationships that you will never have enough training examples.

But you're able to do it during inference, which is like horse wearing hat, or person sitting on fire hydrant. And that really taps into the relationship as well as the compositionality of images. And yeah, there were some quantitative measurement that shows our work at that time-- now it's ancient time-- that does better than the state of the art.

We also went beyond just a contrived labeling of objects or relationships that went into natural language. And there was a series of papers started with my former student Andre Karpathy, many of you know, Justin Johnson, on image captioning, dense captioning, paragraph generation. I want to say one thing that shows you how badly at least me or oftentimes scientists predicts the future.

When I was a graduate student, when I was about to graduate, 2005, I remember it was very clear to me my life dream as a computer vision scientist was to, when I die, I want to see computers can tell a story from a picture. That was my life's dream. I feel that if we can put a picture into the computer and it will tell us what's happening, a story, we've achieved the goal of computer vision.

I never dreamed less than 10 years, just around 10 years after my graduation, this dream was realized collectively, including my own lab, by LSTM at that point, and CNNs. It was just quite a remarkable moment for me to realize. First of all, it's kind of the wrong dream to say that that's the end of the computer vision achievement.

Second, I didn't know how fast it would come. So be careful what you dream of. That was the moral of the story.

But static relationships are easier. Real world is full of dynamic relationships. Dynamic relationships are much more nuanced and more difficult.

So this is fairly recent work. It was I think at NeurIPS two years ago. And we're still doing this work on multi-object, multi-actor activity recognition or understanding.

And that is an ongoing work. I'm not going to get into the technical details. But the video understanding, especially with this level of nuance and details, still excites me.

And it's an unsolved problem. I also want to say that vision as a field has been exciting, not only because I'm doing some work in it. It's because some other people's work.

And none of these are my own work. But I find that the recent progress in 3D vision, in pose estimation, in image segmentation, with Facebook SAM and all the generative AI work has been just incredible progress. So we're not done with building AI to see what humans see.

But we have gone a long way. And part of that is the result of data, compute, algorithms, like neural networks that really brought this deep learning revolution. And as a computer vision scientist, I'm very proud that our field has contributed to this.

And AI's development has been and I continue to believe will be inspired by brain sciences and human cognition. And for this section, I'm very appreciative of all the collaborators, current and former students, and Ranjay you're a part of them, who has contributed. Let's just fast forward to building AI to see what humans don't see.

Well, I just told you humans are super good. But I didn't tell you that we're not good enough. For example, I don't know about you, but I don't think I can recognize all these dinosaurs.

And in fact, recognizing very fine-grained objects is not something humans are good at. There are more than 10,000 types of birds in the world. We put together or we got our hands on a data set of 4,000 types of birds.

And humans typically fail miserably in recognizing all species of birds. And this is an area called fine-grained object categorization. And in fact, it's quite exciting to think about computers at this point can go beyond human ability to train detectors, object detectors, that can do much finer grain understanding of objects beyond humans.

And one of the application papers we did which I find very fascinating, is a fine-grained car recognition. We downloaded 3,000 types of cars, separated by make, model, year that's ever built by 1970s, starting 1970s. We stopped before Tesla was popular.

So people always ask me this question. Where's Tesla? We don't have Tesla.

And after we trained the fine-grained object detector for thousands of cars, 3,000 of cars, we downloaded street view pictures of 100 American cities, most populated cities, two per state. And we also correlated this with all the census data that came out of 2010. And it's incredible to see the world through vision as a lens, the correlation between car detection and human society is stunning, including income, including education level, including voting patterns.

We have a long paper that has dozens and dozens of these correlations. So I just want to show you that even though we don't see it with our individual eyes, but computers can help us see our world, see our society through these kind of lenses in ways that humans can't. OK, to drive home this idea that humans are not that good, even though 10 minutes ago I told you're so good, is this visual illusion called Stroop test.

Try to read out to yourself the color of the word, not the word itself. Go left to right and top to bottom, as fast as possible. It's really hard, right?

I have to do red, orange, green, blah, blah, blah. That's a fun visual illusion. This one some of you probably have seen.

These are two alternating pictures. They look like the same but there's one big chunk that's different. Can you tell?

Raise your hand if you can. It's an IQ test. [LAUGHTER] m so all the faculty were thinking, oh no.

I didn't raise my hand. OK, so it's the engine. Oh.

OK, so it's a huge chunk. This has landed on your retina. And you completely missed it.

OK, good job. [LAUGHTER] It's not that funny, if it's in the real world, when it's a high stake situation. Whether you're passing through airport security or doing surgeries.

So actually not seeing can have dire consequences. Medical error is the third-leading cause of American patients' deaths annually. And in surgery rooms, accounting for all the instruments and glasses and all that is actually a critical task.

If something is missing, on average a surgery will stop for more than one hour, so that the nurses and doctors have to identify where the thing is, and think about all the life risk to the patient. And what do we do today? We use hand and count.

And imagine if we can use computer vision to automatically assist our doctors and nurses to account for small instruments in a surgical setting. That would be very helpful. And this is an ongoing collaboration between my lab's health care team and Stanford Hospital Surgery Department.

This is a demo of accounting for these glasses during a surgical scenario. And this would, if this becomes mature technology, I really hope this would have really good application for these kind of uses. Sometimes seeing is not just attention.

Every example I just showed you there seemed to be attentional deficit. But sometimes seeing is more profound, or not seeing is more profound. This is my really favorite visual illusion, since I was a graduate student, made by Ted Edison at MIT.

And I'm just showing you the answer. This checkerboard illusion, if you look at the top graph checkerboard A and B, no matter what I tell you they look like different gray scales, right? I mean, how could they on Earth have the same gray scale.

But if I added this, you see that they're the same gray scale. So this is a visual illusion. Even if you know the answer, It's hard to not be tricked by your eyes.

For those of you who are old enough, who do you see here? AUDIENCE: Bill Clinton and Al Gore. FEI-FEI LI: Clinton and Gore, right?

Is it? Is it Clinton and Gore? So it turned out they are Clinton and Clinton.

And it's a copy of Clinton's face in Gore's hair, and in a context, that it is very primed for all of us to see them as Clinton and Gore. So being primed is a fundamental thing of human bias. And in computer vision, we have also inherited, if we're not careful, computer vision has inherited human bias, especially through data sets.

So Joy Buolamwini used to be at MIT, had written this beautiful poem that exposes the bias of computer vision. So I'm not nearly as a leading expert as Joy and many other people are. But it's important to point out that not seeing has consequences.

And we need to work really hard to combat these biases that creep into computer vision and AI systems. And these are just really examples of hundreds and hundreds of thousands of papers and work people are doing in combating biases. Now on the flip side, sometimes not seeing is a must, as seeing too much is also really bad.

This brings us to the value of privacy. And my lab has been actually doing quite a bit in the context of health care, but quite a bit of privacy computing in the past few years in terms of how we can protect human dignity, human identity, in computer vision context. One of my favorite works that's not led by me is by Juan Carlos Niebles.

That combines both hardware and software to try to protect human privacy while still recognizing human behaviors that are important. The idea is the following. If you want to look at what humans do, you take a camera you shoot a video and you analyze it.

In this case, a baby is pushing a box. What if you don't want to reveal this kid? What if you don't want to reveal the environment?

Can you design a lens that blurs the raw signal, like you never take the pure pixel signal? What if the designed lens gives you a signal like that? So for humans, you don't even see the baby.

Well, that's exactly what they did. They designed a warped lens. And the lens gives you a raw signal in the top row.

But they also designed an algorithm that retrieves not super resolution, they have no intention to recover the identity of the people, but just to recover the activity they need to know. This way their combined hardware-software approach not only protects privacy, but also reads out the insight that whether you're in transportation cases or health care cases, that is relevant to the application users. So building AI to see what humans don't see is part of computer vision's quest.

It's also important to recognize sometimes what humans don't see is bad, like bias. But we also want to make computer not see the things that we want to preserve privacy for. So in general, AI can amplify and exacerbate many profound issues that has plagued human society for ages, and we must commit to study and forecast and guide AI's impact for human and society.

And many students and former students have contributed to this part of the work. Let's talk about building AI to see what humans want to see. And this is where really putting humans more in the center of designing technology to truly help us.

When you hear the word AI, well, you're kind of a biased audience. But when the general public hears about AI today, what is the number one thing that comes to their mind? Anxiety, right?

A lot of that anxiety is labor landscape, jobs. And this is very important. And if you go to headlines of news, every other day we see that.

But there is actually a lot of cases where human labor is in dire shortage. And again, this brings me back to the health care industry that I also work with. America was missing at least 1 million nurses last year.

And the situation is just worse and worse. I talked about the medical error situation in our health care system. The aging society is exacerbating the issue of lack of caretakers.

And a lot of these burdens fell on women and people of color in very unfair ways. Care-taking is not even counted in GDP. So instead of thinking about AI replacing human capability, it is really valuable to think about AI augmenting humans, and to lift human jobs, and to also give human a hand, especially health care from a vision perspective.

There are so many times and so many scenarios that we're in the dark. We don't know how the patient is doing. We don't know if the care delivery is high quality.

We don't know where that small instrument was missing in the surgical room. We don't know if we're making a pharmaceutical error that might have dire consequences. So in the past 10 years, my lab and I and my collaborators have started this fairly new area of research called ambient intelligence for health care, where we use smart sensors, mostly depth sensors and cameras, and machine learning algorithms to glean health critical insights.

Most of this earlier work was summarized in this Nature article called "Illuminating the Dark Spaces of Healthcare with Ambient Intelligence. " I'll just give you a couple of quick examples. One case study is hand hygiene.

We started this work way before COVID. Everybody thought this is the most boring project. But when COVID came, it became so important.

It turned out that hospital acquired infection kills three times more people in America than car accidents every year. And a lot of that is because of doctors and nurses carrying germs and bacteria from room to room. So WHO has very specific protocols for hand hygiene.

But humans make mistakes. And now the way to monitor that by hospitals is very expensive, sparse, and disruptive. They put humans in front of-- I don't know the patient rooms, and try to remind the doctors and nurses.

You can see this is completely non-scalable. So my students and I have been collaborating with both Stanford Children's Hospital and Utah's Intermountain Hospital by putting depth sensors in front of these hand hygiene gel dispensers, and then using video analysis and activity recognition system to watch if the health care workers are doing the right thing for hand hygiene. And quantitatively, the bottom line is the ground truth of human behavior.

You can see that the computer vision algorithm's precision and recall is very high compared to even human observers that we put in the hospital in front of the hospital room. Another example is ICU Patient Mobility Project where we getting patient to move in the right way in the ICU is really important. It helps our patients to recover.

And on top of that, ICU is so important. It's 1% of US GDP is spent in ICU. Health care is 18%.

So this is where patients fight for death and life. And we want to help them to recover. We work with Stanford Hospital to put these sensors, again RGBD sensors in ICU rooms.

And we study how the patients are being moved. Some of the important movements that doctors want patients to do include getting out of bed, getting in bed, getting in chair, getting out of chair, these things. And we can use computer vision algorithm to help the doctors and nurses to track these movements and so on.

So this is, again, a preliminary work. Last but not least, aging in place. Aging is very important.

But how do we keep our seniors safe, healthy, but also independent in their living? How do we call out early signs of whether it's infection or mobility change, sleep disorder, dietary issues? There are so many things.

It's computer vision plays a big role in this. We are just starting to collaborate actually with Thailand and Singapore right now to get these computer vision algorithms into the homes of seniors, but also keeping in mind the privacy concerns. So these are just examples.

Last but not the least, I'm actually still very excited by the long future where I think no matter what we do, we probably will enter a world where robots collaborate with humans to make our lives better. So ambient intelligence is passive sensors. It can do certain things.

But eventually I think embodied AI will be very, very important in helping people, whether it's firefighters, or doctors, or caretakers, or teachers, or so on. And technically, we need to close the loop between perception and action to bring robots or embodied AI to the world. Well, the gap is still pretty high.

This is a robot. I think-- I don't know. It's a Boston Dynamics robot or some kind of robot.

It's a pretty miserable robot trying to put a box and miserably failed. And I know there are so many-- robotic research is also really progressing really fast. So it's not fair to just show that one example.

But in general, we are still a lot of robotic learning and robotic research right now is still on skill level tasks, short horizon goals, and closed world instruction. I want to share with you one work that at least was attempted towards robotic learning to open world instruction. It's still not fully closing all the gap, and I don't claim to do so.

But at least we're working on one dimension. And that is some of you know our work VoxPoser, just released half a year ago. Where we look at a typical robotic task such as open the door, or whatever, a robotic task in the wild.

And the idea in today's robotic learning is you give a task, and you try to give a training set, and then you try to train an action model. And then you test it. But the problem is, how do you generalize?

How do you hope in the wild generalization? And how do you hope that instruction can be open world? And here's the result.

The focus of this work is motion planning in the wild or using open vocabulary. And the idea is to actually borrow from large language models. From large language model, to compose the task, and from also a visual language model to identify the goal and also the obstacles, and then use a code generated 3D value map to guide to do motion planning.

And I'm not going to get into this. But quickly, so once the robot takes the instruction, open the top drawer, you use LLM to compose the instruction. And because the LLM helps you to identify the objects as well as the actions, you can go use a VLM, visual language model, to identify the objects that you need in the world.

Every time you do that, you're starting to update a planning map. And it helps to, in this case you identify the drawer. The maps sets some values and it focuses on the drawer.

And if you give it an additional instruction of watch out for the vase, and it goes back to LLM and goes back to VLM, and they identify the vase. And then it identifies the planning path with the obstacle, and updates the value map, and recomputes the motion map, and do it recursively till it has more optimized this. So this is the example we see in simulation in real world.

And there are several examples of doing this for articulated objects, deformable manipulations, as well as just everyday manipulation tasks. OK, in the last three minutes, let me just share with you one more project, then we're done. Is that even with VoxPoser, which I just showed you, and many other projects in my lab, I always feel in the back of my mind that compared to where I come from, which is the visual world, is these are very small scale data.

Very small scale anecdotal experimental setup, and there is no standardization, and the tasks were more or less lab specific. And compared to the real world which is so complex, so dynamic, so variable, so interactive, and so multitasking it's just unsatisfying. And how do we make progress in robotic learning?

Vision and NLP has already shown us that large data drives learning so much, and the kind of effective benchmarking drives learning. So how do we combine the goal of large data and effective benchmarking for robotic learning has been something on my mind. And this is the new project that we have been doing.

Actually, it's not so new anymore, for the past three years called BEHAVIOR, benchmark for everyday household activities in virtual interactive ecological environments. And let me just cut to the chase. Instead of small anecdotal tasks that we want to train robots on, we want to do 1,000 tasks, 1,000 tasks that matter to people.

So we started actually by a human centered approach. We literally go to thousands of people and ask them, would you like a robot to help you with-- so let's try this. Would you like a robot to help you with cleaning kitchen floor?

Yeah, sort of, mostly. OK. Shoveling snow?

Yeah. Folding laundry? AUDIENCE: Yeah.

FEI-FEI LI: Yeah, OK. Cooking Breakfast [INTERPOSING VOICES] FEI-FEI LI: OK, I don't know. I get mixed-- Ranjay wants everything.

I get mixed reviews. OK, this one, opening Christmas gift? AUDIENCE: No.

FEI-FEI LI: Right, yeah exactly. OK, I'm glad you're not a robot, Ranjay. So we actually took this human centered approach.

We went to the government data of American and other countries human's daily activities. We go to crowdsourcing platform like Amazon Mechanical Turk. We ask people what they want robots to do.

And we rank thousands of tasks. And then we look at what people want help with, and what people don't want help with. It turned out cleaning, all kinds of cleaning people hate.

But opening Christmas gift or buying a ring, or mix baby cereals, is actually really important for humans. We don't want robots help. So we took the top 1,000 tasks that people want robots help, and put together the list for BEHAVIOR data set.

And then we actually scanned 50 real world environments across eight different things, like apartments, restaurants, grocery stores, offices, and so on. And this compared to one of my favorite works from UW, Object Verse, is very small. But we got thousands and thousands of object assets.

And we created a simulation environment. OK, all right. I want to actually give credits to a lot of good work that came out of UW and many other places.

So robotic simulation is actually a very interesting area of research and excellent work, like Ai2THOR, Habitat, Sapien has been also making a lot of contribution. We collaborated with NVIDIA, especially the Omniverse group, to try to focus on creating a realistic simulation environment for robotic learning that has the good physics, like thermal transitional lighting and all that; good perception which we did some user studies to show that we have very good perceptual experience; and also just interactions. And I'm not going to get into all the details.

We did some comparisons and show the strength of this BEHAVIOR environment for training 1,000 robotic tasks. And right now we are working on a whole bunch of work that is involving benchmarking, robotic learning, multi-sensory robotics, and even economic studies on the impact of household robots. And OK, I actually want to say one thing I'm not showing here.

Is that we are actually doing brain robotic interfacing, using BEHAVIOR environment to use EEG to drive robotic arms to show the brain robot interface. And that was just published this quarter. So I didn't include this slide.

So BEHAVIOR is becoming a very rich research environment hopefully for our community, but at least for our lab's robotic work. And of course, the goal is one day we'll close the gap between robotics and collaborative robots, home robots that can help people. And this part of the research is really trying to identify problems, whether it's health care or embodied AI, where we want to build the AI to see and also to do what humans want it to, whether it's helping patients or helping elderlies.

And I think that's the key emphasis is really augmentation. And a lot of collaborators have participated in this part of the work. This really summarizes the three phases of our work or three different types of our work, and all of this have accumulated to what I would call a human centered AI approach, where we recognize it's so important to develop AI with a concern for human impact.

It's so important to focus AI to augment and enhance humans. And it's actually intellectually still important to be inspired by human intelligence and cognitive sciences and neurosciences. And that was really the foundation of Stanford's Human Centered AI Institute that I co-founded and launched five years ago with faculty from English, Medicine, Economics, Linguistics, Philosophy, Political Science, Law Schook, and all that.

And HAI has been around for five years now almost. We do work from digital economy to Center for Research for Foundational Models, where some of our workers-- like Percy, Chris-- you guys all know them-- are at the forefront of benchmarking and evaluating today's LLMs. And we also work with faculty like Michael Bernstein, some of him very well, on creating ethics and society review process for AI research.

And we also focus on educating not only ethics focused AI to our undergrads, but also really bring that education to the outside world, especially for policymakers, as well as business executives. And we directly engage with the national policy, Congress and Senate and White House to advocate for public sector AI investment, especially right now. In fact, UW is part of the partner and also senators from Washington state are extremely important for this is to advocate the next bill for national AI research cloud.

So this really concludes my talk. That was a pretty dense quick overview of a human centered approach to AI, and I'm happy to take questions. [APPLAUSE] One more slide.

PRESENTER: We have time for maybe two questions. AUDIENCE: What do you think the most interesting breakthrough in the next 5 or 10 years is going to be in computing? FEI-FEI LI: The question is, what do I think the most interesting breakthrough in the next 5 or 10 years.

I just told you in the talk, I'm so bad at predicting. So I think the two things that does excite me, one is really just deepening AI's impact to so many applications in the world. It's not necessarily yet another transformer or anything.

It's just that we have gotten to a point, the technology has so much power and capability. We can use this to do scientific discovery, to make education more personalized, to help health care, to map out the biodiversity of our globe. So I think that deepening and widening of AI applications or from an academic point of view, that deepening and widening of interdisciplinary AI is one thing that really excites me for the next 5 to 10 years.

On the technology side, I'm totally biased. I think computer vision is due for another revolution. We're at the cusp of it.

There's just so much that is converging. And I'm really excited to see the next wave of vision breakthroughs. PRESENTER: Go ahead.

AUDIENCE: So large language models have been impressive because of what they have been able to do with semantic understanding. What do you think the frontier for image, computer vision is in that respect? FEI-FEI LI: Yeah.

This is a very good question. The question is large language model is really encoding semantics so well. What's the frontier of image?

So let me just say something. First of all, the world is fundamentally very rich. Its language-- Ranjay, don't yell at me.

I still think language is a lossy compression of the world. It is very rich. It goes beyond just describing the world.

It goes into reasoning, abstraction, creativity, intention, and all this. But much of language is symbolic, is a compression. Whereas the world itself in 3D in 4D is very, very rich.

And I think there needs to be a model. The world deserves a model. Not just language deserves a model.

There needs to be a new wave of technology that really fundamentally understands the structure of the world. PRESENTER: OK, we have time for one more. Go ahead.

AUDIENCE: I really agree that language can be lossy, like compression of the real world. I'm just wondering, what's your opinion on just how English as a whole is just so much like dominating the research field itself, like all these labeled data sets are labeled in English, while other language might have different ways of describing objects, describing the relationship between objects? That lack of diversity, how do you feel about it?

FEI-FEI LI: Right. So the question is about bias of English in our dominating data sets of our AI. I think you're calling out a very important aspect of what I call the inherited human bias, right?

Our data sets inherit that kind of bias. I do want to say one thing. This is not meant for defense.

It's a fun fact that when we were constructing ImageNet, because the ImageNet was-- George Miller made this lexicon taxonomy in many languages. It was so nice and easy to map the synsets of English ImageNet to French, Italian, Spanish, Portuguese. I think there are also Asian languages we used.

And so even though ImageNet seemed English to you. The data comes from all languages, we could get our hands on the license. But that doesn't really solve the problem you're saying.

I think you're right. I mean we have to be really mindful, even in the BEHAVIOR data set, when we're looking at human daily activities, we started with the US government data. We realized we're very biased.

First of all, you realize you're biased because there's so much TV watching in the data. And then we actually went to Europe. But that does not include the global South.

So we're definitely still very biased. PRESENTER: OK, I think that's all the time we have. Let's thank Fei-Fei.

FEI-FEI LI: Thank you.