[Music] la [Music] for us Todd can you hear us oh yeah we can hear you oh fantastic great are we oh how funny I isn't this early you're you're actually on so you're you're beautifully there and uh you can you know take off if you're ready oh I see great okay well give us give us one second then we this is my God the we we know you're not in the US because you're running 5 minutes early that's that's fantastic well we were running 40 minutes late so somehow we time warped and now we're earlier
so oh my gosh however that worked um we can start with your video if you would like and then you can get ready that's okay I think um I think we're we're probably ready then okay well then then let's take the stage I'm going to I'm going to leave the stage to the side and you're there I I can't hear you too well can you hear me I can hear you Fant fantastically fine yes fantastic good well okay then the St your yeah just just go ahead go for it go for it I don't know
if you made an introduction or not but uh I'm there you go you get the stage Applause oh hello hi everybody uh I'm Todd it's uh really a pleasure to speak with you today I'm very sorry that we're not with you in Munich but we're here at the MIT media lab I guess that's a picture of it but uh Peter can you just swing the camera just for a second second if uh people want to see we're in this uh it's a brand new building uh designed by fumihiko makaki it's uh glass filled with light
a very interdisciplinary lab and uh okay Peter come back to me and um what I want to talk about today uh is about what our group does uh it's called the hyper instruments and Opera of the future group and over the years we've looked at probably more ways than most people about how music can touch people's lives as deeply as possible uh both the most sophisticated music and also music that uh can be accessed by anybody and we've looked at ways that technology can build Bridges to human expression and also to um Human Performance and
creativity through music so um what I want to talk about today are basically two things a little bit a glimpse of some ideas about where I think music might be going because of technology and a little bit about our latest project it's an opera called death and the powers uh with a bunch of robots and we'll show you some video at the end so uh the work that we do to make tools for music making uh started about 20 years ago uh with a project with the cellist Yo-Yo Ma uh yo-yo came and asked us
to think about making a new kind of cello and I play cello myself was very interested in that project and I thought back to when I was a kid probably a lot of you uh know what what this is yeah yes no so when when I was a kid uh Sergeant Peppers by The Beatles came out as you might remember Sergeant Peppers was the first Beatles CD album uh that the Beatles could not play live they designed it to be made in a studio and the sound of the album the music itself uh was a
real inspiration to me but it also bothered me that this fantastic music could only be made offline in a digital Studio seemed like that was the wrong way for technology to go so when we started working with yo-yo we started thinking about how to combine the live performance of a great artist with all the intricacy that you can do in a digital recording studio and we came up with a new technology that we've been developing for about 20 years now called hyper instruments and uh the basic idea of a hyper instrument is um to M
take an instrument uh like a traditional instrument so this this actually is the cello that we built for Yo-Yo Ma uh you can see it looks like a cello it has strings a fingerboard a bridge but it doesn't have any F holes it's it has a funny shape uh that's because we put all kinds of sensors into the instrument it measures the audio it measures the vibration of the instrument and uh we do a lot of work measuring the physical gesture of performance so this for instance is the bow that we designed for yo-yo uh
you can see it has Electronics built into the bow uh little circuit boards on the Frog of the bow switches pressure sensors and actually um it sends out an electric signal at both ends of the bow the cello measures the signal and software lets me understand everything about the expression of BOE so the speed how much bow I'm using uh is a lot of pressure a little pressure even picado and uh the idea is that I the instrument knows what music is being played but also how it's being played the expression so when someone like
yo-yo or any other musici we've done this for classical music for rock music depending on how the expression changes if it's faster slower more intense calmer bring out this note not that note uh the instrument understands and can change and morph it can become a voice or a flute a single line can become a whole Orchestra depending on how it's played so we develop that kind of instrument for for virtuosic musicians and then we realized that the same techniques for measuring Boeing and the same software uh could be used for anybody to make music didn't
have to be just Yo-Yo Ma so we started a whole line of work about 15 years ago to make musical experiences for the general public so um we make instruments like uh this is a rhythm instrument made out of rubber with little sensing pads in it I can make delicate or very uh loud rhythms uh squeezy instruments like this music shaper I can make sounds especially by children just by squeezing or touching this pad a rhythm instrument like this beat bug which lets me tap a rhythm and modify it using these little Bend sensors and
uh we developed a whole range of creative activities and and performance activities for the general public with this kind of instrument uh software like this called hypers score uh lets anybody use these lines and color to create original music and then I can push a button and it transcribes that music for regular musicians to play like orchestras or rock bands and that's used more and more by children by adults by various people and uh this kind of work uh surprisingly the cello uh and the cello bow led to these squeezy instruments uh which then led
to kind of a surprise a couple of my students uh Aran Ag gozi and Alex rolis set up a company down the block and invented rock band and Guitar Hero uh they still have this company harmonics that builds these so the general idea of measuring expression for performance uh led to high-end performance performance for everybody and now to you know a pretty large um public commercial activity uh that lets people make music but this is in itself uh growing very fast so a couple of places that this kind of work is going next I think
is um You probably haven't heard of a company called echonest if you haven't you should Google it uh later today uh econest is another small company that grew out of uh the media lab this company analyzes any music any audio and it analyzes uh Tristan Jan is a specialized specialist in analyzing the sound of the audio Brian Witman looks at what people say about the audio on the internet and together it takes any audio and can tell why it makes an impression on any of us why why it gives us a certain feeling so this
software is now becoming very popular for deciding uh preferences for music if I like this music I'll like another piece of music but it also allows me to take many pieces of music and morph them together to combine them it'll do the same thing with image so it's probably the most powerful package right now for allowing people to manipulate audio in a variety of intuitive ways in turn uh this has led to a company called RJ DJ which you may have heard about they make mobile apps for uh from music this is one of the
most interesting new areas in music what happens with RJ DJ is they take the echon platform they commission musicians usually Young Musicians to write new music and the idea is the music goes on a mobile device and it's not totally finished when it gets to your mobile device so it might be a piece of music that every time you turn it on it starts with the same feeling the same themes the same texture but then it plays out it finishes in a different way every time uh an app like Inception the app which is one
of the most popular apps on the iPhone right now uh takes the soundtrack from the music from the movie Inception puts it in this RJ DJ form and it constantly plays differently if you shake your iPhone the music develops in a certain way if you tilt your iPhone it develops differently it also takes in audio so if you talk to it sing to it uh takes Ambient sound from around you it takes the sounds of your everyday life and turns it into music so this idea of a musical composition which is a collaboration between a
composer a performer and the audience where the music is always different uh is I think a field which is developing very quickly that we intend to work on the general idea of having music adapt to an individual has led us to think very seriously about uh music not just being a fun activity but being something that can really change our minds and change our bodies it can change our health so we have a large body of work here at the media lab which is looking at different diseases es uh just definitely uh your your ordinary
emotional state but also diseases like Parkinson's Strokes Alzheimer's disease uh there's growing evidence and growing research that shows that music has a very very profound and important effect on all of these diseases so music can help you move for Parkinson's Disease where people can stop moving uh music seems to be the only thing that can help you move again Strokes where you lose the ability to move your body especially fine motor control uh music seems to be the best way to restore movement and also to restore speech um music has a very important connection to
our memories music is the only thing that advanced Alzheimer's patients can still remember so we're using music more and more to restore people's memories through their association with music and music is simply one of the best ways for us to communicate and connect with other people so there are many many many contexts where music can be the only way that a person can actually communicate who they are and what they are and what they're feeling to somebody else so one of our long-term collaborators is Dan Elsie who's a young man who lives at a hospital
in the Boston area who has cerebral paly from birth he cannot move very well he cannot speak at all but music is the way he communicates uh we helped develop a tracking device for his head uh that allows him to use hypers score uh to make music compositions and then for the Ted conference the uh three years ago uh you could see this online maybe you've seen it already uh we developed a whole system that uses his head tracking device that measures the movements he's making based on knowledge of how he wants to interpret a
composition and with this information he can sit on stage and perform his own compositions even though he can't always give a beat exactly where he wants to the system knows what he's trying to do and it allows him to communicate through his composition and also through his performance um and so we're uh pushing further with instruments like this the interesting thing is that this whole body of work is what we think of as a personal instrument because the instrument we designed for Dan is not like a cello for Yo-Yo Ma that any chist can play
or a or a squeezy toy for a child this instrument is designed specifically for the skills that Dan has and also the limitations he has so it's it's it's a personal instrument and I think the next stage uh which I'd like to suggest to you is the development of a field of personal music music which is customized for a particular person uh not music which is uh designed to be generally attracted to as many people as possible so another uh new company uh that is associated with the media lab some of our former students started
it uh is called Source tone you can also look this up it's the first company that I know of that allows you to go online to take answer a series of questions about how you're feeling at a particular moment and then it will generate totally new music that doesn't exist before uh based on it's it's like it's like a musical psychiatrist it generates this music based on what it thinks would be most beneficial for you that's of course a big question if I'm depressed do I want Happy music depressing music subtle music what is it
but that's what this company tries to do it's a brand new company you can go online and check it out and I think it's quite interesting but I think the next step is that based on the growing knowledge we have of exactly what goes on in Our Minds when we listen to music why it is that it makes us feel a certain way what aspects of Music correspond to what reactions uh we're going to more and more use this information to be able to customize music not in a general way but in a very particular
way so that a piece of music will have a maximum effect uh that's a cartoon of me you can probably tell um maximum effect on a particular person at a particular time based on your experience your genetics your knowledge of music and what you need and feel at a particular moment so this feel of personal music is what we're working on right now and uh I think what we'll devote our attention to over the coming years um now I'd like to switch to a particular form of personal music and a kind of story about one
man's relationship to music uh it's the opera called death in the powers which we finished uh just about a year ago and um it's we often make large music projects to test out a lot of these ideas we often do them in Opera form because we can bring together music and visuals and technology and story and character all together uh death in the powers is one of the most unusual operas we've done uh it's called the robots Opera so it has not only robots on stage but the whole stage is a kind of robot um
we did uh had fantastic collaborators on this project Robert Pinsky uh wrote the text one of the Great American poets Diane paus is one of the best young theater directors in the United States Alex McDowell is one of the top production designers in Hollywood designs all of Spielberg's movies a lot of movies with um very imaginative sets and uh we all came together to to make this Opera premiered it in Monaco uh Prince Albert was the um honorary Patron last September uh brought it to Boston in March to Chicago in April and uh now it's
uh it's going other places it's a story of a man in his mid to late 60s Rich powerful successful it's also a little bit strange kind of like Bill Gates combined with Walt Disney combined with Howard Hughes maybe Steve Jobs in there a little bit somebody who uh wants to live forever and actually he's tired of the world he wants to leave the world he wants everything about himself to stay so he figures out a way to build something called the system where he can download everything about himself into his environment and then he personally
can leave so the main character uh is a famous singer named James Melina at the beginning of the Opera he's just about to turn on the system so that he can leave and um in fact he does turn on the system he's disintegrates on stage with a bunch of theatrical tricks and he leaves physically the stage you don't see him again as a singer until the end of the Opera uh and the whole stage little by little comes to life so the main character in this Opera really is the stage uh his family his wife
Evie his daughter Miranda and his kind of research assistant Nicholas are all left to try to decide if they like Simon in this new form if he's really still human if they can communicate with him if the system is great and they should follow him uh exactly what is the meaning of this the whole stage as I said comes alive it has three gigantic walls each wall is like a robot it has eyes and ears so it's choreographed and moves on stage has visual displays on each side of the wall um the um the walls
sometimes show pictures but we we try not to show real pictures you see kind of a shadow of people uh often we use them to show the soul of Simon Powers uh his feeling at any moment so you often get like a gigantic burst of light or gesture on the walls and uh to do this since Simon Powers is offstage uh we invented a new technique called disembodied performance which is the next extension of the kind of work that we did years ago with yoyo Ma and uh the best way to to show you about
this is maybe to give you a quick uh demonstration so I'd like to introduce uh if you step in front of the camera uh Ellie Jessup and I'll give Ellie the microphone too um and Ellie's a doctoral student in our group at the media lab and also uh Peter torpe Peter you want to step in front of the you want to wave that's Peter uh Peter's also a doctoral student in our group and uh they've been working on all the aspects of disembodied performance and uh let Ellie tell you about it thanks so as Todd
mentioned one of the kind of key theatrical issues in this production of death and the powers is that our main character dies in the very first scene and transforms himself into the theatrical set so we said how can we capture that live performance and turn it into something non-anthropomorphic that you can still see and relate to so right now I'm wearing a variety of sensors these are the sensors that our main character James Melina has and behind me I have a visualization of one of the things on the LED display walls of the set so
you can see it's reacting to my voice but also to my movement and my breath so all of this together combines to you get the sense of life and Humanity in a character even though he's not physically present Peter want you say a little about the software sure we had to develop a number of control systems for various aspects of of the Opera in general one for choreographing robots but also for interpreting this live performance data and allowing it to control and shape the movement of all the set pieces and the visuals that you see
on the walls and how the performance affects the sound the spatialized sound of the singer's voice who's off stage you know what since we're standing here um we're going to show you a video in a second Ellie do you want to just since you worked on the chandelier you want to point out the chandelier absolutely so as you may have noticed in the tour around the media lab we also actually have this chandelier Which Giant musical instrument one of the ways in which Simon Powers ends up communicating with his wife Evie uh it starts out
hanging above the stage but in the middle of the Opera he inhabits it it comes down and she discovers that it's actually a musical instrument the Teflon strings have sensors around the top so they can detect as she Strokes the strings dampens them plucks them it can excite clusters of strings and shape his voice and the Sonic environment through her performance being able to basically have a very romantic encounter with the chandelier and also here we have one of the robots from the Opera so there are about nine of these on stage at a time
and they act sort of like a Greek chorus they're actually retelling the story of the human characters Simon Powers they can move around the stage freely uh they're completely wireless and autonomous and they can translate move they can Elevate from about 3 and 1/2 ft to 7 ft tall and have lighting throughout them to be expressive and since I know uh time is uh short uh maybe we can run the first video I have two short videos one's a very short video that uh shows these robots in action and uh maybe we can run that
and then I'll just uh say a word um and we can um we can show a little bit from the Opera so can we show video [Music] one tonight we delve into a [Music] world Opera is about the moment when music and voices singing in particular becomes drama how can we make robots that will be alive and expressive we wanted these robots to feel like they had human reactions but didn't look anything like [Music] humans so if I'm I don't know if I'm still there but I know that we're slightly overtime Sunny can we run
two minutes excerpt from the Opera or should we stop there yeah you can just do it just go ahead and and you're still there okay thanks so much so here's just a two-minute excerpt to give you a sense of the Opera and then we'll sign off and Say Goodbye [Music] [Applause] [Music] [Applause] [Music] I have billions of bucks and I can still sign [Music] tracks in the [Music] [Applause] [Music] I came from light and I will return to [Music] light I miss [Music] father [Music] [Applause] [Music] thanks so much sorry for being a little late
and I hope this gives you a little bit of a glimpse of how we're thinking about one possible future of Music uh both for professionals high-end music experiences and for everyone else thanks very much w
