Psych 2134 - Week 4 (part 2 of 2)

Okay so we're back with uh a second recording after what happened last week where I lost part of a recording I feel as though I don't fully trust my computer or fully trust my zoom anymore to save files for me uh so maybe we'll be seeing more two-part lectures in the future we've now gone over uh the physical properties of sound the anatomy of the ear and the acoustic features of speech That can be viewed within a spectrogram I wanted to spend a little bit of extra time on spectrograms because for myself that's something I've

always struggled with so I wanted to give you a little bit more information and let that simmer a little bit more uh so hopefully you understand it a bit better than I did initially we'll next talk about the challenge of decoding speech 1950s scientists became very interested In developing a computer that could read the blind so at Hoskins laboratory in New Haven Connecticut they were looking at how to move an electric eye over a page that would translate the letters on the page and produce a distinct sound for each letter one by one okay it

turns out that that experimental Endeavor was actually quite terrible it sounded awful so the idea then that phonies or like Beads on a string that they could be discrete elements and then just strung together was shown to be false phonemes in spoken language do not exist distinctly from one another when you try and to cut and paste phonemes together within a speech synthesis synthesis system what you end up with is completely incoherent speech is produced in a way uh where phonemes are influenced by what comes Before them and what comes after them the acoustic cues

of individual phonemes are altered by their context they do not exist independently from each other this leads us to discuss something that's known as the lack of invariance the lack of invariance means that there is no consistent acoustic cues that tell us what phoneme we are hearing in a given instance so turning that phrase on its head the lack of invariance means There is a lot of variance and how specific phonemes can be produced the context in which a phoneme is produced is used in perceiving that phoneme so for example we're now familiar with formants

formant Transitions and study State formants and so we can look at a couple of different Productions our different syllables containing the buff phoneme so we have B Bay and bow and what we can actually see is that the performance for the buff phoneme are different in each of these examples we don't need to get much more granular than that other than recognizing that those formant transitions look different and that is because the consonant is changed by the vowel that follows it so this is the lack of invariance problem that you can't simply clip the B

Sound out of the word bat and put it into the word bet because those ba sounds are influenced by the vowels that come before them so the ba it bet sounds different than the ba in bat and the reason for this is something called co-articulation co-articulation is the idea that you're not producing static and discrete sounds although you are planning to produce a specific sound say the ba sound here is in the word Bag what you're actually doing is producing sounds that combine all three of these noises together one after another so they're kind of

smushed together they're really lying on top of one another so fell means overlapping in the speech stream is co-articulation and what this causes is variable production of phonies a phoneme will vary in different contexts depending on what other Phonemes are neighboring it the benefit of this is it increases our rate of speech we can say more words in less time and there's this lovely quote from Hockett um I think this should be 1955 and not 1995 as I have on the slide here um this lovely quote from the textbook from Hockett who we're familiar with

um described who described producing speech as like putting eggs onto a conveyor belts that you're not then Extracting one egg at a time but that instead those eggs are getting mashed together and that so what comes out the other end is eggshell yolk and whites all mixed together and speech perception is piecing all of that back together taking that mess and turning it in to the speech that we as human listeners are capable of perceiving so it really is this decoding problem really is a complex problem Another way of thinking about this co-articulation is to

think about perceptual invariance so the k sound and Cat and caught sound different but your brain hears or perceives in the same way but acoustically they are different they differ in their physical properties of the sound so the sensation of those sounds within your brain within your auditory system will be different but the perception of Those sounds will be the same scientists are able to look at these physical differences and really precise detail using things like spectrograms and other methodologies and show that the K and Cat and caught are physically different and our ability to

accommodate this perceptual invariance is the our ability to perceive sounds that have highly variable acoustic manifestations As instances of the same sound category That we are treating the cup in cat and caught as the same phoneme even though they physically differ we perceive them as the same phone are mental categories actually warp our perception of sounds this is as it turns out a very efficient strategy so instead of us hearing all of the subtle distinctions between the cup and capped and caught and can and Cam and so on our brain treats anything within the Category

of the k sound in the same way this allows us to then ignore differences that are not meaningful foreign so what are the nature of these phoneme categories how does our mind deal with all of these variations how do we set up these category boundaries we do this through a process known as categorical perception so I'll get into categorical perception but I just want to talk about one Physical property of speech first to kind of set the stage for categorical perception just so we'll talk about voicing because voicing is one way in which we can

articulate consonants in different ways so voicing is one characteristic that kind of drives the creation of different consonant categories we're going to talk about speech production more next week so this is Really background um don't dig into this quite yet wait for next week um I just want you to know that the phoning pairs that you're seeing here so the pup and park and the Embark are produced by the same movement of the articulators how we bring our lips together how we move our tongue and our teeth how we move how we move our tongue

and our teeth the only difference is whether or not there's vocal cord Vibration so with a voiced consonant you are vibrating your vocal cords so you can put your fingers I shouldn't appoint a turtleneck today put your fingers actually onto your throat you can say bark and as you're producing that phoneme you can feel the vocal cord vibration so that's a voiced consonant in park there's no vocal cord vibration so that is a voiceless or sometimes called an unvoiced or a d voiced Consonant you can go down and try out the different pairs so the

da da as in down versus as in town so similar articulation different patterns of voicing so these consonants are coupled in how they are articulated they're the same in a lot of different properties the only difference is whether they're voiced or voiceless And so this is an important characteristic to know about to understand our next example looking at categorical perception so some consonants are voiced some consonants are voiceless they can otherwise be the exact same so we can create different categories of sounds based on this feature of voicing so now we're going to talk about

the acoustic cue not just the art not the articulation the articulatory queue That's the last slide but the acoustic cue the main dependent variable is voice onset time in this case so how are we describing the physical property of sound we're using the dependent variable of voice onset time voice onset time is the onset of acoustic phonation or acoustic energy so it's the beginning of your vocal cord vibration voicing is that vibration of the vocal cord So voice onset time is how long does it take before you start vibrating your vocal cords so we've talked

about on the last slide how as in Bear is voiced so the buff and bear that's what's known as a short voice onset time there's a very brief lag between the start of the buff sound and the start of vocal cord vibration compared to the voiceless consonant as in pair Huh has no voicing within it you're not vibrating the vocal cord so the start of the acoustic energy in this case is actually at the start of the vowel in pair so the phonation in this case starts about 50 milliseconds after you release that puff sound

so you've got a longer voice onset time so voice onset time is how long does it take for the voicing to start we generally have two broad categories Of this short voice onset time and long voice on set time short voice onset time corresponds to voiced consonants like the buff and bear where you vibrate your vocal cords long voice onset time corresponds to voiceless consonants like the puff in pair where you're not vibrating your vocal cords to produce that P sound the only difference between these words is a very small acoustic break between the initial

consonant And the vowel that comes after it it is on the order of about 40 milliseconds there's no difference in your articulation so that's something that's really interesting that our mind can treat these as separate categories even though the physical distinction between them is so subtle so giving you here a couple more examples voice onset time is the leg between the release of a stop consonant and the Start of the vowel so like and bear you can have zero voice onset time or no lag in voice onset time where the onset of vocal cord vibration

approximately coincides with the plosive release with the consonant release with the air release so this zero lag creates the perception of hearing ba and so we'll hear an example of this show my sounds on so you're hearing bot in that example Or like in pair on the previous slide you can have a longer leg between the release of the stop consonant and the start of the vowel so this is sometimes called positive voice onset time or long voice on set time so there's a delay in the onset of vocal cord vibration this longer release creates

the perception of hearing paw paw paw Paw so we hear Paul now another category exists but we don't really use this category in English we also have negative voice onset time we're the onset of vocal cord vibration actually precedes it comes before the plosive release so I'll play that for you here so this is something that's used in the French language if you speak French um you may be able to hear this Distinction a little bit more clearly so and let's see we'll go back to our zero voice onset time so in isolation you might

hear that that those are a bit different but you can imagine maybe in fluent speech um if this is something a category you're not used to hearing your brain's not really used to hearing that it may not treat it as a separate category but in English we have zero voice answer Time and positive voice onset time it's distinct categories so all these three examples sound like categorically different phonemes but the only thing that's been manipulated is the voice onset time and it's only manipulated by a handful of milliseconds so like with those consonant pairs we

saw a couple slides ago the articulation in all these cases for BA and PAW the articulation is the same the only thing that's differing is that voice onset Time sorry it's going to play through all of these I'll just skip that okay as it turns out you don't need to have a voice onset time of exactly zero milliseconds or 50 milliseconds to perceive different phonies you can manipulate poissons at time so it's a little bit shorter or a little bit longer so you could have a voice onset time of 10 20 um or 10 or

20 milliseconds and you may Still you will still perceive that as a short poisonset time and you could have it a bit longer it's 30 40 or 50 milliseconds and you'll perceive that as a longer voice onset time voice onset time is a continuous variable it's time time is a continuous variable so experimentally voice onset time can be manipulated in a continuous fashion As a researcher you could play different sounds for your research subjects where voice onset time varies by zero or um or the stepwise function of you know 0 or 10 or 20 30

40 milliseconds so on and so forth but even though those acoustically differ in a continuous fashion we perceive those sounds categorically we don't perceive them continuously This is a super interesting phenomenon known as categorical perception categorical perception happens when you ignore small acoustic variations and how something sounds in order to better perceive it as a category hear the categories we're talking about are categories of consonants we use categorical perception for other kinds of stimuli in our environment as well so if we look at this graph we've got how people responded in this Experiment so this

was an experiment by Lieberman and colleagues a number of different experiments that were conducted between the 1950s and six and 60s and they've got the percentage of people who said that this phoneme is a bar so higher percentages mean people said it was ba lower percentages mean people said it was ha you've got voice onset time which varies along the x-axis so in 10 millisecond Increments you can vary voice on set time linearly so you know 10 20 30 40 50 millisecond increments what's interesting is that when you play these sounds for people play these

consonants with the different poissons at times people actually when you ask them okay which consonant was this treat these categorically they don't treat them linearly so from 0 to 20 milliseconds if the Voice onset time is anywhere between 0 to 20 milliseconds people said pretty reliably almost 100 that sounds like a ba and um if the voice onset time varied between 30 to 50 milliseconds people responded pretty reliably almost 100 saying that sounds like a paw what's interesting is at this kind of you know midline between the two categories um you know Um the difference

between the 20 millisecond voice sounds that time and 30 millisecond poissonset time is equal to the difference between the 30 millisecond voice onset time and the 40 millisecond poissons that time but people don't treat that differences equal they treat that as a distinction between two different categories you have this jump from people saying it's a ball to people saying it's a paw and you don't really have much in between So this non-linear distinction we make between phonemic categories of consonants is US ignoring small irrelevant distinctions of sounds and paying a lot more attention to the

subtle categorical shifts the meaningful shifts um because in the English language we don't have different words that are somewhere in you know require a consonant somewhere in between a ball and a paw we just have ba or buff sounds And Pa sounds we don't have something in between it's not meaningful for us so we treat them as distinct categories this is the nature of categorical perception that we tend to pay attention to subtle changes that tell us about a categorical boundary categorical perception is super important to speech perception it allows us to identify exactly which

speech sound we're listening to so we don't tend to hear the subtle Differences in those speech categories as we'll see in a few slides people tend to have a lot of variability in how they articulate sound depending on who's saying it or in which context and we as listeners tend to ignore these bits of variability and tune into the subtle acoustic cues that denote important shifts in what we are hearing and this again is not limited to speech we use categorical perception for other kinds of information This is one way in which our brains reduce

the noise in our natural environments it's a resource conserving process it also makes sense that we use categorical perception on the end of the perception side of things because we don't have precise control over our articulators there really is a range of acceptable expression of a particular speech sound and our brain can treat that whole range as belonging to the Same category so the previous example I gave you as you know an experimental setup where participants are hearing ba or paw seems really contrived you're hearing single phonemes in isolation this is not what we're doing

when we're decoding speech one word to coding speech we have context cues from our environment that influence our perception we hear phonies in the context of words so I'll present for you here an Experiment from Bach McMurray and his colleagues back in 2008 where participants did a similar task to the previous slide so they're asked to categorize sounds as being either in the category so the voiced category or the category the voiceless category but in this time the phonemes were in the context of real words they were in the words Beach or peach or those

words are the same like bear and pear were the only difference between those words is The difference in voice on set time and for this task there was a matching picture on the computer screen when participants did this experiment so they were shown the two pictures a picture of a beach and a picture of a peach they heard the words Beach and Peach over headphones and they needed to identify what they heard by clicking on the picture like in the previous Example The Voice onset time varied linearly from zero Voice onset time to 10 20

or 5 10 15 20 25 30 35 40 milliseconds so it varied linearly but what did they find in terms of people's perception was perception linear or was it categorical what they found in this experimental setup is that the slope this categorical perception curve was much more graded so this dotted line is showing us something similar to what we saw on the last slide here the y-axis is different So it's the percentage of hearing paw on the last slide it was percentage of hearing ball um so this dotted line is showing us what we saw

on the last slide that paw and isolate or ba in isolation people hear ba and then there's the sharp shift in a category boundary and then people start hearing paw when the voice sounds at times at about 20 milliseconds but when participants heard the phonies in the context of real words in the Words Beach and Peach and we're shown pictures you see a more graded slope and so this experimental finding is shown with the solid line so this experiment showed that when context is involved with real words and with visual cues there's what's known as

gradient sensitivity to our phoneme categories so we treat these categories as a gradient it's not an absolute all or none shift in category boundaries When you've got the context of real words and pictures were a little bit looser with our category boundary so I'll take this um quote I roughly copied it from the paper which I think helps to explain this concept nicely decoding the speech signal must operate in real time at a rate that places extreme demands on processing efficiency to keep Pace with the nearly continuous flow of fluent speech In order to maximize

efficiency under such conditions it's important to both make rapid temporary predictions about the incoming speech signal and use what information you have available to you so you're making very quick decisions about what you're hearing and what you're hearing is transient it doesn't last for very long you don't often have the opportunity to say oh can you repeat yourself and you can't replay sounds over and over again so you use context To help inform some of these decisions what's in your real world that can help you decode speech so two things here are likely true we

do treat phoneme categories as categorical but also real world information can influence our perception so those are two key takeaways from this particular study our brains are using all the information we have available to us to help make the Speech decoding process more efficient we'll now talk about the gnong effect which is described in the textbook so the gnong effect is also called the lexical effect in speech perception so um this is a really interesting manipulation and kind of similar to the last manipulation where voice onset time is being varied um but you've got the

two ends of the Continuum different and important way Where one end of the Continuum in this voice onset time shift is going to be a real word so another phoneme pair you can have that different voice onset time is same articulation duh it's a short voice on set time ta is the long voice onset time um but you can anchor either end of the Continuum by putting those phonemes into the context of real words or non-words so you can have a pair like doot to toot Where toot is a real word and do does not

a real word or you could ask subjects to make a decision on a word pair like Dash to Tash which phoneme are you hearing duh or ta and the answer to that question seems to be influenced by whether or not a word at either or there is a real word at either end of the spectrum so you're presenting people with this foaming pair within the context of a word People's responses about what they're hearing are you hearing a duh or are you hearing a TA is influenced by their expectation of what they're hearing is a

real word or a non-word we have a bias for hearing the real word and this is the gnong effect this lexical effect this bias for hearing the real word I'll show you some data on the next slide here so you've got this ambiguous sound again Varying voice onset time on our x-axis you have that short voice onset time so the duh is a short voice onset time and ta is that longer before it's sunset time but as we've talked about with the other examples you can vary that linearly uh voice on set time it can

be either 0 10 20 30 40. um we'll look at this bottom line first so this um are people saying that they hear da is represented in blue and are people Responding that they hear ta that's represented in purple so the two colors are the proportion of people's responses this bottom line here the phonemes are just in the context of two non-words Das and Tas and if you do that this categorical perception looks like it does for other sorts of categorical perception experiments you've got a sharp shift um and it's right at the midline between

these two categories right where the Difference actually does exist um that 40 millisecond boundary so at 40 milliseconds people start saying they heard tasks but before that 40 millisecond poissons at time so um that short voice onset time people say they heard death so this is replicating a previous finding what we know but these responses differ if you use real words and non-words at either end if they continue So we'll move up to this next line here where we've got dash at one end of the Continuum Dash is a real word Ash is at the

other end of the continuum and what do people say in this experimental manipulation well people seem to widen their category boundary for saying that they heard a duh in these blue responses because that's anchored it's heard in the context of a real word um and so people have a bias for hearing That real word and so they widen their category boundary compared to the previous manipulation where that category boundary people start reporting they heard the difference at 40 milliseconds here people report they heard the difference that about 50 milliseconds so this is a significant difference

now we'll go up to the Top Line our final example where you have toot on the one and on the one end so ta in the Context of a real word and you have dute on the other end so D in the context of this time a non-word and we're now seeing the mirror image people say that they heard a TA until about 30 millisecond voice onset time so that's really interesting again widening their category boundary being more liberal with their category boundary because ta is um because toot is a real word so people are

expecting to hear a real word we Don't expect to hear non-words and so you're less likely to say that it's dute so the gnong effect is a shift in your category boundary depending on whether the phoneme is heard in the context of a real word or a non-word we have a wider category boundary when one end of the spectrum is a real word this is showing us that we do a lot of perceptual cleanup as we hear speech context cues are balanced against the acoustic signal here the context clue is Hearing a real word we

have a bias for hearing a real word we're going to balance that against the acoustic signal so the acoustic signal might be saying one thing but that acoustic signal is driving you down the wrong path well I'm not expecting to hear an onward I'm expecting to hear a real word and so we have that bias for the real word and so this is just showing you the same thing that you saw in the last graph in a slightly different way Um again kind of looking at this X Y axis that we've seen from the other

experiments that we have to shift um the kind of hearing sorry hearing duh or hearing ta um so hearing duh for the short voice onset time um and just hearing those phonemes in isolation um in that solid dark line and people seem to have this categorical shift Um where you go from saying yeah you know I heard a duh sorry I go I heard a duck to I heard a TA and there seems to be a categorical shift when those phonemes are not in the context of real words when you start putting these real words

and non-words into the experiment the red line is showing us the difference um from Dash to Tash So when Dash is the real word hash is the non-word people say they heard Duff or a wider longer amount of time um longer voice onset time they widen their category boundary we see the mirror image when people are looking at the difference between dute and toot where toot is the real word again people wide in their category boundary they say they heard for a little bit of a wider category boundary because It's anchored by a real word

so this is known as a perceptual shift our category boundary shifts to accommodate that people are hearing the real word this is what's known as a top-down effect um we're going to talk about top-down effects more when we get into morphology so don't worry about this piece of information um too too much but just a top-down Effect is that our knowledge of real words in English influences um which is you know our top level knowledge our knowledge of words is influencing whether we hear this low-level phony so top down knowledge of real words affecting what

phony we're hearing all right so I'm just giving you an awful lot of information I feel um we'll take a little bit of a mental Break um and watch this short video on the phonemic restoration effect before we go inside this busy bar can you understand these garbled words this sentence does it sound range the words are hard to make out and if we head inside with music and people talking and coughing it should be even harder to hear them right well let's take another list This sentence does it sound Fringe amazingly it's not so

hard to hear those words now and the sentence doesn't sound so strange at all this is called the phonemic restoration effect your brain anticipates a fluid complete sentence and fills in this missing phonic sound all right so just going off what the narrator was saying so the phonemic restoration effect is an effect in which a non-speech sound That shares certain acoustic properties with a speech sound is heard as both a non-speech and speech sound when embedded within a word that leads to a strong expectation for the speech sound so you are expecting to hear a

speech you are expecting to hear an unbroken flow of words so if something interrupts that like cough your perceptual system accommodates for that and you keep hearing speech as if it kind of were uninterrupted you Restore that missing speech signal excuse me when we're recognizing a word we don't need all of the sensory information to recognize it you can present people with incomplete words and people will still recognize the word people also will report hearing a phoneme that wasn't even there what this is showing us is that top-down information about words is influencing our perception

of speech we expect to Hear words when we're hearing language and so that expectation shapes our perception so I've got a link here um to an activity from the textbook um showing this phonemic restoration effect um she also has some examples for the ganang effect on there as well so uh go check those out play around with those sounds a little bit and it will help you get a better understanding of What are you know these very new Concepts to you and something I can I'm only explaining to you with examples but for you to

actually hear the examples I think would be really useful so go check out that link play around with the sounds a little bit in there as well um and see what you discover about the phonemic restoration effect and the Ganon effect it's we'll now move on to the development of Speech perception which is a really fascinating process so how do we learn about these category boundaries that we have in speech how do we learn about our consonant categories figure out that voice onset time is Meaningful build that word knowledge and you know all of these

other interesting bits of information so the interesting thing when examining development so looking at young young infants in particular is you have to Think up really clever ways to measure children's speech comprehension because you can't ask them um especially for young young infants so I've got videos on the owl site um these really are I would say a must-watch because they're going to demonstrate the concepts that I'm explaining here I think it's just really useful to see these Concepts in practice to understand kind of what I'll be talking about here and what's in the Textbook

we'll first talk about the high amplitude sucking procedure it's also sometimes called the non-nutritive sucking procedure in this sort of experiment a pacifier is hooked up to a computer so you'll have a pacifier with actually like wires on it um and a computer is sensing how often the infant is sucking on that pacifier when the pacifier is sucked it prompts the computer to produce more Of the stimuli so in a speech experiment infants are going to hear a sample of speech each time they suck on the pacifier so they may hear the phony over and

over again each time they suck on the pacifier the sucking rate tells us how interested the baby is in that sound it creates this sort of feedback loop if a child likes the sound they're gonna suck more If they're uninterested in the sound they're going to decrease their sucking rate so you can get a measure of how interested a child is based on their sucking rate so the sucking rate is the dependent variable in this sort of experiment this is what's known as a habituation um a habituation method I'm losing my words here uh sort

of a habituation kind of experiment Um so when the infant is hearing the same sound over and over again they get used to it they get uninterested in it their sucking rate decreases that's what's known as habituation they've habituated to that sound what you can then do is play a new sample to them so the infants habituated the sucking rate decreases play something new for new for them if the sucking rate increases when you play that new stimulus The experimenter then knows that the infant is perceiving those as a different stimuli that X and Y

are two different things because now the infant if the sucking rate increases when you play the new stimulus the infant has disabituated so the sucking rate decreases that's the infant habituating getting used to the sound if you play the new sound the sucking rate increases that's disabituation now the infant is Interested again so let's look at this graphically so here's the high amplitude sucking methodology socking rate is on the y-axis and the different stimuli over time are on the x-axis so remember that pacifier is hooked up to a computer each time the infant sucks the

sound is played for them so if they suck more they're more interested they're going to get more of the sound so it's that feedback loop as their Sucking rate decreases um you know they're getting less of that sound that means they're not interested in hearing it anymore so um so hearing ba over and over again the infant sucking ring increases they're interested but over time that sucking rate starts to decrease and they've habituated to that sound then you play a new stimulus so here you start playing the Paw Paw sound over and over and that

sucking rate increases Again that's showing the disabituation the infant is now re-interested in what they're hearing telling us that they're perceiving ba and PAW as two different um two different phonemes or two different um sorry syllables so they're perceiving those as being different in some way now that we understand the methodology let's see how this can be applied so the high amplitude sucking procedure Can be used to examine whether infants can discriminate between consonants in their language uh so IMS and their colleagues back in 1975 so very early on in language development research uh tested

baby's responses to the ba versus paw syllables and what they found is that infants between one to four months of age were very good at distinguishing these sounds and so this is what we saw on the previous slide uh ba tapaw And we see that infants you know habituate to the boss sound and then disobituate to Paw they're interested so telling us that they're treating those as two different categories so infants can discriminate between consonants in their language what this bottom graph is showing us um is the main takeaway from this one is that infants

only care about the speech sounds that are important in their language So here they're um the two different instances of ba differed in their voice onset time um and importantly differed by the same um the same difference the same number of milliseconds as the difference between bar to Paw these two instances of ba to ba differed by that same magnitude so batapa differed by 20 milliseconds bought to a different instance of ba Also differed in 20 milliseconds in voice onset time but infants treated them as the member of the same category they habituated to ba

they switch to that bar that differs in voice onset time but they still treated it as part of the same category they did not disobituate in this instance what this is telling us is that infants only care about the categorical shifts that are meaningful for the consonants In their language they're not treating all differences is the same only treating uh those differences differently that are meaningful for the consonants in their language so it seems to be that children have learned that ba and PAW are different phonies by between one to four months of age uh

the interesting thing um so yeah so the interesting thing here is We can look at a number of different consonant shifts um how do infants treat consonants you know not just in their native language not say just in English um but in other languages as well because other languages use different consonants other languages have different sounds what's interesting is it looks like infants can discriminate between any phoneme in any language at least initially How do we know this well this is an interesting phenomenon we know that adults are really poor at distinguishing phonemes that do

not exist within their language so you know this is part of the piece of adults being really bad at learning second languages within adulthood will come back to this idea more throughout the course and the general worker conducted this study looking at English speaking Infants or English learning infants and Hindi learning infants um and looked at their ability to discriminate between the contrast between um a Hindi da and the Hindi so the Hindi da is a consonant that does not exist in English so da typically we produce by bringing your tongue just to the roof

of your mouth just to the kind of the top of your um what's called the alveolar Ridge so try saying over again and just the tip of your tongue is hitting the roof of your mouth what's known as the retroflex D so a consonant that exists in Hindi the D sound you actually bring the bottom of your tongue to the roof of your mouth so you curl over your tongue in your mouth bring the bottom of your tongue to a roof of your mouth the D sound is something that doesn't exist in English it's not

a sound that We use in English if you're practicing to make that sound right now maybe it is the first time you've tried to make that sound um I might not be producing it correctly so you're gonna may be able to produce it more correctly than I can uh if you're a native Hindi speaker um so what's janetworker looked at invincibility to perceive this difference so adults can't discriminate between the Da and the D sound but can infants do this um so we'll head to the next slide to talk about the experimental setup for this

so in this experiment uh Janet worker and her colleagues used what's known as a conditioned head turn procedure so in a conditioned head turn procedure you're going to have kind of an experimental booth and you can have speakers on either side of the booth so On the right side and on the left side you'll love the infants sitting in the booth often they're sitting in their caregiver's lap uh their caregivers getting a pair of you know noise canceling headphones and a pair of glasses so they can't manipulate what's happening in the experiment and you play

stimuli in this experimental Booth either on the right side or on the left side and you are looking at where the infant Is orienting to so um what side of the booth are they looking towards are they looking towards the right or are they looking towards the left and that tells us about the infant's interest in what's being played you're going to have an experimenter sitting outside of the booth whose video and audio recording all of this and is coding the infant's head turns uh so before I get into some of the Nitty-gritties of this

you know why do you use this methodology instead of the high amplitude sucking uh High amplitude sucking is really only effective until an infant is about three to four months of age after that point the sucking reflex starts to drop off and you have to use more sophisticated measures to get at an infant's perceptual abilities so that's where you use the head turn preference procedure at this stage in development as well so After about four months of age infants become you know their motor abilities have developed a little bit more they're stronger they can turn

their heads and so a method like this is more appropriate at this stage of development so um four months and older the main outcome measure in this sort of experiment is seeing that infants look longer at stimuli that they're interested in They may have a kind of two different outcomes and it's debated within the literatures which outcome measure you use but they may show a novelty preference so looking longer at novel stimuli or new information or they show a familiarity preference looking longer at familiar stimuli um so there's debate over you know should we be

looking at novelty or familiarity preference really Researchers are looking for discrimination between novel and familiar items so are they just treating the novel and familiar items differently so the question being you know do infants discriminate between paw and ba or not is there just is there a difference in looking times um this is a well-suited experimental setup for infants because there's limited attentional demands limited motor Demands they're just having to turn their head they're not having to do a button press like you might have to do in some of your psychology experiments and there's no

communication demands in this and so going back to this example um with the Hindi sounds can infants perceive this difference and at what age do they lose the ability to perceive this difference so looking at adults first Um can adults perceive this difference they cannot adults can hear the difference between Duff and but looking at adults cannot hear the difference maybe you can't hear the difference in what I'm producing because I have a cold and you're listening to an audio recording and it makes it a little bit harder and you're an adult um what's very

interesting is that English learning babies can hear this Difference um so they're treating the da and the as different phoning categories even though that phoneme contrast does not exist in the language they're learning so they're keeping all their options open very early on by the time they get to eight months of age they lose this ability so they can no longer hear this distinction between the two Hindi consonants so this is a really interesting finding It's showing us that young young infants can actually discriminate between um many different phoneme categories even if they're not an

important distinction for their native language and that this that there's a perceptual narrowing around eight months of age though it varies um towards only perceiving the differences between phoneme contrasts that are meaningful in your native language So the number of contracts or number of categories you care about starts to really only be focused on your native language um because that's the language you're learning that's the language you're hearing and it's more efficient to really just to pay attention to the contrasts that are meaningful in your native language so I've got a link on the our

site um for the experimental setup for this So janetworker walks you through this head turn preference procedure um and so it's really interesting to see this in action and of course with cute babies it's a great thing to watch uh so I'll skip through it um for the interest of time here so if you can watch that on your own time um so Patricia cool as another researcher who's done a lot of work in this area and has shown that this phenomenon um this Phonetic category distinction um and how it changes through development is a

phenomenon that happens across languages so you can look at other languages so in the Japanese language the distinction between the English and La phonemes doesn't exist in Japanese and so Japanese learning infants lose the ability to distinguish rough and love around six months of age but prior to six months of age they can Differentiate those phonies even though they're not a meaningful distinction in their native language different non-native phonemes are lost at different points within development so they all fall between about 8 to 14 months of age so an interesting kind of way to think

about this is that sometimes language acquisition involves learning to ignore things that are not important in your Language the idea is that language learning involves not just learning new things but ignoring things that are not relevant to your target language we I mentioned earlier a little bit about our brain engaging in this resource conservation process when it comes to perceiving language the same thing applies in language acquisition that our brain is trying to do what's most efficient to tackle this very complex Problem of decoding the speech signal so what we've talked about so far is

that speech perception is a really challenging problem for a number of reasons um it's so complicated in fact that you know Microsoft Google Apple have spent collectively billions of dollars to try to get computers to decode speech in the same way that humans do it doesn't seem to be an easy problem to solve there's so much variability in how We produce speech speeches unsegmented it's co-articulated it's a very hard problem so the brain needs to be adapted in some way to recognize speech it's something that we as listeners seem to overcome quite easily we're very

easily able to decode speech because we have tools like categorical perception that help us simplify our acoustic environment when it comes to speech so let's talk about how psycholinguists Think humans tackle the speech perception problem we're going to start with an example that shows how we use auditory and visual information to perceive speech foreign [Music] what do you hear but look what happens when we change the picture foreign it's an illusion known as the mcgurk Effect take another look concentrate first on the right of the screen now to the left of the screen the illusion

occurs because what you are seeing clashes with what you are hearing in the illusion um what we see overrides what we hear so um the mouth movements we see as we look at a face can actually influence what we believe we're hearing if we close our Eyes we actually hear the sound as it is if we open our eyes we actually see how the mouth movements can influence what we're hearing it's a bizarre effect remember the only sound you're hearing is bar with a B what's remarkable about this illusion is even knowing how it's done

doesn't seem to make a difference the effect works no matter how much you know about the effect I've been studying the mcgurk effect for 25 years now and I've been the face in the stimuli I've seen stimuli thousands and thousands of times but the effect still works on me I can't help it [Music] experiment shows us that even when our senses are working normally we can still make mistakes in interpreting their signals all right so it's a really neat illusion of the mcgurk effect uh so maybe go back and watch that video Try closing your

Eyes as you're going through the videos and see what you perceive or hear what you perceive um and then open your eyes and see how the conflicting auditory and visual information work against each other all right sorry for the jump into switching locations as my computer died and I lose my life because it's like four o'clock so it's getting dark uh so what's interesting in this video is that the person is producing the same Acoustic information with conflicting visual information foreign excuse me it's an interesting illusion because you perceive a phoneme differently based on visual

information when the auditory and visual information is in Conflict our brain prefers the visual information speech is being integrated using different kinds of information combining information and recognizing it As a single perceptual fact when you have conflicting information that's what leads to misperceptions this leads us then to the motor theory of speech so one of the earliest examples of the mcgurk effect came as evidence for the motor theory of speech the motor theory of speech states that the perception of speech sounds involves accessing representations of the articulatory gesture that are required To make those speech

sounds so what's the motor command for making that speech sound knowing the motor command for that speech sound is involved in perceiving that speech sound the motor theory of speech is what's known as a modular theory meaning that there is a dedicated speech processor in our head that's not responsible for processing other kinds of information only for speech so speech is a separate module from other cognitive processes You have a dedicated system for one cognitive process that's what it means to be a modular Theory it treats speech is special and that you have the speech

module not used for processing other kinds of sounds this idea of modularity is not limited to language it's thought that we use it in other kinds of cognitive systems as well and that each cognitive module would be adapted to processing other Kinds of information so why is it a motor theory it's a motor theory because it assumes that our system for articulating speech is also used for perceiving speech the theory suggests that we have a central processor or a speech module that's responsible for controlling how we articulate sounds and for controlling how we perceive sounds

when we're perceiving sounds this speech module is mapping that perception to how We would actually articulate these sounds so we use motor signals not just to move our articulators but to decode how other people are moving their articulators and that informs what we hear with our ears this Theory does try to explain a lot of facts about perception that we've talked about so far in today's class there is also as well some neural Imaging evidence for this Theory so here I'm presenting some data we're Participants listened to syllables containing either a lip sound like a

sound or a tongue sound sound produced with your tongue like that sound the left panel um in these fmri scans shows what happens when people actually move their articulators so you ask people to move their lips or move their tongue here they're not making the puff sound or the tough sound they're just moving Um their lips or their tongues respectively the middle panel is the articulation task so what areas of the brain are lighting up oh sorry for the last time I didn't say um so the red areas on the brain um are what's more

active when people are moving their lips the green areas on the brain and this movement scan are showing what areas of the brain are more active or most active When the participants just moving their tongue next we have the articulation task so here people are actually producing the the phonemes the puff sound or the T sound and you can see that this red area um is more active for producing the puff sound and this green area is more active for producing the T sound and finally in this last panel you have A speech perception task

either hearing the P sound um or the TA sound the speech perception task interestingly resulted in activity in the motor region so where these um where these areas are circled with yellow and the motor regions even though this task did not involve moving the tongue or the lips this was a perception task and the pattern of activity in the Perception task matched up with the brain activity found in the motor region for the movement tasks in the articulation tasks so in this movement task in articulation task these are both in motor regions here um so

that's going to be in your parietal cortex this speech perception task there's activity in the auditory cortex but also activity in the motor cortex in The in the um parietal cortex so hearing syllables with the PO sound revealed activity in the lip region of motor cortex hearing syllables with the T sound revealed activity in the tongue region of motor cortex this is giving some evidence to the motor theory which suggests that we're mapping sound onto the articulations used to produce that sound this brain Imaging data is giving us that bit of Evidence which is really

interesting the motor 3 can also be used to explain categorical perception so the the motor theory would state that acoustic parameters are continuous in nature as we talked about with poisson set time but articulations themselves are discrete you can only make you know certain kinds of articulation so saying but uh and so PA does involve the tip of Your tongue a little bit those are discrete motor actions so when you have two different motor programs that's going to drive categorical perception producing two different categories so even though the acoustic parameters like boysons have time very

continuously our motor system is interpreting them as being discrete categories there's no that there are no in-between cases so we Learn these as discrete categories because we have to produce them discreetly so that's the argument for the motor theory and explaining categorical perception the motor theory can also be applied to the mcgurk effect so when I see a person's mouth moving you're get I'm getting extra information um that I'm then using for decoding the motor theory suggests that visual information is robust and reliable So when the acoustic and visual information conflict with one another motor

theories suggests that we use the most reliable source of information which is the visual information the motor information the articulatory gesture is going to win out because what's primary to the motor Fury is the motor command so the motor theory came about in the 1970s it was really popular throughout the 1980s it's not as popular these days And it has faced some criticism in the research one piece is that categorical perception as we've talked about is not speech specific we use categorical perception for other kinds of information for colors for music so perceiving the same

major and minor chords we treat them as discrete entities even though they vary continuously same with color so categorical perception is not specific To speech and the motor theory is modular treat speech is special but this process doesn't seem to be special for speech it also appears as though categorical perception is not human specific so there's been a really interesting experiment where chinchillas were trained on perceiving the difference between da and ta so that voice onset time variation um and even though they themselves do Not talk chinchillas do not talk they don't have um they're

they don't talk they're still able to perceive Dot and talk categorically in the same way that humans do now chinchillas don't have articulators they don't have a motor code for how those phonemes should differ but they're doing the task in the same way that humans do and so that's another criticism for the motor theory Um and that you know our if we're mapping to articulators and that's why categorical perception occurs well how can an animal who doesn't articulate speech map to a motor command to perceive da and toss to distinct categories so there's no reason

to believe that Chinchillas have an innate speech perception module um and that it must have just that this Ability to perceive da and talk categorically must have resulted from General auditory abilities that are not exclusive to humans it's not specific to the ability to produce speech so are there other explanations we also have the general auditory framework this is not in the textbook but it's still important the general auditory framework states that speech is perceived in the Same way as other kinds of auditory information although perceiving speech begins with auditory processing investigation of speech perception

through models like the motor theory have studies speech perception independent of the auditory system the general auditory framework focuses on the idea that the auditory system is good at perceiving many kinds of acoustic information the auditory system Is responsible for categorical perception to occur so masaro back in 1979 forwarded this idea of a fuzzy logic model of speech perception that our auditory categories are fuzzy they don't have hard and fast boundaries or fuzzy boundaries we can take sounds that are not perfect but we can categorize them as all belonging to a single category we use

similar sorts of processes for Other kinds of information in our environment so for example we use it when categorizing animals that we have fuzzy boundaries bird is a fairly fuzzy category and we allow lots of different examples of birds to be in that category Robbins and Penguins don't share a lot of characteristic Robin's why Robins you know eat worms they'll visit a bird feeder penguins live in the Arctic they Don't fly they swim you know they're not eating birds or other bugs um so they are really different from what we expect a bird to be

like but we treat both of these as equally good members of the bird category so the same sort of logic is applied to speech sounds not even those beach sounds might differ they may be articulated slightly differently you know by speakers with different focal qualities we treat them as belonging to You know the same perceptual category we can have good and bad instances of a specific phoneme but we can perceive it as all belonging to the same acoustic category and so this is built off of prototype Theory um oh sorry I mentioned that we treat

Robin and penguin as equally good members of um as equally good members of the bird category we actually treat Robin as a Better member of the bird category um and penguin um then penguin penguin is the worst member of the bird category but we still treat them as two members of the bird category it's a good thing this is not the cognition course um so we do a similar thing when we're perceiving speech there are good instances of phonemes there are not so good instances of phonemes but we allow them to be part of the

same category speech perception within the general Auditory framework is viewed like perception in general we use whatever information is available to us because the incoming speech signal is accompanied by a lot of noise and so we have to use whatever information we can to disambiguate the speech signal so how does the general auditory framework explain the mcgurk effect well it would suggest that we're using information from different modalities what's known as multimodal integration Integrating information from auditory and visual modality our mind weighs the evidence from this incoming information and makes a judgment about the best

evidence in this case auditory information is more likely to be wrong we have frequent misperceptions of auditory information whereas visual information tends to be more reliable so when in doubt we go with visual information when we're integrating both auditory and visual Information and so that's why the mcgurk effect works according to the general auditory framework we're weighing evidence and making judgments about the best evidence none of this appeals to a modular View it's not specific to speech we do this for other kinds of cognitive processes as well and it's not linked to a dedicated neural

processor for speech we use the same kind of fuzzy logic and We weigh evidence when perceiving speech sure music other kinds of auditory information or visual information touch information and so on and so forth now we have this domain General system that applies across different domains not just speech that is involved in perception so here we can contrast the two theories we've got the motor theory which is a modular Theory where we have a dedicated system for processing speech We used um we're mapping speech to the articulatory processes and these motor programs for producing speech

the general auditory framework on the other hand is not modular there's nothing special about speech we process speech and auditory information in the same way that we process other kinds of information speech can be recognized using multiple cues and so it can explain the mcgurk Effect by just saying that the visual information is more reliable so we trust the visual information so something like um asking a question like comparing and contrasting the motor theory and the general auditory framework for explaining the mcgurk effect might be I think a really good exam question um or explaining

you know what is the modular Theory or what is the general auditory Theory or what is motor theory Sorry and what's the general auditory framework um so just keep that in mind as you're moving through your studying you can tell that I've got the midterm on the brain right now and you probably do too so before we leave for today two quick things which is a very quick PSA today we've been talking about perception about our auditory organs I really want you to encourage you especially because You know many of you are young folks um

to really protect your hearing uh if you still have the opportunity to protect your hearing because once you lose your hearing you do not get it back once those hair cells on the vascular membrane are damaged in any way they do not grow back they do not regenerate there are a lot of environmental risks to our hearing so wearing headphones um going to a concert or sporting events now that we're probably going to those a Lot more often than we have in the past two years using power tools or other loud equipment you know mowing

the lawn a good cue is if you need to raise your voice when you're speaking your environment is probably too loud so protect your hearing whenever you can you can use foam or wax earplugs you can use custom earplugs I've got you know pack right here I got from the grocery store I'm seeing fireworks in a couple weeks Um so I've got my foam earplugs ready for such an instance um so take good care of your hearing um it's a crucial organ um and just a couple notes for next week um so I want you

to check the form early next week um for your conversation topic so next week is on speech production we talk about speech production errors so throughout the week I want you to listen For and describe speech production errors um according to what you learn from lecture in the reading so it's important to get started on that early in the week so you have a chance to find it throughout the week you can't leave this one until Sunday night so just keep that in mind for next week you're reading next week as well a little bit

of a mishmash of a couple different things uh so skip section 10.3 And 10.4 in the textbook um instead go back and read section 4.3 where otherwise not reading chapter four and I'm posting a couple videos on owl instead of reading section 10.3 and 10.4 um for those videos one of them is mandatory one of them is optional optional one's really really good um and the mandatory one is really really short so you've got some trade-offs there all right I think that's enough lecture For one week um thank you to all of you for your

attention I'm sorry for the coughing and the coughing breaks um it will eventually be well again I'm sure um look forward to seeing you all next week