[Music] hey everybody welcome to the daniels on research i'm david daniel i'm daniel willingham this is the podcast where david and i pick an article that we think will be especially interesting to educators and unravel it a little bit and have a closer look at it and dan found one that he liked for a whole bunch of reasons and i looked at it and i liked it too so we thought we'd talk about it what's the article dan yeah this is the title is the negative effect of smartphone use on academic performance may be overestimated
evidence from a two-year panel study so this is taking on an issue that i think is really intuitive to most teachers which is we're concerned when we see students in our classroom using tech devices in a way that they're not intended most often that's going to be smart phones and we figure there's going to be learning loss all right so before you get into it i want to get into in case you're thinking out there of publishing sorry i had hand surgery sorry everybody i talked with my hands so you're going to see this a
lot um in case you're trying to publish something read the title again dan yeah the negative effect of smartphone use on academic performance may be overestimated colon evidence from a two-year panel study that's a title right away you think you're right you think you're right you may be wrong people are going to read that yeah that's it that's exactly it and that's what that's what made this article so interesting to me as i was saying before you stop me with that fascinating observation about the title the uh this isn't a this is a finding that
was it was an instance where research seemed to really confirm what would be really intuitive to most of us which is kids using cell phones during their classrooms is not good for learning now there are two types of evidence that have been brought to bear on this and david i think you're probably going to want to elaborate on this a little bit but let me let me get the two types of evidence out on the table briefly one type of evidence is an experimental uh paradigm a randomized control trial and in these sorts of experiments
the researchers are going to for example send a text message to students during class it's understood the students know that if i get a text message i'm supposed to respond to it in some way and then there's going to be some outcome measure of students the extent to which they understood the lecture despite being disrupted the second type of experiment is a correlational study where you're just going to let students do what you know use or not use electronics during a lecture as they would typically do or not do and then you look at their
grades either at the end of the semester or their grades on examinations or whatever it is and you look for a correlation is it the case that kids who use their smartphones a whole lot more tend to earn lower grades than kids who don't use their smartphones during class yeah and you know if you watch this uh even once you're gonna know i like the ecologically valid versions of that the way i think of it is is we can um look in the wild in the classroom find something that we think is going on and
then we can bring into a more controlled situation for to look at mechanisms or to try interventions before pushing it back out to the classroom um so i put i put more confidence in the messier ones from the classroom if we're trying to make recommendations for the classroom right so the the trouble of course with the the the messy version is everyone knows correlation is not causation and so uh researchers try and deal with that by measuring the variables that you think might co-vary with what you're interested in so to make that concrete what if
kids who tend not to do very well in class kind of know class isn't really for me i tend not to do very well i don't really care that much and so they use their phones more during class than kids who are more invested in their education or what if you got a really really not non-engaging teacher who's going over stuff you can get someplace else or they're all powerpoint and you can write down the notes and you have time left over right well if it's if it's a teacher effect that should be the same
for everybody in the class though right if everybody with a phone one class then it's going to be an individual effect rather than a teacher effect but if you're looking at multiple classes yeah then you could have you could have a teacher effect and you wouldn't you wouldn't recognize it well i mentioned that just because a lot of the ones that are during the classes were done in a single teacher's classroom right um so the way that researchers try and deal with that is they'll do things like get the gpas of all the students in
the classroom so then they can control for the gpa to see whether we see what we see as kids who tend not to do well in classes anyway uh are the ones who are using their phones okay so with with all those and so they'll they'll measure prior gpa and they'll measure personality characteristics um you know big five personality characteristics and so on with all those uh with all that battery you still see correlations of of something like negative 0.3 negative 0.4 between cell phone use and final grade in courses okay so with this particular
study all right let's talk about the first the first analysis all right so um to set it up a little bit they basically had students sign up for this with cell phones that had an app that would trace their cell phone usage and correlated that with maybe times in class and gpa yes exactly right it's pretty clever the way they did it and kind of unusual so they end up with this is at a danish university they have about 412 i think it is students in the final analysis um and as you said they they
actually are tracking them they're using the gps feature on the phone to know where they are when their phone screen is on and then they've also got the student schedule so they know you were sitting in an auditorium and your phone screen went on for this period of time and so forth so those are the the data they've got and then they're looking at their gpas here's the way this study is different and you heard the term panel data which not everybody may be familiar with all of our listeners will be familiar with that it's
a term that's more often used in economics than in psychology it basically means because they're going they're collecting data across two years they're going to have data from the same student across multiple classes and they're also going to have the same classroom with different sets of students that now enables them to get at some of these confounding variables that we were talking about they can look within an individual student aha when you took this particular class you on average used your phone this much but in this other class you on average used your phone a
different amount so now i can look within a student and see look at variation in phone use and grade and this allows you to sort of extract out those individual variables that we were talking about like what if this is a student who doesn't care very much about uh about their about their academic outcomes yeah one thing they didn't do one thing they didn't do which i think would be really cool is actually flip it and say okay in these kinds of classes students tend to have their smartphones on more or less they actually could
have yes yeah they didn't do that um if you're watching um or listening authors of this study um there there's your next pub just make a really great title again you know i was that was exactly what i thought when you said that i'm like for all we know that's in the works right in the pipeline that would make sense you've got this huge really cool data set it's probably going to end up being more than one paper um but again the highlight here the what makes this study really worth looking at is they're they're
not going to know what the individual student characteristics are but they can be confident that those individual student and individual classroom characteristics are sucked out of the analysis right and that's that's going to be the important thing for their their their second analysis right um and uh also just really quickly point out um they did they were very transparent about several things one of them was the people who volunteered for this tend to have a different gp a higher gpa and lower gpa were not as well represented i don't know what that means in the
danish system um and that uh the uh you can't tell what was on the on the cell phone so they could have been going anywhere we don't know where they went yeah and they one thing is like they were actually just sampling i think 15 minute bins and all they know is that the screen is on but they did know the duration the screen was on um and they actually their their little rationale for this was so it sounds like like you have no idea what's going on there as you said they were pretty upfront
about that but they also pointed out um the screen the minimum time the screen was on was usually like 35 seconds i think there are very few observations when the screen was on for a shorter period of time than that so they said even if you got a text message and ignored it your phone screen would have you know automatically clicked off in a short uh sooner than 35 seconds so they said we're pretty confident that most of these we don't know what they were doing but we're pretty confident something was actually happening during that
time yeah it was pretty cool that way yeah so that was thoughtful so guess what they found in the first one they found in the first yeah so yeah go ahead give us the rundown of the uh the important analysis they found a negative correlation um between gpa and cell phone use right so that was like doing the garden variety it started trying to replicate the analysis that everybody else has done in the past without controlling for those factors so negative correlation is one goes up the other goes down right right so so people are
just as they're listening um so what what they found is uh the more cell phone used to lower the gpk right and then when they made use of the panel data and uh removed the classroom effects and the individual student effects it kind of went away it went away right so it's still slightly negative in absolute terms but the confidence interval included zero it went down by about two thirds after they included all that right another interesting thing they found the first one is in class smartphone use was almost as strongly correlated with high school
gpa as the college gpa which which opens up um to some ideas with that second analysis there's something else going on here right you know there's some other mediating variables we might want to look at so what's your take on this now it's like here are the data what are let's let's move on to speculation time what do you think is going on we had we we have three things indicating cell phone use during the during class is a bad idea we've got sort of a basic cognitive analysis it's drawing away attention that absolutely makes
sense we've got these intervention studies that we mentioned before we've got the correlational studies we mentioned before now we get this new data which indicates now actually there's no relationship between cell phone use and um and gpa there's there's something else that's accounting for that relationship you've seen elsewhere what's your take on what's going on here well you know how they say um not everything's good for everybody uh what what what this is saying is not everything may be bad for everybody um and to me the the nice thing about this study is it says
look there's more going on here we need to start looking at that as opposed to making these broad general swipes laptops are all bad for but for note note taking cell phones are all bad for sending in the classroom put them in a basket and put them out of the room there's other other measures also like like like fear of missing out measures there's studies that show that if you put it in a basket and the bask leave the basket in the room they don't do as well as you move the basket out of the
room there's there's lots of different variables to be looking at here and what they did with that second also said look there are other variables to be looking at here we have evidence that if we account for some of them it goes away so why why would it be the case that what give us an example of what you think an individual level a student level variable could be that would be accounting for the relationship that's usually observed well i like the one that was kind of alluded to they're very diplomatic but they said look
if it's correlated with college gpa and high school gpa maybe there's another variable that they're bringing with them um so and then we have the low gpa confound so it may be students who like you said earlier who school isn't for me or they're not engaged in school would have have a different consequence from smartphone interactions than the other students it gets washed out at a higher level of aggregated data the other the others and they but they also point out that in le in at least some studies they've used gpa as a correlate and
you still uh you observe the relationship and they so they they speculate that self-control might be at work oh they they have several things and and gpa i think is is almost too gross of a measure um there's a study that i have right here um that i pulled up by christian and and colleagues were they uh looked at a very different way and came out with different results for example what if i actually send you a text at this part of my lecture um or the class the class presentation and then i look at
your test scores to see if that getting that text that time may affected your your your performance on the test on that topic right the authors of this looked at teacher engagement course topic student density they lighting they listed all these possible intervening variables um that can be looked at um i don't some of them you know i don't know about lighting but um the student level ones the really interesting thing was what you were talking about was a self-control um maybe a attention working memory capacity the you know since we're cognitive that we're going
to go right there right it could be age also i'm developmental i'll throw that in but there's a whole lot of things to be looked at before we can do a global swipe that something's good or as the top as the title implies not good right it actually didn't say it's not good it said we need to look a little deeper here scratch beat the service we give you really firm recommendations right yeah that's definitely not the conclusion of the article you know it's not saying like well it turns out cell phones are fine so
like don't don't worry about it anymore and it's also worth revisiting the point we were making very early uh on here which is this is a population of danish university students so you know if you're teaching sixth graders in the united states it's uh that would be very dangerous to assume that the population is going to be exactly the same yeah and they may not even be use their cell phones as much as they do in the united states we just don't know um but the the interesting thing that they implicated which i think was
right on and especially if you're teaching right now uh during covert and you're doing distance learning and those sorts of things the self-control it's it's an issue that we're all having to deal with and that's going to be interviewing variable lots of different things the inhibition of of these sorts of things it's really something neat to look at i'm glad they pulled it in as a possible explanation so tell me make this a little more personal for me what's your policy in your classroom when you're teaching uh and does this uh article make you think
any differently about it my policy can be summed up in one word no um i i discourage its use i have 300 people in my classroom so there's a lot of uh when you there's a lot of distraction that can be going on to the students around you my general philosophy is is you can do whatever you want but don't interfere with the people around you who's learning and there's a couple studies for example laptops that show that even when you change screens the people in your peripheral periphery that distracts their attention if it comes
at the bad time that could affect their their attention and maybe their learning in that moment right so i'm just trying to mitigate the collateral damage as opposed to on the individual student um so i i try to discourage device i also try to teach in a way that you don't want to look at it so that's right that's kind of a nice thing um but in in a you know class with any time you're going to have natural times where you might wander um and when you get kind of full or when you need
a break and this may be students who use that during that time and it may actually be taking the place of that other thing that would happen yeah that's a really that's a really good point right there distraction's going to happen anyway and it's all just a question of what what what's going to happen during it but as you say there's the collateral damage issue yeah so what do you do what do you think about five or six years ago i started um making my class and and like you it's my typical class size is
about 350. uh i met i turned it into a no devices class and i always um during the anonymous student evaluations at the end of the semester i have several questions for students about that um and it's been very consistent ever every semester which is um the modal response from my students is initially i really didn't like this and i was kind of you know angry at you um but by the end i've i saw the point and i i think it's good one of the questions i always ask is should i continue this in
future semesters and it's usually about 85 or 90 percent of students say yeah you should continue this yeah i give the last two rows i get the last two rows if you want to have device you go to the last two rows yeah and i've i always you know make an announcement in class first of all for some students it is um you know it's an accommodation they uh they have trouble writing or whatever and i don't want those students to feel self-conscious so i always make an announcement in class like if you know i
don't want this to get in the way of your learning if you feel like you uh really would prefer to go on laptops like come you know come by and talk to me about it and we'll uh we'll work it out and there's usu usually most students are are quite open to at least trying it um and for a student that doesn't have an accommodation i'll usually say like do you want to try it for just a couple of classes and see and see what you think and the and this is what i started to
interrupt you with the other thing i found is that i think i would have had more trouble doing no devices 10 years ago and now there's a lot of self-awareness on the part of my students anyway i'm on my devices all the time and they actually kind of welcome the idea of okay for 75 minutes i'm just going to do this i'm just going to you know try and think about cognitive psychology and they know themselves well enough to know that if they're taking notes on a laptop or whatever you know popping over and and
checking social media is gonna be almost irresistible for them so they're they're usually they're happy for me to make the rule um so it's like it's like a christmas club it's like you know i need enforced good behavior i need to like you know buy in and then i have to put that money in every week or something you know something terrible happens we have to compare entrance requirements i love the way your students kind of the way you reflect the way you talk about how your students like you know it's actually i know me
well enough to know this is not a bad idea yeah uh anything else you wanted to extract from this article no i think if you do if you do research in this area this is a nice guideline a guideline it's a nice suggestion hey start looking at these other variables there's some really neat things out there and they're probably related to bigger things besides smartphone use so it could be a good boomerang back out to some some bigger ideas and if you're a teacher you probably already know this to some degree there are some kids
that it's more distracting for than others right which makes for a modal policy right what's the what's the best way to thread this needle that to get what you need from your students and and get them to engage in the way you need to well i think that's a great last word for this episode all right man i'll talk to you soon all right thanks see you bye everybody [Music]
