Hi, I’m Adriene Hill and Welcome back to Crash Course Statistics. In this series we’ve been talking a lot about how often you see data and statistics on the news and on social media. There are all sorts of studies and data sets promising the keys to a better life.
Coffee is good for you. No wait. It’s killing you So what do you trust?
INTRO Journalism has many goals to inform, expose, help people make better decisions about their communities and their lives. But, the only way journalism matters--is if people read it or watch it or listen to it. Journalists have to capture the attention of their audience and help them connect with the story.
Case studies and observational studies can be great source materials for articles or podcast. But, it’s always important to look at both the quality of the science. And the quality of the journalism.
If there’s a study without a control group or with non-randomized design it’s going to be less reliable. And if a journalist doesn’t ask those questions, or just doesn’t include the answers in her story, how are you going to know? Back in 2015, newspapers around the world ran stories heralding Chocolate as a way to lose weight.
It sounded great. But turns out it wasn’t good journalism or good science. A science journalist and PhD named John Bohannon created this story by doing a real, randomized study, but one that was intentionally riddled with flaws.
It was meant to be a way to show that academic journals would publish the very flawed study. And so would news outlets. The goal, writes Bohannon, was to demonstrate how easy it is to turn bad science into big headlines.
And it worked. Once the study was published and the press release went out journalists jumped on it. Bohannon says that many outlets ran the story without ever contacting him.
Very few reporters asked about the number of subjects they tested-- only 16--and no he says, reported that number. Also says Bohannon the stories that ran didn’t quote any outside researchers for corroboration. And while it is disappointing that you can’t add some Cadbury to your diet and lose weight, and maybe your aunt keeps quoting this study to you as she downs that 5th and 6th bon-bon, but bad science and bad scientific journalism is not always that harmless.
Most clinical studies base their conclusions on statistical tests that give researchers--and the rest of us a quantifiable way to measure the evidence that the study provides. For example, when a reputable doctor claims that Ibuprofen increases the risk of fertility issues in men, it’s because there was a study with a group that took Ibuprofen and a control group that didn’t, and the subjects taking Ibuprofen showed a tangible increase in some measure of infertility. But which measure?
And was the control group given a placebo? An article that you see on Yahoo health, probably won’t tell you in fact this one doesn’t. For that information, you have to go to the original academic article, and those can be kinda dense.
It turns out that this study did have a placebo control group, and it measured infertility in a clinically respectable way, by measuring levels of fertility related hormones. But these facts are important when considering how trustworthy the conclusion of a study is, and a lot of news articles don’t have them. As a side note if this study was done in rats instead of humans the conclusion that Ibuprofen increases the risk of fertility issues in men would not be as strongly supported!
Let’s go to the Thought Bubble. Imagine that you’re going about your morning as usual, sipping your coffee and scrolling through the latest news when you see an article with the title “Miracle Food causes weight loss! " You want to fit better into your jeans so, you click through.
You see that the miracle food is called Targ, AND the results were “statistically significant”so it’ seems legit. You jump in your car and drive down to the local grocery store and see that Targ is on sale! So you pick up the largest pack and start eating you’re already feeling stronger But then you begin to experience side effects like heartburn and stomach ulcers and a desire to fight.
The article you read didn’t mention that when researchers considered the over 20,000 subjects that the weight loss was only about 1/10th of a pound more for Targ eaters. That’s not very much maybe not worth these side effects. When a study reports something as significant, you probably assume that this means that it’s really gonna matter, but this isn’t always the case, since significant means something different in statistics than in everyday English.
And Science journalists can misuse this confusion by not mentioning how big of an effect was observed. Thanks Thought Bubble! We don’t have time to read all the academic articles on even one topic that affects us.
Take what gets called “text neck”-- a condition that includes sore neck muscles from looking down at your phone and laptop all the time. A Google Scholar search for Academic articles about “Text Neck” returns over 180 results and that’s only since 2013. There’s no way that you could read all of those without exacerbating your already sore neck.
So we need people like scientific journalists who can distill all those articles into digestible--and engaging--pieces for us to consume. It’s helpful to be skeptical but we should keep reading about science. When reading a science story it’s important to note a couple things: who wrote it, who published it, who did the science, and who funded the science.
If an article that tells you that drinking Diet Coke is good for your teeth and is on the Coke website, your suspicion should be raised more so than if it was published by Scientific American. You should also consider who funded and completed the research the article is based on. If you read an article that claims a rare fruit juice will reduce your blood pressure and stave of cancer and you see that study was funded by the juice company.
Be suspicious. Not every study funded by a company is inherently flawed. Science cost money, it can be expensive, and while there are sources of funding from governments and other neutral organizations, the reality is that often the people who are willing to pay to have the research done are the companies who have a vested interest in the results.
Sometimes to get the research done, researchers need to partner with these organization. Privately funded research can be done well. Another thing to watch for in science and health journalism is whether the claims made in the headline actually match the claims made in the story.
You don’t see many stories with headlines like “Ketchup may have mild relationship with weight gain in men over 40”cause who’s going to read that? It might be accurate, but it’s just not as flashy as “Is Ketchup making you fat! ?
” There are a number of reasons we get this splashy headlines. Media outlets from Buzzfeed to Goop to the old-guard newspapers are all fighting for audience these days--Maybe not exactly the same audience but audience. And that competition makes the super-sexy headline, really, really appealing.
Sensational gets clicks. Content creators are under pressure to find and write what’s gonna get shared. The language of correlation is uncertain and, as such, less catchy.
You’ll also spot plenty of causation problems in science and health reporting. When you see an article that claims that “doing yoga cures cancer” you should check to see whether it was an experimental study or whether the claim is based on correlation on a survey between doing yoga and not having cancer. Only experimental studies with randomized designs and control groups have a shot at showing evidence of causation.
Because personally, I can think of a lot of confounding factors for yoga and cancer study. Nowon to another way science can get mischaracterized. There are studies done on mice and rats that get reported as if they were studies on humans.
And while a lot of medical and health related studies get their start in mouse models many of the treatments that work in mice don’t end up being successful in their human counterparts. Similarly, you will see clickbait-y headlines that say “Hydrogen Peroxide kills cancer! ” and lists all the ways you can now incorporate H2O2 into your daily life.
But what the title doesn’t tell you is that these were in vitro studies, which means they’re done on real cancer cells but in a petri dish. In a very simplified sense-- the cells were grown by themselves in a dish and the substance of interest was put into the dish and it killed the cancer cells. But in a dish lots of things we consume everyday will kill cancer cells like coffee, or alcohol.
But they even working in tandem aren’t going to cure cancer. Anyway these misguided “hydrogen peroxide kills cancer” headlines get shared around online and people come up with alternative therapies that involve consuming Hydrogen Peroxide which can be really, really dangerous. Like dead dangerous.
Science stories can make for great journalism. And they can give you something clever to say at your next dinner party. But any time you hear a cable talk show host says the phrase “scientists have found.
. . ” or “a new study suggests…” you should always just look up that study to be sure.
At least before you start spreading it around. And if the results of a study will cause you to make any changes in your life or your family’s life you should really go back and check the science. No matter how reputable the source, it’s always important to be aware of these issues, whether you see it in Buzzfeed or the Economist.
Articles often gloss over all kinds of details. The kind of control group that was used, or whether the study was done in mice or monkeys, all of which can make a huge difference in how strongly you can take the claims of the study. And the bigger the life change you are thinking about making, the more in depth your search for information should be.
Adding a square or two of dark chocolate to your diet is not going to be a big deal. Trying to cure cancer with high doses of Vitamin K just because some study found it kills cancer cells in a dish that is. So this doesn’t mean all the science you read` about on Reddit or watch on your favorite YouTube channel is wrong.
It just means that you need to use statistical thinking to check which claims are reasonable and which aren’t. In order to help us remember some of the rules of thumb we talked about today, our writer Chelsea came up with a limerick: And so without further ado. Crash Course’s FIRST original limerick.
When a study reports correlations Or has mice as its main population The results it declares May not be quite fair So be careful about generalizations Alright, let’s see you do better. Thanks for watching. I'll see you next time.
