why most public research findings are false that's the title of a paper that came out when I was roughly halfway through my PhD I'd spent about 18 months failing to get any of my experiments to work and I was ready to give up but then I realized to an extent that's just how science Works science investigates things nobody has investigated before it's trial and error so studies starts small maybe too small and what looked like a really interesting finding was actually just a fluke most is maybe an exaggeration but certainly some published scientific findings are
wrong what does that mean for you you currently have access to more Health Science information than ever before and more ways to consume it by Tik Tok Facebook YouTube and I love that so what's the problem well the problem is it might just be wrong we all know that feeling of seeing a headline that really resonates but maybe sounds a bit too good to be true how can you tell I'm a scientist and health technologist what I do is help companies compile scientific evidence to support their product and then figure out how reliable that evidence
is I work with medical devices for drug delivery I work with software to help you get better sleep I work with medicines for the most rare and severe types of seizures I am not an expert in any of those fields and you don't need to be either what I do is the science of science the science of science is a thing and it's principles can help you judge Health Science information in the media so some scientific findings are wrong what do we do about it well science self-corrects and one of the main ways it does
that is through something called a systematic review now the origins of the systematic revieww are maybe not what you'd expect in the early 1930s a young medical student called Archie Cochran had a problem in interviews he refers to it as a sexual dysfunction it's the 1930s you've got a sexual dysfunction Archie did what any one of us would do he went to Vienna to find Sigman Freud what Arie didn't know then was that he had a genetic condition not a psychological one psychoanalysis didn't help and he became very frustrated he was seeing one of Freud's
students and he says he believed strongly in intuition well I couldn't help asking why what's the evidence for what you are saying this experience in Vienna really got Archy thinking about evidence and by by the time he returned to medical school he was kind of a troll he says we started a sort of a club of individuals who would ask Consultants what's the evidence that your treatment has any effect we found this great fun there was so little evidence available by the time he left medical school he was obsessed with evidence and it came to
Define his medical career among his many great contributions was the idea that single studies can't be relied upon to make Healthcare decisions he called for research summaries to be produced that compile all the available evidence on a specific question this is what we now know as a systematic review and they are a required part of the treatment approval process the most bust typee of systematic review is called the Cochran review um Cochran reviewers judge how good the evidence is using a tool which is essentially a huge technical questionnaire and no big deal but it was
developed right here in Bristol used globally not just for cockran reviews and it was co-created by the team that I used to work for so I kind of felt that I was okay to mess with it a little bit what I wanted to do was take it out of the hands of the professionals and turn into something anyone can use to judge Health Science information in the media so what I did was take the core principles of Cochran and boil them down to just three questions three questions that I want you to remember and think
about about the next time don't worry I remember them if they go it's fine and you're going to remember them too um I want you to remember them because I want you to think about them the next time you see a too good to be true headline or a weirdly compelling Tik Tok okay so has there been a study done in humans are we sure it's not a red herring has there been a randomized control trial if the answer to any of these questions is no then it's not bad science not not by any means
but it does mean there's a high risk that in time it'll turn out that the findings weren't reliable before we get into it we need to be aware that it will take conscious effort because we want to believe I want to believe I have an autoimmune disease um I'm sick a lot I'm on immunosuppressant medication there's no cure and the plan is for me to be on imuno medication for the rest of my life so when I see a Tik Tok that says it's definitely going to cure me I want to believe I want it
to be true I don't want to question it so it takes conscious effort to stop think and ask has there been a study done in humans are we sure it's not a red herring has there been a randomized control trial this first one's super easy and a real killer question and once you start asking this question and listening out for the answer it'll filter out a lot of Health claims all um medical breakthroughs have to start somewhere and a lot of them start by growing cells in Petri dishes in a lab but that doesn't mean
that all cells in Petri dishes are destined to become great medical breakthroughs a really good way to remember this is that cancer cells grown in a dish in a lab can be very effectively killed with with a hammer all kinds of things will kill cancer cells in Petri dishes but that doesn't make them a viable cure for cancer and we know this because 90% of candidate drugs will fail when they get to the clinical trial stage mostly this is because once we put them in real life humans they just don't work in the way the
evidence suggested they would so if we ask this question has there been a study done in humans and the answer is no we can say okay that's super interesting but maybe it'll turn out to not be a reliable finding if we ask this question has there been a study done in humans and the answer is yes we're going to take it up a notch we're going to say are we sure it's not a red hering in the context of medical research what we're talking about here is confounding founding happens when something is related to both
the health factor that we're interested in and the health outcome causally related to both those things it's it's quite a tricky concept so I'm going to give you an example that's going to change the way you listen to Health Science information in the media nice graph here very clear relationship between whatever's on X and whatever's on y looking at this graph the more X happens the more y happens so if Y is something bad this graph suggests we want to stop doing X this is a real relationship it is not a chance correlation but what's
key is the nature of this relationship so let's see what our variables are on our y AIS we've got total cost of fire damage and on our xaxis we've got number of firefighters at the scene this is a real relationship but it is not a direct causal relationship it's a red herring fire is the culprit fire causes both the total cost of fire damage and the number of firefighters at the scene now we know what firefighters do we understand that system we often don't know what health factors do and we don't necessarily understand the underlying
system this is why it's so important to ask how was the study done what was the study design so what study design would avoid red herrings confounding that would be randomized control trials last question has there been a randomized control trial randomization is the key to getting rid of red herrings because then the red herrings occur as much in the treatment group as they do in the control group at the end of the first year of World War II Arie Cochran found himself as a prisoner of war as a medical officer he was in charge
of the health of around 8,000 British prisoners their diet was awful Archie noticed everyone including himself was beginning to swell they were getting very very sick indeed Archie suspected it was thyine deficiency but Archie is a prisoner of war he is not a doctor in a hospital he can't just order up a bunch of yeast which is what they need Archie needs to convince the guards beyond all doubt that yeast is the treatment they need Archie manages to get a small amount of yeast on the black market he numbers the soldiers and he sends the
odd numbers off to the treatment group the even numbers off to the control group the treatment group get yeast the control group get a placebo it was vitamin C he might have been tempted to give the sicker patients the yeast but then they're not randomized and then yeast isn't the only difference between the groups what if the yeast doesn't help because the sicker patients are too sick and what if the sicker patients are sick because they also smoke what if the sicker patients are more sick because of some fundamental genetic difference when you're a prisoner
of war trying to convince Gods to buy you yeast there can be no what ifs and when we are spending money and resource on thousands of very sick patients in clinical trials there can be no WTH within 3 days there was a noticable difference between the groups Archie wrote up his findings and told the guards the next day he woke up to heaps of yeast the whole Camp got better because of what was essentially one of the first ever randomized controled trials it was even blinded the patients didn't know if they were getting the treatment
or the placebo and this is another really important thing about randomized control trials the expectations and hopes of both the patients and the researchers impact results in trials that aren't double blinded the treatment effect can be exaggerated by as much as 17% Arie Cochran changed the face of Medicine by not accepting what he was being told by asking tough questions even of people of authority and experience ask tough questions even if people are wearing a white coat in their video even if they have the word doctor in their username even if they are stood on
a red dot giving a tedex talk ask tough question the best advice I can give you is that if you see a too good to be true headline or a compelling Tik Tok go to the Cochran online database I'm not affiliated by go with Cochran by the way I should just say but at this time they are considered the gold standard in healthc care evidence and they are used globally to make policy and healthare decisions they may have already asked the tough questions for you but if there isn't a review already on what you're interested
in we can all be more like Archie by asking the tough questions ourself has there been a study done in humans are we sure it's not a red hering has there been a randomized control trial change how you listen to this information because if you're listening to something really interesting and the answer to all these questions is yes then you might want to pay attention thank you very much
