By now I hope you get the hang of playing the game of null hypothesis significance testing. Because of the way we play the game, there are some things to consider when we make a statistical decision with NHST. In general, the goal of NHST is to decide the location of the true parameter that is not known to us based on the sample location.
So even though we do not know about the true reality, a true parameter, in fact, we may never be, we may never be able to, and yet the rules of NHST seem to force us to make such sharp binary decision by either rejecting or failing to reject the null based on a fuzzy threshold of alpha 0. 05. So let's just hold on to it for a moment, accepting that you know that is just the rule of the game but because of that very rule NHST bears some interesting implications after each decision you make.
So let's turn to the table here. So the status of true reality about the null that is unknown to us is listed column-wise so this column, so either null so this is all about no right so in reality which is not known to us at the null may be true or false right and then the rows represent the decision we make about the true reality after running NHST. So this is our decision about the reality, right?
About the null. and based on the NHST, we either reject the null or fail to reject the null right so these are the only decision we make from the NHST. So under this scheme we are making correct decisions in two cases that are when we reject the null and the null is indeed false in reality so in this case, we make a correct decision and in another case, we fail to reject the null and the null is indeed false (I mean to say true, sorry!
) in reality. However, if you reject the null when the null is true in reality then you just made an error called a type 1 error so this is when you claim that there is something when in fact there is nothing so this is basically false alarm the alarm goes off even though there is no fire. On the other hand, if you fail to reject the null after running an NHST but when in fact the null is false then you just made an error called type 2 error.
So this is when you claim that there is nothing when in fact there is something. So this time the alarm doesn't go off when there is fire. So between these two types of error and statistical hypothesis testing in research is heavily weighted against the type 1 error meaning that researchers are more concerned about saying they found something important or significant, interesting when in fact they didn't.
So they're more concerned about lying about their result or the findings or discovery so that concern is echoed in the nominal alpha 0. 05 which is an explicit decision rule to reject the null only when the likelihood of observing the data or statistics as extreme as or more extreme is less than or equal to 1 out of 20 or 5 out of 100 times instead of being 50 50. So to better illustrate this point let's take a rather extreme example.
Imagine a hypothetical courtroom where the death penalty is possible if the defendant is convicted. So here we have two mutually opposing and exclusive verdicts, innocent which is our null or guilty which is our alternative. So understandably they cannot be considered equally weighted as the consequence of the guilty verdict is permanent and irreversible once the sentence is carried out.
Therefore the judge cannot be impartial in evaluating the evidence from the prosecution and the judge will be much more careful not to return a guilty verdict as much as possible unless the presented evidence supports way beyond a just reasonable doubt making the case extremely strong. So assuming that we do not know what truly happened there is always a chance that we may make a mistake in our decision as we can see from the table here. So in this hypothetical example and the decision and that the consequence sorry the consequence of return the guilty verdict so in other words rejecting the null by mistake which is the false-positive, F+ that can be "dead" serious for both defendant as well as the persecution.
So as such, you want to minimise the occurrence of false positives in making a decision in this context. In other words, you became more conservative in rejecting the null by setting the level of significance alpha as low as possible. So in a nutshell it is this example is kind of similar to how statistical hypothesis testing is used in a research context and why the decision rule to reject the null is set so conservative.
However, there are other situations where minimising the false negatives become more desirable such as in medical screening. Now let's think about another hypothetical example where you want to find out whether you or your partner is pregnant or not so here the pregnant is the alternative and not pregnant is your null. so you want to test the pregnancy using one of these pregnancy screening kits as shown here so the test works by detecting the hormone called the human chorionic Gonadotropin which is hCG in short in the urine sample or blood secreted by the placenta after the fertilised egg implants in a female's uterus.
So it is claimed that the sensitivity of the test is 95% where the sensitivity is defined as the performance of a test to correctly identify a condition (i. e. , pregnancy) when in fact the person is pregnant.
So that is true positive like T+ here so that is the true positive this case right but the sensitivity of a test is only one of the components to characterise the accuracy of the test and there's another component of the accuracy of a test is called specificity which is defined as the ability to exclude people who do not have the condition. Again the two hypotheses cannot be equally so cannot they cannot have equal bearings depending upon whom you ask. For example, let's say if you're an unemployed, unmarried teenager, right then missing pregnancy which is a false negative, so that's false-negative here right by mistake bears much more serious consequence later compared to detecting pregnancy by mistake which is false positive like this one.
So in general, medical screening and diagnostic tests are weighted highly toward sensitivity, not to miss the condition so the dilemma here is that sensitivity and specificity have a reciprocal relationship so if one increases then the other typically goes down. So from these two examples, we can understand that statistical hypothesis testing can be flexible enough to test any hypothesis as long as we are aware of the other consequences after whatever decision we make. But at the same time, you need to remember that you do not prove anything with null hypothesis significance testing.
Nothing is proven, I say this again, nothing is proven with NHST. It is not like a mathematical theorem you try to prove okay you always have to remember that whatever decision you make with NHST is just a probabilistic statement and no such statement can be 100% or 0% true or false under the null hypothesis significance testing. So for example, you still have this probability of making the type 1 error at the level of significance you preset to make the decision in case the null is true in reality.
and from that sense rejecting the null does not mean that you accept the alternative as true. I would just say the evidence strongly supports the h1 instead of saying I accept the h1 or I proved my point. Please and I say it again please do not say that you proved anything with NHST because I have an allergic reaction to the expression and I might die from anaphylactic shock.
So by the same token, failing to reject the null does not mean that we have shown that there is no effect. First of all, we assigned a very generous margin of doubt to the null so that is really not fair to say that there is no effect when we fail to reject the null. And secondly, the absence of evidence is not the same as the evidence of absence.
Well so and by the same token here we have the probability so here the big p represents the probability that you will observe a certain data set right data this vertical line is read "given" so the probability that you will see a certain data set given the theory is true is not the same as the probability that the theory is true given a certain data set okay so they are not the same thing. So you cannot just flip back and forth to mean the same thing. They are not necessarily the same right so you always have to remember that the absence of evidence is not the same as the evidence of absence.
Finally, a statistically significant result is not necessarily practically significant. However, if a result is not statistically significant then it's very likely that the result is not practically significant either and but in any circumstances, you should not equate a statistically significant result with an important result because p-value itself does not tell us about how important an effect actually is nor if the hypothesis of interest is true or false. Statistically speaking a very small and unimportant effect can be statistically significant with a large number of samples by the same token very large and important effect can be missed simply because the sample size is too small.
So far I have talked about all these caveats behind NHST because the practice has been criticised for various reasons for a while. So every so often you may hear people make a call to ban significance tests. Papers and books are written, conferences are held and proceedings are published as you can see from the list of references regarding the issue.
For example, the first reference is the statement released from the American statistics society for the first time in its 177 year history at the time of the publication, on the statistical significance and p values with six principles regarding how to interpret them the main reason behind all these unfavourable attention to NHST is that it is the most common practice used across many scientific disciplines including the vision sciences. This is basically what everybody does in the field of science. At the same time, this is probably one of the, if not the most, overused, misused, and abused method of testing research question and it has become the root of all evil practice in science.
There are lots of reasons behind this criticism intertwined with the history of how NHST has evolved in time but one of the main problems I personally see is from the misunderstanding of the P-value as a threshold for success or failure of an experiment. I think this misconception got worse and solidified over over the years combined with the "publish or perish" culture in academia, which in turn led to publication bias where only the significant results are accepted for publication and their over-representation as a consequence. So when the P-value is seen this way you'll never know what people would do to hack p-value, not that I'm saying they do.
It is probably one of the spookiest and dreadful thing for any scientist to have a near significant result like a p = 0. 06 carved in the Halloween pumpkin shown here. So whenever I have this kind of result, should I just ignore the result when it is just a more than 0.
01 away from the significance? Running an experiment is an expensive business both in time and money wise. So let's consider a different view of towering figures in statistics who are responsible for torturing us with an NHST.
So Ronald Fisher was probably the one who popularised the significance testing and suggested to compute a p-value to evaluate the evidence obtained from a sample against the null and the null representing a population distribution. And there was no concept of the alternative hypothesis. For him, p-value represents a continuum of strength of evidence, for example, higher the p-value weaker the evidence.
So even though he did suggest a few guiding criteria on how extreme is extreme to help decision such as 0. 05 or 0. 01 he did not like the idea about making a sharp judgment call by rejecting or accepting hypotheses.
In case of the non-significant result like before p like a 0. 06 he would say to defer your judgment until more data are available and he suggested any final conclusion should be guided with experience in the context of an effort to understand the result, instead of solely relying on the p-value. On the other hand, Jersey Neyman and Egon Pearson demanded two competing hypotheses to be tested, namely h1 and h2 and they required more rigorous mathematics to determine the decision rules which is much more involving than just determining the level of significance alpha before the experiment and define a rejection region for each hypothesis.
Under their setup, you will always accept or reject one or the other based on the ratio between the two likelihood even though accepting a hypothesis does not mean that you think it is true. It is more about what you are going to do after the decision and you would only act as if the conclusion reached by the test is true, minding the consequence of cost-benefit trade-off by acting upon the decision. so unlike Fisher, a more extreme result is not showing more evidence than a just extreme result for them as they consider P-value as a just a mere proxy with no real evidential properties whatsoever.
Of course, they did not get along with each other. In fact, it is pretty well known in the field how much they hated each other and the null hypothesis significance testing as we know of and practice with today is a bit of amalgamation between the two camps.
