The Criticism of NHST - Annotated Bibliography Example

Psychology Research Cohen. J(1994) "The earth is round (p < .05). American Psychologist, 49, 997-1003 In this article, Cohen heavily criticises null hypothesis significance testing (NHST). He points out that as early as the 1930s, psychologists had been finding problems with this approach, and yet it continues to be promoted, used and respected. Indeed, he states that NHST has actually blocked the advancement of psychology as a science. He argues that fundamentally the problem with this approach is that it doesn’t actually provide the information which is sought from it, even though many people believe that it does. What is sought from NHST is to know what the probability is of H0 being true given any particular data set. However, what it actually does is tell the researcher is that, given that H0 is true, what is the probability of the data., which is not the same thing. However collective denial has allowed null NHST to continue unabated. The major flaw in NHST is that it makes the logical reasoning inherent to it probabilistic, not absolute. Cohen then demonstrates how the nil hypothesis is, anyway, fundamentally always false. Cohen then suggests how psychology as a science can move forward. He states that there is no ‘magical alternative’ to NHST, but that rather psychologists should concentrate on better understanding and improving their data. He argues that approaches such as Exploratory Data Analysis offer a better way of understanding data, which avoids generalization. He also suggests that effect sizes be reported in terms of confidence limits. His overarching warning is that when using statistical techniques to help form well founded theories, the researcher must approach them sensibly and with ‘informed judgement’. He concludes that, given the problems associated with statistical induction, psychology, like other longer established sciences, should rely on replication. Abelson R.P. (1997) "On the surprising longevity of flogged horses: why there is a case for the significance test". Psychological Science January Vol. 8 Issue 1 pp 12-15 Abelson refutes total criticism of NHST, arguing that critics which reject it due to the fact that the null hypothesis is virtually always false are, in fact, exaggerating their claims. Although Abelson accepts that they can generate misleading conclusions, he maintains that NHST can be valuable in defending research from the criticism that its the results can explained by random factors or that the direction of the mean difference has not been demonstrated convincingly enough. He accepts the flaws in the tests but claims that alternatives such as effect sizes or meta-analysis are also not perfect. He therefore suggests that alternative be used more often, but that null hypothesis significance tests should not be abandoned completely. Indeed, he argues it is more due to idiotic use that NHST produce meaningless results, rather than the NHST themselves. Abelson makes three main points in his article. The first is that a distinction should be made between different types of significance tests and their various propensities to illogical and meaningless use. He then moves on maintain that the proposed alternative of using confidence limits is workable and makes sense, as Cohen also suggests. However, Abelson registers concern about whether the debate over whether of not zero should be included in the confidence level means that the same problems as with the NHST could arise. He also warns that using confidence limits cannot be regarded as a way of resolving all the problems inherent to significance testing. Finally, he argues that the criticism against the NHST – its categorical nature – could also be used to argue in favour of it. Psychology, maintains Abelson, needs to be able to put forward categorical statements about results, even though they are in fact uncertain and provisional. Leland Wilkinson (1999) "Statistical Methods in Psychology Journals Guidelines and Explanations" American Psychologist Vol. 54, No. 8, 594-604 Wilkinson and the Task Force on Statistical Inference have released these guidelines in response to the kinds of debates and criticism put forward by Cohen and Abelson above. They discuss statistical testing and its alternative in an attempt to come up with a workable recommended approach for researches in the field of psychology. This includes more recent methods which have become possible via computers. The aim was to move beyond null hypothesis significance testing in order to ascertain both whether it still has a place in psychology research and also what valid and workable alternative really exist. The first recommendation that they give concerns the design of the research method. They state that it is important to be clear about what kind of study is being conducted and to clearly state its goals. Making one study appear to be a different type of study simply to elevate its reputation must be avoided. Although some research approaches may be promoted as being more reliable or more valuable than others, in actual fact all methods have their advantages and disadvantages and should be used with due care and attention. They then give some advice about population. They state that the population (participants, stimuli or studies) must be clearly defined, and any control groups must also be clearly defined. This is of extreme importance because how a population is defined affects the conclusions drawn in the research study. Students must therefore take care, the article advises, to distinguish between a class of objects and a statistical population. Inferences about a population are made through statistical methods, they explain, while inferences about a class are made through logical or non-statistical methods. A recommendation is then put forward about the sample. They state that the sampling procedures must be clearly explained and any criteria for inclusion or exclusion must be made explicit. The sample size for each subgroup must also be stated. The article warns that interval estimate for clustered or stratified random sample are different from those for simple random samples, and statistical software can be used for this purpose. A convenience sample (a sample which is not random) can also be used, but the criteria for selection must be clear. It is essential that a convenience sample is not masked as a random sample simply because the academic community regards the random sample as more scientifically rigorous. The article then moves on to advice about assignment. They state that the assignment of units to levels of the causal variable is essential for research involving causal inferences. They recommend that random assignment allows for the strongest possible causal inferences and avoids extraneous assumptions and research bias. However, enough information must be provided to prove that the process for making the assignments is random. They stress that an apparently random assignment can be easily deliberately corrupted and so the researcher must prove that their assignment is genuinely random. A pseudorandom sequence from a computer generator or published tables of random numbers should therefore be used. Where non-random assignment is more suitable, the effects of covariates must be limited. To do this they must be identified, measured and their affects must be adjusted for by analysis or design. It is essential to clearly state strong assumptions, potential sources of bias and plans to minimize dropouts, non compliance and missing data. When dealing with measurement, the variables must be stated, measured and justified. It is important to choose a suitable name for each variable, and preferable one which reflects how the variable is measured. This precision helps to avoid excessive generalisation. In terms of instruments, all necessary information must be stated. For example, with a questionnaire, the psychometric properties of its scores in relation to the way it is used in a population must be summarized. If the research involves the use of physical apparatus, enough detail must be provided to allow another researcher to accurately reproduce the measurement process. Reliability of the instrument must also be demonstrated, and it must be proved to measure only that which the research seeks to measure. The article then advises researchers to in terms of procedure, any anticipated sources of attrition must be described, as well as how this may affect the generalizability of the results. The conditions under which the measurements were taken should be clearly described and methods to combat bias must be made clear. It is essential that a researcher be aware of all possible areas of bias and take measures to overcome them. In terms of power and sample size, the size and justification for the size must be stated. The effect sizes, sampling, measurement assumptions and analytic procedures used in power calculations must also be recorded. The article stresses that it is essential to show how effect-size estimates have been derived from previous research, so as to avoid the accusation that they have been taken from the actual data of the study or even constructed to justify the sample size used in the study. This kind of transparency lends credibility to the research. The article then moves on to give advice about the results section of a research project. In this section they advise that the researcher be immediately open about complications and unanticipated events in data collection as well as methods used to overcome these problems. It must also be made clear how the actual analysis is different to that planned before meeting with unforeseen problems. It is must also be clear in this section what techniques are used to prove that results derived cannot simply be explained by anomalies in the data. In the analysis section, the article advises that the analysis and design be well matched to the research question, and admits that this is not an easy task given the high number of quantitative research methods available to the researcher. It advises to err on the side of simplicity, since although complex designs are sometimes necessary, more simple, classical approaches can often be sufficient. There is no point in choosing a complex or radically new method in order to impress reviewers. It is more scientifically rigorous and therefore preferable to choose the most effective and simplest method possible, according to the needs of the study. Three recommendations From reading the above articles it is clear that the first recommendation to students is to treat statistical tests with caution. While no test should be excluded, whether it be NHST, confidence limits or effect sizes, it is also important not to hold one test as more intrinsically valuable than the others. All tests have their advantages and disadvantages and it is therefore necessary to choose between them based on the needs of the particular study underway, using logic and common sense. The second recommendation would be to be transparent. Make choices about the way the study will be conducted and be clear about them. Do not carry out one type of research masking it as another, because this will simply damage your reputation of objectivity as a researcher. Instead, decide on the best approach for your study and then justify it. Do not use the approach which is most fashionable, but the one that works best. With the appropriate justification, this honest and transparent approach will result in greater scientific credibility. The third recommendation would be to explain, define and outline everything very clearly. Make sure that every decision which is made is valid and has the necessary explanation to back it up. It should be clear that the study can be reproduced by future researchers. This approach will also ensure that researcher bias is avoided. By demonstrating that you are aware of all the issues surrounding your research project and have come up with appropriate approaches to overcome them, it will be clear that the study has been meaningfully constructed and conducted. Read More