The Concepts of Reliability and Validity in Research - Essay Example

The concepts of reliability and validity are fundamental to all forms of scientific research; in fact, “science” could well be defined as the effort to draw reliable, valid conclusions about the world around us. As researchers attempt to study human society, thought, and behavior, the concepts of reliability and validity become more complex and intractable. Since research that is unreliable or invalid—however those terms are defined—is more or less by definition a waste of time and effort, it is important for both quantitative and qualitative researchers working outside of the physical sciences to agree upon meaningful and practical definitions of these terms. Such definitions depend, in turn, upon philosophical views regarding the nature, and even the purpose, of knowledge. The concepts of reliability and validity, along with the rest of the “scientific method”, were formulated as part of the rise of the physical sciences as an organized activity over the last several centuries. These concepts were associated with the worldview that was eventually formalized as Positivism; and to this day, the stereotypical view of science is essentially a Positivist one, even though most philosophers and historians of science consider this view to be somewhat outmoded. Positivism views reality as both objective and knowable, with laws that can be deduced through a process of orderly and meticulous observation and measurement. The fundamental source of knowledge is direct observation, and all notions of how the universe operates are to be tested against past and future observations. Beyond having a desire for knowledge, precision, and elegance, the observer him/herself is not relevant in this system; since objective reality and its laws are “out there” ready to be discovered and understood, the scientist as an individual does not (or is not supposed to) enter into the equation. A Newton or an Einstein might achieve an accurate understanding of an aspect of reality earlier than others might have done, but “Newtonian” mechanics or “Einsteinian” relativity would ultimately exist, and would be essentially the same, without Newton or Einstein. This is the classic view of how science works; and there is no question that the physical sciences, pursuing an essentially Positivist agenda, have enjoyed phenomenal success. In the Positivist scientific worldview, reliability is measured by repeatability—that is, a measurement, observation, or procedure that consistently yields the same result when carried out in the same manner is treated as an established fact. The first way in which such reliability is established is via the mathematical treatment of measured results obtained by the original researcher(s) in order to establish statistical significance; the second stage in establishing reliability is the replication of results by other researchers to ensure that the original results were not affected by faulty equipment, incorrect or poorly documented procedures, or some other form of error. Reliability, in turn, implies validity—at least regarding the facts observed, if not always regarding the conclusions drawn from them (Altheide & Johnson, p. 487). The classic method of establishing validity for a scientific theory is to predict the results of experiments or measurements that have not yet been carried out; a theory that yields accurate predictions is obviously more likely to be correct than one which does not. In some cases a theory may turn out to be invalid even though it was originally based upon reliable data; a classic example is Newtonian physics, which is highly successful in explaining and predicting “garden variety” phenomena, but fails when confronted by very high velocities, very strong gravitational fields, and so on. One important aspect of Positivist science is that it tends to be reductionistic: that is, in order to obtain accurate, reliable measurements, scientists focus on relatively narrow aspects of reality. As systems become increasingly complex, their behavior becomes more difficult to model and predict. As a result, “hard” science does not easily lend itself to the investigation and understanding human beings. Over the course of the 20th Century, it became apparent that the physical universe was less easily knowable than earlier thinkers had believed. In particular, Quantum Mechanics established that there were limitations to the observable, and that the act of observing could alter the behavior of what was observed. As a response to such problems, thinkers like Karl Popper created a revised conceptual underpinning for science: Postpositivism. Postpositivism acknowledges that even in the physical sciences, we can never perceive truth directly—all knowledge has an element of the observer in it. Reality is still considered “objective”, but our knowledge of it is limited and subjective. Accordingly, our ideas about reality cannot be treated as “objectively true”; the best we can do is to say that a particular theory has not yet been falsified by contrary observations. Despite its acceptance of the subjectivity of knowledge, Postpositivism does not materially change the procedures of Positivist science; and reliability and validity are established in the same ways under a Postpositivist worldview as in a Positivist one. Postpositivism offers a somewhat different philosophical understanding of how science works, but it does not materially change the way in which physical scientists go about their business. Quantitative research in the social sciences has traditionally been based largely on the Positivist paradigm, attempting to replicate the kind of success and assurance that the physical sciences have enjoyed (Altheide & Johnson, p. 487). Just as the physical sciences assume that our perceptions and measurements reflect a genuine reality “out there”, Positivist social-science research see people’s attitudes, beliefs, and personal characteristics as “real” phenomena that can be measured through the use of appropriate investigatory methods. Accordingly, such research has typically involved the use of tools such as surveys and structured interviews, designed to collect a relatively limited amount of information from a sufficiently broad sample of respondents to achieve statistically meaningful results (Bryman, p. 104). However, the nature of subjects studied—that is, human beings and the organizations they construct—and the constraints under which researchers must operate impose considerable challenges to researchers attempting to achieve reliable and valid results. The range of subjects for quantitative research is quite broad, including political polling, market research, organizational behavior, and many other fields. In most cases, quantitative researchers are not able to conduct experiments in which they survey subjects before and after changing some condition; instead, they must measure a number of variables at a single time, and then attempt to find correlations among these variables to suggest likely cause-and-effect relationships (Bryman, p. 106, 117-118 et seq.). Key to this effort is the proper identification of independent variables. In some cases, it may be easy to do this; for example, if polling shows a strong correlation between female gender and a preference for a particular political party over another, it is not hard to defend the position that being female increases the likelihood of this political preference rather than that favoring this party increases the likelihood of one’s being female. In other cases, however, it may be much harder to differentiate between dependent and independent variables; and it may even be the case that none of the variables measured are truly causative, but that the real causative factors are outside the bounds of the survey. Other problems in establishing reliability and validity for quantitative research involve the sensitivity of the results obtained to the exact procedures used to gather information. The details of the wording used in surveys and structured interviews can strongly influence subjects’ responses, as can the way in which surveys and interviews are presented. Further, even the order of the questions in a survey or interview can affect how people respond to them—presumably because different sequences of questions can trigger different thought processes in some respondents. Careful research design can minimize some of these problems, and careful statistical analysis (by comparing, for example, responses obtained by different interviewers or different versions of a survey) can identify other issues so that the results can be corrected. Still, the fact that quantitative human research is subject to such challenges means that its findings are often less reliable than researchers could desire. In order to achieve maximum reliability and validity, quantitative research should ideally be performed with a large sample that accurately represents the population being investigated. The best way to ensure a representative sample is to select members of the population at random, adjusting the results to reflect the known composition of the overall population (for example, to adjust for any discrepancy between the sample’s gender composition and that of the sampled population). In reality, however, much quantitative research is carried out using “samples of convenience” rather than truly random population samples (Bryman, p. 107 et seq.); and even when research participants are selected at random, bias is likely to creep in because the characteristics of those who choose not to participate in the research may well be different from the characteristics of those who do (Bryman, p. 112). (As an example of this, pollsters in the recent American elections needed to adjust their results to reflect the observed fact that American conservatives are, on average, less willing to be surveyed than are liberals.) Even when all the issues of research design and sampling are dealt with, the fact remains that quantitative research can reveal relatively little about the people investigated; the kind of brief questions and answers used in this type of research do not really tell very much about the real thought processes of the people studied. In order to discover more of the human “inner reality”, many researchers now prefer to engage in qualitative research. Qualitative research, involving lengthy, detailed interviews, focus groups, and other loosely-structured techniques, investigates subjects in much greater depth than quantitative research, but also necessarily involves much smaller samples than quantitative research. While the knowledge gained can be much “richer” than what is learned in quantitative research, qualitative research must deal with special challenges relating to reliability and validity because of the small samples involved, and because by its very nature qualitative research tends to be highly subjective. Because the qualitative researcher must engage much more closely with his/her research subjects than the quantitative researcher, s/he can no longer even pretend to be an invisible, neutral observer. As Altheide & Johnson put it, “…the scientific observer is part and parcel of the setting, context, and culture he or she is trying to understand and represent” (p. 486). Accordingly, qualitative research is not typically based upon Positivism; instead, most qualitative researchers consider themselves Realists, Postpositivists, or even “radical Postpositivists” (Scheurich, p. 81). Realism can perhaps best be described as an attempt to come as close as possible to a Positivist approach, while realizing that the observer in qualitative research can never be the neutral, uninvolved investigator of the Positivist paradigm. Realism is based on the idea that there is an objective reality “out there”; and the job of the Realistic researcher is to perceive and describe that reality. According to Hammersley (1992, quoted in Altheide & Johnson, p. 488), “An account is valid or true if it represents accurately those features of the phenomena that it is intended to describe, explain, or theorize.” Other researchers have objected that given the inherently subjective nature of observation, there is no way in reality to determine whether this “accurate representation” has been achieved. One response to such objections is “Analytic Realism” (Altheide & Johnson): While Analytic Realists do believe that objective reality is “out there”, they acknowledge that qualitative researchers are inevitably working from their own perspective and can never be truly “objective”. The best that conscientious researchers can do is to be explicit about “where they are coming from”, so that their audience can interpret their research appropriately (p. 495). Other qualitative researchers have rejected Realism entirely; instead of aiming for an “accuracy” that they feel is essentially meaningless, these “radical Postpositivists” substitute any of numerous forms of more purpose- or ideology-based validity; Altheide & Johnson refer to these as “hyphenated validities” (p. 488). For example, researchers may support a standard of validity based on liberation, empowerment, feminism, Marxism, and so on. Such concepts of validity can work well in defining what pieces of research are considered valid and useful by particular communities of researchers and research-users; but this kind of “subjective validity” is not likely to apply well outside such communities. Given the small sample sizes involved in qualitative research and the personal involvement of the qualitative researcher with his/her subjects, reliability can be a very difficult issue. Clearly, the statistical tools used by physical scientists and quantitative researchers are of little value to qualitative researchers dealing with very small samples. One tool for increasing reliability of such research is to have several researchers independently code the recorded responses of research subjects; this approach, while it does not solve the problem of limited sample size, at least avoids some of the subjectivity in the process of determining what each subject expressed (Hoijer, p. 17). At best, qualitative research can do more than quantitative research to “get inside the subject’s head” and determine what (and how) people are really thinking. However, by its nature qualitative research is highly subjective and of suspect reliability; ultimately, we may learn a great deal about how a particular subject thought and felt at a particular time, but this will be of only limited value in determining how other subjects will think and feel. It has even been noted that different researchers studying the same topic have sometimes come up with significantly different findings—surely a warning to both the creator and the consumer of qualitative research (Hoijer p. 15, quoting Le Compte & Goetz 1982, p. 37). References Altheide, DL & Johnson, JM, 1994, “Criteria for Assessing Interpretive Validity in Qualitative Research”, Handbook of Qualitative Research, Denzin, NK & Lincoln, YS (eds), Sage Publications, London, pp. 485-99 Bryman, A, 1999, Research methods and organizational studies, Unwin Hyman Ltd, London, Chapter 4: Survey research, pp. 104-34. Hoijer, B, 1990, “Reliability, validity, and generalizability: three questions for the qualitative reception researcher”, The Nordicom Review, vol. 1, pp. 15-19. Scheurich, JJ, 1997, Research method in the postmodern, The Falmer Press, London, Chapter 4: The Masks of Validity: A Deconstructive Investigation, pp. 80-93. You should make reference to relevant sections of Wimmer & Dominick DON’T HAVE Bauer & Gaskell chs. 18 & 19; DON’T HAVE SR 10.1: Hoijer OK SR 10.2: Altheide & Johnson OK SR 10.3: Scheurich. OK DON’T HAVE ANY OF THE BELOW Asamen, JK & Berry, GL (eds) 1998, Research paradigms, television and social behaviour, Sage, London, pp. 111–28. Dey, I 1993, Qualitative data analysis, Routledge, London, pp. 249–63. Frey, LR, Botan, CH, Friedman, PG & Kreps, GL 1991, ‘Designing valid communication research’, ch. 6 in Investigation communication: an introduction to research methods, Prentice-Hall, Englewood Cliffs, NJ, pp. 118–38. Frey, LR, Botan, CH, Friedman, PG & Kreps, GL 1992, Interpreting communication research, Prentice Hall, NJ, pp. 196–8. This looks at the issue in relation to content analysis. Glesne, C & Peshkin, A 1992, Becoming qualitative researchers: an introduction, Longman, pp. 140–7. Hansen, A, Cottle, S, Negrine, R & Newbold, C 1998, Mass communication research methods, Macmillan, London, pp. 18–19; pp. 120–1 (content analysis). Neuman, WL 1997, Social research methods, 3rd edn, Allyn & Bacon, Boston, pp. 138–46; 190–5 (quantitative research). Smith, MJ 1988, Contemporary communication research methods, Wadsworth, Belmont, CA, pp. 183–91; 46–50. Sarantakos, S 1998, Social research, 2nd edn, Macmillan, South Yarra, pp. 78–86. Stacks, DW & Hocking, JE 1992, Essentials of communication research, Harper Collins, New York, pp. 121–30. Read More