Evaluation approaches - Research Paper Example

?EVALUATION APPROACHES This is a review of the extant literature on three different types of evaluation in research: the archival evaluation, multi-method or systemic evaluation and comparative-descriptive evaluation. These approaches were examined through three different studies with different research subjects. The review revealed that the diversity in evaluation approaches provides researchers a comprehensive array of evaluation frameworks that would best suit the requirements and objectives of their respective research initiatives. Evaluation techniques are crucial in any form of research regardless of whether it is a qualitative or quantitative research or a combination of both. This is because evaluation or measurement determines the interpretation of data gathered and, hence, impacts the findings and conclusions of a particular study. There is an ongoing debate as to which technique of evaluation or measurement can best interpret probability. According to Hand (1996), this debate is stimulated and aggravated by controversy over the legitimacy of applying different classes of statistical methods to data that emerge as a consequence of different kinds of measurement activity. (p. 446) The body of literature supports this theme, from theories to actual research examples. The following studies undertaken to investigate three separate issues with different evaluation and measurement techniques highlight the differences by which the studies went through with their investigations. Archival Evaluation With an aim of investigating the validation approaches in Information Systems research, Boudreau, Gefen and Straub (2001) used an evaluation technique anchored on a comprehensive review and analysis of literature on the subject, specifically comparing two time periods. The researchers refer to it simply as a literature review except that such review was undertaken in a highly systematic manner. Clearly, the study is qualitative in character and it aims to describe the validation trends from 1997 to 1999. The employment of the review of literature as a foundation for inferences and conclusions is valid since qualitative research aims to find depth when quantitative studies seek breadth. Ambert et al. (1995) for instance, explained that because of this focus, evaluation could draw from a smaller group of research subjects or respondents instead of a large representative sample of the population. (p. 880) Bourdeau et al. selected five journals to represent the periods by which the study sought to compare. Out of these journals, the author evaluated a total of 193 articles selected based on specific criteria or attributes. (p. 6) There were at least eleven such attributes (i.e. research type, content and construct validity, research method, etc.) and that they were, in turn, evaluated according to a validation test statistic – a second independent coder. (p. 6) The resulting data allowed for the identification of relationships and the evaluation of the efficacy of validation approaches based on previous documents so that the research objectives are finally met. It was found, for instance, that instruments are more frequently validated today than they were 11 years ago. (p. 7) Multi-method Evaluation Boyd et al. (2007), investigated community health services with emphasis on the need for stakeholder participation and dialogue. For this purpose, they designed a multi-method approach in order to cover the numerous stakeholders involved in community health services: health practitioners, community leaders, marginalized people, etc. Three evaluation frameworks are designed for goals, the stakeholders and the organization. The researchers employed the so-called critical systems thinking (CST) principle in this strategy, which is typified by three fundamental characteristics, namely: Critical awareness for the examination of assumptions in addition to the conditions that led to their emergence; Improvement both on temporary and local levels done in an adequately informed manner; and, Methodological pluralism pertains to the use of several or a combination of methods, using the strengths of each to compensate for the weaknesses of the others. (p. 1306) The multi-method approach also called as systemic approach has been justified by the sheer number and diversity of the sampling population. Each has differing stakes on the issue of community health and their contribution to the data could only be appropriately evaluated by different frameworks. The researchers highlighted the fact that the evaluation strategy is also flexible and responsive, drawing upon aspects of the three evaluation approaches. These supposedly enable the researchers to comprehensively cover their sampling population and, hence, interpret data authoritatively because the approach has the ability to address complexities in addition to the ability to approach organizations effectively either from an objectivist or subjectivist point of view. Comparative Evaluation Comparative evaluation is not unlike a descriptive approach to evaluation by which concepts or samples are defined and explained so that their strengths and weaknesses are identified. This is the approach used by Duffield who evaluated the various sampling methods that can be applied to passive Internet measurement research. His objective was to determine the best technique to comprehensively cover the complexities of the Internet as a large distribution system and the network infrastructure in general. Within this framework, numerous concepts were introduced such as traffic measurement, network management, in addition to the various sampling methods that were evaluated. The proprietary historical review was used but the subject is recent, hence, the extant body of research is limited. But the main issues evaluated where the types of passive sampling methods, which include: polling management information base; packet monitoring; and, flow monitoring. (p. 477) Each of these passive sampling methods were explained, then compared. The subjects allowed for a purely descriptive evaluation approach because they are technical in nature. Checking from a manual or testing it from a machine can easily verify a point or an inference. Here, the evaluation is not about the measurement of probabilities because the process can yield accurate and quantified results. As the reviewed literature highlighted, evaluation is crucial in the outcome of an academic research. The appropriate evaluation techniques determine the validity of the research findings and conclusions. Another important variable that was revealed, however, is that the diversity in evaluation methods do not necessarily mean that research designers could not agree with about what should be the right framework, which underpins the drive for standardized evaluation models. Studies have different goals and different methodologies, with different sampling profile and populations. The large number of techniques can be interpreted as comprehensive and vibrant research frameworks, some of which are integrated into multi-approach evaluation model in order to address complex research objectives. References Ambert, A., Adler, P., Adler, P. and Detzner, D. (1995). "Understanding and Evaluating Qualitative Research." Journal of Marriage and Family, vol. 57, no. 4, pp. 879-893. Boyd, A., Geerling, T., Kagan, C. Midgley, G., Murray, P., and Walsh, M.P. (2007). "Systemic Evaluation: A Participative, Multi-Method Approach." The Journal of Operational Research, vol. 58, no. 10, pp. 1306-1320. Boudreau, M., Gefen, D. and Straub, D. (2001). "Validation in Information Systems Research: A State-of-the-Art Assessment." MIS Quarterly, vol. 25, no. 1. pp. 1-16. Duffield, N. (2004). "Sampling for Passive Internet Measurement: A Review." Statistical Science, vol. 19, no. 3, pp. 472-498. Hand, D.J. (1996). "Statistics and the Theory of Measurement." Journal of the Royal Statistical Society, vol. 159, no. 3, pp. 445-492. Read More