The Impact of Fatigue on Human Raters when Scoring Speaking Responses by G. Ling, P. Mollaun and X. Xi - Article Example

Article 1: Fatigue on Human Raters Introduction The article “A Study of the Impact of Fatigue on Human Raters when Scoring Speaking Responses” was conducted in 2014 by Guangming Ling, Pamela Mollaun and Xiaoming Xi. Informed by a research gap on the impact of time-related fatigue on rating quality in language testing, these authors set out to examine the effects of fatigue on human raters’ scoring performance within the context of TOEFL iBT Speaking Test. The authors succeeded in showing how rating accuracy, quality, and consistency vary across time due to fatigue-related effects. Summary The study employed a quantitative experimental research design to “examine the effects of fatigue on human raters of audio responses by comparing rating accuracy and consistency at various time points throughout a scoring day, under shift conditions that differ by shift length and session length” (Ling, Mollaun, and Xi 479). The authors consulted other sources to show that the burden placed on raters’ concentration and cognitive capabilities in scoring speaking responses may trigger time-related fatigue and also aggravate scoring accuracy and consistency. Overall, the findings proved that (1) scoring productivity and quality differ widely across hours irrespective of shift conditions, (2) shorter scoring shifts (e.g., 6-hours shifts and those that switch to a new speaking task every 2 hours) enjoy higher rating accuracy, greater hourly productivity, and greater rating consistency across time than longer scoring shifts, and (3) longer scoring sessions and expanded scoring length are positively associated with increased fatigue. Critique The main argument of the study is that fatigue occasioned by longer scoring shift length and session length affects the human raters’ rating quality, accuracy, as well as consistency. The writers’ perspective is embedded in the fact that scoring shift length and session length in language testing may indeed occasion undesirable fatigue-related outcomes. This perspective is consistent with that of other authors, who argue that exam proximity (time between cognitive tasks) has a powerful impact on performance due to cognitive fatigue (Fillmore and Pope 12). Drawing from this comparison, it can be argued that the writers’ ideas on fatigue are similar to those of other writers. Indeed, their ideas are effective in demonstrating how scoring shift length and session length affect the rating quality, accuracy, and consistency in language testing. The authors do not show any bias and are qualified to write in this area based on their affiliation with the United States Educational Testing Service. Their main ideas are easily agreeable based on the experiments done in the study to prove them; however, the authors fail to discuss how other factors (e.g., regional traditions of language testing) can impact rating quality, accuracy, and consistency (McNamara and Knoch 556). Although their introduction and literature review sections are easy to understand, other areas (e.g., methodology and results sections) seem complicated to a general reader. Additionally, although the writers use reputable sources to back their arguments, they fail to connect the available literature to their main findings in a manner that could reinforce their main contributions in the area. Overall, the study can be recommended to students and professionals who may want to understand the effects of fatigue in language testing and scoring. Conclusion The article is effective in demonstrating how fatigue-related factors affect the rating accuracy of human raters in language testing. Language testing and scoring professionals may find this article interesting as it provides a glimpse of ideal scoring shift length and session length for optimal productivity in rating quality, accuracy and consistency. Works Cited Fillmore, Ian and Devin G. Pope. The Impact of Time between Cognitive Tasks on Performance: Evidence from Advanced Placement Exams, 2012. Web. 12 Nov. 2015. Ling, Guangming, Pamela Mollaun and Xiaoming Xi. “A Study of the Impact of Fatigue on Human Raters when Scoring Speaking Responses.” Language Testing. 31.4 (2014): 479-499. ERIC. Web. 12 Nov. 2015. McNamara, Tim and Ute Knoch. “The Rasch Wars: The Emergence of Rasch Measurement in Language Testing.” Language Testing. 29.4 (2012): 555-576. ERIC. Web. 12 Nov. 2015. Article 2: Framework for Classroom-Based Assessment Introduction The study titled “Developing a Comprehensive, Empirically Based Research Framework for Classroom-Based Assessment” was conducted in 2011 by Kathryn Hill and Tim McNamara. The central themes of the article revolve around decentralizing responsibility for assessment to classroom teachers, dimensions and scope of classroom-based assessment, and underscoring the immense significance of assessment on learning. Summary This study used a qualitative research design (ethnographic methods) to develop “a comprehensive framework for conducting research on classroom-based assessment” (Hill and McNamara 395). The proposed framework was based on two themes namely dimensions (evidence, interpretation and use) and scope (how language teachers carry out CBA, what they look for when evaluating students, and what theory or standards do the employ in assessing students). The qualitative design was employed to investigate and broaden the variety and type of questions that should be included in CBA research. A summary of the results demonstrates that the classroom based assessment framework was effective in helping language teachers to (1) plan, frame, conduct and use assessment, (2) know what to look for in advance learning, feedback, and reporting, (3) understand the standards to use in teaching Indonesian as a foreign language, and (4) comprehend learner understandings of language learning and assessment. Critique The main argument of the article is that a classroom-based assessment framework based on the summarized themes (assessment dimensions and scope) may be effective in assisting language teachers to assess a whole range of issues in language learning. The writers’ perspective is to develop a classroom-based assessment model that innovates in considering epistemological basis for observed assessment practices, allows for the inclusion of a diverse range of data including the more intuitive forms of teacher decision-making, and uses a school-based foreign language setting. These ideas are more inclusive than those of other writers and they help shape the writers’ argument about the validity and viability of the framework. The authors are university professors, hence are qualified to write in this area. Although the writers appear unbiased in their arguments and reporting, it is difficult to agree with their argument that a process-oriented classroom-based assessment model represents an improvement from earlier models that used criteria and standards. As reported in the literature, criteria and standards of assessment still represent some of the best approaches used in evaluating language learners (Parker et al. 34). Although the introductory sections of the study are easy to understand, it is generally felt that the findings and discussion should have been presented in a much simpler format for ease of understanding. A good practice for presenting the findings revolves around the use of the deductive approach (using a structure or predetermined framework) with the view to achieving a sense of direction and logical flow (Chenail 67). Although the writers use reputable sources to strengthen their arguments, they nevertheless seem to make simple concepts harder based on their way of analyzing the qualitative findings. Overall, the work can be recommended to researchers and classroom practitioners with an interest in language learning and assessment. Conclusion The study has been effective in demonstrating how the actual classroom processes revolving around assessment dimensions and scope can be effective in undertaking research on classroom-based assessment. This article is useful in assisting teachers to develop effective standards and benchmarks for language assessment in a school-based foreign language setting. Works Cited Chenail, Ronald J. “Conducting Qualitative Data Analysis: Reading Line-by-Line, but Analyzing by Meaningful Qualitative Units.” Qualitative Report. 17.1 (2012): 266-269. ERIC. Web. 12 Nov. 2015. Hill, Kathryn and Tim McNamara. “Developing a Comprehensive, Empirically Based Research Framework for Classroom-Based Assessment.” Language Testing. 29.3 (2011): 395-420. ERIC. Web. 12 Nov. 2015. Parker, David C., Mathew K. Burns, Kristen L. McMaster and Edward S. Shapiro. “Extending Curriculum-Based Assessment to Early Writing.” Learning Disabilities Research & Practice. 27.1 (2012): 33-43. Academic Search Premier. Web. 12 Nov. 2015. Read More