Education: Critique of the article Evaluating Workplace English Programs - Essay Example

? Education: Critique of the article “Evaluating Workplace English Programs.” This review is chosen because it deals with an issue that I have observed often in my own working experience: the low quality of assessment that occurs in many workplace learning situations, both at the outset of a program and at its conclusion. This makes it very difficult to measure progress for the individuals involved, and it casts doubt on both the value of workplace English programs, and potentially also the competence of instructors. These are issues which are highly significant for the future of this professional area, and the article is an attempt to clarify some of the theories and practices which contribute to this situation. The authors set the context for the article by describing a boom in the provision of workplace English classes which occurred between 1988 and 1994 when the US Department of Education committed some 133 million dollars to such programs with a view to “better the workers’ reading, writing, and communication skills, which would lead to improved worker productivity.” (Ekkens and Winke, 2006, p. 267) The funding ceased, leaving individual companies with the costs of continuing this initiative, and also with the task of ensuring the suitability and the quality of provision for their own specific needs. While larger companies can afford to hire a specialist service provider to deliver and monitor programs, smaller companies are often in some difficulty with this complex task. The article addresses the need to identify practical and efficient ways of judging the progress that workplace learners of English make. One issue which causes difficulty is the proliferation of standardized tests which could be used to evaluate workplace English programs. The authors list five professionally produced and standardized tests, each of which specifies that training in the testing method must be undertaken before the tests are administered. This in turn implies high costs either in training such personnel or in hiring such staff to undertake the work. The focus of these five is national and general, rather than local and specific which raises the issue of whether they are in fact at all suitable for the needs of small and tightly focused companies which may have priorities and conditions quite different from those of the national testing bodies. So far the authors have outlined the context well, and have identified some of the practical difficulties that practitioners in the field encounter. The literature review at the start of the article summarizes the work of Sticht (1999) on the gap between what testing systems test, and what participants in English language courses and their companies actually require in terms of day to day demands of the job. A case is made for more personalized assessment methods such as “portfolios, journals, observations checklists, ans diaries” (Ekkens & Winke, 2006, p. 269) following research with a pedagogic rather than quality control focus (Huerta-Macias, 1995; Grognet, 1996 and 1997; Shohamy, 2001). Although this research shows that there are clear benefits for individual learners, in terms of becoming more aware of and taking responsibility for their own learning progress, and there are some other advantages such as a rise in learner self-esteem, the authors note that these methods take more time to develop, operate and score, and most significant of all for workplace learning providers, they are often not recognized by funding bodies. Critics of these alternative assessment methods maintain that they are too subjective and therefore unreliable as measures of progress or predictors of workplace success (Brantmeier, 2006). The authors cite an interesting article by Peirce, Swain and Hart (1993) but do not take full account of this article’s insights, albeit in a context of Swedish students learning French, into the serious discrepancy that exists between learners’ own assessment when compared with formal tests. The article formulates the research question very clearly as “investigating the extent to which workplace English language learners demonstrate learning as measured by (a) standardized assessments and (b) alternative assessments, in this case, learning journals.” (Ekkens and Winke, 2006, p. 270). It is therefore a comparative study, describing two different ways of measuring progress in workplace learning. A further aim is mentioned, namely “to qualitatively evaluate the extent to which each contributes to an overall assessment of the benefits of the workplace English classes.” (Ekkens and Winke, 2006, p. 270). These formulations demonstrate that the authors have both narrowed down the scope of the study (to just one alternative assessment method, i.e. learning journals) and opened it up to a much bigger dimension, which is that of overall assessment of benefits. The latter definition seems to include benefits that could be, social, and even personal, and does not prioritise academic or business benefits. The empirical part of the article starts with a description of the methodology. The fieldwork was conducted on three intermediate workplace English courses, anonymized as Alpha, Beta North and Beta South, involving two different teachers and two slightly different delivery timings, but all three consisting of a total of 30 hours of instruction over ten weeks. The sample size consists of 36 individuals in total, of which 21 completed both the pre and post tests and the voluntary learning journal. Demographic factors such as gender, age, length of stay in the United States, length of employment with the current employer, country of origin, and level of education were gathered and presented in tabular form for the final sample of 21. There are two potential problems in the way this fieldwork was set up. The first is the relatively small sample size of 21 individuals, which renders any outcome interesting for those involved, but hardly transferable to a wider context. One of the groups (Beta North) consisted of only four individuals, all female and aged between 47 and 51, for example, which raises the question whether demographic variables (rather than assessment methods) might be the more relevant factor in any analysis. The second difficulty is that one of the researchers was the director of the company providing the training, and it is noted that she “supervised the design and the instruction of the classes” and “was in charge of the assessment program and the reporting of the assessment scores to stakeholders, including the funding agencies.” (Ekkens and Winke, 2006, p. 270) It is commendable that this involvement is clearly revealed, and the consequent weaknesses in the study readily admitted, but on the other hand it is less than ideal for one so involved in the programs to be conducting the research. The likelihood of bias, whether conscious or unconscious, is much greater in instances where the researcher participates in aspects of the program being researched. The fact that she clearly has a financial stake is also an important weakness. Pre and post-tests results for listening and reading were analyzed in a quantitative way using SPSS. The data from the voluntarily completed journals were analyzed using a grounded theory approach. The authors cite Charmaz (2006) but do not elaborate exactly how this theory relates to their own fieldwork. This is a weakness in the article because it means that others would find it hard to replicate this study without knowing about the way grounded theory affected the analysis of the learning journals. The discussion and results sections emphasize an idea that was introduced at the start of the article, namely “The classes contain nontraditional learners, who may, by extension, also be non-traditional test takers, unfamiliar with the standards-based test format, which, consequentially, may negatively impact their test performance.” (Ekkens and Winke, 2009, p. 266). This is suggested as an explanation for the somewhat surprising outcome that more than half of the participant demonstrated a decrease in performance level when starting and closing scores are compared. Another explanation is that the test was administered too soon in the learning cycle, after 30 hours of learning rather than the 100 that Stich (1999) recommends. The authors conclude that formal tests are a poor way of measuring progress. The authors assess the learning that is demonstrated in the learning journals as being much more positive, although the contribution of the learning journals seems to have been in the area of improving participants’ own feelings of “low self-esteem or peceived lack of ability” (Ekkens and Winke, 2009, p. 277). Significantly, conclusions are drawn that imply a beneficial effect in language use and in workplace performance. An important recommendation at the end of the article reveals the researchers’ concern with funding imperatives: they state that program organizers should supply evidence from alternative assessments such as learning journals, along with formal test results, even when these are not required, and that funding bodies should reconsider their current over-reliance on formal tests. The data is explained clearly, and samples of actual journal entries are given which demonstrate the type of data used. One factor which was not discussed is the whole area of ideological issues that underlie many language training decisions, including for example the imperatives of business efficiency and the political dogma that often influences the design and operation of government funding programs. Overall the article is interesting, but not very convincing, since it clearly demonstrates the inadequacy of formal tests, but it fails to demonstrate any true validity for the learning journals, beyond the feelings of participants. Inclusion of the test results of those who chose not to complete the learning journals would have been an interesting piece of information, and it might have considerably changed the results. It may have shown, for example, that those who completed learning journals did less well in the tests, than those who focused on the formal assessment methods. No evidence is given for any measurable improvement in work performance, and this too, is a weakness. In referring to the sponsoring organizations the article also produces an outcome related more to funding issues than pedagogic ones, which suggests the aims and conclusion do not match very well. References Brantmeier, C. (2006) Advanced L2 learners and reading placement: Self-assessment, CBT, and subsequent performance. System 34 (1), 15-35. Charmaz, K. (2006) Constructing grounded theory: A practical guide through qualitative analysis. Thousand Oaks, CA: Sage. Ekkens, K. and Winke, P. (2009) Evaluating Workplace English Language Programs. Language Assessment Quarterly 6, 265-287. Grognet, A.G. (1996) Planning, implementing and evaluation of workplace ESL programs. Washington DC: National Clearinghouse for ESL Literacy Education/Center for Applied Linguistics. Grognet, A. G. (1997) Performance-based curricula and outcomes: The mainstream English language training project (MELT) Updates for the 1990s and beyond. Denver, CO: Spring Institute. Huerta-Macias, A. (1995) Alternative Assessment: Responses to commonly asked questions. TESOL Journal 51 (1), 8-11. Pierce, B.N., Swain, M., and Hart, D. (1993) Self-assessment, French immersion, and locus of control. Applied Linguistics 14 (1), 25-42. Shohamy, E. (2001) Democratic assessment as an alternative. Language Testing 18 (4), 373-391. Sticht, T. (1999) Testing and Accountability in Adult Literacy Education: Focus on Workplace Literacy Resources for Program Design, Assessment, Testing & Evaluation. El Cajon, CA: Applied Behavioral and Cognitive Sciences, Inc. Read More