Reading, Test Design, and Evaluation - Assignment Example

Heading: Reading, Test Design & Evaluation Your name: Course name: Professors’ name: Date Final draft of the prototype (has been edited) Complete the dialogue between two friends: John and Mary. What does John reply his friend Mary? In questions 1-5, tick the right answer on the answer sheet provided. Samples Sample 1: Mary: Hey, John. It is nice to see you again. How did your vacation go? John: 0……….B 0 A B C D E F G H Mary: How is your family? John: 0…………….D 0 A B C D E F G H In the above question, the correct answer is D Mary: Where did you visit this year? Your uncle’s place again? John: I……… Mary: How was the weather? John: II……………. Mary: Fantastic! Did you take photographs? John: III………… Note: I. B; II.E; III.G; IV. H; V.C Mary: Please do! You stayed in a hotel, right? John: IV……………… Mary: That sounds like you had a great time. So, will visit there again? John: V………. Statement of rationale for the design The choice for design of the above prototype test is determined by a number of factors discussed below. Primarily, a Criterion-Referenced Test (CRT) was adopted because an individual test had been designed for each student. Apart from determining whether or not each student achieved the specific skills or concepts, CRTs measured the specific ski;;s that make up a curriculum. Similarly, the test did not consider the performance of other examinees and instead compared each individual with a preset standard of acceptable achievement. The individual test for each student will aid in effectively measuring the intended language skill as the students will be required to write down their answers on their answer sheets (Harmer, 2007). Further, as Harmer, (2007) says, the prototype test assumes a multiple choice testing design. Here, the students are provided with complete and incomplete statements in form of a conversation. They are then required to complete the conversation by choosing the most appropriate answer or completion firm from the alternatives provided consisting of the destructors and key. The test consists of five-question dialogue (one answered question in sample 1 and another question in sample 2) between two friends. The students have to each answer the questions on the provided answer sheet, choosing the correct answers from the multiple choices. In terms of scoring, design adopts the objective test method. Under objective test, the scoring procedure is fully specified to enable agreement among various scorers. It implies that the tester makes the answer key, and the students should select the best or appropriate answers should be similar to the one in the answer key (Hughes, 2003). It is also a language skill test as it enables the tester or teacher to gain information regarding the ability levels of students to use language whenever in a particular situation. This information indicates the real ability level at an actual time. There are three forms of language skill test including writing skill, reading skill, and speaking skill tests. The above prototype test involves a reading and speaking skill since it is about an incomplete conversation. The students are required to read the conversation and complete it with the best answer in the available answer key (Harmer, 2007). The design has also met vital aspects of a suitable prototype for a specific class or students. Firstly, the test considered validity, which entails test measuring what was intended to measure. The test ought to aid inference to certain domain of the intended language use. This implies that for the tester to determine the validity of the test outcomes, it is imperative to state the expectations of a tester on the students’ capacity to use language in real life situations. The tester should then find out whether the test given produces suitable evidence of their potential to do so. Therefore, in this test, validity is achieved from the fact that the test intends to measure students with skills beyond classroom, but in practical life situations (Brown, Hudson, Norris, & Bonk, 2002). Secondly, it is designed to meet reliability requirement. Reliability entails consistency, which requires the test to have reliable scores in order to yield similar results if repeatedly used. This implies that the test would often rank-order a number of candidates in almost the similar way. This does not imply that the same candidates will fail or pass, as the pass mark will be put separately. Dependability has been used in test because of the need for accuracy and consistency. The design of the test also meets ethics and fairness. Normally, test givers ought to ensure that pertinent principles are broadly spread and comprehended in their organizations (Hughes, 2003). In terms of fairness, all tests should fair in terms of the three dimensions: unbiased, equitable handling of the testing process, and fair results. Test givers should try to reduce bias during test designing, for instance avoiding local issues that are unfamiliar to a section of the test takers. In relation to this design, the test considered both ethics and fairness as the questions are applicable to all the target candidates (Brown, 2005). Description of the pilot trial of the test The pilot test was necessary in that it would assess the draft provision and make corrections based on the stakeholders’ realistic experience and recommendations. Upon drafting these specifications, sample materials were produced. Piloting occurred through test takers similar to the intended candidates. This was also important in determining the effectiveness, feasibility, and adequacy of the test techniques. It was also vital in determining whether test technique was realistic and practical. It would also help in the identification of logistic issues that could happen while using the proposed techniques. In addition, pilot testing was essential in the determination of the necessary resources in the main test. It also helps in the development of questions and test plan. It is also indispensable in convincing the stakeholders on the feasibility and effectiveness of the test. It is essential in convincing the stakeholders that the test is knowledgeable and competent; hence the need to support it (Taylor, 2005). According to Hughes (2003), the pilot test was conducted using the following procedures: 1. Finding five students from the same group of candidates who were to be actually measured (target students). 2. Sample students were arranged to do the test under similar conditions as they would in actual test. Some of the factors considered here include location, time, and the test method. In this case, the method was writing, and thus, students were required to complete the conversation with suitable answer in the alternatives given. 3. After all respondents had completed the test, time was taken with each one of them to discuss their experiences. Some of the questions asked concerned the time taken to complete the test; the purpose of the test; clarity of directions; clarity of instructions; and challenges encountered during the test. 4. The completed tests were collected, and responses were read to determine whether students answered correctly. 5. The data was analyzed and the pilot test results were presented as it would be in the real test. 6. The pilot test results were shared with the stakeholders who would the use the data. The test was them modified using the information gathered from the students (Alderson, 2000). Results The test was trailed on a group of five students from the target group. In the reporting of the performance test outcomes, the pilot test conducted was conducted in order to give readers more confidence in the reported results. As per the test results, three out of five students scored high marks because they seemed to follow instructions and understood the questions clearly. Some found problems completing the conversation because they did could not follow the conversation. The results indicated that the pilot test was not very knowledgeable, effective, feasible, and competent (Hughes, 2003). Recommendations As Hughes (2003) maintains, to achieve the best results, the pilot test should be modified to enable all students understand the questions. More needs to be done to equip learners with reading and writing skills, especially through several exercises on conversations. This will also enable them to relate whatever they learn in class to the realities of life. Generally, there is need for slight modification of the test in order to measure the intended abilities of the target candidates and achieve the objectives. ASSESSMENT CRITERIA 1. Quality of the designed test and rationale (15%) a. Test type, purpose, duration, conditions and test taker characteristics are clearly defined (4) √ b. Clear task instructions and formatting (2) c. Scoring method/weighting of tasks are defined (2) √ d. Results of the pilot test are present (1) √ e. Recommendations for improvement have been made (2) √ f. Formal requirements of the assignment are met (as per Unit Outline section 6h) (2) √ g. References are appropriate and presented in the required manner (APA style) (2) √ 2. Meets task requirements (10%) a. Test includes two sample items (1) b. Test focuses on both language points and functional skills (2) c. Test is performance-based and practical (2) d. Test has been revised according to the recommendations (2) √ e. Original and revised test have been appended to the assignment (1) f. Assignment appendix includes all testing instructions, marking schemes and descriptors to enable the test’s administration by a third party (1) √ g. Signed statements of participatory consent have been provided by all pilot test participants (1) √ 3. Clear, substantial, critical, systematic presentation and discussion (15%) a. All stages of test development are present and the procedure is described in a logical, critical and systematic manner (4) √ b. Decisions made during each stage of the test development procedure are justified by references to the literature (2) c. Demonstrated understanding of the relationship between teaching methodology/language acquisition and testing (2) √ d. Evaluation of test results demonstrates an appropriate understanding of item analysis techniques (3) √ e. Recommendations are strongly associated with or justified by pilot test results and evaluation (3) √ f. Clear discussion outlining how the chosen pilot test participants conform to the test sample requirements (1) 4. Demonstrated understanding of language testing principles and practices (15%) a. Test rationale describes how the test adheres to the six qualities of test usefulness (e.g. reliability, validity, authenticity, interactiveness, impact, practicality) (6) √ b. All test development choices and task specifications (format, language focus, scoring method) are described and justified (9) √ 5. Appropriate use of testing terminology (5%) References Alderson, C. (2000). Assessing Reading. Cambridge: Cambridge University Press. Pp. 1-30. Brown, J.D., Hudson, T., Norris, J. & Bonk L. (2002). An investigation of Second Language Task-based Performance Assessments. Second Language Teaching and Curriculum Center, University of Hawaii.Pp. 20-50. Brown, J.D. (2005). Testing in Language Programs: A comprehensive Guide to English Language Assessment. NY: The McGraw-Hill Companies Inc. Pp. 10-40. Harmer, J. (2007). The Practice of English Language Teaching. (4th ed.). Harlow: Longman. Pp. 10-30. Hughes, A. (2003). Testing for Language Teachers. 2nd Ed. Cambridge: Cambridge University Press. Pp. 5-25. Taylor, L. (2005). Key concepts in ELT: Washback and impact. ELT Journal, 59(2), 154- 155. Appendices Appendix 1 First draft of the prototype (to be submitted for editing) Instructions: Complete the dialogue between two friends: John and Mary. What does John reply his friend Mary? In questions 1-5, tick the right answer on the answer sheet provided. Samples Sample 1: Mary: Hey, John. It is nice to see you again. How did your vacation go? John: 0……….B 0 A B C D E F G H In the above question, the answer is B Sample 2: Mary: How is your family? John: 0………D In the above question, the answer is D Mary: Where did visit this year? John: I……… Mary: How was the weather? John: II……………. Mary: Nice! Did you take photographs? Note: I. B; II.E; III.G; IV. H; V.C John: III………… Mary: You stayed in a hotel, right? John: IV……………… Mary: That sounds like you had a great time. So, will visit there again? John: V………. Mary: That will be more wonderful. Appendix 2 Final draft of the prototype (has been edited) Complete the dialogue between two friends: John and Mary. What does John reply his friend Mary? In questions 1-5, tick the right answer on the answer sheet provided. Samples Sample 1: Mary: Hey, John. It is nice to see you again. How did your vacation go? John: 0……….B 0 A B C D E F G H Mary: How is your family? John: 0…………….D 0 A B C D E F G H In the above question, the correct answer is D Mary: Where did you visit this year? Your uncle’s place again? John: I……… Mary: How was the weather? John: II……………. Mary: Fantastic! Did you take photographs? Note: I. B; II.E; III.G; IV. H; V.C John: III………… Mary: Please do! You stayed in a hotel, right? John: IV……………… Mary: That sounds like you had a great time. So, will visit there again? John: V………. Read More