Designing Quality Assessments - Assignment Example

Running head: ASSIGNMENT NUMBER 2 Assignment Number 2 Designing Quality Assessments Assignment Number 2 Introduction In this paper we have selected four learning objectives and developed 4 different assessments (one per objective). We discuss the nature of each objective as well as its potential advantages or disadvantages. We also give justifications for the choice of each different type of assessment. PART ONE (parts a. and b.): The choice of objectives and their assessment methods is the following: The Reference for the Objective What the Objective is How to Assess the Objective A3 Speaks confidently to others Performance Test B4 Works effectively in a team Portfolio C4 Is creative Essay D4 Solves unfamiliar problems within the domain Supply items We will now explain these ways of assessing the objectives in detail. How the performance assessment will be done: (Objective: speaks confidently to others - A3) To assess this objective, we need an assessment method that works at a practical level to allow the person to demonstrate progress after instruction and classes relating to this subject. We can assess progress to this objective relatively rapidly, but it must be done in a "real-life environment". The person being assessed also needs to know how the scoring is done, with respect to the criteria chosen and the important of each criterion for the overall score. This is because "learning increaseswhen learners have a sense of what they are setting out to learn, a statement of explicit standards they must meet and a way of seeing what they have learned" (Loacker, Cromwell & O'Brien, 1986) The assessment will be done as part of the ongoing daily work of the person concerned. We will select a meeting that the person is to lead, for a nominal period of 45 - 60 minutes. We will evaluate the person's performance in speaking confidently to others on the following basis: What external signs of confidence does the person show (rated on a level from one to five, where one is none and five is excellent - if however the person overdoes it, then assessors should subtract points accordingly) makes eye-contact with the person/people listening speaks clearly and audibly asks clear and pertinent questions gives clear and pertinent answers directs the conversation according to the agenda of the meeting These points carry equal weight as each one is important to convey confidence while talking to other people. The assessment will be shared with the person concerned afterwards. Knowledge of the assessors' view point will help the person understand the relationship between the way he/she thinks that he/she replies to these criteria and the way that his/her performance is assessed by others. How the portfolio assessment will be done: (Objective: works effectively in a team - B4) Assessing effective working in a team requires a longer term approach for the assessment. An attempt to measure this by an assessment of very short duration would run the risk of being unrepresentative of the person's performance. For these reasons we chose the portfolio assessment which has the characteristic of being an ongoing, rather than an episodic assessment (AAHE, 1996). In addition to this, completing the portfolio assessment becomes part of the learning process for the person concerned where the goal is as much to affect the learner's growth, instead of just passively measuring it (Courts & McInerney, 1993). The portfolio assessment will be done over a period of time (2 months) and will correspond to stated criteria against which the person being assessed and the assessors have agreed. The person puts together the portfolio with information that demonstrates how he or she works effectively within a team. The portfolio is shown regularly to the assessors as well. At the end of the 2 months, the assessors will review the portfolio with the person to assess the performance of the person concerned on the following basis: evidence of a particular role or function taken on by the person within the context of the team evidence of identifiable results within the team contributed by the person evidence of team feedback which is appreciative of the person's contribution How the essay assessment will be done: Objective: is creative (C4) The essay assessment was chosen here as a way to probe capability. Compared to certain other methods of assessment, the essay has the advantage of being easy and cost-effective to put in place as well as allowing us to determine the depth of capability (University of Colorado, 2008). The person to be assessed will be asked to write an essay. This will be done under "exam conditions", meaning no textbooks or other external information sources allowed and a set time to start and to finish. No particular preparation will be required as the assessment is of the person's creativity, not of their prior knowledge. A sample essay question of this type is as follows: You have been asked to organize the activities for adolescents (11 to 15 years old) on a summer camp lasting for three weeks. The camp will be situated in an open area close to a forest and to a lake. The overall aim of the camp is to increase appreciation of natural resources amongst the adolescents and to maintain a reasonable level of activity and participation for these adolescents who are easily distracted or soon become bored with overly repetitive activities. The camp has sufficient equipment and staff monitors for a variety of activities to be done in safety, although any activities that you suggest should in any case observe the basic rules of personal safety and minimum financial outlay. You may make any reasonable assumptions about the resources available at the camp. With this information in mind: describe a program of 10 activities for the adolescents attending this summer camp that will satisfy both the adolescents and the goals of the camp (10 marks) state why each of the activities in this program will be satisfactory (10 marks). For each activity, give an alternative that could be used in case of resources becoming unavailable, the adolescents expressing discontentment or other reasons preventing the original activity. The alternative should remain consistent in spirit with the theme of the original activity (10 marks) Time to write this essay is 90 minutes. Candidates leaving the room before the end of this time will not be readmitted. A maximum of 30 marks is possible. Instructions for assessors are as follows. The assessors are to inform the person or persons being assessed about the exam conditions of the assessment, the date, time and place of the assessment and the requirement to be punctual or forfeit the possibility to take the assessment. Scoring of assessment will be done according to the criteria that follow. Assessors should award points as follows as indicated in the essay instructions given to the person(s) to be assessed. As the aim is to assess creativity, allow for a suitable range of answers. For example, watching beavers build a dam, constructing aerial walkways in the forest (under correct staff supervision) and building and flying kites to harness wind power are all acceptable, even though each must be justified. On the other hand, power boat racing on the lake is not (fails to satisfy criteria about appreciating nature or observing minimal outlay). Don't penalize for wrong spelling or grammar unless this makes it difficult to read or understand the essay. How the supply items assessment will be done: Objective: Solves unfamiliar problems within the domain (D4) We chose supply items assessment here to make sure that we are assessing for the capability to handle unknown problems without "suggesting" the answer for example, by multiple choice questions. The goal is to test higher levels of understanding of concepts that can then be applied in new situations (Frey, 2008) The supply items assessment as applied to car mechanics and car repair would look like this: 1. A car is brought into a garage by its owner. The owner asks the mechanic to modify the rear suspension of the car. Before making any commitment, the mechanic asks if the owner has validated the modification with ____________________. 2. A customer asks if a hybrid electric car (petrol engine with alternative electric motorization) is really any less polluting than an classical petrol engine car. The mechanic's first question to the customer is on the source of ________________________________. 3. A company with a fleet of cars regularly serviced by a garage asks if there is a cost advantage to retro-fitting fuel economy devices to all of the cars. To answer such a question, you will need to know ___________________ compared to _________________________. To make the assessment, assessors will compare the item (word or phrase) supplied by the person being assessed with a selection of possibilities given in the assessor's guidelines and assign a mark of zero (no relevance), one (partial relevance) or two (corresponds well to at least one of terms in the assessors' list) for each question. (All questions carry equal marks. Assessors will grade the using a scoring rubric that lists possible alternatives for each question. Assessors will then use best judgment to assign a note of 0, 1 or 2 to each question, where 0 is answer missing or completely wrong, 1 is partially correct and 2 is for a correct answer. PART TWO (part c.): The Reference for the Objective What the Objective is How to Assess the Objective A3 Speaks confidently to others Performance Test B4 Works effectively in a team Portfolio C4 Is creative Essay D4 Solves unfamiliar problems within the domain Supply items What will and what won't be measured by these assessments The performance test measures what it is important to know how to do, when speaking confidently with others. It is a "snapshot" of progress. More precisely, it measures the objective over a short period of time in a particular situation. It does not measure knowledge of the underlying principles of how to speak confidently or why these principles are effective. A person doing well in this performance test might be doing well because of working hard to study information and build his or her own capability, or we might just see "natural talent" in action. The portfolio assessment will measure what went well in working effectively in a team. On the other hand, as the person making the portfolio is unlikely to include any material that reflects negatively, we won't know if there are any particular problems or gaps in capability. Therefore the portfolio assessment measures a fraction of the situation, even if it corresponds well to the objective of "works effectively in a team". The essay assessment will measure creativity but more specifically, creativity under (moderate) time pressure. This may alter a person's creative processes, so that we get a different reading under these circumstances than if the creative process was "open-ended" (no time constraints). The person's performance may be better or worse in the other case, but the assessment will not tell us this. The essay assessment will also allow a measure of the student's capability to reason and to determine how and why the student chose to give a particular answer to the question set. In comparison with other assessment methods, the performance test would come closest to being alternative option. Multiple choice would not because there is no indication of the creativity of the person, and neither would portfolio because of the difficulty of identifying what is truly creative compared to what may be copied from other sources. The supply items assessment will measure the person's capability to deal with unknown areas of a subject, but only in the sense that these areas are unknown because they are "corner cases" or not often encountered. If the person has read up a lot on their subject, we will no longer be testing the unknown, instead we will be testing their knowledge retention. To get round this problem, such a supply items assessment would need to be updated regularly with new items that really are unknown. PART THREE (part d.): The Reference for the Objective What the Objective is How to Assess the Objective A3 Speaks confidently to others Performance Test B4 Works effectively in a team Portfolio C4 Is creative Essay D4 Solves unfamiliar problems within the domain Supply items How effective will each instrument be in developing knowledge and capacity of the information processing that is implied in the objective If an assessment instrument is to be effective in this sort of development, the people being assessed will at least have to know about the assessment principles beforehand. In the case of the portfolio for example, the person will have knowledge of the objective and pointers from the assessors as to what kind of material will be appropriate and why for the construction of the portfolio. For the performance test, similar preparation should be done ahead of time, allowing the person to work on the points that count in the assessment. On the other hand, for the essay assessment to assess creativity, we would probably not go further than making sample essay questions available, as we are assessing as much an innate capability as any learned competence. Similarly, for the supply items for unfamiliar problems, any detailed preparation may be counter-productive, in the sense that we end up testing a person's memory rather than his or her inner capability to deal with the unknown. So for the two objectives C4 and D4, the instrument will not be (can not be) as effective in developing the capability; its effectiveness will stay in the area of testing the capability. The portfolio and the performance test assessments have a much better chance of simultaneously developing a person's capacities as the same time as assessing them. This is because the person knows from the beginning what the assessment is based on, and which particular points are to be measured. In addition, the person can assess themselves on these points. For the portfolio, this is done on a continuing basis. For the performance test, the person can get some reading of his or her capabilities in this area from prior practical sessions, before the official performance test session. However, this contribution to a person's development of knowledge and capacity has certain costs, or drawbacks in other areas. The performance and portfolio instruments have a risk of lack of objectivity in measuring the true performance of a person. Subjective factors can intervene ranging from the interrelationship between the assessors and the person (maybe even just because of physical appearance) to the subjective selectiveness that a person will use in putting together their portfolio material. Portfolio and performance test assessments also rate lower than essay and supply item assessments in terms of efficiency, reliability and feasibility. The efficiency is lower because of the time and effort required by both the person being assessed and the assessors (more time-consuming that essay and supply item). The reliability is lower, in the case of portfolios because of the biasing towards only positive elements and for the performance test because you cannot control the actions and reactions of the people to whom the person to be assessed is speaking. Nonetheless, we could argue that the reliability of the essay and supply type assessments is only higher because they are restricted to a narrower domain of assessment. The difference in feasibility is a practical matter: it is easier to have someone sit down and answer an essay question or supply items for an hour or two, than to organize a suitable "live" performance test, let alone a two month portfolio assessment. Criteria such as discrimination and fairness can be equalized among the different assessment methods as long as for the essay and supply items assessments a balanced view is taken in setting the questions, and for the performance and portfolio assessments, assessors strive to remain objective and impartial in their assessments. References AAHE, 1996. Nine Principles of Good Practice for Assessing Student Learning, Retrieved April 19th 2008, http://www.fctel.uncc.edu/pedagogy/assessment/9Principles.html Courts & McInerney, 1993. Assessment in Higher Education: Politics, Pedagogy, and Portfolios., Praeger Publishers, Westport CT. Frey, 2008. An Introduction to Quality Test Construction. Retrieved April 19th 2008, http://www.specialconnections.ku.edu/cgi-bin/cgiwrap/specconn/main.phpcat=assessment§ion=qualitytest/main Loacker, G., Cromwell, L. & O'Brien, K. 1986, Assessment in Higher Education: To Serve the Learner. In "Assessment in Higher Education", U.S. Department of Education, Washington D.C. University of Colorado, 2008. Assessment of Our Students IV - Essay Tests. Retrieved April 19th 2008, http://carbon.cudenver.edu/public/OTE/nn/vol5/5_4.htm Read More