Evaluation methodology - Term Paper Example

?Question #2: Research Methods and Critique Question Evaluation methodology is a form of qualitative research. Charles and Mertler (2002) describe evaluation research as being used to determine the worth of a program or curriculum that has been put into place to serve a particular purpose. Select a program recently implemented in your work environment and design a study to assess its worth using one of the qualitative research methods. Your paper should describe your selected method and explain why you believe it to be the best approach given the nature of your study. Program evaluation is defined as “the systematic collection, analysis, and reporting of descriptive and judgmental information about the merit and worth of a program’s goals, design, process, and outcomes to address improvement, accountability, and dissemination questions and increase understanding of the involved phenomena” (Stufflebeam & Shinkfield, 2007, p. 710). Various program evaluations have been designed and deemed fundamental in helping organizations determine the needs, outcomes, and budgetary issues, of different programs for them to continually improve (Alford, 2010; Mishra, Vijayshri, & Garg, 2009). The researcher will use the CIPP evaluation model (Stufflebeam & Shinkfield, 2007) to guide this program evaluation. The acronym CIPP stands for context evaluation, input evaluation, process evaluation, and product evaluation. Developed by D. L. Stufflebeam, the CIPP evaluation design employs a logic model to describe the program’s elements. Since the early 1970s, the logic model has been used in the educational system, and represents a plausible and sensible model about how a program will work under certain environmental conditions to solve identified problems. The logic model is the core of a certain program’s planning, evaluation, program management, and communications. It holds together everything a program was originally designed for, from its goals, vision, resources, activities, and intended outcomes in a way that is logical to its purpose (Hall, Daly, & Madigan, 2010). The study to be designed will follow its evaluation approaches to test and verify the reality of the advisory program of a target school and how it works. It will also help the researcher focus on appropriate process and outcome measures (University of Wisconsin-Extension, 2003). The CIPP model of program evaluation is the process of delineating, obtaining, providing, and applying descriptive and judgmental information about the merit and worth of some object goals, design, implementation, and outcomes to guide improvement decisions, provide accountability reports, inform institutionalization/dissemination decisions, and improve understanding of the involved phenomena (Kellaghan & Stufflebeam, 2003). It is a simple procedural model that measures program evaluation. It would reflect the efficiency of performance of a certain program. (Wholey, Hart, & Newcomer, 1987; Hong, Teng, & Baum, 2009; Chien, Lee, & Cheng, 2007). The CIPP goes through four stages in the evaluation of programs. The first stage of context evaluation assesses the needs and problems of the program and how set goals are being met. It identifies what needs to be done for improvement. The second stage of input evaluation assesses the organization’s budget and staffing and sees how effective resources are being used. The third stage of process evaluation assesses how plans are being implemented and if the activities being performed produce positive outcomes for the stakeholders. The fourth and last stage of product evaluation assesses the outcomes of the program and checks if initial goals have been fulfilled (Stufflebeam & Shinkfield, 2007). Due to its renowned reputation as an effective model of evaluation, this study has chosen the CIPP to evaluate an educational advisory program (Kellaghan & Stufflebeam, 2003). The results of the evaluation using this model will greatly help decision makers to assess the needs of the program in implementing its various activities. Using the CIPP model will be helpful in deriving the necessary information that can prompt appropriate judgment and/or coming up with decisions for alternative actions. It will also help evaluators of the program to make the adjustments or improvements called for. That is the main purpose of the researcher’s study. Stufflebeam, McKee & McKee (2003) claim that the CIPP model is efficient in facilitating the decision-making, monitoring and judging processes of the evaluation. Moreover, the CIPP model gives key stakeholders a clearer picture of the whole program. School officials, teachers, students and community supporters get to see how the advisory program is supposed to work and understand the underlying theoretical framework, the flow of activities and what is to be expected of it (Poliandri, Cardone, Muzzioli & Romiti, 2010). It follows that key stakeholders will glean the effectiveness of the program and see the gaps that need to be filled out (Chung & Ho, 2009). The program to be evaluated is School X’s Middle School Advisory Program. Wilson (1998) cites that advisory programs in middle schools are based on the concept that each child should be known well by at least one caring adult in the school. Advisory programs have become an essential component of middle school life for students. They are borne out of the need to address issues of adolescents usually in transition from junior high school to middle school, mostly belonging to fifth through eighth grades. School climate has been known to improve with the coming of advisory programs as it addresses the adolescents’ social and emotional needs (Killin & Williams, 1995). Burkhart (1999) found that middle school advisory programs have been identified as a possible cause in increases in students' academic success, decreased vandalism, improved attendance, less student alienation, learning that is more student centered, and a more positive school climate overall. Indeed, students learn best in supportive environments where no child can fall through the cracks. As students leave their elementary grades and enter the unfamiliar world of middle school, they may go through difficult transitional experiences. From a small and intimate environment, moving on to a wider and more challenging environment may be intimidating for them as they are placed in unfamiliar settings with unfamiliar people and events (Ginther, Haveman, & Wolf, 2000). Compounding this difficult educational and social transition, adolescents experience rapid, physical, emotional, and cognitive changes (Jackson & Davis, 2000). It is at this stage when it becomes common for students to experience life challenges such as divorce of their parents, economic hardships, mental health issues, peer pressure and involvement in drug and alcohol issues (Wittmer, 1993). Eccles, Roeser & Sameroff (2000) claim that in the critical time of early adolescence (age 10-15), children are prone to social, emotional or educational distress and if not provided with the support they need for their personal and educational development, they may be at risk of not reaching their full potentials. Adolescents new to middle school life may not know teachers, staff or people in the administration and vice versa, so these youngsters may feel at a loss in their new environment. An apt solution to this is a schoolwide advisory program that provides the students with at least one caring adult to connect to while they ride on the waves of their new and perhaps, intimidating school experiences. Anfara (2006) contend that advisory programs give adolescents opportunities to expand their social circles and learn more about themselves as supervised by a caring adult facilitator. Such experiences and programs enhance the adolescents' ability to self-reflect, and improve their ability to make the critical decisions necessary for personal, academic, and social development (Eccles & Harold, 1993). The middle school advisory groups of School X meet everyday with selected teachers and school administrators as their advisors. Each advisory group has approximately 10 students and their activities vary depending on their needs (Burkhart & Fusco, 1999). If a caring and motivated advisor runs an advisory program, it can generate a nurturing environment needed by adolescents to be comfortable enough to air out their personal issues such as family problems or concerns regarding their peer group. Research has shown adolescents learn best where they feel connected and safe in a supportive environment and where they know an adult is available for them (Schulkind, 2009). Advisory programs are designed to meet specific student needs. Topics discussed in sessions may include issues on drug, alcohol and tobacco use; sex and dating; friends and relationships; conflict resolution and study skills; community service; current events; values and ethical issues; decision making; school spirit; and problems students have at school or at home (Gallassi & Gulledge, 1997). These concepts are learned through various activities such as debates, discussions, peer mediation, group dynamic games, and field trips or special projects (MacIver & MacIver, 2008). Packaged advisory curricula are available or teachers may design their own activities to suit the topic and needs of their advisees. Because important and sensitive issues are dealt with, it may occasionally result in the need for a crisis response that would necessitate the involvement of the school counselor or the needed intervention for specific students. In evaluating School X’s Middle School Advisory Program, focus group interviews shall be conducted with seven to ten students and another one for about the same number of advisory staff. The student group interview will focus on their perception of the school climate and how their advisory programs help them in coping with school challenges. It is assumed that if the students perceive their school experiences to be positive, the advisory program could have been instrumental for such perception. The advisory staff group interview will discuss the advisory programs they facilitate and what improvements they think can be done to make it better. Interviews serve the purpose of enabling participants to be able to discuss their interpretations of a concept, as well as giving participants the opportunity to express their own point of view with regard to certain situations (Cohen, Manion, & Morrison, 2000). The interview method involves questioning or a discussion of issues with one or more people.  It is useful to collect data that may not be accessible through observation or questionnaires (Blaxter, Hughes, & Tight, 2006). Robson (2002) pointed out how the interview is a flexible and adaptable research tool.  In face-to-face interviews, there is the possibility of following up interesting responses.  Frey and Mertens-Oishi (1995) insisted that the interviewer who sensitively guides the questioning could enhance participation and also answer any question the respondent may ask.  Oppenheim (1992) suggested that the response rate is higher in interviews than in questionnaires, due to the fact that participants become more involved. However, there are also disadvantages to using interviews as a research tool.  Robson (2002) mentioned the fact that the whole interview process can be time-consuming.  Beginning with making arrangements, often having to reschedule appointments, the actual interview itself, and then the writing of notes or transcribing the interview, is, when considered in total, a very lengthy and complex process.  A major disadvantage in any interview situation is the possibility of bias (Grinnell & Unrau, 2008). The interviewer may unwittingly divulge his/her opinion or expectations by his/her tone of voice, or in the way questions are asked.  Even when recording the interview, it is important to remain aware of bias having an effect on how answers are understood and transcribed. However, these methods are a quick way to assess participants’ insights, although participants may not always respond truthfully, so honesty will need to be emphasized. Additionally, theoretical orientation may bias questions and the interpretation of answers. The interviews will be recorded and transcribed in verbatim for data analysis. The following questions will lead the discussions in the focus group interviews – one set for the students and another set for the advisory staff. These questions were adapted from the Saucon Valley Middle School (SVMS) Advisory Student Survey for School Climate. The interviewer will use these questions to start off discussions and allow for free sharing of insights and opinions of the participants. Student Focus Interview Questions: How often is there good communication between teachers and students? To what extent is your school a good place to learn? Do you think students are given a chance to succeed in school? Is your school a pleasant place to be? Are you able to talk to an adult in school about your problems? With all the questions asked, do you think your school’s advisory program addresses your needs as a student? Advisory Staff Focus Interview Questions: What are your advisory program’s goals? Do you think your school’s advisory program needs revision regarding its goals? Should the advisor continue with his/her advisees through all the middle school years? Do you think spend enough time in your advisory program? Do advisors need the opportunity to meet individually with advisees? Does there need to be more staff training in specific areas such as assisting students with goal setting? Would you like to share anything else regarding the advisory program? Upon gathering data from these focus group interviews, the responses will be studied to address some major questions guided by the CIPP evaluation of context, input, process, and product. Coleman (2009) suggested the following questions: What needs to be done? How should it be done? Is it being done? Did it succeed? Coleman (2009) further explained how each evaluation domain works: The context part of the evaluation refers to the needs, problems, and opportunities in the program evaluated, in this case, the advisory program. The information gathered from the interviews will determine if the program is meeting the needs of the stakeholders and if opportunities are maximized to help the students in school. Input is determined by the things identified to be done in order to meet the needs of the stakeholders in the program (Osokoya & Adekunle, 2007). For example, what activity is thought to be effective in helping students perceive a more positive school climate? Is it the group dynamic games? The debates? Or just plain discussions? The process component of the CIPP checks if the program is doing what it is supposed to be doing. The data gathered will provide feedback regarding its effectiveness and if it is maintaining the quality expected by all stakeholders. Product “measures, interprets, and judges the achievements of the organization” (Coleman, 2009, p. 40). It specifically evaluates how well the program has met its objectives, so decisions can be made if it will continue, repeat, or extend to other settings. CIPP evaluation seems to be the best choice for School X’s Middle School Advisory Program because it is a guideline to check out all the necessary details for the program to be effective. Evaluation of advisory programs is necessary if quality and meeting of goals is a priority. Helping out adolescents who need to have a positive relationship with at least one adult (advisory program facilitator) and one peer (co-members in the advisory program) greatly affects in the development of his/her confidence, security, and sense of belonging. Students who feel a positive sense of belonging to their school can develop resiliency that may protect them against negative psychological outcomes, such as depression and at-risk behaviors (Blum & Rinehart, 1996). These researchers found students who felt more connected to school showed lower at-risk behaviors and attitudes than did students who did not report feeling connected. In a study of 176 sixth-grade students, Wentzel (1998), found that students’ perceptions of support from their teachers were significantly related to the likelihood that students would develop goals consistent with their classroom objectives. Positive school environments promote strong connections and attachments of students to the school and enhance their social interactions. When students feel accepted and encouraged by their school environment, they are more confident to deal with their problems (Cartland, Ruch-Ross, & Henry, 2003). On the other hand, students who do not develop a connection to their school usually feel socially isolated or even outcast and this results in poor social interactions. This implies that students’ perception of their own connectedness to the school or their sense of belonging depends on the kind of environment fostered by the school (McNeely, Nonnemaker, & Blum, 2002). Strong bonds may be established through respectful encounters between teachers and students. Such alliances allow them to collaborate and share responsibilities in problem solving. This becomes a core element of positive school environments, specifically in advisory programs (Ma, 2003). References Alford, K.R. (2010). Program evaluation: Two Management-oriented samples. Unpublished paper. University of the Cumberlands. Anfara, V.A. (2006). Advisor-Advisee Programs: Important but problematic, Middle School Journal, 36 (1), 54-60. Blaxter, L., Hughes, C., & Tight, M. (2006). How to research (3rd ed.). Berkshire: Open University Press. Blum, R. W., & Rinehart, P. M. (1996). Reducing the risk: Connections that make a difference in the lives of youth. Minneapolis, MN: Division of General Pediatrics and Adolescent Health, University of Minnesota. Burkhart, R. M. (1999). Advisory: Advocacy for every student. Middle School Journal, 30(3), 51-54. Burkhart, R., & Fusco, E. (1999). Advisory: An advocacy program for students. Shoreham, NY: Shoreham-Wading River Middle School. Cartland, J., Ruch-Ross, H. S., & Henry, D. B. (2003). Feeling at home in one’s school: A first look at a new measure. Adolescence, 38, 305-319. Chien, M., Lee, C. & Cheng, Y. (2007). The construction of Taiwan’s educational indicator systems: Experiences and implications. Education Research for Policy Practice, 6, 249-259. Chung, S., & Ho, P. (2009). Quality teaching and learning: A quality assurance framework for initial teacher preparation programmes. International Journal of Management in Education, 3, 302-314. Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education (5th ed.). London: Routledge Falmer. Coleman, T. (2009). Evaluation of an After-School Program in an Urban Middle School. An Applied Dissertation Submitted to the Fischler School of Education and Human Services in Partial Fulfillment of the Requirements for the Degree of Doctor of Education Nova Southeastern University. Eccles, J. & Harold, R.D. (1993). Parent-school involvement during the early adolescent years. Teachers College Record, 94(3), 568-587. Eccles, J. S., Roeser, R. A., & Sameroff, A. J. (2000). Non-subject matter of school counseling. The Elementary School Journal, 100, 443-471. Frey, J. H., & Mertens-Oishi, S. (1995). How to conduct interviews by telephone and in person. London: Sage. Ginther, D., Haverman, R., & Wolf, B. (2000). Neighborhood attributes as determinates of children’s outcomes: How robust are the relationships? The Journal of Human Resources, 35, 603-642. Grinnell, R. M., Jr., & Unrau, Y.A. (2008). Social work research and evaluation: Quantitative and qualitative approaches (8th ed.). New York, New York: Oxford University Press. Hall, M.A., Daly, B.J. & Madigan, E.A. (2010). Use of Anecdotal Notes by Clinical Nursing Faculty: A Descriptive Study. Journal of Nursing Education, 49(3). Hong, J., Teng, C. & Baum, T. (2009). Evaluating the quality of undergraduate hospitality, tourism and leisure programmes. Journal of Hospitality, Leisure, Sport and Tourism Education, 8(1). Jackson, A. W., & Davis, G. A. (2000). Educating adolescents in the 21st century. Williston, VT: Teacher’s College Press. Kellaghan, T., & Stufflebeam, D. L. (2003). International handbook of educational evaluation: Part 1. Dordrecht, The Netherlands: Kluwer Academic Publishing. MacIver, M.A., MacIver, D.J. (2008). Middle School Reform. 21st Century Education: A Reference Handbook (Vol. 2, pp. 309-317). Ma, X. (2003). Sense of belonging to school: Can schools make a difference? Journal of Educational Research, 96, 340-349. McNeely, C. A., Nonnemaker, J. M., & Blum, R. W. (2002). Promoting school connectedness. Journal of School Health, 72, 138-146. Mishra, A., Vijayshri & Garg, S. (2009). Evaluation of the undergraduate physics programme at Indira Gandhi national open university: A case study. International Review of Research in Open and Distance Learning, 10(6). Oppenheim, A. N. (1992). Questionnaire design: Interviewing and attitude measurement. London: Pinter. Osokoya, M.M. & Adekunle, A. (2007). Evaluating the trainability of enrollees of the Leventis Foundation (Nigeria) Agricultural Schools’ programs. Australian Journal of Adult Learning, 47 (1). Poliandri, D., Cardone, M., Muzzioli, P., & Romiti, S. (2010). Dynamic database for quality indicators comparison in education. Working paper for INVALSI – National Institute for the Educational Evaluation of Instruction and Training. Robson, C. (2002). Real world research (2nd ed.). Oxford: Blackwell. Schulkind, S.B. (2009). Creating a culture of connectedness through middle school advisory programs. Middle School Journal, 41(1): 20-27. Stufflebeam, D. L., Mckee, H, & McKee, B. (2003). The CIPP model for evaluation. Retrieved from http://www.wmich.edu/evalcts/pubs/ Stufflebeam, D. L., & Shinkfield, A. J. (2007). Identifying and assessing evaluation opportunities in evaluation theory, models, and applications (pp. 453-464). San Francisco: Jossey-Bass. University of Wisconsin-Extension (2003). Enhancing program performance with logic model. Retrieved from lmcourse@ces.uwex.edu Wentzel, K. R. (1998). Social relationships and motivation in the middle school: The role of the parents, teachers, and peers. Journal of Education Psychology, 90, 202-209 Wholey, J., Hart, H., & Newcomer, k. (2004). Handbook of practical program evaluation (2nd ed.). San Francisco: Jossey-Bass. Wittmer, J. (1993). Developmental school guidance and counseling: Its history and reconceptualization. In J. Wittmer (Ed.), Managing your school counseling program: K?12 developmental strategies (pp. 2-11). Minneapolis, MN:Educational Media Corporation. Read More