Program Evaluation Method - Essay Example

PROGRAM EVALUATION METHOD PROGRAM EVALUATION METHOD M2AI Logic model While there are many forms, logic models specify relationships among program goals, objectives, activities, outputs, and outcomes. Logic models are often developed using graphics or schematics and allow the program manager or evaluator to clearly indicate the theoretical connections among program components: that is, how program activities will lead to the accomplishment of objectives, and how accomplishing objectives will lead to the fulfillment of goals. In addition, logic models used for evaluation include the measures that will be used to determine if activities were carried out as planned (output measures) and if the programs objectives have been met (outcome measures). (Julian, 1997) In trying to obtain information about the project, a logic model maybe useful because: 1. It will develop a common language among stakeholders 2. It will offer a high participatory learning opportunities among project members 3. It will emphasize and document explicit outcomes 4. It will clarify about what works in the project and why 5. It will identify important variables that will be used to measure and enable more efficient use of evaluation resources 6. It will offer a reliable reporting framework 7. It will improve planning, management and design of the project On the other hand, a logic model may present difficulties in obtaining information because: 1. Objectives, activities, outcome measures of the logic model maybe unclear or missing. This is because logic models contain lists of each of the elements of a logic model without specifying which item on one list is related to which item on another list. This can easily lead to confusion regarding the relationship among elements or result in accidental omission of an item on a list of elements. 2. Too much (or too little) information maybe provided on the logic model. The logic model should include only the primary elements related to program/project design and operation. As a general rule, it should provide the "big picture" of the program/project and avoid providing very specific details related to how, for example, interventions will occur, or a list of all the agencies that will serve to improve collaboration efforts. 3. Objectives maybe confused with activities. The project evaluator must make surethat items listed as objectives are in fact objectives rather than activities. Anything related to program implementation or a task that is being carried out in order to accomplish something is an activity rather than an objective. For example, hire 10 staff members is an activity that is being carried out in order to accomplish an objective such as improve response time for incoming phone calls. The impact on the project Even though it leaves out information, a good model represents those aspects of an initiative that, in the view of the stakeholders, are most important for understanding how the effort works. The developers will go through several drafts before producing at a version that the stakeholders agree accurately reflects their story. If the information will become overly complex, it will be possible to create a family of related models, or nested models, each capturing a different level of detail. One model could sketch out the broad pathways of change, whereas others could elaborate on separate components, revealing detailed information about how the program operates on a deeper level. Individually, each model will convey only essential information, and together they provide a more complete overview of how the program or initiative functions. M3A1 Selecting an evaluation approach Qualitative methods Because qualitative methods are open-ended, they are especially valuable at the formative stage of evaluation when programs are pilot testing proposed procedures, activities, and materials. They allow the evaluator unlimited scope to probe the feelings, beliefs, and impressions of the people participating in the evaluation and to do so without prejudicing participants with the evaluator’s own opinions. They also allow the evaluator to judge the intensity of people’s preference for one item or another. For instance, in selecting an evaluation approach for the distribution for a local news paper, below are the approaches the evaluator can use. (Miles, Saldana and Huberman, 2013) 1. Personal interviews- these are personal interviews with broad, open-ended questions are especially useful when the evaluator wants to understand either the strengths or weaknesses of a new or modified program before it is in effect or the cause of a problem should one develop after the program is in effect. Relatively unstructured personal interviews with members of the target population allow interviewees to express their point of view about a program’s good and bad points without being prejudiced by the evaluator’s own beliefs. Open-ended questions allow interviewees to focus on points of importance to them, points that may not have occurred to the evaluator. 2. Focus groups- Focus groups serve much the same function as personal interviews. The main difference is that, with focus groups, the questions are asked of groups. Ideally these groups comprise four to eight people who are likely to regard each other as equals. A feeling of equality allows all members of the group to express their opinions freely. Quantitative approach Quantitative methods are ways of gathering objective data that can be expressed in numbers (e.g., a count of the people with whom a program had contact or the percentage of change in a particular behavior by the target population). Quantitative methods are used during process, impact, and outcome evaluation. 1. Counting system- involves keeping written records of all events pertinent to the program (e.g, each contact with a member of the target population or each item distributed during a product-distribution program). Counting systems are especially useful for process evaluation 2. Surveys- A survey is a systematic, nonexperimental method of collecting information that can be expressed numerically. Surveys may be conducted by interview (in person or on the telephone) or by having respondents complete, in private, survey instruments that are mailed or otherwise given to them. The factors considered during the evaluation are the target population, the demographic factors of the consumers like age, culture, language and income levels. This is an outcome evaluation because it is used to measure how well the program met its overall goal. For instance, how many people are reading newspapers than before the program began. M4A1 `After getting the results of my evaluation finding, I will want to get the information back to the community and the target population. But first, various stakeholders like current and potential funders, community partners, professional colleagues and consumers will be interested in hearing about the program to ensure they will use the evaluation results. The questions will arise out of the goals and objectives in the program description, planning and the stated needs of the stakeholders. These evaluation questions will provide the foundation and direction for the evaluation with the stakeholders. In designing questions, the convergent phase will be used. In this phase, the questions that are most relevant to the evaluation are selected. The questions asked by stakeholders during this phase will give them the opportunity to express their opinions and ideas concerning the program. Everyone in the meeting is allowed to contribute. In that case, it means that the stakeholders contribute to the achievement of effective results and recommendations for the evaluation. After presenting my findings, I will provide a context for my data collection, and then report how the data was collected, including the limitations or issues regarding data quality. This will give the audience the best opportunity to draw conclusions. I think the stakeholders will approve of my evaluation and convince them to fund the program. M5A1 Planning the evaluation Through proper planning of an evaluation, challenges within a program can be avoided. In designing the evaluation, the evaluator will follow the basic steps of developing and evaluation design. 1. Focusing the evaluation 2. Gathering the information 3. Organizing the information 4. Analyzing the information 5. Reporting the information 6. Administering the information By focusing on the data source options will be useful for the stakeholders as they will seek to know about the evaluation. Through data collection by sampling, the evaluator will use the following sampling methods: 1. Representative sampling- this method is used when a sample of a population represents the rest of the population in the study. It is mainly used when the population is large to survey everyone 2. Random sampling- this is used when it is not possible to reach every member of a population for study as a participant 3. Stratified random sampling- this is whereby a subset of the population that shares at least some common characteristics. Validity Internal validity- this specifically refers to whether a study makes a difference or not with adequate proof to support the claim External validity- this refers to the general treatment of outcomes The stakeholders will adopt the recommendations of the evaluation because the procedures used in designing and developing the evaluation will be well stipulated in the evaluation report. M6A1 Stakeholders have been identified to enable or hinder an evaluation before it is conducted, while it is conducted and after the results are collected and are ready to be used. Involving the stakeholders in any research ensures that the right evaluation questions and effective data interpretation is achieved. This also improves the quality and relevance of the interpretation by increasing the chances for field-development. It is important to note that stakeholders usually support any given evaluation and act on its data interpretation and recommendation in cases where they are involved in the evaluation process. This ensures that stakeholders support is required in any evaluation, or else this will result to criticism, resistance and even the sabotage of the evaluation. There are various limitations identified in engaging the stakeholders in data analysis. A lot of time is required for the evaluations, and this excludes various stakeholders from indulging in all the evaluation periods. This factor has been described to decrease the inducement of accountability and result to negative impacts on the responses. There is potential exploitation of consent, and this limits the ability of an evaluation to remain independent in light of other viewers and bias, this also influences the participant’s responses. Many opinions are produced in the evaluation and this makes it impossible for the stakeholders to reach a stable agreement. M7A1 The key stakeholders are the group of individuals and organizations that have effectively directed interest in the program. The group includes of the participants, program implementers, decision makers, and partners. They are usually affected by the evaluation results since they are the major partners in the group. The stakeholders are the current and potential funders who indulge in the completion of the projects report. The involvement of the different stakeholders with diverse perspectives on the program ensures that individuals develop both internal and external support for the evaluation process. In this case, I will use the sustainability report, which will provide details about the governance of the project, the start and progress of the project, financial implications, staffing, the full evaluation report from the start of the project, and the social, cultural and economic factors of the project. An effective delivery strategy is required to ensure that the report is effectively delivered. An important strategy for the report is to initiate categories for the stakeholders. I should focus on what the stakeholders expect from the evaluation and how they plan to use the results. It is also important to assure the stakeholders that I will strive to maintain open communications that address their concerns. This maximizes communication and involves the stakeholders input in the evaluation process. M8A1 In Michael Scriven’s paper, “Evaluating Evaluations: A Meta-Evaluation check list” he provides descriptive criteria for evaluation based on merit. The method is termed by the author as meta-evaluation checklist. The objective of the checklist is to provide a comprehensive and easy to use tool for professionals who may not necessarily be evaluators because of the use of easy-to-understand terms.  The checklist has six components or criteria. They include validity, credibility, clarity, propriety, cost-utility, and generalizability. The first five criteria have “non-zero bars” i.e. pass-mark levels whose corresponding inadequate performance cannot be offset by good performance on other components. Generalizability has no level of achievement set for it and the tool does not depend on it for determination of merit but earns bonus to the work is being (evaluand). Validity is a major evaluation component of the tool. It has four levels of achievement (bars) of which two are dominant. They include: contextual constraint (determination of the kind of evaluation required); coverage and correctness (probability of truth in the conclusions); reliability; and robustness. The tool assesses the validity of different types of work using a comprehensive composition of sub-criteria or bars. The tool, therefore, provides more accurate results on the basis of this consideration. The second item is credibility. This measures the relevance of the work or process being evaluated to stakeholders, staff and intended clients (audiences). By defining the credibility of the process or project, the tool reflects the possibility of bias (e.g. conflict of interest) and the level of expertise utilized. The third component is clarity. This measures the combination of concision and comprehensibility to stakeholders, staff clients, and audiences. It determines the level of acceptance and implementation. This component enables MEC to outperform other tools such as the Yellow book, KEC and PES. The fourth component is propriety which determination of the ethical, cultural, and legal appropriateness of a process (Ibid, p.5). This covers issues of consent, privacy, timeliness, and the lack of consideration of gender, religion, age, ethnicity, race, and sexual orientation. The fifth component is cost-utility which determines the economic nature of process or work. It measures cost-benefit analysis. Generalizability is the sixth component which, in the tool, is not considered a major determinant factor but a source of bonus. This measures re-usability, exportability and sustainability of the process or work. The sum of achievements in the 6 components provides an estimate of the merit and value of the work or process being evaluated. Component six determined is considered a bonus-dimension because the determination of merit or value can attain the highest grade without the inclusion it inclusion. Meta-Evaluation Checklist is a valuable process because it relies on a tool that is less dependent on small errors or variations of measurement of components involved. MEC provides a good lesson to evaluators that accurate evaluation depends on a specific and comprehensive measurement of the various dimensions of the work. However, the issue of generalizability is an important issue. For instance, the issue of sustainability is an important issue in all kinds of work and projects. Work is intended to benefit both the current and future situations. There, a recommendation for change should be that the sixth component should have enough weight to influence the total outcome of an evaluation exercise. References Miles, M, Huberman, A.M & Saldana, J. (2013). Qualitative Data Analysis: A methods source book. 3rd ed. Newbury part. C.A: Sage Publications. Julian, D. (1997). "The utilization of the logic model as a system level planning and evaluation device." Evalua tion and Program Planning 20(3): 251-257. Read More