Questions on Program Analysis, Monitoring and Evaluation - Assignment Example

Questions on Program Analysis, Monitoring and Evaluation Q using your knowledge of measurement in the social sciences, select a public problem with which you are familiar and describe how you would measure it. Sol. there are so many public problem that we see in our daily lives. One major problem is the problem of availability of goods in the market. We often see that there is always a scarcity of public goods in remote areas and it is a major problem in developing nations. a public good is a good that is non-rivaled and non excludable. it fairly means that no one has the right to stop other in taking the benefit of the goods and consumption of the good by one individual does not reduce the availability of the good for consumption by others. Public goods are very good example of market failure. Market failure exists when the production or use of goods and services by the market is not efficient. It leads to the free rider problem. Causian solution is a very fair solution to this problem. It is a mechanism by which the potential beneficiaries of a public band together and pool their band together and pool their resources based on their willingness to pay to create the public good. Me rover if the transaction cost between the beneficiaries are sufficiently low then an adequate level of public good production can occur even under competitive free market condition. Q.2 what are the essential elements of the classical experimental design (CED) in evaluation research Define the concept "threat to validity," and identify the threats to validity which a classical experiment does and does not address. Sol. evaluation is a systematic determination of merit, worth, and significance of something. it someone using criteria using a set of standards it is a methodological area than is closely related to, but distinguishable from more traditional social research while the design of experiments is a theory concerning minimum number of experiments necessary to develop an empirical model of a research question and methodology for setting up the necessary experiments. There are various elements of classical experimental design related to research evaluation research such as replications, variations, internal validity, and reproducibility. Concept of validity: the validity of study reflects the accuracy of the results. Matter of validity can bring complexity. So the principal concerns of validity should be straightforward and its consideration is very necessary in every evaluation. Validity can be classified as internal and external validity. Empirical there are too many problem that needs to be solved experimentally or by observations. Different research designs are susceptible to various types of validity threats such as the common pre/post design. However it is not valid for what we know as malnutrition effect .another common threat to validity is knows as history effects. It is different from that of malnutrition effects. Selection effects, mortality effects and threats to external validity are next in the cue. These above are the threats to validity that a classical experiment does addresses Q.3 Describe the steps in the "prospective evaluation" methodology recommended by the GAO. Address the question, Should all new public programs or policies be subjected to prospective evaluation Sol. GAO has developed the PES as a systematic method for fulfilling congressional request for analyzing proposed legislation and helping and helping identify top-priority problems. Tools of evaluation methodology are very helpful in providing best possible information's on the likely out comes of proposed programs Following are the steps involved (1) A careful, skilled textual analysis of proposed program, designed to clarify the implied goals of that program and what is assumed to get results (2) A review and synthesis of evaluation studies from similar programs. (3) Summary judgments of likely success, given a future context that is not too different from the past. Prospective analysis is a very promising tool when it comes to analyze new public program. It gives expert design assistance to a new program at the point when it is most needed and helps convince others of the basic logic and likely success program. If we see in a public policy perspective, it provides understanding about how a program is likely to work and thereby increasing its chance for success. Q.4 Describe the Van Meter and Van Horn model of the policy implementation process and discuss it relevance to the designing studies of implementation as part of program evaluations. Sol. In the past three decades, models based on econometric information and information from participation in large governmental programs have become important components of the policy making process. It is likely that models based on Geographic Information Systems shall follow suit. There is temptation to believe that models, by providing superior factual guidance, significantly transform the policy making process. The most comprehensive of implementation perspective to date is the attempts to integrate several previous conceptualizations are proposed by van horn and van meter in 1979. they claimed that four variables most affect a program's implementation: 1-the strength of the statues and ensuring regulators 2-the presence of a fixer, i.e. a key legislator attuned to program requirement 3-the resources of various constituency groups 4-the commitment and leadership of agency officials Q.5 Describe logic models (theories of change, log-frame, results chains, etc.), sketch the model of a program with which you are familiar and discuss their uses in policy and program evaluations. Sol. the logic model is a general structure for unfolding work in an organization. Since work is often put together in programs, it is often referred to as the program logic model. In its simplest form, logic model analysis works into four groups or steps that are input activities, outputs and outcomes. Example of a logic model: inputs processes outputs short-term outcome(s) intermediate outcomes long-term outcomes - Free articles and other pamphlet on the Web - coworkers - Free Management Library - Funders - Self-directed learners - Volunteers - Computers - Web - provisions - Provide peer-assistance models in which learners support each other - Provide free, online training program: Basics of sovereign Learning - gives free, online training program: Basic Life Skills - Provide free, online training program: Passing your GED Exam - 30 groups that used peer models - 100 completed training programs - 900 learners who finished Basics of autonomous Learning - 900 beginners who finished Basic Life Skills - 900 learners who passed their GED to gain high-school diploma - high school diploma for alumnae - improved attitude toward self and society for alumnae - enhanced family life for family of alumnae - around the clock employment for beginners (in job that required high-school education) - increased dependability and improved verdict of learners - enhanced publicity and public relations for SDLC - independent living for learner (by using salary to rent apartment) - strong basic life skills for beginners - improved love life for learner who's now in a commitment - increased probability and interest for learner to be present at college Logic model is not meant to achieve financial benefits, perhaps it is used but government or by such an organization who has no relation with the benefits of the outcomes .however in such a scenario, where profit is not the question, it can roof a bit tricky to keep an eye on progress towards outcomes. A program logic model provides such indicators, in terms of output and outcome measures of performance. Q.6 Explain how the performance measurement process used by managers differs from systematic program impact evaluation. Sol. performance measurement is the method whereby an organization creates the parameters within which programs, investments, and acquisitions are reaching the desired result. This process of measurement performance frequently requires the use of statically confirmation to find out growth toward specific defined organizational objectives. An impact evaluation measures the program's effects and the level to which its objectives were attained. Even though evaluation designs may create valuable information about a program's efficiency, some may produce more valuable information than others. Q.7 Public policy and program evaluation does not occur in a vacuum. Discuss the place of the environment in logic models and some of its implications for evaluation. Sol. Public Policy is the base for all government achievement. All public service, all publicly directed finances, every public bureaucrat is guided by policy. Public Policy is the very structure for supremacy - making it more available, easier to value and receptive is critical in improving the odds for achieving success. It is a well kwon fact that government consists of tree elements. Government works best when these elements can congregate and we believe that Information Technology is perhaps the best facilitating medium for making that happen. Policy Integration builds upon Policy construction and is premised on the notion that "actions do not occur in a vacuum. The program theory is the formal description of the program's concept and design. This is also called a logic model or impact pathways. The program theory breaks down the components of the program and shows anticipated short- and long-term effects. An analysis of the program theory examines how the program is organized and how that organization will lead to desired outcomes. It will also reveal unintended or unforeseen consequences of a program, both positive and negative. The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders. Q.8.Enumerate and define the multiple criteria used to assess measures used in evaluations using as examples program measures with which you are familiar. Sol. the numerous performance criteria performance evaluation includes: 1. The cost of designing and collecting multiple criteria is high. 2. Non-profit performance criteria are difficult to understand and evaluate because of their partisanship. 3. A single comprehensive measure is often needed for decision-making purposes. For e.g. a situation where member of a decision making committee must decide on which of three candidates to endorse, the decision maker will compare three criteria (headship, human relations, and financial management ability) in pairs to develop a position. The comparisons would be: 1. Is leadership more important than human relations skills for this job 2. Is leadership more important than financial management ability for this job 3. Are human relations skills more important than financial management ability for this job A ratio scale of 1 to 9 is added to the ordinal ranking provided by the responses to these issues to provide the relative importance of one criterion over another. The twosome comparisons are summarized in a square matrix and the Eigen values and preference vector are then computed to determine the relative ranking of the criteria. Then each of the candidates is compared on the basis of the criteria as follows: 1. Is candidate A superior to Candidate B in leadership skills 2. I s candidate A superior to Candidate C in leadership skills 3. I s candidate B superior to Candidate C in leadership skills Q.9 There is a number of quasi experimental designs that are used to evaluate policies and programs. Using a specific quasi-experimental evaluation (study) with which you are familiar, describe the evaluation design and assess its success in addressing threats to validity. Sol. Regression-discontinuity design is a type of quasi experimental design. RD design is seen as a valuable method for determining whether a program or treatment is efficient. In its simplest most traditional form, the RD design is a pretest-posttest program-comparison group plan. The unique features which lay down the RD designs apart from other pre-post group designs is the method by which research contributors are allocated to circumstances. In RD designs, contributors are assigned to program or comparison groups only on the basis of a cutoff score on a pre-program measure. Thus the RD design is distinguished from randomized experiments (or randomized clinical trials) and from other quasi-experimental strategies by its unique method of obligation. This cutoff criterion involves the major advantage of RD designs -- they are appropriate when we wish to target a program or treatment to those who most need or deserve it. Thus, unlike its randomized or quasi-experimental alternatives, the RD design does not require us to assign potentially needy individuals to a no-program comparison group in order to evaluate the effectiveness of a program. The Internal Validity of the RD Design, Internal validity refers to whether one can deduce that the treatment or program being examined caused a change in outcome indicators. Internal validity as imagine is not concerned with our ability to simplify but rather centralized on whether a fundamental relationship can be established for the immediate research context. Research designs which address causal questions are often compared on their relative ability to yield internally valid result. The first step in any project is to develop a plan for the work to be done. The plan for an evaluation project is called a "design" and is a particularly crucial step to give an appropriate evaluation. A good design offers an chance to make the most of the quality of the evaluation, helps diminish and justify the time and cost necessary to perform the work, and increases the strength of the key findings and recommendations by ensuring that threats to valid results are minimized. Reference: 1. Wright, Benjamin D. (1983). Fundamental Measurement in Social Science and Education. Research Memorandum No. 33a MESA Psychometric Laboratory. 8 April, 2009. 2. Wright, Benjamin D. A HISTORY OF SOCIAL SCIENCE MEASUREMENT. Measurement for Social Science and Education. MESA Psychometric Laboratory. 8 April, 2009. < http://www.rasch.org/memo62.htm > 3. Ohlund, Barbara & Yu, Chong-ho. (1999). Threats to validity of Research Design. 8 April, 2009. < http://www.creative-wisdom.com/teaching/WBI/threat.shtml > 4. Coldren, Jr. James R. "Chip". (1989). Evaluating Juvenile Justice Programs A Design Monograph for State Planners. Office of Juvenile Justice and Delinquency Prevention. 8 April, 2009. < http://www.jrsa.org/evaluationwebsite/guide/documents/documentg.html> 5. Trochim, William M.K. (2006). Introduction to Evaluation. Research Method Knowledge Base. 8 April, 2009 < http://www.socialresearchmethods.net/kb/intreval.htm > 6. Office of Program Evolution, Enforcement & Review. Evaluation Design. USDA Food Safety and Inspection Service. 8 April, 2009. 7. Chan, Y. L. & Lynn, B. E. (1991). Performance evaluation and the analytic hierarchy process. Journal of Management Accounting Research. 57-87. 8 April 2009. 8. Public Policy Sources. (1999). Public Policy Sources #37: Growth theory: the causes of economic growth. 8 April 2009. 9. Evaluating Drug Control and System Improvement Projects. (1992) Guidelines for Projects Supported by the Bureau of Justice Assistance. National Institute of Justice. Bureau of Justice Assistance, U.S. Department of Justice. 8 April, 2009. 10. DEVELOPMENT PROCESSES. PERFORMANCE MEASUREMENT PROCESS. Trade. 8 April, 2009. 11. Prospective Evaluation Method. The prospective Evaluation Synthesis.(1990). Program Evaluation and Methodology Division. United States General Accounting Office. GAO. 8 April, 2009. 12. McNamara, Carter. (1997). Guidelines and Framework for Designing Basic Logic Model. Guidelines and Framework for Designing Basic Logic Model. 8 April, 2009 13. Research Methods & Experimental Design. (2004). Human Supervisory Control. 8 April, 2009. 14. Sutton, Rebecca. (1999). THE POLICY PROCESS: AN OVERVIEW. Overseas Development Institute Portland House. Stag Place London SW1E 5DP. Chameleon Press Ltd, London SW18 4SG. 8 April, 2009. < http://www.odi.org.uk/resources/odi-publications/working-papers/118-policy-process.pdf > 15. King, John Leslie & Kraemer, Kenneth L. Models, Facts, and the Policy Process: The Political Ecology of Estimated Truth, Center for Research on Information Systems and Organizations (CRITO). Graduate School of Management and Department of Information and Computer Science. University of California, Irvine. Working Paper #URB-006. 8 April, 2009. < http://www.crito.uci.edu/research-archives/pdf/urb-006.pdf > 16. United States political science documents. (1985). University of Pittsburgh. University Center for International Studies. American Political Science Association Published by University Center for International Studies, University of Pittsburgh. Item notes: v. 10, pt. 2. Original from the University of Michigan. (2008). 8 April, 2009. Read More