Student Support Services Program Evaluation - Research Proposal Example

Chapter Introduction Background The East Mississippi Community College (EMCC) is a supported associate degree Commission on Colleges of theSouthern Association of Colleges (SACS-COC) accredited Level I Community College serving the counties of Clay, Kemper, Lauderdale, Lowndes, Noxubee, and Oktibbeha for almost 80 years (Students, 2009).  Because averagely 21.28% of students at EMCC are of the low-income family and first generation students, EMCC’s appropriation cannot sustain programs that support this particular aggregate of students. The Student Support Services (SSS) offers the school assistance in this matter. The SSS program is a federal grant funded through the U. S. Department of Education for the purpose of increasing both retention and graduation rates, increasing rates of students who continue on to 4-year institutions and the generation of an environment that impel successful low-income and first-generation students; and students with disabilities. The implementation of the SSS program at EMCC is guided by 3 major objectives: outcome objectives, performance objectives, and process objectives. Outcome objectives evaluates measures of retention rates, graduation rates, good academic standing rate, transfer rates to four-year universities and campus climate. Performance objectives, on the other hand, are designed to measure the success of the SSS activities and services towards achieving the first objective. This includes special orientation for project participants, completion of applications to 4-year institutions, and tutoring. And the last, process objectives, is intended measure the efforts the SSS staff to put into action the program’s services and activities including identifying and selecting project participants, assessing the need for services, monitor progress of the participants, financial and economic literacy counseling, personal and academic counseling, mentoring and cultural activities and academic programs. The complete report of the objectives is included in the appendix section. Problem Statement To be eligible for the Student Support Services program, a student must meet any of the following criteria for eligibility namely: belongs to a low-income family, a first-generation college student, documented disability, and must want to obtain a bachelor’s degree from a 4-year institution. It is encouraging to note that EMCC’s graduation rate for SSS-eligible students is significantly higher when compared to the non-eligible students (74:23% for 2008). At the same time, student enrollment statistics for 2006–2008 clearly shows that the retention rate of SSS eligible students are higher that that of non-eligible students (54:91; 55:77; 55:70. Conversely, it is also clear that while non-SSS eligible students have maintained their retention or persistence rate, SSS eligible students’ persistence rate has dramatically dropped from 91% in 2006, to 70% in 2008. The Office of Disability Services reports that in the Fall of 2008, 889 students were SSS eligible accounting for 21.28% of the total population of students enrolled and actually mirrors the number of SSS eligible students since 2006. This means designing activities and services that would enhance the eligible students’ chances of pursuing a college degree is essential. With the Mississippi State’s 15 Community Colleges sharing the 6.5 % of the State’s education budget and even with the 6.49 percent increase in the allocation awarded this year, the resources of the school will not be enough to support the scores of SSS eligible students. Nevertheless, the SSS program still aspires for the implementation of its activities and services directed towards helping EMCC in producing SSS eligible graduates that are 4-year institution ready. The outcome objectives of the SSS program are as follows: persistence rate to increase by 11%, from 54% to 65%; graduation rate of 45% among the participants within six years, representing 20% increase from 25% in 2006; a transfer rate of 45% from 26% within four years; and an 85% of student will report a supportive institutional climate. These might seem like ambitious targets for those who are on the outside looking in. The program proponents, however, assure that these are attainable goals albeit appearances towards the contrary. The evaluation of the Student Support Services Program is therefore crucial in order to appraise its effectiveness and its progress, as well as to identify areas that would need strengthening, improvement, or modification. In doing so, eligible students will be assured of continued support and services, both from the EMCC and the Student Support Services; that is both successful and effective. Definitions 1. Persistence Rate – Persistence rate is conceptually defined as the rate of retention of SSS program participants from the time of entry to EMCC until they graduate. 2. Graduation rate – Graduation rate is conceptually defined as the percentage of SSS program participants that graduate from their two-year program from EMCC. 3. Transfer rate – Transfer rate is conceptually defined as the percentage of SSS program participants that proceed to a 4-year university or institution after graduating from EMCC. 4. Supportive institutional climate- Supportive institutional climate is conceptually defined as the SSS program participants’ perceptions of EMCC’s provision of an environment that is supportive of their success. Research Questions The purpose of this study is to evaluate the effectiveness of the current Student Support Services Program being implemented at the East Mississippi Community College. Specifically, it aims to measure the progress in the achievement of the program’s outcome objectives, as well as to determine the effectiveness of the activities and services provided by the program in the achievement of the outcomes objectives objectives, and the achievement of the outcomes objectives. This study proposes to answer the following questions: 1. What is the demographic data of SSS program participants in terms of? a. Gender b. Age c. Educational attainment d. Socio-economic status e. Educational attainment of parents f. Documented disability 2. What is the current status of the Student Support Services Program performance objectives in terms of? a. Special orientation for Project participants b. Completion of applications to four-year universities c. Tutoring 3. What is the current status of the Student Support Services Program process objectives in terms of? a. Identify and select project participants b. Assess need for services and monitor progress c. Financial and Economic Literacy Counseling d. Personal and academic counseling e. Mentoring f. Cultural activities and Academic Programs 4. What is the status of the Student Support Services Program outcome objectives before program implementation in terms of? a. Persistence rate among SSS program participants? b. Academic standing rate among SSS program participants? c. Graduation rate among SSS program participants? d. Transfer rate to four-year universities among SSS program participants? e. Perception of campus climate SSS program participants? 4. What is the status of the Student Support Services Program outcome objectives one year after program implementation in terms of? a. Persistence rate among SSS program participants? b. Academic standing rate among SSS program participants? c. Graduation rate among SSS program participants? d. Transfer rate to four-year universities among SSS program participants? e. Perception of campus climate SSS program participants? 5. Is there a difference in the status of the Student Support Services Program outcome objectives before and one year after implementation in terms of? a. Persistence rate among SSS program participants? b. Academic standing rate among SSS program participants? c. Graduation rate among SSS program participants? d. Transfer rate to four-year universities among SSS program participants? e. Perception of campus climate SSS program participants? 6. Is there a relationship between the performance objectives, process objectives and the outcome objectives? Chapter 2 Literature Review Introduction When implementing projects, constant evaluation is an integral and necessary undertaking. Each phase of the project need to be scrutinized in order to assess the project’s progress and ensure that it is heading in the right direction. Process evaluation Process evaluation is a program evaluation method that measures the coverage of the program and the process by which the program is being implemented (WHOa, 2000). It allows program evaluators to establish the connection between the coverage and the process (McNamara, 2008; McCawley, 2001). Coverage identifies the success of the program in reaching its target population. Process, on the other hand, explores the accountability of the program, that is, that the activities and services of the program produced the quality and quantity of the output specified in the objectives (W. K. Kellog Foundation, 2004) . The first step in conducting a process evaluation is to define the characteristics of program participants in order to determine if the intended population is actually reached by the program. The next step is to explore the performance of activities and services designed to achieve the main objective of the program, to identify problematic areas within the process, and map out possible areas for improvement (CDC, 2009). Performance evaluation Performance evaluations are conducted in order to appraise the program’s performance of established performance parameters designed to achieve the main goal of the program (Maryland, Speros, 2001). It aims to determine the productivity of the program in terms of compliance with the programs requirements and guidelines such as meeting targets, timetables and deadlines and helps in identifying the benefits and limitations of the program (Arh & Schwartz, 2009). Through formative feedback, wasteful processes can be eliminated and areas for improvement can be dealt with accordingly, ensuring that efforts and resources are focused on the set objectives of the program (APS, 2002; Barlow, 2009). Outcome evaluation Outcomes evaluation is a program evaluation method that uses a quantitative approach to assess the impact or effectiveness of the treatment to the dependent variable by tracking the performance of acceptable outcomes have been demonstrated and ascertaining that the outcome is indeed the result of the intervention (McCawley, 2001; Springer, 2001; WHOb, 2000).  Outcomes evaluation uses three designs: Randomized controlled trial, control group design and pre-post test design.  Randomized controlled trials (RTC) exercise the most rigor in that it utilizes control methods in order to minimize bias and accurately predict the causation of a phenomenon such as randomization and the use of a control group.  The control group design, on the other hand, utilizes a control group similar to the RTC.  However, it lacks the randomization factor of the RTC.  The groups in comparison are matched as much as possible in their characteristics and whatever differences there remains can be controlled through the use of statistical treatments.  The third design: the pre-post design is the design that utilizes the least rigor.  Still, it is a method that is trusted to provide useful data to determine if the program objectives are being achieved; for the purpose of program improvements.  The pre-post test design compares the data before and after the introduction of the treatment in order to determine its effectiveness.   Chapter 3 Methodology Problem Statement The purpose of this study is to evaluate the effectiveness of the current Student Support Services Program being implemented at the East Mississippi Community College. Specifically, it aims to measure the progress in the achievement of the program’s outcome objectives, as well as to determine the effectiveness of the activities and services provided by the program in the achievement of the outcomes objectives. Research Questions 1. What is the demographic data of program-eligible students in terms of? a. Gender b. Age c. Educational attainment d. Socio-economic status e. Educational attainment of parents f. Documented disability 2. What is the current status of the Student Support Services Program performance objectives in terms of? a. Special orientation for Project participants b. Completion of applications to four-year universities c. Tutoring 3. What is the current status of the Student Support Services Program process objectives in terms of? a. Identify and select project participants b. Assess need for services and monitor progress c. Financial and Economic Literacy Counseling d. Personal and academic counseling e. Mentoring f. Cultural activities and Academic Programs 4. What is the status of the Student Support Services Program outcome objectives before program implementation in terms of? a. Persistence rate among SSS program participants? b. Academic standing rate among SSS program participants? c. Graduation rate among SSS program participants? d. Transfer rate to four-year universities among SSS program participants? e. Perception of campus climate SSS program participants? 5. What is the status of the Student Support Services Program outcome objectives after program implementation in terms of? a. Persistence rate among SSS program participants? b. Academic standing rate among SSS program participants? c. Graduation rate among SSS program participants? d. Transfer rate to four-year universities among SSS program participants? e. Perception of campus climate SSS program participants? 6. Is there a difference in the status of the Student Support Services Program outcome objectives before and one year after implementation in terms of? a. Persistence rate among SSS program participants? b. Academic standing rate among SSS program participants? c. Graduation rate among SSS program participants? d. Transfer rate to four-year universities among SSS program participants? e. Perception of campus climate SSS program participants? 7. Is there a relationship between the performance objectives, process objectives and the outcome objectives?  Sample Selection The sample for this study will be derived from the total SSS program participants. All SSS program participants will be deemed eligible to be included as a sample for this study. The researcher will meet the participants in order to discuss the nature of the study. It will be explained to them that they can choose whether or not to participate in the study, or withdraw their participation at any time within the study period.  They will be assured full confidentiality of any information collected and such information shall never be revealed for any reason, other than to clarify necessary information for this particular study. They will be given a chance to ask questions and clarifications and shall be answered by the researcher without delay.  The Slovin’s formula will be used to determine the sample size using the equation below: n = N / (1+Nα 2) Where: n = number of samples N = total population α = margin of error The α will be set at 0.05 permitting a 5% margin of error. The sample size computed through the Slovin’s formula will be drawn using a random sampling technique. Those who will consent to participate will be assigned numbers and drawn through the use of a random number generator program from the Microsoft Office 2007 Excel program.  The researcher’s contact number will be given to the sample in case of any questions or clarifications or change in the decision to participate. Instrumentation The instrument that will be utilized for this study will be a researcher developed questionnaire, items of which will be derived from the SSS program objectives. For example, the item for evaluating the effectiveness of the process objective tutoring will be measured through their academic performance such as GPA. The questionnaire will be subjected to validity and reliability testing before implementation of the research.   Data Collection and Analysis A pre-post test design will be utilized for this study, specifically directed towards investigating the achievement of the program’s outcomes objectives and to explore the effectiveness of the process and performance objectives in influencing the achievement of the outcomes objectives. Participants will be surveyed using a closed-ended questionnaire to be developed by the researcher. For research question number 1, the following demographic variables will be collected: Gender, age, educational attainment, socio-economic status, educational attainment of parents, and documented disability. These will be analyzed through the use of descriptive statistics specifically distribution, measures of central tendency and percentile. For research question number 2, the following data on performance objectives will be collected: The program participants’ awareness of the existing activities and services of the program (Special Orientation), completion of applications to four-year universities and their academic standing (tutoring). These will be analyzed through the use of descriptive statistics specifically distribution and percentile. For research question number 3, the following data on process objectives will be collected: rate of recruitment and representation of students from the different target groups, rate of participant monitoring, rate of financial and economic literacy counseling, rate of academic and personal counseling, rate of participants mentored, and the rate of participation to cultural and academic activities. These will be analyzed through the use of descriptive statistics specifically distribution and percentile. For research question number 4, the following data on outcome objectives will be collected: The retention rate among the program participants, academic standing, graduation rate transfer rate and campus climate before the implementation of the SSS program. These will be analyzed through the use of descriptive statistics specifically distribution and percentile. For research question number 5, the following data on outcome objectives will be collected: The retention rate among the program participants, academic standing, graduation rate transfer rate and campus climate one year after the implementation of the SSS program. These will be analyzed through the use of descriptive statistics specifically distribution and percentile. For research question number 6, the difference between the retention rate among the program participants, academic standing, graduation rate transfer rate and campus climate before and one year after the implementation of the SSS program will be compared. The comparison between the pre-test data and the post-test data will be through a Mann-Whitney U test for ordinal/rank data. For research question number 7, the relationship between the process and performance data and the progress of the outcome objectives will be explored. The Spearman Rank Order coefficients will be utilized to explore relationships between the variables in order to identify factors that contribute to the progress and success of the program. Timeline Considering that the program plan was implemented in August 2009, the post-test data will be collected after August 2010 to allow for 1 year implementation time. Hereforth is the proposed timeline for this study. Contact Person/s Activity Date of completion Adviser Approval of questionnaire January 15, 2010 Program participants Pilot Testing of questionnaire/ Validity and reliability testing/ Revision February 15, 2010 SSS facilitators and participants Meeting with SSS facilitators and participants April 15, 2010 SSS facilitators and participants Selection of respondents August 1, 2010 Sample Obtain consent/ Distribution of Questionnaires August 15, 2010 Sample Collection of Questionnaires August 30, 2010 Collation of data/ Statistical Analysis September 10, 2010 Writing of chapters 4 and 5 October 20, 2010 Panel Defense November 15, 2010   Expected Outcomes The scope of the study includes the program plan currently under implementation At EMCC. Although some objectives are achievable within four to six years, still, it is important to conduct performance and process evaluations in order to track progress and make necessary adjustments. There is a big percentage of students at EMCC who are in need of a superior program to assist them in their endeavor to eventually get a degree from a 4-year university or institution. The strength of this study will be in the use a random sample for which results can be generalizable to the population of SSS program participants. This study may be limited in such that it will evaluate the program only at one point in time, and that is one year after the program implementation. It is this researcher’s recommendation therefore, that concurrent evaluations be made in order to ensure that the program’s activities and services are aligned to its outcome objectives. The success of this program will not only be significant to the proponents and implementers of the program, but more so, to the significant number of SSS-eligible students, and the elevation of educational attainment in East Mississippi. References Aitken, William; OConor, Darren; OConor, Darren (Editor). Planning and Controlling Projects. Retrieved from http://site.ebrary.com/lib/ucbirmingham/Doc?id=10040397&ppg=135 Arh, G., & Schwartz, R. (2009). Supporting program management by developing, implementing, and transferring knowledge from the performance indicator monitoring systems (PIMS). Public Performance & Management Review, 33(2), 241-254. doi:10.2753/PMR1530-9576330204. Barlow, R. (2009). News: Redefining value analysis practices for a healthcare reform- minded industry: are healthcare organizations generating true value or spinning wheels?. Healthcare Purchasing News, 33(10), 8-11. Retrieved from CINAHL Plus with Full Text database. Center for Disease Control (2009). Chapter 1 – Program planning and evaluation. Retrieved from http://www.cdc.gov/vaccines/vac- gen/policies/ipom/downloads/chp-01-program-plan-and-eval.pdf East Mississippi Community College (2009). TRIO: Program and eligibility. Retrieved from http://www.eastms.edu/index.php?option=com_content&task=section&id=22&Ite mid=91 McCawley, P. (2001) The logic model for planning and evaluation. University of Idaho Extension Retrieved from http://uidaho.edu/extension/LogicModel.pdf McNamara, C. (2008). Basic guide to program evaluation. Retrieved from http://www.managementhelp.org/evaluatn/fnl_eval.htm#anchor1581634 World Health Organization (WHOa), (2009). Workbook 4: Process Evaluations. Retrieved from http://www.unodc.org/docs/treatment//process_evaluation.pdf World Health Organization (WHOb), (2009). Workbook 7: Outcome Evaluation. Retrieved from http://www.unodc.org/docs/treatment//outcome_evaluation.pdf Maryland Department of Budget & Management. Maryland Performance Planning and Evaluation Program Users Handbook. Retrieved from http://www.dhmh.state.md.us/ohr/pep/pephandbook.pdf Speros, C. (2001). A system to evaluate provider performance within the Shelby County healthcare network. Retrieved from CINAHL Plus with Full Text database. Springer, M. L. (2005). Program Management: A Comprehensive Overview of the Discipline. Purdue University Press West Lafayette, IN Technical Assistance Cooperative (2005). Chapter four: Program performance evaluation process Retrieved from http://www.tacinc.org/index/admin/index/_uploads/docs/MEA$URE_UP_Chapter _4_Process.pdf The American Physiological Society (2002). Program evaluation: The key to a successful program. Retrieved from http://www.the- aps.org/education/promote/promote.html W.K.Kellogg Foundation (2009). Logic model development guide. Clear and concise discussion of the use of logic models. Retrieved from (www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf) Read More