Student Rating in Higher Education - Research Proposal Example

Running Head: Research Proposal Research Proposal Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Name Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Course Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Lecture Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Date Contents Research Proposal 3 1.0 Research Topic: 3 2.0 Introduction 3 2.1 Research Hypothesis 5 3.0 Literature Review 6 3.1 Primary Study 1 6 3.2 Primary Study 2 7 3.3 Primary Study 3 8 3.4 Primary Study 4 9 3.5 Primary Study 5 10 3.6 Primary Study 6 12 3.7 Primary Study 7 13 3.7.1 Procedure 13 3.7.2 Instrumentation and procedure 14 4.0 Research Methodology 14 4. 1 Research Setting 14 4.2 Research Sampling 15 4.3 Instrumentation 15 4.4 Research procedure 16 4. 5 Data Analysis 16 4.6 Research custom to be followed 17 5.0 Timing and Research Budget 17 5.1 Budget 17 5. 2 Research Budget 17 6.0 Weakness and Strength of the Study 17 7.0 References 18 Research Proposal 1.0 Research Topic: Investigation of teaching effectiveness using student rating and faculty self-rating in higher education. 2.0 Introduction When delivering instruction in a classroom a teacher employs various methods of teaching. The amount of understanding gained by a student is a very good indicator of whether a particular teaching method is effective. According to Galbraith, Merrill and Kline (2012) teaching methods are defined as principles and tactics used to deliver instructions. The teaching method used impacts hugely on whether a teacher is considered effective by his students or not. According to Galbraith, Merrill and Kline (2012) learning should be informative, productive, satisfying and worthwhile from the student’s perspective. Students are uniquely positioned to give the most useful feedback on the effectiveness of any teaching method. There is a lot of research on the education on the way teachers should deliver their instruction in class in order for learning to be enhanced. Other research is indicates to teachers what not to do in classes. Despite this teaching effectiveness has not been well investigated using both students and faculty perceptions. Sometimes teachers use a method that they are comfortable with without any regard for those they are instructing. According to Theall and Franklin (2010) student satisfaction is linked to the effectiveness of a teaching method although it is not a direct measure of instructor effectiveness. Studies on the student ratings of their instructors have found them to be valid, unbiased, valid and useful in evaluating instructor effectiveness (Calkins and Micari, 2010). Presently, most institutions of higher learning use student rating to evaluate the teaching effectiveness of their faculty. One of the most useful application of student ratings is to assist instructors improve their teaching skills. Furthermore, they are now being widely applied in by human resource management in educational institutions to make personnel decisions. Ory and Ryan (2001) say that student ratings are used to make decisions such as personnel retention, promotion, tenure and award of honors or assigning tutors to specific classes. According to Ory (2001) over 90 per cent of universities in the United States use student’s ratings for evaluating tutors and courses. However, faculty self-assessment has been poorly applied in studies on teaching effectiveness. Teaching effectiveness is defined by Theall and Franklin (2010) as producing beneficial and purposeful student learning through appropriate procedures. On the other hand an effective teaching method according to Centra (2002) is where an environment for appropriate comprehension of instructions is created. The legitimacy of student feedback as an indicator of teaching effectiveness has been questioned in various studies (Calkins and Micari, 2010). Therefore student evaluation needs to be combined with other methods of evaluating teaching effectiveness. According to Centra (2002) students are exposed to various teaching methods including explaining, demonstrations, lecturers, readings among others. Thus as consumers of these methods they are able to discern the quality, usefulness, relevance and how instructor interacts with the students. However as the producer of instruction tutor’s also have a legitimate voice in evaluating teaching effectiveness. 2.1 Research Hypothesis Evaluation of teaching effectiveness using both the student evaluation and faculty evaluation are equally valid methods of assessing teaching effectiveness in higher education. 2.2 Research Objective To find out if student evaluation of teaching effectiveness and faculty self-rating produce similar results. 3.0 Literature Review 3.1 Primary Study 1 Sajjad, S. (2011). Effective teaching methods at higher education level was a study on the effectiveness of teaching methods conducted set in the University of Karachi. The aims of the study were set as follows; To find out which methods are used in various departments in delivering instructions. To investigate student perception on whether a particular method was interesting. To find out why students perceive a teaching method as effective. To suggest strategies for improving the quality of teaching. The study uses purposive sampling to gather evidence of teaching effectiveness from a sample drawn from the faculty of arts at the university. Students selected for the sample were drawn from the psychology, sociology, social work, special education, mass communication, and international relations, Education, Urde, Persian and Islamic Learning (Sajjad, 2011). The sample selected 20 students in each of the department for the study. The non-probability sampling method of convenience sampling was used where the most accessible and proximate participants were considered for the study. The participants chosen for the study were questioned about the effectiveness of teaching methods used by their tutors, tutors where also asked to give the reason for rating a teaching method lower or higher than others. The total sample for this research was 220 students (Sajjad, 2011). The study used interviews and questionnaires to obtain feedback from the participants. Teaching methods were categorized into; Lectures, Group Discussion, Individual presentations, Assignments, Seminars, Workshop, Conferences, Brainstorming, Role play and Case Study (Sajjad 2011). On each method the students were asked to rate the teaching effectiveness on the scale of 1 to 5; with one being least effective and 5 being most effective (Sajjad 2011). The second part of the study where students were asked the reason for a particular rating was an open ended question. 3.2 Primary Study 2 Spiller, D., & Ferguson, P. B. (2011). Student evaluations: do lecturers value them and use them to engage with student learning needs? This is a study used to evaluate the perception of lecturers of the validity of students evaluating teachers model. According to Spiller and Ferguson (2011) the aim of the research was to explore how lecturers perceive the formal student appraisal process and whether they use the feedback to improve student learning. The research used electronically mailed questionnaires that were sent to academics in three New Zealand Universities. The response rate to the questionnaires was 47.19% (Otago University), 37% (University of Waikato and 45.42% (Otago University) (Spiller and Ferguson, 2011). The questions sent to the respondents gathered qualitative responses from the academics that had chosen to respond to the sample. Purposive sampling was used to select interviews among those who had answered the questionnaires. The selection criteria for interviewees included academic level, career stage, experience and academic discipline. The results reported by this paper are partial reports of the study and only presents results from a sample of 20 academicians at the Waikato University. The qualitative data provided by the lecturers was analyzed to find their perception of student’s judgment of their teaching effectiveness. Secondly, the analysis aimed to find out how teachers respond to student evaluation to improve their own teaching. The research questions for this paper were: “What do academics think about the usefulness of the information that is provided by the students in the formal evaluations process and how does student feedback influence their teaching and assessment practices?” (Spiller and Ferguson, 2011). One of the major weaknesses of this study is its sampling technique. The paper also makes use of questionnaires which are shown to elicit low response rates among respondents. Furthermore, the process of sending questionnaires is selective as they are sent to specific targets of the research. Further use of purposive sampling while selecting the interviews means the research has a substantial sampling bias. Therefore, the results of the study may not be representative of the perception of the whole population of the Student appraisal process. Furthermore the use of volunteers in the research may also be pointers to a possible existence of bias in the research results. From this research one is able to learn the weaknesses that are inherent in using volunteers for a research study. The enthusiasm of volunteer respondents in the research most of the time means results are skewed. 3.3 Primary Study 3 Burden, P. (2009). A case study into teacher perceptions of the introduction of student evaluation of teaching surveys (SETs) in Japanese tertiary education is a descriptive case study into the perceptions of teachers on student’s evaluation of teachers (SETs). The study seeks to show how SETs are being received in a real-life context rather than an experimental context. The research selected six university teachers from a recently incorporated Japanese University as the sample for the study. The study was aimed at; Investigating how teachers perceive the purpose of the evaluation process. How are the questions in SETs useful in improving instructional delivery? How is the student feedback useful in instructional improvement? How SETs affect the daily classroom teaching. Whether sets are inline with teacher’s conception of teaching. The study used interviews to access the perception of teachers of the Student evaluation process. The 6 respondents were divided into three groups; full time “tenured” teachers, those on “Limited or “fixed”) term and part time teachers (Burden, 2009). In this study data was collected from the three respondents seven months after the administration of SETs. The research employed semi-structured interviews but it also allowed the researcher to proceed with the interview based on the responses of the respondents (Burden, 2009). The main weakness of this study is its selection of sample for the study. By selecting a sample of only 6 lecturers, the study is a poor representation of the whole population (Burden, 2009). On the other hand, the study selects its participants from a number of categories therefore providing more categories for use in analyzing the information. The use of face to face interviews in conducting the interview also means that the researcher was at a better position to determine the quality of the respondent’s answers. 3.4 Primary Study 4 Centra, J. A. (2002). Will teachers receive higher student evaluations by giving higher grades and less coursework is a study on whether student evaluation of their teachers can be influenced by favorable treatment by teachers. The study employed the Student Intructional Report II (SIR 11) to collect data to try and establish a correlation between the independent variables; self reported student’s expected grades and an evaluation of difficulty/workload and students evaluation of their teachers. This study sampled over 55,000 classes where the SIR II had been used between 1995 and 1999 (Centra, 2002). 68 per cent of the respondents were attending four year colleges while 32 per cent of the respondents were in two year colleges. Due to the large number of participants, the class was chosen as the unit of analysis instead of individual students. The major strength of this study is its selection of a large sample. This large sample means that the results of the research are more likely to be representative of the whole population. However, this large sampling is a disadvantage as the unit of analysis has to be coalesced at the class level and therefore the results presented are affected by this generalization. Furthermore, interviewing such a large sample of respondents is an expensive undertaking. 3.5 Primary Study 5 Barnett, C.W Matthews, H.W., and Jackson, R.A. (2003). A Comparison between Student Ratings and Faculty Self-Ratings of Instructional Effectiveness is a study that compares the two popular methods of evaluating the effectiveness of learning. The study was set in Mercer University School of Pharmacy and involved 13 lecturers who where teaching a one semester course as part of the three year Doctor of Pharmacy degree. The 13 lecturers were required to fill self evaluation forms on how they perceive their teaching effectiveness. The other sample was gathered from 113 to 128 students and filled in-class evaluation of a teacher at the end of his teaching. The following research procedure was followed while conducting the research: during the filling-in of student evaluations no faculty member was present, the evaluation forms were distributed and collected by class officers (Barnett, Matthews and Jackson, 2003). The evaluation followed the school policy where each member of faculty has to be evaluated but the filling in of evaluation forms is voluntary. The evaluation criteria for faculty members and students was the same and consisted of 11 items on specific aspects of teaching and 1 item on the overall teaching ability. The evaluation items used in this study had been in use in student evaluation for several years. Students could either respond (1=poor, 2=below average, 3=average, 4=above average, 5=excellent) (Barnett, Matthews and Jackson, 2003). On the other hand faculty self-evaluation made us of an analod scale which consisted of a four inch line with the terms “poor” and “excellent” at each end respectively (Barnett, Matthews and Jackson, 2003). The lecturers had to place a mark at a particular point in the scale as a response to each aspect of their teaching. This mark was them converted to a numerical number. To enable the linking of faculty evaluation to courses their responses were not anonymous unlike those of the students. The evaluation items used in the study included were (Barnett, Matthews and Jackson, 2003): Pace of material presentation in class. Encouraging class participation. Stimulation of thinking on the content area. Is the instructor willing to discuss content outside class? Overall ability of instructor. Handouts. Use of audio-visual in instruction delivery. Course Organization Did objectives of course represent the content delivered? Organization of the unit content. Is the instructor knowledgeable in the content area? Are exam questions related to the unit objectives? Instructor’s knowledge The main weakness of this study is that students were required to fill in the evaluation voluntarily. Thus the number of students filling the questionnaires could not be determined before their administration. However, the large sample selected for the study mean the results are more representative of the whole population. The procedure of this research and the items used in the evaluation both for students and lecturers are useful as variables on any future research on the subject of teaching effectiveness evaluation. 3.6 Primary Study 6 Shakurnia, A. H., & Karami, A. (2011). A comparison between student ratings and faculty self-ratings at School of Pharmacy in AJUMS in Iran is a descriptive cross-sectional study to investigate the effectiveness of both student rating and faculty self-rating. The study involved thirty-one member of the AJUMS school of pharmacy faculty. The tutors filled in self-evaluations for every course they handled. At the conclusion of course, the students filled another set of in-class evaluations. Similar to the Barnett, Matthews and Jackson (2003), the students and the faculty made use of the same evaluation criteria. In this study there were 14 items in the evaluation criteria. The 14 items were Shakurnia and Karami (2011). Punctuality and time awareness of tutor during class time. Does instructor highlight the purpose and content for every lesson. Is the delivery coherent and consistent to the content area (s). Are the examples comprehensive and exercise proper. Is instructor knowledgeable and capable of answering questions on the knowledge area. Outlining the content area and sticking to the course outline throughout the semester. Use of updated and extensive sources of information. Attention to class attendance. Ability to manage the class. Willingness to deliver instruction. Encouraging class participation. Motivating further studies and research Continuous assessment of learning. Social behavior and mutual respect with students. Overall ability of the instructor. Similar to Barnett, Matthews and Jackson (2003) among the items of the study was one which assessed the overall ability of an instructor. The responses were on 5-point Likert scale; score 1 represented poor while score 5 represented excellent (Shakurnia and Karami, 2011). To make the responses of the lecturer’s more anonymous each was assigned a code name and then asked to fill the evaluation questionnaire. Where a lecturer did not complete his self-evaluation or was not rated by students their responses were omitted. Data analysis was carried out using SPSS-15 program. Mean and standard deviation were used to summarize the data. Further research analysis methods included t test and ANOVA for comparing the averages and Pearson correlation coefficient. The main strength of the study is its sampling technique as it sampled an entire faculty for investigation. The validity of the lecturer’s responses was also enhanced by using codes for their names to retain anonymity. The researcher further provides items that can be used in assessing the effectiveness of teachers and their teaching methods. 3.7 Primary Study 7 Anderson, M. R., Ingram, J. M., Buford, B. J., Rosli, R., Bledsoe, M. L., & Onwuegbuzie, A. J. (2012). Doctoral Students’ Perceptions of Characteristics of Effective College Teachers: A Mixed Analysis is a study to investigate the tutor’s characteristics as perceived by doctoral students as making a teacher effective. The study sampled 205 doctoral students in a United States public university. The university was highly ranked as research Institution in the Carnegie Foundation for the Advancement of Teaching classifications. It used purposive sampling that only considered students who were pursuing a Doctoral degree. According to Anderson et al (2012) the large sample selected for the study was justified by the need to have an external generalization in the study. 3.7.1 Procedure The study identified a number of evening and fulltime doctoral classes. The researchers obtained permission of the respective tutors to survey their classes for the study. The largest group of respondents for the study were studying for a Doctorate of Philosophy (n=174) with around 59.2 per cent being female (Anderson et al, 2012). The ages of those sampled ranged between 22 and 56 years. The research gathered both qualitative and quantitative data from the students and thus the mixed sampling design was applied. Most of the students involved in the research had completed a course in statistics, mathematics or statistics. Ethical considerations for the research involved compliance with adherence to the research protocols of the University. The research employed a mixed research design where both qualitative and quantitative data was given equal weight. Both qualitative and quantitative approaches were employed in data analysis, data collection and data interpretation. 3.7.2 Instrumentation and procedure The study started by training the data collectors to ensure uniform data collection. When Data collection started, the data collectors read out a set of instructions naming the members of faculty to be assessed by the study and the intent of the study. From there the data collectors distributed informed consent forms and questionnaires to the participants. Each participant was asked to write down six characteristics of an effective college tutor. Secondly, they were asked to give reasons for stating the six characteristics. Each respondents was given 15 to 30 minutes to fill the questionnaire, the whole data collection phase took 5 days 4.0 Research Methodology The research will be conducted within the descriptive research paradigm. The descriptive research paradigm was chosen for the study as it would make it possible to make descriptive comparisons between student’s evaluation and self –evaluation. 4. 1 Research Setting The research will be set in Central Queensland University Rockhampton campus 4.2 Research Sampling The research will survey 15 lecturers drawn from the University’s Faculty of Eductation and 80 students pursuing a bachelor of education degree. The selection of the sample for the study will employ the purposive sampling technique. Purposive sampling is applied to make sure the research obtains the desired information for the study (Anderson et al, 2012). The purposive sampling was preferred as the design of the study meant University students were the only ones who could provide the right information (Greimel-Fuhrmann and Geyer, 2003). The size of the sample chosen was not too small to make sure it was representation of the whole population, while making it any larger would be too expensive. 4.3 Instrumentation The questionnaire will be the main instrument of data collection employed in this study. Similar to Shakurnia and Karami (2011) the same questionnaire will be employed for both Faculty members and the students. The questionnaire will access teaching effectiveness based on the following 12 items: Use of appropriate teaching method. Encouraging class participation. Stimulation of thinking on the content area. Is the delivery coherent and consistent to the content area(s). Continuous assessment of learning. Social behavior and mutual respect with students. Handouts. Use of audio-visual in instruction delivery. Course Organization Did objectives of course represent the content delivered. Overall ability of the instructor. Is instructor knowledgeable and capable of answering questions on the knowledge area. Outlining the content area and sticking to the course outline throughout the semester. The responses will be on 5-point Likert scale; score 1 represented poor while score 5 represented excellent (Sajjad, 2011). According to Sajjad (2011) the Likert scale is easy to understand and facilitates easier analysis of the data. Anonymity of the lecturer’s responses while be enhanced by using codes instead of writing their names on the questionnaire as applied in Shakurnia and Karami (2011). The questionnaire were a student has to respond to aspects of class teaching had been widely applied in evaluating the effectiveness of teaching. 4.4 Research procedure 4.4.1 Ethical issue One of the ethical issue facing this research is the wish of respondents to remain anonymous. 4.4.2 Data Collection: Students Step 1: The research will begin by training those who will engaged in data collection. Step 2: Obtain the permission of tutors to sample their classes. Step 3: Read out research instructions detailing the members of the faculty to be evaluated by the study. Step 4: Distribute Consent forms and questionnaires. Step 5: Give each Student 10 to 15 minutes to fill up the questionnaires (Anderson et al, 2012). Step 6: Collect the questionnaires Step 8: Record the information collected. Step 9: Analyze the Information 4. 5 Data Analysis Mean and standard deviation will be used to summarize the data (Field, 2009). The t test and ANOVA will be employed for comparing the results for student evaluation and faculty self-rating for comparing the averages and Pearson correlation coefficient (Shakurnia and Karami 2011). Similarly, the Shakurnia and Karami (2011) employed the mean and standard deviation to summarize data. 4.6 Research custom to be followed No member of the faculty should be present during the student evaluation 5.0 Timing and Research Budget 5.1 Budget The research will be conducted within a four week period: Week 1: Literature review and identification of research topic. Week 2: Obtaining and Awaiting Ethical approval. Week 3: Data Collection. Week 4: Data Analysis Week 5: Presentation of results and Conclusion 5. 2 Research Budget Equipment 15,000 Travel 4,500 Materials and Supplies 9,500 Consultants 2,500 Printing 1,500 Tuition Remission 3,100 $41,500 6.0 Weakness and Strength of the Study The study major weakness is its use of purposive sampling instead of random sampling. However, the study main strength is its procedure that combines methodologies that have been used in the seven previous studies reviewed for this research. 7.0 References Barnett, C.W., Matthews, H.W. & Jackson, R.A. (2003). A Comparison Between Student Ratings and Faculty Self-Ratings of Instructional Effectiveness. American Journal of Pharmaceutical Education, 67 (4): 17 Burden, P. (2009). A case study into teacher perceptions of the introduction of student evaluation of teaching surveys (SETs) in Japanese tertiary education. The Asian EFL Journal Quarterly March 2009 Volume 11, Issue, 128. Calkins, S. & Micari, M. (2010). Less-than-perfect judges: Evaluating student evaluations. The NEA Higher Education Journal, 7-22. Centra, J. A. (2003). Will teachers receive higher student evaluations by giving higher grades and less course work?. Research in Higher Education, 44(5), 495-518. Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London, England: Sage. Galbraith, C. S., Merrill, G. B., & Kline, D. M. (2012). Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses. Research in Higher Education, 53(3), 353-374. Greimel-Fuhrmann, B. & Geyer, A. (2003). Students’ evaluation of teachers and instructional quality –Anaysis of relevant factors based on empirical evaluation. Assessment and Evaluation in Higher Education 28, 229-238. Onwuegbuzie, A. J., Witcher, A. E., Collins, K. M., Filer, J. D., Wiedmaier, C. D., & Moore, C. W. (2007). Students’ perceptions of characteristics of effective college teachers: A validity study of a teaching evaluation form using a mixed-methods analysis. American Educational Research Journal, 44(1), 113-160. Ory, J. C. (2001). Faculty thoughts and concerns about student ratings. In K. G. Lewis (Ed.),Techniques and strategies for interpreting student evaluations. New Directions for Teaching and Learning, no. 87. (pp.3-15) San Francisco, Ca: Jossey Bass.23. Ory, J. C., & Ryan, K.(2001). How does Student Ratings Measure up to a New Validity Framework? In M. Theall, P. Abrami, and L. Mets (eds.), The Student Ratings Debate: Are they Valid? How can we best Use Them? New Directions for Institutional Research, no. 109, San Francisco: Jossey Bass. Sajjad, S. (2011). Effective teaching methods at higher education level was a study on the effectiveness of teaching methods. Available from: http://www. wfate. org, Shakurnia, A. H., & Karami, A. (2011). A comparison between student ratings and faculty self-ratings at School of Pharmacy in AJUMS in Iran. Educational Research (ISSN: 2141-5161), 2(10), 1589-1594. Spiller, D., & Ferguson, P. B. (2011). Student evaluations: do lecturers value them and use them to engage with student learning needs?. Salford: Salford University. Theall, M., & Franklin, J. L. (2010). Assessing teaching practices and effectiveness for formative purposes. A guide to faculty development, 151-168. Read More