Personnel Selection and Assessment - Research Paper Example

Personnel Selection and Assessment October 2009 Intake Module Assignment 2 Evaluating Selection Methods When evaluating a given selection method, a number topics need to be considered: reliability; validity; utility, fairness (APA, 2010). The following will describe each of these aspects, as well as how they apply to the evaluation of a selection method. Reliability pertains to the quality of a measure. It is the consistency, repeatability, or accuracy of a measure. Therefore, a reliable test is one which can be consistently interpreted across different situations. The most straightforward way of assessing the reliability of a measure is to test the same group of people twice, and reliable measure will be seen to produce similar scores at both points in time. There are a number of ways to test reliability. One test of reliability is called test-retest reliability, and is designed to test things that are stable across time, such as intelligence (Field, 2009). Inter-rater reliability is assessed by having two or more independent observers or raters. The scores are then compared to determine the consistency between the raters. This method is preferred for something less stable, such a mood or productivity or interview performance. The degree of inter-rater reliability can be expressed as a reliability coefficient, or as a percentage of agreement between the raters’ data sets (Mertens, 2009). Intra-rater reliability is similar but differs in that the comparisons the consistency between sets of data collected by a single rater while observing the same performance. Another test of reliability is called parallel-forms reliability testing (Trochim, 2006). To execute parallel-forms reliability testing researchers generate a number of questions that address a specific construct. Then, they randomly divide them into multiple sets and administer both sets of questions to the same group of people. The statistical correlation between sets is an indication of reliability of the measure. The MMPI is calibrated this way. Finally, internal consistency reliability refers to the correlating of similar items within a single test. Since similar items are supposed to test similar things, the test taker should respond similarly to both questions, which is an indication of internal consistency reliability. According to Field (2009) when testing for validity, one is effectively testing for ‘evidence that a study allows correct inferences about the question it was aimed to answer or that at a test measures what it set out to measure conceptually’ (p. 795). There are three main types of validity. Construct validity refers to whether a test correlates to the theorised construct that it purports to measure. For example, an intelligence test is a test which requires strong evidence for construct validity, since researchers need to be sensitive to the extent to which they are measuring intelligence rather than language or culture differences (Mertens, 2009). A second type of validity is content validity, which is relevant to test which aim to evaluate a specific body of knowledge. For example, a depression scale may lack content validity if it does not represent the entire range of possible items, such as the behavioural dimension. Finally, criterion-related validity is a measure of how well a test is effective at predicting of a construct. This type of validity testing is divided into two types, namely concurrent and predictive validity, where concurrent validity is assessed by correlating a test with another test that has been previously validated. These two tests are taken at the same time. Predictive validity differs in that the one test is taken earlier and is meant as a predictor to a later measure (Mertens, 2009). The utility of selection methods refers to the value it has on the process of selection. Companies making use of measures which do not directly address the specified needs within the organisation can stand to lose a great deal of money. It is therefore important to determine the useful or the utility of the chosen selection method. The evaluation of the utility of a method of selection is done so with the intention to ensure that the test will serve the information needs of its users. Mertens (2009) outlines a set of utility standards which can be summarised as follows: 1. Does the method of selection address the needs of those involved or affected by the evaluation? 2. Is the person conducting the evaluation trustworthy and competent to do so? 3. Is the information collected specific and relevant to the client needs? 4. Are the perspectives, procedures, and rational used in the interpretation of the findings clear and carefully described? 5. Does the evaluation report provide essential information that is easily understood? It is noteworthy to mention that there may be no direct correlation between validity and utility. For example, a measure may be quite valid in terms of its construct but have little value in its practical application as a measure for personnel selection. Fairness is an important concern in all aspect of testing. According to (Kunnan, 2007), ‘if a test is not fair there is very little value in a test having qualities such as validity and reliability of test scores’ (p. 40). Tests have historically been viewed as beneficial to society, since it offers a level for equal opportunity and confronts the system of privilege. Nevertheless, tests may not always succeed in doing so. With respect to intelligence testing, cultural differences often confound observations. However, testing has became increasingly popular in the late 19th century and early 20th century, and with this popularity there has been growing interest in the fairness of tests (Kunnan, 2007). Fairness studies have focused on issues such as test bias and the impact of subgroup differences on expected organisational productivity as well as on subgroup hiring rates (Gilliland, 1993). Furthermore, just as it is important to establish psychometrically fair selection methods from ethical, business and legal perspectives, Gilliland (1993) points to the applicant’s perception of test fairness. According to the American Psychological Association (APA, 2010: par.2), ‘careful standardization of tests and administration conditions helps to ensure that all test takers are given a comparable opportunity to demonstrate what they know and how they can perform in the area being tested’. The APA (2010) declare that testers are required to use tests that are undiscriminating in terms of race, age, gender, disability, ethnicity, national origin, religion, sexual orientation, and linguistic background. In order to determine the fairness of a selection measure, it should meet the standards outlined here. Intelligence Testing in Personnel Selection Since its inception, intelligence tests have prompted a great deal of controversy and debate. Some schools of thought, including radical behaviourism, interpretism, and social constructivism, believe there is ‘little value in including intelligence as a basis for describing the human condition’ (Gottfredson & Saklofske, 2009, p. 185). Critics of intelligence tests have claimed that they, among other things, measure nothing more than test-taking skills, that they are bias against certain racial and socio-economic groups, that they have low predictive abilities and that they are developed and promoted by those in power in order to maintain the ‘status quo’ (Snyderman & Rothman, 1987). Despite these denunciations, a comprehensive survey of expert opinion on intelligence and aptitude testing found that experts, in areas of intelligence testing, hold an overall positive attitude to the validity and usefulness of intelligence tests (Snyderman & Rothman, 1987). These researchers reported that experts believe that intelligence tests measure the most important aspects of intelligence and that these measures are important to our society. However, this survey also found that most experts maintain that intelligence tests are to a large extent racially and socio-economically biased, and as such, experts generally favour their decreased use in employment. Despite these shortcomings, some intelligence measures do accurately describe the certain intellectual properties of many people. In the face of this, tests of intelligence continue to be one of the most widely used measures for educational and occupational as well as for clinical diagnosis (Richardson, 2002). The following paper aims to address the debates present in the literature regarding intelligence testing, paying particular attention to the reliability, validity, utility and fairness as a selection method. The discussion will commence with an examination of the definition of intelligence, looking at past and present debates. This will be followed by a discussion on what intelligence tests actually, and how useful the results are for selection purpose. Finally, this paper will consider test fairness, which appears to be the point of most contention and concern. The search for a definition of intelligence requires far more attention that can be given in this brief discussion of the subject. Answering this question requires a description of an empirical phenomenon, one which necessitates a cross discipline analysis in order to explain the ‘roots, meaning, and social consequences of human cognition diversity’ (Gottfredson & Saklofske, 2009, p.187). The first attempt to estimate intelligence was made by French Psychologist, Alfred Binet in 1905. Following extensive observations of children, analysing the kind of knowledge and skills they were required to learn, Binet devised a measure of intelligence based on short memory tasks, verbal and mathematical reasoning, sentence construction and further multiples of school type tasks (Richardson, 2002). INTELLEGENCE Since this initial estimate of intelligence, there have been a number of attempts to arrive at an agreeable definition. In the previously mentioned survey of expert opinions on intelligence and aptitude tests, researchers, Snyderman and Rothman (1987), identified the following elements of intelligence for which there were good for some, yet only moderate for others, indicators of agreement. These included: abstract thinking and reasoning; the capacity to acquire new knowledge; problem solving ability; adaptability to one’s environment; creativity; memory; mental speed; linguistic competence; achievement motivation; and goal directedness. As such, there is no consensus as to what constitutes intelligence in general. Nevertheless, one of the more applicable definitions with respect to the workplace is that of emotional intelligence. At the absence of a general consensus, it begs the question, what are intelligence tests, and what are they testing? What exactly so intelligence tests test? There is consensus in the literature that intelligence tests are both highly reliable and valid tool for measuring what it set out to measure. It has been argued that intelligence tests do not measure intellectual ability, but rather they predict performance simply because it reflects a person’s social privilege. Other definitions have been proposed. Fluid intelligence is the capacity to think logically and solve problems in novel situations, independent of previous knowledge (Cattell, 1971). Crystallized intelligence is the ability to use skills, knowledge, and experience. The general intelligence factor (abbreviated g) is a construct used to quantify what is common to the scores of all intelligence tests (Carroll, 1993). Instead of viewing intelligence as a single, general ability, primary abilities theory focused on the seven different aspects of verbal comprehension, reasoning, perceptual speed, numerical ability, word fluency, associative memory, and spatial visualization (Thurstone, 1938). Following on from this line of reasoning, the only independent value of tests is that which we randomly assign to it (Gottfredson & Salkofske, 2009). It has further been questioned whether ‘overall intelligence is just the summation of independent or semi-independent abilities that we might choose to add, or not, to the IQ measurement pot’ (Gottfredson & Salkofske, 2009, p. 187). Such questioning and debate has moved the inquiry into intelligence and intelligence testing away from the psychometric level, and towards questions regarding developmental course, physiological correlates, genetic roots and social consequences of human variation in intelligence. According to Richardson (2002), variances in intelligence testing are not primarily cognitive but a result of sociocognitive-affective factors which determine an individual’s preparedness for the demands of the intelligence test. Richardson (2002) identifies the following factors: ‘(a) the extent to which people of different social classes and culture have acquired a specific form of intelligence (or forms of knowledge and reasoning); (b) related variation in ‘academic orientation’ and ‘self efficacy beliefs’; and (c) related variation in test anxiety, self-confidence, and so on, which affect performance in testing situations irrespective of actual ability’ (p. 288). Cognitive Preparedness Theories of socio-history and ecology very cognitive abilities as evolving systems which are required to adapt to unpredictable events and environmental changes, as such cognitive processes can be viewed as being knowledge-based and structured through the employment of cultural tools (Richardson, 2002). Following on from this viewpoint, Richardson (2002) argues that intelligence tests do not assess ability for complex cognition, but rather screen for a particular set of cultural tools. Nevertheless, if a tool can effectively predict the performance of a job candidate, does it really matter that it is not testing some abstract concept? Affective Preparedness Intelligence tests are constructed on the assumption that test takers operate in a social and effective vacuum, however, there are number of human complexities which make a person more or less well prepared for a specific test situation. Richardson (2002) points, for example, to the role of self-efficacy in test performance and indicates that such factors are strongly linked to social class. It can thus been appreciated that variance in test results may be linked to affective preparedness. Emotional intelligence has been associated witth higher task performance and organizational citizenship (Petrides, Frederickson, & Furnham, 2005). Performance Preparedness Performance preparedness relates to affective preparedness in that lowered affective preparedness impacts cognitive engagement and confidence (Richardson, 2002). In responses to these critical observations, there have been noticeable changes in recent years, as intelligence testing has been moving away from intellectual classification and become more integrated into describing the multifaceted person. The study of intelligence now involves most areas of psychology, as well as sciences such as neurobiology and behaviour genetics, which has further expanded the view of the person and intelligence testing (Gottfredson & Salkofske, 2009). The outcome has been a recent focus on emotional intelligence measures as predictors of job performance. Intelligence Test Utility in Personnel Selection Intelligence tests are widely used measures for selection and placement; this is mainly because they provide a large body of useful information for decision makers, in a valid and feasible way (Gottfredson & Salkofske, 2009). However, as noted by Richardson (2002) the correlation between intelligence tests and job performance is difficult to establish due to ‘uncertain suitability of tests, poor reliability of performance ratings, contradictory findings, and...selectivity of reporting and citation’ (p. 302). Still, uncertainty is the nature of behavioural science especially with respect to its predictive ability. An early study by Ghiselli and Brown (1948), which examined the effectiveness of intelligence tests in the selection of workers, these researchers concluded the validity of intelligence tests for selection across eight occupational groups. Intelligence tests were found to be useful for selection of clerical workers; however, they were found to be of little use for selection of sales clerks and unskilled workers. There was some good evidence for use in the selection of supervisors, salesmen, and skilled workers, however they pointed to the needed for further data in this domain. Intelligence measures were found to be of no use in the selection of semi-skilled workers, and called for further research into its utility in the selection of protective services. Researchers Cote and Miners (2006) offer a compensatory model of emotional intelligence that posits job performance becomes better as cognitive intelligence decreases. Studies by Neisser et al. (1996) and Wagner (1997) report low correlations between intelligence testing and job performance. Equally, Hunlin, Henry and Noon (1990) found that once an employee has been in a job for a period of time, and performance anxiety decreases and confidence has increased, there is no relation between performance and intelligence testing. Fairness Since the being of intelligence tests there has been concern from both the public as well as test developers that tests test people fairly. Jensen (1980) argues that intelligence tests should be renamed cultural background tests because instead of testing intelligence, they test your culturally embedded cognitive alertness. According to Jensen (1980) and Richardson (2002), intelligence tests are built in a vacuum of a particular society and culture, biasing those from other societies. A test is bias when it understates the ability of the test taker. Intelligence tests have been found to bias most strongly against women and ethnic minorities, calling into question the ethical and practical of intelligence tests for selection procedures. Still, clear correlations exist between test scores and job performance. It may be important here to note the difference between bias and fairness, as the concepts may become blurred with their overlapping nature. A fair test refers to how a test is used, whereas a biased test is one in which there is systematic differences in meaning of test scores in relationship to specific groups. A test can be fair and biased, depending on how the information obtained from the test is used (Gottfredson & Salkofske, 2009). For example, in terms of racial differences in intelligence scores, making use of a racial norm would then make the test fair, despite its bias. However, Gottfredson and Salkofske (2009) point to the debate as to whether or not the use of such is fair. Snyderman and Rothman (1987) also noted the concern amongst experts with regards to fairness in intelligence testing. Summary Discussion Intelligence testing has a long and controversial history, and has been the subject of numerous studies. This paper has identified a number of key points. The main controversy surrounding intelligence testing has been in identifying a definition of intelligence, and determining what is actually being tested. The literature, it is suggested that intelligence tests are merely tools which are specifically designed to examine latent traits (Gottfredson & Salkofske, 2009). Intelligence paradigms have been moving away from the psychometrics toward explaining test variances and issues regarding preparedness which questions the ability of a test to filter through sociocognitive-affective factors (Richardson, 2002). Notably, Gottfredson and Salkofske (2009) suggest that tests claiming to be without bias and providing an equal base need to take further steps to account for perhaps obscure test biases. Finally, intelligence testing does not appear to be a reliable predictor of job performance (Hulin, Henry & Noon, 1990; Neisser et al., 1996; Wagner, 1997). Finding suggests intelligence tests maybe the single best predictors of job performance we have. Despite advances in testing and the viewing of the ‘whole person’, it does not present as a reliable, fair, useful test for personnel selection. Reference American Psychological Association. (2010). Code of Fair Testing. Accessed on the 28th of January 2010, from www.apa.org. Carroll, J.B. (1993). Human Cognitive Abilities. Cambridge University Press. Cattell, R. B. (1971). Abilities: Their structure, growth, and action. New York: Houghton Mifflin. Cote, S. and Miners, C.T.H. (2006). "Emotional intelligence, cognitive intelligence and job performance. Administrative Science Quarterly, 51(1), pp1-28. Field, A. (2009). Discovering Statistics using SPSS, third ed. London: Sage. Gilliland, S.W. (1993).The perceived fairness of selection systems: An organisational justice perspective. The Academy of Management Review, 18(4), 694-734. Gottfredson, L. & Salkofske, D.H. (2009). Intelligence: Foundations and issues in assessment. Canadian Psychology, 50(3), 183-195. Jensen, A.R. (1998). The g-factor. CT: Praeger. Kunnan, A. J. (2000). (Ed.) Fairness and Validation in Language Assessment. Cambridge, U.K.: CUP. Merten, D. M. (2009). Research and Evaluation in Education and Psychology: Integrating diversity with quantitative, qualitative, and mixed methods (2nd ed.). London: Sage. Neisser, U., Boodoo, G., Bouchard, T.J., Boykin, A.W. Brody, N., Ceci, S.J., Halpern, D.F., Loehlin, J.C., Perloff, R., Sternberg, R.J. & Urbina, S. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101. Pérez, J.C., Petrides, K.V., & Furnham, A. (2005). Measuring trait emotional intelligence. In R. Schulze and R.D. Roberts (Eds.), International Handbook of Emotional Intelligence (pp.181-201). Cambridge, MA: Hogrefe & Huber. Popham, J. (1991). Why standardized tests don’t measure educational quality. Educational Leadership, 56(6), 8–15. Richardson, K. (2002). What IQ tests test. Theory Psychology, 12(3), 283-314. Sternberg, R.J. & Berg, (1986). Quantitative integration: Definitions of intelligence: A comparison of1921 and 1986 symposia. In D.K. Detterman and R.J. Sternberg (Eds.). What is Intelligence? Contemporary viewpoints on its nature and definition. NJ: Ablex. Synderman, M. & Rothman, S. (1987). Survey of expert opinion on intelligence and aptitude testing. American Psychologist, 42(2), 137-144. Thurstone, L.L. (1938). Primary mental abilities. Chicago: University of Chicago Press. Trochim, W. (2006). Types of Reliability. Accessed on 28th of January 2010, from www.socialresearchmethods.net. Read More