Predictive Validity - Coursework Example

Harinath Babu BM Academia Research November 18, 2007 Predictive Validity In today's highly competitive world markets personnel assessment and selection is one of the key methods available to organizations to ensure that they have effective workforces (Smith & Robertson, 1993). Acquiring and retaining high quality personnel is seen as key to organizational success (Shakleton & Newell, 1997), and successful selection and assessment of staff therefore presents a significant challenge to organizations, particularly in the face of growing skill shortages (Cunrow, 1989) and forecasts of a considerable drop in the numbers of young people that will be available for work in the decade ahead (Warr, 1996). And yet, in the absence of any perfect method of assessment and selection, organizations continue to use a variety of imperfect methods in their attempts to predict which individuals will be successful in particular job roles. This paper will examine the continuing widespread preference of organizations for using assessment and selection methods with less predictive validity than others, and will consider what can be done to persuade human resources practitioners to modify their organizations' approaches towards ensuring better matches between individuals and job roles. In particular, the prevalence of the traditional employment interview as the primary selection method will be compared to the alternative methods of cognitive ability tests and biographical data. Whilst the assessment and selection process provides information for decisions by both the employer and the potential employee, this is not the traditional view as employment decisions have long been regarded as a management prerogative (Torrington & Hall, 1991). However, given the predicted skill shortages and the fact that selection is also concerned with the future life plans of individuals, the predictive validity of selection methods is an important issue (Meijer, 1998) both for organizations and for individuals. Predictive validity refers to the extent to which an assessment measure can predict subsequent job performance (Smith et al, 1993) such as error rate, production rate, appraisal scores, absence rate, or other criteria that may be important to the organisation. Relationships between assessment outcomes and future performance are expressed as correlation coefficients (r), where r = 1 represents a perfect relationship, and r = 0 signifies that no relationship whatsoever exists. A correlation of, say, r = 0.4 is regarded as comparatively good in assessment and selection (Torrington et al, 1991), but this does illustrate that there are no methods of selection that represent outstanding predictors of future performance. In their survey of management selection methods used in French and British organizations, Shackleton & Newell (1991) illustrated that, although there was an increasing use of personality assessments, cognitive ability tests, assessment centers and biographical data between the years of 1984 and 1989, the traditional method of face to face interviews continued to represent the dominant method of assessment and selection in the UK. The format of interviews may range from totally unstructured where no objectives are set, to highly structured, pre-planned processes in which topic areas are closely related to job behavior, with varying degrees of semi-structure in-between (Anderson & Shackleton, 1993). Huffcutt & Arthur (1994) used meta-analysis to demonstrate that predictive validity increased as interview structure increased; the degree of standardization of questions and response scoring mechanisms resulted in validities ranging from r = 0.20 for less structured interviews to r = 0.56 for more highly structured interviews. Nevertheless, even highly structured interviews may vary in their predictive validity. In a comparison of situational interviews (future-orientated) and behavior description interviews (past-orientated) Campion, Campion & Hudson (1994) found higher validities for past-orientated (r = 0.51) than for future-orientated (r = 0.39) interviews. Similarly, Pulakos & Schmitt (1995) compared situational versus past behavior questions in structured interview settings and reported that only the past behavior questions were valid in predicting future supervisor ratings. In spite of the evidence for the generally greater predictive validity of structured interviews, organizations still largely prefer unstructured interviews (Graves & Karen, 1996, cited in Hough & Oswald, 2000). This preference may be attributed to a general reliance on intuition to make decisions (Beach, 1990; Dawes, 1988, cited in Dipboye, 1997) and a tendency for interviewers to have faith in the validity of their own judgments (Dipboye, 1997). In addition it has been suggested that, for interviewers who value the role of interviewing and see it as an important part of their function, an unstructured interview provides greater challenge and autonomy whereas a highly structured interview could be seen as deskilling the role and reducing it to something of a monotonous exercise (Dipboye, 1997); as such, less structure is more attractive to managers because it gives greater emphasis to their authority (Torrington et al, 1991). Research has also illustrated that applicants generally prefer unstructured interviews, possibly because they allow more control over the situation (Schuler, 1993, Latham & Finnegan, 1993, cited in Dipboye, 1997). Human Resources practitioners usually play a large part in the assessment and selection process (Torrington et al, 1991) and, as such, can exercise significant influence on the design and application of selection procedures. The conventional starting point for the assessment and selection process is a thorough job analysis resulting in a job description and person specification to identify the knowledge and skills required for effective performance of the role. However, Torrington et al (1991) found that less than half of personnel departments use job analysis and its outcomes for assessment and selection purposes. Greater emphasis on these basic procedures and their impact upon the predictive validity of selection methods therefore needs to be conveyed to practitioners through their early training and continuous professional development as a means of engineering a change of attitudes in organizations towards more robust assessment and selection methods. High validity of selection techniques have been shown to produce significant financial savings for organizations (Hunter & Hunter, 1984) and it is arguable therefore that predictive validity in recruitment methods should represent a significant priority for human resources practitioners. Whilst research indicates that general managers have not been easily persuaded by utility analyses that demonstrate the financial benefits of sound selection procedures (Latham & White, cited in Dipboye, 1997), they may be more convinced by demonstrable cost savings to their own budgets. Nevertheless, the literature has clearly had some impact in changing organizational attitudes in this respect as research illustrates that there has been a steadily increasing use of cognitive ability tests (Shackleton et al, 1991) to determine, for example, verbal, numerical and spatial skills, and in the use of biographical data (Shackleton et al, 1997) in organizational selection procedures. Hunter & Hunter (1984) found that the average validity of cognitive ability tests was r = 0.45 for performance ratings and r = 0.54 for training success. Indeed Huffcutt & Arthur's (1994) review of the Hunter & Hunter (1984) work, in which they assessed interview validity in relation to that of cognitive ability tests, demonstrated that the higher levels of structure in interviews were comparable to the validities for ability tests. Furthermore, if cognitive ability tests are combined with a second predictor, the multiple correlation increases still further (Saldago, 1999). Where interview procedures have a tendency to be relatively unstructured, the use of such tests can add greater potential accuracy to selection decisions. However, many candidates feel that, whilst they can improve their prospects with a good interview "performance," they see the use of tests as representing something of a dispassionate routine which minimizes control over their destinies (Torrington et al, 1991). Whilst it has been shown that a person's ability to score highly on such tests correlates with the capacity to retain new knowledge, to pass exams and to succeed at work (Plumley, 1985), several studies have also suggested that psychometric "g" (which is generally seen as the principal component of cognitive ability test), accounts for the majority of the predictive power and that the remaining variance accounts for little or no additional variance in the criterion (Borman, Hanson & Hedge, 1997). Nevertheless, such tests clearly are strong predictors of job performance, and the predictive validity also increases in relation to job complexity (Pettersen & Tziner, 1995). But, in the light of these factors, it has been suggested that an element of "self-selection" may precede test-taking in that job seekers may select into or "gravitate" towards jobs with ability requirements matching their own cognitive ability (Wilk & Sackett, 1996, cited in Hough & Oswald, 2000). Perhaps a greater degree of self selection would be desirable in cases where large numbers of potential applicants may be anticipated, by means of providing prospective applicants with more information about the organization and the job; for example the use of "a realistic job preview" would enable candidates to assess their own suitability to a much greater extent (Torrington et al, 1991). Given that the selection process is a two-way decision making process, applicant attitudes towards an organization may be influenced by the selection practices used which may, in turn, impact upon job-choice decisions. Rynes & Connerley (1993, cited in Salgado, 1999) found that the main factor that affected applicant attitudes towards selection methods was the apparent validity and faith in the evaluation of the procedures, perception of the employer's need to know, and beliefs about likely self-performance. Kluger & Rothstein (1993, cited in Borman et al, 1997) found that business students were more comfortable with biodata than with an ability test in that they viewed themselves as having more control over their performance, thought it was fairer and less difficult, and believed that the procedure best captured who they were. Biodata methods of selection have been described as "permitting the respondent to describe himself in terms of demographic, experiential or attitudinal variables presumed or demonstrated to be related to personality structure, personal adjustment or success in social, educational or occupational pursuits" (Owens, 1976, cited in Drakeley, 1989). These are usually in the format of multiple-choice, self-report questionnaires and are based on the same principles of job analysis that apply to other selection methods to ensure job relatedness (Drakeley, 1989). In addition to "hard" information that is verifiable, for example via references or educational attainments, biodata questionnaires usually include "soft" items of a more abstract nature such as value judgments, aspirations, motivations, attitudes and expectations (Drakeley, 1989) - human attributes that are not so easily tapped with cognitive ability tests or interview methods (Lefkowitz, Gebbia, Balsam & Dunn, 1999). This method can significantly raise the reliability and validity of the selection process (Bliesner, 1996, cited in Harvey-Cook & Taffler, 2000), by making use of information about past events in an objective and systematic way to predict future job success (Drakeley, 1989). It is particularly useful at the pre-selection stage where the employer may need to reduce large numbers of common format applications to a manageable set (Harvey-Cook et al, 2000), thus helping to minimize the early rejection of candidates on the basis of subjective information or information which may not be specifically related to job performance (Drakeley, 1989). Whilst cognitive ability tests are generally accepted to provide the best indication of future performance in terms of predictive validity, biodata can be equally effective (Hunter & Hunter, 1984). Reilly & Chao (1982) also reviewed alternatives to the cognitive ability tests for predicting a range of criteria and found mean validities for biodata ranged from 0.32 for job tenure to 0.46 for productivity. Notwithstanding the fact that the findings show strong support for considering biodata as a valid predictor for personnel selection, the development of a robust biodata questionnaire is a time consuming process and the method has been criticized for its organizational-specificity - that is, biodata questionnaires are tailor-made for individual organizations (Hunter & Hunter, 1984) and therefore limited in their general use. However, there is evidence that biodata predictors of job behavior can be developed to generalize across organizations and across jobs (Wilkinson, 1997, cited in Harvey-Cook et al, 2000), which may be a particularly useful at, say, professional entry level. Like the structured interview, the same questions are asked of everyone with the biodata method and, like the cognitive test, the answers given are assessed in a consistent way (Drakeley, 1989). But in contrast to less structured interviews and to cognitive ability tests in general, biodata has generally been found to have relatively little or not adverse impact on minorities, and is generally seen by applicants as less intrusive (Lefkowitz, et al, 1999). Legislation promoting equality of opportunity has given emphasis to the importance of using well-validated selection processes which discriminate fairly and not unfairly between applicants (Torrington et al, 1991), but there is still a growing trend for unpopular or controversial selection procedures to lead to litigation against organizations. Unstructured interviews account for the majority of cases, followed by cognitive ability and physical ability tests; together they have been found to be discriminatory in around 40 percent of cases (Terpstra, 1989, cited in Hough & Oswald, 2000). Nevertheless, structured interviews do treat applicants in a consistent manner and mean differences by race have been found to be more likely reduced in structured interviews containing content related to non-cognitive constructs, especially for high-complexity jobs (Huffcutt & Roth, 1998, cited in Hough & Oswald, 2000). There is clearly no prescriptive single best method of assessment and selection but the evidence strongly suggests that the use of a combination of appropriate methods will generate greater predictive validity. Whilst the literature indicates that organization are beginning to move in this direction, the pace of change is slow, in contrast the ever increasing pace of globalization and organizational change in response to world competition (Shackleton et al, 1997). Given the competitive labor market, organizational attractiveness is of prime importance such that positive applicant reactions may significantly increase the chances of employing the best applicants, and avoid the possibility of costly litigation (Hough & Oswald, 2000). If organizations are to compete successfully for a limited pool of high quality staff, they must look at the financial costs of poor recruitment methods, recognize the selection procedure as a two-way decision procedure, and consider validity as a crucial factor in the value of their selection methods (Smith & Robertson, 1989). Organizations will undoubtedly continue with their preferred methods of assessment and selection, and the interview looks set to continue to represent the standard approach (Robertson & Makin, 1986). Perhaps the search for better predictors would be better directed towards improving those that will be used anyway (Smith & Robertson, 1989) but, in parallel, organization may be persuaded to consider additional methods to improve their success; it seems that much can be done to improve the preferred methods by means of engaging in job analysis, building more structure into the interview process, and by considering a variety of selection methods. These small modifications could do much to contribute both to an organization's reputation and to improving its success in selecting personnel. References Anderson, N.R. & Shackleton, V.J. (1993). Successful Selection Interviewing. Oxford: Blackwell. Borman, W.C., Hanson, M.A., & Hedge, J.W. (1997). Personnel Selection. Annual Review of Psychology. 48, 299 - 337. Campion, M.A., Campion, J.E., & Hudson, J.P. (1994). Structured Interviewing: A Note on Incremental Validity and Alternative Question Types. Journal of Applied Psychology. 79, 998 - 1002. Cunrow, B. (1989). Recruit, Retrain, retain; Personnel Management and the three R's. Personnel Management, November, pp. 40 - 47. Dipboye, R.L. (1997). Structured Selection Interviews: Why do They Work Why are they Underutilized: In Anderson, N. & Herriot, P. (Eds). International Handbook of Selection and Assessment. London: Wiley. Drakeley, R.J. (1989). Biographical Data. In Herriot P. (Ed). Assessment & Selection in Organizations - Methods & Practice for Recruitment & Appraisal. Chichester: Wiley. Harvey-Cook, J.E. & Taffler, R.J. (2000). Biodata in Professional, Entry Level Selection: Statistical Scoring of Common Format Applications. Journal of Occupational & Organisational Psychology. 73, 103 - 118. Hough, L.M. & Oswald, F.L. (2000). Personnel Selection: Looking Toward the Future - Remembering the Past. Annual Review of Psychology. 51, 631 - 664. Huffcutt, A.I. & Arthur, W. (1994). Hunter & Hunter (1984) Revisited: Interview Validity for Entry-Level Jobs. Journal of Applied Psychology. 79, 184 - 190. Hunter, J.E. & Hunter, R.F. (1984). Validity & Utility of Alternative Predictors of Job Performance. Psychological Bulletin. 96(1), 72 - 98. Lefkowitz, J., Gebbia, M.I., Balsam, T. & Dunn, L. (1999). Dimensions of Biodata Items and their Relationship to Item Validity. Journal of Occupational & Organisational Psychology. 72, 331 - 350. Meijer, R. (1998). Consistency of Test Behavior and Individual Difference in Precision of Prediction. Journal of Occupational and Organisational Psychology 71, 147 - 160. Pettersen, N. & Tziner, A. (1995). The Cognitive Ability Test as a Predictor of Job Performance: Is its Validity affected by Job Complexity and Tenure within the Organisation International Journal of Selection & Assessment. 3, 237 - 241. Plumbley, P.R. (1985). Recruitment & Selection (4th Edn). London: Institute of Personnel Management. Pulakos, E.D. & Schmitt, N. (1995). Experience-Based and Situational Interview Questions: Studies of Validity. Personnel Psychology. 48, 289 - 308. Reilly & Chao (1982). Validity & Fairness of Some Alternative Employee Selection Procedures. Personnel Psychology, 35, 1 - 62. Robertson, I.T. & Makin, P.J. (1986). Management Selection in Britain: A Survey & Critique. Journal of Occupational Psychology. 59, 45 - 57. Salgado, J.F. (1999). Personnel Selection Methods. In Cooper, C.L. & Robertson, I.T. (Eds). International Review of Industrial and Organisational Psychology. London: Wiley. Shackleton, V. & Newall, S. (1991). Management Selection: A Comparative Survey of Methods used in Top British and French Companies. Journal of Occupational Psychology. 64, 23 - 36. Shackleton, V. & Newall, S. (1997). International Recruitment & Selection. In. Anderson, N. & Herriot, P. (Eds). (1997). International Handbook of Selection & Assessment. Chichester: Wiley. Smith, M. & Robertson, I.T. (1993). The Theory & Practice of Systematic Personnel Selection. London: The Macmillan Press Ltd. Torrington, D. & Hall, L. (1991). Personnel Management - A New Approach (2nd Edn). Hemel Hempstead: Prentice Hall. Warr, P. (1996). Younger and Older Workers. In Warr, P. (Ed). (1996). Psychology at Work. London: Penguin. Read More