Validity and Reliability Matrix - Essay Example

of Phoenix Material Validity and Reliability Matrix For each of the tests of reliability and validity listed on the matrix, prepare a 50-100-word description of test's application and under what conditions these types of reliability would be used as well as when it would be inappropriate. Then, prepare a 50-100-word description of each test's strengths and a 50-100-word description of each test's weaknesses. TEST of Reliability Application and APPROPRIATENESS Strengths Weaknesses Internal Consistency It measures the extent to which tests or procedures assess the same characteristic, skill or quality. It is a measure of the precision between the observers or of the measuring instruments used in a study. It refers to how well all the test items relate to each other. This form of reliability is used to judge the consistency of results across items on the same test. Essentially, test items that measure the same construct are measured to determine the tests internal consistency. The multiple indicators of a property increase the measure's reliability. Rival hypotheses are ruled out because of the use of range of multiple indicators of the property being studied and measured. This test often helps researchers interpret data and predict the value of scores and the limits of the relationship among variables. It accounts for error due to content sampling, usually the largest single component of measurement error (Lawrence, 1993). A property is measured in several different ways, in which the most typical form for measurement is through questionnaire, and then measures obtained were combined into a single score, hence reliability across different parts of an instrument might be difficult to establish. The measures are specifics of a property which will eventually be collated to reflect a single information. Split-half Half of the test items (even numbered) are correlated to other half (odd numbered) to obtain reliability coefficient. This is done by randomly dividing all items that purport to measure the same construct into two sets. The entire instrument is administered to a sample of people and the total score for each randomly divided half is calculated. The split-half reliability estimate is simply the correlation between these two total scores. It only requires a single test administration. It is therefore resources-wise test. Cost and time will be used efficiently. It is limited to estimating differences on one dimension (usually the number of items, or raters). The resultant coefficient will vary as a function of how the test was split. It is also not appropriate on tests in which speed is a factor (that is, where students' scores are influenced by how many items they reached in the allotted time). Test/retest It is an index of score consistency over a brief time period, typically several weeks. It tells how much the individual's normative score is likely to change on near-term retesting. This index of score is obtained by administering the same test twice, with a certain amount of time between administrations, and then correlating the two score sets. Each subject should score different than the other subjects, but if the test is reliable then each subject should score the same in both test. The closer the results, the greater the test-retest reliability of the survey instrument. The test is easy to administer hence, it is the most popular indicator of survey reliability. The consistency of a measure from one time to another is measured and assessed. It assumes that there will be no change in the quality or construct being measured. "It is an excellent measure of score consistency because it allows the direct measurement of consistency from administration to administration" (Lawrence, R. et al, 2001) Administration of test for the second time may produce the "practice effect" - respondents "learn" to answer the same questions in the first test and this affects their responses in the next test. Score change could be caused by day-to-day fluctuation in performance, or the individual's recollection of the earlier administration. Considerably different estimates depending on the interval can be obtained. Parallel and alternate forms Two parallel forms are created. One way to accomplish this is to create a large set of questions that address the same construct and then randomly divide the questions into two sets. You administer both instruments to the same sample of people. The correlation between the two parallel forms is the estimate of reliability (Trochim, 2006). Calculating the Pearson r between the scores found by the two forms produces the alternate forms reliability coefficient. Naturally, we expect a high correlation to be found. If not, then the forms are not truly equivalent and should not be substituted for each other. The administration of two equivalent tests within a short time interval to the same group of subjects minimizes threat to internal validity. Duplication of items and comparing the results effectively ensures a reliability that the assessment is indeed valid regardless of difference in time frame, though short as it is. Alternate-form reliability coefficients provide estimates of the extent to which individuals can be expected to rank the same on alternate forms of a test Large number of items that reflect the same construct needs to be generated. Construction of the large number of items is time consuming and entails other reliability and validity tests. It is therefore cost and time bounded. Administration of the forms is done with the assumption that the randomly divided halves are parallel or equivalent. Time factor is essential. Test of Validity Application and APPROPRIATENESS Strengths Weaknesses Face validity Face validity generally defines how closely the test appears to measure what it's supposed to measure. It is concerned with how a measure or procedure appears. It further answers the following questions: does it seem like a reasonable way to gain the information the researchers are attempting to obtain Does it seem well designed Does it seem as though it will work reliably Experts are essentially required to check the validity of the constructs of the instrument, hence high quality of tool has been ensured in terms of operations of constructs involved in the assessment. As long as the experts agree on its validity to measure, it does not depend on established theories for support. The validity can only be attained by subjecting the instrument through the experts' judgment, hence it is generally subjective. The validity therefore may change depending on the consensus of the experts. This can be literally described as validity through spot-checking. It is probably the weakest way to try to demonstrate construct validity. Content validity Content validity is about whether the group of items adequately covers the domain being measured. It is concerned with how well the sample of test items represents the content the test is designed to measure. Put another way, the instrument's parts should tap the full range of different aspects of the construct. Whatever we call it, any measure of this construct should include items tapping the full range of perceptions about the outcomes of exercising that people might hold. Adequate coverage of domains being measured is required thus, researchers need to define the very domains they are attempting to study. All aspects of the domain is explored for definition and usage all throughout the instrument. Experts are essentially required to check the validity of the constructs of the instrument, hence high quality of tool has been ensured in terms of operations of constructs involved in the assessment. Unlike face validity, it depends on established theories for support. The assessment relies upon the developers of an instrument having a thorough understanding of the construct it is designed to measure and/or on previous research that has already established the domain of interest. Though dependent on established knowledge, the content validity is subjective to the experts consensus. Criterion related The performance of the operations of the instrument is set against some criterion. Criterion-related validity is concerned with the degree to which scores from an instrument correlate with scores on some relevant criterion variable. A common scenario where this is assessed is when scores on a new instrument are correlated with scores on an older instrument (Research Methods Philosophy of Science and Research Design, n. d.) . Standards are used to ensure the validity of the instrument. The validity is defined by the highly-accepted set of requirements. It allows prediction about how the operationalization will perform based on our theory of the construct. the criteria they use as the standard for judgment are not fixed. Even the standards or criteria for judgment are not measured, hence the instrument to tested therefore cannot be claimed for validity. Construct It is an assessment of how well you translated your ideas or theories into actual programs or measures. It is concerned with how well a particular test can be shown to measure a particular construct (a theoretical construction about the nature of human behavior, such as intelligence, anxiety, or creativity). Construct validity is determined by seeing how well the test distinguishes between two groups of subjects, one that exhibits a high degree of the construct and one that doesn't. a test has construct validity if it accurately measures a theoretical, non-observable construct or trait. The construct validity of a test is worked out over a period of time on the basis of an accumulation of evidence. Construct validity requires experts to examine the constructs of the instrument, hence high quality of tool has been ensured in terms of definitions of constructs involved in the assessment. As long as the experts agree on its validity to measure, it does not depend on established theories for support. Constructs are difficult to measure because they are not directly observable but they are inferred from their effect on behavior. Operational definitions might not be adequate to encompass the meaning of the construct. Definitions of the construct should be strictly specified and defined such that only a single interpretation could be generated from the subjects hence can measure a unified information in response to the construct being measured. References Lawrence, R. (1993). Test Evaluation. Retrieved from http://ericae.net/seltips.txt on April 23, 2008. Lawrence, R., Shafer, M. and William, D. (2001). Reliability. ERIC Digest. ERIC Clearinghouse on Assessment and Evaluation College Park MD. Retrieved from http://www.ericdigests.org/2002-2/reliability.htm on April 23, 2008. Research Methods Philosophy of Science and Research Design, (n. d.) Reliability and validity of measurement. Retrieved from http://www.bangor.ac.uk/pes004/resmeth/measure.htm on April 23, 2008. Trochim, W. (2006).Types of Reliability. Research Methods Knowledge Base. Online: http://www.socialresearchmethods.net/kb/reltypes.php on April 23, 2008. Read More