Effect of Leadership Development Program on Organizational Performance - Dissertation Example

?Answers to queries Contents Reliability and Validity 3 Reliability 3 Estimating Reliability 3 Different Methods for Assessing Reliability 3 Test-Retest Reliability 3 Inter-rater Reliability 3 Internal Consistency Reliability 4 Estimating the Validity of a Measure 4 Estimating Validity 4 Convergent and Discriminant Validity 5 Criterion-Related Validity 5 Continuity correction 7 Applications 7 Likelihood ratio 8 Fisher's exact test of independence 10 When to use it 10 Null hypothesis 10 How it works 10 Examples 11 Graphing the results 12 Similar tests 12 Basic concepts of hypothesis testing 13 Null hypothesis 14 Biological vs. statistical null hypotheses 15 Testing the null hypothesis 15 P-values 16 Significance levels 17 One-tailed vs. two-tailed probabilities 17 Reporting your results 18 Confidence interval 20 p=0.891 represents the probability of occurring the event. Reliability and Validity Reliability The reliability of a measure is an inverse function of measurement error: The more error, the less reliable the measure Reliable measures provide consistent measurement from occasion to occasion Estimating Reliability Reliability can range from 0 to 1.0 When a reliability coefficient equals 0, the scores reflect nothing but measurement error Rule of Thumb: measures with reliability coefficients of 70% or greater have acceptable reliability Different Methods for Assessing Reliability Test-Retest Reliability Inter-rater Reliability Internal Consistency Reliability Test-Retest Reliability Test-retest reliability refers to the consistency of participant’s responses over time (usually a few weeks, why?) Assumes the characteristic being measured is stable over time—not expected to change between test and retest Inter-rater Reliability If a measurement involves behavioral ratings by an observer/rater, we would expect consistency among raters for a reliable measure Best to use at least 2 independent raters, ‘blind’ to the ratings of other observers Precise operational definitions and well-trained observers improve inter-rater reliability Internal Consistency Reliability Relevant for measures that consist of more than 1 item (e.g., total scores on scales, or when several behavioral observations are used to obtain a single score) Internal consistency refers to inter-item reliability, and assesses the degree of consistency among the items in a scale, or the different observations used to derive a score Want to be sure that all the items (or observations) are measuring the same construct Estimates of Internal Consistency Item-total score consistency Split-half reliability: randomly divide items into 2 subsets and examine the consistency in total scores across the 2 subsets (any drawbacks?) Cronbach’s Alpha: conceptually, it is the average consistency across all possible split-half reliabilities Cronbach’s Alpha can be directly computed from data Estimating the Validity of a Measure A good measure must not only be reliable, but also valid A valid measure measures what it is intended to measure Validity is not a property of a measure, but an indication of the extent to which an assessment measures a particular construct in a particular context—thus a measure may be valid for one purpose but not another A measure cannot be valid unless it is reliable, but a reliable measure may not be valid Estimating Validity Like reliability, validity is not absolute Validity is the degree to which variability (individual differences) in participant’s scores on a particular measure, reflect individual differences in the characteristic or construct we want to measure Three types of measurement validity: Face Validity Construct Validity Criterion Validity Face Validity Face validity refers to the extent to which a measure ‘appears’ to measure what it is supposed to measure Not statistical—involves the judgment of the researcher (and the participants) A measure has face validity—’if people think it does’ Just because a measure has face validity does not ensure that it is a valid measure (and measures lacking face validity can be valid) Construct Validity Most scientific investigations involve hypothetical constructs—entities that cannot be directly observed but are inferred from empirical evidence (e.g., intelligence) Construct validity is assessed by studying the relationships between the measure of a construct and scores on measures of other constructs We assess construct validity by seeing whether a particular measure relates as it should to other measures Self-Esteem Example Scores on a measure of self-esteem should be positively related to measures of confidence and optimism But, negatively related to measures of insecurity and anxiety Convergent and Discriminant Validity To have construct validity, a measure should both: Correlate with other measures that it should be related to (convergent validity) And, not correlate with measures that it should not correlate with (discriminant validity) Criterion-Related Validity Refers to the extent to which a measure distinguishes participants on the basis of a particular behavioral criterion The Scholastic Aptitude Test (SAT) is valid to the extent that it distinguishes between students that do well in college versus those that do not A valid measure of marital conflict should correlate with behavioral observations (e.g., number of fights) A valid measure of depressive symptoms should distinguish between subjects in treatment for depression and those who are not in treatment SAT Example High school seniors who score high on the the SAT are better prepared for college than low scorers (concurrent validity) Probably of greater interest to college admissions administrators, SAT scores predict academic performance four years later (predictive validity) Continuity correction In probability theory, if a random variable X has a binomial distribution with parameters n and p, i.e., X is distributed as the number of "successes" in n independent Bernoulli trials with probability p of success on each trial, then for any x ? {0, 1, 2, ... n}. If np and n(1 ? p) are large (sometimes taken to mean ? 5), then the probability above is fairly well approximated by where Y is a normally distributed random variable with the same expected value and the same variance as X, i.e., E(Y) = np and var(Y) = np(1 ? p). This addition of 1/2 to x is a continuity correction. A continuity correction can also be applied when other discrete distributions supported on the integers are approximated by the normal distribution. For example, if X has a Poisson distribution with expected value ? then the variance of X is also ?, and if Y is normally distributed with expectation and variance both ?. Applications Before the ready availability of statistical software having the ability to evaluate probability distribution functions accurately, continuity corrections played an important role in the practical application ofstatistical tests in which the test statistic has a discrete distribution: it was a special importance for manual calculations. A particular example of this is the binomial test, involving the binomial distribution, as in checking whether a coin is fair. Where extreme accuracy is not necessary, computer calculations for some ranges of parameters may still rely on using continuity corrections to improve accuracy while retaining simplicity. Source: Wikipedia Likelihood ratio Conditional probabilities can assist when estimating the probability that evidence came from an identified source. The probability estimate is based on calculation of a likelihood ratio (LR) .03 The likelihood ratio is the ratio of two probabilities of the same event under different hypotheses. Thus for events A and B, the probability of A given that B is true (hypothesis #1), divided by the probability of event A given that B is false (hypothesis #2) gives a likelihood ratio. The likelihood ratio is a ratio of probabilities, and can take a value between zero and infinity.02 The higher the ratio, the more likely it is that the first hypothesis is true. In forensic biology, likelihood ratios are usually constructed with the numerator being the probability of the evidence if the identified person is the source of the evidence, and the denominator being probability of the evidence if an unidentified person is the source of the evidence. The results can be interpreted as follows: LR < 1 — the genetic evidence has more support from the denominator hypothesis LR = 1 — the genetic evidence has equal support from both numerator and denominator hypotheses LR > 1 — the genetic evidence has more support from the numerator hypothesis16 These likelihood ratios can be translated into verbal equivalents that depict, in a relative way, the strength of the particular likelihood ratio in consideration.16,03 These verbal equivalents, however, are only a guide.17 Table of Verbal Equivalents Limited evidence to support LR 10000 The following equation can be used to determine the probability of the evidence given that a presumed individual is the contributor rather than a random individual in the population. LR  =  P(E/H1) / P(E/H0) P(E/H1) is the probability of the evidence given a presumed individual is the contributor. P(E/H0) is the probability of the evidence given the presumed individual is not the contributor of the evidence. In the case of a single source sample, the hypothesis for the numerator (the suspect is the source of the DNA) is a given, and thus reduces to 1. This reduces to: LR  =  1/ P(E/H0) which is simply 1/P, where P is the genotype frequency. The use of the likelihood ratio for single source samples is simply another way of stating the probability of the genotype and, while stated differently, is the same as the random match probability approach. Source: http://www.nfstc.org/pdi/Subject07/pdi_s07_m02_05_a.htm Fisher's exact test of independence When to use it Fisher's exact test is used when you have two nominal variables. A data set like this is often called an "R?C table," where R is the number of rows and C is the number of columns. Fisher's exact test is more accurate than the chi-squared test or G-test of independence when the expected numbers are small. See the web page on small sample sizes for further discussion. The most common use of Fisher's exact test is for 2?2 tables, so that's mostly what I'll describe here. You can do Fisher's exact test for greater than two rows and columns. Null hypothesis The null hypothesis is that the relative proportions of one variable are independent of the second variable. For example, if you counted the number of male and female mice in two barns, the null hypothesis would be that the proportion of male mice is the same in the two barns. How it works The hypogeometric distribution is used to calculate the probability of getting the observed data, and all data sets with more extreme deviations, under the null hypothesis that the proportions are the same. For example, if one barn has 3 male and 7 female mice, and the other barn has 15 male and 5 female mice, the probability of getting 3 males in the first barn and 15 males in the second, or 2 and 16, or 1 and 17, or 0 and 18, is calculated. For the usual two-tailed test, the probability of getting deviations as extreme as the observed, but in the opposite direction, is also calculated. This is an exact calculation of the probability; unlike most statistical tests, there is no intermediate step of calculating a test statistic whose probability is approximately known. When there are more than two rows or columns, you have to decide how you measure deviations from the null expectation, so you can tell what data sets would be more extreme than the observed. The usual method is to calculate the chi-square statistic (formally, it's the Pearson chi-square statistic) for each possible set of numbers, and those with chi-square values equal to or greater than the observed data are considered as extreme as the observed data. (Note—Fisher's exact test assumes that the row and column totals are fixed. An example would be putting 12 female hermit crabs and 9 male hermit crabs in an aquarium with 7 red snail shells and 14 blue snail shells, then counting how many crabs of each sex chose each color. The total number of female crabs is fixed at 12, and the total number of male crabs, red shells, and blue shells are also fixed. There are few biological experiments where this assumption is true. In the much more common design, the row totals and/or column totals are free to vary. For example, if you took a sample of mice from two barns and counted the number of males and females, you wouldn't know the total number of male mice before doing the experiment; it would be free to vary. In this case, the Fisher's exact test is not, strictly speaking, exact. It is still considered to be more accurate than the chi-square or G-test, and you should feel comfortable using it for any test of independence with small numbers.) Examples McDonald and Kreitman (1991) sequenced the alcohol dehydrogenase gene in several individuals of three species of Drosophila. Varying sites were classified as synonymous (the nucleotide variation does not change an amino acid) or amino acid replacements, and they were also classified as polymorphic (varying within a species) or fixed differences between species. The two nominal variables are thus synonymicity ("synonymous" or "replacement") and fixity ("polymorphic" or "fixed"). In the absence of natural selection, the ratio of synonymous to replacement sites should be the same for polymorphisms and fixed differences. There were 43 synonymous polymorphisms, 2 replacement polymorphisms, 17 synonymous fixed differences, and 7 replacement fixed differences.   synonymous replacement polymorphisms 43 2 fixed 17 7 The result is P=0.0067, indicating that the null hypothesis can be rejected; there is a significant difference in synonymous/replacement ratio between polymorphisms and fixed differences. Eastern chipmunk, Tamias striatus. The eastern chipmunk trills when pursued by a predator, possibly to warn other chipmunks. Burke da Silva et al. (2002) released chipmunks either 10 or 100 meters from their home burrow, then chased them (to simulate predator pursuit). Out of 24 female chipmunks released 10 m from their burrow, 16 trilled and 8 did not trill. When released 100 m from their burrow, only 3 female chipmunks trilled, while 18 did not trill. Applying Fisher's exact test, the proportion of chipmunks trilling is signficantly higher (P=0.0007) when they are closer to their burrow. Descamps et al. (2009) tagged 50 king penguins (Aptenodytes patagonicus) in each of three nesting areas (lower, middle, and upper) on Possession Island in the Crozet Archipelago, then counted the number that were still alive a year later. Seven penguins had died in the lower area, six had died in the middle area, and only one had died in the uppper area. Descamps et al. analyzed the data with a G-test of independence, yielding a significant (P=0.048) difference in survival among the areas; however, analyzing the data with Fisher's exact test yields a non-significant (P=0.090) result. Custer and Galli (2002) flew a light plane to follow great blue herons (Ardea herodias) and great egrets (Casmerodius albus) from their resting site to their first feeding site at Peltier Lake, Minnesota, and recorded the type of substrate each bird landed on.   Heron Egret Vegetation 15 8 Shoreline 20 5 Water 14 7 Structures 6 1 Fisher's exact test yields P=0.54, so there is no evidence that the two species of birds use the substrates in different proportions. Graphing the results You plot the results of Fisher's exact test the same way would any other test of independence. Similar tests The chi-squared test of independence or the G-test of independence may be used on the same kind of data as Fisher's exact test. When some of the expected values are small, Fisher's exact test is more accurate than the chi-squared or G-test of independence. If all of the expected values are very large, Fisher's exact test becomes computationally impractical; fortunately, the chi-squared or G-test will then give an accurate result. See the web page on small sample sizes for further discussion. If the number of rows, number of columns, or total sample size become too large, the program you're using may not be able to perform the calculations for Fisher's exact test in a reasonable length of time, or it may fail entirely. If Fisher's doesn't work, you can use the randomization test of independence. McNemar's test is used when the two samples are not independent, but instead are two sets of observations on the same individuals. For example, let's say you have 92 children who don't like broccoli and 77 children who like broccoli. You give then your new BroccoYumTM pills for a week, then observe that 14 of the children switched from not liking broccoli before taking the pills to liking broccoli after taking the pills. Three of the children switched in the opposite direction (from liking broccoli to not liking broccoli), and the remaining children stayed the same. The statistical null hypothesis is that the number of switchers in one direction is equal to the number of switchers in the opposite direction. McNemar's test compares the observed data to the null expectation using a goodness-of-fit test. The numbers are almost always small enough that you can make this comparison using the exact binomial test. For the example data of 14 switchers in one direction and 3 in the other direction, P=0.013. Source: http://udel.edu/~mcdonald/statfishers.html Basic concepts of hypothesis testing Null hypothesis A giant concrete chicken in Lang Con Ca, Vietnam. The null hypothesis is a statement that you want to test. In general, the null hypothesis is that things are the same as each other, or the same as a theoretical expectation. For example, if you measure the size of the feet of male and female chickens, the null hypothesis could be that the average foot size in male chickens is the same as the average foot size in female chickens. If you count the number of male and female chickens born to a set of hens, the null hypothesis could be that the ratio of males to females is equal to the theoretical expectation of a 1:1 ratio. The alternative hypothesis is that things are different from each other, or different from a theoretical expectation. For example, one alternative hypothesis would be that male chickens have a different average foot size than female chickens; another would be that the sex ratio is different from 1:1. Usually, the null hypothesis is boring and the alternative hypothesis is interesting. Finding that male chickens have bigger feet than female chickens might lead to all kinds of exciting discoveries about developmental biology, endocrine physiology, or sexual selection in chickens. Finding that male and female chickens have the same size feet wouldn't lead to anything except a boring paper in the world's most obscure chicken journal. It's therefore tempting to look for patterns in your data that support the exciting alternative hypothesis. For example, you might measure the feet of 10 male chickens and 10 female chickens and find that the mean is 0.1 mm longer for males. You're almost certain to get some difference in the means, just due to chance, so before you get all happy and start buying formal wear for the Nobel Prize ceremony, you need to ask "What's the probability of getting a difference in the means of 0.1 mm, just by chance, if the boring null hypothesis is really true?" Only when that probability is low can you reject the null hypothesis. The goal of statistical hypothesis testing is to estimate the probability of getting your observed results under the null hypothesis. Biological vs. statistical null hypotheses It is important to distinguish between biological null and alternative hypotheses and statisticalnull and alternative hypotheses. "Sexual selection by females has caused male chickens to evolve bigger feet than females" is a biological alternative hypothesis; it says something about biological processes, in this case sexual selection. "Male chickens have a different average foot size than females" is a statistical alternative hypothesis; it says something about the numbers, but nothing about what caused those numbers to be different. The biological null and alternative hypotheses are the first that you should think of, as they describe something interesting about biology; they are two possible answers to the biological question you are interested in ("What affects foot size in chickens?"). The statistical null and alternative hypotheses are statements about the data that should follow from the biological hypotheses: if sexual selection favors bigger feet in male chickens (a biological hypothesis), then the average foot size in male chickens should be larger than the average in females (a statistical hypothesis). If you reject the statistical null hypothesis, you then have to decide whether that's enough evidence that you can reject your biological null hypothesis. For example, if you don't find a significant difference in foot size between male and female chickens, you could conclude "There is no significant evidence that sexual selection has caused male chickens to have bigger feet." If you do find a statistically significant difference in foot size, that might not be enough for you to conclude that sexual selection caused the bigger feet; it might be that males eat more, or that the bigger feet are a developmental byproduct of the roosters' combs, or that males run around more and the exercise makes their feet bigger. When there are multiple biological interpretations of a statistical result, you need to think of additional experiments to test the different possibilities. Testing the null hypothesis The primary goal of a statistical test is to determine whether an observed data set is so different from what you would expect under the null hypothesis that you should reject the null hypothesis. For example, let's say you've given up on chicken feet and now are studying sex determination in chickens. For breeds of chickens that are bred to lay lots of eggs, female chicks are more valuable than male chicks, so if you could figure out a way to manipulate the sex ratio, you could make a lot of chicken farmers very happy. You've tested a treatment, and you get 25 female chicks and 23 male chicks. Anyone would look at those numbers and see that they could easily result from chance; there would be no reason to reject the null hypothesis of a 1:1 ratio of females to males. If you tried a different treatment and got 47 females and 1 male, most people would look at those numbers and see that they would be extremely unlikely to happen due to luck, if the null hypothesis were true; you would reject the null hypothesis and conclude that your treatment really changed the sex ratio. However, what if you had 31 females and 17 males? That's definitely more females than males, but is it really so unlikely to occur due to chance that you can reject the null hypothesis? To answer that, you need more than common sense, you need to calculate the probability of getting a deviation that large due to chance. P-values Probability of getting different numbers of males out of 48, if the parametric proportion of males is 0.5. In the figure above, the BINOMDIST function of Excel was used to calculate the probability of getting each possible number of males, from 0 to 48, under the null hypothesis that 0.5 are male. As you can see, the probability of getting 17 males out of 48 total chickens is about 0.015. That seems like a pretty small probability, doesn't it? However, that's the probability of getting exactly17 males. What you want to know is the probability of getting 17 or fewer males. If you were going to accept 17 males as evidence that the sex ratio was biased, you would also have accepted 16, or 15, or 14,… males as evidence for a biased sex ratio. You therefore need to add together the probabilities of all these outcomes. The probability of getting 17 or fewer males out of 48, under the null hypothesis, is 0.030. That means that if you had an infinite number of chickens, half males and half females, and you took a bunch of random samples of 48 chickens, 3.0% of the samples would have 17 or fewer males. This number, 0.030, is the P-value. It is defined as the probability of getting the observed result, or a more extreme result, if the null hypothesis is true. So "P=0.030" is a shorthand way of saying "The probability of getting 17 or fewer male chickens out of 48 total chickens, IF the null hypothesis is true that 50 percent of chickens are male, is 0.030." Significance levels Does a probability of 0.030 mean that you should reject the null hypothesis, and conclude that your treatment really caused a change in the sex ratio? The convention in most biological research is to use a significance level of 0.05. This means that if the probability value (P) is less than 0.05, you reject the null hypothesis; if P is greater than or equal to 0.05, you don't reject the null hypothesis. There is nothing mathematically magic about 0.05; people could have agreed upon 0.04, or 0.025, or 0.071 as the conventional significance level. The significance level you use depends on the costs of different kinds of errors. With a significance level of 0.05, you have a 5 percent chance of rejecting the null hypothesis, even if it is true. If you try 100 treatments on your chickens, and none of them really work, 5 percent of your experiments will give you data that are significantly different from a 1:1 sex ratio, just by chance. This is called a "Type I error," or "false positive." If there really is a deviation from the null hypothesis, and you fail to reject it, that is called a "Type II error," or "false negative." If you use a higher significance level than the conventional 0.05, such as 0.10, you will increase your chance of a false positive to 0.10 (therefore increasing your chance of an embarrassingly wrong conclusion), but you will also decrease your chance of a false negative (increasing your chance of detecting a subtle effect). If you use a lower significance level than the conventional 0.05, such as 0.01, you decrease your chance of an embarrassing false positive, but you also make it less likely that you'll detect a real deviation from the null hypothesis if there is one. You must choose your significance level before you collect the data, of course. If you choose to use a different significance level than the conventional 0.05, be prepared for some skepticism; you must be able to justify your choice. If you were screening a bunch of potential sex-ratio-changing treatments, the cost of a false positive would be the cost of a few additional tests, which would show that your initial results were a false positive. The cost of a false negative, however, would be that you would miss out on a tremendously valuable discovery. You might therefore set your significance value to 0.10 or more. On the other hand, once your sex-ratio-changing treatment is undergoing final trials before being sold to farmers, you'd want to be very confident that it really worked, not that you were just getting a false positive. Otherwise, if you sell the chicken farmers a sex-ratio treatment that turns out to not really work (it was a false positive), they'll sue the pants off of you. Therefore, you might want to set your significance level to 0.01, or even lower. One-tailed vs. two-tailed probabilities The probability that was calculated above, 0.030, is the probability of getting 17 or fewer males out of 48. It would be significant, using the conventional P Read More