Evaluation of The Impact of Goals on Software Project Management Paper by Abdel-Hamid - Assignment Example

Review of the paper 2 against guidelines of paper 1 This is an evaluation of a paper by Abdel-Hamid, T.K., K. Sengupta, and C. Swett - The Impact of Goals on Software Project Management: An Experimental Investigation- against the guidelines given by Barbara A Kitchenham et al. in Preliminary Guidelines for Empirical Research in Software Engineering. Context Guidelines According to the context guidelines there are certain clauses that must be followed in the research papers. According to C1, as much industrial context must be specified as possible. The entities, attributes and measures that capture the contextual information must be clearly defined. The paper under consideration is based on exploring the impact of project goals on managerial decision making in software project environments. The study mentions that this topic has been studied in various other contexts like clerical tasks, prose learning, typing etc. that cover the importance of goal setting in specific terms in any kind of managerial decisions. According to the information available, goal setting has more effect on simple tasks but have weaker effects on complex tasks. They have identified software project management as a complex task. That is why it requires a continuous process of identifying goals and reconciling and multiple goal decision making. It is good enough context explanation. C2- - If a specific hypothesis is being tested, state it clearly prior to performing the study and discuss the theory from which it is derived, so that its implications are apparent. If a specific hypothesis is being tested, state it clearly prior to performing the study and discuss the theory from which it is derived, so that its implications are apparent. The study mentions that “it was designed to investigate the impact of different project goals on project management decision (planning and resource allocation) and project performance (cost, schedule and quality)”. It also mentions Weinberg and Schulman experiment which concluded that “programming is such a complex activity that programmers have an almost infinite number of choices in terms of how they will write a program in order to meet certain specification and if given specific objectives, programmers can make programming choices in such a way that they will meet those objectives – provided they do not conflict with other specific objectives.” It also mentions multi-goal study by Locke and Bryan(1969) C3- If the research is exploratory, state clearly and, prior to data analysis, what questions the investigation is intended to address and how it will address them. The study has framed the research question as follows:- “Do difference in Goal sets result in different planning and staff allocation strategies in the management of software projects?” The possibility of goals raises the follow-up question:-“Do differences in project planning and staff allocation strategies result in different levels of project performance (defined in terms of cost, schedule, and quality)?” For investigation it says - “The research questions were explored in the context of a simulation game in which subjects played the role of project managers running the programming phase of a software project.” C4- Describe research that is similar to, or has a bearing on, the current research and how current work relates to it. This study was an experiment done by Weinberg and Schulman in which five teams were given the same programming assignment but different directions were given to each team about what to optimize while doing the job. This study differs from Weinberg and Schulman study as it focuses on managerial planning and resource allocation. The study also mentions a study done by Abdel-Hamid and Madnick in 1991 that is about the impact of staff additions on project completion time. The study in consideration goes further to analyze other factors that affect successful completion of a project. Experimental design D1- Identify the population from which the subjects and objects are drawn. As prior research shows that project costs, schedule estimates and staff allocation decisions impact the performance dimensions, this study tries to find out whether the difference in project planning and staff allocation strategies will result in different level of project performance in terms of cost, schedule and quality. For this they have chosen a simulation game in which subjects played the role of project managers running the programming phase of a software project. D2- Define the process by which the subjects and objects were selected. The subjects and objects were selected according to a simulation game recently published in many studies like Abdel-Hamid et al 1993, 1994; and Sengupta and Abdel-Hamid 1993, 1996. The subjects were second-year masters' students in an information technology management curriculum at a U.S. university. All subjects were mid-level managers and had an average of 12.6 years work experience. D3- Define the process by which subjects and objects are assigned to treatments. In order to prepare the subjects for the simulation, a one-hour classroom lecture was given to explain and demonstrate the computer interface and explain the task. After this session, all subjects spent a second one-hour session period playing the game (using a practice project). The experiment was conducted two days later in a computer lab. Subjects took between 90 and 120 minutes to complete the experiment. After the experiment, subjects filled up a debriefing questionnaire. The questionnaire provided subjects with an opportunity to express views regarding the simulation. It also contained questions on demographics, the specifics of the subjects' project management experience (if any), the ease of use of the simulator, and the understandability of the intrusions. D4- Restrict yourself to simple study designs or, at least, to designs that are fully analyzed in the statistical literature. If you are not using a well-documented design and analysis method, you should consult a statistician to see whether yours is the most effective design for what you want to accomplish. As mentioned above the study adopts a method used many times by many other studies. So it has the support of existing study designs. D5- Define the experimental Unit The experimental unit was two groups of 13 subjects each. Since one of the subjects fell sick, one of the groups had 12 members. One group with 13 members was given the cost/schedule goal condition and another group with 12 members was given quality/Schedule goal condition. D6- For formal experiments, perform a pre-experiment or precalculation to identify or estimate the minimum required sample size. The study has mentioned the details of its sample size and also mentioned that it conducted a pilot study with seven subjects before expanding it to 26 subjects. D7- Use appropriate levels of blinding. The study does not mention any blinding method adopted for unbiased results. D8- If you cannot avoid evaluating your own work, then make explicit any vested interests (including your sources of support) and report what you have done to minimize bias. The study has no mention of any such measures taken. It does not mention the possibility of any vested interests. D9- Avoid the use of controls unless you are sure the control situation can be unambiguously defined. The study was done in a real life situation and no control was used except for dividing the subjects in two groups based on cost and quality. D10- Fully define all treatments (interventions) According to the guidelines, treatments must be properly defined if experiments are to be replicated and/or their results are to be taken up by industry. This study says that software project performance is typically defined in terms of three main outcome factors: cost, duration, and quality. The challenge of software development is represented as the "push-pull" attempt to control and manage the three major outcome factors: cost, schedule, and quality. These outcome factors are viewed as "springs" that are dynamically compressed or decompressed during the life of a project. Project schedules can also affect product quality. A tight schedule leads to schedule pressures, and people under time pressure do not necessarily work better, they just work faster. The research was based to find answers to these questions that are relevant to every organization, especially software management organizations. The findings can be applied to all these organizations as required by the guidelines. D11- Justify the choice of outcome measures in terms of their relevance to the objectives of the empirical study. The outcomes were analyzed using MANOVA or multivariate analysis of variance model of repeated measures as the study involved many variables. Data collection guidelines DC1- Define all software measures fully, including the entity, attribute, unit and counting rules. Project performance was made operational through three dependent measures: final cost (in person-days), completion time (in days), and remaining undetected defects at the end of the programming phase (in number of defects). The number of remaining undetected defects indicated the quality of the software product, i.e., fewer remaining undetected defects indicating higher quality software, and vice versa. In addition, the subjects' four decision variables—desired total staff level (in people), percent of staff allocated to QA (percent), and the updated cost and schedule estimates (in person days and days respectively)—were captured at every 40-day interval. Following the suggestion in Wofford et al. (1992), the updated cost and schedule estimates were operationalized as the delta between the subject's latest estimates and the project's initial estimates. DC2- For subjective measures, present a measure of inter-rater agreement, such as the kappa statistic or the intra-class correlation coefficient for continuous measures Although there is detailed depiction of data analysis but no such term has been mentioned. DC3- Describe any quality control method used to ensure completeness and accuracy of data collection. The only method mentioned here is announcement to the subjects that this project will be carrying 20% of their course evaluation, as they were all management students. DC4- For surveys, monitor and report the response rate and discuss the representative-ness of the responses and the impact of non-response. The sample size is representative of the real life situation in organizations. Since one of the group was lesser in strength, the result compilation had considered that also but it was found that it did not affect the result. Although there is no specific mention of taking such a measure as mentioned in the guideline but it seems that it has been taken care of. DC5- For observational studies and experiments, record data about subjects who drop out from the studies There is no such mention of any subject dropping out of the study except for one subject who could not join the study since the very beginning as he fell sick before the study could start. That factor also has been considered while compiling the results. DC6- For observational studies and experiments, record data about other performance measures that may be affected by the treatment, even if they are not the main focus of the study. We do not find any such mention in the study. Analysis guidelines A1: Specify any procedures used to control for multiple testing. The study has used MANOVA and has justified the significance level. A2: Consider using blind analysis. There is no mention in the study if they have used any such method. There is no mention of any possibility that calls for blinding. A3: Perform sensitivity analyses Since there were no extreme results, there is no mention of taking care of this aspect. A4: Ensure that the data do not violate the assumptions of the tests used on them. It seems that there is no such violation possibility because the data collection and analysis has been described in detail but no such possibility has been mentioned. A5: Apply appropriate quality control procedures to verify your results. To maintain quality control, adjustments to the project's cost and schedule as well as the two staffing decisions were analyzed using a multivariate-analysis-of-variance model for repeated measures Because some of the subjects completed their projects as early as day 240, while others finished it later, there were missing values from interval eight (day 280) onward. Therefore, repeated measures analyses were conducted through period seven (day 240). From day 0 through 240 for the two staffing decisions, and from day 40 through 240 for the cost and schedule adjustments were made. Since the number of subjects in the two groups was not equal, the analysis was conducted with the General Linear Models procedure in SAS, 1987. This shows that quality control has been taken care of to get proper results. Presentation Guidelines P1: Describe or cite a reference for all statistical procedures used. References have been used in most of the places while describing any statistical procedure like “While minimizing schedule overruns was a common goal to both the groups the cost group's greater willingness to extend the project's completion time may be attributed to the widely touted "lesson" that tighter schedules often lead to higher project costs (Boehm 1981; Brooks 1995; Londeix 1987).” OR “studies of performance on moderately or extremely complex tasks should look for time lag effects (Wood and Locke 1990, p. 99).” P2: Report the statistical package used There is complete detail of the statistical process used. They have used MANOVA for analyzing project cost and schedule and staffing decisions. To match the results of two unequal group General Linear Models procedure was used for analysis. The results were plotted on charts and they were shown in figure 1 to 6. These graphs had summary statistics, means and standard deviations. It also shows univariate analysis of variance results. P3: Present quantitative results as well as significance levels. Quantitative results should show the magnitude of effects and the confidence limits. This has been complied fully with charts and details. The study shows summary statistics) for the groups' final cost, completion time, and remaining defects. A MANOVA shows that the two goal conditions were significantly different (F(4,l 8) = 9.0993; p Read More