Software Engineering Testing tools - Research Paper Example

? Software Engineering Testing Tools are Largely Unreliable Due to their Misuse in Development as a Result of Poor Selection - Draft Larry Stewart Abstract This research paper details a study about the unreliability of software engineering testing tools as a result of the misuse of these testing tools in development. Testing activities that are geared towards evaluating the capabilities and attributes of systems or programs with a view to ascertaining whether the program produces the required results will be discussed. This paper also aims to compare and identify some new practices that can be used in augmenting current software development practices. Introduction Software engineering testing tools can be said to be generally unreliable due to the misuse of these testing tools in development. As Myers, (1979) notes, “Software Testing is the process of executing a system or program with the aim of searching out errors in the program.” (Myers, Glenford J., 1979). Software Testing also involves activities that are geared towards evaluating the capabilities and attributes of a particular system or program and ascertaining that the program produces the required results. “Software engineering is not unlike other physical processes where inputs are received and outputs are produced. Where software differs is in the manner in which it fails. Most physical systems fail in a fixed and reasonably small set of ways. By contrast, software can fail in many bizarre ways, and detecting all of the different failure modes for software is generally infeasible.” (Musa, John D., 1997) There are numerous software engineering testing tools available today. Many of these tools have come to be adopted and utilized as solutions to the rising demand for assurance of software quality in the software engineering industry. Within the past decade, there has been a steady increase in black-box testing techniques which are generally called model-based testing (MBT) techniques. This steady increase in black-box testing techniques may be due to the growing popularity of object orientation and modeling in software engineering. Model based testing is a general term used to indicate that a particular testing approach bases common testing tasks (like test result evaluation and test case generation) on a model of the application that is being tested. (Jorgensen 1995) Engineering software is usually implemented in critical applications, and so improper testing can result in errors that may have severe outcomes that may cause huge losses. In spite of the inherent limitations, software testing is an integral part of software development. It is usual for more than half of the software development time to be spent on testing the software, and this entails a critical process of selection. The Software engineering testing tools in use today are unreliable largely due to their misuse in development, as a result of poor selection Testing Purposes Testing of engineering software can be broadly deployed in various phases of the software development cycle. Testing of software involves a process of verification and validation of software programs or applications to ensure that the software is functioning as expected, and meets the technical and business requirements that the designers had in mind when designing and developing the software. Software testing is also aimed at identifying important errors, flaws or defects, in software programs that have to be corrected. (Hetzel, William C., 1988) During test planning it must be decided what to view as an important defect, through the reviewing of the design requirements and documents. In a broader view, we may start to question the utmost purpose of testing. Why do we need more effective testing methods anyway, since finding defects and removing the defects does not necessarily lead to better quality? The value to be derived from engineering software testing is that such tests go far beyond doing tests about the underlying code. Such activities also involve an examination of the functional behavior of the program being tested. “Behavior is a function of the code, but it doesn’t always follow that if the behavior is bad then the code is bad. It’s entirely possible that the code is solid but the requirements were inaccurately or incompletely collected and communicated.” (Pan, Jiantao 2007). In some cases, an application may not be working exactly as planned, but this can be because the application is not getting the correct inputs or commands to function as desired. Software engineering testing includes an evaluation of the functional and technical design requirements, regulatory, business requirements, and the programming code, configuration of the hardware, corporate standards, cultural issues and language differences. It is very difficult to test every combination of commands or keystrokes and indeed this might prove to be even impossible, thereby creating room for unstructured testing In software testing, quality signifies the conformance of the software to the specified design requirements. As humans, we are imperfect, and thus it is almost impossible to create a correct program of moderately complexity with the first try. Finding the problems and get them fixed. This is the purpose of debugging in programming phase, and no single practice can provide a universal solution for software testing.” (Petrenko 2000) Model-based Testing Model-based testing is rooted in hardware testing applications such as telephone switches, and has recently been extended to a wide range of software domains. There are numerous published works discussing some case studies involving model-based testing in industrial, as well as academic settings. (Gronau 2000). This signifies a growing general interest in model based testing. The most cost-efficient approach to software testing incorporates thinking beyond normative features, while also incorporating this idea into the whole process of development. There are 3 main aims of software testing: validation, verification and finding of defects in the software. Validation The validation process is used to confirm that the software being tested actually meets the business requirements of the application. Software testing has been the most extensively used method for the detection of software problems and failure. During the testing process, it is important for the developers to generate the programs test inputs, and run those test inputs on the program, as well as checking for correctness in the test execution. It is a recognized fact that software testing can be quite expensive, but automating the software testing process can be used to reduce the laborious human effort involved in the testing, thereby reducing inefficacy in software testing. There are some technical challenges involved in such automation of software testing, including generating sufficient test inputs and the checking of correctness in the test execution. Program specifications may also be valuable in addressing the challenges faced in this type of software testing. Verification The process of verification is aimed at confirming that a given application meets the technical specifications involved. This specification entails a proper description of the functions in terms of an output value that can be measured, given specific input values under specific pre-conditions. Finding of defects Defect in software constitute a difference between the actual outcome of the program and the initially expected results. The ultimate source of such a defect can be from faults in the specification, design, or development stage of the software. It is essential to anticipate some form of abnormal behavior in the program in order to create reliable and secure software. Misuse or abuse cases in software testing should be discussed and prepared for, as this can enable an organization to be able to view the software involved from the same perspective as the end user. By thinking beyond normative features and at the same time contemplating unexpected eventuality, software security professionals can understand better how more reliable software can be created, and tested. Since software features can be intentionally abused or unwittingly misused, it is essential for professionals with a good knowledge of the vulnerabilities of these features and how software can be protected to have an active role in the analysis of such misuse and unreliability. Software engineering involves the use of tools, processes and techniques that will enable the software engineer to address various programming and development issues in various phases of the software development life cycle (SDLC). Testing of software enables the evaluation of the capabilities and attributes of systems or program or in order to ensure that the software meets the required needs and accomplishes the required results. (Hetzel, William C., 1988). Software engineering testing tools are largely unreliable as a result of misuse in development. This is mostly due to a limited understanding of software testing principles, even though Software engineering testing is a crucial aspect of ensuring software quality, and is widely utilized both by programmers. Another reason for the misuse and hence unreliability of Software engineering testing tools is the degree of complexity of the software involved. Software engineering testing is more than mere debugging of the program, and since the major aims of testing include validation, verification, estimation of reliability and quality assurance, such testing can also be utilized as a generic metric. Software testing is a trade-off between quality, budget, and the available time. Reliability and correctness testing are known to be 2 major areas of software testing. Unlike most physical systems, most of the defects in software are design errors, not manufacturing defects. Software does not suffer from corrosion, wear-and-tear The main aim of Software Engineering is the building of better, software that will be as much as possible, free of defects. Some software-intensive systems which are constructed with the use of more securely developed software tend to have more ability to continue functioning properly in the presence of attacks. This is because these will be able to tolerate any failures and resist the exploitation of weaknesses in the software by attackers. It has been noted that software engineering testing is still not mature, and remains an art, due to the fact that the parties involved are still finding it very hard to make this a science. Outdated testing techniques which were invented as far back as 30 years ago are still being employed in some cases, and some of these are heuristic or crafted methods which lack the proper engineering methods needed. Although software testing can be quite expensive, failure to test the software is even costlier, especially in situations in which human lives are at stake. Yang, et al (2006) details a survey that was intended to study and compare coverage-based software testing tools. The study included a comparison of Automation and reporting, Code coverage measurement, Coverage measurement criteria and Code coverage measurement features. Wong, E. W., Lei, Y. & Ma, X. (2005) also used a coverage-based approach by using graph models of the concurrent software. Structural or white-box testing often involves constructing a reachability graph. In the survey carried out by Yang et al. (2006), additional features like debugging assistance, software prioritization for testing, automatic generation of test cases and customization of test reports were also considered. The results from that research suggest that test tools are tailored to a specific software domain given their unique features. Thus, it will be effective to use the findings of that study in the selection of which coverage-based test tool are most suited for each specific software domain. The Process of Tool Evaluation Effective selection of tools and evaluation processes are major factors in software engineering, used for improving development efficiency. Since the distributed application of methods is a function of simultaneous work and effective tools for collaboration, it is necessary to employ the proper approaches for addressing the additional issues. In their research, Poston et al provided a general purpose workflow for evaluation of software testing tools that seems appropriate for testing tool evaluation and selection. “Based on a wide range of available solutions, a systematic approach for tool evaluation is required.” (Poston, R.M., Sexton 2011). Poston and Sexton (2011) also identified 4 main phase of the testing tools selection and evaluation process: Analysis and classification of requirements and expected tool properties Definition of tool selection criteria including prioritization Search for tools and classification of candidate tools Evaluation of candidate tools with respect to a given evaluation framework. (Poston, R.M., Sexton 2011). Many software engineering testing tools are flawed and thus, unreliable due to their misuse in development. For instance with coverage testing, there is no definite proof. Inspection has been suggested as a cost-efficient alternative to unit testing, and previous experimental results point to the fact that “code reading by stepwise abstraction is at least as effective as on-line functional and structural testing in terms of number and cost of faults observed.” (Myers, Glenford J. 1979). Functionality (exterior quality) Engineering (interior quality) Adaptability (future quality) Correctness Efficiency Flexibility Reliability Testability Reusability Usability Documentation Maintainability Integrity Structure Table 1. Typical Software Quality Factors In the use of software engineering testing tools, caution must be exercised, and there should be critical and curious, albeit non-judgmental, good communication. Although testing every combination of keystrokes or commands is difficult at best and may well be impossible, it must be noted that features don’t function in isolation from each other. It is a general consensus among software engineering testing experts that test plans are usually biased towards functional testing. In the process of this functional testing, each feature is likely to be singularly tested in a unit test, with a systems integration test being a simple series of these unit tests that has been joined or strung together. A major problem inherent in this approach is that it when each feature is tested alone and a bunch of these kinds of tests are strung together, it becomes very difficult to realize that a series of steps (for example open a document, edit, print, and save the document, edit a single page, print it, save as new document) does not work. On the other hand, users will probably discover this very quickly. The actual environment in which software is tested will always affect the results of such tests. Some enterprises believe that the software vendors should be responsible for testing the applications and systems, but this is not so. Neglecting to use negative testing practices during the various stages of deploying critical systems can be considered negligent. Conclusion Testing of software enables the evaluation of the capabilities and attributes of systems or program or in order to ensure that the software meets the required needs and accomplishes the required results. During such tests, software features can be intentionally abused or unwittingly misused, it is essential for professionals with a good knowledge of the vulnerabilities of these features and how software can be protected to have an active role in the analysis of such misuse and unreliability. It has been noted that software engineering testing is still not mature, and remains an art, due to the fact that the parties involved are still finding it very hard to make this a science. Outdated testing techniques which were invented as far back as 30 years ago are still being employed in some cases, and some of these are heuristic or crafted methods which lack the proper engineering methods needed. Although software testing can be quite expensive, failure to test the software is even costlier, especially in situations in which human lives are at stake. Many software engineering testing tools are flawed and thus, unreliable due to their misuse in development. For instance with coverage testing, there is no definite proof. Also, the actual environment in which software is tested will always affect the results of such tests. Some enterprises believe that the software vendors should be responsible for testing the applications and systems, but this is not so. Neglecting to use negative testing practices during the various stages of deploying critical systems can be considered negligent. There are numerous software engineering testing tools available today. Many of these tools have come to be adopted and utilized as solutions to the rising demand for assurance of software quality in the software engineering industry. Within the past decade, there has been a steady increase in black-box testing techniques which are generally called model-based testing (MBT) techniques. In spite of the inherent limitations, software testing is an integral part of software development. Testing of engineering software can be broadly deployed in various phases of the software development cycle. It is usual for more than half of the software development time to be spent on testing the software, but software features can be intentionally abused or unwittingly misused, leading to a situation in which software engineering testing tools are generally unreliable due to the misuse of these testing tools in development. References Hetzel, William C., (1988) The Complete Guide to Software Testing. Wellesley: QED Information Sciences. Myers, Glenford J. (1979). The Art of Software Testing. New York: John Wiley and Sons. pp. 145–146. Musa, John D.., (1997) Software-reliability-engineered testing practice. International conference on Software engineering, Pages 628 - 629 Pan, Jiantao (2007) Software Testing. Carnegie: Mellon University Yang, Q., Li, J.J., Weiss, D. (2006). A Survey of Coverage Based Testing Tools. Proceedings of the 2006 international workshop on Automation of software test, May 2006, pp. 99-103. Poston, R.M.; Sexton, M.P.: Evaluating and selecting testing tools. IEEE Software, Volume: 9 Issue: 3, May 1992,S. 33 -42 Wong, E. W., Lei, Y. & Ma, X. (2005). Effective generation of test sequences for structural testing of concurrent programs. Proceedings of the 10th IEEE International Conference on Engineering of Complex Computer Systems’, IEEE Computer Society, Washington, DC, USA, pp. 539–548. Read More