Integration Testing and Systems Testing - Research Paper Example

Integration Testing and Systems Testing Introduction: Systems Engineering is associated with conceptualizing, design, development, testing & make-live of large scale & complex engineering systems for various industrial & software applications. It comprises of complete life cycle of building complex engineering systems using the style & methodology of an engineering management system keeping in view the requirements of the applications for which the systems are being created. Largely Systems Engineering is concerned with computerization of engineering systems using embedded software or controlling software for managing complex functionalities & automation of the overall system to deliver accurate & efficient results and to ensure optimum safety of the individuals working on them. The primary objectives of Systems Engineering is to deliver the systems with optimum practical effectiveness including reliability, efficiency, safety, quality, maintainability, scalability, upgradeability and cost effectiveness. Systems engineering is carried out in a structured process framework called the Systems Engineering Life Cycle processes and is defined by the ISO/IEC 15288 standard [Magee, Stan. 2005]. The process framework is presented in the figure below at a high level: Figure 1: The process framework of Systems Engineering (Source: Bahill, Terry A. and Dean, Frank F. Dec. 1994.) The Integration and System testing activities are carried out after the “integrate” stage of this lifecycle before the “launch” stage. Integration testing is carried out based on the architectural knowledge of the entire system comprising of multiple components that interact with each other to deliver the desired functionalities of the system. Integration testing essentially is carried out after all the components (units are tested individually) and found to be compliant to the test plans. Integration test plans are focused on the behavior of these components in the collaborative environment where the components interface with each other. The testing is carried out to test three elements – data elements, processing elements and connecting elements. The data elements comprise of the information that shall be used & transformed, the processing elements perform transformations on the data elements and the connecting elements perform communications among the different components (pieces of architecture) together. [Bertolino and Inverardi et al. 2002] The next step after integration testing is System Testing that essentially is an end to end testing of the entire system using analytics & artifacts like detailed UML diagrams, sequence diagrams, class diagrams, collaboration diagrams etc. in which all components are expected to perform as per the functional specifications of the system. System testing also involves regression testing & stress testing (load testing) of the entire system environment on the test bed. [Briand, Lionel and Labiche, Yvan. 2002] System integration testing and system testing means completely different things to different companies. System and integration testing is the time to verify systems, subsystems and validate component interoperability, adherence to design constraints and the correct processing of all potential exception conditions. The interaction between integration testing and system testing is very imported relating to verifying systems and sub-systems of a project. Systems test engineers today are designing and testing models to solve integration and system testing problems. Systems integration is a critical task that demands continual participation throughout the project design, construction, and systems testing ensure compatibility between the various systems, other pertinent elements, and the overall project. The testing plan must be structured in a manner that builds from the simple to the complex and from the systems to the end-user systems. This paper investigates system integration testing, system testing, and the roles than play within a project. The role of Testing in Systems Engineering Before the detailed aspects of Integration and System Testing are covered, it is important to discuss the role of testing in Systems Engineering. Testing activities should not be carried out by the people who are developing & programming the system. This is because, psychologically, it is not easy for individuals to find out defects in self developed systems and moreover it is an extremely challenging & creative work that should be left to the individuals that are experts in the same. Hence, as per the global best practices testing should be carried out by people that are not involved in development & programming operations of the systems. The testers should be independent quality assurance personnel that design & execute their own programs for testing the systems in isolated environment that is similar to the development environment but is completely segregated from the same. As a ground rule, developers should not be provided any access & modification privileges in the test environment such that test environments are utilized for running tests only and not for coding improvements. All coding improvements should be carried out in development environment after the codes are sent back from the development environment along with bug reports. As a ground rule, further development on the codes that have gone into testing should not be carried out although some rapid development models allow this partially with calculated risks. The rapid development cycles are adopted by many modern world business driven systems software development companies that can flexibly shift the baselines due to changing requirements, Such methodologies allow rapid removal of defects as reported continuously by the test environment. In such cases, the developers do not wait for testers to complete their test reports rather are in continuous touch with them to get daily updates about defects such that they can be corrected in the development environment. The codes thus corrected are shipped to the test environment rapidly to carry out further cycles of testing. Many experts do not recognize this model as such but the model itself is gaining popularity given its potential of saving time & efforts and hence cost (Boehm, Barry. 2005). The following picture presents the concept of variable baseline: Figure 2: Spiral Model of flexible increments in Rapid Systems Development Cycle (Source: Boehm, Barry. 2005. pp12-14] The objectives of testing should be very clearly understood & articulated. Normally, the expectations of developers from the testing team are that they will demonstrate the functioning of programs as per specifications. However, best practices suggest that a positive approach towards testing doesn’t make it effective. Psychologically, the testers should be trained on organized negative thinking such that their objective is to discover as many errors as possible by carefully designing & executing test cases. Hence, testing team is always positioned on the other side of the table facing the development team such that they “never team up”. Testing should be carried out following bottom to top approach whereby the modules (units) are first tested in standalone environment and then in integrated environment. In broad perspective testing methodologies can be categorized in two forms – Black Box Testing and White Box Testing. In the Black Box testing (also called input/output testing) methodology, the testers are not concerned with the internal programming & its behavior that is carried out in the system. The data for testing is derived from the specifications of the system that has been agreed with the customer in the business user requirement specifications. The test cases are designed by defining multiple combinations of the inputs by populating the entire system with mock data elements and then testing the specifications as agreed with the end users. The objective of black box testing is to find out as many errors as possible in a finite number of test cases. The techniques used in black box testing are cause-effect graphing, equivalence partitioning, boundary value analysis and error guessing. In the White Box testing (also called logic driven testing) methodology, the testers are concerned with the internal program structure & the logic and not the functional specifications of the program. The most basic white box testing is the code review process (code inspection) in which a separate group of people (different from programmers) review every syntax, line by line to arrive at errors in the code writing itself. The objective is to detect and correct as many syntax errors as possible by inspecting data reference errors, data declaration errors, computation errors, comparison errors, control flow errors, interface errors, input/output errors, etc. This is essentially not a testing process as such but is an essential pre-testing process before the codes are uploaded into the testing environment for white box testers. The testers design test cases by writing codes that perform “decision coverage” and “branch coverage” testing in such a way that the results possess at least one true or false decision outcome and that all the statements in the program are executed at least once. Overall, white box testing comprises of statement coverage, decision coverage, condition coverage, decision - condition coverage and multiple condition coverage. [Myers and Badgett et al. 2004; Adrion and Branstad et al. 1992] Tests are conducted with the help of tools that comprise of end to end cycle for testing of the system codes. A framework of testing software shall comprise of: (a) A translator that converts the source data into appropriate test data removing production data elements and replacing them by mock data under same data types. For example, if the data extracted for testing has actual mobile numbers of customers, they are modified by randomly generated numbers just to ensure that no production data gets into the test environment. (b) A driver that initializes & runs the test environment (c) Testing stubs that run the simulation of the modules (d) A monitor to observe the flow through the test object and also to graphically represent the traces of actions by modules in the test environment (e) A result validation engine (f) A file manager for test files (g) A reporting engine to manage test reports [Adrion and Branstad et al. 1992] Black Box and White Box testing require completely different expertise and hence should be managed by different teams & managers. However, the overall supervisor of testing should be the same that is not governed by the development management but should have reporting line to the roles that are overall in-charge of delivery to the customers. Integration Testing in Systems Engineering Integration Testing lies in the White Box Testing category. We present Integration Testing concepts & roles applicable in the world of Software Engineering that comprises of majority of application areas in Systems Engineering. Software Engineering world has two broad methodologies for programming a system – Object Oriented Programming and Modular Programming. Object Oriented Programming is the legacy methodology in which discrete application entities are created that encapsulates data and the behavior of the entity as per the target functionality. The objects for certain functionalities can be generalized and reused by allowing it to interact with other objects using standard protocols. In spite of these two major advantages of object oriented software, Modular programming has rapidly found acceptance over object based programming due to their better visibility & understandability and flexibility to adapt to changes in the systems architecture. Objects may at times hold reference information about other objects which may change dynamically thus make their reuse difficult. Components are independent modules of the overall architecture that are self contained & organized comprising of their own functionalities and features. The interconnections among the modules are coherent & much simpler unlike objects where the interconnections are relatively complex. A System architecture is viewed as modular if it has decomposability, composability, understandability, continuity and security. Integration testing in modular software and object oriented systems are different. In modular software, the integration testing is simpler due to simpler interfaces among the modules. However, in case of object oriented systems the integration testing is very complex specifically if the objects are reused and hence are “sealed”. The integration testers need to test multiple features in case of object oriented systems – message passing, synchronization, dynamic binding, persistence, encapsulation, inheritance, polymorphism, etc. among the objects that may be interacting with unforeseen combinations and invocations. The testing is carried out at algorithm level, class level, cluster level and the overall system level. Following are some of the popular integration testing techniques used in Object Oriented Programmed Systems as well as Components Based Programmed Systems: (a) State based integration testing: In this method, the objects or components are viewed as finite state machines that are systematically arranged in a hierarchy and “State Transition Diagrams” are created to define the interface processes. Thereafter the testers employ graph minimization techniques to remove the infeasible or unnecessary state transitions. The resultant model is then verified for the selected properties as defined in the test plan. (b) Event based integration testing: The states in the previous method are replaced by events whereby the synchronization sequence of programs acting at same instance is viewed as pairs of synchronization events. The graphs comprise of event sequences rather than components or objects. Testing is carried out based on relationships between event pairs – always valid, possibly valid or never valid. Possibly valid relationship can be possibly true or possibly false. Thus the testing requires vetting of event sequence as per the test specifications. (c) Deterministic and Reachability integration testing: In this technique the program is forced to execute according to a desired path (forced execution order testing) and the effects are traced, recorded, analyzed and reported. This however is not viewed as an extensive integration testing system because the number of paths possible in a complex program can go in billions thus making the testing ineffective. (d) UML based integration testing: Unified Modeling Language is very popular among systems architects in creation of class, activity, sequence, collaboration, and state diagrams. These diagrams are already designed as a part of the architecture and the tester uses them for verifying execution & interactions effects using different diagrams for different perspectives – example, activity diagrams from user functionality perspective and sequence diagrams for inter component/object interaction perspective. [Chan and Chen et al. Jul. 2002] System Testing in Systems Engineering System Testing is an extensive testing cycle that takes into account the entire system end to end that involves functional testing and performance testing. Hence the testers make use of almost every architecture & design UML documentation that the developers might have created – Use cases, corresponding sequence & collaboration diagrams, class diagrams, object constraint language expressions, etc. From these documentation the system testers need to derive detailed test requirements, detailed design information, test cases, test drivers, etc. in order to devise the test plan and its size, duration, resource requirements and life cycle. The test cases are generated through test programs that can take inputs from the UML artifacts and test requirements. A lot depends upon the “testability” & “automation” of the software at the system testing stage which depends upon whether sufficient information in the UML artifacts is available or not. A basic set required for system testing are: (a) Use case diagrams (b) Use case descriptions (c) Sequence/Collaboration diagrams for each use case (d) Class diagrams showing the application classes (e) Data dictionary describing all the classes, methods and attributes (f) Object constraint language expressions for the classes Figure 3: System Testing planning (Source: Briand and Labiche. 2002) System Testing is kind of a never ending exercise and is carried out in pre-release stage, post-release stage and post make-live stage (field defects). The system testers try to find as many defects as possible through their test plans and practically can never be able to achieve zero-defect system given the complexity & limitations of documentation required for system testing. [Briand, Lionel and Labiche, Yvan. 2002; Gittens, Mechelle and Godwin, David et al. 2002] Positioning & role play of Integration and Systems Testing in the overall Systems Development Life Cycle It is well understood and appreciated that testing exercises are very complex, challenging and innovative and require a lot of support & documentation from the developers. Hence, looking at the bigger picture, the author hereby argues that testers cannot be kept on the other side of the table against developers rather should be allowed to team up with developers irrespective of separate reporting lines. The way testers are made aware of developer’s requirements, even the developers should also be made aware of the tester’s requirements. It is in the larger interest of the project to ensure that the system is “testable” most preferably in “automated mode”. In the modern development, the testing cycle in projects is pushed to the last stage after all the iterations & enhancements are carried out amidst continuous input of requirements from the business. The modern rapid action development ensures that development can also become iterative using the shifting baseline concept as introduced earlier in this paper. However, just shifting baselines in development environment doesn’t guarantee effective testing at the end rather the test cases also need to be continuously modified in line with the shifting code baselines. As discussed earlier that testing is completely dependent upon a large framework of documentation by the development teams. If the developers do not document a substantial part of them given that they did not perceive them to be important from development perspective then the testing (especially system testing) can be substantially impacted. Hence, it is important that culturally & psychologically the developers should be educated on the bigger picture of the development & testing framework such that they keep the testing requirements in mind from day one of the project life cycle. The author proposes a strategic partnership between developers and testers to facilitate this approach. This may begin at the time when business requirement specifications are being translated into software requirement specifications. The developers are testers need to work together to carry out the following collaborative activities: (a) Mutually agree on the required set of documentation that is required to be prepared for effective testing exercises later (all required UML diagrams and corresponding documentation). (b) Functional specifications (c) Reviews & inspection logs (d) Formal entry and exit criteria definitions (e) Functional test cases (based on use cases) (f) All functional variations to be documented and modified in the functional test cases periodically (g) Multi platform testing (h) Development of beta environments and their documentation (i) Designs & considerations of automated test case environment generation to whatever extent possible (j) Development of simulation environments for integration testing, system testing, stress testing and their documentation (k) Structured mechanism for defect tracking (l) Centralized code version control system and code repository for both development and test systems in which the “code transition” logs (example from development to testing) in the form of check-in records and check-out records are very clearly identified [Chillarege, Ram. 2002; Whittaker, James A. 2000; Talby, David and Keren, Arie et al. 2006] Testing requirements, efforts, cost, etc. should be planned right at the time of preparing the costing of the project. Each stage of test preparation and testing activities should be planned in the beginning itself. Normally organizations maintain a centralized pool of testers for multiple software projects that are in progress. However, given the engagement procedure of testers & developers defined above, each development team may have to engage a strategic testing partner in the beginning itself who will work part time and will keep an eye on all the documentation created by developers such that she/he can raise alerts about deficiencies at the strategic checkpoints as the project is in progress. Hence, the project manager should include the cost of efforts by the testing partner in addition to the costing of consolidated testing that shall be carried out at the end of development stages – like component building, component integration and final build release. The strategic partnership with a tester will ensure that all information & documentation required for building test environments and automating the testing are consolidated by the individual such that last moment surprises like missing UML diagrams or missing specification details do not occur letting the project manager & test manager get into an embarrassing situation of damage control by compromising with lesser testing efforts. In large projects involving extreme programming (agile development methodology) the testing team gets their consolidated work stack when the integration of components is over and integration testing is triggered. The module testing in such large projects involves huge intermittent testing exercises almost continuously and hence the developers need test cases to be run during the development phase itself. The strategic test partnership concept can prove to be very handy as all such test coding can be carried out by the tester that has partnered with the development team on a part time basis. The coordination process between developers and testing partners definitely is a challenge but can be made easier by deploying test director tools that can partially serve as a testing helpdesk for the developers with clear documentation of accountability of the testers holding the testing helpdesk. The defect management process also becomes very easy by using such test director tools given that all defects are open for viewing by everyone, defects can be dynamically assigned to development resources by the testers and the resources in turn can close the defects and update the status on the tool. The project manager can get consolidated view of all the closed and pending defects in the test director tool. The idea shall be to fix the defects as soon as possible after their discovery by the testers and logging them on the test director tool. [Talby, David and Keren, Arie et al. 2006] The other important consideration in the overall systems development life cycle is including testing considerations in the Project Risk Management Framework. Normally, the logging & closing of project risks are carried out using a centralized Risk Management system and database. The system should be used proactively to log testing risks as well in addition to development risks. The testing partners may be provided access to the tool such that they evaluate the proceeding of the project and its documentation from their perspective and log risks like “emerging non-compliance to testing requirements thus making the software difficult for testing”. The testing partners would have to think selfishly and be very clear about what is required for effective testing to be carried out by her/his colleagues and what is not coming up effectively as per her/his expectations. Project scheduling is another important part of the project management where testing should be considered adequately. The schedule of the project should take into account the end to end life cycle of testing of the project based on the high level test plans and should keep enough flexibility just in case the defect density is not in control and the testing span requires to be stretched. Delivery management will always keep the project manager stretched for timelines but under no circumstances the testing should be compromised. Strategic Testing partners should be actively involved in the Quality Assurance group and the Software Engineering Process Group (SEPG). This is because testers are themselves quality assurance personnel in their own rights. Hence, they should be allowed to carry out formal technical reviews and code inspection much before even the component level testing is triggered. To achieve complete compliance with the quality assurance rights of the testers as a part of the requirement of the project the following needs to be inspected and complied with: (a) The Major test phases have been properly identified and sequenced (b) The traceability to validation criteria have been established as a part of the overall software requirement analysis (c) Major test functions have been identified early in the project (d) Testing schedule has been defined explicitly (e) Test resources are identified and booked from the central resource pool much in advance by the project (f) Testing tools (tools to generate test cases, run test cases, monitor progress, behavior, etc. tools to carry out stress testing, tools to log testing outputs and reports, etc.) are identified and procured (g) Test recording mechanism has been established (h) Test drivers and stubs have been identified & documented and work to develop them is included as a part of the overall project schedule (i) Both black box and white box testing are identified (j) Testing mechanisms (UML based, State based, Execution path based, etc.) are defined and documented clearly in the test plan (k) Decision on error handling and boundary value testing has been taken (l) Decision on performance testing has been taken (m) Acceptable variations from the expected test results (setting expectations from testing) have been defined (n) Software configuration management procedures have taken into account separate management of codes that have transitioned to testing environments and those that have passed testing (o) All requirement of testing documentation has been defined (p) The metrics for measuring testing effectiveness (key performance indicators are defined) [Pressman and Associates. 2001] Conclusion: Testing is one such major component of Systems Development Life Cycle that is not yet understood completely by the project management. Normally, testing is pushed to the fag end of the project after all the coding & modifications are carried out. Due to timeline pressures from the delivery management, the developers take short cut approaches thus skipping a number of critical documentation that are very important for planning & running effective test cases. This is the reason that the project management should keep the testability of their product from the very beginning and include testing as a part of all their components of planning – estimation, scheduling, quality assurance requirements, Resourcing, etc. This paper critically evaluated the role of testing in systems engineering, the details of integration & systems testing and the overall role of testers in the systems development life cycle defining their organizational positioning, engagement aspects, reporting aspects, accountabilities, and tasks. The outcome of this paper has been derived from literature review as detailed out in the references section and in the citations at various placed in the paper. Reference List: Adrion, W. Richards and Branstad, Martha A. et al. (Jun. 1992). Validation, Verification and Testing of Computer Software. Computing Surveys. Vol.14. No.2. ACM. pp164-175. Bertolino and Inverardi et al. (Jun. 2002). Consiglio Nazionale delle Ricerche and Universita’ di L’Aquila. Italy. pp1-2. Boehm, Barry. (Dec. 2005). Some Future Trends and Implications for Systems and Software Engineering Processes. Systems Engineering. Vol. 9. No. 1. Wiley Periodicals. pp12-14. Briand, Lionel and Labiche, Yvan. (Jun. 2002). A UML based approach to System Testing. Carleton University. Version 4. pp1-6. Chan and Chen et al. (Jul. 2002). An Overview of Integration Testing Techniques for Object – Oriented Programs. Proceedings of the 2nd ACIS Annual International Conference on Computer and Information Science. International Association for Computer and Information Science. pp1-6 Chillarege, Ram. (Sept. 2002). Centre for Software Engineering. Software Testing Best Practices. Technical Report. IBM Research. pp1-10. Gittens, Mechelle and Godwin, David et al. (Aug. 2002). An Empirical Evaluation of System and Regression Testing. University of Western Ontario in collaboration with IBM. pp3. Magee, Stan. (Apr. 2005). Evidence Product Checklist for ISO/IEC Standard 15288 – System Engineering Life Cycle Processes. Software Engineering Process Technology. pp3. Meyer, Bertrand. (Sept. 1996). Object Oriented Software Construction. Published by Interactive Software Engineering. Second Edition. pp61-62. Myers, Glenford and Badgett, Tom et al. (Jul. 2004). The Art of Software Testing. Second Edition. John Wiley and Sons. pp5-20, 24-26, 43-52. Sage, Andrew P. (1992). Systems Engineering. Published by Wiley – IEEE. pp8-9. Software Engineering – A Practioner’s Approach. (2001). Published by R.S. Pressman and Associates Inc. pp36-42. Talby, David and Keren, Arie et al. (Jun. 2006). Agile Software Testing in a Large Scale Project. IEEE Software. Published by IEEE Computer Society. pp2-7 Whittaker, James A. (Feb. 2000). What is Software Testing? And Why is it so Hard?. IEEE Software. Published by IEEE Computer Society. pp1-3 End of Document Read More