Measuring Test Quality in Embedded Systems - Research Paper Example

Measuring Test Quality in Embedded Systems Introduction The recent years have witnessed an increased focus on embedded systems quality and better measuring methodology following the increased number of high profile examples of product defects being caused by errors in embedded systems. Modern embedded systems have been included with more line codes compared to what was being witnessed a few years ago. Now that the latest developments have led to the number of bugs to be out of proportion in relation to the number of line codes, the process of eradicating the bugs and other errors as well as improving the measuring process has become significant and sensitive more than ever. The process of measuring to determine if the testing is of the best quality to assure system’s quality has become a challenge on its own (Baresel 67). In most situations, the test quality is conducted informally resulting to very little or at time no quantitative evidence gathered by the tester to be used in determining the test coverage level. It is needless to achieve a test success that is 100% and yet the test quality can only manage to cover a small code fraction leaving the better portion of the codes untested completely. This situation has called for the development of better testing methods that can be used to overcome the mentioned above setbacks experienced by the traditional embedded systems through offering the exact test quality information even in situations where the measuring and testing processes are carried out when the application under investigation is being run in an embedded target board (Conrad 78) Some malfunction software have been introduced as the quality problem culprit in the products of embedded systems. Software testing has therefore received more focus in the recent years and is commonly being used as a means adapted by companies to produce products that are of high quality within the desired time duration to their customers. The traditional systems that made use of manual methods for testing software have been unable to keep up with the rapid increasing code amounts in the contemporary implementation of products. Thus better systems to ensure that the tests conducted are efficiently executed and the tests actions are reported, monitored and analyzed in better ways so as to keep up with the pace of modern requirements are needed (Grchtmann 27). Most of the programs used these days in embedded systems produce codes that are needlessly complex. While such released codes might still be functional, they are normally more complicated than they are required to be. Codes that are complicated tend to transform and become codes that are problem-riddled. Most technicians believe that in an event that a code is not released, then it better be allowed to escape. This will check on the overly complex abundance of the software within the embedded system. Some of such software have been applied in safety-critical systems in this new generation thus resulting in the enactment of specific guidelines which have challenged the operations of some of the traditional programs (Gross and Eyres 28). The basic problem of the traditional software used in embedded systems whose testing quality needs to be improved is the writing of most of such software in ways that are complex beyond the level needed to perform their respective function. Complex software has been associated with a number of problems when used in embedded systems. The problems include; most likely to be defective, very hard to understand, hard to change, difficult when needed to be reused and takes longer periods to produce (Lehman 45) This study is out to seek the understanding of the embedded systems future testing taking into consideration both the practical and theoretical perspectives and goes further to try and come up with modern methodologies that are in a position to meet the stakeholder’s needs. To achieve this, the paper will look at suitable algorithms made use by the testing method that it will propose. The safety of the testing process adapted by the methodologies will also be taken into consideration. By the end of the paper, it should be clear to see how the measuring of the test quality of embedded systems can be improved (Puscher and Nossal 140) Evolutionary Testing This proper proposes the Evolutionary Testing methodology as the most convenient meta-heuristic testing method in the modern generation. This method is convenient as it transforms the testing of an embedded system into an optimization problem. The input domain of the system becomes the search space where the searching for the testing data takes place so at to meet the desired test aim. Evolutionary Testing has its software in a non-linear form which is included with loops and if-statements allowing the test problem conversion to have the tasks that previously resulted in discontinuous, non-linear and complex results being optimized. The methodology uses meta-heuristic approaches to search for data; that is, simulated annealing, evolutionary algorithms and tabu search. Evolutionary algorithms have proved to stand the test of time as they come up with test data now that their suitability and robustness for different testing tasks solutions have been proven already (Schultz and De Jong 12) Evolutionary Algorithms Evolutionary algorithm is an adaptive search procedure and technique that has been based on natural genetic processes. The algorithm is characterized by a procedure that is iterative and works parallel with several potential solutions for individual populations. Its values on permissible solutions are encoded in every individual for optimization problem variables. The main concept behind evolutionary algorithm is evolving the successive generations of better combinations that have been on the increase of the parameters which greatly affect the embedded system’s overall performance (McGraw 83). Beginning from a selected sample, evolutionary algorithm aims at achieving the possible optimum solution through exchanging information randomly between independent random change probability and the fit samples. The adaption of this algorithm is accomplished through the reinsertion and the selection of procedures that have been developed taking into consideration their fitness (Redmil 12). The selection procedures in question involve the control where the samples are selected depending on their fitness values. The reinsertion strategy on the other hand is used to determine the number and the type of samples taken from a given population. Fitness value refers to a numerical value which is used to express a sample performance in relation to the existing optimum so as to allow the comparison of different samples. The fitness notion is fundamental to the evolutionary algorithm application in that its level of success while using them depend heavily on a fitness definition that keep on changing not too slowly and not too rapidly with the parameters designed in the embedded system. This fitness function has to guarantee that the samples to be tested are in a position to be differentiated in accordance to how suitable they are in solving the problem on optimization (Graff 45) Software Testing Application For one to be able to automate software testing with the help on evolutionary algorithms, the aim of carrying out the testing itself must be changed into an optimization task in the first place. Following this, the test aiming numeric representation is important from which a fitness function that is most suitable for the test data which have been generated to be evaluated can be possibly derived. Depending on the test aim that is being pursued, several varying fitness functions come up for the evaluation of test data. The evaluation testing can only take place if the fitness function that is most appropriate has been defined. The testing application is normally done at random in situations where the test data happen to be have been obtained already in systematic tests conducted previously. Thus the evolutionary testing process benefits from the knowledge of the tester of the embedded system under investigation. Each sample to be tested is a representation of a test datum which the execution of the test objects takes place. For every test datum presented, the execution process has to be monitored while the fitness values found for each of the corresponding sample. One of the important considerations during the testing is that the generated test data has to be in the input domain of the system being tested (Kaner 61). Structural Testing Generation Structural testing has been widely applied in various industrial practices and has also been stipulated in several developments of software standards. The most common applications of such structural testing include; branch, condition and statement testing. The main objective of having evolutionary testing being applied on structural testing is to come up with a set of test data that would lead to coverage which will be of the highest level in relation to the structural test criterion that will have been selected. Structural testing methodologies fall under four main categories and the classification has been based on the purpose of the test required and the control-flow graph. The categories include; methods that are path oriented, node-oriented, node-node oriented and node-path oriented (Menachem 64). Methods that are node oriented call for specific nodes execution in the graph involving control-flow. The methods widely known to be in this category include condition and statement testing. Methods that are oriented on the path taken need the execution to be carried out on specified paths found in the control-flow graph. Such methods include path testing. Methods that are oriented on node-path call for specific node achievement and from this particular node, a specific path execution occurs. The simplest example of such methods is the branch test. Methods that are oriented in node to node involve several nodes execution taking place in a sequence that is pre-determined but without taking a concrete path on the control-flow graph. This category includes methods such as all-uses, all, defs and all-defuse-chains which are all oriented on data flow (Boehm 56). For one to make use of evolutionary testing in structural testing automation, the testing process has to be divided into aims that are partial. Partial aim identification is based on the programed control-flow graph. Each of the selected partial aim represents a structure that has been programed which requires execution so as to have full coverage like branch, statement or legal values condition (Chidamber 482). Safety Testing Embedded systems are often viewed as being safety-relevant. This means that safety analysis in relation to embedded systems such as software-hazard analysis and fault-tree analysis have to be considered as safety requirements. Such requirements are indispensible for the components of the system now that they have been derived from system behaviors that have to be avoided in all means. In case there exists any possibility of violation of the safety requirements that have been specified then the system is regarded as not being safe. Following the above situation, the test aim is normally directed towards finding input situations that result in safety requirement violation. In the event that such types of input situations are identified, then the embedded system is not safe thus has to be corrected. The fitness evaluation witnessed in having Evolutionary Test applied on safety tests is the same as structural testing fitness evaluation (Ebert 91). The difference between the two is that fitness function has not been based on the program’s branch predicates but on specified post-conditions for individual components. This can be explained by the fact that if the speed of an output signal of a given component is not permitted to be negative, then the individual’s fitness values can be easily set in accordance to every output speed value produced. Individuals with small speed values obtain fitness values that are higher as compared to individuals having high speed values. When the Evolutionary Test is in a position to detect an individual with a negative speed value, then this is a sure proof that the safety requirements have been violated. In order to have a safety automation that is complete, the Evolutionary Test can be integrated with Time Partition Testing for embedded system’s integration test (Hamill 23). Temporal Testing Behavior Most of embedded systems normally undergo temporal requirements at some point. This is as a result of operational comfort; the system’s reaction times to user commands which are normally short or as a result of technical processes requirements that are controlled by the embedded system. This means that embedded systems have to be tested thoroughly with regards to both their functional behavior and with the aim of detecting any deficiencies existing in temporal behavior. The traditional methods are unstable in examining the correctness of the temporal. Even for a tester who is experienced, the process becomes almost impossible especially when it comes to finding input situations that are most important and are relevant in the thorough temporal behavior examination which is achieved through testing and analyzing manuals of complex systems (Jackson 25). Evolutionary testing has proved to provide a lasting solution to this problem as it offers a testing approach that is promising for temporal behavior testing of embedded systems and real-time. During the process of testing the system’s temporal behavior, the main aim is to find out the existence of input situations for which the embedded system goes against its timing constraints that have been specified. In this process, a violation takes place as the outputs are normally produced prematurely or the computation process takes longer than expected. The test task and consequently the Evolutionary Test is to examine the input situation with more focus being on executions times that are either short or long so as to find out the chances of a temporal error being produced. The process of determining execution times that are longest or shortest using evolutionary testing involves the measuring of the execution time for each test datum (Martin 67). The individual’s fitness evaluations that have already been generated are based on the times of execution measured for the test data that corresponds to them. The testing process is normally terminated in the event that an error is detected in the temporal behavior or a termination criterion that was specified has been reached at. In case a violation of the predetermined temporal limits of the system is detected, the test will be considered as being successful and the system rendered corrected. Evolutionary testing thus permits for a complete automated search for execution times that are extreme. The test benefits especially considering the triviality of the temporal behavior evaluation concern (McCabe 150). Alternative Methodology Atollic TrueANALYZER is one of the latest developments when it comes automation embedded tests that are in-target. This testing tool is best fit to meet the testing needs of the application software used today. In embedded system, a report indicating that the test process was 100% successful may not necessarily mean that the application has been successfully tested. This is because most software tests are normally run on a small portion of the entire applications. This makes it almost impossible to be sure if every line code has been exercised. Chances are that the fraction under test may behave as the tester might expect yet the rest of the software might still be untested and thus possibly contain several bugs yet to be discovered. This makes the understanding of the test procedure’s quality to be critical in any judgment of whether a product has been well tested before it is released. The analysis of dynamic execution flow can be applicable in code coverage analysis which happens to be away of test quality measuring. Very many code types coverage have come up ranging from the very stringent to the ones that are very basic. The code types that are more stringent need more trails and efforts when used in coverage analysis but still manage to reveal the potential problem. Atollic TrueANALYZER analyzes the code coverage types that are most stringent without putting in any extra efforts from either the tester or the developer (Spinellis 43). Conclusion In order to achieve a quality test while measuring embedded systems, thorough test have to be conducted which in most cases include several testing tasks that are demanding. Such tests are normally difficult to master with regards to the conventional functions basis that have been oriented and the testing methods that are structure-oriented. In addition to this, the automation process has also proved to be problematic. Such a process includes test-cases generation to cover varying structural testing criteria, the compliance test that includes the safety requirements for the system and temporal behavior testing. The Evolutionary Test has proved to be an approach that is promising in the complete automation of the task involving complex testing. The testing process allows for complete test case automation for structural testing, the temporal behavior testing and the safety properties testing. Evolutionary testing has had recommended results in the past with reference to the three mentioned application areas (McConell). Following the Evolutionary Test’s complete automation, an embedded system can be easily tested with large different input situation numbers both in terms of safety tests and temporal behavior testing. Such a process involves the generation of thousands of sets of test data which are then executed with the process lasts only for a few minutes. In case the specified constraints are not violated in any way, then the tester can find confidence in the system’s correct functioning as this will have increased to a great extent. Evolutionary Test brags of very few prerequisites in its application. The testing process requires the system’s interface specification that is being tested to guarantee valid input values generation. The structural testing process makes use of the test object’s source code as one of the requirements (Halstead 51). Evolutionary Test application has been proved to be successfully in a number of case studies including; industrial applications in the engine electronics field where the testing yielded desirable results, efficiency and effectiveness test process have been improved in different fields through the use of Evolutionary Test, the process has also been part of quality improvement and development cost reduction and robustness and functional tests. Evolutionary testing has also played a big role in evolutionary structural testings that are currently underway with more emphasis being put on programs testability basis assessment on software metrics that have been statistically determined. With the use of the most appropriate information, the selection of the evolutionary algorithms that is most suitable for the test is possible (Garmus 78). Works Cited Baresel, Amstrong. Automation of Structural Testing using Evolutionary Algorithms. Berlin: Humboldt University Press, 2000. Print. Beizer, Betty. Software Testing Technique. New York: Nostrand Reinhold, 1999. Print Boehm, Bobby, et el. Characteristics of Software Quality, Amsterdam: North-Holland, 1997. Print Chidamber, Kemerer. “A Metrics Suite for Object oriented Design.” IEEE Transaction on Software Engineering, 20 (6). 476-493 Conrad, Fey. Model-Based Generation and Structured Representation of Test Scenarios, Proceedings of the Workshop on Software-Embedded Systems Testing. New Jersey: Auerbach Publications, 2007. Print Ebert, Christof & Dumke, Reiner. Software Measurement. Kindle Edition. P.91 EC-Council. Penetration testing: Procedures & Methodologies. New York: Course Technology, 2010. Print Garmus, Herron. Function Point Analysis. New Jersey: Addison Wesley, 2001. Print Graff, Miller & Wyk, Ken. Secure Coding: Principles and Practices. California: O’Reilly, 2003. Print Grchtmann, Wenger. Evolutionary Testing of Temporal Correctness. Berlin: Humboldt University Press, 2005. Print. Gross, Jones & Eyres Davids. “Structural performance measure of evolutionary testing applied to worst-case timing of real-time sytstems.”IEE Proc, Vol. 147, No. 2, pp 25-30 Halstead, Emmanuel. Elements of Software Science. Amsterdam: Elsevier North-Holland, 1997.Print Hamill, Goseva. “Common faults in Software fault and failure data.” IEEE Transactions of Software Engineering, 35(4): 484-496 Hetzel, Welington. The Complete Guide to Software Testing. Wellesley: Information Science, 1998. Print Jackson, Donald. “A direct path to dependable software.” Communication of the ACM, 52(4) Jones, Beril and Sthamer, Heith. “A strategy for using Genetic Algorithms to Automate Branch and Fault-based Testing.” The Computer Journal, Vol. 41, 2: 234-278 Jones, Beril and Sthamer, Heith. The Automatic Genertaion of Software Test Data Sets using Adaptive Search Techniques. New Jersey: Auerbach Publications, 2007. Print Kaner, Cain & Hung, Nelson. Testing Computer Software. New York: John Wiley & Sons, 1999. Print Lehman, Eliene. Time Partition Testing: A Method for Testing Dynamic Functional Behavior. London: Oxford University Publishers 2000, Print. Lehmann, Eliene and Wegener, Jessica. Test Case Design by Means of the CTE XL. New Jersey: Auerbach Publications, 2007. Print Marciniak, John. Encyclopedia of Software Engineering. New York: John Wiley & Sons, 1999. Print Marick, Beril. The Craft of Software Testing. Prentice Hall: New Jersey, 1994. Print Martin, Ronny. “Managing vulnerabilities in networked systems.” IEEE Computer McCabe, Tonny. A complexity measure.” IEEE Transaction Software Engineering, 4: 123-167 McConell, Steve. Code Complete. New york: Microsoft Press, 1993.Print McGraw, Gerald & Potter, Becky. “ Software Security Testing.” IEEE Security and Privacy, Vol. 2: 81-85 Menachem, Ben. Software Quality, Producing Practical and Consistent Software. New York: Thomson Computer Press, 1999. Print Mueller, Funnel & Wegener Jessica. A Comparison of Static Analysis and Evolutionary Testing for Verification of Timing Constraints. Berlin: Humboldt University Press, 2001. Print. Mueller, Funnel. Generalizing Timing Predictions to Set-Associative Caches. New York: Wiley, 1997, Print Puscher, Phylvan & Nossal, Richards. “ Testing the Results of Static Worst-Case Execution-Time Analysis.” Real-Time Systems Symposium, 134-143 Puschner, Peter & Vrchoticky, Allan. Problems in Static Worst-Case Execution Time Analysis. Berlin: Humboldt University Press, 2000. Print. Redmil, Faith. “Exploring Risk-Based Testing and Its Implications.” Software Testing, Verification and Reliability, Vol. 14:3-15 Schultz, Collins & De Jong. “Test and Evaluation by Genetic Algorithms.” IEEE Expert 8: 9-14 Spinellis, Daniels. Code Quality. New York: Addison Wesley, 2006.Print Sthamer, Hillary. The Automatic Generation of Software Test Data Using Genetic Algorithms. Wales: Pontyprid, 1996, Print Tracey, Nocholas. An Automated Framework for Structural Test-Data Generation. Wales: Pontyprid, 1996, Print Viega, Jensen & McGraw, Gerald. Building Secure Software. Boston : Addison Wesley, 2004. Print Read More