A topic on software testing - Thesis Proposal Example

Measuring test quality in Embedded Systems Introduction Modern systems and products are largely based on ‘embedded systems’ to facilitate control, action and enhanced functionality. In fact, embedded systems used in these scenarios comprise both hardware and software components that work in a highly customized framework to achieve a specific workflow or operation. Although embedded systems are sophisticated computers, they seldom share or provide a similar interface like PCs and are capable of providing a highly dedicated and distinct functionality based on their structure and configuration. A typical application of embedded systems is in precision-driven measurement and control systems, wherein a particular process or parameter such as temperature is constantly monitored and measured through sensors, converted and delivered as a stream of digital signals and processed to modulate the regulation of the underlying process (heat controlled according to predefined constraints). Embedded systems are also in vehicles to measure carbon emissions through a sensory arrangement and processed through a feedback controller network to store and monitor the condition of the engine (Rajski and Tyszer). Over the past few decades, embedded systems have become integral parts of every notable electronic system including safety-critical components. For instance, embedded systems are used to control sensitive and critical functions such as aircraft navigation and ensure failsafe availability of financial networks. Thus, in these scenarios, the primary expectation from these systems is to prevent the slightest occurrence of delay and downtime, which can otherwise be life threatening and financially unacceptable. In fact, Broekman and Notenboom categorize malfunctions as the primary quality issue that plague embedded systems (78). As such, he emphasizes on the relevance and importance of software testing as an effective way of developing high quality systems and products for diverse purposes. Drechsler and Drechsler say that modern software is built on complex code that cannot be tested and evaluated using manual testing techniques (105). Given the huge volume of software that needs to be developed to interact with large hardware structures and requirements, he argues that better tools, which can perform tests efficiently along with better analysis and reporting, are the need of the hour. In other words, software testing must also evolve over time in tune with changing requirements and should be capable of adapting to more complex scenarios. This paper comprises a research proposal aimed at determining the quality of software testing in embedded systems. In addition to a comprehensive literature review, this proposal outlines the various objectives and subject areas that will guide the research process. All techniques and methodologies employed in undertaking the study have also been described apart from a preliminary overview of the expected results. Literature Review Although the embedded systems industry understands the importance of software testing, Fernandes notes that there is a wide disparity over the current quality of various testing methodologies (56). The significance of testing quality is evident especially in the case of embedded systems as any rectification in the form of updates and firmware patches involve difficult steps and are very expensive to design and implement (Kamal, 153). As such, firms emphasize on the importance of testing every feature of embedded systems to the highest levels of confidence. Current limitations Although the importance of testing quality is understood and emphasized by all stakeholders associated with the development and production of embedded systems, the range and sophistication of such systems that exist today is very limited. Most embedded systems are evaluated using methods like debugging, ad-hoc testing and functional validation (black box testing) (Giladi 92). However, embedded systems have certain limitations as they depend on the interaction of both hardware and software for useful operations. Thus, such systems can only be tested upon the integration of both classes of components. This implies that any delays in system development can have significant effects on the time, tolerance and resources available for the testing and quality assurance phases as the delivery times approach (Kobayashi 98). Marwedel states that fault correction costs multiply as the testing progresses along each inherent phase since it is time consuming and tedious to converge to an earlier stage where the fix needs to be provided (146). As such, the emphasis is on identifying a majority of the issues during the preliminary stages of development. According to Pries, modern software development has adopted a test-driven approach over the past decade (178). Under this approach, design and development are continuously evaluated by automated tests that are capable of identifying inconsistencies even during the architectural design of the system (Petrenko 127). Thus, development begins with the writing of tests, which serve as the foundation for the required functionalities and user interfaces. This modular approach also facilitates the incorporation of any name changes (through refactoring methods) and allows the inclusion of new tests as development continues (Pierre 65). As a result, the frequency of bugs can be contained and new features will be evaluated thoroughly to eliminate the introduction of new discrepancies into the system. Despite these apparent benefits, the different characteristics of embedded system develop have restricted the effectiveness with which test based development can be realized (Shibu 160). Unlike conventional software, embedded systems are highly dependent on the underlying hardware, thus adding a new dimension into the list of potential bugs and issues (Noergaard 261). Lee notes that specific hardware is often unavailable to the programmer during the initial phases of the development cycle (45). The use of customized hardware may introduce unforeseen behavior due to the very distinct nature of the hardware platform (Giese 54). Moreover, tests that account for both the hardware and software elements are highly dependent on hardware drivers and may thus have to encompass many more test cases of higher complexity (Givargis 23). In addition, the conventional testing process works concurrently with the base program (Abran 186). For instance, developers write a piece of code and analyze results from the automated tests thereafter. Changes required to remove the detected bugs are developed and incorporated into the program immediately and the testing is performed once again. Thus, this process of writing code, testing and error correction are carried out in a repetitive loop throughout the traditional development cycle. In contrast, the software for embedded systems is developed externally and migrated into the latter, often through a very intricate and time-consuming process. Thus, any traditional testing approach that is fast paced in nature may simply not be feasible in the case of embedded systems. Hommel says that developers will not be able to deploy new updates frequently without acceding to a tradeoff on development time, effort and cost (72). Besides, while traditional software development has evolved towards modular, object-oriented programming and rapid development, a majority of embedded software development is still carried out in C and C++. As a result, Tian says that many automated testing frameworks are unsuitable for evaluating embedded systems given the latter’s reliance on legacy code (378). Industry-wide testing practices Analyzing the root causes of failures and the defects that caused them can positively affect the quality, reliability and availability of systems (Richard 142). It is imperative that the underlying source of a software or hardware defect is identified when a system becomes unavailable. Even if the system has returned to full operation, this root cause information is necessary so that corrective action can be taken to improve the design of the system and to mitigate the effects of defects (Buhrer and Kundig 286). Comprehensive design validation, quality assurance and manufacturing tests, although essential to a products success, cannot possibly anticipate all possible failure mechanisms in the field (Sherman 59). Embedding diagnostics As mentioned previously, several methods exist to test and identify defects and potential pitfalls. Many of these methods can also be used to store such analysis and use them for subsequent benchmarking and analysis (Radetzki 34). For instance, using interrupt handlers to identify software failures and failure logging mechanisms have long been the usual methods of testing hardware platforms. Sally identifies characteristics such as robustness, availability and reliability that must be measured to evaluate performance of embedded systems (128): Robustness refers to the ease with which embedded systems are resilient towards potential problems and catastrophic failures. A system can be made more robust by eliminating the root causes of identified errors and by adding failsafe procedures that can subvert unexpected behavior in real time. An embedded system is considered reliable if and only if it does not relay corrupt information. In fact, there must be adequate mechanisms in place to validate data in real time and correct erroneous information. In extreme cases, reliability rests on restricting the corrupt data and reporting such an occurrence. Availability is the most crucial factor in safety critical systems. Thus, such systems should be resilient even against partial or temporary outages and can be achieved by delinking the malfunctioning component and avoiding an impact on the remaining parts. This ensures minimal availability at the very least. IBM has been the leader within the testing industry in the above approach for many years. Branded as the RAS methodology, this technique has been useful in measuring server efficiency, hardware failures and system diagnostics for IBM’s server products (Ganssle 256). IBM is also active in using embedded diagnostics for the development of nonstop systems. Latest advances utilize probabilistic techniques such multivariate state analysis that are capable of determining potential impact under different scenarios. Hardware chips, that form the core of failsafe systems, contain inbuilt diagnostic features thereby facilitate rapid prototype development much like conventional software programs (Hemida 195). As such, many of today’s embedded devices and hardware chips have onboard circuitry and software that enhance their resilience and availability. The proximity of these elements to the device circuitry is useful in validating every basic feature of the chip from close quarters. One of the simplest examples of such methods is the BIST (Built-in self test) that are utilized under light and temperature control systems (Kleinjohann 83). Modern instrumentation frameworks for embedded devices are moving towards compatibility with popular processor platforms (like Intel and AMD) to leverage the availability of high-speed I/O (Input-Output) operations for faster testing and error resolution (Luo 28). Other methods used by in-built testing instruments include state transition maps and circuit analysis for effective debugging and error detection (Schmidt 54). By injecting errors into various circuit paths, these techniques can then identify erroneous circuit segments by estimating the signal eye width (Wu 187). Additional methods consist of analyzing network packets and circuit performance, which have proven equally capable at improving the quality of embedded devices. Objectives and Significance The increasing pace at which embedded systems are being developed to fulfill complex requirements, together with the increasing use of high-density hardware chips is expected to enhance the importance of reliability and resilience in the coming years. In the current scenario, even a minimal downtime is not acceptable in the smooth operation of servers running financial software. Without this, even small outages are likely to have a catastrophic loss on the markets. The failure of a single chip should not cause the failure of the entire navigation system of an aircraft or space shuttle. Richard notes that the modern expectation from embedded systems is to have a high level of recoverability in the case of unexpected behavior. This implies that engineers must reinvent the methods of testing such systems and develop better alternatives that can provide alternate operational channels to existing workflows (97). Sally says that many system manufacturers have realized the benefits of embedding the testing logic into the silicon substrate of embedded systems, which can be useful in diagnosing defects during normal operational routines and initiate preemptive failsafe routines in such instances. The proposed study aims to identify and develop related techniques that can be used to test and evaluate the quality of embedded systems. The study will also undertake an analysis of the economic benefits provided by features such as feedback loops and failure recovery. The research shall also explore opportunities where existing testing methods may be optimized to suit the evolving needs of the industry. In doing so, the study shall also establish specific benchmarks that need to be realized by embedded devices that belong to the safety critical category. With the increasing use of virtual computing and cloud networks, quality control methods need to be devised that can leverage the benefits delivered by these new mediums of communication and software development. With growing interconnection and intelligence of embedded systems, it is thus necessary that testing methods accommodate these emerging trends. References Abran, Alain. Software process and product measurement: international conferences. IWSM 2008, MetriKon 2008, and Mensura 2008. Munich, Germany, November 18-19, 2008. Atkinson, Colin. Component-based software development for embedded systems: an overview of current research trends. Berlin: Birkhäuser, 2005. Bin, Eyal. Hardware and software, verification and testing. New York: Springer, 2007. Broekman, Bart, and Notenboom, Edwin. Testing embedded software. New York: Pearson Education, 2003. Print. Broy. Model-based testing of reactive systems: advanced lectures. New York: Springer, 2005. Buhrer, Richard, and Kundig, Albert. Embedded systems: new approaches to their formal description and design. London: Routledge, 2007. Print. Chis, Monica. Evolutionary computation and optimization algorithms in software engineering: applications and techniques. London: Idea Group, 2010. Drechsler, Rolf, and Drechsler, Nicole. Evolutionary algorithms for embedded system design. New York: Springer, 2007. Print. Fernandes, Joao. Behavioral modeling for embedded systems and technologies: applications for design and implementation. Boston: Idea Group Inc, 2009. Print. Ganssle, Jack. Embedded systems. Chicago: Newnes, 2007. Print. Richard, Jack. The art of designing embedded systems. Chicago: Newnes, 2008. Print. Giese, Holger. Model-Based Engineering of Embedded Real-Time Systems. New York: Springer, 2010. Print. Giladi, Ran. Security and embedded systems. London: IOS Press, 2006. Print. Givargis, Tony, and Russo, Stefano. Software technologies for embedded and ubiquitous systems: 6th IFIP WG 10.2 International Workshop. SEUS: 2008. Print. Hemida, Roman. Advanced techniques for embedded systems design and test. London: Routledge, 2008. Print. Hommel, Gunter. Embedded systems - modeling, technology, and applications. Technische Universität Berlin, 2006. Print. Kamal, Raj. Embedded systems: architecture, programming and design. Delhi: McGraw Hill, 2010. Print. Kleinjohann, Bernd. Architecture and design of distributed embedded systems. New York: Springer, 2005. Print. Kobayashi, Naoki. Programming languages and systems: 4th Asian symposium, APLAS 2006. Sydney, 2006. Print. Lee, Roger. Software Engineering Research, Management and Applications 2010. Computational Intelligence, 2010. Print. Luo, Yuhua. Cooperative Design, Visualization, and Engineering: 6th International Conference. CDVE 2009, Luxembourg, Luxembourg, September 20-23, 2009. Print. Marwedel, Peter. Embedded System Design: Embedded Systems Foundations of Cyber-Physical Systems. New York: Springer, 2010. Print. Noergaard, Tammy. Embedded systems architecture: a comprehensive guide for engineers and programmers. Chicago: Newnes, 2005. Print. Notenboom, Edwin. Testing embedded software. New York: Pearson Education, 2006. Print. Petrenko, Alexandre. Formal approaches to software testing: Third International Workshop on Formal Approaches to Testing of Software : FATES 2003. Montreal, Quebec, Canada, 2003. Print. Pierre, Laurence. Correct hardware design and verification methods. New York: Springer, 2009. Print. Pries, Kim. Testing Complex and Embedded Systems. London: CRC Press, 2010. Print. Radetzki, Martin. Languages for Embedded Systems and Their Applications: Selected Contributions on Specification, Design, and Verification from FDL08. New York: Springer, 2009. Print. Rajski, Janusz, and Tyszer, Jerzy. Arithmetic built-in self-test for embedded systems. University of Michigan, 2008. Print. Roychoudhry, Abhik. Embedded systems and software validation. Berlin: Morgan Kaufmann, 2009. Print. Sally, Gene. Pro Linux Embedded Systems. Bristol: APress, 2009. Schmidt, Andrew. Embedded Systems Design with Platform FPGAs: Principles and Practices. Berlin: Morgan Kaufmann, 2010. Print. Sherman, Ripplinger. Using quality attributes to identify and specify embedded system requirements. Arizona State University, 2008. Print. Shibu, Auteur. Intro To Embedded Systems 1E. New York: McGraw Hill, 2009. Print. Takanen, Ari. Fuzzing for software security testing and quality assurance. New York: Artech House, 2008. Tian, Jeff. Software quality engineering: testing, quality assurance, and quantifiable improvement. New York: John Wiley, 2007. Print. Wu, Zhaohui. Embedded software and systems. Berlin: Birkhauser, 2009. Print. Research Contents Quality Assurance Maintaining effective control over the process of embedded systems development is a challenging task for any team. As described in the preceding sections, companies face many hurdles in securing a clear overview of the development process, and thus face numerous issues in monitoring the design and development process. Thus, the proposed study shall analyze embedded system testing as a process and identify deficient areas, which lead to slippage and losses for customers (Bin 186). Thus, a major portion of this study will be dedicated towards developing a new framework for assuring system quality and shall ascertain whether this needs to be a dedicate function that is distinct from other roles associated with embedded systems development (Drechsler 87). It is necessary to dedicate some efforts towards quality assurance as this helps develop contingency mechanisms should any undesirable behavior continue to occur within the embedded system despite rigorous testing. The study shall therefore determine the level of ‘eyesight’ that the team needs to be provided, especially during the testing phase. This is necessary if the behavior resulting from the injection of a defective test case needs to be detected, stored and analyzed for subsequent improvement (Kamal, 145). Quality assurance thus holds utmost importance for the propose study as it is seen as a confidence building measure as part of any proposed testing framework. Expected features of proposed framework In addition to containing features that facilitate the evaluation of complex requirements, the proposed framework shall ensure the reduction of all costs associated with quality assurance. This shall be achieved through two unique methods. As mentioned previously, all solutions shall be based on model-driven testing methodologies that are capable of generating test cases in an automated fashion throughout all stages of the development lifecycle. Using test-driven development from the early stages will help eliminate large potential errors from the very outset besides helping optimize the underlying code based on space and time complexities (Giese 214). Apart from automated test case generation and verification, it is believed that frequent manual testing is also necessary to ensure the integrity and operational efficiency of the testing process. Using static analyses, test engineers will be able to evaluate the new testing framework and the inherent models that drive the automated testing (Radetzki 95). The specific details of such static analyses will be developed during the actual study. The proposed framework shall also include innovative techniques for performance evaluation, test data storage and monitoring of system execution. Given the importance of the underlying processor architecture for effective system performance, the proposed study shall also develop the foundations for a hardware emulator that can help in testing at the machine code level and ensure conformity between the architectural design and the code under execution. At a higher level, the software that powers the system shall be evaluated for reliability and reusability (Roychoudhry 196). Additionally, procedures for evaluating the interfaces that allow software communication with the hardware shall be developed. In such a scenario, the framework is expected to switch between assembly and high-level code at the user’s convenience. All test cases under the proposed framework will monitor results that should conform to a maximum, minimum and an average benchmark. For example, the response times for a safety critical system will be compared against a predefined benchmark to determine whether the embedded system is capable of responding within a required time band (Notenboom 279). Additionally, the average response time will provide a sense of directionality to the user and determine the potential for crossing the desired barrier. Wherever possible, such benchmarks will be configurable in both absolute and relative terms. In addition, measures to ascertain the percentage of times when the system responds within the predetermined benchmark will be developed. In safety critical system, this ratio can be as high as 99% (Pries 38). Research Methods Since the proposed study aims to develop new methods to evaluate modern embedded systems, it is necessary to understand similar ideas developed by industry peers and researchers. For this purpose, the study shall utilize information from both primary and secondary sources of information. Primary sources for this study will comprise professionals currently working in the embedded devices industry who can offer critical insights into the latest trends and developments in this area. Besides, interviewing professionals will also provide useful information on some of the issues that professionals face when determining and improving the quality of their systems. In addition to this knowledge, the study also intends to source related information from a number of secondary sources including books, journals, magazine articles and any other relevant sources. Thus, the proposed study will seek to understand the future of embedded system testing from both theoretical and practical perspectives and work towards developing a new methodology that can fulfill the needs of both stakeholders. The study shall also include the development of a suitable algorithm that will be used to demonstrate the usefulness of the proposed testing method(s). Given the highly technical nature of the analysis, the proposed research shall rely on both qualitative and quantitative methods to evaluate the proposed testing framework(s). More specifically, the study proposes to use a value-based approach in addition to dependency analysis in order to evaluate the effectiveness of the new framework. For instance, a set of embedded systems can be tested using the new framework and the resulting outcomes can then be used to determine a risk exposure estimate (Pierre 240). The magnitude of this risk exposure can be deduced based on the error in the outcome and the probability of obtaining such erroneous output. Additionally, the improvement in the testing model will also be analyzed by calculating the outcome after error correction and the outcome prior to such resolution. The difference in these results can help understand the cost-benefit ratio of a given code structure or hardware logic (Kleinjohann 46). In addition, the new framework shall also provide a dependency among test cases. Evaluating this dependency is important is necessary to prioritize test cases that provide the highest benefit per unit costs (Atkinson 154). Many other research methods that can help improve the efficiency of the new testing model will be adopted accordingly. Experiment processes and expected results This section provides a brief overview of the main experiments and tests that will be performed to achieve the objectives described above. As mentioned previously, this study will not focus on either functional or structural testing as it is not possible to generate device or industry specific test cases using either methods. Further, it is also not possible to automate the testing process using these methods due to certain limits on the capabilities of these techniques. Currently, there are no specific tools or techniques to analyze the temporal behavior of systems. Hence, this study shall introduce an evolutionary element into the application of structural testing, thereby eliminating the constraint of manual generation of test cases. Since structural testing is followed widely throughout the industry, finding suitable software packages will not be an issue. However, the evolutionary component will be developed as an algorithm and be used as part of a selected structural testing program in the form of a plugin (or through any other convenient arrangement). Using this approach, test cases can then be generated for testing patterns such as loops, conditions, statements and code branches. Thus, this project aims to attain data sufficiency by helping generating thousands of test cases for every possible scenario and criteria that can influence the underlying embedded system. Read More