Embedded software is at the core of current and future telecommunication, automotive, multimedia, and industrial automation systems. The success of practically any industrial application depends on the embedded software system’s dependability, and one method to verify the dependability of a system is testing its robustness. The motivation behind this study is to provide a knowledge base of the state of the practice in robustness testing of embedded software systems and to compare this to the state of the art. We have gathered information on the state of the practice in robustness testing from seven different industrial domains (telecommunication, automotive, multimedia, critical infrastructure, aerospace, consumer products, and banking) by conducting thirteen semi-structured interviews. We investigate different aspects of robustness testing, such as the general view of robustness, relation to requirements engineering and design, test execution, failures, and tools. We highlight knowledge from the state of the practice of robustness testing of embedded software systems. We found different robustness testing practices that have not been previously described. Our study shows that the state of the practice, when it comes to robustness testing, differs between organizations and is quite different from the state of the art described in the scientific literature. For example, methods commonly described in the literature (e.g., the fuzzy approach) are not used in the organizations we studied. Instead, the interviewees described several ad-hoc approaches that take specific scenarios into account (e.g., power failure or overload). Other differences we found concern classification of robustness failures, the hypothesized root causes of robustness failures, and the types of tools used for robustness testing. The study is a first step in capturing the state of the practice of robustness testing of embedded software systems. The results can be used by both researchers and- practitioners. Researchers can use our findings to understand the gap between the state of the art and the state of the practice and develop their studies to fill this gap. Practitioners can also learn from this knowledge base regarding how they can improve their practice and acquire other practices.
Institute of Electrical and Electronics Engineers (IEEE), 2016. Vol. 4, 1859-1871 p., 7438745