Högskolan i Skövde

his.sePublications
Change search
Link to record
Permanent link

Direct link
Grindal, Mats
Publications (8 of 8) Show all publications
Grindal, M. (2007). Handling Combinatorial Explosion in Software Testing. (Doctoral dissertation). Linköping University, Department of Computer and Information Science
Open this publication in new window or tab >>Handling Combinatorial Explosion in Software Testing
2007 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

In this thesis, the overall conclusion is that combination strategies, (i.e., test case selection methods that manage the combinatorial explosion of possible things to test), can improve the software testing in most organizations. The research underlying this thesis emphasizes relevance by working in close relationship with industry.

Input parameter models of test objects play a crucial role for combination strategies. These models consist of parameters with corresponding parameter values and represent the input space and possibly other properties, such as state, of the test object. Test case selection is then defined as the selection of combinations of parameter values from these models.

This research describes a complete test process, adapted to combination strategies. Guidelines and step-by-step descriptions of the activities in process are included in the presentation. In particular, selection of suitable combination strategies, input parameter modeling and handling of conflicts in the input parameter models are addressed. It is also shown that several of the steps in the test process can be automated.

The test process is validated through a set of experiments and case studies involving industrial testers as well as actual test problems as they occur in industry. In conjunction with the validation of the test process, aspects of applicability of the combination strategy test process (e.g., usability, scalability and performance) are studied. Identification and discussion of barriers for the introduction of the combination strategy test process in industrial projects are also included.

This research also presents a comprehensive survey of existing combination strategies, complete with classifications and descriptions of their different properties. Further, this thesis contains a survey of the testing maturity of twelve software-producing organizations. The data indicate low test maturity in most of the investigated organizations. Test managers are often aware of this but have trouble improving. Combination strategies are suitable improvement enablers, due to their low introduction costs.

Place, publisher, year, edition, pages
Linköping University, Department of Computer and Information Science, 2007
Series
Linköping studies in science and technology. Dissertations, ISSN 0345-7524 ; 1073
Keywords
Input parameter models, software development, software testing, test process
National Category
Software Engineering
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-1994 (URN)978-91-85715-74-9 (ISBN)
Public defence
(English)
Available from: 2008-04-21 Created: 2008-04-21 Last updated: 2020-05-07
Lindström, B., Nilsson, R., Ericsson, A., Grindal, M., Andler, S. F., Eftring, B. & Offutt, J. (2007). Six Issues in Testing Event-Triggered Real-Time Systems. Skövde: Institutionen för kommunikation och information
Open this publication in new window or tab >>Six Issues in Testing Event-Triggered Real-Time Systems
Show others...
2007 (English)Report (Other academic)
Abstract [en]

Verification of real-time systems is a complex task, with problems coming from issues like concurrency. A previous paper suggested dealing with these problems by using a time-triggered design, which gives good support both for testing and formal analysis. However, a

time-triggered solution is not always feasible and an event-triggered design is needed. Event-triggered systems are far more difficult to test than time-triggered systems.

This paper revisits previously identified testing problems from a new perspective and identifies additional problems for event-triggered systems. The paper also presents an approach to deal with these problems. The TETReS project assumes a model-driven development

process. We combine research within three different fields: (i) transformation of rule sets between timed automata specifications and ECA rules with maintained semantics, (ii) increasing testability in event-triggered system, and (iii) development of test case generation methods for event-triggered systems.

Place, publisher, year, edition, pages
Skövde: Institutionen för kommunikation och information, 2007. p. 10
Series
IKI Technical Reports ; HS-IKI-TR-07-005
National Category
Information Systems
Identifiers
urn:nbn:se:his:diva-1269 (URN)
Available from: 2008-06-17 Created: 2008-06-17 Last updated: 2018-09-07Bibliographically approved
Grindal, M., Lindström, B., Offutt, J. & Andler, S. F. (2006). An Evaluation of Combination Strategies for Test Case Selection. Empirical Software Engineering, 11(4), 583-611
Open this publication in new window or tab >>An Evaluation of Combination Strategies for Test Case Selection
2006 (English)In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 11, no 4, p. 583-611Article in journal (Refereed) Published
Abstract [en]

This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.

Place, publisher, year, edition, pages
Springer, 2006
Keywords
Combination strategies, Orthogonal arrays, AETG, Test case selection, Testing experiment
National Category
Engineering and Technology
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-1995 (URN)10.1007/s10664-006-9024-2 (DOI)000242805000005 ()
Available from: 2008-04-21 Created: 2008-04-21 Last updated: 2022-09-15Bibliographically approved
Grindal, M., Offutt, J. & Mellin, J. (2006). Handling Constraints in the Input Space when Using Combination Strategies for Software Testing. Skövde: Institutionen för kommunikation och information
Open this publication in new window or tab >>Handling Constraints in the Input Space when Using Combination Strategies for Software Testing
2006 (English)Report (Other academic)
Abstract [en]

This study compares seven different methods for handling constraints in input parameter models when using combination strategies to select test cases. Combination strategies are used to select test cases based on input parameter models. An input parameter model is a representation of the input space of the system under test via a set of parameters and values for these parameters. A test case is one specific combination of values for all the parameters. Sometimes the input parameter model may contain parameters that are not independent. Some sub-combinations of values of the dependent parameters may not be valid, i.e., these sub-combinations do not make sense. Combination strategies, in their basic forms, do not take into account any semantic information. Thus, invalid sub-combinations may be included in test cases in the test suite. This paper proposes four new constraint handling methods and compares these with three existing methods in an experiment in which the seven constraint handling methods are used to handle a number of different constraints in different sized input parameter models under three different coverage criteria. All in all, 2568 test suites with a total of 634,263 test cases have been generated within the scope of this experiment.

Place, publisher, year, edition, pages
Skövde: Institutionen för kommunikation och information, 2006. p. 39
Series
IKI Technical Reports ; HS-IKI-TR-06-001
National Category
Information Systems
Identifiers
urn:nbn:se:his:diva-1263 (URN)
Available from: 2008-06-17 Created: 2008-06-17 Last updated: 2018-09-07Bibliographically approved
Grindal, M., Offutt, J. & Mellin, J. (2006). On the Testing Maturity of Software Producing Organizations. In: Proceedings - Testing: Academic and Industrial Conference: Practice and Research Techniques, TAIC PART 2006. Paper presented at Practice And Research Techniques (TAIC PART 2006), August 29-31, Cumberland Lodge, Windsor, UK (pp. 171-180). Los Alamitos, California: IEEE Computer Society
Open this publication in new window or tab >>On the Testing Maturity of Software Producing Organizations
2006 (English)In: Proceedings - Testing: Academic and Industrial Conference: Practice and Research Techniques, TAIC PART 2006, Los Alamitos, California: IEEE Computer Society, 2006, p. 171-180Conference paper, Published paper (Other academic)
Abstract [en]

This paper presents data from a study of the current state of practice of software testing. Test managers from twelve different software organizations were interviewed. The interviews focused on the amount of resources spent on testing, how the testing is conducted, and the knowledge of the personnel in the test organizations. The data indicate that the overall test maturity is low. Test managers are aware of this but have trouble improving. One problem is that the organizations are commercially successful, suggesting that products must already be "good enough". Also, the current lack of structured testing in practice makes it difficult to quantify the current level of maturity and thereby articulate the potential gain from increasing testing maturity to upper management and developers

Place, publisher, year, edition, pages
Los Alamitos, California: IEEE Computer Society, 2006
Identifiers
urn:nbn:se:his:diva-1919 (URN)10.1109/TAIC-PART.2006.20 (DOI)000242106900020 ()2-s2.0-70349943447 (Scopus ID)0-7695-2672-1 (ISBN)978-0-7695-2672-0 (ISBN)
Conference
Practice And Research Techniques (TAIC PART 2006), August 29-31, Cumberland Lodge, Windsor, UK
Available from: 2007-09-21 Created: 2007-09-21 Last updated: 2020-03-19Bibliographically approved
Grindal, M., Offutt, J. & Andler, S. F. (2005). Combination Testing Strategies: A Survey. Software testing, verification & reliability, 15(3), 167-199
Open this publication in new window or tab >>Combination Testing Strategies: A Survey
2005 (English)In: Software testing, verification & reliability, ISSN 0960-0833, E-ISSN 1099-1689, Vol. 15, no 3, p. 167-199Article in journal (Refereed) Published
Abstract [en]

Combination strategies are test case selection methods that identify test cases by combining values of the different test object input parameters based on some combinatorial strategy. This survey presents 16 different combination strategies, covering more than 40 papers that focus on one or several combination strategies. This collection represents most of the existing work performed on combination strategies. This survey describes the basic algorithms used by the combination strategies. Some properties of combination strategies, including coverage criteria and theoretical bounds on the size of test suites, are also included in this description. This survey paper also includes a subsumption hierarchy that attempts to relate the various coverage criteria associated with the identified combination strategies

Place, publisher, year, edition, pages
John Wiley & Sons, 2005
National Category
Engineering and Technology
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-1694 (URN)10.1002/stvr.319 (DOI)000232009700003 ()2-s2.0-24944512941 (Scopus ID)
Available from: 2007-08-13 Created: 2007-08-13 Last updated: 2017-12-12Bibliographically approved
Grindal, M. (2004). Thesis Proposal: Evaluation of Combination Strategies for Practical Testing. Skövde: Institutionen för kommunikation och information
Open this publication in new window or tab >>Thesis Proposal: Evaluation of Combination Strategies for Practical Testing
2004 (English)Report (Other academic)
Abstract [en]

A number of combination strategies have been proposed during the last fifteen years. Combination strategies are test case selection methods where test cases are identified by combining interesting values of the test object's input parameters. Although some results, achieved from small isolated experiments and investigations, point in the direction that these methods are useful in practical testing. Few attempts have been made to investigate these methods under realistic testing conditions. We outline a thesis proposal that is an attempt to determine if combination strategies are feasible alternatives to the currently used test case selection methods in practical testing.

For combination strategies to be feasible alternatives to use in practical testing we require two things. Firstly, the combination strategies need to be effective in finding faults, at least as effective as currently used methods. Secondly, the cost per fault found when using combination strategies should not exceed the corresponding cost for the currently used methods.

To investigate the effectiveness and efficiency of combination strategies we need to establish a benchmark from practical testing and then compare that with how combination strategies perform in the same or similar situations.

Further, we need a testing process targeted for the use of combination strategies to be able to assess the complete cost of using combination strategies. Thus, an important part of this research project is to develop a combination strategies testing process. In particular, the activities of selecting combination strategies to use and transforming the requirements on the test object into a format suitable for combination strategies are focused on. These activities are specific to combination strategies and not very well understood.

The methods used for achieving our research goal include literature surveys, investigation of the state-of-practice, with respect to used test case selection methods and cost of testing, experiments, tool implementations, and proof-of-concept, in the form of a case study. In addition to the direct results of our investigations we expect this research to result in detailed information about how to use the suggested test process. This information will include work instructions covering the manual parts. The process information will also include functional descriptions of the tools as well as interface descriptions of the input and output formats of each tool. These tool descriptions will make the test process generic in the sense that alternative tool implementations can be evaluated keeping everything else constant.

Place, publisher, year, edition, pages
Skövde: Institutionen för kommunikation och information, 2004. p. 62
Series
IKI Technical Reports ; HS-IKI-TR-04-003
National Category
Information Systems
Identifiers
urn:nbn:se:his:diva-1264 (URN)
Available from: 2008-06-17 Created: 2008-06-17 Last updated: 2018-09-07Bibliographically approved
Lindström, B., Grindal, M. & Offutt, J. (2004). Using an Existing Suite of Test Objects: Experience from a Testing Experiment. In: ACM SIGSOFT Software Engineering Notes: SECTION: Workshop on empirical research in software testing papers. Paper presented at Workshop on empirical research in software testing papers (WERST) (pp. 1-3). , 29
Open this publication in new window or tab >>Using an Existing Suite of Test Objects: Experience from a Testing Experiment
2004 (English)In: ACM SIGSOFT Software Engineering Notes: SECTION: Workshop on empirical research in software testing papers, 2004, Vol. 29, p. 1-3Conference paper, Published paper (Other academic)
Abstract [en]

This workshop paper presents lessons learned from a recent experiment to compare several test strategies. The test strategies were compared in terms of the number of tests needed to satisfy them and in terms of faults found. The experimental design and conduct are discussed, and frank assessments of the decisions that were made are provided. The paper closes with a summary of the lessons that were learned.

Series
Software Engineering Notes, ISSN 0163-5948, E-ISSN 1943-5843 ; 29(5)
Keywords
Software testing, fault seeding, representativity, independence
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-1505 (URN)10.1145/1022494.1022535 (DOI)
Conference
Workshop on empirical research in software testing papers (WERST)
Available from: 2007-07-02 Created: 2007-07-02 Last updated: 2017-11-27Bibliographically approved
Organisations

Search in DiVA

Show all publications