his.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Handling Combinatorial Explosion in Software Testing
University of Skövde, School of Humanities and Informatics. University of Skövde, The Informatics Research Centre.
2007 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

In this thesis, the overall conclusion is that combination strategies, (i.e., test case selection methods that manage the combinatorial explosion of possible things to test), can improve the software testing in most organizations. The research underlying this thesis emphasizes relevance by working in close relationship with industry.

Input parameter models of test objects play a crucial role for combination strategies. These models consist of parameters with corresponding parameter values and represent the input space and possibly other properties, such as state, of the test object. Test case selection is then defined as the selection of combinations of parameter values from these models.

This research describes a complete test process, adapted to combination strategies. Guidelines and step-by-step descriptions of the activities in process are included in the presentation. In particular, selection of suitable combination strategies, input parameter modeling and handling of conflicts in the input parameter models are addressed. It is also shown that several of the steps in the test process can be automated.

The test process is validated through a set of experiments and case studies involving industrial testers as well as actual test problems as they occur in industry. In conjunction with the validation of the test process, aspects of applicability of the combination strategy test process (e.g., usability, scalability and performance) are studied. Identification and discussion of barriers for the introduction of the combination strategy test process in industrial projects are also included.

This research also presents a comprehensive survey of existing combination strategies, complete with classifications and descriptions of their different properties. Further, this thesis contains a survey of the testing maturity of twelve software-producing organizations. The data indicate low test maturity in most of the investigated organizations. Test managers are often aware of this but have trouble improving. Combination strategies are suitable improvement enablers, due to their low introduction costs.

Place, publisher, year, edition, pages
Linköping University, Department of Computer and Information Science , 2007.
Series
Linköping studies in science and technology. Dissertations, ISSN 0345-7524 ; 1073
Keyword [en]
Input parameter models, software development, software testing, test process
National Category
Software Engineering
Research subject
Technology
Identifiers
URN: urn:nbn:se:his:diva-1994ISBN: 978-91-87515-74-9 (print)OAI: oai:DiVA.org:his-1994DiVA: diva2:32270
Public defence
(English)
Available from: 2008-04-21 Created: 2008-04-21 Last updated: 2013-03-26
List of papers
1. Combination Testing Strategies: A Survey
Open this publication in new window or tab >>Combination Testing Strategies: A Survey
2005 (English)In: Software testing, verification & reliability, ISSN 0960-0833, E-ISSN 1099-1689, Vol. 15, no 3, 167-199 p.Article in journal (Refereed) Published
Abstract [en]

Combination strategies are test case selection methods that identify test cases by combining values of the different test object input parameters based on some combinatorial strategy. This survey presents 16 different combination strategies, covering more than 40 papers that focus on one or several combination strategies. This collection represents most of the existing work performed on combination strategies. This survey describes the basic algorithms used by the combination strategies. Some properties of combination strategies, including coverage criteria and theoretical bounds on the size of test suites, are also included in this description. This survey paper also includes a subsumption hierarchy that attempts to relate the various coverage criteria associated with the identified combination strategies

Place, publisher, year, edition, pages
John Wiley & Sons, 2005
National Category
Engineering and Technology
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-1694 (URN)10.1002/stvr.319 (DOI)000232009700003 ()2-s2.0-24944512941 (Scopus ID)
Available from: 2007-08-13 Created: 2007-08-13 Last updated: 2013-04-09Bibliographically approved
2. An Evaluation of Combination Strategies for Test Case Selection
Open this publication in new window or tab >>An Evaluation of Combination Strategies for Test Case Selection
2006 (English)In: Journal of Empirical Software Engineering, ISSN 1382-3256, Vol. 11, no 4, 583-611 p.Article in journal (Refereed) Published
Abstract [en]

 This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.

Keyword
Combination strategies, Orthogonal arrays, AETG, Test case selection, Testing experiment
National Category
Engineering and Technology
Identifiers
urn:nbn:se:liu:diva-12971 (URN)10.1007/s10664-006-9024-2 (DOI)
Available from: 2008-02-29 Created: 2008-02-29
3. Managing Conflicts when Using Combination Strategies to Test Software
Open this publication in new window or tab >>Managing Conflicts when Using Combination Strategies to Test Software
2007 (English)In: Proceedings of 18th Australian Conference on Software Engineering (ASWEC2007), 10-13 April, Melbourne, Australia, 255-264 p.Article in journal (Refereed) Published
Abstract [en]

Testers often represent systems under test in input parameter models. These contain parameters with associated values. Combinations of parameter values, with one value for each parameter, are potential test cases. In most models, some values of two or more parameters cannot be combined. Testers must then detect and avoid or remove these conflicts. This paper proposes two new methods for automatically handling such conflicts and compares these with two existing methods, based on the sizes of the final conflict-free test suites. A test suite reduction method, usable with three of the four investigated methods is also included in the study, resulting in seven studied conflict handling methods. In the experiment, the number and types of conflicts, as well as the size of the input parameter model and the coverage criterion used, are varied. All in all, 3854 test suites with a total of 929, 158 test cases were generated. Two methods stand out as tractable and complementary. The best method (called the avoid methods) with respect to test suite size is to avoid selection of test cases with conflicts. However, this method cannot always be used. The second best method (called the replace method), removing conflicts from the final test suite, is completely general.

Place, publisher, year, edition, pages
IEEE, 2007
National Category
Computer Science
Identifiers
urn:nbn:se:liu:diva-12972 (URN)10.1109/ASWEC.2007.27 (DOI)
Available from: 2008-02-29 Created: 2008-02-29 Last updated: 2010-04-28
4. Input Parameter Modeling for Combination Strategies
Open this publication in new window or tab >>Input Parameter Modeling for Combination Strategies
2007 (English)In: Proceedings of the IASTED International Conference on Software Engineering (SE2007), Feb 13-15, Innsbruck, Austria, 2007, 255-260 p.Conference paper, (Refereed)
Abstract [en]

Combination strategies are test methods that generate test cases based on input parameter models. This paper suggests a structured modeling method used to translate requirements expressed in a general format into an input parameter model suitable for combination strategies.

This paper also describes results from two initial experiments exploring the efficiency and effectiveness of the modeling method. These results indicate that the resulting models may contain enough information to detect the vast majority of faults in the system under test. Further, results indicate that the modeling method is simple enough to use in practical testing.

Keyword
category partition, combinatorial testing, model-based testing, software testing
National Category
Computer Science
Identifiers
urn:nbn:se:liu:diva-12973 (URN)
Available from: 2008-02-29 Created: 2008-02-29
5. Using Combination Strategies for Software Testing in Practice: A Proof-of-Concept
Open this publication in new window or tab >>Using Combination Strategies for Software Testing in Practice: A Proof-of-Concept
2006 (English)Report (Other (popular science, discussion, etc.))
Series
Technical Report HS-IKI-TR-06-010. School of Humanities and Informatics, University of Skövde
National Category
Engineering and Technology
Identifiers
urn:nbn:se:liu:diva-12974 (URN)
Available from: 2008-02-29 Created: 2008-02-29

Open Access in DiVA

No full text

Other links

http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-11116

Search in DiVA

By author/editor
Grindal, Mats
By organisation
School of Humanities and InformaticsThe Informatics Research Centre
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar

Total: 150 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf