This study evaluates a theory of subsumption relations among mutants created by the ROR mutation operator, thus making most of these mutants redundant. A redundant mutant can be skipped during mutation analysis without decreasing the quality of the resulting test suite. This theory is interesting since mutation testing is computationally expensive and the theory states that the set of ROR mutants can be reduced by 57%. The reduced set of ROR mutants has therefore, been used in several recent studies. However, we provide proof that this theory do not hold for strong mutation and that part of the theory is incorrect. The theory itself has to our knowledge, never before been evaluated empirically.
By finding counter examples, we prove that a test suite, which is 100% adequate for the non-redundant ROR mutants might not be 100% adequate for the mutants, which are supposed to be redundant. The subsumption relations do not hold for strong mutation. We have also proved that more than one top-level mutant can be detected by the same test. This should not be possible according to the theory. Hence, this part of the theory is incorrect, independent of strong or weak mutation.
Our findings are important since strong mutation is frequently used to evaluate test suites and testing criteria. Just as redundant mutants can give an overestimation of the mutation score for a test suite, using the reduced set can give an underestimation. Results reported from such studies should therefore, be accompanied by information on whether the reduced or complete set of ROR was used and if the researchers used strong or weak mutation.
HS-IIT-TR-15-002