Assessing the Impact of Using Fault Prediction in Industry

Assessing the Impact of Using Fault Prediction in Industry,10.1109/ICSTW.2011.75,Robert M. Bell,Elaine J. Weyuker,Thomas J. Ostrand

Assessing the Impact of Using Fault Prediction in Industry  
BibTex | RIS | RefWorks Download
Do fault prediction models that guide testing and other efforts to improve software reliability lead to finding different or additional faults in the next release, to an improved process for finding the same faults that would occur were the models not used, or do they have no impact at all? In this challenge paper, we describe the difficulties involved in estimating effects of this sort of intervention and discuss ways to empirically answer that question and ways of assessing any changes, if present. We present several experimental design options and discuss the pros and cons of each. Keywords-experimental design; estimating causal effects; software faults; software testing; industrial systems I. INTRODUCTION For the last several years, we have been developing fault prediction models based on historical information about code changes and defects as well as static code characteristics such as size and programming language. We have performed substantial empirical studies to validate the effectiveness of these models and generally found them to be very good at identifying the files likely to contain the largest numbers of faults in the next release. Because the models have not been used to provide real-time feedback to software testers or developers for any of these systems, we have not been able to observe the effects of providing the predictions in practice. Instead, these past studies indicate the potential for impact if testers can use the information to redirect their searches for faults. We have built a tool that automatically extracts the nec- essary data from a project data base, builds the models, and provides the user with an ordered list of the files predicted to have the most defects in the next release (3). Use of this tool on active software systems offers the promise of assessing the actual effects of feedback from the models. A key question for potential users is whether accurate fault predictions can improve the efficiency of software testing. By this we mean, can they increase the ratio of faults detected to effort expended, compared to what would be achieved without the benefit of such predictions. It is also important to assess whether any such effects decay over the course of time. In addition, we would like to be able to determine whether there is a change in the nature of the faults detected. Although there are plausible arguments about why the use of prediction models should or should not affect the faults made and faults detected, they are not a substitute for solid empirical evidence. In this challenge paper, we describe the difficulties involved in estimating the effects of using fault predictions and discuss the pros and cons of possible experimental designs for doing so.
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.