Academic
Publications
A practical guide for using statistical tests to assess randomized algorithms in software engineering

A practical guide for using statistical tests to assess randomized algorithms in software engineering,10.1145/1985793.1985795,Andrea Arcuri,Lionel C.

A practical guide for using statistical tests to assess randomized algorithms in software engineering   (Citations: 9)
BibTex | RIS | RefWorks Download
Randomized algorithms have been used tosuccessfully address many different types of software engineering problems. This type of algorithms employ a degree of randomness as part of their logic. Randomized algorithms are useful for difficult problems where a precise solution cannot be derived in a deterministic way within reasonable time. However, randomized algorithms produce different results on every run when applied to the same problem instance. It ishence important to assess the effectiveness of randomized algorithms by collecting data from a large enough number of runs. The use of rigorous statistical tests is then essential to provide support to the conclusions derived by analyzing such data. In this paper, we provide a systematic review of the use of randomized algorithms in selected software engineering venues in 2009. Its goal is not to perform a complete survey but to get a representative snapshot of current practice in software engineering research. We show that randomized algorithms are used in a significant percentage of papers but that, in most cases, randomness is not properly accounted for. This casts doubts on the validity of most empirical results assessing randomized algorithms. There are numerous statistical tests, based on different assumptions, and it is not always clear when and how to use these tests. We hence provide practical guidelines to support empirical research on randomized algorithms in software engineering.
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
    • ...Once our prototype will be able to handle different specific Event-B characteristics, as extracted from the model repository, and once we have implemented different fitness functions, a detailed comparison among the fitness functions but also against the model checking approach will be performed following a sound statistical approach [19]...

    Alin Stefanescuet al. Towards Search-Based Testing for Event-B Models

    • ...To compare whether a configuration A is better than another configuration B on a branch, we follow the follow procedure, as described in more detail by Arcuri and Briand [25]...
    • ...In case there is no statistical difference in the success rates, we can analyze the time an algorithm takes to find an optimal solution for the runs in which it is successful [25]...
    • ...b observations/values). As discussed in [25], we use a Mann-Whitney U-test (with # =0 .05 )t o asses which conf iguration requires less computational effort to find optimal solutions...
    • ...This a very large number of comparisons, which can lead to a high probability of Type I error [25] if we consider the hypothesis that all tests are significant at the same time...
    • ...We do not use corrections such as the Bonferroni one, for reasons that are discussed in detail and at length in [25]...

    Gordon Fraseret al. It is Not the Length That Matters, It is How You Control It

    • ...1) Experiment Design: We designed our experiment using the guidelines proposed in [7, 28]...
    • ...We used odds ratio [28] for this purpose, as the results of our experiments are dichotomous...
    • ...We chose Fisher’s exact test because it is appropriate for dichotomous data where proportions must be compared, thus matching our case [28]...

    Shaukat Aliet al. A Search-Based OCL Constraint Solver for Model-Based Test Data Generat...

    • ...To analyze these data by taking into account the random components of the techniques, we followed a rigorous statistical procedure [8]...

    Rohan Sharmaet al. Testing Container Classes: Random or Systematic?

    • ...A rigorous statistical procedure has been used to evaluate and compare the effectiveness of these randomized algorithms [14]...
    • ...g statistical tests to assess ware engineering [14]...

    Hadi Hemmatiet al. Empirical Investigation of the Effects of Test Suite Properties on Sim...

Sort by: