Sign in
Author

Conference

Journal

Organization

Year

DOI
Look for results that meet for the following criteria:
since
equal to
before
between
and
Search in all fields of study
Limit my searches in the following fields of study
Agriculture Science
Arts & Humanities
Biology
Chemistry
Computer Science
Economics & Business
Engineering
Environmental Sciences
Geosciences
Material Science
Mathematics
Medicine
Physics
Social Science
Multidisciplinary
Keywords
(10)
Algorithm Design
Confidence Interval
Empirical Research
Practice Guideline
Randomized Algorithm
Software Engineering
Statistical Analysis
Statistical Test
Systematic Review
Effect Size
Subscribe
Academic
Publications
A practical guide for using statistical tests to assess randomized algorithms in software engineering
A practical guide for using statistical tests to assess randomized algorithms in software engineering,10.1145/1985793.1985795,Andrea Arcuri,Lionel C.
Edit
A practical guide for using statistical tests to assess randomized algorithms in software engineering
(
Citations: 9
)
BibTex

RIS

RefWorks
Download
Andrea Arcuri
,
Lionel C. Briand
Randomized algorithms have been used tosuccessfully address many different types of
software engineering
problems. This type of algorithms employ a degree of randomness as part of their logic. Randomized algorithms are useful for difficult problems where a precise solution cannot be derived in a deterministic way within reasonable time. However, randomized algorithms produce different results on every run when applied to the same problem instance. It ishence important to assess the effectiveness of randomized algorithms by collecting data from a large enough number of runs. The use of rigorous statistical tests is then essential to provide support to the conclusions derived by analyzing such data. In this paper, we provide a
systematic review
of the use of randomized algorithms in selected
software engineering
venues in 2009. Its goal is not to perform a complete survey but to get a representative snapshot of current practice in
software engineering
research. We show that randomized algorithms are used in a significant percentage of papers but that, in most cases, randomness is not properly accounted for. This casts doubts on the validity of most empirical results assessing randomized algorithms. There are numerous statistical tests, based on different assumptions, and it is not always clear when and how to use these tests. We hence provide practical guidelines to support
empirical research
on randomized algorithms in software engineering.
Conference:
International Conference on Software Engineering  ICSE
, pp. 110, 2011
DOI:
10.1145/1985793.1985795
Cumulative
Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
(
portal.acm.org
)
(
dl.acm.org
)
(
www.informatik.unitrier.de
)
(
doi.acm.org
)
(
ieeexplore.ieee.org
)
(
ieeexplore.ieee.org
)
More »
Citation Context
(9)
...Once our prototype will be able to handle different specific EventB characteristics, as extracted from the model repository, and once we have implemented different fitness functions, a detailed comparison among the fitness functions but also against the model checking approach will be performed following a sound statistical approach [
19
]...
Alin Stefanescu
,
et al.
Towards SearchBased Testing for EventB Models
...To compare whether a configuration A is better than another configuration B on a branch, we follow the follow procedure, as described in more detail by Arcuri and Briand [
25
]...
...In case there is no statistical difference in the success rates, we can analyze the time an algorithm takes to find an optimal solution for the runs in which it is successful [
25
]...
...b observations/values). As discussed in [
25
], we use a MannWhitney Utest (with # =0 .05 )t o asses which conf iguration requires less computational effort to find optimal solutions...
...This a very large number of comparisons, which can lead to a high probability of Type I error [
25
] if we consider the hypothesis that all tests are significant at the same time...
...We do not use corrections such as the Bonferroni one, for reasons that are discussed in detail and at length in [
25
]...
Gordon Fraser
,
et al.
It is Not the Length That Matters, It is How You Control It
...1) Experiment Design: We designed our experiment using the guidelines proposed in [7,
28
]...
...We used odds ratio [
28
] for this purpose, as the results of our experiments are dichotomous...
...We chose Fisher’s exact test because it is appropriate for dichotomous data where proportions must be compared, thus matching our case [
28
]...
Shaukat Ali
,
et al.
A SearchBased OCL Constraint Solver for ModelBased Test Data Generat...
...To analyze these data by taking into account the random components of the techniques, we followed a rigorous statistical procedure [
8
]...
Rohan Sharma
,
et al.
Testing Container Classes: Random or Systematic?
...A rigorous statistical procedure has been used to evaluate and compare the effectiveness of these randomized algorithms [
14
]...
...g statistical tests to assess ware engineering [
14
]...
Hadi Hemmati
,
et al.
Empirical Investigation of the Effects of Test Suite Properties on Sim...
References
(50)
Mutation Operators for Spreadsheets
(
Citations: 9
)
Robin Abraham
,
Martin Erwig
Journal:
IEEE Transactions on Software Engineering  TSE
, vol. 35, no. 1, pp. 94108, 2009
An evolutionary approach to estimating software development projects
(
Citations: 38
)
Jesús S. Aguilarruiz
,
Isabel Ramos
,
José Cristóbal Riquelme Santos
,
Miguel Toro
Journal:
Information & Software Technology  INFSOF
, vol. 43, no. 14, pp. 875882, 2001
Full Theoretical Runtime Analysis of Alternating Variable Method on the Triangle Classification Problem
(
Citations: 4
)
Andrea Arcuri
,
Birmingham B
Conference:
International Symposium on Search Based Software Engineering  SSBSE
, 2009
BlackBox System Testing of RealTime Embedded Systems Using Random and SearchBased Testing
(
Citations: 5
)
Andrea Arcuri
,
Muhammad Zohaib Z. Iqbal
,
Lionel C. Briand
Conference:
IFIP International Conference on Testing of Communicating Systems  TestCom
, pp. 95110, 2010
Formal analysis of the effectiveness and predictability of random testing
(
Citations: 4
)
Andrea Arcuri
,
Muhammad Zohaib Z. Iqbal
,
Lionel C. Briand
Conference:
International Symposium on Software Testing and Analysis  ISSTA
, pp. 219230, 2010
Sort by:
Citations
(9)
Towards SearchBased Testing for EventB Models
(
Citations: 1
)
Alin Stefanescu
,
Florentin Ipate
,
Raluca Lefticaru
,
Cristina Tudose
Conference:
International Conference on Software Testing, Verification, and Validation  ICST
, 2011
It is Not the Length That Matters, It is How You Control It
(
Citations: 1
)
Gordon Fraser
,
Andrea Arcuri
Conference:
International Conference on Software Testing, Verification, and Validation  ICST
, pp. 150159, 2011
A SearchBased OCL Constraint Solver for ModelBased Test Data Generation
(
Citations: 1
)
Shaukat Ali
,
Muhammad Zohaib Iqbal
,
Andrea Arcuri
,
Lionel Briand
Conference:
International Conference on Quality Software  QSIC
, pp. 4150, 2011
Testing Container Classes: Random or Systematic?
Rohan Sharma
,
Milos Gligoric
,
Andrea Arcuri
,
Gordon Fraser
,
Darko Marinov
Conference:
Fundamental Approaches to Software Engineering  FASE
, pp. 262277, 2011
Empirical Investigation of the Effects of Test Suite Properties on SimilarityBased Test Case Selection
Hadi Hemmati
,
Andrea Arcuri
,
Lionel C. Briand
Conference:
International Conference on Software Testing, Verification, and Validation  ICST
, pp. 327336, 2011