Repetitions calculator
Calculate how many runs, cross-validation repetitions, datasets or users your engineering experiment needs to detect the difference you care about.
How is it calculated?
Every calculation looks for the smallest number that reaches the desired power, with noncentral t distributions as in G*Power (Faul et al., 2007):
- Runs: Mann-Whitney U test with the asymptotic relative efficiency method (Lehmann, 1975), which adjusts the two-group t by 3/π. Â12 is converted to d with d = √2·Φ⁻¹(Â12) (Vargha & Delaney, 2000).
- Cross-validation: Nadeau-Bengio (2003) corrected t-test, in which the variance is multiplied by (1/J + 1/(k − 1)), with J folds in total. That is why power has a ceiling even with many repetitions.
- Datasets: Wilcoxon signed-rank test (Demšar, 2006) with the asymptotic relative efficiency method. The minimum with which the exact test can give a significant result is also required.
- Users: paired or independent-samples t-test (Cohen, 1988) to compare versions; to find problems, n = ln(1 − goal) / ln(1 − p) (Nielsen & Landauer, 1993).
If you compare your method with several others, α is divided by the number of comparisons (Bonferroni). It is conservative planning: the analysis with the Holm correction, as done by Experimental Evaluation, is as powerful or more.
References
Arcuri, A., & Briand, L. (2011). A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (pp. 1–10). ACM. https://doi.org/10.1145/1985793.1985795
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
Lehmann, E. L. (1975). Nonparametrics: Statistical methods based on ranks. Holden-Day.
Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239–281. https://doi.org/10.1023/A:1024068626366
Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (pp. 206–213). ACM. https://doi.org/10.1145/169059.169166
Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2), 101–132. https://doi.org/10.3102/10769986025002101
How to cite this tool
AtheneaLab. (2026). Repetitions calculator [Web application]. https://athenealab.com/en/sample-size/repetitions/
Want to know more? Write to us at [email protected]