Sample size calculation

Repetitions calculator

Calculate how many runs, cross-validation repetitions, datasets or users your engineering experiment needs to detect the difference you care about.

What do you want to calculate?
How do you know the size of the difference?

In the units of your metric (for example, 2 F1 points, 15 ms or 20 km).

In the same units as the difference.

0.2, 0.5 and 0.8 are small, medium and large effects (Cohen, 1988).

5 or 10 is usual.

With more than two, the calculation corrects α for the number of comparisons of your method with the others.

0.05 is usual.

Probability of detecting the difference if it exists. 0.80 is usual.

Average proportion. 0.31 is the value found by Nielsen and Landauer (1993).

Between 1 and 99%.

Result

—

How is it calculated?

Every calculation looks for the smallest number that reaches the desired power, with noncentral t distributions as in G*Power (Faul et al., 2007):

  • Runs: Mann-Whitney U test with the asymptotic relative efficiency method (Lehmann, 1975), which adjusts the two-group t by 3/π. Â12 is converted to d with d = √2·Φ⁻¹(Â12) (Vargha & Delaney, 2000).
  • Cross-validation: Nadeau-Bengio (2003) corrected t-test, in which the variance is multiplied by (1/J + 1/(k − 1)), with J folds in total. That is why power has a ceiling even with many repetitions.
  • Datasets: Wilcoxon signed-rank test (Demšar, 2006) with the asymptotic relative efficiency method. The minimum with which the exact test can give a significant result is also required.
  • Users: paired or independent-samples t-test (Cohen, 1988) to compare versions; to find problems, n = ln(1 − goal) / ln(1 − p) (Nielsen & Landauer, 1993).

If you compare your method with several others, α is divided by the number of comparisons (Bonferroni). It is conservative planning: the analysis with the Holm correction, as done by Experimental Evaluation, is as powerful or more.

References

Arcuri, A., & Briand, L. (2011). A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (pp. 1–10). ACM. https://doi.org/10.1145/1985793.1985795

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146

Lehmann, E. L. (1975). Nonparametrics: Statistical methods based on ranks. Holden-Day.

Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239–281. https://doi.org/10.1023/A:1024068626366

Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (pp. 206–213). ACM. https://doi.org/10.1145/169059.169166

Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2), 101–132. https://doi.org/10.3102/10769986025002101

How to cite this tool

AtheneaLab. (2026). Repetitions calculator [Web application]. https://athenealab.com/en/sample-size/repetitions/

Want to know more? Write to us at [email protected]