This guide is specific to engineering. It covers comparing methods or algorithms with repeated experiments. In the social sciences, presenting results is covered in What is expected of the results.
Almost every engineering paper ends in a comparison: your solution against others. A table of averages is not enough for a reviewer to accept it. The difference must be confirmed with a statistical test that fits how the experiment was run, and reported together with its size (Demšar, 2006; Arcuri & Briand, 2011). This guide explains what to prepare before the analysis: which projects it applies to, what to record while you experiment and how to organize the data.
Can your project be evaluated this way?
Yes, if you compare your method with at least one other using a numeric metric.
| Type of project | Typical metric | Does it apply? |
|---|---|---|
| Classification or detection (intrusions, faults, images) | Accuracy, F1, AUC | Yes |
| Prediction or regression (demand, power consumption) | Error (RMSE, MAE) | Yes |
| Optimization (routes, schedules, resource allocation) | Cost, distance, time | Yes |
| Networks and systems (protocols, architectures) | Latency, throughput, energy | Yes |
| Software engineering (testing, refactoring) | Time, defects found, coverage | Yes |
| A prototype with no other solution to compare it with | — | No: describe its performance and check that it meets the requirements |
| Acceptance or satisfaction survey (TAM, UTAUT) | Questionnaire scores | No: use Instrument Validation |
| Experiment with physical factors (temperature, mix, load) | Strength, yield | No: it is a classical experimental design (ANOVA) |
What to record while you experiment
- Every repetition, not just the average. Record the value of each run or fold. The average is computed at the end; without the individual values no test is possible.
- The same conditions for all. Every method is tested with the same datasets, the same metric and the same budget: time, iterations and hyperparameter tuning.
- Enough repetitions. If your method is randomized (seeds, random initialization), run each method at least 30 times (Arcuri & Briand, 2011). If you compare across several datasets, use at least 6: with 5 or fewer, no difference can come out significant. You can calculate the exact number with the Repetitions calculator.
- How each value was obtained. Write down the seed, the data split and the version of each method. You will need it to describe the methods and so others can repeat your evaluation.
Runs or folds: the key question
The tool asks what your repetitions are, because each case needs a different test.
| Independent runs | Cross-validation folds | |
|---|---|---|
| What changes between repetitions | The algorithm’s seed | Which part of the data is used for testing |
| Example | A genetic algorithm run 30 times | A model evaluated with 10-fold cross-validation |
| What the methods must share | The same problem | Exactly the same folds |
| Test used | Mann-Whitney U test | Nadeau-Bengio corrected t-test |
If you compare across several datasets, the unit of comparison is the dataset: on each one the repetitions are averaged and the methods are compared with the Friedman and Wilcoxon tests (Demšar, 2006).
How to organize your data
Use an Excel sheet with one of these two layouts. The first row holds the column names.
Wide format: one problem
| run | Proposed | GA | ACO |
|---|---|---|---|
| 1 | 1242.5 | 1291.0 | 1268.3 |
| 2 | 1238.1 | 1302.7 | 1259.9 |
| 3 | 1251.0 | 1287.4 | 1272.6 |
| … | … | … | … |
Long format: several datasets
| dataset | run | method | metric | value |
|---|---|---|---|---|
| NSL-KDD | 1 | Proposed | F1 | 0.921 |
| NSL-KDD | 1 | SVM | F1 | 0.848 |
| CTU-13 | 1 | Proposed | F1 | 0.903 |
| … | … | … | … | … |
- Column names. The tool recognizes, among others, method, dataset, run, fold or seed, metric and value, in English or Spanish.
- Clean numbers. The decimal separator can be a point or a comma. Do not merge cells or add rows of totals or averages.
- Everything complete. Each method needs results on every dataset, and its name must always be written the same way.
Download the templates
Each template includes an example you can load as is in the tool to see what it delivers, and an instructions sheet.
- One problem, several runs or foldsWide format · Excel (.xlsx)
- Several datasetsLong format · Excel (.xlsx)
Before and after
| Weak version | Problem | Improved version |
|---|---|---|
| The proposed method achieved 94.2% accuracy versus 91.8% for SVM. | An average with no variability or statistical test | The proposed method achieved F1 = 0.92 ± 0.01 over 30 runs versus 0.89 ± 0.02 for SVM (Mann-Whitney U, p < 0.001, Â12 = 0.86). |
| The method was compared on 4 datasets and judged superior. | With fewer than 6 datasets, the Wilcoxon test can never give p < 0.05 | Use at least 6 datasets (ideally 10 or more) or present the results as descriptive. |
| The 10 folds were treated as 10 independent samples with an ordinary t-test. | Folds share data: the test finds differences that do not exist | Use the Nadeau-Bengio (2003) corrected t-test. |
| The results of the other methods were copied from their original papers. | Different data, splits and hardware: the comparison is not fair | Run every method under the same conditions; if that is not possible, state it as a limitation. |
What each journal or conference decides. Some ask you to publish the code and data so the results can be verified, or to evaluate on specific benchmark datasets. Check their author guidelines before designing your experiments.
Before you load your data
- Did you record the value of each run or fold, not just the average?
- Does every method have results on every dataset, with the same metric?
- Do you know whether your repetitions are independent runs or cross-validation folds?
- Do you have at least 30 runs, 10 folds or 6 datasets?
- Does the first row of your sheet hold the column names?
With your data ready, Experimental Evaluation chooses the test, calculates the effect size and gives you the table, the figure and the text for your paper.
References
Arcuri, A., & Briand, L. (2011). A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (pp. 1–10). ACM. https://doi.org/10.1145/1985793.1985795
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30. https://jmlr.org/papers/v7/demsar06a.html
Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239–281. https://doi.org/10.1023/A:1024068626366