Blog5 · Results

How to compare your method with others

Before claiming your solution is better you need results that can be compared. Which projects apply, what to record while you experiment and how to organize the data, with templates.

Social sciencesEngineering

This guide is specific to engineering. It covers comparing methods or algorithms with repeated experiments. In the social sciences, presenting results is covered in What is expected of the results.

Almost every engineering paper ends in a comparison: your solution against others. A table of averages is not enough for a reviewer to accept it. The difference must be confirmed with a statistical test that fits how the experiment was run, and reported together with its size (Demšar, 2006; Arcuri & Briand, 2011). This guide explains what to prepare before the analysis: which projects it applies to, what to record while you experiment and how to organize the data.

Can your project be evaluated this way?

Yes, if you compare your method with at least one other using a numeric metric.

Type of projectTypical metricDoes it apply?
Classification or detection (intrusions, faults, images)Accuracy, F1, AUCYes
Prediction or regression (demand, power consumption)Error (RMSE, MAE)Yes
Optimization (routes, schedules, resource allocation)Cost, distance, timeYes
Networks and systems (protocols, architectures)Latency, throughput, energyYes
Software engineering (testing, refactoring)Time, defects found, coverageYes
A prototype with no other solution to compare it with—No: describe its performance and check that it meets the requirements
Acceptance or satisfaction survey (TAM, UTAUT)Questionnaire scoresNo: use Instrument Validation
Experiment with physical factors (temperature, mix, load)Strength, yieldNo: it is a classical experimental design (ANOVA)

What to record while you experiment

  • Every repetition, not just the average. Record the value of each run or fold. The average is computed at the end; without the individual values no test is possible.
  • The same conditions for all. Every method is tested with the same datasets, the same metric and the same budget: time, iterations and hyperparameter tuning.
  • Enough repetitions. If your method is randomized (seeds, random initialization), run each method at least 30 times (Arcuri & Briand, 2011). If you compare across several datasets, use at least 6: with 5 or fewer, no difference can come out significant. You can calculate the exact number with the Repetitions calculator.
  • How each value was obtained. Write down the seed, the data split and the version of each method. You will need it to describe the methods and so others can repeat your evaluation.

Runs or folds: the key question

The tool asks what your repetitions are, because each case needs a different test.

Independent runsCross-validation folds
What changes between repetitionsThe algorithm’s seedWhich part of the data is used for testing
ExampleA genetic algorithm run 30 timesA model evaluated with 10-fold cross-validation
What the methods must shareThe same problemExactly the same folds
Test usedMann-Whitney U testNadeau-Bengio corrected t-test

If you compare across several datasets, the unit of comparison is the dataset: on each one the repetitions are averaged and the methods are compared with the Friedman and Wilcoxon tests (Demšar, 2006).

How to organize your data

Use an Excel sheet with one of these two layouts. The first row holds the column names.

Wide format: one problem

runProposedGAACO
11242.51291.01268.3
21238.11302.71259.9
31251.01287.41272.6
…………

Long format: several datasets

datasetrunmethodmetricvalue
NSL-KDD1ProposedF10.921
NSL-KDD1SVMF10.848
CTU-131ProposedF10.903
……………
In the wide format each method is a column and each row is a run or fold. In the long format each row is one result; it works for several datasets and for recording several metrics.
  • Column names. The tool recognizes, among others, method, dataset, run, fold or seed, metric and value, in English or Spanish.
  • Clean numbers. The decimal separator can be a point or a comma. Do not merge cells or add rows of totals or averages.
  • Everything complete. Each method needs results on every dataset, and its name must always be written the same way.

Download the templates

Each template includes an example you can load as is in the tool to see what it delivers, and an instructions sheet.

Before and after

Weak versionProblemImproved version
The proposed method achieved 94.2% accuracy versus 91.8% for SVM.An average with no variability or statistical testThe proposed method achieved F1 = 0.92 ± 0.01 over 30 runs versus 0.89 ± 0.02 for SVM (Mann-Whitney U, p < 0.001, Â12 = 0.86).
The method was compared on 4 datasets and judged superior.With fewer than 6 datasets, the Wilcoxon test can never give p < 0.05Use at least 6 datasets (ideally 10 or more) or present the results as descriptive.
The 10 folds were treated as 10 independent samples with an ordinary t-test.Folds share data: the test finds differences that do not existUse the Nadeau-Bengio (2003) corrected t-test.
The results of the other methods were copied from their original papers.Different data, splits and hardware: the comparison is not fairRun every method under the same conditions; if that is not possible, state it as a limitation.

What each journal or conference decides. Some ask you to publish the code and data so the results can be verified, or to evaluate on specific benchmark datasets. Check their author guidelines before designing your experiments.

Before you load your data

  • Did you record the value of each run or fold, not just the average?
  • Does every method have results on every dataset, with the same metric?
  • Do you know whether your repetitions are independent runs or cross-validation folds?
  • Do you have at least 30 runs, 10 folds or 6 datasets?
  • Does the first row of your sheet hold the column names?

With your data ready, Experimental Evaluation chooses the test, calculates the effect size and gives you the table, the figure and the text for your paper.

References

Arcuri, A., & Briand, L. (2011). A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (pp. 1–10). ACM. https://doi.org/10.1145/1985793.1985795

Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30. https://jmlr.org/papers/v7/demsar06a.html

Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239–281. https://doi.org/10.1023/A:1024068626366

Share this guide

How to cite this document

AtheneaLab. (2026, November 14). How to compare your method with others. AtheneaLab Blog. https://athenealab.com/en/blog/how-to-compare-your-method-with-others/