Blog4 · Methods

How to run an expert judgment

Before you administer your instrument, expert judges assess whether each item measures what it should. What they rate, how Aiken's V decides, how many judges you need and what to report.

Social sciencesEngineering

Expert judgment provides validity evidence based on test content: before the instrument is administered, people with experience in the topic assess whether each item measures what it should, whether it is understood and whether it suits the population (American Educational Research Association et al., 2014). It is expected whenever you build an instrument or adapt one to another population or language. It does not replace factor analysis: it comes before it.

The path

  1. Define the variable and its dimensionsWith a conceptual definition of each: it is what judges use to decide whether an item belongs there.
  2. Choose the judgesPeople with experience in the topic, the population or measurement, who did not write the items.
  3. Blind classificationEach judge places every item in the dimension they think it measures, without seeing the intended one.
  4. RatingsRelevance, clarity and cultural fit from 1 to 4, with comments and alternative wordings.
  5. CoverageWhether any dimension is missing an important aspect.
  6. Analysis and decisionAiken's V with its interval for each item and criterion; items that fall short are revised.
  7. ReportThe procedure, the judges, the values and what you changed.
The sequence of an expert judgment. Blind classification comes before the ratings so that judges are not swayed by the intended dimension.

What each judge rates

CriterionQuestion for the judge
RelevanceDoes the item measure the intended dimension?
ClarityIs it understood without ambiguity on first reading?
Cultural fitAre the language and the situation natural for the target population?

Each criterion is rated from 1 (does not meet it) to 4 (fully meets it). Blind classification answers a different question: whether an item "looks like" another dimension. When judges already know where an item goes, they tend to side with the author; blind, an item that most judges place elsewhere becomes visible.

How the decision is made: Aiken's V and its interval

Aiken's V summarizes the judges' agreement as a value from 0 to 1: V = (x̄ − l) / k, where x̄ is the mean rating, l the lowest value of the scale and k the difference between the highest and the lowest (Aiken, 1985). On a 1-to-4 scale, a V of 1 means every judge gave a 4.

The value alone is not enough, because with few judges it varies a lot. Its 95% confidence interval is computed (Penfield & Giacobbi, 2004) and the lower limit must reach the criterion: .70 in the conservative version or .50 in the lenient one (Merino & Livia, 2009). An item meets the standard when it does so on all three criteria.

How many judges?

The number of judges decides how much disagreement the criterion allows. The table shows how many points below 4 an item can receive on one criterion, adding up all the judges, and still meet the standard.

JudgesWith the .70 criterionWith the .50 criterion
2Never reachedNone
3None: all must give a 41
4None: all must give a 42
513
614
726
827
1049

With three or four judges and the .70 criterion, a single 3 is enough for the item to fall short. That is why you should have at least five, and seven or more if you can get them. The values were computed with the 1-to-4 scale and the 95% interval.

An item that falls short is not simply deleted. Read the comments and the wordings the judges proposed: the problem is often clarity, and rewriting the item solves it. If you change items substantially, a second round with the same judges is advisable.

What to report

AspectWhat to report
JudgesHow many took part, their training and their years of experience
ProcedureHow and when it was done, whether there was blind classification, the criteria and the scale
AnalysisAiken's V with its 95% interval and the criterion for the lower limit
ResultsHow many items met the standard, the mean V per criterion, the agreement in the blind classification and which items were revised
ChangesWhich items were rewritten or removed and why

Before you call it done

  • Does each dimension have a definition judges can use to decide?
  • Do you have at least five judges, and do you describe their profile?
  • Do you decide with the lower limit of the interval, not just the value of V?
  • Were the items that fell short revised using the judges' comments?
  • Do you report what you changed after the judgment?

Do it online with Expert Validation in your Research Tracker (in Spanish): you build the variable, its dimensions and the items; invite the judges by email or with a link; see who has already answered and, once you reach the number of judges you set, download the PDF report, a certificate for each judge, the Excel file and the text for your methods. Later, once you have data, test the structure with Instrument Validation.

References

Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Merino Soto, C., & Livia Segovia, J. (2009). Intervalos de confianza asimétricos para el índice de validez de contenido: Un programa Visual Basic para la V de Aiken. Anales de Psicología, 25(1), 169–171.

Penfield, R. D., & Giacobbi, P. R., Jr. (2004). Applying a score confidence interval to Aiken's item content-relevance index. Measurement in Physical Education and Exercise Science, 8(4), 213–225.

This guide is specific to the social sciences. It covers validating questionnaires and scales. In engineering, evaluating a solution is covered in What is expected of the methods.

Share this guide

How to cite this document

AtheneaLab. (2026, November 16). How to run an expert judgment. AtheneaLab Blog. https://athenealab.com/en/blog/how-to-run-an-expert-judgment/