Expert judgment provides validity evidence based on test content: before the instrument is administered, people with experience in the topic assess whether each item measures what it should, whether it is understood and whether it suits the population (American Educational Research Association et al., 2014). It is expected whenever you build an instrument or adapt one to another population or language. It does not replace factor analysis: it comes before it.
The path
- Define the variable and its dimensionsWith a conceptual definition of each: it is what judges use to decide whether an item belongs there.
- Choose the judgesPeople with experience in the topic, the population or measurement, who did not write the items.
- Blind classificationEach judge places every item in the dimension they think it measures, without seeing the intended one.
- RatingsRelevance, clarity and cultural fit from 1 to 4, with comments and alternative wordings.
- CoverageWhether any dimension is missing an important aspect.
- Analysis and decisionAiken's V with its interval for each item and criterion; items that fall short are revised.
- ReportThe procedure, the judges, the values and what you changed.
What each judge rates
| Criterion | Question for the judge |
|---|---|
| Relevance | Does the item measure the intended dimension? |
| Clarity | Is it understood without ambiguity on first reading? |
| Cultural fit | Are the language and the situation natural for the target population? |
Each criterion is rated from 1 (does not meet it) to 4 (fully meets it). Blind classification answers a different question: whether an item "looks like" another dimension. When judges already know where an item goes, they tend to side with the author; blind, an item that most judges place elsewhere becomes visible.
How the decision is made: Aiken's V and its interval
Aiken's V summarizes the judges' agreement as a value from 0 to 1: V = (x̄ − l) / k, where x̄ is the mean rating, l the lowest value of the scale and k the difference between the highest and the lowest (Aiken, 1985). On a 1-to-4 scale, a V of 1 means every judge gave a 4.
The value alone is not enough, because with few judges it varies a lot. Its 95% confidence interval is computed (Penfield & Giacobbi, 2004) and the lower limit must reach the criterion: .70 in the conservative version or .50 in the lenient one (Merino & Livia, 2009). An item meets the standard when it does so on all three criteria.
How many judges?
The number of judges decides how much disagreement the criterion allows. The table shows how many points below 4 an item can receive on one criterion, adding up all the judges, and still meet the standard.
| Judges | With the .70 criterion | With the .50 criterion |
|---|---|---|
| 2 | Never reached | None |
| 3 | None: all must give a 4 | 1 |
| 4 | None: all must give a 4 | 2 |
| 5 | 1 | 3 |
| 6 | 1 | 4 |
| 7 | 2 | 6 |
| 8 | 2 | 7 |
| 10 | 4 | 9 |
With three or four judges and the .70 criterion, a single 3 is enough for the item to fall short. That is why you should have at least five, and seven or more if you can get them. The values were computed with the 1-to-4 scale and the 95% interval.
An item that falls short is not simply deleted. Read the comments and the wordings the judges proposed: the problem is often clarity, and rewriting the item solves it. If you change items substantially, a second round with the same judges is advisable.
What to report
| Aspect | What to report |
|---|---|
| Judges | How many took part, their training and their years of experience |
| Procedure | How and when it was done, whether there was blind classification, the criteria and the scale |
| Analysis | Aiken's V with its 95% interval and the criterion for the lower limit |
| Results | How many items met the standard, the mean V per criterion, the agreement in the blind classification and which items were revised |
| Changes | Which items were rewritten or removed and why |
Before you call it done
- Does each dimension have a definition judges can use to decide?
- Do you have at least five judges, and do you describe their profile?
- Do you decide with the lower limit of the interval, not just the value of V?
- Were the items that fell short revised using the judges' comments?
- Do you report what you changed after the judgment?
Do it online with Expert Validation in your Research Tracker (in Spanish): you build the variable, its dimensions and the items; invite the judges by email or with a link; see who has already answered and, once you reach the number of judges you set, download the PDF report, a certificate for each judge, the Excel file and the text for your methods. Later, once you have data, test the structure with Instrument Validation.
References
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Merino Soto, C., & Livia Segovia, J. (2009). Intervalos de confianza asimétricos para el índice de validez de contenido: Un programa Visual Basic para la V de Aiken. Anales de Psicología, 25(1), 169–171.
Penfield, R. D., & Giacobbi, P. R., Jr. (2004). Applying a score confidence interval to Aiken's item content-relevance index. Measurement in Physical Education and Exercise Science, 8(4), 213–225.
This guide is specific to the social sciences. It covers validating questionnaires and scales. In engineering, evaluating a solution is covered in What is expected of the methods.