Validating an instrument means gathering evidence that it measures what it claims to measure and that it does so precisely in your sample. Citing the original author's validation is not enough: validity is not a fixed property of the test but of its scores in a specific population, so your own evidence is expected.
The path
- Clean the datasetWithout careless, incomplete or invariant responses, which distort the correlations among items.
- Analyze the itemsItem-rest correlation, α if item deleted, skewness and kurtosis.
3 · Does the structure have theoretical support and prior evidence?
- Reliabilityα and ω for each dimension.
- Convergent and discriminant validityAVE, Fornell-Larcker criterion and HTMT.
- ReportThe values, the decisions you made and the limitations.
The key decision: EFA or CFA?
The CFA tests a structure specified in advance: how many dimensions there are and which items belong to each one (Brown, 2015). If theory defines it and a prior study supported it, going straight to the CFA is appropriate, especially when adapting a test with a known, clear structure (Ferrando et al., 2022). The EFA is reserved for when that structure is unknown.
An EFA may fail to reproduce a correct structure, because its result depends on the number of factors, the extraction and the rotation. Do not abandon a theoretical structure just because an EFA did not find it. If you use both analyses, run them in different halves of the sample: in the same one, the CFA confirms something tailored to those data (Fokkema & Greiff, 2017).
What to report
| Evidence | What to report | Guiding criterion |
|---|---|---|
| Internal structure | χ²(df), CFI, TLI, RMSEA with its 90% CI, SRMR and standardized loadings | CFI and TLI ≥ .90; RMSEA and SRMR ≤ .08; loadings ≥ .50 |
| Reliability | α and ω for each dimension | ≥ .70 |
| Convergent validity | Average variance extracted (AVE) | ≥ .50 |
| Discriminant validity | Fornell-Larcker and HTMT | HTMT < .85 (Henseler et al., 2015) |
Cutoffs guide; they do not decide. A value close to the limit can be reported if you justify it and acknowledge it as a limitation. What you should not do is hide it or tweak the model until the numbers comply.
Before you call it done
- Do you report the evidence obtained in your sample and not only the original author's?
- Do you justify why you ran a direct CFA or an EFA followed by a CFA?
- Does every removed item or correlated error have a content reason and is it reported?
- Do you report α and ω by dimension, plus convergent and discriminant validity?
- Do weak results appear as limitations in the discussion?
Do it step by step with Instrument Validation: it explains every number, writes the text for your methods and proposes the limitations. Before that, clean your dataset with Data Cleaning.
References
Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Press.
Ferrando, P. J., Lorenzo-Seva, U., Hernández-Dorado, A., & Muñiz, J. (2022). Decálogo para el análisis factorial de los ítems de un test. Psicothema, 34(1), 7–17. https://doi.org/10.7334/psicothema2021.456
Fokkema, M., & Greiff, S. (2017). How performing PCA and CFA on the same data equals trouble: Overfitting in the assessment of internal structure and some editorial thoughts on it. European Journal of Psychological Assessment, 33(6), 399–402. https://doi.org/10.1027/1015-5759/a000460
Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135. https://doi.org/10.1007/s11747-014-0403-8