Research Article
BibTex RIS Cite

Exploring number of response categories in factor analysis: Implications for sample size

Year 2025, Volume: 12 Issue: 2, 341 - 352

Fatih Orçan

https://doi.org/10.21449/ijate.1581217

Abstract

Factor analysis is a statistical method to explore the relationships among observed variables and identify latent structures. It is crucial in scale development and validity analysis. Key factors affecting the accuracy of factor analysis results include the type of data, sample size, and the number of response categories. While some studies suggest that reliability improves with more response categories, others find no significant relationship between the number of response categories and reliability. A key consideration is that increasing the number of response categories can introduce measurement errors, especially when there are too many categories for participants to respond accurately. The study examines how different numbers of response categories affect sample size requirements in factor analysis, particularly under misspecified and correctly specified models. MonteCarloSEM package in R was used to simulate data sets based on sample size, number of response categories, model specification, and test length. Results show that a higher number of categories helps reduce bias and improve model fit, especially in smaller samples. However, when sample sizes are small or when fewer categories are used, increasing the number of items or the number of categories can improve parameter estimation. The findings suggest that for optimal results, researchers should carefully balance sample size, number of items, and response categories, particularly in studies with categorical data.

Keywords

Factor analysis Sample size Number of response categories Number of items

References

  • Abulela, M.A.A., & Khalaf, M.A. (2024). Does the number of response categories impact validity evidence in self report measures? A scoping review. Sage Open, 14(1), 1 16. https://doi.org/10.1177/21582440241230363
  • Abdelsamea, M. (2020). The effect of the number of response categories on the assumptions and outputs of item exploratory and confirmatory factor analyses of measurement instruments in psychological research. Journal of Education Sohag UNV, 76, 1153-1222. https://doi.org/10.21608/edusohag.2020.103373
  • Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9(2), 151 160. https://doi.org/10.1207/s15324818ame0902_4
  • Flora, D.B., & Curran, P.J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466
  • Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
  • Kline, R.B. (2011). Principles and Practice of Structural Equation Modeling. Guilford Press.
  • Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25(4), 987 995. https://doi.org/10.1177/001316446502500404
  • Kılıç, A.F. (2022). The effect of categories and distribution of variables on correlation coefficients. Ege Eğitim Dergisi, 23(1), 50-80. https://doi.org/10.12984/egeefd.890104
  • Kyriazos, T.A. (2018). Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology, 9(8), 2207 2230. https://doi.org/10.4236/psych.2018.98126
  • Lozano, L., García-Cueto, E., & Muñiz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology 4(2). 73 79. https://doi.org/10.1027/1614-2241.4.2.73
  • MacCallum, R.C., Widaman, K.F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989X.4.1.84
  • Matell, M.S., & Jacoby, J. (1971). Is there an optimal number of alternatives for Likert scale items? I. Reliability and validity. Educational and Psychological Measurement, 31, 657-674. https://doi.org/10.1177/001316447103100307
  • Maydeu-Olivares, A., Fairchild, A.J., & Hall, A.G. (2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 24(4), 495 505. https://doi.org/10.1080/10705511.2017.1289816
  • Orçan, F. (2021). MonteCarloSEM: An R package to simulate data for SEM. International Journal of Assessment Tools in Education, 8(3), 704 713. https://doi.org/10.21449/ijate.804203
  • Orçan, F., & Yanyun, Y. (2016). A note on the use of item parceling in structural equation modeling with missing data. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 59-72. https://doi.org/10.21031/epod.88204
  • Preston, C.C., & Colman, A.M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1–15. https://doi.org/10.1016/S0001-6918(99)00050-5
  • R Core Team. (2020). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
  • Rhemtulla, M., Brosseau-Liard, P.É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354 373. https://doi.org/10.1037/a0029315
  • Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
  • Shi, D., Siceloff, E.R., Castellanos, R.E., Bridges, R.M., Jiang, Z., Flory, K., & Benson, K. (2021). Revisiting the effect of varying the number of response alternatives in clinical assessment: Evidence from measuring ADHD symptoms. Assessment, 28(5), 1287-1300. https://doi.org/10.1177/1073191120952885
  • Simms, L.J., Zelazny, K., Williams, T.F., & Bernstein, L. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557-566. http://dx.doi.org/10.1037/pas0000648
  • Yoon, G. (2024) No one optimal way to measure people’s attitudes? Preferred length of scales in advertising research. Journal of Current Issues & Research in Advertising, 45(1), 43-70, https://doi.org/10.1080/10641734.2023.2246049
  • Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological distance between categories in the Likert scale: Comparing different numbers of options. Educational and Psychological Measurement, 72(4), 533-546. https://doi.org/10.1177/0013164411431162

Exploring number of response categories in factor analysis: Implications for sample size

Year 2025, Volume: 12 Issue: 2, 341 - 352

Fatih Orçan

https://doi.org/10.21449/ijate.1581217

Abstract

Factor analysis is a statistical method to explore the relationships among observed variables and identify latent structures. It is crucial in scale development and validity analysis. Key factors affecting the accuracy of factor analysis results include the type of data, sample size, and the number of response categories. While some studies suggest that reliability improves with more response categories, others find no significant relationship between the number of response categories and reliability. A key consideration is that increasing the number of response categories can introduce measurement errors, especially when there are too many categories for participants to respond accurately. The study examines how different numbers of response categories affect sample size requirements in factor analysis, particularly under misspecified and correctly specified models. MonteCarloSEM package in R was used to simulate data sets based on sample size, number of response categories, model specification, and test length. Results show that a higher number of categories helps reduce bias and improve model fit, especially in smaller samples. However, when sample sizes are small or when fewer categories are used, increasing the number of items or the number of categories can improve parameter estimation. The findings suggest that for optimal results, researchers should carefully balance sample size, number of items, and response categories, particularly in studies with categorical data.

Keywords

Factor analysis Sample size Number of response categories Number of items

References

  • Abulela, M.A.A., & Khalaf, M.A. (2024). Does the number of response categories impact validity evidence in self report measures? A scoping review. Sage Open, 14(1), 1 16. https://doi.org/10.1177/21582440241230363
  • Abdelsamea, M. (2020). The effect of the number of response categories on the assumptions and outputs of item exploratory and confirmatory factor analyses of measurement instruments in psychological research. Journal of Education Sohag UNV, 76, 1153-1222. https://doi.org/10.21608/edusohag.2020.103373
  • Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9(2), 151 160. https://doi.org/10.1207/s15324818ame0902_4
  • Flora, D.B., & Curran, P.J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466
  • Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
  • Kline, R.B. (2011). Principles and Practice of Structural Equation Modeling. Guilford Press.
  • Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25(4), 987 995. https://doi.org/10.1177/001316446502500404
  • Kılıç, A.F. (2022). The effect of categories and distribution of variables on correlation coefficients. Ege Eğitim Dergisi, 23(1), 50-80. https://doi.org/10.12984/egeefd.890104
  • Kyriazos, T.A. (2018). Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology, 9(8), 2207 2230. https://doi.org/10.4236/psych.2018.98126
  • Lozano, L., García-Cueto, E., & Muñiz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology 4(2). 73 79. https://doi.org/10.1027/1614-2241.4.2.73
  • MacCallum, R.C., Widaman, K.F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989X.4.1.84
  • Matell, M.S., & Jacoby, J. (1971). Is there an optimal number of alternatives for Likert scale items? I. Reliability and validity. Educational and Psychological Measurement, 31, 657-674. https://doi.org/10.1177/001316447103100307
  • Maydeu-Olivares, A., Fairchild, A.J., & Hall, A.G. (2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 24(4), 495 505. https://doi.org/10.1080/10705511.2017.1289816
  • Orçan, F. (2021). MonteCarloSEM: An R package to simulate data for SEM. International Journal of Assessment Tools in Education, 8(3), 704 713. https://doi.org/10.21449/ijate.804203
  • Orçan, F., & Yanyun, Y. (2016). A note on the use of item parceling in structural equation modeling with missing data. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 59-72. https://doi.org/10.21031/epod.88204
  • Preston, C.C., & Colman, A.M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1–15. https://doi.org/10.1016/S0001-6918(99)00050-5
  • R Core Team. (2020). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
  • Rhemtulla, M., Brosseau-Liard, P.É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354 373. https://doi.org/10.1037/a0029315
  • Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
  • Shi, D., Siceloff, E.R., Castellanos, R.E., Bridges, R.M., Jiang, Z., Flory, K., & Benson, K. (2021). Revisiting the effect of varying the number of response alternatives in clinical assessment: Evidence from measuring ADHD symptoms. Assessment, 28(5), 1287-1300. https://doi.org/10.1177/1073191120952885
  • Simms, L.J., Zelazny, K., Williams, T.F., & Bernstein, L. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557-566. http://dx.doi.org/10.1037/pas0000648
  • Yoon, G. (2024) No one optimal way to measure people’s attitudes? Preferred length of scales in advertising research. Journal of Current Issues & Research in Advertising, 45(1), 43-70, https://doi.org/10.1080/10641734.2023.2246049
  • Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological distance between categories in the Likert scale: Comparing different numbers of options. Educational and Psychological Measurement, 72(4), 533-546. https://doi.org/10.1177/0013164411431162
There are 23 citations in total.

Details

Primary Language English
Subjects Similation Study
Journal Section Articles
Authors

Fatih Orçan 0000-0003-1727-0456

Early Pub Date May 1, 2025
Publication Date
Submission Date November 8, 2024
Acceptance Date January 31, 2025
Published in Issue Year 2025 Volume: 12 Issue: 2

Cite

APA Orçan, F. (2025). Exploring number of response categories in factor analysis: Implications for sample size. International Journal of Assessment Tools in Education, 12(2), 341-352. https://doi.org/10.21449/ijate.1581217

23823             23825             23824