In factor analysis, determining the number of factors underlying measurement indicators is important. An incorrect decision on the number of factors may mislead practitioners in terms of estimating parameters in factor analysis, reporting students' scores, calibrating items through an item response theory model, equating or linking different test forms, estimating reliability, examining differential item functioning, and investigating validity. Exploratory factor analysis, parallel analysis, Kaiser's rule, and Cattell's scree test are commonly used methods for deciding on the number of factors in educational and psychological assessments. When a test consists of multiple-choice or ordinal-scaled items, some test takers might find the correct answers to the items through guessing. The guessing effect might impact the correlation coefficients among items. Thus, it might also impact the decisions on the number of factors via exploratory factor analysis, parallel analysis, Kaiser's rule, and Cattell's scree test. These four methods do not consider the guessing effects through modeling a guessing parameter when examining the dimensionality of data. The main purpose of this study is to investigate the impact of guessing on the performance of exploratory factor analysis, parallel analysis, Kaiser's rule, and Cattell's scree test in determining the number of factors underlying measurement indicators. Among these four methods, Cattell's scree test is a subjective method because the determination of the elbow point in the scree plot requires the user to make a judgmental call. Therefore, another purpose of this study is to propose a method that may allow for a more objective evaluation of Cattell's scree test, specifically, through calculating angles in the scree plot. A Monte Carlo study was conducted to examine the performance of exploratory factor analysis, Kaiser's rule, parallel analysis, and the revised scree test in determining the dimensionality of data when guessing effects were present. The following design factors were manipulated: factor structure, sample size, test length, the number of factors, values of the pseudo-guessing parameters, and the correlation between factors. The study results showed that all four methods performed worse for determining the number of factors under the presence of guessing effects than under the absence of guessing effect. In other words, none of the four methods was robust to the presence of guessing effects. Among the four methods, parallel analysis performed the best. The study results also showed that all four methods tended to retain fewer factors as the guessing effects became greater. Across all levels of guessing effects, parallel analysis was the best method for identifying the number of factors under conditions with simple structures, while exploratory factor analysis using the chi-square difference test was the best method for determining the dimensionality of bifactor models. In terms of the methods for estimating polychoric correlations, the maximum likelihood and Bayesian methods performed almost identically and led to similar estimated numbers of factors via the four methods. The current study design indicated that two different cutoff values were reasonable to use for determining the number of factors via the revised Cattell's scree test: 161 for simple-structure models and 173 for bifactor models. The revised Cattell's scree test performed better for determining the number of factors under conditions with simple structures than with bifactor models using these two cutoff values. Although practitioners and researchers may consider using the revised Cattell's scree test to evaluate a scree plot in a more objective way, it is important to use the indicated cutoff values with caution in that they may not be applicable under other study conditions.
