As I struggle to make sense of the multitude of methods available for “factor analysis”, I found a paper by William Revelle that offers an interesting suggestion:

“Until computer packages such as BMD or SPSS will accept commands only from users who have passed an interactive test of their psychometric knowledge and ability, or until there are levels of programs available for various levels of user ability, there is a great benefit in using scaling algorithms that are simple to understand and robust to violations of their assumptions.”

-William Revelle, Hierarchical  Cluster Analysis  and the Internal  Structure of  Tests
Multivariate Behavioral Research, 1979,14, 57-74

I wonder–if we collectively decided to go ahead with this suggestion–what would the test look like? And how broadly would we define competence in an area? There are many things that require licenses (such as cars or guns), and many skills require a license to practice (medicine, running a restaurant, cutting hair, etc).  On the other hand, misapplied statistical methods aren’t the only reason studies end up with erroneous conclusions.

In addition, I find the ICLUST implementation in R’s psych package  (still maintained by Revelle!) to be straightforward in both its usage and interpretation.