Does anyone else think it is strange that in scale development, you are advised to generate a large pool of items, fit a model to everything, and then throw out items that don't fit well to finalize the scale? Why are arbitrary statistical fit indices prioritized over theory?

I mean, if I am developing a model of some behavioral process, I am not going to throw out trials where the model fails to predict a subject's behavioral responses. Instead, the misfit tells me where the model (i.e. the theory) needs improvement.

It seems like philosophies of measurement are very different from those focused on modeling data-generating processes. One view suggests we prune the data until the our model fits (or add covariances), whereas the other suggests we modify the model in fundamental ways.

IMO, theory should dictate what items/types of behaviors I am interested in, and it is then my job to develop a model that can adequately account for theoretically relevant patterns observed across those items/behaviors.

(this thread brought to you by my confusion when discussing CFA with a colleague )