New preprint! " Replication Under Underdetermination: Introducing Systematic Replications Framework" w/ @tunc_necip. In this paper we discuss a core epistemic problem of hypothesis-testing wrt replication studies and offer a replication strategy (SRF) 1/

A core issue in philosophy of science is the underdetermination of theories by evidence: Theories have empirical implications only in conjunction with various auxiliary hypotheses, 2/

thus non-corroborative results of single tests leave us underdetermined in our decision as to whether the main hypothesis or some auxiliary hypotheses should bear the burden of falsification. 3/

Auxiliary hypotheses pertaining to the operationalizations of the theoretical terms are hardest to disentangle from the main hypothesis, because these fulfill the crucial task of making a theoretical claim testable in the first place. 4/

We have at least two such auxiliaries: a hypothesis that links the theoretical predictor to the observable predictor (AHpre) and another that links the theoretical outcome to the observational outcome (AHout). 5/

In this regard, every single isolated hypothesis test poses in fact at least three independent questions (i.e., whether to retain or reject the main hypothesis, AHpre, and AHout), to all of which it receives only a single answer. 6/

This creates an uncertainty which is amplified in the social sciences, where operationalizations play a very central role, but are much less theory-driven and less strongly linked to the theoretical terms. 7/

A central aim of replications is actually tackling underdetermination: close replications test for the validity of auxiliary assumptions such as the reliability of the instruments or the absence of artefacts, 8/

and conceptual replications test for the validity of auxiliaries pertaining to operationalizations. 9/

However, both close and conceptual replications fail in mitigating underdetermination when they are conducted as isolated tests. Close replications inform only about the stability/instability of the relationship between particular operationalizations of theoretical variables, 10/

thus their results do not bear on the main hypothesis. Negative results in conceptual replications, on the other hand, can be easily relegated to the new set of auxiliaries. Therefore, the main hypothesis can be easily "saved" from falsification via ad hoc explanations. 11/

In order to ground the link between replicability and falsification/corroboration of scientific hypotheses, we need to garner the advantages of both close and conceptual replications while controlling for their respective weaknesses. 12/

SRF aims to disentangle the implications of non-corroborative findings for the main and operationalization-related auxiliary hypotheses by organizing tests into a series of logically interlinked close and conceptual replications. 13/

SRF starts with replications as close to the original as possible and proceeds towards more conceptual replications by varying one operationalization at a time while keeping the other constant and then repeating the procedure for the other leg. 14/

SRF mitigates underdetermination in original studies as well as in close and conceptual replications. Primarily, it allows for non-corroborative evidence to have differential implications for the components of the TH & AHs bundle. 15/

Thereby these components can receive blame not collectively but in terms of a weighted distribution. 16/

In certain cases where it is not possible to achieve this, it allows demarcating, on which pairings from possible AHpre and AHout formulations the truth-value of the TH is conditional 17/

In all cases, the confounding effects deriving from the AHs can be relatively isolated. In section 2.1 we explicate how SRF operates in various scenarios (e.g. failed close replication or AH-variations with mixed results) 18/

The suggestion that tests should be logically interconnected is not entirely new. Sidman's (1960) "systematic replication," recent suggestions for designing meta-studies (Baribault et al. 2018) or introducing different operationalizations as random factors ( @talyarkoni 2019), 19/

and the notion of triangulation due to Munafò and Smith (2018) have apparent similarities to the idea behind SRF, but none of these aim to distinguish the confounding effects of various AHs on the TH. 20/

Systematic variation of the design elements does not help with the problem of underdetermination unless it is undertaken with a view to disentangle the questions we collectively address to each individual test thorough a series of systematically interconnected tests. 21/