Read on Twitter
Okay, #ica20, let& #39;s do this. Here are my thoughts on #openscience and #opencomm for the more qualitative approaches in our field!
(Beware, long thread.)
(Also, please be kind.)
(Beware, long thread.)
(Also, please be kind.)
>Background: I was VERY curious how our agenda for #opencomm would be received in our field (if you don& #39;t know what I& #39;m talking about, here& #39;s the link: https://academic.oup.com/joc/advance-article/doi/10.1093/joc/jqz052/5803422).">https://academic.oup.com/joc/advan...
>Basically, we& #39;re calling for making research more available, transparent, & traceable. We encourage the sharing of data (where ever possible) & analysis scripts, and the introduction of preregistration, registered reports, large scale collaborations, the TOP guidelines, etc.
>We explicitly criticize QRPs such as p-hacking, harking, story telling, or the general incentive structures that prioritize publication quantity over research quality.
>Although we explicitly focus on quantitative approaches in the hypothetico-deductive context, we also offer some tentative suggestions how #opencomm might work for the more qualitative approaches.
(for more information, see talk by @lindseybier https://www.youtube.com/watch?v=nm-19808jSo&list=PL_wlr6EiCKuXAj0j-EoJnTmfWkbxpFVnn&index=3)">https://www.youtube.com/watch...
(for more information, see talk by @lindseybier https://www.youtube.com/watch?v=nm-19808jSo&list=PL_wlr6EiCKuXAj0j-EoJnTmfWkbxpFVnn&index=3)">https://www.youtube.com/watch...
>So, again, I was VERY curious re. how the agenda would be received by our community.
Overall, the feedback was overwhelmingly positive, and I& #39;m really happy & thankful for all the nice & kind words we& #39;ve received :)
Overall, the feedback was overwhelmingly positive, and I& #39;m really happy & thankful for all the nice & kind words we& #39;ve received :)
>However, both in our reviews from JOC and the subsequent discussion at #ica20, we did get a bit of pushback coming from one particular direction -- which, tbh, somewhat surprised me.
>(But, to be crystal clear here: Feedback was fair, constructive, good spirited, and self-reflective -- just the way it should be.)
>Interestingly, the agenda was not criticized by our field& #39;s industrious p-hackers, which was what happened in psych, and for whom consequences indeed are dire (mb they& #39;ve learned from the mistakes of their psych kin, bc arguments simply don& #39;t hold up?)
>Instead, pushback came from the more qualitative approaches. I& #39;ve read the following objections & concerns:
(pls. feel free to correct wrong understandings or to add novel points.)
(pls. feel free to correct wrong understandings or to add novel points.)
>1. Replicability not relevant, bc. all knowledge is contextual. 2. Through self-reflectively explicating lenses & biases, qual. is already doing most of this anyway. 3. The researcher is part of the knowledge process, hence cannot be separated from the results.
>4. Preregistration doesn& #39;t make sense, bc. we don& #39;t know what we& #39;ll find beforehand. 5. Knowledge is co-created together with participants, hence it& #39;s a process that cannot be planned/explicated/replicated. 6. Results need to be interpreted and are never absolute/general.
> 7. It& #39;s not about replicability or generalizability, but transferability. 7. In a lot / most (?) cases, sharing data is not possible/useful: Too risky for participants, even when anonymized/deidentified, and too much effort for too little value of what& #39;s left.
>Here& #39;s my reply:
>Qualitative research is not the *problem*. Qualitative research is the *solution*.
>Exploratory research is also not the problem. The problem is selling exploratory research as confirmatory. Exploratory research is CRUCIAL for knowledge building. Exploratory research is part of the *solution*.
>Preregistration & Registered Reports do *not* prevent exploration. This is a common & false misunderstanding. Instead, by explicitly labeling unplanned/emergent/novel analyses as exploratory, we make them *more* prominent. We elevate them.
>Preregistration is mainly for hypothetico-deductive, quantitative approaches. If you& #39;re 100% exploratory, just skip preregistration! It& #39;s *not* mandatory.
But most of the time we start with a theoretical framework, perspective, assumption, or hunch -- this you can preregister.
But most of the time we start with a theoretical framework, perspective, assumption, or hunch -- this you can preregister.
>Yes, you absolutely cannot share specific kinds of data. End of story.
>Yes, qualitative research is *much* more contextual. BUT: It& #39;s not a 100% contextual. Bc if it were a 100% contextual/idiosyncratic, we could, by definition, not transfer it. But we can.
>And, granted, we do: Just read the discussion sections, you& #39;ll often find many (relevant!, important!) claims re. underlying power structures, generally disadvantaged groups, oppression mechanisms, etc.
>BUT: You cannot have the cake and eat it. It& #39;s a truism that for the very same reasons, the transferability of insights coming from small scale qualitative studies is also more limited.
>Let& #39;s not forget: Objectivity >= Reliability >= Validity.
Let& #39;s also not engage in jingle-jangle fallacies: Transferability ~ generalizability.
Let& #39;s also not engage in jingle-jangle fallacies: Transferability ~ generalizability.
>Awareness of reduced generalizability is not a problem, it& #39;s again part of the solution. Quant researchers can learn much from qual researchers:
Our knowledge is highly contextual. Don& #39;t overclaim. Know yourself. Know the structure you& #39;re working in. Be humble.
(The irony.)
Our knowledge is highly contextual. Don& #39;t overclaim. Know yourself. Know the structure you& #39;re working in. Be humble.
(The irony.)
>I& #39;m perfectly aware that all this is highly delicate: E.g., there are different understandings of what qual. research itself even is. In our agenda, we explicate that. I& #39;m no expert on qual. I& #39;m hence happy to have other jump in here. These are my thoughts, my understanding.
>I honestly believe that the replication crisis has shown that we should be more humble, more descriptive, more nuanced. If I& #39;m not mistaken, these are core values of qual research.
>I would love to encourage qual research to see the many opportunities #openscience offers. This is really not about paternalizing qual. & exploratory research. This is about *empowering* qual & exploratory research!
>Don& #39;t believe me? Well, then read @talyarkoni magnificient paper "the generalizability crisis".
(Background: Tal is a leading figure of the #openscience movement in psych and, without doubt, currently the smartest person living on this earth).
This is what he writes:
(Background: Tal is a leading figure of the #openscience movement in psych and, without doubt, currently the smartest person living on this earth).
This is what he writes:
>Conclusion: #openscience and #opencomm will lead to knowledge claims that are more robust. In this process, qualitative research is not part of the problem but part of the solution.
>Let& #39;s all join this discussion, continue to exchange thoughts and concerns, see what works and what not. I truly believe that #openscience has much to offer for *all* of us.
> (Please don& #39;t kill me.)
(To read this on one single page, see https://tobiasdienlin.com/2020/05/24/open-science-and-qualitative-research-yes-we-can-do-this/)">https://tobiasdienlin.com/2020/05/2...
