Yes, peer-review has its problems. However, without peers, there& #39;s no peer-review at all. Therefore, publishing original #statistics research in non- #stats journals is - well, not the best idea.

Yet, #sportsscience did it (again).

A thread ...
1/n

Simultaneous testing problems belong to the tricky things in statistics. The #sportsscience journal Gait & Posture published a paper (Niiler, 2020) on a new alpha-level correction ... which is "more efficient than the Bonferroni correction".

2/n https://doi.org/10.1016/j.gaitpost.2019.10.028">https://doi.org/10.1016/j...

Unfortunately, the method of Niiler (2020) tends to reject a correct null-hypothesis far more often than the chosen significance level alpha (https://abs.twimg.com/emoji/v2/... draggable="false" alt="➡️" title="Pfeil nach rechts" aria-label="Emoji: Pfeil nach rechts">invalid inference https://abs.twimg.com/emoji/v2/... draggable="false" alt="‼️" title="Doppeltes Ausrufezeichen" aria-label="Emoji: Doppeltes Ausrufezeichen">).

Here& #39;s a simulation study to demonstrate that the size of the test is way larger than alpha=0.05 https://abs.twimg.com/emoji/v2/... draggable="false" alt="⤵️" title="Nach rechts zeigender Pfeil mit Krümmung nach unten" aria-label="Emoji: Nach rechts zeigender Pfeil mit Krümmung nach unten">
3/n

With a method like this, you can reject the null hypothesis at a significance level of, e.g., alpha=0.05 even though nothing is actually statistically significant. That& #39;s not good. Damage is happening as the method is already used in the literature:
4/n https://link.springer.com/article/10.1007/s00221-020-05722-0">https://link.springer.com/article/1...

This thread is to make the #AcademicTwitter community aware of the flawed method in Niiler (2020).

In case you& #39;re wondering:
Yes, I contacted the author and tried to start a dialog, but got no response.

So, I submitted a "letter to the editors" of Gait & Posture.
5/n

It felt really strange to do this, but doing nothing wasn& #39;t an option either. My letter can be found here:

http://www.dliebl.com/files/Liebl_2020_LttE_G&P.pdf

I& #39;ll">https://www.dliebl.com/files/Lie... keep you posted about the process.

6/end

PS: Yes, there& #39;s still research besides COVID research https://abs.twimg.com/emoji/v2/... draggable="false" alt="😉" title="Zwinkerndes Gesicht" aria-label="Emoji: Zwinkerndes Gesicht">.

Btw: This is the alpha level correction in Niiler(2020). M is the number of tests, alpha is the signif level (e.g. alpha=0.05) and rho is the first-order auto-correlation in a time-series of test-statistics. All this is completely ad-hoc and there& #39;s no theoretical justification.

Niiler (2020) did an MC-simulation to "demonstrate" the correct size. However, he chose a data generating process for which the series of test statistics are constant. So, for this process, there was no multiple testing problem. He didn& #39;t see it since his method is also biased.