Read on Twitter
FOUR CARDINAL RULES OF STATISTICS 
A thread
1/
A thread
1/
ONE: CORRELATION DOES NOT IMPLY CAUSATION.
Yes, I know you know this, but itâs so easy to forget!
Yeah, YOU OVER THERE, you with the p-value of 0.0000001 â yes, YOU!! Thatâs not causation.
2/
Yes, I know you know this, but itâs so easy to forget!
Yeah, YOU OVER THERE, you with the p-value of 0.0000001 â yes, YOU!! Thatâs not causation.
2/
No matter how small the p-value for a regression of IQ onto shoe size is, that doesnât mean that big feet cause smarts!!
It just means that grown-ups tend to have bigger feet and higher IQs than kids.
3/
It just means that grown-ups tend to have bigger feet and higher IQs than kids.
3/
So, unless you can design your study to uncover causation (very hard to do in most practical settings â the field of causal inference is devoted to understanding the settings in which it is possible), the best you can do is to discover correlations.
Sad but true.
4/
Sad but true.
4/
TWO: A P-VALUE IS JUST A TEST OF SAMPLE SIZE.
Read that again â I mean what I said!
If your null hypothesis doesnât hold (and null hypotheses never hold IRL) then the larger your sample size, the smaller your p-value will tend to be.
5/
Read that again â I mean what I said!
If your null hypothesis doesnât hold (and null hypotheses never hold IRL) then the larger your sample size, the smaller your p-value will tend to be.
5/
If youâre testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE.
6/
6/
Why does this matter?
In many contemporary settings (think: the internet), sample sizes are so huge that we can get TINY p-values even when the deviation from the null hypothesis is negligible.
In other words, we can have STATISTICAL significance w/o PRACTICAL significance.
7/
In many contemporary settings (think: the internet), sample sizes are so huge that we can get TINY p-values even when the deviation from the null hypothesis is negligible.
In other words, we can have STATISTICAL significance w/o PRACTICAL significance.
7/
Often, people focus on that tiny p-value, and the fact that the effect is of **literally no practical relevance** is totally lost.
8/
8/
This also means that with a large enough sample size we can reject basically ANY null hypothesis (since the null hypothesis never exactly holds IRL, but it might be âclose enoughâ that the violation of the null hypothesis is not important).
9/
9/
Want to write a paper saying Lucky Charms consumption is correlated w/blood type? W/a large enough sample size, you can get a small p-value. (Provided thereâs some super convoluted mechanism with some teeny effect size⊠which there probably is, b/c IRL null never holds)
10/
10/
THREE: SEEK AND YOU SHALL FIND.
If you look at your data for long enough, you will find something interesting, even if only by chance!
In principle, we know that we need to perform a correction for multiple testing if we conduct a bunch of tests.
11/
If you look at your data for long enough, you will find something interesting, even if only by chance!
In principle, we know that we need to perform a correction for multiple testing if we conduct a bunch of tests.
11/
But in practice, what if we decide what test(s) to conduct AFTER we look at data? Our p-value will be misleadingly small because we peeked at the data.
Pre-specifying our analysis plan in advance keeps us honest⊠but in reality, itâs hard to do!!!
12/
Pre-specifying our analysis plan in advance keeps us honest⊠but in reality, itâs hard to do!!!
12/
Thatâs it for today. Have a great weekend! 
13/13
13/13
