Ok, so there's a lot to unpack here. This was a randomized controlled trial looking at the impacts of fasting on lying in lean and obese people

Headline finding - obese people lie more than lean ones https://twitter.com/YoniFreedhoff/status/1291126474616045571

Now, leaving aside the abhorrent nature of the study, let's look at the science here because I think it is VERY REVEALING

The study was a bit of a headache to unravel, but despite the confusing writing the design seems relatively simple:

People were randomized to either get breakfast before or after they played a dice game

The dice in this game had three evenly distributed colours - blue, yellow, and red

People had to report the colour they got on their first dice roll. If it was blue, they got no money, yellow they got 3 Euros, and red, 5 Euros

Because the game was random, you can use the reported numbers to compare to the numbers you'd get if people reported the colours accurately, and determine whether they were probably lying

Here's the main finding in graphical form. If you look at the columns on the right, they are higher than the ones on the left, so obese people lie more!

Except, there's something a little weird here. The statistical tests don't actually compare obese and lean people

If you look carefully, all of these statistical tests are comparing within-group differences - i.e. how likely it is that within the obese/fasted group the results would've been observed due to chance

But they AREN'T comparing obese and lean people! What you expect to see, going by the abstract and discussion of the study, is a statistical test comparing the proportions BETWEEN groups i.e. comparing obese with lean

Fortunately, the authors did this analysis as well!

You can find it in table 4 of the supplementary materials

What do we see?

Well, here's the table. The bottom two rows are the interesting ones

NO significant differences between reported proportions for pretty much any subgroup of lean vs obese people

In fact, it appears as if the main finding of this paper completely contradicts the results of this analysis. There does not appear to be any statistically significant differences in the reported values when comparing lean and obese people at all!

Now, that's pretty bad. But it gets worse

This study was probably not designed to test the question of obese vs lean

How do we know? Look at the sample size calculation

So, they've computed their sample size based on the idea that they want to detect an effect size of 0.25 between two groups

But...they didn't include any indication of obese vs lean here. The eventual sample size (150) shows us that

Instead, it seems almost certain that the original study just looked at lying in fasted vs breakfasted people

We can actually see this even more clearly because they ran 36 people through the entire procedure only to exclude them after the fact!

This starts to get a bit murky, because I cannot find a pre-registration for the study

That's worrying, because changing your hypothesis after running a study is a classic sign of p-hacking

Another sign is the statistical analysis. I count upwards of 100 comparisons (Fisher's exact test, chi squared etc) with no correction for multiple comparisons

That's...worrying

If you apply a Bonferroni correction to the results, pretty much every statistically significant finding completely disappears, which is not surprising given that they ran SO MANY tests

Bringing it back home, we have this sentence in the discussion

According to supplementary table 4, this simply isn't true!

Obese people had differences in behaviour, but the statistical comparisons DIDN'T SHOW A SIGNIFICANT DIFFERENCE

Pretty major issue, that

Anyway, the paper is abhorrent regardless, but I think it also shows some worrying signs of being constructed after the fact from a dataset of a trial with different aims

Oh, another issue - the paper makes an inherently misleading claim about causality. The primary findings were of a subgroup analysis of non-randomized groups (lean vs obese) and so it's not clear whether this was causal anyway

Because the randomization was simply fasted vs breakfast, the causal attribution for this study should be comparing those two groups, not the subgroups of obese vs lean