Thread by @dskamper, Since the apparent conclusion of this study is not one I prefer, [...]

Since the apparent conclusion of this study is not one I prefer, I thought I& #39;d read through it to challenge my presumptions.

Tldr;While It& #39;s great that scholars are asking these questions. This paper comes nowhere close to saying what @davidshor claims it does.

A thread:

1/ https://twitter.com/davidshor/status/1385669767193825283">https://twitter.com/davidshor...

https://twitter.com/davidshor/status/1385669767193825283

Let me first begin by saying what I& #39;m *not* trying to do here. First, I don& #39;t know stats well enough to challenge their stats. Second, I& #39;m not going to use ad hominems to claim the authors& #39; political views are fatal flaws. The work should be judged on its own merits.

2/

On its merits, there is significant reason to doubt that this study is telling us much of anything.

What did they do?

They got a large # of people to do an online survey. They were asked to read a policy proposal and rate their support on a scale.

The proposal...

3/

...was written with either a "neutral" framing, or a "race,", "class," or "race-class" framing. The authors claim their data shows the "class" framing increases policy support the most.

Now, I said I& #39;m not going to do stats, but...

4/

...the first thing I& #39;ll note is that the size of this effect seems pretty small. Respondents rated the proposals on a 1-7 scale. The class frame was better than the neutral frame by all of 0.09 points - meaning if the average neutral score was 4, the class score was 4.09.

5/

Bear in mind "statistically significant" means the math strongly suggests the difference isn& #39;t chance. It doesn& #39;t say that the difference would be a huge deal in the real world.

But my main problem is in the design of the survey.

6/

The design of the survey assumes two things I find deeply problematic:

1) the test messages used in the survey were chosen without bias, and;

2) the responses were measuring what the survey designers intended to measure.

Let me tackle one at a time.

7/

I had hoped, that, at minimum, the authors were relying upon some previous work to guide their creation of the messages tested in the survey. They weren& #39;t. They basically sat down and wrote out some messages.

Now, I& #39;m sure they tried hard, I am.

8/

But understand how bias could play a role here. If the study& #39;s authors were *in any way* predisposed to believe the hypothesis they supposedly proved in this study, then they should never have been the ones to write the messages. I mean, come on, that& #39;s survey design 101.

9/

But that& #39;s what they did. They looked around at media sources and grabbed some stuff and shaped it into policy statements. Sorry if that is disrespectful but that& #39;s what they did.

Given the small size of the effect, even a slight bias (even unconscious)...

10/

...in the shaping of these messages could& #39;ve accounted for the results.

I presume there& #39;s tons of research on this, none referenced by the authors. Nor did they seem to take a basic step of running these test messages by people who had different biases to get their read.

11/

(Let me make clear - I& #39;m not suggesting a conscious effort to produce a false study. I& #39;m suggesting the authors are human and have biases that could easily have included the wording of the statements. and they don& #39;t appear to have made any effort to correct for that.)

11a/

Put more simply:

A basic tenet of the scientific method is to make sure your experiment isn& #39;t designed in a way that tilts the scales to produce the result you want. At this basic level, this survey design fails.

12/

Now, let me move onto point 2 - did the survey measure what the authors intended it to measure? Did people read the class framing and think of it as class framing? Did they read the race framing and think of it that way?

Now, that& #39;s a simplistic way to put it, I know.

13/

But, still, the point stands - do we know that what respondents were responding to was what the authors wanted them to respond to?

I think there& #39;s good reason to believe the answer could be no.

A few examples.

14/

First, in every one of the six policy areas where they had test messages, the "race + class" framing message was notably *longer* than the others. On student debt, for example, the race+class message was 17 lines, the class-only message 11. I& #39;ll bet all the money...

15/

...in my pockets against all the money in your pockets that there& #39;s research out there someplace that says shorter messages are often better. Do the authors speak to it at all? No.

But I feel like my second example is even more telling.

16/

Look at these three: the "race" and "class" messages on marijuana, minimum wage, and climate, probably in the wrong order because I do photos badly On Here.

See any pattern?

I see at least one:

17/

the class messages note positive effects of the new policy, with inspiring language, while the race messages center on how the current policy is bad.

Minimum wage, race: "boost incomes for people of color". Class: "lift nearly a million people out of poverty."

18/

Climate change, class: "improve the economy by giving people jobs... create over ten million new jobs.

Race: "Black and brown communities bear the brunt of pollution and environmental degradation."

19/

Marijuana, class: "the profits and employment that legalization can bring."

Race: "in every state Black people are far more likely to be arrested."

The minimum wage "race" message nods in the direction of improvements, but the class one centers improvement.

20/

And in the other two, "race" message only talks about the harms currently suffered, with nothing at all about the impact of a policy change.

So, what is this study measuring? Is it measuring "race" vs "class" framing, or is it measuring "positive" vs "negative" framing?

21/

I& #39;m gonna go ahead and guess that there& #39;s research on positive vs negative messages, too, research that once again the authors did not mention in their piece.

There are other points I could make but I& #39;ll leave it with those for now because I& #39;m hitting the thread limit.

22/

I& #39;ll just close by saying that the content of the study itself does not bear out the enthusiasm for it expressed by @davidshor, @JamesSurowiecki, and others. By all means, this subject should continue to be investigated. But it& #39;s going to need a lot more rigor than this.

End/

Latest Threads Unrolled: