Thread by @MatthewAKraft, #academictwitter It's time to update how we teach about effect sizes & [...]

#academictwitter It's time to update how we teach about effect sizes & the policy relevance of research findings.

Here are some resources:

1) I'm hopeful my paper can serve as an accessible introduction to interpreting the importance of effect sizes.

https://journals.sagepub.com/doi/abs/10.3102/0013189X20912798?journalCode=edra

2) We need to teach students how to "translate" results into terms that are more widely understood (but still accurate!).

See for example work by Bard & @JohnFPane https://journals.sagepub.com/doi/abs/10.3102/0013189X19848729

Translating Standardized Effects of Education Programs Into More Interpretable Metrics - Matthew D....

Evaluators report effects of education initiatives as standardized effect sizes, a scale that has merits but obscures interpretation of the effects’ practical i...

https://journals.sagepub.com/doi/abs/10.3102/0013189X19848729

3) A picture is worth 1,000 words. We need to help students to create better visualizations of research evidence.

Explore work by @EdwardTufte

Or even simple visual guides like this from @Stata
https://www.stata.com/support/faqs/graphics/gph/stata-graphs/

4) We need to teach students about opportunity costs and unintended consequences to contextualize policies and programs.

This @J_HumanResource paper by Jepsen & Rivkin on reducing class sizes in CA is one of my favorite examples:

http://jhr.uwpress.org/content/44/1/223.refs

5) We have to teach students to think about program costs and cost-effectiveness:

Check out the great work by @abrooksbowden including:
https://journals.sagepub.com/doi/10.3102/0013189X18814447

A classic paper by @douglasharris99:
https://journals.sagepub.com/doi/10.3102/0162373708327524

@cbcse has a range of great resources.

Using Resource and Cost Considerations to Support Educational Evaluation: Six Domains - Clive R....

Cost, cost-effectiveness, and benefit-cost analysis are methods used by economists to evaluate public policies. Essentially, these methods rely on impact evalua...

https://journals.sagepub.com/doi/10.3102/0013189X18814447

6) Let's encourage students to ask - can that program or policy be effective at scale?

This @IESResearch i3 report about attempts to scale promising education interventions is sobering

https://ies.ed.gov/ncee/pubs/20184013/pdf/20184013.pdf

7) We also have to reaffirm for students the value of the "gut-check."

Does an effect seem plausible & make common sense?

Is there a theory of action & causal chain of events that would explain the findings?

Check out @NoraEGordon & @clconaway new book for a practical guide:

10) Finally, we should build & refine empirical ES benchmarks for specific fields, contexts & outcomes

@DaveEvansPhD & @Yuan_Fei_ have done just this in a new review of interventions that impact educational outcomes in low- & middle-income countries.

https://drive.google.com/file/d/1HkaK76YJTWGdKWmJYQvVJ3kgW9ieVkO4/view

10) Finally, we should build & refine empirical ES benchmarks for specific fields, contexts & outcomes @DaveEvansPhD & @Yuan_Fei_ have done just this in a new review of interventions that impact educational outcomes in low- & middle-income countries. https://drive.google.com/file/d/1HkaK76YJTWGdKWmJYQvVJ3kgW9ieVkO4/view

. . . and here is a paper that provides empirical ES benchmarks for metal health and prevention programs.

https://pubmed.ncbi.nlm.nih.gov/30136245/

P.S. here is the open-access working paper version of my Effect Size paper and an excellent summary blog post.

Working Paper
https://scholar.harvard.edu/mkraft/publications/interpreting-effect-sizes-education-interventions

Blog Post by Matt Di Carlo via @shankerinst https://shankerinstitute.org/blog/interpreting-effect-sizes-education-research

Interpreting Effect Sizes In Education Research

Interpreting “effect sizes” is one of the trickier checkpoints on the road between research and policy. Effect sizes, put simply, are statistics measuring the size of the association between two

https://scholar.harvard.edu/mkraft/publications/interpreting-effect-sizes-education-interventions

Latest Threads Unrolled: