"Protestantism makes Filipinos more industrious and enables them to rise out of poverty while also proving their dependence on material incentives" is a take I would've expected from Harvard men in 1920 but here we are https://twitter.com/QJEHarvard/status/1272598345836400641
Some questions at first glance

1.) Where are the Filipino scholars in the works cited? I'm sure they'd have something to say about religiosity and poverty in their own country

2.) How does one "randomly remove religion" from an evangelization program?
I'm skeptical that religion can be removed from a program using (as the authors cite Weber) Protestant theology as the basis for its assumptions about how income and work connect to spirituality

If the solution was "just don't teach the Jesus parts" then that seems insufficient
Looking over the website for the organization the authors partnered with (International Care Ministries) I'm even less convinced that you can just remove religion from their program but maybe that's also true of American philanthropy in general https://www.caremin.com/about-icm 
What were the consenting procedures for participation in the study groups? Did the authors just take participation in the overall ICM program as consent for being in the study? Given how poor the participants are compared to the research incentives that's already coercive
I ask having already gone through (and passed) an IRB to work in an indigenous community in the Visayas

My proposal wasn't evangelical but 30-60 minute interviews on what community members thought about folklore and still got raked over the coals so this one passing surprises me
So they did "just take out the Jesus part" by randomly removing the Protestant Values (V) module but again how exactly do you remove Protestant values from the curriculum when the same assumptions underlying that one module are also built into the other two?
"At six months, we find that those who were invited to receive the V curriculum have significantly higher religiosity"

Well yes because if I know my Protestant missionary and finance programs (which I do) then the groups are set up to keep going and "hold members accountable"
But how is religiosity quantified in this study? I don't see a citation in this section that explains how that was done

Is it based on church attendance? Scripture reading? Survey responses to test agreement with statements like they did to measure "grit"?
"Relative to the no-curriculum control, the HL group has significantly higher income and perceived relative economic status"

What heightens "perceived relative economic status"? Feeling more richly blessed than their neighbors, making it necessarily theological in nature?
Again I wonder about the coercion element since the authors conducted their random selection of test and control communities within ICM's selection criteria (impoverished enough that feeding programs would be a sufficiently motivating incentive to do the 16-week course)
"a significant aim of our study is to establish, in the context of a randomized controlled trial, that intentional exposure to a religious program can generate the critical first stage: an exogenous change in religiosity"

The other thing that establishes this is colonial history
Really questioning the need to do a whole partnership with an American-led missionary organization if the big takeaway is that targeting communities for religious instruction supported by external incentives contributes to changes in religious behavior

Kinda goes without saying
I still find it hard to believe that religious instructors were actually convinced to remove "religion" from their program even though they did agree to remove the module labelled Protestant theology

That is if we view religion as belief and behavior rather than just the latter
Even if ICM could and did remove religion from their program in the experimental groups, there's still the major ethical concern of what the data will be used to support

ICM is certainly excited to see that their program (now supported by a study!) makes people more Protestant
While obviously researchers don't have absolute control over what their data is used to justify, it's still a huge issue that this study's conclusions ultimately feed back into what ICM is already doing

There's no way they'll use the data in a way that excludes the Values module
Which makes this study look less like a rigorous examination of what constitutes religious behavior or how religiosity correlates to economic activity and more like a testbed for ICM to test ways it can improve the program's effectiveness over its 30-month duration
These statements don't really square up

Communities targeted as experimental and control groups were ones selected by the pastors specifically because they weren't already part of ICM's reach, so really what the authors did was facilitate expansion to 320 new communities
Treating that expansion as inevitable ("work they were doing anyhow") obscures the authors' role in instigating the conditions for missionization to occur in all targeted communities, since I find it hard to believe that ICM just pulls out of the control groups after the study
This all just circles back to my earlier tweet about the ethical questions researchers should ask themselves first about what their data cold be used to support

In this case the data directly supports ICM's mission and makes suggestions for how it could improve its tactics
So the authors paid ICM to target new communities where they could carry out a modified version of their missionary program which will definitely be used later to support actual missionary efforts in these same communities and others

Red flags were raised
I really can't see how this all doesn't just benefit ICM in the end

They get NYT-level publicity from economists who pay them to make contact lists of potential new converts and write up a formal IRB-approved and peer-reviewed set of recommendations for how they can get better
One of the authors got the group of which he's the president and founder to write up the informed consent documents such that the experimental groups didn't know they were going to be taught by missionaries

This is by definition not "informed" and can hardly be called "consent"
There are blind market research studies and then there's "we paid this religious organization to teach you a randomly selected curriculum that could've included religious instruction and worded our post survey specifically so that most of you would be too dumb to figure it out"
Like honestly these research methods are so manipulative and ignorant and colonial and anachronistic that I'm having to ask myself if it really is as bad as it looks because boy does it look bad
Of course none of this looks better in light of Karlan (the "agnostic Jew" as he described himself in the NYT article that covered the study) treating it all as comedy that economists would write about religion this way

I'm reading and laughing but not because it's funny
Also raises another question of why the authors thought partnering with and supporting a Protestant missionary group in Mindanao (specifically its offices in Koronadal and General Santos City) would be a good idea given the last oh 500 years of colonialism, external and internal
idk seems like a problem that 2/3 of the households in their survey that's supposed to track changes over time never had baseline data to begin with so they had to extrapolate that from only 1/3 of the data
The overtly religious questions may only randomly appear in the questionnaire but how will the authors address the stigmas in Philippine society against outward expressions of areligiosity, especially to strangers who aren't revealing that they're working with missionaries?
Once again the lack of citations by anyone working in the Philippines or even being of Filipino descent is really telling because none of the many social/historical factors that influence the role religion plays in Filipino public life get brought up at all
Those outcomes certainly are of interest to ICM because their system relies on the three modules (Values, Health, and Livelihood) supporting each other such that I still doubt that they're really separable
And why would Protestantism, sanitation, and work be separable when the last 100+ years of American missionary work in the Philippines post-1898 has largely relied on those qualities being conflated with righteousness, whiteness, and civilization? That's basic background stuff
The authors stated earlier that the pastors who chose who got invited to participate usually selected the female head of household

What roles do gender play in religious and economic life in the Philippines and how might this affect the data? This study is silent on that point
lol at economists calling theology "counterintuitive"

Given that Christianity has been arguing over grace vs works for 2000 years and individual interpretations can vary so widely, I question how this survey can prove that one answer or another indicates more Protestantism
ICM even states that their mission in teaching values and livelihood together is to make program participants feel that God is in control and their suffering has a salvific effect so saying "they thought they were better off even though they weren't" shouldn't be surprising
So all program participants (even the experimental groups) reported increased religiosity in the 6-month post-survey questions? I wonder why that is when the program was run by missionaries who selected households based on their own receptivity criteria and not the authors'
Again, because the authors set out to survey 7999 households spread across four locations (Bacolod, Dumaguete, General Santos, and Koronadal) then ran out of time to collect baseline data but went ahead with the study anyway
Hoping this section explains how putting pastors in charge of teaching the modules according to the research design (which a few didn't do) is supposed to control for the socialization that goes on outside the actual lessons

How could that be free of religious instruction?
My running concern through this whole article is that while it still might've been ethically dubious to work with extant ICM groups, what the authors did was incentivize ICM to make entirely new groups and then failed to assemble the baseline data needed for proper comparison
Meaning the authors weren't just observing something that was already going on, as I've pointed out before

They were the direct cause of the missionary growth that created the conditions for their experiment and Karlan used his existing organization to further both
So Karlan pays his own employees at Innovations for Poverty Action with Gates Foundation money to facilitate a missionary program that he was also paying with Gates Foundation money so that he could watch what the missionary program did with it and report back to them

Huh
Now they're getting into specific questions from the survey and I'd like to know about the wording, namely what language(s) they used since between the three islands (Negros, Cebu, and Mindanao) where their sites are located you've got at least four major languages in use
Working in majority Christian towns in Mindanao means most of your respondents will be speakers of either Hiligaynon or Cebuano with Maguindanao, Tagalog, Blaan, and T'boli mixed in

Am I to expect the authors made translations in all these languages or did IRB just let it slide?
Since my training and experience are much more qualitative than quantitative I'd need someone with the latter experience to run through the statistical methodology

Even so my main problems with the study aren't with the statistical models but with the premise and research design
Seems fairly obvious that contact with a Values module telling participants that not working hard is sinful combined with contact with Americans and more affluent Filipinos would result in lower perceived income levels

If only there were baseline data for 2/3 of the sample
Again, just mentioning that "insurgent violence stopped us from following up on Christian missionary work in Mindanao" without talking about why there is insurgent violence in Mindanao (hint: it's in large part because of Christian missionary work) shows a lack of care to context
As discussed in this thread and in scholarly literature, internal colonialism from migration of Christian Filipinos into majority-Muslim regions of Mindanao post-WW2 plus Spanish occupation means that researchers helping missionaries isn't value-neutral https://twitter.com/stephen_acabado/status/1272263393852837888?s=20
Another predictable aspect of this study is that there's overlap between who the pastors recall as having participated in all groups, as well as indications that even some in the placebo groups were evangelized

Did the authors really expect pastors to not use the contact lists?
It's that or the pastors are remembering wrongly who participated in which group, which the authors also acknowledge, but the fact that it was the pastors who made those contact lists doesn't let the authors off the hook for setting it in motion
Geographic distance between communities that each pastor selected themselves, so likely in the same province or municipality

The authors had no way to control for return visits or other communication after the study period was over so how can they "believe" it didn't take place?
Greater exposure to Protestant missionaries leads to higher rates of people identifying as Protestants

Truly thrilling stuff, a real breakthrough for economics and the social sciences at large

Again, why do the authors need to pay pastors in order to come to this conclusion?
Putting the whole conclusion here for reference

In terms of its contributions to religious studies it seems like a complete nothing burger

In terms of its contributions to ethics I'm eager to see it mentioned in IRB trainings of the future as an example of what not to do
Really this whole thing strikes me as equally obvious (missionization causes changes in religious behaviors and attitudes) and dubious (paying missionaries to find potential converts and then lying by omission about what the participants were taught)
Just a wild throwback to the colonial period that takes a missionary organization at its word that Filipinos are spiritually, medically, and economically deficient and then actively supports them in expanding their operation into historically contested regions of the country
If the authors really can't be bothered to do the bare minimum and cite Filipino scholars, work with Filipinos outside ICM to find a less harmful way to do this research, and do some historical background work to explain their results, then they shouldn't be working there
Some other useful critiques
https://twitter.com/DivDecEcon/status/1272896993539559432
https://twitter.com/rinewithoutacat/status/1272907590012481537?s=20
https://twitter.com/LandonSchnabel/status/1272718839218757632
https://twitter.com/gchelwa/status/1272790160648921088?s=20
https://twitter.com/ErikKatovich/status/1272948927155798017?s=20
You can follow @DGoweyAuthor.
Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled: