My online experiment for children has been trolled by fake/ not-serious participants!

A thread on some lessons I’ve learned that I hope can be helpful to others.

1/9 After Covid hit, I spent a lot of time turning an in-person experiment for 4-6-year-olds into an online touchscreen game that kids could play at home with parent consent and supervision.

2/ 9 At first recruitment and testing went great. I successfully piloted around 70 children in a month and results looked similar to in-lab data. Participants were recruited through Facebook Ads and emails to past @ChangingBrain participants.

3/9 Then recruitment slowed down, so I increased compensation from $5 to $10 on our Facebook Ad. This increased our signups from ~4/day to ~50/day! But then the data started looking fishy...and I remembered all the horror stories of MTURK data quality assurance. So I investigated

4/9 We anticipated needing to verify that children, not adults, were actually playing the game so we audio record the participant saying their name and age in the game. From these recordings, it was clear that some new participants weren’t children.

5/9 How else did we spot fake participants? Many participated at weird times (4am, 10pm), played the game using different subject codes back to back, and either took a really short or really long time to play.

6/9 Some fake participants also marked that they found out about the study through @helping_science, but we are still waiting for our study to be posted there. So strangely that was a helpful screener.

7/9 We think 65/ 88 people who we invited to participate were non-serious participants!

The bottom line: If you are running asynchronous studies with children online, you need many checks to ensure data quality!

8/9 Some checks I recommend implementing to catch fake participants: audio recording parts of the experiment, asking questions in the experiment that can be matched to a prior survey, having catch questions about say, recruitment avenue, in the consent and checking timestamps.

9/9 If you have good data quality checks for asynchronous child online testing, I’d love to hear about them!