A Shared Scientific Infrastructure to do Randomized Controlled Trials and Longitudinal Studies

By Neil Heffernan, Worcester Polytechnic University

Shared scientific instruments help address big challenges in research. The Hubble Space Telescope has helped hundreds of astronomers do studies that none of them could have done if they had to also build the instrument they use. Physicists benefit from shared particle accelerators that are expensive to build and they test their own ideas to better our understanding of the universe. Shared scientific instruments make science more efficient since no single scientist need to create and validate the instruments.

How social sciences can benefit from the shift to a digital world

Social sciences could benefit from such an instrument to continue the forward push to do more randomized controlled experiments. Social science is reaping the benefits of our cultural shift to the digital world and the many ways people interact with online platforms. There is a whole conference on digital experimentation held at MIT every year where the big companies (Microsoft, Facebook, Amazon etc) come and talk about how they are using their platforms to do massive amounts of AB testing. (Don’t get confused: business folks tend to call randomized controlled experiments/trials AB testing but its the same underlying idea of randomizing users to conditions to learn what leads to the best outcomes). Google, Facebook, and other web-based companies run hundreds of experiments a day on all of their users to maximize things like revenue from ads and user experience. What if we harnessed that power for education? What would a shared scientific instrument for education look like that could run experiments to help the world maximize student learning?

ASSISTments: An example of a shared platform

I am running one such platform utilizing ASSISTments. Over 50,000 students use it in class and for homework that teachers assign. The assigned content is from our library with a wide range of topics including math, science, and foreign language through the largest content library is middle and high school math. Content is also created and shared by teachers. Due to funding from NSF, the Department of Education, and others, we can make the whole system free to students, teachers, and researchers.

ASSISTments blends ASSISTance with assessment. It works by giving students feedback as they go, while teachers get actionable, timely reports. I have used this tool to publish two dozen studies so far, but now, due to a new NSF grant, we created ASSISTmentTestBed.org which allows external researchers to propose better ways to help students learn with technology. Researchers get their results through anonymized data emailed to them, with some preliminary automated data analysis telling them if there is an effect on their desired outcomes. A dozen researchers have submitted studies while 5 have already resulted in published peer-reviewed papers. We open these data sets so others can write their own papers using these data sets; already 25 papers have been written by others that use our open data sets.

In my view, we as education researchers need to think more holistically about supporting shared education research platforms. Astronomers and physicists know they need to lobby the government to get funding for large scale instruments, but education researchers don’t see the forest for the trees, failing to ask for larger tools they cooperatively share. This commonly devolves into educational researchers Balkanized into competing for scarce research funds, and policymakers consequently wondering if educational researchers are serious about doing science. I believe nonprofit, university-based platforms could help more researchers do better open science more efficiently, and increase our understanding of human learning dramatically, while at the same time deliver better educational outcomes. Come to this talk and learn how to create a study.