Large-scale randomized experiments reveals that machine learning-based instruction helps people memorize more effectively

https://www.nature.com/articles/s41539-021-00105-8, Data and code

We perform a large-scale randomized controlled trial involving ~50,700 learners of at least 18 years of age in Germany who use an app to study for the written portion of the driver’s permit from December 2019 to July 2020 and gave consent to participate in the trial.

During the randomized controlled trial, each learner was randomly assigned to a “select”, a “difficulty”, or a “random” group throughout her entire usage of the app. In the “select” group (n = 10,151 learners), the questions of each study session were chosen using our machine learning algorithm. In the “difficulty” group (n = 34,029), they were chosen in circular order proportionally to the initial difficulty, i.e., easier questions first. In the “random” group (n = 13,600), they were chosen uniformly at random with replacement. The only difference in app functionality across groups was due to the item selection algorithm and learners do not know to which item selection algorithm they have been assigned.

After controlling for the length and frequency of study, we find that learners for whom a machine learning algorithm determines which questions to include in their study sessions remember the content over ~69% longer.

In terms of engagement, learners of the “select” (“difficulty”) group were 50.6% (47.6%) more likely, in median, to return to the app within 4–7 days than learners of the “random” group. However, learners of the “select” group were also more likely to stop using the app in the initial 2 days than those of the other groups.