Dustin Palea

Large classes make it challenging to provide individualized feedback, especially for complex skills like qualitative coding. Annota leverages AI and peer-based insights to generate accurate hints, guiding students to refine their annotations. Our research shows that this approach enhances learning, provides accurate feedback with minimal student input, and turns large classrooms into an advantage rather than a limitation for learning at scale.

In large classes, providing individualized feedback is a major challenge, especially for complex skills like qualitative coding. Annota is a peer-based AI system designed to address this issue by leveraging the collective work of students. Using the Dawid-Skene expectation maximization algorithm, Annota compares student annotations across the class and intelligently aggregates them to generate hints, helping students refine their work. These hints suggest when students may have missed an important annotation or included an unnecessary one, prompting them to reflect and improve. Our research shows that Annota produces highly accurate feedback, performing on par with the best student in the class, and requires input from only a small number of students to generate accurate hints. Students reported that it helped them better understand research questions, evaluate their own annotations more critically, and understand when they were over- or under-annotating. Ultimately, this work demonstrates that large classes, rather than being a limitation, can actually be leveraged to enhance learning at scale.