Facilities

Home / Facilities / IDEA Laboratory / Projects / Multimodal Affective Pedagogical Agents for Different Types of Learners

Multimodal Affective Pedagogical Agents for Different Types of Learners

Faculty: Nicoletta Adamo, Purdue (PI), Richard E. Mayer, UCSB (co-PI), Bedrich Benes, Purdue (co-PI)

Sponsor: NSF - IIS - Cyberlearning, award # 1821894 (2018-2021), Purdue Instructional Innovation Grant (2018- 2020)

 

IDEA LabAlthough the preponderance of research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective aspects. The project uniquely integrates several areas of research: computer graphics research on life-like and believable representation of emotion in embodied agents, advanced methods and techniques from artificial intelligence (deep learning) and computer vision for real-time recognition of emotions, cognitive psychology research on how people learn from affective agents, and education research on the efficacy of affective agents for improving students’ learning of STEM concepts. The project will: (1) research and develop novel algorithms for emotion recognition and for life-like emotion representation in embodied animated agents, which will be integrated in a new system for creating affective pedagogical agents to be embedded in digital lessons; and (2) develop an empirically grounded research base that will guide the design of affective pedagogical agents that are effective for different types of learners. The PIs will conduct a series of experiments to determine the effects of the agent’s emotional style and emotional intelligence on a diverse population of students in the context of online learning of introductory statistics. The goals of the studies are to determine how to design agents that exhibit emotions that best foster student learning, whether students learn better when the system detects the learner's emotional state and alters the agent accordingly, and whether learning effects are different for different types of learners.

Intellectual merit

The project enables the intellectual partnership of researchers in complementary domains and promises to advance the state of the art in agent design and implementation by integrating findings on effective emotion regulation in teaching with algorithms that produce life like expression of emotions in embodied agents. The project will bring innovation in two areas. (1) It will contribute to our understanding of the effects of affective agents on people’s learning and the interaction between agent’s features, learner’s characteristics and learning effects; and (2) it will contribute to the development of new, effective algorithmic approaches to emotion recognition and emotion animation. In addition to its scientific merit, the proposed project will develop a system of affective animated pedagogical agents that adapt to the learner’s characteristics and can be used by learners of all ages, for education and training in a variety of subject matters and settings.

Broader Impacts

This project seeks to improve student learning of STEM material by examining how to design effective learning environments with pedagogical agents that incorporate both affective and cognitive principles. In terms of instructional impact, this project will result in evidence-based design principles and technologies for how to incorporate emotional intelligence into learning environments involving pedagogical agents, so they not only follow cognitive principles of learning but also affective principles of learning. In terms of technological impact, this project will result in new algorithms and systems that will allow emotionally-sensitive agents to become part of the standard tool-set that educators use to create e-learning applications. The project holds great promise for scalability. Currently, creating believable APAs is a difficult task requiring expertise in animation and programming. When completed, the algorithms and APA system will be available to the public through the project website. Educators will be able to enhance any type of digital learning material with life like agents that display adequate adaptive emotional behaviors for different types of learners, including underrepresented groups, and that are available to students anywhere anytime. The agents resulting from the proposed research are intended to be an effective alternative to expert and caring human tutors or learning companions, and also prepare learners to function in highly technological environments in which the differences between human-computer interactions and human-to-human interactions are becoming less and less pronounced.

  1. You can access a demo video of the system here:  https://www.dropbox.com/s/3ftntror1qcp1dm/%5BUnity%5D%20Procedural%20Animation%20System.mp4?dl=0

  2. A list of publications related to the project can be accessed here: https://www.dropbox.com/scl/fi/seccvygiqfyo9dals5bjc/NSF-IIS-1821894-publications.docx?rlkey=rik41ecdj9eygs3bisgreqyug&dl=0

  3. The Unity executable file of the Affective Agents Animation System and related documentation are available upon request. Please email Nicoletta Adamo at nadamovi@purdue.edu to request a copy.

  4. Sample animated lecture videos produced with the system can be accessed below: