Modern tablets, smart phones, and computers easily deliver interactive learning activities with high quality graphics, but it takes expertise in computer programming and animation to create animated instructors able to teach mathematics and other subjects with natural human-like gestures and speech. Nicoletta Adamo-Villani, professor of computer graphics technology and director of the Idea Lab, and Voicu Popescu, associate professor of computer science, want to automate the process of creating lifelike computer-animated instructors which are able to speak, gesture, and even write on an animated whiteboard without sacrificing delivery eloquence.
“We know that animators can make realistic, believable, and appealing avatars. You see this in movies all the time,” Adamo-Villani said. “With advancements in animation technology and computer power, it’s becoming possible to create real-time interactive realistic avatars. We want to bring movie-quality avatars into educational materials and e-learning.”
By creating software which generates computer-animated instructors, or teaching avatars, from a script, Adamo-Villani wants to make the authoring of digital learning activities available to educators who don’t have programming or animation expertise.
Using the system that Adamo-Villani and her research team is developing, instructors will choose the specific math concepts to be taught (for example, a learning activity involving the linear equation “y = 2x + 48”) and write a textual script which specifies what and when the teaching avatar speaks and gestures. The software will automatically generate a teaching avatar who speaks, writes on the animated whiteboard, and uses context-appropriate gestures.
Automating the creation of effective gestures is challenging, Adamo-Villani said.
“Speech analysis research provides some of the rules for how to make the teaching avatar talk and produce certain language-specific gestures,” she said. “We are figuring out charisma, or personality, gestures and researching how to generate them automatically based on the script.”
Although math problems can be presented in textual form, learning is made easier through an instructor’s speech and gestures, Adamo-Villani said. Young learners find speech and gesture to be important, especially if they are not proficient in reading or fluent in English.
Adamo-Villani partnered with Susan Wagner Cook, an educational psychologist at the University of Iowa. Cook studies how gestures improve comprehension for children learning math.
Although some charisma gestures need to be explicitly written into the script, Adamo-Villani said that some math-related gestures can be automated. She is also researching how the speed of the avatar’s gestures affects perception.
Automating avatar creation will broaden math education
Scaling the creation of interactive math instruction has traditionally been difficult, said Adamo-Villani. A talented human teacher could record a video in which she eloquently delivers a learning activity around the “y = 2x + 48” linear equation. But if the learner wants to interactively change the equation’s slope (i.e., 2) and/or the y-intercept (i.e., 48) to other values, one needs software which could automatically create a teaching avatar able to deliver the learning activity for any numerical values.
Teaching avatars will also enhance research in gesture effectiveness
Adamo-Villani also partnered with University of Wisconsin-Madison researchers Mitchell Nathan and Martha Alibali to study the effectiveness of specific gestures in math education. For example, when discussing the equilibrium between the left and right sides of an equation, is comprehension enhanced when the instructor makes the iconic balance gesture with her left and right hands? To test this, Nathan and Alibali will use Adamo-Villani’s software since it’s able to generate animations in which the only characteristic of the presentation which varies is the presence or absence of the gesture.
“When trying to present the same learning activity to two test subjects, once using the balance gesture and once without, a human instructor will have other minor variances in her delivery,” Adamo-Villani said. “Small changes in speech or other physical movements can invalidate the comparison. With the teaching avatar, we can precisely control every aspect of her delivery.”
Adamo-Villani’s current project evolved from earlier research in which she used gesturing avatars to teach math and science via sign language to children who are deaf.
“I was trying to help them learn subjects that they struggled with,” she said. “The deaf don’t usually find careers in math and science. We had math lessons already prepared, so it was logical to try using avatars.”
In the future, learners might be able to change the avatar’s appearance to suit personal preference and/or cultural norms. Adamo-Villani’s software currently employs two avatar characters: a casually dressed teenager and a professionally dressed young adult.
The children in the earlier study liked the teenager avatar. “We did some studies and learned the kids could relate to the teenager. They liked the stylized look,” Adamo-Villani said. Her research partners at Wisconsin asked for the young adult avatar.
Avatars won’t replace human teachers, Adamo-Villani said.
“Face to face interaction is very important,” Adamo-Villani said. “But I think we can help people in remote locations and people who might not have access to high quality education. You can learn more from a good teacher than from books only. We want to make e-learning scalable, and avatars are scalable.”
Adamo-Villani hopes to expand beyond mathematics into the creation of teaching avatars for surveying-related education in construction management technology as well as computer programming.