“Basically, we wanted to build a learning framework that optimizes the learning experience according to the emotions and performance of each individual learner. We were looking at all possible cues: your face, your posture, the way you speak, how well you were doing with the tasks. Furthermore, MaTHiSiS had to run on anything from classical PCs all the way up to robots”, says Dr. Stelios Asteriadis, associate professor at the Department of Data Science and Knowledge Engineering (DKE).

“Our job was to combine all those different cues, and to set up the automated recognition of facial expressions. We also enabled the use of artificial intelligence to map simple digital interactions to emotions and, at the same time, made sure all of this could be used to optimize the learning experience.”

A second challenge comes from the fact that people spend every waking moment learning. “Nowadays, we are surrounded by social media and all sorts of devices. We continuously learn in different ways, even if we are unaware of it.” Learning things you already know isn’t particularly engaging. With so many informal opportunities to learn, the level of prior knowledge varies wildly. This too calls for a personalised approach to learning.

“Suppose the big goal is to learn math”, says Dr. Asteriadis. “You need to learn addition, subtraction, division, and so on.” Teaching these concepts in a fixed order risks boredom in learners who are already familiar with some of them. Therefore, the team went with a non-linear way of teaching. “We broke learning material down into chunks of smaller learning subgoals, so that the algorithm could jump from one subgoal to another. The algorithm determined your learning path based on your performance and your emotions. The Maastricht team was responsible for this adaptation as well, for allowing the system to recommend the best material for a particular student in a particular moment.”

But… Wait, this kinda works

MaTHiSiS was extensively tested in five different countries across Europe, with a focus on students with special needs. “The results were very encouraging. Among other things, we looked at the engagement of learners compared to the emotion recognition we derived, and found that they were correlated.” Put simply, that means that the emotion recognition algorithms could successfully identify when someone was engaged in learning. “Pedagogists also compared learning outcomes with and without MaTHiSiS, and with and without emotion recognition. We are now exploring opportunities to refine, reuse and retest the system in different environments.”

In real life situations, recognising emotions isn’t as easy as it seems. “Don’t forget that different people express their emotions in different ways. It even changes within the same person: one day I may express my emotions in one way. The next day, I could be moody and I express my emotions in a different way. This is challenging to overcome for a machine.”