The robot acts as a facilitator of social communication with children. Image: César Hernández (CSIC)

CSIC Communication/ICMAT Communication

A new social robot developed by researchers from the Spanish National Research Council (CSIC), an agency attached to the Ministry of Science, Innovation and Universities, assists a team of therapists in the treatment of children with autism. The device uses mathematical models and artificial intelligence to make autonomous decisions and express its own emotions based on its interaction with the children, with the aim of improving their social communication and mood. Initial trials were conducted during June in Madrid through learning activities designed by specialists from Deletrea, a centre for the treatment of people with autism and language development disorders.

The device represents a support tool with great potential for therapies for children with Autism Spectrum Disorder (ASD), a neurobiological condition that affects communication and social interaction. In Spain, more than 90,000 pupils have this disorder, representing over 30% of students with special educational needs, according to data from the Ministry of Education, Vocational Training and Sport for the 2023–2024 academic year.

The robot, developed within the Emorobcare project, “is designed to interact with people, is low cost and has multiple potential uses,” explains David Ríos Insua, Research Professor at CSIC at the Institute of Mathematical Sciences (ICMAT), who co-leads the project together with Juan Antonio Rodríguez Aguilar, Research Professor at CSIC at the Artificial Intelligence Research Institute (IIIA). In this initial phase, the robot has been programmed to improve the language skills of children with autism.

“The robot is a co-therapist: a mediator and facilitator of social communication with the children,” explains Sandra Freire, one of the founding partners of Deletrea. “In autism, without motivation, it is very difficult to teach. And the robot is an extremely motivating element, providing specific stimuli (lights, colours),” she adds.

Unlike other social robots available on the market, this one stands out for its emotional and decision-making model, developed by the ICMAT team. “This model enables the robot to decide how to act and express emotions depending on its interaction with people,” says Ríos Insua.

To communicate with children and therapists, the device integrates different artificial intelligence (AI) models. In a conversation, it first uses speech recognition models that convert what people say into text; then, through language models, it generates a response appropriate to the context of the conversation; and finally, voice generators convert the text into audio in the robot’s voice. It also uses vision models to recognise, among other things, head and hand gestures and to estimate the attention of its interlocutor, combining the visual and oral information it perceives.

“One of the main challenges has been ensuring that all these AI models provide smooth interaction with the user, despite running in an environment with limited computational resources,” notes Juan Antonio Rodríguez Aguilar.

Autonomous decision-making
The robot makes decisions autonomously based on a set of predefined “vital objectives”, which depend on the application context. In this case, “its most basic need is to have energy. Once satisfied, it tries to interact with the child and then seeks to teach them things and contribute to their well-being,” explains Ríos Insua.

At every moment, the models calculate which action—telling a joke, continuing a game or switching therapeutic activity—maximises the expected utility of these objectives, prioritising the most basic ones. Moreover, to do so, the system predicts how both the child and the therapist might react to its actions.

The consequences of these decisions directly influence the robot’s emotional state, which in turn modifies its own objectives and behaviours. “Basically, if the outcome of a decision is positive, the robot becomes happier; the more unexpected that outcome is, the greater the intensity of that emotion. If it is negative, the opposite occurs,” explains Ríos Insua. These emotional variations are reflected in its facial expression, its way of speaking and its behaviour: when it is happier, it tends to pursue more advanced objectives; when it is sadder, it focuses on more basic ones.

The robot makes decisions autonomously based on a set of predefined “vital objectives”. Image: César Hernández (CSIC)

AI and design adapted to therapeutic communication
The team has adapted speech generation models to fit the typical dynamics of therapeutic sessions. “Therapists often give clues to children by stretching out sounds—for example, ‘the apple is rrrrr…’ to suggest its colour—or by using whispering as a communicative resource,” explains Rodríguez Aguilar. In addition, they use different tones of voice—emphatic or cheerful, neutral, sad or angry—that facilitate the child’s understanding and response.

To reproduce these nuances, the team has cloned the voices of an actor and an actress, from which the robot can generate the different tones of speech, whispers and drawn-out sounds required in therapeutic activities.

In the branch of AI that enables robots to interpret the visual world, known as computer vision, the gesture recognition model has also been adapted, among others. The system takes into account forms of non-verbal communication common in some people with autism, such as pointing at objects, moving the head to express affirmation or negation, and making hand gestures to request, greet or indicate “stop”.

Moreover, the robot’s facial and bodily expressions have required particularly careful and tailored design. A limited set of expressions has been created to make them easily recognisable. “Emorobcare offers interaction with fewer social stimuli (gaze, body movements, gestures, facial expressions, etc.), which can cause confusion, so it may be clearer for children,” says Freire.

Beyond this application, the robot has many other uses that leverage its affective and decision-making capabilities, for example in supporting older people, in language teaching or in assisting long-term patients in hospitals.

The work is carried out in the frame of the Emorobcare project, funded by the AI Excellence programme of the State Secretariat for Digitalisation and Artificial Intelligence of the Ministry for Digital Transformation and Public Function.

ICMAT and IIIA team that developed the project. Image: César Hernández (CSIC)

—

This text is an automatic translation from the original Spanish version.