[LUM#20] Metaverse in motion

[LUM Magazine, Podcast] - Published on December 20, 2023 in Science-Society

Improving communication between individuals who encounter each other in virtual worlds is one of the objectives of the European ShareSpace project, in which the EuroMov laboratory is participating. This challenge requires reintroducing movement in all its dimensions into these metaverses in order to facilitate social interactions. Benoît Bardy, a researcher specializing in the study of human movement, explains.¹.

Imagine a coworker offering you a cup of coffee. Their hand reaches for the cup on the table in front of them and picks it up. Are they going to bring it to their mouth to drink or are they going to hand it to you? In reality, you probably don't need to ask yourself this question, because you have already identified their intention. "The way his gaze moves from the cup to you, the way he picks it up, the angle of his elbow, the direction of his wrist—these are all signs that give very early indications of your conversation partner's intention, " explains Benoît Bardy, a researcher at the EuroMov laboratory.

Sensorimotor communication

All these cues, which researchers call "movement primitives," form the basis of sensorimotor communication, which plays a major role in social interactions. "Looks, gestures, emotions—these expressions are central to the way we communicate, " explains the movement specialist. We are able to perceive all these cues when we are face-to-face with someone, but what happens when the person we are talking to is not in the same room? "Then we lose much of our ability to detect these intentions in the gestures of those we are talking to."

This is a real research challenge at a time when virtual reality is developing and the metaverse, defined as a hybrid world that can go beyond the real world, particularly thanks to augmented reality, is becoming increasingly talked about. How can we find these driving forces of intention in digital universes? This is one of the challenges of the European ShareSpace research program, in which the EuroMov laboratory is participating.

A project with an innovative vision: the creation of future hybrid social spaces shared by humans and avatars. Here, the famous social sensorimotor primitives are captured using innovative connected mobile sensors, then reconstructed using new extended reality technology. "This allows us to promote virtual communication by recreating these gestures, which can be amplified or even attenuated depending on the application, " explains Benoît Bardy.

Avatar

And to invest in the metaverse, ShareSpace integrates four levels of interaction into its scenarios, blending the real and the virtual. At the first level, called L0, humans interact in the real world to better assess, identify, and calibrate sensorimotor situations. "At this level, we offer synchronization games, which are recognized as vectors of empathy and social affiliation, " explains the movement specialist.

At the next level, or L1, all humans find themselves in a virtual world, where each person has an avatar that copies their movements. "This is a non-autonomous avatar that reproduces our movements and interacts with the avatars of other human beings." In practice, each participant stays "at home" equipped with a virtual reality headset and meets up with others in the metaverse.

The upper level, L2, is characterized by a certain degree of autonomy given to avatars. "For example, when playing a synchronization game together, if one of the participants begins to fall out of sync with the rest of the group, their avatar can adapt its sensorimotor response to get back in unison with the group." Finally, at the last level (L3), we meet Sarah, an artificial intelligence built using a mathematical model. "If one of the participants is lagging behind in the synchronization game, Sarah is able to notice this, go and find them, and take them aside in a personalized way to resynchronize them with the group. We are creating a hybrid world that combines reality, virtual reality, and augmented reality."

Virtual rehabilitation

What are the practical applications for this shared metaverse? "SharesSpace focuses on three areas: health, sport, and art," replies Benoît Bardy. In terms of health, the project aims to treat lower back pain by enabling patients suffering from chronic back pain to participate in group rehabilitation sessions with a therapist. These sessions are unlike any other, with real and virtual patients following the advice of an autonomous avatar. "A therapist represented by their avatar can closely monitor whether the patient is performing the necessary exercises correctly and correct them if necessary, " explains the EuroMov laboratory specialist.

For the sporting aspect, ShareSpace offers cyclists the opportunity to learn how to ride together in a peloton in a safe environment. "It's an app that's very well suited to children, who will learn, for example, how to avoid an obstacle with the help of Sarah, who is there to amplify their parents' gestures to make it easier for them to understand their intentions, " explains Benoît Bardy. An app will also be offered to cyclists with a competitive mindset, "where, on the contrary, it is sometimes necessary to hide your intentions in order to better prepare for a breakaway from the peloton," explains the researcher. See you at the ShareSpace stand in Paris during the 2024 Olympic Games.

Finally, with regard to the artistic dimension, the project's creators have given carte blanche to a collective of artists from the European Ars Electronica Foundation, who will use this technology to create an immersive choreography for the Art and Technology Festival in Vienna in September 2024. This show will be available to watch both in reality and virtually.

1. Euromov (UM, IMT Mines Ales)
   ↩︎

Keywords: LUM#20, LumLu, LUM Magazine, Research, Science, Company