Tech moves to holograms, fine dining and teleporting classes for more immersive experience

Researchers are pushing telepresence technology beyond video calls, from Star Wars-style holograms to virtual fine dining and classrooms that leap across space.

By Tom Cassauwers

For Maurizio Murroni, associate professor of telecommunications at the University of Cagliari in Italy, Star Wars is much more than a science fiction film franchise – it is a source of inspiration for his work.

“In the Star Wars universe, they regularly project holograms of people to have conversations across the galaxy,” Murroni said, explaining his fascination. “That’s what we are building right now.”

Murroni leads an EU-funded initiative called HEAT, which is developing immersive telepresence technologies with low-latency – keeping delays close to zero – to bring holographic communication closer to everyday reality.

How would that work? A standard camera, such as the one on a smartphone, could scan a person. That scan can then be transmitted across the globe and turned into a hologram, visible through virtual reality headsets or projectors.

Holograms have been around for a while. In 2012, for example, a hologram of deceased US rap singer Tupac was projected onto the stage of the Coachella Valley Music & Arts Festival.

So far, however, they have remained mostly a showbiz gimmick, without much impact on everyday life. Many of these early systems were expensive and unwieldy to use, and required very fast internet connections.

Initiatives like HEAT are trying to change that. The researchers aim to make immersive telepresence practical at home, through holograms and a range of other technologies, from virtual and augmented reality to realistic avatars in virtual worlds.

Such systems could be crucial for uses ranging from teleconferencing to healthcare, connecting people who are physically distant.

“During COVID-19 we all saw how important it was to offer a real experience to people,” said Murroni. “We want to bring people together.”

HEAT is not alone in this effort. Another EU-funded initiative, SPIRIT, is working towards a similar goal. With testbeds in Germany, Belgium and the UK, it is focused on making immersive telepresence more accessible.

Behind the holograms, the real hurdle is the network carrying all that data. Much of SPIRIT’s work targets this underlying infrastructure – a key bottleneck for the technology’s wider adoption.

Transporting data efficiently is essential for data-intensive applications like real-time telepresence. The SPIRIT team has tested and improved network protocols to enable this.

“When you are doing a video call, you use a protocol to send packets of data,” said Tim Wauters, a postdoctoral researcher at the University of Ghent and the research centre Imec, in Belgium.

“We improved some of these protocols and reduced the latency, or delay on the signal, from a number of seconds to milliseconds.”

That can make a huge difference for certain tasks at a distance. Short delays allow people to interact more naturally.

Musicians could play seamlessly together online, conversations will flow and doctors might be able to assess health issues via a video stream.

The SPIRIT team has tested these network advances in real-world trials. For example, Mugaritz, a Basque restaurant with two Michelin stars near San Sebastián, Spain, approached the team to help recreate their restaurant experience virtually.

Together with SPIRIT researchers, they designed an immersive restaurant setting that participants could engage with from home.

“You consume the dishes at home, but have the feeling of being in a restaurant,” said Peter Van Daele, a professor at Ghent University and Imec, and SPIRIT project coordinator. “It’s an extreme example, but it shows what we are capable of.”

They ran a small pilot involving 50 people who used virtual reality glasses to engage with each other during dinner.

In addition to their own research, the SPIRIT researchers also invited people from across Europe to propose their own telepresence projects and get SPIRIT support. The Mugaritz dining experience was one of several pilots.

Other projects go far beyond restaurant visits. Telepresence could support psychological therapy, remote robot control and more engaging forms of training. It may also change how we receive healthcare.

“A doctor might use this technology to talk to and look at a patient,” said Van Daele. “We even developed tactile systems whereby the doctor can feel parts of the patient’s body from a distance.”

Education is another likely beneficiary of telepresence. “Imagine if students could jump across the solar system,” said Murroni. “They could get close to the Sun and feel its heat. Or they could go to Mercury’s craters and feel the cold on the dark side of the planet and the heat on its light side.

The breakthroughs achieved by the SPIRIT and HEAT teams are helping to spark a wave of creative experimentation in other fields. “We are providing tools to creatives, where they can go beyond the realities of the normal world,” said Murroni.

The HEAT researchers, for example, have worked with theatres. “What if actors could use smells to create emotions in their theatre audience?” asked Murroni.

“We are working with theatres on this. A headset could be used to transfer smell. They are now creating an artistic act using this. Audiences could smell wine and cheese during a lunch scene.”

Working together with artists and creatives is, according to Murroni, the biggest advantage of a project like HEAT.

“As a scientist, I find it great to interact with artists,” said Murroni. “They are not engineers and they offer these wholly fresh perspectives.”

This article was originally published in Horizon, the EU Research and Innovation Magazine.