BAZKARIA: eating together through immersive telepresence

The Challenge

Immersive telepresence leverages eXtended Reality (XR) technologies to establish remote connection and interaction that overcome physical distances. In comparison with current video-mediated communication systems, it promises closer to real-life interactions, more realism and improved spatial collaboration. However, despite the last couple of decades of basic research in these technologies, they are still nascent. Significant challenges exist related to the representation of users, their interaction with the space, their communication with others, the coherent combination of digital and real objects, or the application of these technologies to real and social environments outside laboratories.

Commensality–shared eating as a culturally and socially significant act that fosters connection, belonging and common meaning–is a natural, social experience common to all. Eating in the company of others, in opposition with eating alone or accompanied by our digital devices, has been proven beneficial for our health and wellbeing. A dining table is a reflection of cultures, hierarchies and social norms. Due to these benefits of eating with others and complexity, commensality is an ideal use case to study how people socialize and communicate. Immersive telepresence technologies developed for commensality will efficiently transfer to many other application domains.

BAZKARIA (Basque word for lunch) looked at the future of immersive telepresence technology in the use case of two persons eating together at different tables. Sharing a dining table at a restaurant remotely implies teleporting the commensals, but also other elements of the experience (e.g. food). Besides, digital-teleported elements must be well integrated with elements at the local environment of users, such as their table or their food. Waiters are also an integral part of the experience, so they must be part of it as well.

The Proposed Solution

BAZKARIA used Vicomtech’s VHOLO as its core technology for immersive telepresence. This solution was refined and evolved through an iterative co-creation process to accommodate the needs of the use case and the deployment space at Mugaritz. It was also iteratively evaluated gathering data from the system and feedback from the users. The most relevant aspects of the solution are:

How do we teleport the elements of the experience?

We used a solution based on pointcloud volumetric video. This technology mixes colour and depth information from multiple cameras to create holograms. We use two RGBD cameras to capture each table. Our system uses the colour and depth data from these cameras to reconstruct a pointcloud representing the user, the food and so on.

One relevant aspect of creating a telepresence experience, is ensuring that all important elements are captured, and elements that should not be in the experience are left out, e.g. for privacy reasons. We started explorating this idea through the development of a segmentation pipeline segmentation for volumetric video that can divide the scene in the existent elements and used as a starting point to create an object-based immersive telepresence experience where each element is managed differently.

How do we create the illusion of sharing a table?

After the capturing of experience elements and the creation of holograms for each table through the volumetric video system, we need to present these to the user on the other table. For that purpose, we implemented a web-based mixed reality interface that works across different Head Mounted Displays (HMD), i.e. VR glasses, in the market. Mixed reality uses pass-through video captured by the external cameras of the HMD to create a 3D canvas where digital objects can be rendered.

In BAZKARIA, we leveraged this technological development to create the illusion of sharing a table, by aligning holograms of the other, remote-digital, table with the local-physical table. Besides seeing each other, commensals could speak with each other real-time, and also with the waiter. When the waiter approached one of the tables physically, his or her hologram was seen at the other table, and the three users could maintain a conversation, creating a coherent situation for users.

The Result

This project produced a diverse set of results:

• BAZKARIA prototype for immersive telepresence in a restaurant. A prototype including the technological evolution required to carry out our pilots and to deploy our systems in Mugaritz. Part of these enhancements feed back to our core solution VHOLO, e.g. the volumetric video segmentation pipeline.
• Evaluation with over 50 users in different gastronomic experiences. The data captured includes video and audio recordings, movement sensors, interactions and feedback surveys or interviews.

The combination of qualitative and quantitative constitutes a rich dataset that increases our understanding of immersive technology and its use in a social scenario.