Vicomtech joins the European LUMINOUS project revolutionizing XR interaction through natural language technologies

Vicomtech is part of the European consortium leading LUMINOUS (Language Augmentation for Humanverse), an ambitious project that aims to transform how people interact with extended reality (XR) environments by leveraging natural language and multimodal large language models (MLLMs). The goal: to build XR systems that adapt to non-predefined users and environments, enabling fluid communication with the surroundings and instant access to up-to-date global and domain-specific knowledge.

Vicomtech leads the Natural Language Processing (NLP) efforts alongside HiTZ, within a multidisciplinary consortium made up of 12 partners from seven European countries, including top research centres, universities, and innovative tech companies. The project, funded by the European Commission through the Horizon Europe programme, brings together expertise in AI, Augmented Vision, Computer Graphics, Neurorehabilitation, and Ethics, with a strong user-centric approach.

LUMINOUS will use speech interfaces and adaptable avatars to assist users in task execution, providing visual aids, personalised instructions, and context-aware interaction. It will also explore how MLLMs can identify and describe previously unknown objects, broadening their real-world usability.

The project includes three pilot applications:

• Neurorehabilitation, supporting stroke patients with aphasia (led by MindMaze and UCL);
• Immersive industrial safety training (led by Ludus);
• 3D BIM architectural design review (led by Mindesk).

With this project, Vicomtech reinforces its commitment to developing cutting-edge technologies that serve people and strengthens its position as a reference in applying natural language to immersive and intelligent environments.