IKUN Project: Large Multimodal Models to Revolutionize Smart Industry

Funded by the Basque Government’s ELKARTEK program, the IKUN project has achieved key advances in anomaly detection, synthetic data generation for quality assurance, and the development of conversational assistants for plant operators.

The main objective of the project has been the research and adaptation of Large Multimodal Models (MLLMs) to the demanding reality of manufacturing plants. The challenge was to overcome barriers such as the lack of large volumes of labeled data for training AI systems, thereby improving quality systems and providing workers with more natural interfaces to consult dashboards and interact with machinery. In this context, Vicomtech has led the overall development of the project, ensuring the robustness of the models under real operating conditions.

Among the most notable results is the creation of complex multimodal datasets. For example, systems have been developed for anomaly detection in casting jets in foundry processes and in machining centers, combining more than 1.5 million sensor data points processed at the edge with thousands of high-resolution images of cutting tools.

The combination of sensor and image data enhances anomaly detection in industrial processes.

To address the lack of real failure examples, the team employed latent diffusion models and generative algorithms to synthetically create visual defects. This enabled the successful generation of artificial images of paint defects (such as craters or orange peel), faults in printed circuit boards (PCBs) according to technical standards, and even hyperspectral images of electronic waste.

Results show that synthetic data generation enables model training in scenarios with scarce real failure data.

At the same time, IKUN has achieved significant progress in human–factory interaction. Virtual assistants based on Retrieval-Augmented Generation (RAG) architectures have been developed, integrating technical manuals and enabling real-time telemetry analysis. As a demonstration of its capabilities, the system can generate code on demand to calculate performance indicators such as OEE and understand machine-tool-specific vocabulary, reducing speech recognition error rates from 33.33% to 9.56% in noisy industrial environments.

All these achievements have been validated in four real demonstrators focused on anomaly detection in foundry and machining, classification of construction waste using hyperspectral cameras to improve recycling, synthetic image generation, and multimodal conversational assistance for resolving technical incidents.

Led by Vicomtech, the IKUN team has included the specialized expertise of TECNALIA, IKERLAN, TEKNIKER, and AZTERLAN, along with the participation of the University of the Basque Country (EHU), the corporate R&D unit IKOR Technology Center, and the intermediary agent IMH Campus. All research has been supported and funded by the ELKARTEK 2024 program, promoted by the Basque Government.