Smart data management server and tooling for next generation AI fuction development

The project aims to create an advanced laboratory that provides high-performance computing and smart data-management capabilities. The proposal included three key components: a parallel storage system capable of distributing user data transparently across multiple servers, a computing cluster designed for distributed processing with native Big Data technologies, and a real-time distributed data-streaming platform to support the entire data lifecycle and the development of machine learning models.
With this new infrastructure, Vicomtech can now host tools for distributed processing and Big Data, offer comprehensive storage services for research and development initiatives, and manage all stages of data ingestion, processing, storage, and exploitation. It also enables the preparation, training, and deployment of continuous machine-learning solutions, incorporating a security model for multi-tenant environments with different user roles.
The main tasks carried out during the project included designing the laboratory, identifying suppliers, acquiring the selected equipment, and installing, testing, and validating the infrastructure. As a result, Vicomtech now has a state-of-the-art environment for research on advanced Big Data technologies and model-training systems for perception and data management.