Linguistic Data Optimization Tool

Translation processes are essential for overcoming the linguistic barriers that significantly hinder socioeconomic activities, particularly in multilingual communities such as the European Union or the Basque Autonomous Community. In response to the exponential growth of digital content, translation activities increasingly rely on specialized technologies such as Computer-Assisted Translation (CAT) tools, built on Translation Memories (TM), and Machine Translation (MT).

In the field of MT, recent advances in Artificial Intelligence (AI), especially neural networks and Deep Learning, have significantly accelerated progress. Neural Machine Translation (NMT) has emerged as the new scientific and commercial paradigm and is becoming increasingly integrated into multilingual content production, particularly in professional translation workflows involving post-editing.

To deliver high-quality automatic translations, NMT systems require large parallel linguistic datasets —sets of aligned sentences in two languages— to model translation knowledge across language pairs. These resources must be of high quality, as noise in the data, such as misalignments, corrupted characters, or incorrect encoding, directly impacts system performance. Similarly, errors in translation memories reduce the productivity of human translators. In practice, substantial noise in linguistic corpora and translation memories is widespread and significantly undermines the efficiency and quality of translation processes.

The LIDO project aims to research and develop a multilingual linguistic data optimization system using Artificial Intelligence technologies. The optimization will be approached through three main axes, leveraging dedicated AI models, including neural language models, multilingual vector-based semantic representation models, and highly portable statistical models.