Tutorials

The landscape of machine translation (MT) has evolved dramatically since the advent of neural MT (NMT), which marked a breakthrough in translation quality and fluency. More recently, the rise of large language models (LLMs) has reshaped this landscape once again, introducing a new paradigm that merges translation, adaptation, and post-editing within a unified framework of multilingual text generation. These advances are expanding the possibilities for translators and language professionals, offering tools that can be tailored to domain-specific needs and local workflows. While commercial systems such as DeepL, Google Translate, ChatGPT, or Gemini dominate public attention, a vibrant ecosystem of free and open-source NMT and LLM resources has emerged. Projects like OPUS-MT, NLLB, and translation-oriented open LLMs such as Tower and Salamandra make it increasingly feasible to build and adapt high-quality MT pipelines for specific languages, domains, or institutional contexts. Yet, integrating these tools—each with its own dependencies and APIs—into professional computer-assisted translation (CAT) environments remains a technical challenge.

The MTUOC project addresses this gap by providing a comprehensive open-source framework that simplifies deployment and integration. This hands-on tutorial will guide participants in building and customizing their own tailored MT ecosystems using fully open and free technologies. Attendees will learn how to (1) set up the MTUOC-server, (2) deploy leading open models such as OpusMT and NLLB, (3) integrate translation-specialized LLMs (Tower, Salamandra) through MTUOC components, and (4) connect all these tools seamlessly within OmegaT, a widely used open-source CAT platform. By the end of the session, participants will have a fully operational and reproducible open-source translation workflow capable of combining neural MT and LLM-based translation within a professional environment. Since both MTUOC and OmegaT are distributed under the GNU-GPL license, the entire solution remains free, extensible, and adaptable to the needs of individual translators, research groups, and institutions.

Tutorial materials: https://github.com/zouharvi/humeval-tutorial

Human evaluation is the gold standard for multilingual NLP but is frequently omitted due to operational complexity. This tutorial demonstrates how to design and execute rigorous human evaluation campaigns focusing on multilingual tasks (e.g. translation, multilingual, or multimodal evaluation), covering the full lifecycle: data selection, protocol selection, setting up the evaluation campaign, annotator management, and analysis of results. The practical focus will be on setting up the evaluation campaign with examples, while the theoretical part will be devoted to modern statistical techniques, such as turning pairwise preferences into absolute scores, or modelling benchmarking competitions. At the end, participants will have detailed knowledge of how to design, implement, and run high-quality human evaluation in their scientific and industry applications.

Tutorial materials: https://github.com/YuriBalashov/eamt2026-eval-tutorial

Before the tutorial: please complete the quick start steps outlined here: https://github.com/YuriBalashov/eamt2026-eval-tutorial#quick-start-before-the-tutorial

A half-day hands-on tutorial which introduces automatic translation quality evaluation methods and tools to an audience that has not traditionally used them: freelance translators, small language service providers (LSPs), translation project managers, and translation studies students with little or no programming experience. Evaluation techniques long reserved for MT research and large-scale industry workflows are now within reach of individual language professionals, thanks to two converging developments: user-friendly no-code web toolkits such as MATEO (Vanroy et al., 2023), and modern large language models (LLMs) that can serve as on-demand coding partners. Building on the emerging concept of Translation Analytics, the tutorial unfolds in four parts. Part 1 surveys manual and automatic evaluation, from MQM and direct assessment to BLEU, chrF, TER, COMET, BLEURT, BERTScore, and current developments (xCOMET, MetricX, LLM-based metrics). Part 2 walks participants through MATEO, where they run BLEU, chrF, TER, and COMET on multilingual evaluation sets in EN–DE, EN–RU, EN–JA, or EN–ZH. Part 3 interprets the outputs: score tables, confidence intervals, and sentence-level COMET in Excel. Part 4 introduces lightweight statistics (means, variance, p-values; Pearson, Spearman, and Kendall correlations) using Excel and LLM-assisted Python. All materials are openly available in a GitHub repository.

Vanroy, Bram, Arda Tezcan, and Lieve Macken. 2023. MATEO: MAchine Translation Evaluation Online. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 499–500, Tampere, Finland. European Association for Machine Translation.