Abstract
This paper aims to help fill a current research gap by assessing the quality of legal translation produced by LLMs. It focuses on the handling of legal terminology and the disambiguation of legal homonyms with the help of reasoning models (RMs). The paper starts from a real-world use case: South Tyrol is a bilingual province in Italy where Italian and German are co-official languages and legal translation from Italian into a non-dominant legal variety of German constitutes a daily activity for many organisations.
Results indicate that RMs show a strong capacity to effectively distinguish between different legal contexts or legal subdomains. Enabling reasoning in LLMs increases the rate of accurate homonym disambiguation. Longer reasoning also correlates with correct homonym disambiguation. However, complying with system-bound legal terminology, especially in a non-dominant variety, remains a challenge. RMs tend to translate as if there were only one variety of legal German. Homonym disambiguation may rely on legally debatable criteria (e.g. frequency, formality). While reasoning successfully helps disambiguate homonyms, the rest of the text is not necessarily adapted in consequence of the decision. Further research is needed to assess whether systematic pivoting via English (or reasoning in English) influences translation quality. Results stress the importance of feeding LLMs not only with domain-specific terminology but also with data that enable them to distinguish between legal varieties (e.g. from terminology databases).