Abstract
Machine translation (MT) in the legal domain is growing, despite it being a high-stakes domain. Technological support is particularly needed in multilingual legal systems. Currently, there is a research gap on how MT – especially using Large Language Models – handles both highly domain-specific terminology and terminological variation within pluricentric languages (e.g. German, Spanish).
To help address these gaps, we published LegISTyr, a manually curated test set of 2,000 Italian sentences with legal terms. The source terms are paired with their target terms in South Tyrolean German (STG). STG is a non-dominant variety of German used by public authorities in South Tyrol (Italy), where German is a co-official language. LegISTyr supports the – automatic and manual – evaluation of how MT tools address challenges during translation from Italian into STG:
* respecting the system-boundness of legal terminology by avoiding terms pertaining to other, dominant legal systems (e.g. Germany)
* enforcing terms officially standardised by the local Terminology Commission
* correctly disambiguating legal homonyms
* handling legal abbreviations
* handling gender-inclusive language following local legal drafting requirements
Creating a test set enables automatic testing [6] that can inform decisions on whether and which MT tools to use for legal translation in specific language pairs and where to focus attention during post-editing.
We illustrate the method used to create the test set, our choices and (open) challenges. It can be replicated for other legal varieties, pluricentric languages and language combinations.