Artificial intelligence system dealing with (human) natural language rely on language models, predictions of which words occur together. To better understand how such models work -- and where they fail -- when applied to Swedish texts we need Swedish test data. A collection of test data addressing various aspects of understanding and generating text allows us to evaluate and compare models.
During the autumn of 2020 we have started working on developing evaluation data for Swedish language models at Språkbanken Text. This effort is part of the project SwedishGLUE (in Swedish SuperLim) which aims to create a standardized national benchmark for natural language understanding models. The project is a collaboration with RISE (coordinating part), the National Library of Sweden, and AI Sweden, funded by Vinnova until May 2021.
The resulting resource is modeled after the English GLUE and SuperGLUE collections with evaluation data for Natural Language Understanding models. For English, available datasets were collected, and in some cases adapted, and made available for benchmarking systems, with the explicit goal of rewarding general systems that can handle different linguistic tasks across different domains.
The first few months of the SwedishGLUE project were spent on identifying which types of data should be prioritized, and which data are available or could easily be created, given the current funding. The resulting inventory is summarized in a new report. We plan to create or convert 12 new evaluation resources during the project, to test a range of areas, from coreference, entailment, semantic similarity or relatedness, reading comprehension, and bias. However, the intent of this resource is that more data will be added in the future, both by us at Språkbanken Text and by others. It is already clear that more types of tests are needed, but also larger data sets which can be used as training or tuning data.
The collection of resources will be made available through the Språkbanken Text website. More information about the project and the project members is available on the SwedishGlue project page.