Last year, Språkbanken Text together with KB-Lab at the National Library of Sweden, RISE and AI Sweden created SuperLim, a collection of thirteen Swedish test sets for natural language understanding tasks, from sentiment analysis and semantic similarity estimation to coreference resolution and gender bias detection. SuperLim, inspired by the English collection(s) (Super)Glue, can be used to evaluate Swedish language models.
At the end 2021, the Vinnova agency, which funded the development of SuperLim, announced that it will also fund the subsequent project, SuperLim 2.0! The goal of the project is to complement the existing SuperLim with three important components which are currently missing:
- training data for all the test sets that lack it
- a baseline (a reference implementation with results, which the models can be compared against)
- a leaderboard (a system where the evaluation results for different models can be continuously published and compared in a convenient way).
The work is expected to be completed by the end of 2022.
Photo by Florian Schmetz on Unsplash