Large-scale Swedish language models have long been missing. Recently, several such models have been developed (in particular, by KBLab, the datalab at the National Library of Sweden), and more are on the way. Language models are being trained on enormous quantities of text in order to learn to understand language and analyze texts. The models can perform various tasks, for instance, summarise texts, measure how similar texts are, or analyse the attitudes expressed in parts of the text. Language models can improve all applications of language technology to Swedish texts, making them useful in research and in the private and public sectors.
“While it’s great that we finally have Swedish language models, it is difficult to evaluate them. We have therefore now developed SuperLim, a suite of 13 resources that can be used as benchmarks for various tasks. Anybody can use the suite to test models and assess how well they understand the language,” explains Aleksandrs Berdicevskis, one of the project members.
The SuperLim project – a collaboration between Språkbanken Text, KBLab, RISE Research Institutes of Sweden and AI Sweden – partly follows the approach of the English benchmark SuperGLUE.
One important element of evaluation is to identify whether the models have any bias, and if yes, to deal with them appropriately.
“Previous studies demonstrate that language models are sensitive to the data on which they are trained and often reflect human prejudices included in the training data. For example, it may pick up racism or assume that a doctor is a man and a nurse a woman. It is important to evaluate and improve language models so that we can counter the prejudices that get encoded into the models,” says project team member Yvonne Adesam.