SweParaphrase

The downloadable 'sweparaphrase-dev-165.csv' file contains 8 tab-separated columns:
(1) Sentence ID from the automatically translated Swedish dataset;
(2) Genre from source (captions, news, forum);
(3) File from source (images, headlines, answers);
(4) and (5) manually corrected Swedish sentence pairs;
(6) Similarity score from source (based on the English sentence pairs done by Crowdsourcing through Mechanical Turk);
(7) and (8) English sentence pairs from source.

Data source(s)*

The original STS benchmark comprises 8628 sentence pairs, collected from SemEval 2012 (task 6), 2014 (task 10), 2015 (task 2), 2016 (task 1), 2017 (task 1) and *SEM 2013.

Data collection method(s)*

Isbister and Sahlgren, 2020 [1] translated the complete English STS-B http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark#Reference. The original English set is collected from datasets from the SemEval shared tasks.

Data selection and filtering*

This subset is taken from the automatically translated version of STS-B. First we focused only on the development version. Second, we selected only sentences which were deemed accurate translations.

Data preprocessing*

English sentence pairs were tab-seperated. Large chunks of texts appearing after the full stop of the sentence were removed. Scores with decimals longer than 4 were shortened.

Data labeling*

No additional labeling was added. In the English version each sentence pair is annotated with a score (0-5). This annotation was done by Crowdsourcing through Mechanical Turk. Scores were assigned to the source English pairs.

Native speaker of Swedish; fluent non-native speaker of Swedish.

IV. ETHICS AND CAVEATS

Ethical considerations

Things to watch out for

The similarity scores are based on the English data and are not necessarily representative for the Swedish counter parts.

V. ABOUT DOCUMENTATION

Data last updated*

2021-05-31, v1.0

Which changes have been made, compared to the previous version*

This is the first official version

Access to previous versions

This document created*

2021-05-31, Dana Dannélls

This document last updated*

2021-06-07, Dana Dannélls

Where to look for further details

[1],[2],[3],[4]

Documentation template version*

v1.0

VI. OTHER

Related projects

Language models for Swedish authorities Vinnova (grant no. 2019-02996)

References

[1] Isbister, T. and Sahlgren, M. (2020): Why not simply translate? A first swedish evaluation benchmark for semantic similarity. Proceedings of the Eighth Swedish Language Technology Conference (SLTC), University of Gothenburg. https://gubox.box.com/v/SLTC-2020-paper-15. The automatically translated dataset https://svn.spraakbanken.gu.se/sb-arkiv/pub/sweparaphrase/stsb-mt-sv.zip

[2] Yvonne Adesam, Aleksandrs Berdicevskis, Felix Morger (2020): SwedishGLUE – Towards a Swedish Test Set for Evaluating Natural Language Understanding Models. University of Gothenburg. https://gupea.ub.gu.se/bitstream/2077/67179/1/gupea_2077_67179_1.pdf

[3] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018): GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. https://arxiv.org/pdf/1804.07461.pdf

[4] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia (2017): Semeval-2017 task1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. In 11th International Workshop on Semantic Evaluations, 2017. https://www.aclweb.org/anthology/S17-2001.pdf