The WordReference corpus is a very large corpus (170M+ words) of native and non-native natural written production in four languages: English, Spanish, French and Italian. It has been scraped from WordReference forums.
The corpus consists of four tab-separated files (one per language), each containing seven tab-separated columns: message id, poster's nickname, poster' native language(s), the text of the message (post) itself, the id of the topic in response to which the message has been posted (0 if this message is the topic, i.e. the first in the thread), topicstarter's nickname ("topicstarter" if the poster is the topicstarter), native language of the topicstarter. These are the "raw" data with as little processing as possible (but http links and explicit quotes of other users' posts are removed during the download).
Several useful scripts for working with the corpus (including the one which labels all participants as L1 or L2-speakers in the respective subforum) can be found in this repository.
For a more detailed description of the corpus, its creation and examples of how it can be used, see the following paper:
Aleksandrs Berdicevskis
(2020):
Foreigner-directed speech is simpler than native-directed: Evidence from social media, in
Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, NLP+CSS 2020, November 20, 2020, Online. Pp. 163-172
. Please cite the paper if you are using the corpus.
