swe-tokenization-sparv-betterword

The tokenizer is initially based on NLTK's PunktWordTokenizer (which is no longer exposed by NLTK). Sparv's version is custom-made for Swedish using a wordlist and a configuration file with regular expressions, a list of common abbreviations, a list of words containing special characters etc. It is, however, possible to configure the tokenizer for other languages.

This analysis is used with Sparv. Check out Sparv's quick start guide to get started!

To use this analysis, add the following line under export.annotations in the Sparv corpus configuration file:

- segment.token  # Token segments

For more info on how to use Sparv, check out the Sparv documentation.

Example output:

<token>Det</token>
<token>här</token>
<token>är</token>
<token>en</token>
<token>korpus</token>
<token>.</token>