Språkbanken Text is a department within Språkbanken.

swe-tokenization-sparv-betterword

Citation Information

Språkbanken Text (2021). swe-tokenization-sparv-betterword (updated: 2021-05-07). [Analysis]. Språkbanken Text.
Tokenizes text, custom-made for Swedish

The tokenizer is initially based on NLTK's PunktWordTokenizer (which is no longer exposed by NLTK). Sparv's version is custom-made for Swedish using a wordlist and a configuration file with regular expressions, a list of common abbreviations, a list of words containing special characters etc. It is, however, possible to configure the tokenizer for other languages.

Example

This analysis is used with Sparv. Check out Sparv's quick start guide to get started!

To use this analysis, add the following line under export.annotations in the Sparv corpus configuration file:

- segment.token  # Token segments

For more info on how to use Sparv, check out the Sparv documentation.

Example output:

<token>Det</token>
<token>här</token>
<token>är</token>
<token>en</token>
<token>korpus</token>
<token>.</token>

Collection

standard-analyses-swemink-analyses

Type

  • Analysis

Task

tokenization

Unit

token

Tool

NLTK

Model

Trained on

Created

2010-12-15

Updated

2021-05-07

Contact

Språkbanken Text
sb-info@svenska.gu.se
Page manager: Språkbanken Text
|
Last update: 2024-11-22