Tokenizes text into tokens by linebreaks using the RegexpTokenizer from NLTK
The documentation for NLTK's RegexpTokenizer can be found here.
Citation
Språkbanken Text (2021). tokenization-sparv-linebreaks (updated: 2021-05-07). [Analysis]. Språkbanken Text.
Standard reference
Bird, Steven, Edward Loper and Ewan Klein (2009), Natural Language Processing with Python. O’Reilly Media Inc.
Example
This analysis is used with Sparv. Check out Sparv's quick start guide to get started!
To use this analysis, add the following line under export.annotations in the Sparv corpus configuration file:
- segment.token # Token segments
In order to use this tokenizer you need to add the following setting to your Sparv corpus configuration file:
segment:
token_segmenter: linebreaks
For more info on how to use Sparv, check out the Sparv documentation.
Example output:
<token>Det</token>
<token>här</token>
<token>är</token>
<token>en</token>
<token>korpus</token>
<token>.</token>