Tokenization with Stanza's standard model for English
Citation
Språkbanken Text (2022). eng-tokenization-stanza (updated: 2022-08-10). [Analysis]. Språkbanken Text.
Example
This analysis is used with Sparv. Check out Sparv's quick start guide to get started!
To use this analysis, add the following line under export.annotations in the Sparv corpus configuration file:
- stanza.token # Token segments
For more info on how to use Sparv, check out the Sparv documentation.
Example output:
<token>This</token>
<token>is</token>
<token>a</token>
<token>corpus</token>
<token>.</token>
Other references
Stanza: Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton and Christopher D. Manning. 2020
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Association for Computational Linguistics (ACL) System Demonstrations. 2020