Skip to main content
Språkbanken Text is a part of Språkbanken.

eng-tokenization-stanza

Analysis citation Information

Språkbanken Text (2022). eng-tokenization-stanza (updated: 2022-08-10). [Analysis]. Språkbanken Text.
BibTeX
Tokenization with Stanza's standard model for English

Example

This analysis is used with Sparv. Check out Sparv's quick start guide to get started!

To use this analysis, add the following line under export.annotations in the Sparv corpus configuration file:

- stanza.token  # Token segments

In order to use this annotation you need to add the following settings to your Sparv corpus configuration file:

metadata:
  language: eng

classes:
  token: stanza.token

For more info on how to use Sparv, check out the Sparv documentation.

Example output:

<token>This</token>
<token>is</token>
<token>a</token>
<token>corpus</token>
<token>.</token>

Other references

  • Stanza: Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton and Christopher D. Manning. 2020

  • Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Association for Computational Linguistics (ACL) System Demonstrations. 2020

Type

  • Analysis

Task

  • tokenization

Unit

  • token

Tool

Stanza

Model

Stanza standard model for English (https://stanfordnlp.github.io/stanza/models.html)

Keyword

  • stanza

Created

2022-08-10

Updated

2022-08-10

Contact

Språkbanken Text
sb-info@svenska.gu.se