* The Annotation

- Terminal level:

  id: word identifier [obligatory]
  word: word form [obligatory]
  read_as: how the annotator interpreted the word form in the
      annotation (typically for typos, non-standard orthography, etc)
      [optional]
  pos: part-of-speech [obligatory]
  msd: part-of-speech specific morpho-syntactic features [obligatory]
  msd2: general morpho-syntactic features [obligatory]
  lemma: lemma [obligatory], unlemmatized POS-categories get value "_"
  saldo: Saldo (v2.0) identifier [optional]
  sicid: sic identifier for blog tokens [optional]
  sense: Saldo sense identifier [optional]
  sense_quality: A confidence value for 'sense', where '4' means
      manually checked, '3' means a sense annotated by at least two
      annotators without conflict, '2' means annotated by one
      annotator, '1' means multiple conflicting annotations with one
      majority alternative, and '0' means multiple conflicting
      annotations without one majority alternative.
  sense_ann: sense annotations, where the first four positions are
      different annotators and the fifth contains an automatic
      heuristic, mainly for auxiliary 'ha' ('have') and existential
      'det' ('it') [optional]
  connected: whether there is non-white space material directly
      preceding and/or following the token [obligatory]


- Nonterminal level:

  nt id: nonterminal (phrase) identifier [obligatory]
  nt cat: phrase label [obligatory]
  edge idref: child identifier [obligatory]
  edge label: function label [obligatory]
  secedge idref: secondary edge child identifier [obligatory]
  secedge label: secondary edge function label [obligatory]


* Relation to the text source

The annotated corpora in the Annotations directory can be related to
the flat texts in the Sources directory in the following way.

- The texts for the 5 divisions are in their own directories in
  Sources
- The annotation files are divided into subcorpora, whose id forms the
  basename of the corresponding text file in Sources
- The the tokens in the annotation files are marked with an attribute
  "connected", whose value is "neither", "front", "rear", or "both". A
  token is considered connected at the front/rear if there is a
  visible character directly preceding/following the token in the flat
  text. This annotation is agnostic about the type of white space or
  structural division that may seperate tokens.
- A subcorpus in a file in Annotations and its corresponding flat text
  in Sources contain the same string, modulo the type of white space
  (i.e., newlines, spaces, thin spaces, non-breaking spaces,
  repetitions of the above, etc are all considered to be the same)

Note that Eukalyptus allows tokens that contain white space. These
must then be string identical between the version in Sources and in
Annotations.
