Orthography Standardizer

The Orthography Standardizer annotator takes tokens from the word layer and 'cleans up' their labels, creating tags on a new layer with standardized labels, which should be optimal for looking up lexicons, frequency computation, etc.

Configuration parameters are encoded as a JSON string, e.g.

{
    "tokenLayerId":"word",
    "lowerCase":"true",
    "replacements":{
      "\\s" : "",
      "’" : "'",
      "[“”]" : "\"",
      "—" : "-",
      "[\\p{Punct}&&[^-~:']]" : "",
      "^[-']+" : "",
      "[-']+$" : ""
    },
    "orthographyLayerId":"orthography"
}

The parameters are:

tokenLayerId

Input layer from which raw word tokens are selected.

lowerCase

"true" or "on" for conversion to lowercase, absent otherwise.

orthographyLayerId

Output layer on which new annotations are added.

replacements

An object in which the keys are regular expressions to match and the values are what to replace the corresponding matches with.