Package weka.core.tokenizers

Class CharacterNGramTokenizer

  • All Implemented Interfaces:
    Serializable, Enumeration<String>, OptionHandler, RevisionHandler
    public class CharacterNGramTokenizer
extends Tokenizer
    Splits a string into an n-gram with min and max grams.

    Valid options are: 

     -max <int>
  The max size of the Ngram (default = 3).
 
     -min <int>
  The min size of the Ngram (default = 1).
    Version:
    $Revision: 10971 $
    Author:
    Sebastian Germesin (sebastian.germesin@dfki.de), Eibe Frank (eibe@cs.waikato.ac.nz)
    See Also:
    Serialized Form

    • Constructor Detail

      • CharacterNGramTokenizer

        public CharacterNGramTokenizer()

    • Method Detail

      • globalInfo

        public String globalInfo()
        Returns a string describing the tokenizer
        Specified by:
        globalInfo in class Tokenizer
        Returns:
        a description suitable for displaying in the explorer/experimenter GUI

      • getOptions

        public String[] getOptions()
        Gets the current option settings for the OptionHandler.
        Specified by:
        getOptions in interface OptionHandler
        Overrides:
        getOptions in class Tokenizer
        Returns:
        the list of current option settings as an array of strings

      • setOptions

        public void setOptions​(String[] options)
                throws Exception
        Parses a given list of options.

        Valid options are: 

         -max <int>
  The max size of the Ngram (default = 3).
 
         -min <int>
  The min size of the Ngram (default = 1).
        Specified by:
        setOptions in interface OptionHandler
        Overrides:
        setOptions in class Tokenizer
        Parameters:
        options - the list of options as an array of strings
        Throws:
        Exception - if an option is not supported

      • getNGramMaxSize

        public int getNGramMaxSize()
        Gets the max N of the NGram.
        Returns:
        the size (N) of the NGram.

      • setNGramMaxSize

        public void setNGramMaxSize​(int value)
        Sets the max size of the Ngram.
        Parameters:
        value - the size of the NGram.

      • NGramMaxSizeTipText

        public String NGramMaxSizeTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

      • setNGramMinSize

        public void setNGramMinSize​(int value)
        Sets the min size of the Ngram.
        Parameters:
        value - the size of the NGram.

      • getNGramMinSize

        public int getNGramMinSize()
        Gets the min N of the NGram.
        Returns:
        the size (N) of the NGram.

      • NGramMinSizeTipText

        public String NGramMinSizeTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

      • tokenize

        public void tokenize​(String s)
        Sets the string to tokenize.
        Specified by:
        tokenize in class Tokenizer
        Parameters:
        s - the string to tokenize

      • getRevision

        public String getRevision()
        Returns the revision string.
        Returns:
        the revision

      • main

        public static void main​(String[] args)
        Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.
        Parameters:
        args - the commandline options and strings to tokenize