opennlp.tools.util.featuregen

Class TokenPatternFeatureGenerator

java.lang.Object

opennlp.tools.util.featuregen.FeatureGeneratorAdapter

opennlp.tools.util.featuregen.TokenPatternFeatureGenerator

All Implemented Interfaces:

AdaptiveFeatureGenerator

public class TokenPatternFeatureGenerator
extends FeatureGeneratorAdapter

Partitions tokens into sub-tokens based on character classes and generates class features for each of the sub-tokens and combinations of those sub-tokens.

Constructor Summary

Constructors

Constructor and Description
TokenPatternFeatureGenerator() Initializes a new instance.
TokenPatternFeatureGenerator(Tokenizer supportTokenizer) Initializes a new instance.

Method Summary

Modifier and Type	Method and Description
void	createFeatures(java.util.List<java.lang.String> feats, java.lang.String[] toks, int index, java.lang.String[] preds) Adds the appropriate features for the token at the specified index with the specified array of previous outcomes to the specified list of features.

Methods inherited from class opennlp.tools.util.featuregen.FeatureGeneratorAdapter

clearAdaptiveData, updateAdaptiveData

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenPatternFeatureGenerator

public TokenPatternFeatureGenerator()

Initializes a new instance. For tokinization the SimpleTokenizer is used.

TokenPatternFeatureGenerator

public TokenPatternFeatureGenerator(Tokenizer supportTokenizer)

Initializes a new instance.

Parameters:

supportTokenizer -

Method Detail

createFeatures

public void createFeatures(java.util.List<java.lang.String> feats,
                           java.lang.String[] toks,
                           int index,
                           java.lang.String[] preds)

Description copied from interface: AdaptiveFeatureGenerator

Adds the appropriate features for the token at the specified index with the specified array of previous outcomes to the specified list of features.

Parameters:

feats - The list of features to be added to.

toks - The tokens of the sentence or other text unit being processed.

index - The index of the token which is currently being processed.

preds - The outcomes for the tokens prior to the specified index.