Package opennlp.tools.tokenize
Interface Detokenizer
- 
- All Known Implementing Classes:
 DictionaryDetokenizer
public interface DetokenizerA Detokenizer merges tokens back to their untokenized representation. 
- 
- 
Nested Class Summary
Nested Classes Modifier and Type Interface Description static classDetokenizer.DetokenizationOperationThis enum contains an operation for every token to merge the tokens together to their detokenized form. 
- 
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description Detokenizer.DetokenizationOperation[]detokenize(java.lang.String[] tokens)Detokenize the input tokens.java.lang.Stringdetokenize(java.lang.String[] tokens, java.lang.String splitMarker)Detokenize the input tokens into a String. 
 - 
 
- 
- 
Method Detail
- 
detokenize
Detokenizer.DetokenizationOperation[] detokenize(java.lang.String[] tokens)
Detokenize the input tokens.- Parameters:
 tokens- the tokens to detokenize.- Returns:
 - the merge operations to detokenize the input tokens.
 
 
- 
detokenize
java.lang.String detokenize(java.lang.String[] tokens, java.lang.String splitMarker)Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.- Parameters:
 tokens- the token which should be concatenatedsplitMarker- the split marker or null- Returns:
 - the concatenated tokens
 
 
 - 
 
 -