Package opennlp.tools.tokenize
Interface Detokenizer
-
- All Known Implementing Classes:
DictionaryDetokenizer
public interface Detokenizer
A Detokenizer merges tokens back to their untokenized representation.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static class
Detokenizer.DetokenizationOperation
This enum contains an operation for every token to merge the tokens together to their detokenized form.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description Detokenizer.DetokenizationOperation[]
detokenize(java.lang.String[] tokens)
Detokenize the input tokens.java.lang.String
detokenize(java.lang.String[] tokens, java.lang.String splitMarker)
Detokenize the input tokens into a String.
-
-
-
Method Detail
-
detokenize
Detokenizer.DetokenizationOperation[] detokenize(java.lang.String[] tokens)
Detokenize the input tokens.- Parameters:
tokens
- the tokens to detokenize.- Returns:
- the merge operations to detokenize the input tokens.
-
detokenize
java.lang.String detokenize(java.lang.String[] tokens, java.lang.String splitMarker)
Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.- Parameters:
tokens
- the token which should be concatenatedsplitMarker
- the split marker or null- Returns:
- the concatenated tokens
-
-