Package org.languagetool.tokenizers
Interface Tokenizer
-
- All Known Subinterfaces:
CompoundWordTokenizer
,SentenceTokenizer
- All Known Implementing Classes:
SimpleSentenceTokenizer
,SRXSentenceTokenizer
,WordTokenizer
public interface Tokenizer
Interface for classes that tokenize text into smaller units.- Author:
- Daniel Naber
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description List<String>
tokenize(String text)
-