Package org.apache.lucene.analysis.ngram
Class NGramTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.ngram.NGramTokenizer
- All Implemented Interfaces:
Closeable,AutoCloseable
- Direct Known Subclasses:
EdgeNGramTokenizer
Tokenizes the input into n-grams of the given size(s).
On the contrary to NGramTokenFilter, this class sets offsets so that characters
between startOffset and endOffset in the original stream are the same as the term chars.
For example, "abcde" would be tokenized as (minGram=2, maxGram=3):
| Term | ab | abc | bc | bcd | cd | cde | de |
|---|---|---|---|---|---|---|---|
| Position increment | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Position length | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Offsets | [0,2[ | [0,3[ | [1,3[ | [1,4[ | [2,4[ | [2,5[ | [3,5[ |
This tokenizer changed a lot in Lucene 4.4 in order to:
- tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
- count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
- give the ability to
pre-tokenizethe stream before computing n-grams.
Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final intstatic final intFields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY -
Constructor Summary
ConstructorsConstructorDescriptionCreates NGramTokenizer with default min and max n-grams.NGramTokenizer(int minGram, int maxGram) Creates NGramTokenizer with given min and max n-grams.NGramTokenizer(AttributeFactory factory, int minGram, int maxGram) Creates NGramTokenizer with given min and max n-grams. -
Method Summary
Modifier and TypeMethodDescriptionfinal voidend()final booleanprotected booleanisTokenChar(int chr) Only collect characters which satisfy this condition.final voidreset()Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset, setReader, setReaderTestPointMethods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
Field Details
-
DEFAULT_MIN_NGRAM_SIZE
public static final int DEFAULT_MIN_NGRAM_SIZE- See Also:
-
DEFAULT_MAX_NGRAM_SIZE
public static final int DEFAULT_MAX_NGRAM_SIZE- See Also:
-
-
Constructor Details
-
NGramTokenizer
public NGramTokenizer(int minGram, int maxGram) Creates NGramTokenizer with given min and max n-grams.- Parameters:
minGram- the smallest n-gram to generatemaxGram- the largest n-gram to generate
-
NGramTokenizer
Creates NGramTokenizer with given min and max n-grams.- Parameters:
factory-AttributeFactoryto useminGram- the smallest n-gram to generatemaxGram- the largest n-gram to generate
-
NGramTokenizer
public NGramTokenizer()Creates NGramTokenizer with default min and max n-grams.
-
-
Method Details
-
incrementToken
- Specified by:
incrementTokenin classTokenStream- Throws:
IOException
-
isTokenChar
protected boolean isTokenChar(int chr) Only collect characters which satisfy this condition. -
end
- Overrides:
endin classTokenStream- Throws:
IOException
-
reset
- Overrides:
resetin classTokenizer- Throws:
IOException
-