Package org.apache.lucene.analysis.ja
Class Token
java.lang.Object
org.apache.lucene.analysis.ja.Token
Analyzed token with morphological data from its dictionary.
-
Constructor Summary
ConstructorDescriptionToken
(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizer.Type type, int position, Dictionary dictionary) -
Method Summary
Modifier and TypeMethodDescriptionint
int
int
Get index of this token in input textint
Get the length (in tokens) of this token.char[]
getType()
Returns the type of this tokenboolean
isKnown()
Returns true if this token is known wordboolean
Returns true if this token is unknown wordboolean
isUser()
Returns true if this token is defined in user dictionaryvoid
setPositionLength
(int positionLength) Set the position length (in tokens) of this token.toString()
-
Constructor Details
-
Token
public Token(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizer.Type type, int position, Dictionary dictionary)
-
-
Method Details
-
toString
-
getSurfaceForm
public char[] getSurfaceForm()- Returns:
- surfaceForm
-
getOffset
public int getOffset()- Returns:
- offset into surfaceForm
-
getLength
public int getLength()- Returns:
- length of surfaceForm
-
getSurfaceFormString
- Returns:
- surfaceForm as a String
-
getReading
- Returns:
- reading. null if token doesn't have reading.
-
getPronunciation
- Returns:
- pronunciation. null if token doesn't have pronunciation.
-
getPartOfSpeech
- Returns:
- part of speech.
-
getInflectionType
- Returns:
- inflection type or null
-
getInflectionForm
- Returns:
- inflection form or null
-
getBaseForm
- Returns:
- base form or null if token is not inflected
-
getType
Returns the type of this token- Returns:
- token type, not null
-
isKnown
public boolean isKnown()Returns true if this token is known word- Returns:
- true if this token is in standard dictionary. false if not.
-
isUnknown
public boolean isUnknown()Returns true if this token is unknown word- Returns:
- true if this token is unknown word. false if not.
-
isUser
public boolean isUser()Returns true if this token is defined in user dictionary- Returns:
- true if this token is in user dictionary. false if not.
-
getPosition
public int getPosition()Get index of this token in input text- Returns:
- position of token
-
setPositionLength
public void setPositionLength(int positionLength) Set the position length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1. -
getPositionLength
public int getPositionLength()Get the length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.- Returns:
- position length of token
-