Package org.apache.lucene.analysis.ja
Class Token
- java.lang.Object
-
- org.apache.lucene.analysis.ja.Token
-
public class Token extends Object
Analyzed token with morphological data from its dictionary.
-
-
Constructor Summary
Constructors Constructor Description Token(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizer.Type type, int position, Dictionary dictionary)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description String
getBaseForm()
String
getInflectionForm()
String
getInflectionType()
int
getLength()
int
getOffset()
String
getPartOfSpeech()
int
getPosition()
Get index of this token in input textint
getPositionLength()
Get the length (in tokens) of this token.String
getPronunciation()
String
getReading()
char[]
getSurfaceForm()
String
getSurfaceFormString()
JapaneseTokenizer.Type
getType()
Returns the type of this tokenboolean
isKnown()
Returns true if this token is known wordboolean
isUnknown()
Returns true if this token is unknown wordboolean
isUser()
Returns true if this token is defined in user dictionaryvoid
setPositionLength(int positionLength)
Set the position length (in tokens) of this token.String
toString()
-
-
-
Constructor Detail
-
Token
public Token(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizer.Type type, int position, Dictionary dictionary)
-
-
Method Detail
-
getSurfaceForm
public char[] getSurfaceForm()
- Returns:
- surfaceForm
-
getOffset
public int getOffset()
- Returns:
- offset into surfaceForm
-
getLength
public int getLength()
- Returns:
- length of surfaceForm
-
getSurfaceFormString
public String getSurfaceFormString()
- Returns:
- surfaceForm as a String
-
getReading
public String getReading()
- Returns:
- reading. null if token doesn't have reading.
-
getPronunciation
public String getPronunciation()
- Returns:
- pronunciation. null if token doesn't have pronunciation.
-
getPartOfSpeech
public String getPartOfSpeech()
- Returns:
- part of speech.
-
getInflectionType
public String getInflectionType()
- Returns:
- inflection type or null
-
getInflectionForm
public String getInflectionForm()
- Returns:
- inflection form or null
-
getBaseForm
public String getBaseForm()
- Returns:
- base form or null if token is not inflected
-
getType
public JapaneseTokenizer.Type getType()
Returns the type of this token- Returns:
- token type, not null
-
isKnown
public boolean isKnown()
Returns true if this token is known word- Returns:
- true if this token is in standard dictionary. false if not.
-
isUnknown
public boolean isUnknown()
Returns true if this token is unknown word- Returns:
- true if this token is unknown word. false if not.
-
isUser
public boolean isUser()
Returns true if this token is defined in user dictionary- Returns:
- true if this token is in user dictionary. false if not.
-
getPosition
public int getPosition()
Get index of this token in input text- Returns:
- position of token
-
setPositionLength
public void setPositionLength(int positionLength)
Set the position length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.
-
getPositionLength
public int getPositionLength()
Get the length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.- Returns:
- position length of token
-
-