public class TextProfileSignature extends MD5Signature
This implementation is copied from Apache Nutch.
An implementation of a page signature. It calculates an MD5 hash of a plain text "profile" of a page.
The algorithm to calculate a page "profile" takes the plain text version of a page and performs the following steps:
- remove all characters except letters and digits, and bring all characters to lower case,
- split the text into tokens (all consecutive non-whitespace characters),
- discard tokens equal or shorter than MIN_TOKEN_LEN (default 2 characters),
- sort the list of tokens by decreasing frequency,
- round down the counts of tokens to the nearest multiple of QUANT
QUANT = QUANT_RATE * maxFreq, where
QUANT_RATEis 0.01f by default, and
maxFreqis the maximum token frequency). If
maxFreqis higher than 1, then QUANT is always higher than 2 (which means that tokens with frequency 1 are always discarded).
- tokens, which frequency after quantization falls below QUANT, are discarded.
- create a list of tokens and their quantized frequency, separated by spaces, in the order of decreasing frequency.
Constructors Constructor Description
All Methods Instance Methods Concrete Methods Modifier and Type Method Description