Converts the query string to a Collection of Lucene tokens using a regular expression.
Boolean operators AND, OR, NOT are skipped.
Each term is checked to determine if it is optional, required or prohibited. Required
terms output a
Prohibited terms output a
set. If the query uses the plus (+) and minus (-) to denote required and prohibited, this
determination will be accurate. In the case boolean AND/OR/NOTs are used, this
converter makes an uninformed guess as to whether the term would likely behave as if it
is Required or Prohibited and sets the flags accordingly. These flags are used downstream
to generate collations for
, in cases where an original
term is split up into multiple Tokens.