First cut of the tokeniser library. Modified from drupal search.module code (see copyrights). The library converts any text/html into an array of tokens with their score (weight). Supports stop_words, cjk basic tokeniser (for indexers) and different modes of handling numbers.
First cut of the tokeniser library. Modified from drupal search.module
code (see copyrights). The library converts any text/html into an array
of tokens with their score (weight). Supports stop_words, cjk basic
tokeniser (for indexers) and different modes of handling numbers.

Merged from MOODLE_18_STABLE

dilawar/moodle