Tokenizer speedup.
Tokenizer speedup.

Moved the tokenizer away from regex.  Reduced Personal Stats runtime
(scanning sent messages) from about 1.6s to about 0.8s.  The tokenizer
is used in other areas as well, so this should speed up Interpersonal
Stats and Language ID.

ktrnka/droidling,ktrnka/droidling,ktrnka/droidling