LUCENE-5042: Fix the n-gram tokenizers and filters.
LUCENE-5042: Fix the n-gram tokenizers and filters.

This commit fixes n-gram tokenizers and filters so that they handle
supplementary characters correctly and adds the ability to pre-tokenize the
stream in tokenizers.


git-svn-id: 308d55f399f3bd9aa0560a10e81a003040006c48@1492185 13f79535-47bb-0310-9956-ffa450edef68

cscorley/solr-only-mirror,cscorley/solr-only-mirror,cscorley/solr-only-mirror