Removing all use of map/reduce, reworking the logic for full-text search using straight queries. I think this is the best way to go for now. We need to somehow bound the number of candidates we consider after breaking the query into n-grams, but that would mean a query like
Removing all use of map/reduce, reworking the logic for full-text search using
straight queries. I think this is the best way to go for now. We need to
somehow bound the number of candidates we consider after breaking the query
into n-grams, but that would mean a query like

  {ngram: {$in: ['mon','ong','ngo','god','odb']}}

(or a similar query using $or) along with a sort-and-limit by the score field
(e.g.,get the 1000 records with the highest scores that satisfy that query.)
But MongoDB currently won't use an index on {ngram: 1, score: -1} to run that
query, instead preferring to do a full table scan for the ngram matches.

So we're going to treat the ngram query in a similar fashion to how we might
treat an AND query: get frequency counts of all of the ngrams and use those
to direct some direct queries into the index collection, using the score
field to return only the top-scoring results.

Most tests pass, still some work to do.

dblock/mongoid_fulltext,vanboom/mongoid_fulltext,tomg65/mongoid_fulltext,artsy/mongoid_fulltext