Fixed crashing when using Trie based KenLM models with State Minimization
Fixed crashing when using Trie based KenLM models with State Minimization

There are two separate issues that are causing very rare crashes when using KenLM with Joshua.  The only configuration under which we've seen crashing, is when using more than one Trie-based state-minimizing model.  One of these issues is addressed in this patch, another will have to be contributed to the KenLM repo.

This first patch deals with hash collisions in the mapping between a given KenLM state, and the stored memory address of the state in KenLM’s memory pool.  Previously these collisions were ignored.  The problem with ignoring collisions here is that the two models can have different Vocabulary sizes.  This means if you collide and take the state of a model with a larger vocabulary you could potentially have a unigram word id that is out of range.  When searched for this will cause a crash.

The second issue (to be patched in KenLM) is that the equality operator in state.hh isn't comparing enough values to properly differentiate between states during a collision.

thammegowda/incubator-joshua,thammegowda/incubator-joshua,fhieber/incubator-joshua,fhieber/incubator-joshua,fhieber/incubator-joshua,fhieber/incubator-joshua,thammegowda/incubator-joshua,thammegowda/incubator-joshua,fhieber/incubator-joshua,fhieber/incubator-joshua,fhieber/incubator-joshua,thammegowda/incubator-joshua,fhieber/incubator-joshua,thammegowda/incubator-joshua,thammegowda/incubator-joshua,thammegowda/incubator-joshua,thammegowda/incubator-joshua,fhieber/incubator-joshua,thammegowda/incubator-joshua