fixing bugs in generation of tokens, other changes to functions
fixing bugs in generation of tokens, other changes to functions

googleinterns/contextual-query-rewrites,googleinterns/contextual-query-rewrites,googleinterns/contextual-query-rewrites