Pure tokenizer: simplify the choice of obtaining a new token
Pure tokenizer: simplify the choice of obtaining a new token

Fix #1768
Closes gh-1774

jquery/esprima,bcbcarl/esprima,ChromeDevTools/esprima,bcbcarl/esprima,ChromeDevTools/esprima,jquery/esprima,ariya/esprima,bcbcarl/esprima,ariya/esprima,jquery/esprima,ChromeDevTools/esprima,ariya/esprima