small refactorings of the tokenizer
small refactorings of the tokenizer

flurischt/jedi,WoLpH/jedi,WoLpH/jedi,mfussenegger/jedi,tjwei/jedi,mfussenegger/jedi,jonashaag/jedi,dwillmer/jedi,dwillmer/jedi,tjwei/jedi,jonashaag/jedi,flurischt/jedi