added Tokenize function, fixed Scan algorithm, updated id pattern and added eof... -sai
added Tokenize function, fixed Scan algorithm, updated id pattern and added eof... -sai

scottidler/sota,scottidler/sota,scottidler/sota