refactor & optimize tokenization
refactor & optimize tokenization

instead of probing regexes on the whole doc as string,
do state machine style tokenization while looping
through the input stream.

This yields a 10x acceleration on tokenization.

At the moment, input format autodetect has been
dropped, alternative format(s) has to be specified
explicitely.

csabahenk/HTMLi