Modified analyze_tokens to tokenize the unicode and utf-8 versions of the file.
Modified analyze_tokens to tokenize the unicode and utf-8 versions of the file.

Python's tokenize library will accept either unicode or strings, and
will give you column offsets for the tokens in characters for unicode
and in bytes for strings.

We now pass both versions of the file to tokenize, and build up the
node_start_table using the utf-8 byte offset that the ast module's
nodes would provide as the keys, and the tuple of unicode character
offsets of both the start and end of the token for the values.  And
since we now provide the full start and end, don't bother to calculate
it based on the node name length in get_node_start_end.

pelmers/dxr,pombredanne/dxr,jay-z007/dxr,jbradberry/dxr,kleintom/dxr,pombredanne/dxr,pelmers/dxr,kleintom/dxr,jbradberry/dxr,jbradberry/dxr,kleintom/dxr,pelmers/dxr,jbradberry/dxr,pombredanne/dxr,jay-z007/dxr,pelmers/dxr,jbradberry/dxr,pelmers/dxr,pombredanne/dxr,pelmers/dxr,jbradberry/dxr,pombredanne/dxr,pelmers/dxr,pombredanne/dxr,jay-z007/dxr,jay-z007/dxr,jbradberry/dxr,jay-z007/dxr,kleintom/dxr,kleintom/dxr,kleintom/dxr,jay-z007/dxr,pombredanne/dxr,kleintom/dxr,jay-z007/dxr