- Fixed bug in WSTokenizer.xtokenize (it was ignoring source/unit) - Added a LineTokenizer, which tokenizes a file into sentences, based   on newline characters. - Fixed minor docstring bug in WSTokenizer - Changed Token to print verbose string rep from __repr__.  I've been   going back and forth on how we should use different tostring() type   methods.  This will all get standardized/regularized at some point.
- Fixed bug in WSTokenizer.xtokenize (it was ignoring source/unit)
- Added a LineTokenizer, which tokenizes a file into sentences, based
  on newline characters.
- Fixed minor docstring bug in WSTokenizer
- Changed Token to print verbose string rep from __repr__.  I've been
  going back and forth on how we should use different tostring() type
  methods.  This will all get standardized/regularized at some point.


svn/trunk@303

nltk/nltk,nltk/nltk,nltk/nltk