- Added TokenReaderI, a general interface for reading strings -> tokens - Added some implementations of TokenReaderI:   - TokenizerBasedTokenReader   - WhitespaceSeparatedTokenReader   - NewlineSeparatedTokenReader - Added SubtokenContextPointer.subtokens() - Changed SubtokenContextPointer repr (don't print the container) - Added TreeContextPointer
- Added TokenReaderI, a general interface for reading strings -> tokens
- Added some implementations of TokenReaderI:
  - TokenizerBasedTokenReader
  - WhitespaceSeparatedTokenReader
  - NewlineSeparatedTokenReader
- Added SubtokenContextPointer.subtokens()
- Changed SubtokenContextPointer repr (don't print the container)
- Added TreeContextPointer


svn/trunk@1979

nltk/nltk,nltk/nltk,nltk/nltk