Following Edward's suggestion, definition of _read_block has been modified to avoid iterating over stream. The reader now retrieves 94 articles, which conforms to results of grepping for <DOCNO>.
Following Edward's suggestion, definition of _read_block has been
modified to avoid iterating over stream. The reader now retrieves 94
articles, which conforms to results of grepping for <DOCNO>.

Also inserted a hack to filter out the empty document which seems to
be emitted at the end of each file in the ieer directory -- haven't
yet figured out the cause.

According to grep, two of the documents are missing headlines.


svn/trunk@5309

nltk/nltk,nltk/nltk,nltk/nltk