creation of clean_html and clean_visible now works pretty well for the raw HTML that was initially fetched from the Web for the Wikipedia Links Corpus.
creation of clean_html and clean_visible now works pretty well for the
raw HTML that was initially fetched from the Web for the Wikipedia
Links Corpus.

lxml is superior to BeautifulSoup.  Not just because it is faster.  It
also incorporates BeautifulSoup and provides recommendations on how to
use both.  BeautifulSoup can fix tag soups that are more broken than
lxml can handle, and it also has better character set detection and
conversion than lxml, so it is a good fallback.  It's parsing is regex
based, so 10x slower than lxml.

See:
   http://lxml.de/elementsoup.html

I need to put in some checks to verify that we are not storing bogus
documents as clean_html.  The error path I'm most worried about first
is that the fetch from UMass did not record Content-type headers, so
the few hundred thousand binary files (PDF, XLS, etc) are being
processed as though they were HTML.  That might appear to work
sometimes, and we should reject those for now.

Currently, kba.pipeline.hyperlinks does not generate a LabelSet from
the HTML anchor tags to Wikipedia.  That's next.  Also, next is making
this part of a proper "pipeline" so that kba.pipeline can also run a
tagger like LingPipe and build Sentence objects on each StreamItem.

After that, we can go back and process PDFs and XLS etc into HTML with
an earlier binary file conversion stage in the pipeline.

trec-kba/streamcorpus-pipeline,trec-kba/streamcorpus-pipeline,trec-kba/streamcorpus-pipeline