feature: beefed up stopword list and factored out stopword list creation as a function
feature: beefed up stopword list and factored out stopword list creation as a function

Our stopwords come in several flavors: 1) standard English stopwords 2) common words that are not in the stopword list but that we suspect hold no semantic content for this corpus 3) HTML artifacts 4) email-specific noise; that is, unsemantic tokens that are in the corpus because it is email 5) corpus-specific noise.
We don't have all of these factored out exactly; we have 4 groups instead of 5.
The now-current code separates the stopword-creation code as a function and includes the items above, more or less.

...

mikedelong/aarhus