@@ -0,0 +1,792 @@
+package org.antlr.codebuff.validation;%0A%0A/** Walk corpus and group feature vectors and then consider the%0A *  diversity index / entropy of the predicted categories. This%0A *  is a measure of how consistent a corpus is.  If each specific%0A *  context predicts many different categories and mostly at%0A *  same likelihood, entropy is high meaning one or both of:%0A *%0A *  a) model does not distinguish between contexts well enough%0A *  b) corpus is highly inconsistent in formatting%0A *%0A *  To tease out (a) and (b), we can train on a single file%0A *  and measure entropy. That should tell us how much the model captures%0A *  assuming a single file is internally consistent (not always true;%0A *  in principle, we could do divide up files into regions such as methods.)%0A */%0Apublic class MeasureEntropy %7B%0A%7D%0A

add start of entropy file.