Redo per-term AUC analysis to show bias much more clearly.
Redo per-term AUC analysis to show bias much more clearly.

Now, we include an equally sized random sample from the dataset as well. This is
because we don't care about the model's ability to distinguish just within the
term-mentioning subset, but rather its ability to distinguish in the wider
context of the rest of the dataset. In effect, we're oversampling comments that
mention various terms in order to compare performance for those terms.

conversationai/unintended-ml-bias-analysis,conversationai/unintended-ml-bias-analysis