Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Mitigating Gender Bias in Distilled Language Models via Counterfactual
  Role Reversal   Language models excel at generating coherent text, and model compression
techniques such as knowledge distillation have enabled their use in
resource-constrained settings. However, these models can be biased in multiple
ways, including the unfounded association of male and female genders with
gender-neutral professions. Therefore, knowledge distillation without any
fairness constraints may preserve or exaggerate the teacher model's biases onto
the distilled model. To this end, we present a novel approach to mitigate
gender disparity in text generation by learning a fair model during knowledge
distillation. We propose two modifications to the base knowledge distillation
based on counterfactual role reversal$\unicode{x2014}$modifying teacher
probabilities and augmenting the training set. We evaluate gender polarity
across professions in open-ended text generated from the resulting distilled
and finetuned GPT$\unicode{x2012}$2 models and demonstrate a substantial
reduction in gender disparity with only a minor compromise in utility. Finally,
we observe that language models that reduce gender polarity in language
generation do not improve embedding fairness or downstream classification
fairness.

25
The categories of this paper are: patt-sol
7
The categories of this paper are: hep-th
12
The categories of this paper are: nucl-th
16
The categories of this paper are: stat
34
The categories of this paper are: chem-ph
15
The categories of this paper are: nucl-ex
27
The categories of this paper are: comp-gas
4
The categories of this paper are: cond-mat
35
The categories of this paper are: mtrl-th
5
The categories of this paper are: gr-qc
8
The categories of this paper are: hep-ex
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
6
The categories of this paper are: astro-ph
9
The categories of this paper are: nlin
10
The categories of this paper are: q-bio