Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: An Efficient DP-SGD Mechanism for Large Scale NLP Models   Recent advances in deep learning have drastically improved performance on
many Natural Language Understanding (NLU) tasks. However, the data used to
train NLU models may contain private information such as addresses or phone
numbers, particularly when drawn from human subjects. It is desirable that
underlying models do not expose private information contained in the training
data. Differentially Private Stochastic Gradient Descent (DP-SGD) has been
proposed as a mechanism to build privacy-preserving models. However, DP-SGD can
be prohibitively slow to train. In this work, we propose a more efficient
DP-SGD for training using a GPU infrastructure and apply it to fine-tuning
models based on LSTM and transformer architectures. We report faster training
times, alongside accuracy, theoretical privacy guarantees and success of
Membership inference attacks for our models and observe that fine-tuning with
proposed variant of DP-SGD can yield competitive models without significant
degradation in training time and improvement in privacy protection. We also
make observations such as looser theoretical $\epsilon, \delta$ can translate
into significant practical privacy gains.

35
The categories of this paper are: mtrl-th
2
The categories of this paper are: math
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
26
The categories of this paper are: bayes-an
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
12
The categories of this paper are: nucl-th
8
The categories of this paper are: hep-ex
37
The categories of this paper are: supr-con
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
6
The categories of this paper are: astro-ph
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc