Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: LightHuBERT: Lightweight and Configurable Speech Representation Learning
  with Once-for-All Hidden-Unit BERT   Self-supervised speech representation learning has shown promising results in
various speech processing tasks. However, the pre-trained models, e.g., HuBERT,
are storage-intensive Transformers, limiting their scope of applications under
low-resource settings. To this end, we propose LightHuBERT, a once-for-all
Transformer compression framework, to find the desired architectures
automatically by pruning structured parameters. More precisely, we create a
Transformer-based supernet that is nested with thousands of weight-sharing
subnets and design a two-stage distillation strategy to leverage the
contextualized latent representations from HuBERT. Experiments on automatic
speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT
enables over $10^9$ architectures concerning the embedding dimension, attention
dimension, head number, feed-forward network ratio, and network depth.
LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with
the HuBERT size, achieves comparable performance to the teacher model in most
tasks with a reduction of 29% parameters, and obtains a $3.5\times$ compression
ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword
spotting, and intent classification, with a slight accuracy loss. The code and
pre-trained models are available at
https://github.com/mechanicalsea/lighthubert.

23
The categories of this paper are: solv-int
12
The categories of this paper are: nucl-th
30
The categories of this paper are: q-alg
25
The categories of this paper are: patt-sol
18
The categories of this paper are: econ
16
The categories of this paper are: stat
3
The categories of this paper are: physics
4
The categories of this paper are: cond-mat
21
The categories of this paper are: adap-org
5
The categories of this paper are: gr-qc
24
The categories of this paper are: dg-ga
31
The categories of this paper are: ao-sci
7
The categories of this paper are: hep-th
10
The categories of this paper are: q-bio
32
The categories of this paper are: atom-ph
29
The categories of this paper are: funct-an