Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning Speech Representations from Raw Audio by Joint Audiovisual
  Self-Supervision   The intuitive interaction between the audio and visual modalities is valuable
for cross-modal self-supervised learning. This concept has been demonstrated
for generic audiovisual tasks like video action recognition and acoustic scene
classification. However, self-supervision remains under-explored for
audiovisual speech. We propose a method to learn self-supervised speech
representations from the raw audio waveform. We train a raw audio encoder by
combining audio-only self-supervision (by predicting informative audio
attributes) with visual self-supervision (by generating talking faces from
audio). The visual pretext task drives the audio representations to capture
information related to lip movements. This enriches the audio encoder with
visual information and the encoder can be used for evaluation without the
visual modality. Our method attains competitive performance with respect to
existing self-supervised audio features on established isolated word
classification benchmarks, and significantly outperforms other methods at
learning from fewer labels. Notably, our method also outperforms fully
supervised training, thus providing a strong initialization for speech related
tasks. Our results demonstrate the potential of multimodal self-supervision in
audiovisual speech for learning good audio representations.

17
The categories of this paper are: q-fin
16
The categories of this paper are: stat
26
The categories of this paper are: bayes-an
25
The categories of this paper are: patt-sol
18
The categories of this paper are: econ
2
The categories of this paper are: math
37
The categories of this paper are: supr-con
15
The categories of this paper are: nucl-ex
27
The categories of this paper are: comp-gas
21
The categories of this paper are: adap-org
7
The categories of this paper are: hep-th
30
The categories of this paper are: q-alg
12
The categories of this paper are: nucl-th
34
The categories of this paper are: chem-ph
24
The categories of this paper are: dg-ga
0
The categories of this paper are: hep-ph