Instruct: Identify the main category of Biorxiv papers based on the titles and abstracts
Query: Speech-driven Facial Animations Improve Speech-in-Noise Comprehension of Humans Understanding speech becomes a demanding task when the environment is noisy. Comprehension of speech in noise can be substantially improved by looking at the speakers face, and this audiovisual benefit is even more pronounced in people with hearing impairment. Recent advances in AI have allowed to synthesize photorealistic talking faces from a speech recording and a still image of a persons face in an end-to-end manner. However, it has remained unknown whether such facial animations improve speech-in-noise comprehension. Here we consider facial animations produced by a recently introduced generative adversarial network (GAN), and show that humans cannot distinguish between the synthesized and the natural videos. Importantly, we then show that the end- to-end synthesized videos significantly aid humans in understanding speech in noise, although the natural facial motions yield a yet higher audiovisual benefit. We further find that an audiovisual speech recognizer benefits from the synthesized facial animations as well. Our results suggest that synthesizing facial motions from speech can be used to aid speech comprehension in difficult listening environments.
16
The categories of this paper are: microbiology
0
The categories of this paper are: genetics
26
The categories of this paper are: biochemistry
25
The categories of this paper are: bioengineering
20
The categories of this paper are: pharmacology
24
The categories of this paper are: cancer
7
The categories of this paper are: ecology
32
The categories of this paper are: zoology
10
The categories of this paper are: and
2
The categories of this paper are: biology
27
The categories of this paper are: pathology
15
The categories of this paper are: developmental
12
The categories of this paper are: behavior
5
The categories of this paper are: cell
14
The categories of this paper are: biophysics
19
The categories of this paper are: epidemiology