|
|
--- |
|
|
tags: |
|
|
- text-to-speech |
|
|
license: cc-by-nc-sa-4.0 |
|
|
language: |
|
|
- zh |
|
|
- en |
|
|
- de |
|
|
- ja |
|
|
- fr |
|
|
- es |
|
|
- ko |
|
|
- ar |
|
|
- nl |
|
|
- ru |
|
|
- it |
|
|
- pl |
|
|
- pt |
|
|
pipeline_tag: text-to-speech |
|
|
inference: false |
|
|
extra_gated_prompt: >- |
|
|
You agree to not use the model to generate contents that violate DMCA or local |
|
|
laws. |
|
|
extra_gated_fields: |
|
|
Country: country |
|
|
Specific date: date_picker |
|
|
I agree to use this model for non-commercial use ONLY: checkbox |
|
|
--- |
|
|
|
|
|
|
|
|
# OpenAudio S1 |
|
|
|
|
|
**OpenAudio S1** is a leading text-to-speech (TTS) model trained on more than 2 million hours of audio data in multiple languages. |
|
|
|
|
|
Supported languages: |
|
|
- English (en) |
|
|
- Chinese (zh) |
|
|
- Japanese (ja) |
|
|
- German (de) |
|
|
- French (fr) |
|
|
- Spanish (es) |
|
|
- Korean (ko) |
|
|
- Arabic (ar) |
|
|
- Russian (ru) |
|
|
- Dutch (nl) |
|
|
- Italian (it) |
|
|
- Polish (pl) |
|
|
- Portuguese (pt) |
|
|
|
|
|
Please refer to [Fish Speech Github](https://github.com/fishaudio/fish-speech) for more info. |
|
|
Demo available at [Fish Audio Playground](https://fish.audio). |
|
|
Visit the [OpenAudio website](https://openaudio.com) for blog & tech report. |
|
|
|
|
|
## Emotion and Tone Support |
|
|
|
|
|
OpenAudio S1 supports a variety of emotional, tone, and special markers to enhance speech synthesis: |
|
|
|
|
|
**1. Emotional markers:** |
|
|
(angry) (sad) (disdainful) (excited) (surprised) (satisfied) (unhappy) (anxious) (hysterical) (delighted) (scared) (worried) (indifferent) (upset) (impatient) (nervous) (guilty) (scornful) (frustrated) (depressed) (panicked) (furious) (empathetic) (embarrassed) (reluctant) (disgusted) (keen) (moved) (proud) (relaxed) (grateful) (confident) (interested) (curious) (confused) (joyful) (disapproving) (negative) (denying) (astonished) (serious) (sarcastic) (conciliative) (comforting) (sincere) (sneering) (hesitating) (yielding) (painful) (awkward) (amused) |
|
|
|
|
|
**2. Tone markers:** |
|
|
(in a hurry tone) (shouting) (screaming) (whispering) (soft tone) |
|
|
|
|
|
**3. Special markers:** |
|
|
(laughing) (chuckling) (sobbing) (crying loudly) (sighing) (panting) (groaning) (crowd laughing) (background laughter) (audience laughing) |
|
|
|
|
|
**Special markers with corresponding onomatopoeia:** |
|
|
- Laughing: Ha,ha,ha |
|
|
- Chuckling: Hmm,hmm |
|
|
|
|
|
## Model Variants and Performance |
|
|
|
|
|
OpenAudio S1 includes the following models: |
|
|
- **S1 (4B, proprietary):** The full-sized model. |
|
|
- **S1-mini (0.5B):** A distilled version of S1. |
|
|
|
|
|
Both S1 and S1-mini incorporate online Reinforcement Learning from Human Feedback (RLHF). |
|
|
|
|
|
**Seed TTS Eval Metrics (English, auto eval, based on OpenAI gpt-4o-transcribe, speaker distance using Revai/pyannote-wespeaker-voxceleb-resnet34-LM):** |
|
|
|
|
|
- **S1:** |
|
|
- WER (Word Error Rate): **0.008** |
|
|
- CER (Character Error Rate): **0.004** |
|
|
- Distance: **0.332** |
|
|
- **S1-mini:** |
|
|
- WER (Word Error Rate): **0.011** |
|
|
- CER (Character Error Rate): **0.005** |
|
|
- Distance: **0.380** |
|
|
|
|
|
## License |
|
|
|
|
|
This model is permissively licensed under the CC-BY-NC-SA-4.0 license. |
|
|
|
