Upload folder using huggingface_hub

README.md

@@ -0,0 +1,128 @@
+---
+license: cc-by-4.0
+language:
+  - en
+  - de
+  - fr
+  - es
+  - zh
+library_name: transformers
+pipeline_tag: feature-extraction
+tags:
+  - audio
+  - speech
+  - emotion
+  - clap
+  - contrastive
+  - voice
+  - commercial-use
+base_model: laion/voiceclap-small
+---
+
+# VoiceCLAP-Commercial (small)
+
+A **fully commercially-licensable** voice-text contrastive (CLAP-style) embedding
+model for speech-emotion and talking-style retrieval. Same 110M dual-tower
+architecture as [`laion/voiceclap-small`](https://huggingface.co/laion/voiceclap-small),
+but trained **only on data that permits commercial use** — and, perhaps
+surprisingly, it matches or beats the non-commercial-data model on 4 of 5
+benchmarks.
+
+## Why this model exists
+
+The standard VoiceCLAP training mix contains **CC BY-NC** (non-commercial)
+corpora: Expresso and EARS (both Meta, CC BY-NC 4.0), and the bulk of
+[Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset) (the original
+101k-hour split is CC BY-NC 4.0). That makes models trained on the full mix
+unusable for commercial purposes.
+
+This model removes every non-commercial source and keeps only commercially
+usable data. A controlled ablation (below) shows the small model loses
+**nothing** by doing so.
+
+## Training data (all commercially licensable)
+
+| Corpus | License | Role |
+|---|---|---|
+| **Emilia-YODAS** (the CC BY 4.0 subset of Emilia, ~19% of emolia-balanced) | CC BY 4.0 | emotion / voice captions |
+| **LAION's Got Talent** | LAION-released | talking-style captions |
+| **Majestrino** | in-house | voice captions |
+
+The non-commercial Emilia clips are filtered out **at training time** by clip-id
+(original-Emilia ids look like `EN_B00087_S08178_W000004`; the retained
+YODAS clips carry YouTube-style ids). Expresso and EARS are not used.
+Captions use the **`__moss_short__`** scheme: the MOSS-Audio-8B-Thinking
+emotion sentence plus one sampled talking-style sentence, 50/50 mixed with the
+corpus's own caption.
+
+## Architecture
+
+Identical to voiceclap-small — a dual-tower CLAP:
+
+| | |
+| --- | --- |
+| Audio encoder | BUD-E-Whisper-Small: 12 layers × 768 dim × 12 heads, 80-mel @ 16 kHz |
+| Text encoder | `all-MiniLM-L6-v2`: 6 layers × 384 dim, mean-pooled |
+| Joint embedding | 768-d, L2-normalised |
+| Loss | SigLIP sigmoid contrastive + Prototypical Contrastive (PCL, w=0.2) |
+| Total parameters | ~110 M |
+
+**PCL** adds 39 learned emotion prototypes and a cross-entropy term on
+z-scored pseudo-labels derived from emolia's emotion-annotation scalars — a
+small auxiliary loss that sharpens the emotion subspace.
+
+## Results — commercial data costs nothing
+
+Controlled ablation, all arms trained identically (1 node, `__moss_short__` +
+PCL w=0.2, 15 epochs), evaluated at each arm's best epoch:
+
+| Training data | emonet top1 | emonet ρ | VoiceNet-Emo bal@pp | emolia ρ | MAEB-voice |
+|---|---|---|---|---|---|
+| full mix incl. **non-commercial** Emilia | 0.0712 | **0.2308** | 0.6112 | 0.2019 | 0.3472 |
+| **this model — commercial only** | **0.0721** | 0.2061 | **0.6227** | **0.2034** | **0.3564** |
+
+**This model wins 4 of 5 metrics** — emonet top-1, VoiceNet-Emo balanced
+accuracy (+0.0115), emolia Spearman ρ, and MAEB-voice (+0.009). It trails only
+on emonet ρ (−0.025, fine-grained intensity *ranking* on synthetic audio,
+where the dropped Emilia diversity helped). On the in-domain emolia benchmark
+and the 8-task MAEB-voice suite it is strictly better than the
+non-commercial-data model.
+
+Two recovery experiments were tried and discarded: adding VoxCeleb1/2 (CC BY)
+*hurt*, and upweighting the existing safe corpora was flat. Plain commercial
+data is best.
+
+> Absolute numbers are from a 1-node training scale used for the controlled
+> ablation; treat them as relative (commercial-vs-noncommercial deltas), not as
+> the maximum achievable with a full multi-node run.
+
+## Usage
+
+```python
+import torch, soundfile as sf
+from transformers import AutoModel, AutoTokenizer
+
+model = AutoModel.from_pretrained("laion/voiceclap-commercial", trust_remote_code=True).eval()
+tok = AutoTokenizer.from_pretrained("laion/voiceclap-commercial")
+
+# audio: raw mono waveform @ 16 kHz
+wav, sr = sf.read("clip.wav", dtype="float32")
+audio_emb = model.encode_waveform(torch.from_numpy(wav))
+
+# text
+t = tok(["a person speaking with quiet pride in their voice"], padding=True, return_tensors="pt")
+text_emb = model.encode_text(t["input_ids"], attention_mask=t["attention_mask"])
+
+score = (audio_emb @ text_emb.T).item()  # cosine similarity (both L2-normalised)
+```
+
+Conversion from the training checkpoint was verified functionally against the
+original open_clip implementation (cosine ≥ 0.99999 on both towers).
+
+## License
+
+**CC-BY-4.0** — all training data is commercially usable (Emilia-YODAS CC BY 4.0,
+LAION's Got Talent, in-house Majestrino), and the architecture/weights carry no
+non-commercial restriction. This is the distinguishing feature of this model
+versus the standard VoiceCLAP releases, which inherit a non-commercial
+restriction from their CC BY-NC training data.

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "VoiceCLAPSmall"
+  ],
+  "dtype": "float32",
+  "embed_dim": 768,
+  "model_type": "voiceclap-small",
+  "n_ctx": 1500,
+  "n_head": 12,
+  "n_layer": 12,
+  "n_mels": 80,
+  "n_state": 768,
+  "text_hidden_dim": 384,
+  "text_intermediate_size": 1536,
+  "text_layer_norm_eps": 1e-12,
+  "text_max_position_embeddings": 512,
+  "text_num_heads": 12,
+  "text_num_layers": 6,
+  "text_pad_token_id": 0,
+  "text_proj_hidden": 576,
+  "text_vocab_size": 30522,
+  "transformers_version": "5.7.0",
+  "auto_map": {
+    "AutoConfig": "configuration_voiceclap.VoiceCLAPSmallConfig",
+    "AutoModel": "modeling_voiceclap.VoiceCLAPSmall"
+  }
+}

configuration_voiceclap.py

@@ -0,0 +1,42 @@
+"""VoiceCLAP-Small config."""
+from transformers import PretrainedConfig
+
+
+class VoiceCLAPSmallConfig(PretrainedConfig):
+    model_type = "voiceclap-small"
+
+    def __init__(
+        self,
+        embed_dim: int = 768,
+        n_mels: int = 80,
+        n_ctx: int = 1500,
+        n_state: int = 768,
+        n_head: int = 12,
+        n_layer: int = 12,
+        text_hidden_dim: int = 384,
+        text_proj_hidden: int = 576,
+        text_vocab_size: int = 30522,
+        text_intermediate_size: int = 1536,
+        text_num_layers: int = 6,
+        text_num_heads: int = 12,
+        text_max_position_embeddings: int = 512,
+        text_layer_norm_eps: float = 1e-12,
+        text_pad_token_id: int = 0,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.embed_dim = embed_dim
+        self.n_mels = n_mels
+        self.n_ctx = n_ctx
+        self.n_state = n_state
+        self.n_head = n_head
+        self.n_layer = n_layer
+        self.text_hidden_dim = text_hidden_dim
+        self.text_proj_hidden = text_proj_hidden
+        self.text_vocab_size = text_vocab_size
+        self.text_intermediate_size = text_intermediate_size
+        self.text_num_layers = text_num_layers
+        self.text_num_heads = text_num_heads
+        self.text_max_position_embeddings = text_max_position_embeddings
+        self.text_layer_norm_eps = text_layer_norm_eps
+        self.text_pad_token_id = text_pad_token_id

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07849965a3e852b43ddb1b782d7b006ec8b36fe0b91abcdc7fbc7438c81513d7
+size 452717296

modeling_voiceclap.py

@@ -0,0 +1,231 @@
+"""VoiceCLAP-Small: dual-tower CLAP using BUD-E-Whisper-Small + MiniLM.
+
+Standalone single-file implementation. Only depends on PyTorch and
+HuggingFace `transformers` (for `BertModel`, `PreTrainedModel`, and
+`PretrainedConfig`).
+"""
+import math
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import BertConfig, BertModel, PreTrainedModel
+
+try:
+    from .configuration_voiceclap import VoiceCLAPSmallConfig
+except ImportError:
+    from configuration_voiceclap import VoiceCLAPSmallConfig
+
+
+# 30 s of 16 kHz audio = 480_000 samples = 3000 STFT frames (hop 160) =
+# 1500 post-conv frames, which matches the encoder's positional embedding.
+_CHUNK_SAMPLES = 30 * 16000
+
+
+class _LayerNorm(nn.LayerNorm):
+    def forward(self, x):
+        return super().forward(x.float()).type(x.dtype)
+
+
+def _sinusoids(length: int, channels: int, max_timescale: float = 10000.0) -> torch.Tensor:
+    assert channels % 2 == 0
+    log_timescale_increment = math.log(max_timescale) / (channels // 2 - 1)
+    inv_timescales = torch.exp(-log_timescale_increment * torch.arange(channels // 2))
+    scaled_time = torch.arange(length)[:, None] * inv_timescales[None, :]
+    return torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=1)
+
+
+class _MultiHeadAttention(nn.Module):
+    def __init__(self, n_state: int, n_head: int):
+        super().__init__()
+        self.n_head = n_head
+        self.query = nn.Linear(n_state, n_state)
+        self.key = nn.Linear(n_state, n_state, bias=False)
+        self.value = nn.Linear(n_state, n_state)
+        self.out = nn.Linear(n_state, n_state)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        q = self.query(x)
+        k = self.key(x)
+        v = self.value(x)
+        n_batch, n_ctx, n_state = q.shape
+        head_dim = n_state // self.n_head
+        q = q.view(n_batch, n_ctx, self.n_head, head_dim).transpose(1, 2)
+        k = k.view(n_batch, n_ctx, self.n_head, head_dim).transpose(1, 2)
+        v = v.view(n_batch, n_ctx, self.n_head, head_dim).transpose(1, 2)
+        out = F.scaled_dot_product_attention(q, k, v)
+        out = out.transpose(1, 2).reshape(n_batch, n_ctx, n_state)
+        return self.out(out)
+
+
+class _ResidualAttentionBlock(nn.Module):
+    def __init__(self, n_state: int, n_head: int):
+        super().__init__()
+        self.attn = _MultiHeadAttention(n_state, n_head)
+        self.attn_ln = _LayerNorm(n_state)
+        n_mlp = n_state * 4
+        self.mlp = nn.Sequential(nn.Linear(n_state, n_mlp), nn.GELU(), nn.Linear(n_mlp, n_state))
+        self.mlp_ln = _LayerNorm(n_state)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = x + self.attn(self.attn_ln(x))
+        x = x + self.mlp(self.mlp_ln(x))
+        return x
+
+
+class _WhisperAudioEncoder(nn.Module):
+    """Whisper-style audio encoder. Takes a precomputed log-mel spectrogram."""
+
+    def __init__(
+        self,
+        n_mels: int = 80,
+        n_ctx: int = 1500,
+        n_state: int = 768,
+        n_head: int = 12,
+        n_layer: int = 12,
+        output_dim: int = 768,
+    ):
+        super().__init__()
+        self.conv1 = nn.Conv1d(n_mels, n_state, kernel_size=3, padding=1)
+        self.conv2 = nn.Conv1d(n_state, n_state, kernel_size=3, stride=2, padding=1)
+        self.register_buffer("positional_embedding", _sinusoids(n_ctx, n_state))
+        self.blocks = nn.ModuleList(
+            [_ResidualAttentionBlock(n_state, n_head) for _ in range(n_layer)]
+        )
+        self.ln_post = _LayerNorm(n_state)
+        self.avg_pooler = nn.AvgPool1d(kernel_size=2, stride=2)
+        self.proj = nn.Linear(n_state, output_dim)
+
+    def forward(self, mel: torch.Tensor) -> torch.Tensor:
+        # mel: (B, n_mels, T_mel)
+        x = F.gelu(self.conv1(mel))
+        x = F.gelu(self.conv2(x))
+        x = x.permute(0, 2, 1)  # (B, T', D)
+        T = x.size(1)
+        x = x + self.positional_embedding[:T].to(dtype=x.dtype, device=x.device)
+        for block in self.blocks:
+            x = block(x)
+        x = x.permute(0, 2, 1)
+        x = self.avg_pooler(x)
+        x = x.permute(0, 2, 1)
+        x = self.ln_post(x)
+        x = self.proj(x)
+        return x
+
+
+class VoiceCLAPSmall(PreTrainedModel):
+    config_class = VoiceCLAPSmallConfig
+
+    def __init__(self, config: VoiceCLAPSmallConfig):
+        super().__init__(config)
+        self.audio_encoder = _WhisperAudioEncoder(
+            n_mels=config.n_mels,
+            n_ctx=config.n_ctx,
+            n_state=config.n_state,
+            n_head=config.n_head,
+            n_layer=config.n_layer,
+            output_dim=config.embed_dim,
+        )
+        self.audio_proj = nn.Sequential(
+            nn.Linear(config.embed_dim, config.embed_dim),
+            nn.GELU(),
+            nn.Linear(config.embed_dim, config.embed_dim),
+        )
+        bert_config = BertConfig(
+            vocab_size=config.text_vocab_size,
+            hidden_size=config.text_hidden_dim,
+            num_hidden_layers=config.text_num_layers,
+            num_attention_heads=config.text_num_heads,
+            intermediate_size=config.text_intermediate_size,
+            max_position_embeddings=config.text_max_position_embeddings,
+            layer_norm_eps=config.text_layer_norm_eps,
+            pad_token_id=config.text_pad_token_id,
+        )
+        self.text_encoder = BertModel(bert_config, add_pooling_layer=False)
+        self.text_proj = nn.Sequential(
+            nn.Linear(config.text_hidden_dim, config.text_proj_hidden, bias=False),
+            nn.GELU(),
+            nn.Linear(config.text_proj_hidden, config.embed_dim, bias=False),
+        )
+        self.logit_scale = nn.Parameter(torch.zeros(()))
+        self.logit_bias = nn.Parameter(torch.zeros(()))
+
+        # Mel filterbank used by encode_waveform / compute_log_mel.
+        # 80 mel bins x 201 freq bins for n_fft=400, sr=16000 (Whisper-style).
+        self.register_buffer(
+            "mel_filters",
+            torch.zeros(config.n_mels, 201),
+            persistent=True,
+        )
+        self.post_init()
+
+    @torch.no_grad()
+    def compute_log_mel(
+        self, waveform: torch.Tensor, sample_rate: int = 16000
+    ) -> torch.Tensor:
+        """Whisper-style log-mel spectrogram. waveform: (B, T) or (T,) at 16 kHz.
+
+        Returns (B, n_mels, T_mel). Matches the training-time preprocessing
+        bit-exactly so embeddings reproduce the published results.
+        """
+        if sample_rate != 16000:
+            raise ValueError(f"sample_rate must be 16000, got {sample_rate}")
+        if waveform.dim() == 1:
+            waveform = waveform.unsqueeze(0)
+        device = self.mel_filters.device
+        waveform = waveform.to(device=device, dtype=torch.float32)
+        window = torch.hann_window(400, device=device)
+        stft = torch.stft(waveform, n_fft=400, hop_length=160, window=window, return_complex=True)
+        magnitudes = stft[..., :-1].abs() ** 2
+        mel = self.mel_filters.to(magnitudes.dtype) @ magnitudes
+        log_spec = torch.clamp(mel, min=1e-10).log10()
+        log_spec = torch.maximum(log_spec, log_spec.amax(dim=(-2, -1), keepdim=True) - 8.0)
+        log_spec = (log_spec + 4.0) / 4.0
+        return log_spec
+
+    def encode_waveform(self, waveform: torch.Tensor, sample_rate: int = 16000) -> torch.Tensor:
+        """Encode raw 16 kHz waveform; calls ``compute_log_mel`` then ``encode_audio``.
+
+        Handles any-length audio. Clips at or below 30 s are zero-padded to a
+        full 30 s window (matching Whisper's training-time padding). Longer
+        clips are split into non-overlapping 30 s chunks; each chunk is encoded
+        independently and the resulting embeddings are averaged and
+        re-normalised. Output shape: ``(B, embed_dim)``.
+        """
+        if sample_rate != 16000:
+            raise ValueError(f"sample_rate must be 16000, got {sample_rate}")
+        if waveform.dim() == 1:
+            waveform = waveform.unsqueeze(0)
+        B, T = waveform.shape
+        n_chunks = max(1, (T + _CHUNK_SAMPLES - 1) // _CHUNK_SAMPLES)
+        pad = n_chunks * _CHUNK_SAMPLES - T
+        if pad > 0:
+            waveform = F.pad(waveform, (0, pad))
+        chunks = waveform.view(B, n_chunks, _CHUNK_SAMPLES).reshape(B * n_chunks, _CHUNK_SAMPLES)
+        mel = self.compute_log_mel(chunks, sample_rate=sample_rate)
+        feats = self.encode_audio(mel)
+        if n_chunks == 1:
+            return feats
+        feats = feats.view(B, n_chunks, -1).mean(dim=1)
+        return F.normalize(feats, dim=-1)
+
+    def encode_audio(self, mel: torch.Tensor) -> torch.Tensor:
+        feats = self.audio_encoder(mel)            # (B, T', D)
+        feats = feats.mean(dim=1)                  # clip-level mean
+        feats = self.audio_proj(feats)
+        return F.normalize(feats, dim=-1)
+
+    def encode_text(
+        self,
+        input_ids: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+    ) -> torch.Tensor:
+        if attention_mask is None:
+            attention_mask = (input_ids != self.config.text_pad_token_id).long()
+        out = self.text_encoder(input_ids=input_ids, attention_mask=attention_mask)
+        hidden = out.last_hidden_state                                  # (B, T, H)
+        mask = attention_mask.unsqueeze(-1).to(hidden.dtype)
+        pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)
+        feats = self.text_proj(pooled)
+        return F.normalize(feats, dim=-1)

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "chunk_length": 30,
+  "dither": 0.0,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 80,
+  "hop_length": 160,
+  "n_fft": 400,
+  "n_samples": 480000,
+  "nb_max_frames": 3000,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": false,
+  "sampling_rate": 16000
+}

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}