bekko-embedding-v1
Collection
Compact multilingual embedding models with 8M and 25M active parameters, including pretrained checkpoints and retrieval-fine-tuned releases. • 7 items • Updated • 1
bekko-embedding-v1-a25m-pt
bekko-embedding-v1-a25m-pt is the pretrained model for bekko-embedding-v1-a25m. It is a starting point for additional training on a downstream task, not a model intended for direct use.
Fine-tune this checkpoint for your target task before using it. It has not received the final supervised retrieval fine-tuning and should not be used directly as a general-purpose embedding or retrieval model. For general-purpose retrieval, use bekko-embedding-v1-a25m.
Model Details
| Item | Value |
|---|---|
| Model type | Pretrained Sentence Transformer dense embedding model |
| Architecture | mmBERT (ModernBERT-style) encoder, 13 layers, hidden size 384 |
| Base model | hotchpotch/mmBERT-L13H384-pruned |
| Original backbone | jhu-clsp/mmBERT-small |
| Active parameters | 24,930,432 |
| Total parameters | 123,234,432 |
| Embedding dimension | 384 |
| Supported truncate dimensions | 256, 128, 64 |
| Max sequence length | 8192 tokens |
| Pooling | Mean pooling |
| Similarity | Cosine similarity |
| Query/document prefixes | None |
The model was pretrained with multilingual text pairs using Matryoshka representation learning and quantization-aware training. The 13-layer backbone retains layers 0–11 and 18 from mmBERT-small.
The released a25m fine-tuned model starts from an equal-weight linear merge of this pretraining run's final checkpoint and a strong intermediate checkpoint from the same trajectory.
Intended Use
Use this checkpoint as initialization for further training on a specific downstream task. Appropriate uses include:
- fine-tuning for retrieval, classification, reranking, or another target task,
- reproducing or adapting the bekko fine-tuning pipeline,
- pretrain-versus-fine-tuned comparisons and ablation studies.
License
MIT
- Downloads last month
- 18