File size: 4,837 Bytes
874b09f 7fa63d0 874b09f feadc03 874b09f cde8a23 874b09f feadc03 874b09f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 |
---
license: mit
---
<h1 align="center">KaLM-Embedding-V2</h1>
**KaLM-Embedding-V2** is a versatile and compact embedding model, which achieves impressive performance in general-purpose text embedding tasks by leveraging superior training techniques and data.
KaLM-embedding-multilingual-mini-instruct-v2 is trained from [Qwen/Qwen2-0.5B](https://huggingface.co/Qwen/Qwen2-0.5B) with massive weakly-supervised pre-training and high-quality supervised fine-tuning data.
The model incorporates several innovative designs:
- Architectural Design: integration of bidirectional attention, enhancing representation learning.
- Training Recipe: multi-stage training strategy, progressively improving the generalization and performance.
- Training Objective: focal-style reweighting mechanism and online hard-negative mixing strategy to improve the efficiency and continuity of embedding training.
- Training Data: 20 categories of data for pre-training and 100 categories of data for fine-tuning, as well as comprehensive recipes for curating training datasets.
## Model Information
- Model Size: 0.5B
- Embedding Dimension: 896
- Max Input Tokens: 32k
- MRL: 896 512 256 128 64
## 📑 Open-source Plan
- [x] Model Checkpoint
- [x] [KaLM-embedding-multilingual-mini-v1](https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-v1)
- [x] [KaLM-embedding-multilingual-mini-instruct-v1](https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1)
- [x] [KaLM-embedding-multilingual-mini-instruct-v1.5](https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5)
- [x] [KaLM-embedding-multilingual-mini-instruct-v2](https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v2)
- [x] Training and Evaluation Code: [HITsz-TMG/KaLM-Embedding](https://github.com/HITsz-TMG/KaLM-Embedding)
- [x] Technical Report: [KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model](https://arxiv.org/abs/2501.01028)
- [ ] Training Data
## Evaluation
### Overall results on MTEB (cmn, v1) and MTEB (eng, v1).
### Detailed model performance on MTEB (cmn, v1).
### Detailed model performance on MTEB (eng, v1).
## Requirements
Since we have used the Qwen2 model, we advise you to install `transformers>=4.37.0`, or you might encounter the following error:
```
KeyError: 'qwen2'
```
## Usage
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
```
pip install -U sentence-transformers
```
Then you can use the model like this:
```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer("{MODEL_NAME_OR_PATH}", trust_remote_code=True, truncate_dim=None, model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"})
model.max_seq_length = 512
embeddings = model.encode(
sentences,
normalize_embeddings=True,
batch_size=256,
show_progress_bar=True
)
print(embeddings)
```
We add task instructions for queries in asymmetric tasks: retrieval, reranking, classification, and clustering.
And, we add task instructions for both queries and passages in symmetric tasks: STS and pair classification.
If you want to add task instructions to the query, you can use the model like this:
```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer("{MODEL_NAME_OR_PATH}", trust_remote_code=True, truncate_dim=None, model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"})
model.max_seq_length = 512
prompt = "Instruct: Classifying the category of french news. \n Query: "
embeddings = model.encode(
sentences,
prompt=prompt,
normalize_embeddings=True,
batch_size=256,
show_progress_bar=True
)
print(embeddings)
```
## Citation
If you find this model useful, please consider giving a star and citation.
```
@article{zhao2025kalmv2,
title={KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model},
author={},
journal={},
year={2025}
}
@article{hu2025kalm,
title={KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model},
author={Hu, Xinshuo and Shan, Zifei and Zhao, Xinping and Sun, Zetian and Liu, Zhenyu and Li, Dongfang and Ye, Shaolin and Wei, Xinyuan and Chen, Qian and Hu, Baotian and others},
journal={arXiv preprint arXiv:2501.01028},
year={2025}
}
```
## Contact
If you encounter any issue, feel free to contact us via the email: <[email protected]>, <[email protected]>
|