Update README.md
Browse files
README.md
CHANGED
|
@@ -6,7 +6,7 @@ base_model:
|
|
| 6 |
# CompassJudger-1
|
| 7 |
|
| 8 |
<p align="center">
|
| 9 |
-
🤗 <a href="https://huggingface.co/opencompass">Hugging Face</a>   |   🤖 <a href="https://modelscope.cn/organization/opencompass">ModelScope</a>   |    📑 <a href="">Paper</a>   
|
| 10 |
<br>
|
| 11 |
|
| 12 |
</p>
|
|
@@ -27,7 +27,7 @@ Here provides a code to show you how to load the tokenizer and model and how to
|
|
| 27 |
```python
|
| 28 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 29 |
|
| 30 |
-
model_name = "opencompass/CompassJudger-1"
|
| 31 |
|
| 32 |
model = AutoModelForCausalLM.from_pretrained(
|
| 33 |
model_name,
|
|
@@ -63,7 +63,7 @@ print(response)
|
|
| 63 |
|
| 64 |
|
| 65 |
We also provide some examples for different usage situations:
|
| 66 |
-
###
|
| 67 |
|
| 68 |
```
|
| 69 |
**Input**: Hello, can you help me to judge something?
|
|
@@ -173,7 +173,9 @@ cd opencompass
|
|
| 173 |
pip install -e .
|
| 174 |
python run.py configs/eval_judgerbench.py --mode all --reuse latest
|
| 175 |
```
|
|
|
|
| 176 |
|
|
|
|
| 177 |
|
| 178 |
## Use CompassJudger-1 to Test Subjective Datasets in OpenCompass
|
| 179 |
|
|
@@ -224,8 +226,8 @@ infer = dict(
|
|
| 224 |
judge_models = [dict(
|
| 225 |
dict(
|
| 226 |
type=TurboMindModelwithChatTemplate,
|
| 227 |
-
abbr='CompassJudger-1-7B,
|
| 228 |
-
path='
|
| 229 |
engine_config=dict(session_len=16384, max_batch_size=16, tp=1),
|
| 230 |
gen_config=dict(top_k=1, temperature=1e-6, top_p=0.9, max_new_tokens=2048),
|
| 231 |
max_seq_len=16384,
|
|
@@ -253,13 +255,17 @@ For more detailed subjective evaluation guidelines, please refer to: https://git
|
|
| 253 |
|
| 254 |
To facilitate better comparisons within the community, we have tested the subjective performance of some models using CompassJudger-1.
|
| 255 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 256 |
## Citation
|
| 257 |
|
| 258 |
```bib
|
| 259 |
@article{cao2024compass,
|
| 260 |
title={CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution},
|
| 261 |
author={Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, Kai Chen},
|
| 262 |
-
journal={arXiv preprint arXiv:2410.
|
| 263 |
year={2024}
|
| 264 |
}
|
| 265 |
```
|
|
@@ -269,4 +275,4 @@ To facilitate better comparisons within the community, we have tested the subjec
|
|
| 269 |
- https://github.com/open-compass/opencompass
|
| 270 |
- https://github.com/InternLM/InternLM
|
| 271 |
- https://github.com/QwenLM/Qwen2.5
|
| 272 |
-
- https://github.com/InternLM/xtuner
|
|
|
|
| 6 |
# CompassJudger-1
|
| 7 |
|
| 8 |
<p align="center">
|
| 9 |
+
🤗 <a href="https://huggingface.co/opencompass">Hugging Face</a>   |   🤖 <a href="https://modelscope.cn/organization/opencompass">ModelScope</a>   |    📑 <a href="https://arxiv.org/pdf/2410.16256">Paper</a>    |    🎖️ <a href="https://huggingface.co/spaces/opencompass/judgerbench_leaderboard">Leaderboard</a>   
|
| 10 |
<br>
|
| 11 |
|
| 12 |
</p>
|
|
|
|
| 27 |
```python
|
| 28 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 29 |
|
| 30 |
+
model_name = "opencompass/CompassJudger-1-7B-Instruct"
|
| 31 |
|
| 32 |
model = AutoModelForCausalLM.from_pretrained(
|
| 33 |
model_name,
|
|
|
|
| 63 |
|
| 64 |
|
| 65 |
We also provide some examples for different usage situations:
|
| 66 |
+
### General Chat
|
| 67 |
|
| 68 |
```
|
| 69 |
**Input**: Hello, can you help me to judge something?
|
|
|
|
| 173 |
pip install -e .
|
| 174 |
python run.py configs/eval_judgerbench.py --mode all --reuse latest
|
| 175 |
```
|
| 176 |
+
We also provided a leaderboard for JudgerBench: https://huggingface.co/spaces/opencompass/judgerbench_leaderboard
|
| 177 |
|
| 178 |
+
If you want to add your model to this leaderboard, welcome to add an issue in this Repository.
|
| 179 |
|
| 180 |
## Use CompassJudger-1 to Test Subjective Datasets in OpenCompass
|
| 181 |
|
|
|
|
| 226 |
judge_models = [dict(
|
| 227 |
dict(
|
| 228 |
type=TurboMindModelwithChatTemplate,
|
| 229 |
+
abbr='CompassJudger-1-7B-Instruct',
|
| 230 |
+
path='opencompass/CompassJudger-1-7B-Instruct',
|
| 231 |
engine_config=dict(session_len=16384, max_batch_size=16, tp=1),
|
| 232 |
gen_config=dict(top_k=1, temperature=1e-6, top_p=0.9, max_new_tokens=2048),
|
| 233 |
max_seq_len=16384,
|
|
|
|
| 255 |
|
| 256 |
To facilitate better comparisons within the community, we have tested the subjective performance of some models using CompassJudger-1.
|
| 257 |
|
| 258 |
+
See in: [https://huggingface.co/spaces/opencompass/judgerbench_leaderboard](https://huggingface.co/spaces/opencompass/compassjudger_subj_eval_leaderboard)
|
| 259 |
+
|
| 260 |
+
If you want to add your model to this leaderboard, welcome to add an issue in this Repository.
|
| 261 |
+
|
| 262 |
## Citation
|
| 263 |
|
| 264 |
```bib
|
| 265 |
@article{cao2024compass,
|
| 266 |
title={CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution},
|
| 267 |
author={Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, Kai Chen},
|
| 268 |
+
journal={arXiv preprint arXiv:2410.16256},
|
| 269 |
year={2024}
|
| 270 |
}
|
| 271 |
```
|
|
|
|
| 275 |
- https://github.com/open-compass/opencompass
|
| 276 |
- https://github.com/InternLM/InternLM
|
| 277 |
- https://github.com/QwenLM/Qwen2.5
|
| 278 |
+
- https://github.com/InternLM/xtuner
|