Add pipeline_tag, library_name, and paper abstract to model card

This PR improves the model card by adding the `pipeline_tag: text-ranking` and `library_name: transformers` to the metadata. This will enhance discoverability on the Hugging Face Hub (e.g., at https://huggingface.co/models?pipeline_tag=text-ranking) and enable the "how to use" widget for Transformers models.

Additionally, the paper abstract has been added to provide more comprehensive information about the model's purpose and methodology.

README.md

@@ -1,16 +1,19 @@
 ---
-license: apache-2.0
+base_model:
+- internlm/internlm2_5-1_8b
 language:
 - en
 - zh
-base_model:
-- internlm/internlm2_5-1_8b
+license: apache-2.0
 tags:
 - Reward
 - RL
 - RFT
 - Reward Model
+pipeline_tag: text-ranking
+library_name: transformers
 ---
+
 <div align="center">
 
 <img src="./misc/logo.png" width="400"/><br>
@@ -31,17 +34,21 @@ tags:
 
 </div>
 
+# Abstract
+
+We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards a target policy with desired behaviors. Based on this conceptual insight, we propose a scalable pre-training method named Policy Discriminative Learning (POLAR), which trains a reward model (RM) to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between one policy and an arbitrary target policy, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships. Leveraging the POLAR pre-training paradigm, we present a series of RMs with parameter scales from 1.8B to 7B. Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA baselines. POLAR also shows robust generalization capabilities in RLHF using Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly enhancing policy performance--improving LLaMa3.1-8B from an average of 47.36% to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. Moreover, scaling experiments reveal a clear power-law relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. The impressive performance, strong generalization, and scaling properties suggest that POLAR is a promising direction for developing general and strong reward models.
+
 # Introduction
 
 POLAR represents a significant breakthrough in scalar-based reward models achieved through large-scale pre-training. It leverages the innovative **POL**icy Discrimin**A**tive Lea**R**ning (**POLAR**) paradigm——a scalable, high-level optimization objective——to effectively discriminate between policies using a large-scale synthetic corpora. Following pre-training, POLAR RMs are fine-tuned with minimal preference data, rapidly aligning with human preferences. Key features of POLAR include:
 
-* **Innovative Pre-training Paradigm:**  POLAR trains a reward model to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between two policies, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships.
+*   **Innovative Pre-training Paradigm:** POLAR trains a reward model to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between two policies, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships.
 
-* **Tailored for Reinforcement Fine-tuning:** POLAR assigns rewards to LLM trajectories based on given references, perfectly aligning with the Reinforcement Fine-tuning (RFT) framework. POLAR provides a promising solution for applying RFT in generic scenarios.
+*   **Tailored for Reinforcement Fine-tuning:** POLAR assigns rewards to LLM trajectories based on given references, perfectly aligning with the Reinforcement Fine-tuning (RFT) framework. POLAR provides a promising solution for applying RFT in generic scenarios.
 
-* **Superior Performance and Generalization:** POLAR achieves state-of-the-art results on downstream reinforcement learning tasks, consistently delivering accurate and reliable reward signals that generalize effectively to unseen scenarios and significantly reducing reward hacking.
+*   **Superior Performance and Generalization:** POLAR achieves state-of-the-art results on downstream reinforcement learning tasks, consistently delivering accurate and reliable reward signals that generalize effectively to unseen scenarios and significantly reducing reward hacking.
 
-* **Easy to Customize:** Pre-trained checkpoints of POLAR are available, enabling researchers to conveniently fine-tune the RM for various customized scenarios, thus facilitating straightforward adaptation and expansion tailored to specific applications and experimental requirements.
+*   **Easy to Customize:** Pre-trained checkpoints of POLAR are available, enabling researchers to conveniently fine-tune the RM for various customized scenarios, thus facilitating straightforward adaptation and expansion tailored to specific applications and experimental requirements.
 
 <img src="./misc/intro.jpeg"/><br>
 
@@ -60,18 +67,18 @@ We conducted a comprehensive evaluation of POLAR-1.8B via the Proximal Policy Op
 
 You could employ the latest [xtuner](https://github.com/InternLM/xtuner) to fine-tune and use POLAR. Xtuner is an efficient, flexible and full-featured toolkit for fine-tuning LLMs.
 
-- It is recommended to build a Python-3.10 virtual environment using conda
+-   It is recommended to build a Python-3.10 virtual environment using conda
 
-  ```bash
-  conda create --name xtuner-env python=3.10 -y
-  conda activate xtuner-env
-  ```
+    ```bash
+    conda create --name xtuner-env python=3.10 -y
+    conda activate xtuner-env
+    ```
 
-- Install xtuner via pip
+-   Install xtuner via pip
 
-  ```shell
-  pip install 'git+https://github.com/InternLM/xtuner.git@main#egg=xtuner[deepspeed]'
-  ```
+    ```shell
+    pip install 'git+https://github.com/InternLM/xtuner.git@main#egg=xtuner[deepspeed]'
+    ```
 
 ## Inference
 
@@ -128,7 +135,7 @@ print(rewards)
 
 #### Requirements
 
-- lmdeploy >= 0.9.1
+-   lmdeploy >= 0.9.1
 
 #### Server Launch
 
@@ -158,7 +165,7 @@ print(rewards)
 
 #### Requirements
 
-- 0.4.3.post4 <= sglang <= 0.4.4.post1
+-   0.4.3.post4 <= sglang <= 0.4.4.post1
 
 #### Server Launch
 
@@ -189,7 +196,7 @@ print(rewards)
 
 #### Requirements
 
-- vllm >= 0.8.0
+-   vllm >= 0.8.0
 
 #### Server Launch
 
@@ -220,8 +227,8 @@ print(rewards)
 
 ### Requirements
 
-- flash_attn
-- tensorboard
+-   flash_attn
+-   tensorboard
 
 ### Data format
 
@@ -238,31 +245,31 @@ Unlike traditional reward models, POLAR requires an additional reference traject
 
 ### Training steps
 
-- **Step 0:** Prepare the config. We provide examplar ready-to-use configs [here](https://github.com/InternLM/POLAR/blob/main/examples/xtuner_configs/POLAR_1_8B_full_varlenattn_custom_dataset.py). If the provided configs cannot meet the requirements, please copy the provided config and do modification following the [xtuner guideline](https://github.com/InternLM/xtuner/blob/main/docs/en/get_started/quickstart.md). For more details of reward model training settings, please see the xtuner [reward model guideline](https://github.com/InternLM/xtuner/blob/main/docs/en/reward_model/modify_settings.md).
+-   **Step 0:** Prepare the config. We provide examplar ready-to-use configs [here](https://github.com/InternLM/POLAR/blob/main/examples/xtuner_configs/POLAR_1_8B_full_varlenattn_custom_dataset.py). If the provided configs cannot meet the requirements, please copy the provided config and do modification following the [xtuner guideline](https://github.com/InternLM/xtuner/blob/main/docs/en/get_started/quickstart.md). For more details of reward model training settings, please see the xtuner [reward model guideline](https://github.com/InternLM/xtuner/blob/main/docs/en/reward_model/modify_settings.md).
 
-- **Step 1:** Start fine-tuning.
+-   **Step 1:** Start fine-tuning.
 
     ```shell
     xtuner train ${CONFIG_FILE_PATH}
     ```
 
-  For example, you can start the fine-tuning of POLAR-1_8B-Base by
+    For example, you can start the fine-tuning of POLAR-1_8B-Base by
 
-  ```shell
-  # On a single GPU
-  xtuner train /path/to/POLAR_1_8B_full_varlenattn_custom_dataset.py --deepspeed deepspeed_zero2
+    ```shell
+    # On a single GPU
+    xtuner train /path/to/POLAR_1_8B_full_varlenattn_custom_dataset.py --deepspeed deepspeed_zero2
 
-  # On multiple GPUs
-  NPROC_PER_NODE=${GPU_NUM} xtuner train /path/to/POLAR_1_8B_full_varlenattn_custom_dataset.py --deepspeed deepspeed_zero2
-  ```
+    # On multiple GPUs
+    NPROC_PER_NODE=${GPU_NUM} xtuner train /path/to/POLAR_1_8B_full_varlenattn_custom_dataset.py --deepspeed deepspeed_zero2
+    ```
 
-  Here, `--deepspeed` means using [DeepSpeed](https://github.com/microsoft/DeepSpeed) to optimize the training. Xtuner comes with several integrated strategies including ZeRO-1, ZeRO-2, and ZeRO-3. If you wish to disable this feature, simply remove this argument.
+    Here, `--deepspeed` means using [DeepSpeed](https://github.com/microsoft/DeepSpeed) to optimize the training. Xtuner comes with several integrated strategies including ZeRO-1, ZeRO-2, and ZeRO-3. If you wish to disable this feature, simply remove this argument.
 
-- **Step 2:** Convert the saved PTH model (if using DeepSpeed, it will be a directory) to Hugging Face model, by
+-   **Step 2:** Convert the saved PTH model (if using DeepSpeed, it will be a directory) to Hugging Face model, by
 
-  ```shell
-  xtuner convert pth_to_hf ${CONFIG_FILE_PATH} ${PTH} ${SAVE_PATH}
-  ```
+    ```shell
+    xtuner convert pth_to_hf ${CONFIG_FILE_PATH} ${PTH} ${SAVE_PATH}
+    ```
 
 # Examples
 
@@ -297,7 +304,9 @@ rewards = client(data)
 sorted_res = sorted(zip(outputs, rewards), key=lambda x: x[1], reverse=True)
 
 for output, reward in sorted_res:
-    print(f"Output: {output}\nReward: {reward}\n")
+    print(f"Output: {output}
+Reward: {reward}
+")
 ```
 
 ```txt
@@ -351,7 +360,9 @@ rewards = client(data)
 sorted_res = sorted(zip(outputs, rewards), key=lambda x: x[1], reverse=True)
 
 for output, reward in sorted_res:
-    print(f"Output: {output}\nReward: {reward}\n")
+    print(f"Output: {output}
+Reward: {reward}
+")
 ```
 
 ```txt