Update README.md

added results and modified model summary.

README.md

@@ -13,12 +13,13 @@ base_model:
 # Granite-3.3-2B-Instruct
 
 **Model Summary:**
-Granite-3.3-2B-Instruct is an 2-billion parameter long-context AI model fine-tuned for superior reasoning and instruction-following capabilities. Built on top of Granite-3.3-2B-Base, the model delivers significant gains on benchmarks for measuring generic performance including Alpaca and Arena-Hard, and significant improvements in mathematics, coding, and instruction following. It incorporates Fill-in-the-Middle (FIM) capabilities and supports structured reasoning through toggleable custom tags (\<think\>\<\/think\> and \<response\>\<\/response\>), providing clear separation between internal reasoning processes and final outputs. The model has been trained on a carefully balanced combination of permissively licensed data and curated synthetic tasks.
+Granite-3.3-2B-Instruct is a 2-billion parameter 128K context length language model fine-tuned for improved reasoning and instruction-following capabilities. Built on top of Granite-3.3-2B-Base, the model delivers significant gains on benchmarks for measuring generic performance including AlpacaEval-2.0 and Arena-Hard, and improvements in mathematics, coding, and instruction following. It also supports Fill-in-the-Middle (FIM) for code completion tasks and structured reasoning through \<think\>\<\/think\> and \<response\>\<\/response\> tags, providing clear separation between internal thoughts and final outputs. The model has been trained on a carefully balanced combination of permissively licensed data and curated synthetic tasks.
 
 
 - **Developers:** Granite Team, IBM
+- **GitHub Repository:** [ibm-granite/granite-3.3-language-models](https://github.com/ibm-granite/granite-3.3-language-models)
 - **Website**: [Granite Docs](https://www.ibm.com/granite/docs/)
-- **Release Date**: April 10th, 2025
+- **Release Date**: April 16th, 2025
 - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
 **Supported Languages:** 
@@ -37,6 +38,7 @@ This model is designed to handle general instruction-following tasks and can be
 * Code related tasks
 * Function-calling tasks
 * Multilingual dialog use cases
+* **Fill-in-the-middle**
 * Long-context tasks including long document/meeting summarization, long document QA, etc.
 
 
@@ -82,6 +84,7 @@ prediction = tokenizer.decode(output[0, input_ids["input_ids"].shape[1]:], skip_
 print(prediction)
 ```
 
+**Example Outputs**
 **Example Outputs**
 - thinking=True
 ```md
@@ -205,10 +208,11 @@ By implementing this innovative prevention strategy, we can significantly reduce
 <table>
   
 <thead>
+    <caption style="text-align:center"><b>Comparison with different models over various benchmarks. Scores of AlpacaEval-2.0 and Arena-Hard are calculated with thinking=True</b></caption>
   <tr>
     <th style="text-align:left; background-color: #001d6c; color: white;">Models</th>
     <th style="text-align:center; background-color: #001d6c; color: white;">ArenaHard</th>
-    <th style="text-align:center; background-color: #001d6c; color: white;">Alpaca-Eval-2</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">AlpacaEval-2.0</th>
     <th style="text-align:center; background-color: #001d6c; color: white;">MMLU</th>
     <th style="text-align:center; background-color: #001d6c; color: white;">PopQA</th>
     <th style="text-align:center; background-color: #001d6c; color: white;">TruthfulQA</th>
@@ -218,8 +222,55 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <th style="text-align:center; background-color: #001d6c; color: white;">HumanEval</th>
    <th style="text-align:center; background-color: #001d6c; color: white;">HumanEval+</th>
   <th style="text-align:center; background-color: #001d6c; color: white;">IFEval</th>
+  <th style="text-align:center; background-color: #001d6c; color: white;">Attaq</th>
   </tr></thead>
   <tbody>
+<tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-2B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">23.3</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">27.17</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">57.11</td> 
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">20.55</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">59.79</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">54.46</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">18.68</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">67.55</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">79.45</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">75.26</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">63.59</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">84.7</td>
+  </tr>
+  <tr>
+      <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-2B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">24.86</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">34.51</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">57.18</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">20.56</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">59.8</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">52.27</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">21.12</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">67.02</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">80.13</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">73.39</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">61.55</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">83.23</td>
+  </tr>
+  <tr>
+      <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.3-2B-Instruct</b></td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 28.86 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 43.45 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 55.88 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 18.4 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 58.97 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 5.41 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 35.98 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 72.48 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 80.51 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 75.68 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 65.8 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">87.47</td>
+      </tr>
+      
   <tr>
     <td style="text-align:left; background-color: #DAE8FF; color: black;">Llama-3.1-8B-Instruct</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">36.43</td>
@@ -233,6 +284,7 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;">85.32</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">80.15</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">79.10</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">83.43</td>
 
   </tr>
            
@@ -249,7 +301,7 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;">67.54</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">62.91</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">66.50</td>
-
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">42.87</td>
   </tr>
       
   <tr>
@@ -265,7 +317,7 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;">93.35</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">89.91</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">74.90</td>
-
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">81.90</td>
   </tr>
       
   <tr>
@@ -281,7 +333,7 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;">79.89</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">78.43</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">59.10</td>
-
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">42.45</td>
   </tr>
 
   <tr>
@@ -297,25 +349,9 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;">89.63</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">85.79</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">73.20</td>
-
-  </tr>
-      
-      
-<tr>
-    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-2B-Instruct</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">23.3</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">27.17</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">57.11</td> 
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">20.55</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">59.79</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">54.46</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">18.68</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">67.55</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">79.45</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">75.26</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">63.59</td>
-
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">85.73</td>
   </tr>
+            
 
 <tr>
       <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-8B-Instruct</td>
@@ -330,24 +366,8 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;">89.35</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">85.72</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;">74.31</td>
- 
+     <td style="text-align:center; background-color: #DAE8FF; color: black;">84.7</td>
   </tr>
- 
-  <tr>
-      <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.2-2B-Instruct</b></td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">24.86</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">34.51</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">57.18</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">20.56</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">59.8</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">52.27</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">21.12</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">67.02</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">80.13</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">73.39</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;">61.55</td>
-  </tr> 
-      
   <tr>
       <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.3-8B-Instruct</b></td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 57.56 </td>
@@ -361,22 +381,8 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 89.73 </td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 86.09 </td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 74.82 </td>
-      </tr>
-      
-  <tr>
-      <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.3-2B-Instruct</b></td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 28.86 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 43.45 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 55.88 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 18.4 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 58.97 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 5.41 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 35.98 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 72.48 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 80.51 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 75.68 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 65.8 </td>
-      </tr>           
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">88.5</td>
+      </tr>                 
 </tbody></table>
 
 <table>
@@ -388,37 +394,127 @@ By implementing this innovative prevention strategy, we can significantly reduce
     <th style="text-align:center; background-color: #001d6c; color: white;">MATH500</th>
   </tr></thead>
   <tbody>
-  <tr>
-    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-8B-Instruct</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 1.97 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 48.73 </td>
-  </tr>
   <tr>
     <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-2B-Instruct</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 0.89 </td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 35.07 </td>
   </tr>
-  <tr>
-    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-8B-Instruct</td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 2.43 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 52.8 </td>
-  </tr>
   <tr>
     <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-2B-Instruct</td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 0.89 </td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 35.54 </td>
   </tr>
+  <tr>
+      <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.3-2B-Instruct</b></td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 3.28 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 58.09 </td>
+  </tr>
+  <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-8B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 1.97 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 48.73 </td>
+  </tr>
+  <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-8B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 2.43 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 52.8 </td>
+  </tr>
   <tr>
       <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.3-8B-Instruct</b></td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 8.12 </td>
     <td style="text-align:center; background-color: #DAE8FF; color: black;"> 69.02 </td>
   </tr>
+    </tbody></table>
+    
+</tbody></table>
+
+<table>
+  <caption><b>Thinking Ablation</b></caption>
+<thead>
   <tr>
-      <td style="text-align:left; background-color: #DAE8FF; color: black;"><b>Granite-3.3-2B-Instruct</b></td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 3.28 </td>
-    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 58.09 </td>
+    <th rowspan="4" style="text-align:left; background-color: #001d6c; color: white;">Models</th>
+    <th colspan="4" style="text-align:center; background-color: #001d6c; color: white;">Thinking=False</th>
+    <th colspan="4" style="text-align:center; background-color: #001d6c; color: white;">Thinking=True</th>
   </tr>
-    </tbody></table>
+  <tr>
+    <th style="text-align:center; background-color: #001d6c; color: white;">ArenaHard</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">Alpaca-Eval-2</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">AIME24</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">MATH500</th>      
+    <th style="text-align:center; background-color: #001d6c; color: white;">ArenaHard</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">Alpaca-Eval-2</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">AIME24</th>
+    <th style="text-align:center; background-color: #001d6c; color: white;">MATH500</th>    
+  </tr></thead>
+    <tbody>
+         <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-2B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">23.3</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">27.17</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">0.89</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">35.07</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+        </tr>
+         <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-2B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">30.42</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">31.65</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">0.94</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">37.15</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">26.6</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">34.51</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">0.89</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">35.54</td>
+        </tr>
+         <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.3-2B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> - </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> - </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 28.86 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 43.45 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">3.28</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">58.09</td>
+        </tr>
+         <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.1-8B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">37.58</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">30.34</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">1.97</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">48.73</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+        </tr>
+         <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.2-8B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">40.54</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">36.89</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">3.13</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">50.78</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">55.25</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">61.19</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">2.43</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">52.8</td>
+        </tr>
+         <tr>
+    <td style="text-align:left; background-color: #DAE8FF; color: black;">Granite-3.3-8B-Instruct</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> - </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> - </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 57.56 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;"> 62.68 </td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">8.12</td>
+    <td style="text-align:center; background-color: #DAE8FF; color: black;">69.02</td>
+        </tr>
+    </table>
+    <tbody>
 
 **Training Data:** 
 Overall, our training data is largely comprised of two key sources: (1) publicly available datasets with permissive license, (2) internal synthetically generated data targeted to enhance reasoning capabilites.