Update README.md
Browse files
README.md
CHANGED
|
@@ -29,19 +29,12 @@ SPLADE models are a fine balance between retrieval effectiveness (quality) and r
|
|
| 29 |
*(Pure MLE folks should not conflate efficiency to model inference efficiency. Our main focus is on retrieval efficiency. Hereinafter efficiency is a short hand for retrieval efficiency unless explicitly qualified otherwise. Not that inference efficiency is not important, we will address that subsequently.)*
|
| 30 |
|
| 31 |
**TL;DR of Our attempt & results**
|
| 32 |
-
1. FLOPS tuning:
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
2. and a retrieval latency of - **47.27ms**. (multi-threaded)
|
| 39 |
-
3. On **mono-GPU** with **only 5 negatives per query**.
|
| 40 |
-
4. For Industry setting
|
| 41 |
-
- Effectiveness on custom domains needs more than just **Trading FLOPS for tiny gains**.
|
| 42 |
-
- The Premise "SPLADE++ are not well suited to mono-cpu retrieval" does not hold.
|
| 43 |
-
|
| 44 |
-
<img src="./ID.png" width=500 height=350/>
|
| 45 |
|
| 46 |
*Note: The paper refers to the best performing models as SPLADE++, hence for consistency we are reusing the same.*
|
| 47 |
|
|
|
|
| 29 |
*(Pure MLE folks should not conflate efficiency to model inference efficiency. Our main focus is on retrieval efficiency. Hereinafter efficiency is a short hand for retrieval efficiency unless explicitly qualified otherwise. Not that inference efficiency is not important, we will address that subsequently.)*
|
| 30 |
|
| 31 |
**TL;DR of Our attempt & results**
|
| 32 |
+
1. FLOPS tuning: Seperate **Seq lens and Severely restrictive token budget** doc(128) & query(24) NOT 256 unlike Official SPLADE++. Inspired from **SparseEmbed** (instead of 2 models for query & doc).
|
| 33 |
+
3. Init Weights: **MLM adapted on MS MARCO corpus**.
|
| 34 |
+
4. Achieves a modest yet competitive effectiveness **MRR@10 37.22** in ID data (& OOD) and a retrieval latency of - **47.27ms**. (multi-threaded) all On **mono-GPU** with **only 5 negatives per query**.
|
| 35 |
+
4. For Industry setting: Effectiveness on custom domains needs more than just **Trading FLOPS for tiny gains** and The Premise "SPLADE++ are not well suited to mono-cpu retrieval" does not hold.
|
| 36 |
+
|
| 37 |
+
<img src="./ID.png" width=550 height=450/>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
*Note: The paper refers to the best performing models as SPLADE++, hence for consistency we are reusing the same.*
|
| 40 |
|