Update README.md

README.md

@@ -1,13 +1,16 @@
 ---
-base_model: answerdotai/ModernBERT-base
+base_model:
+- answerdotai/ModernBERT-base
+- sentence-transformers/all-MiniLM-L6-v2
 library_name: peft
+license: apache-2.0
 ---
 
-# Model Card for Model ID
+# Model Card for ddosdub/DualEncoderModernBERT
 
 <!-- Provide a quick summary of what the model is/does. -->
 
-
+This is a binary classification model that combines ModernBERT and SBERT embeddings to detect whether a piece of evidence supports a given claim (evidence detection). This is a deep learning approach underpinned by transformer architecture.
 
 ## Model Details
 
@@ -15,23 +18,24 @@ library_name: peft
 
 <!-- Provide a longer summary of what this model is. -->
 
+This model uses a dual embedding approach that combines contextualized embeddings from ModernBERT-base with sentence embeddings from SBERT (all-MiniLM-L6-v2). The model first processes claim-evidence pairs through both embedding models, then concatenates the embeddings and passes them through a classifier to predict whether the evidence supports the claim.
+
+The model is fine-tuned using QLoRA (Quantized Low-Rank Adaptation) with 4-bit quantization and flash-attention for efficient training and inference.
 
+Text preprocessing includes removing reference tags, normalizing accented characters using unidecode, cleaning up irregular spacing around punctuation, and normalizing whitespace. Data augmentation was applied to the positive class (minority) using synonym replacement to address class imbalance.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** Dhruv Sharma and Tuan Chuong Goh
+- **Model type:** Supervised
+- **Language(s) (NLP):** English
+- **License:** cc-by-4.0
+- **Finetuned from model:** ModernBERT-base and SBERT (all-MiniLM-L6-v2)
 
-### Model Sources [optional]
+### Model Sources
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/chuongg3/NLU-EvidenceDetection
+- **Paper:** https://huggingface.co/answerdotai/ModernBERT-base
 
 ## Uses
 
@@ -41,37 +45,76 @@ library_name: peft
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-[More Information Needed]
+This model can be directly used for evidence detection tasks, where the goal is to determine whether a given piece of evidence supports a specific claim. It processes claim-evidence pairs and outputs a binary classification result.
 
-### Downstream Use [optional]
+### Downstream Use
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-[More Information Needed]
+The model can be integrated into fact-checking systems, academic research tools, or information verification applications. It can also serve as a component in larger natural language understanding pipelines for tasks requiring evidence assessment.
 
 ### Out-of-Scope Use
 
 <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-[More Information Needed]
+This model is not designed to:
+- Process non-English text
+- Handle multi-class classification beyond binary evidence detection
+- Serve as a standalone fact-checker without human oversight
+- Generate text or provide explanations for its decisions
 
 ## Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
+The model uses an optimal threshold of 0.5433 determined through validation data to convert probabilities to binary predictions. The 4-bit quantization may introduce some precision loss compared to full-precision models, although the performance metrics indicate this has minimal impact on model quality. The original dataset had class imbalance which was addressed through data augmentation for the positive class.
 
 ### Recommendations
 
 <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+Users (both direct and downstream) should be aware that:
+- The model works best with properly preprocessed text inputs
+- Performance may vary across different domains or types of claims
+- The model should be used as a decision support tool rather than the sole arbiter of evidence validity
+- Regular evaluation on new data is recommended to monitor potential performance drift
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+Use the code below to get started with the model:
+
+```python
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+from sentence_transformers import SentenceTransformer
+import torch
+
+# Load models
+modernbert_tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
+modernbert_model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base")
+sbert_model = SentenceTransformer("all-MiniLM-L6-v2")
+
+# Load the fine-tuned model
+# Replace with actual path when available
+model = torch.load("path/to/h25471ds-m19364tg-ED")
+
+# Process input
+def predict(claim, evidence):
+    # Preprocess text
+    # ... preprocessing code here ...
+    
+    # Get ModernBERT embeddings
+    inputs = modernbert_tokenizer(claim, evidence, return_tensors="pt")
+    modernbert_output = modernbert_model(**inputs)
+    
+    # Get SBERT embeddings
+    sbert_claim = sbert_model.encode(claim)
+    sbert_evidence = sbert_model.encode(evidence)
+    
+    # Combine embeddings and predict
+    # ... model inference code here ...
+    
+    return prediction
+```
 
 ## Training Details
 
@@ -79,26 +122,41 @@ Use the code below to get started with the model.
 
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-[More Information Needed]
+Training data consisted of claim-evidence pairs for evidence detection tasks. Data augmentation was applied to the positive class (minority) using synonym replacement to address class imbalance.
 
 ### Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-#### Preprocessing [optional]
-
-[More Information Needed]
+#### Preprocessing
 
+The preprocessing pipeline includes:
+1. Removing reference tags like [REF], [REF, REF]
+2. Normalizing accented characters using unidecode
+3. Cleaning up irregular spacing around punctuation
+4. Normalizing whitespace
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+- **Training regime:** 4-bit (nf4) quantization with QLoRA
+- **learning_rate:** 0.0002643238333834569
+- **batch_size:** 64
+- **num_epochs:** 5
+- **weight_decay:** 0.048207625326781293
+- **warmup_ratio:** 0.19552784843595056
+- **gradient_accumulation_steps:** 4
+- **lora_r:** 56
+- **lora_alpha:** 40
+- **lora_dropout:** 0.07644825534662132
+- **classifier_dropout:** 0.2659719581055393
+- **classifier_hidden_size:** 768
+- **max_length:** 8192
+
+#### Speeds, Sizes, Times
+
+- **Model size:** The base ModernBERT model is loaded in 4-bit quantization
+- **SBERT embeddings dimension:** 384
+- **Memory footprint:** Reduced due to 4-bit quantization and parameter-efficient fine-tuning
 
 ## Evaluation
 
@@ -108,95 +166,87 @@ Use the code below to get started with the model.
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+Development set with claim-evidence pairs for evidence detection.
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+The evaluation focused on the model's ability to correctly classify evidence as supporting or not supporting claims across various domains and claim types.
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+The following metrics were used to evaluate model performance:
+- Accuracy: Proportion of correct predictions
+- Precision: Proportion of positive identifications that were actually correct
+- Recall: Proportion of actual positives that were identified correctly
+- F1-Score: Harmonic mean of precision and recall
+- Matthews Correlation Coefficient: Correlation coefficient between observed and predicted binary classifications
 
 ### Results
 
-[More Information Needed]
-
 #### Summary
 
+- **Accuracy:** 0.87377657779278
+- **Macro Precision:** 0.83764094620994
+- **Macro Recall:** 0.86135532021442
+- **Macro F1-Score:** 0.84790707217937
+- **Weighted Precision:** 0.88028808321627
+- **Weighted Recall:** 0.87377657779278
+- **Weighted F1-Score:** 0.87591472842040
+- **Matthews Correlation Coefficient:** 0.69859387983347
 
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
+The model achieved a Macro F1-score of 0.848 (84.8%) and an accuracy of 0.874 (87.4%) on the development set.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. -->
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+- **Hardware Type:** CUDA-compatible GPU with T4 (Turing) architecture or newer
+- **Hours used:** Not specified
+- **Cloud Provider:** Not specified
+- **Compute Region:** Not specified
+- **Carbon Emitted:** Not calculated, but the use of 4-bit quantization and QLoRA significantly reduces the computational requirements compared to full-precision fine-tuning
 
-## Technical Specifications [optional]
+## Technical Specifications
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+The model combines ModernBERT's contextual understanding with SBERT's semantic similarity capabilities. It first extracts the [CLS] token embedding from ModernBERT, then concatenates it with SBERT embeddings before passing through the classification layers.
 
 ### Compute Infrastructure
 
-[More Information Needed]
-
 #### Hardware
 
-[More Information Needed]
+- **RAM:** at least 16 GB
+- **Storage:** at least 2GB
+- **GPU:** CUDA-compatible GPU with T4 (Turing) architecture or newer
+- **Training requirements:** T4 or newer GPU architecture to support flash-attention
+- **Inference requirements:** Can be performed on less powerful GPUs with 4-bit quantization
 
 #### Software
 
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
+- **torch:** 2.6.0+cu126
+- **transformers**
+- **peft:** 0.15.1 (for QLoRA implementation)
+- **bitsandbytes:** (for 4-bit quantization)
+- **flash-attn:** (for efficient attention computation)
+- **sentence-transformers**
+- **sklearn**
+- **numpy**
+- **pandas**
+- **unidecode:** (for text normalization)
+- **re:** (for text cleaning)
 
-**APA:**
+## More Information
 
-[More Information Needed]
+The model combines the strengths of ModernBERT's long context understanding with SBERT's semantic similarity capabilities. The use of QLoRA and 4-bit quantization enables efficient fine-tuning with significantly reduced memory requirements compared to full-precision fine-tuning. Flash-attention provides computational speedups during training and inference on compatible hardware.
 
-## Glossary [optional]
+Hyperparameters were optimized using a systematic search process to find the optimal configuration.
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
+Important references:
+- QLoRA: Efficient Finetuning of Quantized LLMs (2023) - https://arxiv.org/abs/2305.14314
+- Hugging Face 4-bit Transformers with bitsandbytes - https://huggingface.co/blog/4bit-transformers-bitsandbytes
+- PEFT: Parameter-Efficient Fine-Tuning Documentation - https://huggingface.co/docs/peft/en/index
 
 ## Model Card Contact
 
-[More Information Needed]
-### Framework versions
-
-- PEFT 0.15.1
+For inquiries about this model, please contact through the GitHub repository: https://github.com/chuongg3/NLU-EvidenceDetection