Update README.md
Browse files
README.md
CHANGED
|
@@ -128,31 +128,6 @@ Training the model used 2.01kWh of energy emmitting 416.73 grams of CO2
|
|
| 128 |
Stammers M, Gwiggner M, Nouraei R, Metcalf C, Batchelor J. From Rule-Based to DeepSeek R1: A Robust Comparative Evaluation of Fifty Years of Natural Language Processing (NLP) Models To Identify Inflammatory Bowel Disease Cohorts. medRxiv. 2025:2025-07.
|
| 129 |
MedRxiv- [MedRxiv Paper](https://www.medrxiv.org/content/10.1101/2025.07.06.25330961v1)
|
| 130 |
|
| 131 |
-
## Glossary
|
| 132 |
-
|
| 133 |
-
| Term | Description |
|
| 134 |
-
|-------------------------------------|-------------|
|
| 135 |
-
| **Accuracy** | The percentage of results that were correct among all results from the system. Calc: (TP + TN) / (TP + FP + TN + FN). |
|
| 136 |
-
| **Precision (PPV)** | Also called positive predictive value (PPV), it is the percentage of true positive results among all results that the system flagged as positive. Calc: TP / (TP + FP). |
|
| 137 |
-
| **Negative Predictive Value (NPV)** | The percentage of results that were true negative (TN) among all results that the system flagged as negative. Calc: TN / (TN + FN). |
|
| 138 |
-
| **Recall** | Also called sensitivity. The percentage of results flagged positive among all results that should have been obtained. Calc: TP / (TP + FN). |
|
| 139 |
-
| **Specificity** | The percentage of results that were flagged negative among all negative results. Calc: TN / (TN + FP). |
|
| 140 |
-
| **F1-Score** | The harmonic mean of PPV/precision and sensitivity/recall. Calc: 2 × (Precision × Recall) / (Precision + Recall). Moderately useful in the context of class imbalance. |
|
| 141 |
-
| **Matthews’ Correlation Coefficient (MCC)** | A statistical measure used to evaluate the quality of binary classifications. Unlike other metrics, MCC considers all four categories of a confusion matrix. Calc: (TP × TN − FP × FN) / √((TP + FP)(TP + FN)(TN + FP)(TN + FN)). |
|
| 142 |
-
| **Precision / Recall AUC** | Represents the area under the Precision-Recall curve, which plots Precision against Recall at various threshold settings. It is more resistant to class imbalance than alternatives like AUROC. |
|
| 143 |
-
| **Demographic Parity (DP)** | Demographic Parity, also known as Statistical Parity, requires that the probability of a positive prediction is the same across different demographic groups. Calc: DP = P(Ŷ=1∣A=a) = P(Ŷ=1∣A=b). This figure is given as an absolute difference where positive values suggest the more privileged group gains and negative values the reverse. |
|
| 144 |
-
| **Equal Opportunity (EO)** | Equal Opportunity focuses on equalising the true positive rates across groups. Among those who truly belong to the positive class, the model should predict positive outcomes at equal rates across groups. Calc: EO = P(Ŷ=1∣Y=1, A=a) = P(Ŷ=1∣Y=1, A=b). A higher value indicates a bias against the more vulnerable group. |
|
| 145 |
-
| **Disparate Impact (DI)** | Divides the protected group’s positive prediction rate by that of the most-favoured group. If the ratio is below 0.8 or above 1.25, disparate impact is considered present. Calc: DI = P(Ŷ=1∣A=unfavoured) / P(Ŷ=1∣A=favoured). Values outside 0.8–1.25 range suggest bias. |
|
| 146 |
-
| **Execution Time / Energy / CO₂ Emissions** | Measured in minutes and total energy consumption in kilowatt-hours (kWh), which is then converted to CO₂ emissions using a factor of 0.20705 Kg CO₂e per kWh. |
|
| 147 |
-
|
| 148 |
-
## Model Card Authors
|
| 149 |
-
|
| 150 |
-
Matt Stammers - Computational Gastroenterologist
|
| 151 |
-
|
| 152 |
-
## Model Card Contact
|
| 153 |
-
|
| 154 | |
| 155 |
-
|
| 156 |
## Base Training Data for the Model
|
| 157 |
|
| 158 |
#### Training data
|
|
@@ -195,4 +170,37 @@ We sampled each dataset given a weighted probability which configuration is deta
|
|
| 195 |
| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
|
| 196 |
| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
|
| 197 |
| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
|
| 198 |
-
| **Total** | | **1,170,060,424** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
Stammers M, Gwiggner M, Nouraei R, Metcalf C, Batchelor J. From Rule-Based to DeepSeek R1: A Robust Comparative Evaluation of Fifty Years of Natural Language Processing (NLP) Models To Identify Inflammatory Bowel Disease Cohorts. medRxiv. 2025:2025-07.
|
| 129 |
MedRxiv- [MedRxiv Paper](https://www.medrxiv.org/content/10.1101/2025.07.06.25330961v1)
|
| 130 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
## Base Training Data for the Model
|
| 132 |
|
| 133 |
#### Training data
|
|
|
|
| 170 |
| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
|
| 171 |
| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
|
| 172 |
| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
|
| 173 |
+
| **Total** | | **1,170,060,424** |
|
| 174 |
+
|
| 175 |
+
## Glossary
|
| 176 |
+
|
| 177 |
+
| Term | Description |
|
| 178 |
+
|-------------------------------------|-------------|
|
| 179 |
+
| **Accuracy** | The percentage of results that were correct among all results from the system. Calc: (TP + TN) / (TP + FP + TN + FN). |
|
| 180 |
+
| **Precision (PPV)** | Also called positive predictive value (PPV), it is the percentage of true positive results among all results that the system flagged as positive. Calc: TP / (TP + FP). |
|
| 181 |
+
| **Negative Predictive Value (NPV)** | The percentage of results that were true negative (TN) among all results that the system flagged as negative. Calc: TN / (TN + FN). |
|
| 182 |
+
| **Recall** | Also called sensitivity. The percentage of results flagged positive among all results that should have been obtained. Calc: TP / (TP + FN). |
|
| 183 |
+
| **Specificity** | The percentage of results that were flagged negative among all negative results. Calc: TN / (TN + FP). |
|
| 184 |
+
| **F1-Score** | The harmonic mean of PPV/precision and sensitivity/recall. Calc: 2 × (Precision × Recall) / (Precision + Recall). Moderately useful in the context of class imbalance. |
|
| 185 |
+
| **Matthews’ Correlation Coefficient (MCC)** | A statistical measure used to evaluate the quality of binary classifications. Unlike other metrics, MCC considers all four categories of a confusion matrix. Calc: (TP × TN − FP × FN) / √((TP + FP)(TP + FN)(TN + FP)(TN + FN)). |
|
| 186 |
+
| **Precision / Recall AUC** | Represents the area under the Precision-Recall curve, which plots Precision against Recall at various threshold settings. It is more resistant to class imbalance than alternatives like AUROC. |
|
| 187 |
+
| **Demographic Parity (DP)** | Demographic Parity, also known as Statistical Parity, requires that the probability of a positive prediction is the same across different demographic groups. Calc: DP = P(Ŷ=1∣A=a) = P(Ŷ=1∣A=b). This figure is given as an absolute difference where positive values suggest the more privileged group gains and negative values the reverse. |
|
| 188 |
+
| **Equal Opportunity (EO)** | Equal Opportunity focuses on equalising the true positive rates across groups. Among those who truly belong to the positive class, the model should predict positive outcomes at equal rates across groups. Calc: EO = P(Ŷ=1∣Y=1, A=a) = P(Ŷ=1∣Y=1, A=b). A higher value indicates a bias against the more vulnerable group. |
|
| 189 |
+
| **Disparate Impact (DI)** | Divides the protected group’s positive prediction rate by that of the most-favoured group. If the ratio is below 0.8 or above 1.25, disparate impact is considered present. Calc: DI = P(Ŷ=1∣A=unfavoured) / P(Ŷ=1∣A=favoured). Values outside 0.8–1.25 range suggest bias. |
|
| 190 |
+
| **Execution Time / Energy / CO₂ Emissions** | Measured in minutes and total energy consumption in kilowatt-hours (kWh), which is then converted to CO₂ emissions using a factor of 0.20705 Kg CO₂e per kWh. |
|
| 191 |
+
|
| 192 |
+
## Model Card Authors
|
| 193 |
+
|
| 194 |
+
Matt Stammers - Computational Gastroenterologist
|
| 195 |
+
|
| 196 |
+
## Model Card Contact
|
| 197 |
+
|
| 198 | |
| 199 |
+
|
| 200 |
+
## Legal
|
| 201 |
+
|
| 202 |
+
1. No guarantee is given of model performance in any production capacity whatsoever.
|
| 203 |
+
2. These models should be used in full accordance with the EU AI Act - Regulation 2024/1689.
|
| 204 |
+
3. These models are not CE marked medical devices and are suitable at this point only for research and development / experimentation at users own discretion.
|
| 205 |
+
4. They can be improved but any improvements should be published openly and shared openly with the community.
|
| 206 |
+
5. UHSFT and the author own the copyright and are choosing to share them freely under a CC BY-NC 4.0 Licence for the benefit of the wider research community but not for commercial organisations who are breaking copyright law and infringing upon NHS intellectual property if they try to sell/market these models for profit.
|