Taja Kuzman
commited on
Update README.md
Browse files
README.md
CHANGED
|
@@ -144,11 +144,15 @@ results = classifier(texts)
|
|
| 144 |
for result in results:
|
| 145 |
print(result)
|
| 146 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 147 |
```
|
| 148 |
|
| 149 |
## IPTC Media Topic categories
|
| 150 |
|
| 151 |
-
The classifier uses the top-level of the IPTC Media Topic NewsCodes schema, consisting of 17 labels.
|
| 152 |
|
| 153 |
List of labels:
|
| 154 |
```
|
|
@@ -166,7 +170,7 @@ labels_map={0: 'education', 1: 'human interest', 2: 'society', 3: 'sport', 4: 'c
|
|
| 166 |
Description of labels:
|
| 167 |
|
| 168 |
The descriptions of the labels are based on the descriptions provided in the [IPTC Media Topic NewsCodes schema](https://www.iptc.org/std/NewsCodes/treeview/mediatopic/mediatopic-en-GB.html)
|
| 169 |
-
and enriched with information which specific subtopics belong to the top-level topics, based on the IPTC Media Topic hierarchy.
|
| 170 |
|
| 171 |
| Label | Description |
|
| 172 |
|:------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
@@ -218,8 +222,8 @@ Label distribution in the training dataset:
|
|
| 218 |
|
| 219 |
## Performance
|
| 220 |
|
| 221 |
-
The model was evaluated on a manually-annotated test set in four languages (Croatian, Slovenian, Catalan and Greek), consisting of 1
|
| 222 |
-
The test set contains
|
| 223 |
|
| 224 |
The model was shown to achieve accuracy of 0.78 and macro-F1 scores of 0.72. The results for the entire test set and per language:
|
| 225 |
|
|
@@ -235,7 +239,7 @@ The model was shown to achieve accuracy of 0.78 and macro-F1 scores of 0.72. The
|
|
| 235 |
For downstream tasks, **we advise you to use only labels that were predicted with confidence score
|
| 236 |
higher than 0.90 which further improves the performance**.
|
| 237 |
|
| 238 |
-
When we remove instances
|
| 239 |
|
| 240 |
| Language | Accuracy | Macro-F1 | No. of instances |
|
| 241 |
|:-------|-----------:|-----------:|-----------:|
|
|
|
|
| 144 |
for result in results:
|
| 145 |
print(result)
|
| 146 |
|
| 147 |
+
## Output
|
| 148 |
+
## {'label': 'sport', 'score': 0.9985264539718628}
|
| 149 |
+
## {'label': 'disaster, accident and emergency incident', 'score': 0.9957459568977356}
|
| 150 |
+
|
| 151 |
```
|
| 152 |
|
| 153 |
## IPTC Media Topic categories
|
| 154 |
|
| 155 |
+
The classifier uses the top-level of the [IPTC Media Topic NewsCodes](https://iptc.org/std/NewsCodes/guidelines/#_what_are_the_iptc_newscodes) schema, consisting of 17 labels.
|
| 156 |
|
| 157 |
List of labels:
|
| 158 |
```
|
|
|
|
| 170 |
Description of labels:
|
| 171 |
|
| 172 |
The descriptions of the labels are based on the descriptions provided in the [IPTC Media Topic NewsCodes schema](https://www.iptc.org/std/NewsCodes/treeview/mediatopic/mediatopic-en-GB.html)
|
| 173 |
+
and enriched with information which specific subtopics belong to the top-level topics, based on the IPTC Media Topic label hierarchy.
|
| 174 |
|
| 175 |
| Label | Description |
|
| 176 |
|:------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
|
|
| 222 |
|
| 223 |
## Performance
|
| 224 |
|
| 225 |
+
The model was evaluated on a manually-annotated test set in four languages (Croatian, Slovenian, Catalan and Greek), consisting of 1,130 instances.
|
| 226 |
+
The test set contains similar amounts of texts from the four languages and is more or less balanced across labels.
|
| 227 |
|
| 228 |
The model was shown to achieve accuracy of 0.78 and macro-F1 scores of 0.72. The results for the entire test set and per language:
|
| 229 |
|
|
|
|
| 239 |
For downstream tasks, **we advise you to use only labels that were predicted with confidence score
|
| 240 |
higher than 0.90 which further improves the performance**.
|
| 241 |
|
| 242 |
+
When we remove instances predicted with lower confidence (229 instances - 20%), the scores are the following:
|
| 243 |
|
| 244 |
| Language | Accuracy | Macro-F1 | No. of instances |
|
| 245 |
|:-------|-----------:|-----------:|-----------:|
|