Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,259 @@
+---
+library_name: transformers
+language:
+  - multilingual
+  - af
+  - am
+  - ar
+  - as
+  - az
+  - be
+  - bg
+  - bm
+  - bn
+  - br
+  - bs
+  - ca
+  - cs
+  - cy
+  - da
+  - de
+  - el
+  - en
+  - eo
+  - es
+  - et
+  - eu
+  - fa
+  - ff
+  - fi
+  - fr
+  - fy
+  - ga
+  - gd
+  - gl
+  - gn
+  - gu
+  - ha
+  - he
+  - hi
+  - hr
+  - ht
+  - hu
+  - hy
+  - id
+  - ig
+  - is
+  - it
+  - ja
+  - jv
+  - ka
+  - kg
+  - kk
+  - km
+  - kn
+  - ko
+  - ku
+  - ky
+  - la
+  - lg
+  - ln
+  - lo
+  - lt
+  - lv
+  - mg
+  - mk
+  - ml
+  - mn
+  - mr
+  - ms
+  - my
+  - ne
+  - nl
+  - no
+  - om
+  - or
+  - pa
+  - pl
+  - ps
+  - pt
+  - qu
+  - ro
+  - ru
+  - sa
+  - sd
+  - si
+  - sk
+  - sl
+  - so
+  - sq
+  - sr
+  - ss
+  - su
+  - sv
+  - sw
+  - ta
+  - te
+  - th
+  - ti
+  - tl
+  - tn
+  - tr
+  - uk
+  - ur
+  - uz
+  - vi
+  - wo
+  - xh
+  - yo
+  - zh
+
+license: agpl-3.0
+tags:
+  - v1.0.0
+  - entity-disambiguation
+  - named-entity-linking
+  - entity-linking
+  - text2text-generation
+---
+
+# Model Card for `impresso-project/nel-mgenre-multilingual`
+
+The **Impresso multilingual named entity linking (NEL)** model is based on **mGENRE** (multilingual Generative ENtity
+REtrieval) proposed by [De Cao et al](https://arxiv.org/abs/2103.12528), a sequence-to-sequence architecture for entity
+disambiguation based on [mBART](https://arxiv.org/abs/2001.08210). It uses **constrained generation** to output entity
+names mapped to Wikidata/QIDs.
+
+This model was adapted for historical texts and fine-tuned on
+the [HIPE-2022 dataset](https://github.com/hipe-eval/HIPE-2022-data), which includes a variety of historical document
+types and languages.
+
+## Model Details
+
+### Model Description
+
+- **Developed by:** EPFL from the [Impresso team](https://impresso-project.ch). The project is an interdisciplinary
+  project focused on historical media analysis across languages, time, and modalities. Funded by the Swiss National
+  Science
+  Foundation ([CRSII5_173719](http://p3.snf.ch/project-173719), [CRSII5_213585](https://data.snf.ch/grants/grant/213585))
+  and the Luxembourg National Research Fund (grant No. 17498891).
+- **Model type:** mBART-based sequence-to-sequence model with constrained beam search for named entity linking
+- **Languages:** Multilingual (100+ languages, optimized for French, German, and English)
+- **License:** [AGPL v3+](https://github.com/impresso/impresso-pyindexation/blob/master/LICENSE)
+- **Finetuned from:** [`facebook/mgenre-wiki`](https://huggingface.co/facebook/mgenre-wiki)
+
+### Model Architecture
+
+- **Architecture:** mBART-based seq2seq with constrained beam search
+
+## Training Details
+
+### Training Data
+
+The model was trained on the following datasets:
+
+| Dataset alias | README                                     | Document type          | Languages      | Suitable for               | Project                                                     | License                                                                                                                                                 |
+|---------------|--------------------------------------------|------------------------|----------------|----------------------------|-------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
+| ajmc          | [link](documentation/README-ajmc.md)       | classical commentaries | de, fr, en     | NERC-Coarse, NERC-Fine, EL | [AjMC](https://mromanello.github.io/ajax-multi-commentary/) | [![License: CC BY 4.0](https://img.shields.io/badge/License-CC_BY_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)                     |
+| hipe2020      | [link](documentation/README-hipe2020.md)   | historical newspapers  | de, fr, en     | NERC-Coarse, NERC-Fine, EL | [CLEF-HIPE-2020](https://impresso.github.io/CLEF-HIPE-2020) | [![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC_BY--NC--SA_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/) |
+| topres19th    | [link](documentation/README-topres19th.md) | historical newspapers  | en             | NERC-Coarse, EL            | [Living with Machines](https://livingwithmachines.ac.uk/)   | [![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC_BY--NC--SA_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/) |
+| newseye       | [link](documentation/README-newseye.md)    | historical newspapers  | de, fi, fr, sv | NERC-Coarse, NERC-Fine, EL | [NewsEye](https://www.newseye.eu/)                          | [![License: CC BY 4.0](https://img.shields.io/badge/License-CC_BY_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)                     |
+| sonar         | [link](documentation/README-sonar.md)      | historical newspapers  | de             | NERC-Coarse, EL            | [SoNAR](https://sonar.fh-potsdam.de/)                       | [![License: CC BY 4.0](https://img.shields.io/badge/License-CC_BY_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)                     |
+
+## How to Use
+
+```python
+from transformers import AutoTokenizer, pipeline
+
+NEL_MODEL_NAME = "impresso-project/nel-mgenre-multilingual"
+nel_tokenizer = AutoTokenizer.from_pretrained(NEL_MODEL_NAME)
+
+nel_pipeline = pipeline("generic-nel", model=NEL_MODEL_NAME,
+                        tokenizer=nel_tokenizer,
+                        trust_remote_code=True,
+                        device='cpu')
+
+sentence = "Le 0ctobre 1894, [START] Dreyfvs [END] est arrêté à Paris, accusé d'espionnage pour l'Allemagne — un événement qui déch1ra la société fr4nçaise pendant des années."
+print(nel_pipeline(sentence))
+```
+
+### Output Format
+
+```python
+[
+    {
+        'surface': 'Dreyfvs',
+        'wkd_id': 'Q171826',
+        'wkpedia_pagename': 'Alfred Dreyfus',
+        'wkpedia_url': 'https://fr.wikipedia.org/wiki/Alfred_Dreyfus',
+        'type': 'UNK',
+        'confidence_nel': 99.98,
+        'lOffset': 24,
+        'rOffset': 33}]
+```
+
+----
+
+### Batched predictions
+
+```python
+from transformers import AutoTokenizer, pipeline
+
+NEL_MODEL_NAME = "impresso-project/nel-mgenre-multilingual"
+
+nel_tokenizer = AutoTokenizer.from_pretrained(NEL_MODEL_NAME)
+
+nel_pipeline = pipeline(
+    "generic-nel",
+    model=NEL_MODEL_NAME,
+    tokenizer=nel_tokenizer,
+    trust_remote_code=True,
+    device="cpu",
+)
+
+sentences = [
+    "Le 0ctobre 1894, [START] Dreyfvs [END] est arrêté à Paris.",
+    "En 1912, [START] Giollitti [END] était encore au pouvoir en Italie.",
+    "La ville de [START] Lpzbourg [END] est importante dans la région."
+]
+
+results = nel_pipeline(sentences)
+
+from pprint import pprint
+pprint(results)
+```
+
+The type of the entity is `UNK` because the model was not trained on the entity type. The `confidence_nel` score
+indicates the model's confidence in the prediction.
+
+## Use Cases
+
+- Entity disambiguation in noisy OCR settings
+- Linking historical names to modern Wikidata entities
+- Assisting downstream event extraction and biography generation from historical archives
+
+> 🐱**Easter Egg**:  
+> While the pipeline’s main goal is to link surface forms to QIDs and Wikipedia pages,  
+> it sometimes slips in an extra surprise: a coarse Wikidata type label (`pers`, `org`, or `loc`).  
+> Not strictly required — but a handy little bonus when it shows up!
+
+## Limitations
+
+- Sensitive to tokenisation and malformed spans
+- Accuracy degrades on non-Wikidata entities or in highly ambiguous contexts
+- Focused on historical entity mentions — performance may vary on modern texts
+
+## Environmental Impact
+
+- **Hardware:** 1x A100 (80GB) for finetuning
+- **Training time:** ~12 hours
+- **Estimated CO₂ Emissions:** ~2.3 kg CO₂eq
+
+## Contact
+
+- Website: [https://impresso-project.ch](https://impresso-project.ch)
+
+<p align="center">
+  <img src="https://github.com/impresso/impresso.github.io/blob/master/assets/images/3x1--Yellow-Impresso-Black-on-White--transparent.png?raw=true" width="300" alt="Impresso Logo"/>
+</p>
+

config.json

@@ -0,0 +1,44 @@
+{
+  "_name_or_path": "facebook/mgenre-wiki",
+  "activation_dropout": 0.0,
+  "activation_function": "gelu",
+  "architectures": [
+    "MBartForConditionalGeneration"
+  ],
+  "custom_pipelines": {
+    "generic-nel": {
+      "impl": "generic_nel.NelPipeline",
+      "pt": [
+        "MBartForConditionalGeneration"
+      ],
+      "tf": []
+    }
+  },
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.0,
+  "d_model": 1024,
+  "decoder_attention_heads": 16,
+  "decoder_ffn_dim": 4096,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 12,
+  "decoder_start_token_id": 2,
+  "dropout": 0.1,
+  "encoder_attention_heads": 16,
+  "encoder_ffn_dim": 4096,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 12,
+  "eos_token_id": 2,
+  "forced_eos_token_id": 2,
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "max_position_embeddings": 1024,
+  "model_type": "mbart",
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "scale_embedding": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.31.0",
+  "use_cache": true,
+  "vocab_size": 256001
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "bos_token_id": 0,
+  "decoder_start_token_id": 2,
+  "eos_token_id": 2,
+  "forced_eos_token_id": 2,
+  "pad_token_id": 1,
+  "transformers_version": "4.46.0.dev0"
+}

generic_nel.py

@@ -0,0 +1,590 @@
+from transformers import Pipeline
+import requests
+import torch
+import time
+import os
+import random
+import logging
+from filelock import FileLock
+import requests_cache
+from requests.adapters import HTTPAdapter
+from urllib3.util.retry import Retry
+
+logger = logging.getLogger("nel_wikidata")
+
+NEL_MODEL = "nel-mgenre-multilingual"
+
+# Required by Wikimedia: identify your app + contact
+HEADERS = {"User-Agent": "NEL-HF-Space/0.1 (contact: emanuela.boros@epfl.ch)"}
+
+
+def safe_get(url, params, retries=3, backoff=2):
+    """
+    Perform a safe HTTP GET request with retry logic.
+
+    This function wraps `requests.get` and automatically retries the request if it fails
+    due to common transient errors such as HTTP 403 (forbidden, often due to missing/blocked
+    User-Agent headers) or HTTP 429 (rate-limited). Retries use exponential backoff.
+
+    Args:
+        url (str): The target URL for the request.
+        params (dict): Query parameters to send with the GET request.
+        retries (int, optional): Maximum number of attempts before giving up. Default is 3.
+        backoff (int, optional): Base for exponential backoff in seconds. Default is 2.
+            - Example: with retries=3 and backoff=2, wait times will be 1s, 2s, 4s.
+
+    Returns:
+        requests.Response or None:
+            - A `Response` object if the request eventually succeeds (status=200).
+            - `None` if all retries fail or an unrecoverable error occurs.
+
+    Notes:
+        - Adds `HEADERS` to all requests, which should include a descriptive User-Agent
+          as required by Wikimedia APIs.
+        - Raises no exceptions; instead prints errors and returns None if the request
+          cannot be completed successfully.
+    """
+    for attempt in range(retries):
+        try:
+            resp = requests.get(url, params=params, headers=HEADERS, timeout=10)
+            if resp.status_code == 200:
+                return resp
+            elif resp.status_code in (403, 429):  # forbidden or rate limited
+                wait = backoff**attempt
+                print(f"Retrying {url} in {wait}s (status={resp.status_code})")
+                time.sleep(wait)
+                continue
+            else:
+                resp.raise_for_status()
+        except Exception as e:
+            print(f"Request error: {e}")
+            time.sleep(backoff**attempt)
+    return None
+
+
+# ---- Persistent cache (shared on disk across processes) ----
+# 30 days is fine for QID/title/type lookups
+requests_cache.install_cache(
+    cache_name=os.getenv("WIKIMEDIA_CACHE", "wikimedia_cache"),
+    backend="sqlite",
+    expire_after=int(os.getenv("WIKIMEDIA_CACHE_TTL", str(60 * 60 * 24 * 30))),
+    allowable_methods=("GET",),
+)
+
+# ---- One Session per process (connection pooling + retries) ----
+_session = requests.Session()
+_session.headers.update(HEADERS)
+
+_retry = Retry(
+    total=int(os.getenv("WIKIMEDIA_RETRIES", "5")),
+    connect=int(os.getenv("WIKIMEDIA_RETRIES", "5")),
+    read=int(os.getenv("WIKIMEDIA_RETRIES", "5")),
+    backoff_factor=float(os.getenv("WIKIMEDIA_BACKOFF", "1.5")),
+    status_forcelist=(429, 500, 502, 503, 504),
+    allowed_methods=frozenset(["GET"]),
+    respect_retry_after_header=False,  # <-- IMPORTANT
+    raise_on_status=False,
+)
+
+_adapter = HTTPAdapter(max_retries=_retry, pool_connections=50, pool_maxsize=50)
+_session.mount("https://", _adapter)
+_session.mount("http://", _adapter)
+
+# ---- Cross-process rate limit (global QPS across all workers on this machine) ----
+# Choose a conservative QPS; Wikimedia is happier with low QPS.
+WIKIMEDIA_QPS = float(os.getenv("WIKIMEDIA_QPS", "3.0"))  # total across ALL processes
+_MIN_INTERVAL = 1.0 / max(WIKIMEDIA_QPS, 0.1)
+
+_LOCK = FileLock(os.getenv("WIKIMEDIA_RATELOCK", "/tmp/wikimedia_rate.lock"))
+_TS_FILE = os.getenv("WIKIMEDIA_RATETS", "/tmp/wikimedia_rate.ts")
+
+
+def _throttle_global():
+    """Global rate limiter shared across processes via file lock + timestamp."""
+    with _LOCK:
+        now = time.time()
+        last = 0.0
+        try:
+            with open(_TS_FILE, "r") as f:
+                last = float(f.read().strip() or "0")
+        except FileNotFoundError:
+            last = 0.0
+
+        wait = (last + _MIN_INTERVAL) - now
+        if wait > 0:
+            time.sleep(wait)
+
+        with open(_TS_FILE, "w") as f:
+            f.write(str(time.time()))
+
+
+def safe_get_old(url, params, timeout=10):
+    """
+    Cached + globally rate-limited + retrying GET.
+    Returns Response or None.
+    """
+    _throttle_global()
+    try:
+        resp = _session.get(url, params=params, timeout=timeout)
+
+        # requests-cache sets this attribute; useful to log
+        from_cache = getattr(resp, "from_cache", False)
+
+        if resp.status_code == 200:
+            if from_cache:
+                logger.debug(f"[WIKI] 200 (cache) {url} params={_short(params)}")
+            return resp
+
+        if resp.status_code == 429:
+            # Even with Retry-After handling, sometimes you still get here.
+            ra = resp.headers.get("Retry-After")
+            sleep = float(ra) if (ra and ra.isdigit()) else (1.0 + random.random())
+            logger.warning(
+                f"[WIKI] 429 rate limited. retry-after={ra} sleeping={sleep:.2f}s url={url}"
+            )
+            time.sleep(sleep)
+            return resp  # caller may treat as failure; Retry already happened.
+
+        logger.warning(
+            f"[WIKI] HTTP {resp.status_code} url={url} params={_short(params)}"
+        )
+        return resp
+    except Exception as e:
+        logger.exception(f"[WIKI] Request error: {e} url={url}")
+        return None
+
+
+def _short(params: dict, maxlen: int = 120) -> str:
+    s = str(params)
+    return s if len(s) <= maxlen else s[:maxlen] + "..."
+
+
+def get_wikipedia_page_props(input_str: str):
+    """
+    Retrieve the Wikidata QID for a given Wikipedia page name.
+
+    This function attempts to resolve a text prediction (e.g. `"Paris >> fr"`)
+    into a Wikidata identifier (`Q90`) and its language code.
+
+    Resolution happens in two steps:
+      1. Query the specified language Wikipedia API (`prop=pageprops`) to check if
+         the page is associated with a `wikibase_item` (QID).
+      2. If that fails, fall back to the Wikidata `wbsearchentities` API, which
+         searches for entities matching the page name in the given language.
+
+    Args:
+        input_str (str): A string in the form `"page_name >> language"`.
+            - Example: `"Paris >> fr"` → page_name="Paris", language="fr".
+            - If no `>>` separator is found, defaults to `"en"` as the language.
+
+    Returns:
+        tuple(str, str):
+            - QID of the page (e.g., `"Q90"`) if found, otherwise `"NIL"`.
+            - Language code (e.g., `"fr"`, `"en"`).
+
+    Notes:
+        - Uses `safe_get()` internally for API calls, with retry logic and
+          a proper User-Agent header (required by Wikimedia).
+        - Falls back gracefully to `"NIL"` if both Wikipedia and Wikidata lookups fail.
+        - This function does not fetch human-readable titles or URLs;
+          use `get_wikipedia_title()` for that.
+    """
+
+    if ">>" in input_str:
+        page_name, language = [x.strip() for x in input_str.split(">>", 1)]
+    else:
+        page_name, language = input_str.strip(), "en"
+
+    qid = "NIL"
+
+    # --- Step 1: Try Wikipedia API ---
+    wikipedia_url = f"https://{language}.wikipedia.org/w/api.php"
+    wikipedia_params = {
+        "action": "query",
+        "prop": "pageprops",
+        "format": "json",
+        "titles": page_name,
+    }
+
+    resp = safe_get(wikipedia_url, wikipedia_params)
+    if resp:
+        data = resp.json()
+        if "pages" in data.get("query", {}):
+            page_id = list(data["query"]["pages"].keys())[0]
+            page_props = data["query"]["pages"][page_id].get("pageprops", {})
+            if "wikibase_item" in page_props:
+                return page_props["wikibase_item"], language
+
+    # --- Step 2: Fallback to Wikidata search ---
+    wd_url = "https://www.wikidata.org/w/api.php"
+    wd_params = {
+        "action": "wbsearchentities",
+        "search": page_name,
+        "language": language,
+        "format": "json",
+    }
+
+    resp = safe_get(wd_url, wd_params)
+    if resp:
+        data = resp.json()
+        if data.get("search"):
+            first_hit = data["search"][0]
+            return first_hit["id"], language
+
+    return qid, language
+
+
+def resolve_type_from_wikidata(p31_qids, visited=None, depth=0, max_depth=3):
+    """
+    Resolve a Wikidata entity's high-level type ("pers", "org", "loc")
+    by checking its 'instance of' (P31) values and, if necessary,
+    recursively climbing 'subclass of' (P279) relations.
+
+    Args:
+        p31_qids (list[str]): List of QIDs from 'instance of' (P31) claims.
+        visited (set, optional): Set of QIDs already checked (to avoid cycles).
+        depth (int, optional): Current recursion depth (default=0).
+        max_depth (int, optional): Maximum recursion depth for P279 traversal.
+
+    Returns:
+        str: One of {"pers", "org", "loc", "UNK"}.
+             "UNK" is returned if no mapping is found within max_depth.
+
+    Notes:
+        - Anchors:
+            Q5 → person
+            Any in `qid_to_organization_label_keys` or TYPE_MAP=organization → org
+            Any in `qid_to_location_label_keys` or TYPE_MAP=location → loc
+        - Subclass traversal:
+            If P31 is not directly mapped, checks P279 (subclass of)
+            recursively up to `max_depth`.
+    """
+
+    if visited is None:
+        visited = set()
+
+    for q in p31_qids:
+        if q in visited:
+            continue
+        visited.add(q)
+
+        # direct anchors
+        if q == "Q5":
+            return "pers"
+        if (
+            q in qid_to_organization_label_keys
+            or q in TYPE_MAP
+            and TYPE_MAP[q] == "organization"
+        ):
+            return "org"
+        if (
+            q in qid_to_location_label_keys
+            or q in TYPE_MAP
+            and TYPE_MAP[q] == "location"
+        ):
+            return "loc"
+
+        # subclass traversal
+        if depth < max_depth:
+            wd_url = "https://www.wikidata.org/w/api.php"
+            params = {
+                "action": "wbgetentities",
+                "format": "json",
+                "ids": q,
+                "props": "claims",
+            }
+            resp = safe_get(wd_url, params)
+            if resp:
+                data = resp.json()
+                entity = data["entities"].get(q, {})
+                claims = entity.get("claims", {})
+                if "P279" in claims:  # subclass of
+                    parents = [
+                        c["mainsnak"]["datavalue"]["value"]["id"]
+                        for c in claims["P279"]
+                        if "datavalue" in c["mainsnak"]
+                    ]
+                    resolved = resolve_type_from_wikidata(
+                        parents, visited, depth + 1, max_depth
+                    )
+                    if resolved != "UNK":
+                        return resolved
+
+    return "UNK"
+
+
+def get_wikipedia_title(qid, language="en"):
+    """
+    Retrieve a Wikipedia title, URL, and a mapped coarse entity type from a Wikidata QID.
+
+    This function queries Wikidata to fetch:
+      - The title of the corresponding Wikipedia article in the given language.
+      - The article URL.
+      - A coarse-grained entity type label ("pers", "org", "loc", or "UNK"),
+        derived from the entity's "instance of" (P31) and, if necessary,
+        recursively from its "subclass of" (P279) hierarchy.
+
+    Args:
+        qid (str): The Wikidata identifier (e.g., "Q90" for Paris).
+                   If "NIL" or empty, returns ("NIL", "None", "UNK").
+        language (str, optional): ISO language code for the Wikipedia site
+                   (default is "en"). Example: "fr" for French Wikipedia.
+
+    Returns:
+        tuple:
+            - title (str): The Wikipedia page title if available, otherwise "NIL".
+            - wiki_url (str): The Wikipedia page URL if available, otherwise "None".
+            - type_label (str): A mapped entity type:
+                * "pers" → person (Q5, or subclass thereof)
+                * "org" → organization (Q43229, Q95074, etc. or subclass thereof)
+                * "loc" → location (Q515, Q6256, Q484170, etc. or subclass thereof)
+                * "UNK" → unknown if not resolved within recursion depth
+
+    Notes:
+        - Uses `safe_get()` internally for robust requests with retries and
+          a proper User-Agent header (required by Wikimedia).
+        - Calls `resolve_type_from_wikidata()` to map P31/P279 values into
+          coarse-grained types.
+        - Limits subclass-of (P279) traversal to `max_depth` defined in
+          `resolve_type_from_wikidata` (default=3).
+        - Returns gracefully with ("NIL", "None", "UNK") if the QID is invalid
+          or Wikidata returns no useful data.
+    """
+
+    if not qid or qid == "NIL":
+        return "NIL", "None", "UNK"
+
+    url = "https://www.wikidata.org/w/api.php"
+    params = {
+        "action": "wbgetentities",
+        "format": "json",
+        "ids": qid,
+        "props": "sitelinks/urls|claims",
+        "sitefilter": f"{language}wiki",
+    }
+
+    resp = safe_get(url, params)
+    if not resp:
+        return "NIL", "None", "UNK"
+
+    try:
+        data = resp.json()
+        entity = data["entities"][qid]
+
+        # --- title + url ---
+        sitelinks = entity.get("sitelinks", {})
+        if f"{language}wiki" in sitelinks:
+            title = sitelinks[f"{language}wiki"]["title"]
+            wiki_url = sitelinks[f"{language}wiki"]["url"]
+        else:
+            title, wiki_url = "NIL", "None"
+
+        # --- detect type from P31 ---
+        claims = entity.get("claims", {})
+
+        if "P31" in claims:
+            p31_qids = [
+                c["mainsnak"]["datavalue"]["value"]["id"]
+                for c in claims["P31"]
+                if "datavalue" in c["mainsnak"]
+            ]
+            type_label = resolve_type_from_wikidata(p31_qids)
+        else:
+            type_label = "UNK"
+
+        return title, wiki_url, type_label
+
+    except Exception as e:
+        print(f"Error parsing Wikidata for {qid}: {e}")
+        return "NIL", "None", "UNK"
+
+
+class NelPipeline(Pipeline):
+    """
+    Hugging Face Pipeline for NEL that supports:
+      - Single input:  text with [START] ... [END]
+      - Batch input:   list of texts, each with one mention
+    """
+
+    def _sanitize_parameters(self, **kwargs):
+        preprocess_kwargs = {}
+        if "text" in kwargs:
+            preprocess_kwargs["text"] = kwargs["text"]
+        return preprocess_kwargs, {}, {}
+
+    def preprocess(self, text, **kwargs):
+        """
+        Accepts:
+           text  → string with [START]...[END]
+           text  → list[str] (batched)
+        Returns:
+           list of dict items (one per mention)
+        """
+        if isinstance(text, str):
+            text_list = [text]
+        else:
+            text_list = text
+
+        batch = []
+        for t in text_list:
+            start_token = "[START]"
+            end_token = "[END]"
+
+            if start_token in t and end_token in t:
+                start_idx = t.index(start_token) + len(start_token)
+                end_idx = t.index(end_token)
+                enclosed = t[start_idx:end_idx].strip()
+                lOffset = start_idx
+                rOffset = end_idx
+            else:
+                enclosed, lOffset, rOffset = None, None, None
+
+            batch.append(
+                {"text": t, "surface": enclosed, "lOffset": lOffset, "rOffset": rOffset}
+            )
+
+        return batch
+
+    def _forward(self, batch):
+        """
+        batch is a list of entries from preprocess()
+        We batch all texts into one generate() call.
+        """
+        texts = [b["text"] for b in batch]
+
+        enc = self.tokenizer(
+            texts, return_tensors="pt", padding=True, truncation=True
+        ).to(self.device)
+
+        outputs = self.model.generate(
+            **enc,
+            num_beams=1,
+            num_return_sequences=1,
+            max_new_tokens=30,
+            return_dict_in_generate=True,
+            output_scores=True,
+        )
+
+        # decode predictions
+        decoded = self.tokenizer.batch_decode(
+            outputs.sequences, skip_special_tokens=True
+        )
+
+        # compute per-sequence confidence
+        transition_scores = self.model.compute_transition_scores(
+            outputs.sequences, outputs.scores, normalize_logits=True
+        )
+
+        confidences = []
+        for i in range(len(decoded)):
+            log_prob = sum(transition_scores[i])
+            prob = torch.exp(log_prob).cpu().numpy()
+            confidences.append(float(prob))
+
+        # attach outputs to batch
+        for i, item in enumerate(batch):
+            item["wikipedia_prediction"] = decoded[i]
+            item["confidence"] = confidences[i]
+
+        return batch
+
+    def postprocess(self, batch, **kwargs):
+        """
+        N items → N linked entities
+        """
+
+        results = []
+        for item in batch:
+            pred = item["wikipedia_prediction"]
+            surface = item["surface"]
+            lOffset = item["lOffset"]
+            rOffset = item["rOffset"]
+
+            # Wikidata resolution
+            qid, lang = get_wikipedia_page_props(pred)
+            title, url, ent_type = get_wikipedia_title(qid, language=lang)
+
+            # normalize confidence into [0,1]
+            conf = item["confidence"]
+            if conf > 1:
+                conf = conf / 100
+            conf = round(conf, 4)
+
+            results.append(
+                {
+                    "surface": surface,
+                    "wkd_id": qid,
+                    "wkpedia_pagename": title,
+                    "wkpedia_url": url,
+                    "type": ent_type,
+                    "confidence_nel": conf,
+                    # "lOffset": lOffset,
+                    # "rOffset": rOffset,
+                }
+            )
+
+        return results
+
+
+qid_to_location_label = {
+    "Q82794": "geographical region",
+    "Q6256": "country",
+    "Q3624078": "sovereign state",
+    "Q35657": "U.S. state",
+    "Q515": "city",
+    "Q532": "village",
+    "Q3957": "town",
+    "Q2983893": "municipality",
+    "Q486972": "human settlement",
+    "Q23442": "island",
+    "Q8502": "mountain",
+    "Q23397": "lake",
+    "Q4022": "river",
+    "Q123705": "peninsula",
+    "Q13221722": "bay",
+    "Q165": "sea",
+    "Q124714": "ocean",
+    "Q42124": "continent",
+    "Q23413": "territory",
+    "Q1048835": "province",
+    "Q107390": "county",
+    "Q13220204": "district",
+    "Q13218630": "subdistrict",
+    "Q618123": "location",  # geographical object
+    "Q2221906": "location",  # geographical location
+    "Q17334923": "location",  # geographical region
+}
+TYPE_MAP = {
+    "Q5": "person",  # human
+    "Q43229": "organization",  # organization
+    "Q95074": "organization",  # company
+    "Q783794": "organization",  # non-profit
+    "Q618123": "location",  # geographical object
+    "Q2221906": "location",  # geographical location
+    "Q17334923": "location",  # geographical region
+    "Q6256": "location",  # country
+    "Q515": "location",  # city
+}
+
+qid_to_location_label_keys = set(qid_to_location_label.keys())
+qid_to_organization_label = {
+    "Q43229": "organization",  # organization
+    "Q95074": "company",  # company
+    "Q783794": "non-profit",  # non-profit organization
+    "Q891723": "international organization",
+    "Q163740": "political party",
+    "Q178706": "trade union",
+    "Q189004": "government agency",
+    "Q327333": "educational institution",
+    "Q31855": "university",
+    "Q219885": "research institute",
+    "Q12973014": "NGO",  # non-governmental organization
+    "Q7210356": "television network",
+    "Q33231": "military unit",
+    "Q15911314": "sports club",
+    "Q57660343": "media organization",
+    "Q167037": "religious organization",
+    "Q4438121": "publishing company",
+    "Q2088357": "academic publisher",
+}
+qid_to_organization_label_keys = set(qid_to_organization_label.keys())

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8cfb5bf9aa521336b586ae37eecac31ed7e86327a1be1802d32551472988633
+size 2468961388

requirements.txt

@@ -0,0 +1,5 @@
+torch
+transformers
+requests
+requests-cache
+filelock

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc00424e1006d8552c992d3bde1acf8d4282909093c4e18a2112a6e6b087b217
+size 1064

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ee4dc054a17c18fe81f76c0b1cda00e9fc1cfd9e0f1a16cb6d77009e2076653
+size 4870365

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "256001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}

trainer_state.json

@@ -0,0 +1,207 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 10.0,
+  "eval_steps": 500,
+  "global_step": 6480,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.7716049382716049,
+      "grad_norm": 1.0553990602493286,
+      "learning_rate": 1.846913580246914e-05,
+      "loss": 0.9346,
+      "step": 500
+    },
+    {
+      "epoch": 1.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.0959,
+      "eval_loss": 0.15309424698352814,
+      "eval_runtime": 7.8759,
+      "eval_samples_per_second": 154.903,
+      "eval_steps_per_second": 2.539,
+      "step": 648
+    },
+    {
+      "epoch": 1.5432098765432098,
+      "grad_norm": 0.7297214269638062,
+      "learning_rate": 1.6925925925925926e-05,
+      "loss": 0.0763,
+      "step": 1000
+    },
+    {
+      "epoch": 2.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.159,
+      "eval_loss": 0.16104426980018616,
+      "eval_runtime": 7.6405,
+      "eval_samples_per_second": 159.674,
+      "eval_steps_per_second": 2.618,
+      "step": 1296
+    },
+    {
+      "epoch": 2.314814814814815,
+      "grad_norm": 0.44237253069877625,
+      "learning_rate": 1.5382716049382717e-05,
+      "loss": 0.0446,
+      "step": 1500
+    },
+    {
+      "epoch": 3.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.0426,
+      "eval_loss": 0.17489495873451233,
+      "eval_runtime": 7.8385,
+      "eval_samples_per_second": 155.642,
+      "eval_steps_per_second": 2.552,
+      "step": 1944
+    },
+    {
+      "epoch": 3.0864197530864197,
+      "grad_norm": 0.3801327049732208,
+      "learning_rate": 1.3839506172839507e-05,
+      "loss": 0.0275,
+      "step": 2000
+    },
+    {
+      "epoch": 3.8580246913580245,
+      "grad_norm": 0.29495081305503845,
+      "learning_rate": 1.2296296296296298e-05,
+      "loss": 0.0162,
+      "step": 2500
+    },
+    {
+      "epoch": 4.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.1139,
+      "eval_loss": 0.1843736320734024,
+      "eval_runtime": 7.7649,
+      "eval_samples_per_second": 157.118,
+      "eval_steps_per_second": 2.576,
+      "step": 2592
+    },
+    {
+      "epoch": 4.62962962962963,
+      "grad_norm": 0.29735738039016724,
+      "learning_rate": 1.0753086419753086e-05,
+      "loss": 0.0106,
+      "step": 3000
+    },
+    {
+      "epoch": 5.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 9.9508,
+      "eval_loss": 0.19341909885406494,
+      "eval_runtime": 7.6995,
+      "eval_samples_per_second": 158.452,
+      "eval_steps_per_second": 2.598,
+      "step": 3240
+    },
+    {
+      "epoch": 5.401234567901234,
+      "grad_norm": 0.07027166336774826,
+      "learning_rate": 9.209876543209877e-06,
+      "loss": 0.0076,
+      "step": 3500
+    },
+    {
+      "epoch": 6.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 9.9377,
+      "eval_loss": 0.20017552375793457,
+      "eval_runtime": 7.6996,
+      "eval_samples_per_second": 158.45,
+      "eval_steps_per_second": 2.598,
+      "step": 3888
+    },
+    {
+      "epoch": 6.172839506172839,
+      "grad_norm": 0.1504916250705719,
+      "learning_rate": 7.666666666666667e-06,
+      "loss": 0.0059,
+      "step": 4000
+    },
+    {
+      "epoch": 6.944444444444445,
+      "grad_norm": 0.24264627695083618,
+      "learning_rate": 6.123456790123458e-06,
+      "loss": 0.0043,
+      "step": 4500
+    },
+    {
+      "epoch": 7.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.0279,
+      "eval_loss": 0.20386986434459686,
+      "eval_runtime": 7.7944,
+      "eval_samples_per_second": 156.523,
+      "eval_steps_per_second": 2.566,
+      "step": 4536
+    },
+    {
+      "epoch": 7.716049382716049,
+      "grad_norm": 0.08363181352615356,
+      "learning_rate": 4.580246913580247e-06,
+      "loss": 0.0035,
+      "step": 5000
+    },
+    {
+      "epoch": 8.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.1566,
+      "eval_loss": 0.20531675219535828,
+      "eval_runtime": 7.6989,
+      "eval_samples_per_second": 158.465,
+      "eval_steps_per_second": 2.598,
+      "step": 5184
+    },
+    {
+      "epoch": 8.487654320987655,
+      "grad_norm": 0.13225023448467255,
+      "learning_rate": 3.0370370370370372e-06,
+      "loss": 0.0029,
+      "step": 5500
+    },
+    {
+      "epoch": 9.0,
+      "eval_bleu": 0.0,
+      "eval_gen_len": 10.0689,
+      "eval_loss": 0.20702147483825684,
+      "eval_runtime": 7.6619,
+      "eval_samples_per_second": 159.23,
+      "eval_steps_per_second": 2.61,
+      "step": 5832
+    },
+    {
+      "epoch": 9.25925925925926,
+      "grad_norm": 0.022540247067809105,
+      "learning_rate": 1.4938271604938272e-06,
+      "loss": 0.003,
+      "step": 6000
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 6480,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 10,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.288315707563704e+17,
+  "train_batch_size": 64,
+  "trial_name": null,
+  "trial_params": null
+}

utils.py

@@ -0,0 +1,62 @@
+qid_to_location_label = {
+    "Q82794": "geographical region",
+    "Q6256": "country",
+    "Q3624078": "sovereign state",
+    "Q35657": "U.S. state",
+    "Q515": "city",
+    "Q532": "village",
+    "Q3957": "town",
+    "Q2983893": "municipality",
+    "Q486972": "human settlement",
+    "Q23442": "island",
+    "Q8502": "mountain",
+    "Q23397": "lake",
+    "Q4022": "river",
+    "Q123705": "peninsula",
+    "Q13221722": "bay",
+    "Q165": "sea",
+    "Q124714": "ocean",
+    "Q42124": "continent",
+    "Q23413": "territory",
+    "Q1048835": "province",
+    "Q107390": "county",
+    "Q13220204": "district",
+    "Q13218630": "subdistrict",
+    "Q618123": "location",  # geographical object
+    "Q2221906": "location",  # geographical location
+    "Q17334923": "location",  # geographical region
+}
+TYPE_MAP = {
+    "Q5": "person",  # human
+    "Q43229": "organization",  # organization
+    "Q95074": "organization",  # company
+    "Q783794": "organization",  # non-profit
+    "Q618123": "location",  # geographical object
+    "Q2221906": "location",  # geographical location
+    "Q17334923": "location",  # geographical region
+    "Q6256": "location",  # country
+    "Q515": "location",  # city
+}
+
+qid_to_location_label_keys = set(qid_to_location_label.keys())
+qid_to_organization_label = {
+    "Q43229": "organization",  # organization
+    "Q95074": "company",  # company
+    "Q783794": "non-profit",  # non-profit organization
+    "Q891723": "international organization",
+    "Q163740": "political party",
+    "Q178706": "trade union",
+    "Q189004": "government agency",
+    "Q327333": "educational institution",
+    "Q31855": "university",
+    "Q219885": "research institute",
+    "Q12973014": "NGO",  # non-governmental organization
+    "Q7210356": "television network",
+    "Q33231": "military unit",
+    "Q15911314": "sports club",
+    "Q57660343": "media organization",
+    "Q167037": "religious organization",
+    "Q4438121": "publishing company",
+    "Q2088357": "academic publisher",
+}
+qid_to_organization_label_keys = set(qid_to_organization_label.keys())