| --- |
| license: apache-2.0 |
| base_model: google/muril-base-cased |
| language: |
| - hi |
| - en |
| library_name: transformers |
| pipeline_tag: token-classification |
| tags: |
| - token-classification |
| - language-identification |
| - code-switching |
| - hinglish |
| - code-mixed |
| - romanized-hindi |
| - text-to-speech |
| widget: |
| - text: "Mujhe calculus ka doubt samajh nahi aaya, please explain." |
| - text: "Main is question ka main point samajh nahi paya." |
| - text: "Is chapter mein bahut kam time hai, kam se kam revision kar lo." |
| model-index: |
| - name: muril-hinglish-lid |
| results: |
| - task: |
| type: token-classification |
| name: Language Identification (word-level) |
| dataset: |
| name: LinCE Hindi-English (dev, lang1+lang2 subset) |
| type: lince |
| metrics: |
| - type: accuracy |
| value: 0.9643 |
| - type: f1 |
| value: 0.9755 |
| name: ENG F1 |
| - type: f1 |
| value: 0.9342 |
| name: HIN F1 |
| --- |
| |
| # muril-hinglish-lid |
|
|
| **Context-aware Hindi/English language identification for romanized Hinglish.** |
|
|
| Labels every word of code-mixed Latin-script text as `HIN` or `ENG`, using the |
| surrounding sentence rather than a dictionary. |
|
|
| ## Example |
|
|
| Input: |
|
|
| ``` |
| Mujhe calculus ka doubt samajh nahi aaya, please explain. |
| ``` |
|
|
| Output: |
|
|
| | word | label | |
| |---|---| |
| | Mujhe | HIN | |
| | calculus | ENG | |
| | ka | HIN | |
| | doubt | ENG | |
| | samajh | HIN | |
| | nahi | HIN | |
| | aaya | HIN | |
| | please | ENG | |
| | explain | ENG | |
|
|
| ## Why context matters |
|
|
| The same string can be two different languages in one sentence: |
|
|
| ``` |
| Main is question ka main point samajh nahi paya |
| HIN HIN ENG HIN ENG ENG HIN HIN HIN |
| └─ मैं └─ इस └─ English "main" |
| ``` |
|
|
| `Main` is मैं; four words later `main` is the English adjective. `is` is इस, not |
| the English copula. No lookup table can express this — it has one row per string. |
|
|
| That is not a contrived example. In our evaluation data the word `to` occurs 775 |
| times, splitting **435 Hindi (तो) / 340 English**. A perfect lookup table gets at |
| most **56%** of those by always guessing the majority. This model gets **98.32%**. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForTokenClassification |
| |
| MODEL = "PhysicsWallahAI/muril-hinglish-lid" |
| tok = AutoTokenizer.from_pretrained(MODEL) |
| model = AutoModelForTokenClassification.from_pretrained(MODEL).eval() |
| |
| |
| def tag_words(words: list[str]) -> list[str]: |
| """One label per word, read off the word's FIRST sub-token.""" |
| enc = tok(words, is_split_into_words=True, truncation=True, |
| max_length=256, return_tensors="pt") |
| with torch.no_grad(): |
| pred = model(**enc).logits[0].argmax(-1).tolist() |
| out, prev = [], None |
| for pos, wid in enumerate(enc.word_ids()): |
| if wid is not None and wid != prev: |
| out.append(model.config.id2label[pred[pos]]) |
| prev = wid |
| return out |
| |
| |
| words = "Mujhe calculus ka doubt samajh nahi aaya please explain".split() |
| print(list(zip(words, tag_words(words)))) |
| # [('Mujhe','HIN'), ('calculus','ENG'), ('ka','HIN'), ('doubt','ENG'), |
| # ('samajh','HIN'), ('nahi','HIN'), ('aaya','HIN'), ('please','ENG'), |
| # ('explain','ENG')] |
| ``` |
|
|
| **Use this helper rather than `pipeline(..., aggregation_strategy=...)`.** The |
| pipeline's aggregation is designed for NER: it merges *consecutive tokens sharing |
| a label* into one span, which is right for `New York City` → one `LOC` and wrong |
| here, where adjacent words routinely share a language. On the example above it |
| returns 7 spans instead of 9 words, fusing `samajh nahi aaya,` into a single |
| `HIN` blob. `aggregation_strategy="none"` is not the fix either — it returns |
| sub-word pieces (`'Mu'`, `'##jhe'`). |
| |
| The helper matches the training-time contract exactly: words pre-split, one label |
| per word taken from its first sub-token. Continuation pieces were masked out of |
| the loss during training and carry no supervision, so reading them — or averaging |
| over them, which would let a long word's many pieces outvote a short word's one — |
| asks the model something it was never taught to answer. |
| |
| Tag in windows of ~100 words, which is the window used in training. |
| |
| ## Model |
| |
| - MuRIL backbone (`google/muril-base-cased`), pretrained on 17 Indian languages |
| **and their romanized forms** |
| - Token-classification head, 2 labels |
| - 237 M parameters · 12 layers · hidden 768 · WordPiece vocab 197,285 |
| - Labels: `HIN` / `ENG` |
| - Context-aware prediction |
|
|
| ## Performance |
|
|
| ### Held-out gold set |
|
|
| 874 answers / 108,432 tokens. The evaluation protocol is the part worth reading: |
| these labels were annotated **from the source text** by a separate model that |
| wrote none of the training corpus. Training labels were derived by aligning |
| Hinglish text to its Devanagari rewrite; the gold labels were not. So this is the |
| only measurement here that is independent of the pipeline that produced the |
| training data. |
|
|
| | metric | value | |
| |---|---| |
| | accuracy | **0.9920** | |
| | HIN F1 | 0.9927 | |
| | ENG F1 | 0.9912 | |
| | contextual homographs (8,444 occurrences) | 0.9861 | |
| | false-Hindi rate on English-only text | 0.02% (1 of 4,363 tokens) | |
|
|
| On its own auto-labelled test split the model reads 0.9934 — but that split is |
| the instrument that cannot see its own errors. The gold figure is the one to quote. |
|
|
| ### Hard cases |
|
|
| | word | accuracy | note | |
| |---|---|---| |
| | `to` | 0.9832 | 775 occurrences, 435 तो / 340 English. Lookup ceiling: 56% | |
| | `use` | 0.7799 | उसे vs English "use". The weakest case, and unchanged across two independently retrained checkpoints — the signature of genuine ambiguity rather than a data defect | |
| | `beta` | — | बेटा (term of address) vs β, the physics symbol | |
|
|
| ### Out of domain: LinCE Hindi-English |
|
|
| [LinCE](https://ritual.uh.edu/lince/) is human-annotated, public, and from a |
| different domain entirely — code-mixed **tweets**, not tutoring text. |
|
|
| | | tokens | accuracy | HIN F1 | ENG F1 | |
| |---|---|---|---|---| |
| | LinCE dev | 12,303 | **0.9643** | 0.9342 | 0.9755 | |
|
|
| > **Not comparable to the LinCE leaderboard.** LinCE has eight classes; this model |
| > has two. We map `lang1`→ENG, `lang2`→HIN and drop the rest — `other` |
| > (punctuation, handles, URLs, emoji; 2,231 tokens), `ne` (named entities; 875), |
| > and 37 tokens of `fw`/`mixed`/`unk`/`ambiguous`. Scoring only the two mappable |
| > classes is an **easier** task than LinCE's. Compare this number to our in-domain |
| > accuracy, and to nothing else. |
|
|
| ### Latency |
|
|
| 0.099 ms/word (108 words in 10.7 ms), NVIDIA A10, fp32, batch 1. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | data | Hinglish tutoring answers, labelled by aligning each answer to its Devanagari rewrite | |
| | train | 23,727 answers / 2,783,768 labelled tokens | |
| | dev | 1,402 answers / 162,047 labelled tokens | |
| | test | 2,771 answers / 322,569 labelled tokens | |
| | splits | **by answer** — the same answer never appears on two sides | |
| | hyperparameters | 3 epochs · batch 32 · lr 3e-5 · warmup ratio 0.1 · weight decay 0.01 · fp16 · max length 256 · 100-word windows | |
| | selection | best epoch by masked accuracy on dev; test read once, at the end | |
|
|
| Tokens the aligner could not place — math variables, symbols, ambiguous residue — |
| were emitted as `O` and **masked out of the loss** rather than guessed at, so the |
| model was never trained to invent a label for something the labelling process did |
| not actually know. |
|
|
| ## Intended use |
|
|
| - Hinglish language identification |
| - Preprocessing for text-to-speech |
| - Transliteration pipelines |
| - Code-switched Hindi/English text |
|
|
| ## Not intended for |
|
|
| - **General language identification across arbitrary languages.** Two labels |
| only. Other Indian languages in Latin script will be forced into `HIN` or `ENG`. |
| - **Devanagari transliteration itself.** This model decides *what* to |
| transliterate. It does not transliterate. |
| - **Determining whether a whole sentence is Hindi or English.** It is a |
| token-level model. Aggregating its labels to a sentence verdict is not what it |
| was built or evaluated for. |
|
|
| ## Limitations |
|
|
| **SMS and chat shorthand.** The largest out-of-domain error class is abbreviated |
| social-media spelling, absent from tutoring text: `ur`, `u`, `h`, `k`, `b`, `r`, |
| `g`. On chat-register text expect worse than the LinCE number suggests — that |
| number already contains these errors, but diluted by well-formed tokens. |
|
|
| **Short fragments.** Accuracy tracks available context, not language. On short |
| homograph-dense English — *"Let me know so I can do it in the morning"* — error |
| rates rise sharply, because `let`, `me`, `know`, `so`, `can`, `do` are all Hindi |
| words in other contexts and six words condition almost nothing. On full-length |
| English text the false-Hindi rate is ~1%; on hand-picked short fragments it was |
| 12.8%. Length is the variable. |
|
|
| **Named entities are out of scope.** The model emits only `HIN`/`ENG` and was |
| never trained on an NE class. Personal and place names receive *some* label, and |
| which one is not meaningful. Handle names separately. |
|
|
| **Domain.** Indian K-12 / exam-prep tutoring text: explanatory, second-person, |
| mathematics- and science-heavy, generally well-formed sentences. That is where |
| 0.9920 holds. |
|
|
| ## Data |
|
|
| The training data is proprietary tutoring text and is **not released**. No |
| training data is included in this repository — weights, config and tokenizer only. |
|
|
| The data reflects the register, subject matter and code-mixing conventions of one |
| setting: Hindi-English as used in Indian exam preparation. It is not a sample of |
| Hinglish in general, and the model's notion of "which words are Hindi" is that |
| community's. |
|
|
| Worth stating plainly, since this model exists to serve a TTS front-end: a `HIN` |
| label causes a word to be transliterated to Devanagari and pronounced with Hindi |
| phonology. A mislabelled word is therefore **mispronounced**, not dropped. The |
| failure is audible — the right direction for a failure to go, but it does mean |
| errors reach listeners directly. |
|
|
| ## License |
|
|
| Apache-2.0, inherited from `google/muril-base-cased`. Training data is not |
| included and is not licensed for redistribution. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{muril-hinglish-lid, |
| title = {muril-hinglish-lid: context-aware Hindi/English language identification for romanized Hinglish}, |
| author = {PhysicsWallah AI}, |
| year = {2026}, |
| url = {https://huggingface.co/PhysicsWallahAI/muril-hinglish-lid} |
| } |
| ``` |
|
|
| Base model: |
|
|
| ```bibtex |
| @article{khanuja2021muril, |
| title = {MuRIL: Multilingual Representations for Indian Languages}, |
| author = {Khanuja, Simran and Bansal, Diksha and Mehtani, Sarvesh and others}, |
| journal = {arXiv preprint arXiv:2103.10730}, |
| year = {2021} |
| } |
| ``` |
|
|
