# Contributing to Multilingual Evaluations

Lighteval supports multilingual evaluations through a comprehensive system of translation literals and language-adapted templates.

## Contributing Translation Literals

### What Are Translation Literals?

We define 19 `literals`, basic keywords or punctuation signs used when creating evaluation prompts in an automatic manner, such as `yes`, `no`, `because`, etc.

These literals are essential for:
- **Consistent prompt formatting** across languages
- **Automatic prompt generation** for multilingual tasks
- **Proper localization** of evaluation templates

### How to Contribute Translations

We welcome translations in your language! To contribute:

1. **Open the translation literals file**: [translation_literals.py](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/utils/translation_literals.py)

2. **Edit the file** to add or expand the literal for your language of interest

3. **Open a PR** with your modifications

### Translation Literals Structure

```python
Language.ENGLISH: TranslationLiterals(
    language=Language.ENGLISH,
    question_word="question",  # Usage: "Question: How are you?"
    answer="answer",  # Usage: "Answer: I am fine"
    confirmation_word="right",  # Usage: "He is smart, right?"
    yes="yes",  # Usage: "Yes, he is"
    no="no",  # Usage: "No, he is not"
    also="also",  # Usage: "Also, she is smart."
    cause_word="because",  # Usage: "She is smart, because she is tall"
    effect_word="therefore",  # Usage: "He is tall therefore he is smart"
    or_word="or",  # Usage: "He is tall or small"
    true="true",  # Usage: "He is smart, true, false or neither?"
    false="false",  # Usage: "He is smart, true, false or neither?"
    neither="neither",  # Usage: "He is smart, true, false or neither?"
    # Punctuation and spacing: only adjust if your language uses something different than in English
    full_stop=".",
    comma=",",
    question_mark="?",
    exclamation_mark="!",
    word_space=" ",
    sentence_space=" ",
    colon=":",
    # The first characters of your alphabet used in enumerations, if different from English
    indices=["A", "B", "C", ...]
)
```

## Contributing New Multilingual Tasks

### Prerequisites

Before creating a new multilingual task, you should:

1. **Read the custom task guide**: [Adding a Custom Task](adding-a-custom-task)
2. **Understand multilingual task structure**: Review the [multilingual tasks](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/multilingual/tasks.py) file
3. **Browse available templates**: Check the [templates directory](https://github.com/huggingface/lighteval/tree/main/src/lighteval/tasks/templates)

### Key Concepts

#### Language-Adapted Templates
For multilingual evaluations, the `prompt_function` should be implemented using language-adapted templates. These templates handle:
- **Correct formatting** for each language
- **Consistent usage** of language-adjusted prompt anchors (e.g., Question/Answer)
- **Proper punctuation** and spacing conventions

#### Template Types
Available template types include:
- **XNLI**: Natural language inference tasks - [`get_nli_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/nli.py#L162)
- **COPA**: Causal reasoning tasks - [`get_copa_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/copa.py#L76)
- **Multiple Choice**: Standard multiple choice questions - [`get_mcq_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/multichoice.py#L81)
- **Question Answering**: Open-ended question answering - [`get_qa_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/qa.py#L46)
- **Custom**: Specialized task templates

#### Formulation Types

##### Multiple Choice Formulation (MCF)
Used for standard multiple choice questions where the model selects from lettered options:
```python
MCFFormulation()
```

**Example output:**
```
Question: What is the capital of France?
A. London
B. Paris
C. Berlin
D. Rome
Answer: | A/B/C/D
```

##### Classification Formulation (CF)
Used for classification tasks where the model generates the answer directly:
```python
CFFormulation()
```

**Example output:**
```
Question: What is the capital of France?
Answer: | Paris
```

##### Hybrid Formulation
Used for tasks that present choices but expect the full answer text:
```python
HybridFormulation()
```

**Example output:**
```
Question: What is the capital of France?
A. London
B. Paris
C. Berlin
D. Rome
Answer: | Paris
```


### Creating Your Multilingual Task

#### Step 1: Create the Task File
Create a Python file following the custom task guide structure.

#### Step 2: Import Required Components
```python
from lighteval.tasks.lighteval_task import LightevalTaskConfig
from lighteval.tasks.multilingual.language import Language
from lighteval.tasks.multilingual.formulations import MCFFormulation, CFFormulation, HybridFormulation
from lighteval.tasks.multilingual.templates import get_template_prompt_function
from lighteval.tasks.multilingual.metrics import get_metrics_for_formulation, loglikelihood_acc_metric
from lighteval.tasks.multilingual.normalization import LogProbTokenNorm, LogProbCharNorm
```

#### Step 3: Define Your Tasks
```python
your_tasks = [
    LightevalTaskConfig(
        # Name of your evaluation
        name=f"evalname_{language.value}_{formulation.name.lower()}",
        # The evaluation is community contributed
        suite=["community"],
        # This will automatically get the correct metrics for your chosen formulation
        metric=get_metrics_for_formulation(
            formulation,
            [
                LogLikelihoodAccMetric(normalization=None),
                LogLikelihoodAccMetric(normalization=LogProbTokenNorm()),
                LogLikelihoodAccMetric(normalization=LogProbCharNorm()),
            ],
        ),
        # In this function, you choose which template to follow and for which language and formulation
        prompt_function=get_template_prompt_function(
            language=language,
            # Use the adapter to define the mapping between the
            # keys of the template (left), and the keys of your dataset
            # (right)
            # To know which template keys are required and available,
            # consult the appropriate adapter type and doc-string.
            adapter=lambda line: {
                "key": line["relevant_key"],
                # Add more mappings as needed
            },
            formulation=formulation,
        ),
        # You can also add specific filters to remove irrelevant samples
        hf_filter=lambda line: line["label"] in <condition>,
        # You then select your huggingface dataset as well as
        # the splits available for evaluation
        hf_repo=<dataset>,
        hf_subset=<subset>,
        evaluation_splits=["train"],
        hf_avail_splits=["train"],
    )
    for language in [
        Language.YOUR_LANGUAGE,  # Add your target languages
        # Language.SPANISH,
        # Language.FRENCH,
        # etc.
    ]
    for formulation in [MCFFormulation(), CFFormulation(), HybridFormulation()]
]
```

#### Step 4: Test Your Implementation
Follow the custom task guide to test if your task is correctly implemented.

> [!TIP]
> All [LightevalTaskConfig](/docs/lighteval/pr_1017/en/package_reference/tasks#lighteval.tasks.lighteval_task.LightevalTaskConfig) parameters are strongly typed, including the inputs to the template function. Make sure to take advantage of your IDE's functionality to make it easier to correctly fill these parameters.

### Validation Checklist
- [ ] Translation literals are accurate and complete
- [ ] Task works correctly across all target languages
- [ ] Metrics are appropriate for the task type
- [ ] Documentation is clear and comprehensive
- [ ] Code follows project conventions

### Getting Help

- **GitHub Issues**: Report bugs or ask questions
- **Discussions**: Join community discussions
- **Documentation**: Review existing guides and examples


<EditOnGithub source="https://github.com/huggingface/lighteval/blob/main/docs/source/contributing-to-multilingual-evaluations.mdx" />