# Inference pipelines with the ONNX Runtime accelerator

The [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function makes it simple to use models from the [Model Hub](https://huggingface.co/models)
for accelerated inference on a variety of tasks such as text classification, question answering and image classification.

ONNX Runtime pipelines are a drop-in replacement for [Transformers pipelines](https://huggingface.co/docs/transformers/en/main_classes/pipelines) that automatically use ONNX/ONNX Runtime as the backend for model inference. 
This means you get:

- **Faster inference**: ONNX Runtime's optimized execution engine provides significant speedups
- **Cross-platform support**: Works across different hardware accelerators (CPU, GPU, etc.)
- **Same API**: Identical interface to transformers pipelines - no code changes needed
- **Automatic model loading**: Seamlessly loads or exports ONNX models

You can also use the
[pipeline()](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#pipelines) function from
Transformers and provide your Optimum model and tokenizer/feature-extractor to it.

```python
>>> from transformers import pipeline
>>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer
>>> model = ORTModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
>>> onnx_pipeline = pipeline("text-classification", model=model, tokenizer=tokenizer)
>>> onnx_pipeline("I love you")
[{'label': 'POSITIVE', 'score': 0.999870002746582}]
```

Currently the supported tasks are:

* `audio-classification`: Classify audio inputs into predefined categories.
* `automatic-speech-recognition`: Convert spoken language into text.
* `feature-extraction`: Extract features from text or images using pre-trained models.
* `fill-mask`: Predict missing words in a sentence.
* `image-classification`: Classify images into predefined categories.
* `image-segmentation`: Segment images into different regions based on their content.
* `image-to-image`: Transform images from one domain to another (e.g., style transfer).
* `image-to-text`: Generate textual descriptions for images.
* `question-answering`: Answer questions based on a given context.
* `summarization`: Generate concise summaries of longer text documents.
* `text2text-generation`: Generate text based on a given input text.
* `text-classification`: Classify text into predefined categories (e.g., sentiment analysis).
* `text-generation`: Generate text based on a given prompt.
* `token-classification`: Classify individual tokens in a text (e.g., named entity recognition).
* `translation`: Translate text from one language to another.
* `zero-shot-classification`: Classify text into categories without prior training on those categories.

## ONNX Runtime pipeline usage

While each task has an associated pipeline class, it is simpler to use the general [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function which wraps all the task-specific pipelines in one object.
The [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function automatically loads a default model and tokenizer/feature-extractor capable of performing inference for your task.

1. Start by creating a pipeline by specifying an inference task:

```python
>>> from optimum.onnxruntime import pipeline

>>> classifier = pipeline(task="text-classification")
```

2. Pass your input text/image to the [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function:

```python
>>> classifier("I like you. I love you.")
[{'label': 'POSITIVE', 'score': 0.9998838901519775}]
```

_Note: The default models used in the [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function are not optimized for inference or quantized, so there won't be a performance improvement compared to their PyTorch counterparts._

### Using vanilla Transformers model and converting to ONNX

The [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function accepts any supported model from the [Hugging Face Hub](https://huggingface.co/models).
There are tags on the Model Hub that allow you to filter for a model you'd like to use for your task.

To be able to load the model with the ONNX Runtime backend, the export to ONNX needs to be supported for the considered architecture.

You can check the list of supported architectures [here](https://huggingface.co/docs/optimum/exporters/onnx/overview#overview).

Once you have picked an appropriate model, you can create the [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) by specifying the model repo:

```python
>>> from optimum.onnxruntime import pipeline

# The model will be loaded to an ORTModelForQuestionAnswering.
>>> onnx_qa = pipeline("question-answering", model="deepset/roberta-base-squad2")
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

It is also possible to load it with the `from_pretrained(model_name_or_path, export=True)`
method associated with the `ORTModelForXXX` class.

For example, here is how you can load the [ORTModelForQuestionAnswering](/docs/optimum/pr_102/en/onnxruntime/package_reference/modeling#optimum.onnxruntime.ORTModelForQuestionAnswering) class for question answering:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering
>>> from optimum.onnxruntime import pipeline

>>> tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")

>>> # Loading the PyTorch checkpoint and converting to the ONNX format by providing
>>> # export=True
>>> model = ORTModelForQuestionAnswering.from_pretrained(
...     "deepset/roberta-base-squad2",
...     export=True
... )

>>> onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

### Using Optimum models

The [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function is tightly integrated with the [Hugging Face Hub](https://huggingface.co/models) and can load ONNX models directly.

```python
>>> from optimum.onnxruntime import pipeline

>>> onnx_qa = pipeline("question-answering", model="optimum/roberta-base-squad2")
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

It is also possible to load it with the `from_pretrained(model_name_or_path)`
method associated with the `ORTModelForXXX` class.

For example, here is how you can load the [ORTModelForQuestionAnswering](/docs/optimum/pr_102/en/onnxruntime/package_reference/modeling#optimum.onnxruntime.ORTModelForQuestionAnswering) class for question answering:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering
>>> from optimum.onnxruntime import pipeline

>>> tokenizer = AutoTokenizer.from_pretrained("optimum/roberta-base-squad2")

>>> # Loading directly an ONNX model from a model repo.
>>> model = ORTModelForQuestionAnswering.from_pretrained("optimum/roberta-base-squad2")

>>> onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

## Optimizing and quantizing in pipelines

The [pipeline()](/docs/optimum/pr_102/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function can not only run inference on vanilla ONNX Runtime checkpoints - you can also use
checkpoints optimized with the [ORTQuantizer](/docs/optimum/pr_102/en/onnxruntime/package_reference/quantization#optimum.onnxruntime.ORTQuantizer) and the [ORTOptimizer](/docs/optimum/pr_102/en/onnxruntime/package_reference/optimization#optimum.onnxruntime.ORTOptimizer).

Below you can find two examples of how you could use the [ORTOptimizer](/docs/optimum/pr_102/en/onnxruntime/package_reference/optimization#optimum.onnxruntime.ORTOptimizer) and the
[ORTQuantizer](/docs/optimum/pr_102/en/onnxruntime/package_reference/quantization#optimum.onnxruntime.ORTQuantizer) to optimize/quantize your model and use it for inference afterwards.

### Quantizing with the `ORTQuantizer`

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import (
...     AutoQuantizationConfig,
...     ORTModelForSequenceClassification,
...     ORTQuantizer
... )
>>> from optimum.onnxruntime import pipeline

>>> # Load the tokenizer and export the model to the ONNX format
>>> model_id = "distilbert-base-uncased-finetuned-sst-2-english"
>>> save_dir = "distilbert_quantized"

>>> model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)

>>> # Load the quantization configuration detailing the quantization we wish to apply
>>> qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=True)
>>> quantizer = ORTQuantizer.from_pretrained(model)

>>> # Apply dynamic quantization and save the resulting model
>>> quantizer.quantize(save_dir=save_dir, quantization_config=qconfig)
>>> # Load the quantized model from a local repository
>>> model = ORTModelForSequenceClassification.from_pretrained(save_dir)

>>> # Create the transformers pipeline
>>> onnx_clx = pipeline("text-classification", model=model)
>>> text = "I like the new ORT pipeline"
>>> pred = onnx_clx(text)
>>> print(pred)
>>> # [{'label': 'POSITIVE', 'score': 0.9974810481071472}]

>>> # Save and push the model to the hub (in practice save_dir could be used here instead)
>>> model.save_pretrained("new_path_for_directory")
>>> model.push_to_hub("new_path_for_directory", repository_id="my-onnx-repo", use_auth_token=True)
```

### Optimizing with `ORTOptimizer`

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import (
...     AutoOptimizationConfig,
...     ORTModelForSequenceClassification,
...     ORTOptimizer
... )
>>> from optimum.onnxruntime.configuration import OptimizationConfig
>>> from optimum.onnxruntime import pipeline

>>> # Load the tokenizer and export the model to the ONNX format
>>> model_id = "distilbert-base-uncased-finetuned-sst-2-english"
>>> save_dir = "distilbert_optimized"

>>> tokenizer = AutoTokenizer.from_pretrained(model_id)
>>> model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)

>>> # Load the optimization configuration detailing the optimization we wish to apply
>>> optimization_config = AutoOptimizationConfig.O3()
>>> optimizer = ORTOptimizer.from_pretrained(model)

>>> optimizer.optimize(save_dir=save_dir, optimization_config=optimization_config)
# Load the optimized model from a local repository
>>> model = ORTModelForSequenceClassification.from_pretrained(save_dir)

# Create the transformers pipeline
>>> onnx_clx = pipeline("text-classification", model=model)
>>> text = "I like the new ORT pipeline"
>>> pred = onnx_clx(text)
>>> print(pred)
>>> # [{'label': 'POSITIVE', 'score': 0.9973127245903015}]

# Save and push the model to the hub
>>> tokenizer.save_pretrained("new_path_for_directory")
>>> model.save_pretrained("new_path_for_directory")
>>> model.push_to_hub("new_path_for_directory", repository_id="my-onnx-repo", use_auth_token=True)
```