# Inference pipelines with the ONNX Runtime accelerator

The [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function makes it simple to use models from the [Model Hub](https://huggingface.co/models)
for accelerated inference on a variety of tasks such as text classification, question answering and image classification.

ONNX Runtime pipelines are a drop-in replacement for [Transformers pipelines](https://huggingface.co/docs/transformers/en/main_classes/pipelines) that automatically use ONNX/ONNX Runtime as the backend for model inference. 
This means you get:

- **Faster inference**: ONNX Runtime's optimized execution engine provides significant speedups
- **Cross-platform support**: Works across different hardware accelerators (CPU, GPU, etc.)
- **Same API**: Identical interface to transformers pipelines - no code changes needed
- **Automatic model loading**: Seamlessly loads or exports ONNX models

<Tip>

You can also use the
[pipeline()](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#pipelines) function from
Transformers and provide your Optimum model and tokenizer/feature-extractor to it.

```python
>>> from transformers import pipeline
>>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer
>>> model = ORTModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
>>> onnx_pipeline = pipeline("text-classification", model=model, tokenizer=tokenizer)
>>> onnx_pipeline("I love you")
[{'label': 'POSITIVE', 'score': 0.999870002746582}]
```

</Tip>

Currently the supported tasks are:

* `audio-classification`: Classify audio inputs into predefined categories.
* `automatic-speech-recognition`: Convert spoken language into text.
* `feature-extraction`: Extract features from text or images using pre-trained models.
* `fill-mask`: Predict missing words in a sentence.
* `image-classification`: Classify images into predefined categories.
* `image-segmentation`: Segment images into different regions based on their content.
* `image-to-image`: Transform images from one domain to another (e.g., style transfer).
* `image-to-text`: Generate textual descriptions for images.
* `question-answering`: Answer questions based on a given context.
* `summarization`: Generate concise summaries of longer text documents.
* `text2text-generation`: Generate text based on a given input text.
* `text-classification`: Classify text into predefined categories (e.g., sentiment analysis).
* `text-generation`: Generate text based on a given prompt.
* `token-classification`: Classify individual tokens in a text (e.g., named entity recognition).
* `translation`: Translate text from one language to another.
* `zero-shot-classification`: Classify text into categories without prior training on those categories.

## ONNX Runtime pipeline usage

While each task has an associated pipeline class, it is simpler to use the general [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function which wraps all the task-specific pipelines in one object.
The [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function automatically loads a default model and tokenizer/feature-extractor capable of performing inference for your task.

1. Start by creating a pipeline by specifying an inference task:

```python
>>> from optimum.onnxruntime import pipeline

>>> classifier = pipeline(task="text-classification")
```

2. Pass your input text/image to the [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function:

```python
>>> classifier("I like you. I love you.")
[{'label': 'POSITIVE', 'score': 0.9998838901519775}]
```

_Note: The default models used in the [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function are not optimized for inference or quantized, so there won't be a performance improvement compared to their PyTorch counterparts._

### Using vanilla Transformers model and converting to ONNX

The [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function accepts any supported model from the [Hugging Face Hub](https://huggingface.co/models).
There are tags on the Model Hub that allow you to filter for a model you'd like to use for your task.

<Tip>

To be able to load the model with the ONNX Runtime backend, the export to ONNX needs to be supported for the considered architecture.

You can check the list of supported architectures [here](https://huggingface.co/docs/optimum/exporters/onnx/overview#overview).

</Tip>

Once you have picked an appropriate model, you can create the [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) by specifying the model repo:

```python
>>> from optimum.onnxruntime import pipeline

# The model will be loaded to an ORTModelForQuestionAnswering.
>>> onnx_qa = pipeline("question-answering", model="deepset/roberta-base-squad2")
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

It is also possible to load it with the `from_pretrained(model_name_or_path, export=True)`
method associated with the `ORTModelForXXX` class.

For example, here is how you can load the [ORTModelForQuestionAnswering](/docs/optimum/pr_68/en/onnxruntime/package_reference/modeling_ort#optimum.onnxruntime.ORTModelForQuestionAnswering) class for question answering:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering
>>> from optimum.onnxruntime import pipeline

>>> tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")

>>> # Loading the PyTorch checkpoint and converting to the ONNX format by providing
>>> # export=True
>>> model = ORTModelForQuestionAnswering.from_pretrained(
...     "deepset/roberta-base-squad2",
...     export=True
... )

>>> onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

### Using Optimum models

The [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function is tightly integrated with the [Hugging Face Hub](https://huggingface.co/models) and can load ONNX models directly.

```python
>>> from optimum.onnxruntime import pipeline

>>> onnx_qa = pipeline("question-answering", model="optimum/roberta-base-squad2")
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```

It is also possible to load it with the `from_pretrained(model_name_or_path)`
method associated with the `ORTModelForXXX` class.

For example, here is how you can load the [ORTModelForQuestionAnswering](/docs/optimum/pr_68/en/onnxruntime/package_reference/modeling_ort#optimum.onnxruntime.ORTModelForQuestionAnswering) class for question answering:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering
>>> from optimum.onnxruntime import pipeline

>>> tokenizer = AutoTokenizer.from_pretrained("optimum/roberta-base-squad2")

>>> # Loading directly an ONNX model from a model repo.
>>> model = ORTModelForQuestionAnswering.from_pretrained("optimum/roberta-base-squad2")

>>> onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
>>> question = "What's my name?"
>>> context = "My name is Philipp and I live in Nuremberg."

>>> pred = onnx_qa(question=question, context=context)
```


## Optimizing and quantizing in pipelines

The [pipeline()](/docs/optimum/pr_68/en/onnxruntime/package_reference/pipelines#optimum.onnxruntime.pipeline) function can not only run inference on vanilla ONNX Runtime checkpoints - you can also use
checkpoints optimized with the [ORTQuantizer](/docs/optimum/pr_68/en/onnxruntime/package_reference/quantization#optimum.onnxruntime.ORTQuantizer) and the [ORTOptimizer](/docs/optimum/pr_68/en/onnxruntime/package_reference/optimization#optimum.onnxruntime.ORTOptimizer).

Below you can find two examples of how you could use the [ORTOptimizer](/docs/optimum/pr_68/en/onnxruntime/package_reference/optimization#optimum.onnxruntime.ORTOptimizer) and the
[ORTQuantizer](/docs/optimum/pr_68/en/onnxruntime/package_reference/quantization#optimum.onnxruntime.ORTQuantizer) to optimize/quantize your model and use it for inference afterwards.

### Quantizing with the `ORTQuantizer`

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import (
...     AutoQuantizationConfig,
...     ORTModelForSequenceClassification,
...     ORTQuantizer
... )
>>> from optimum.onnxruntime import pipeline

>>> # Load the tokenizer and export the model to the ONNX format
>>> model_id = "distilbert-base-uncased-finetuned-sst-2-english"
>>> save_dir = "distilbert_quantized"

>>> model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)

>>> # Load the quantization configuration detailing the quantization we wish to apply
>>> qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=True)
>>> quantizer = ORTQuantizer.from_pretrained(model)

>>> # Apply dynamic quantization and save the resulting model
>>> quantizer.quantize(save_dir=save_dir, quantization_config=qconfig)
>>> # Load the quantized model from a local repository
>>> model = ORTModelForSequenceClassification.from_pretrained(save_dir)

>>> # Create the transformers pipeline
>>> onnx_clx = pipeline("text-classification", model=model)
>>> text = "I like the new ORT pipeline"
>>> pred = onnx_clx(text)
>>> print(pred)
>>> # [{'label': 'POSITIVE', 'score': 0.9974810481071472}]

>>> # Save and push the model to the hub (in practice save_dir could be used here instead)
>>> model.save_pretrained("new_path_for_directory")
>>> model.push_to_hub("new_path_for_directory", repository_id="my-onnx-repo", use_auth_token=True)
```

### Optimizing with `ORTOptimizer`

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import (
...     AutoOptimizationConfig,
...     ORTModelForSequenceClassification,
...     ORTOptimizer
... )
>>> from optimum.onnxruntime.configuration import OptimizationConfig
>>> from optimum.onnxruntime import pipeline

>>> # Load the tokenizer and export the model to the ONNX format
>>> model_id = "distilbert-base-uncased-finetuned-sst-2-english"
>>> save_dir = "distilbert_optimized"

>>> tokenizer = AutoTokenizer.from_pretrained(model_id)
>>> model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)

>>> # Load the optimization configuration detailing the optimization we wish to apply
>>> optimization_config = AutoOptimizationConfig.O3()
>>> optimizer = ORTOptimizer.from_pretrained(model)

>>> optimizer.optimize(save_dir=save_dir, optimization_config=optimization_config)
# Load the optimized model from a local repository
>>> model = ORTModelForSequenceClassification.from_pretrained(save_dir)

# Create the transformers pipeline
>>> onnx_clx = pipeline("text-classification", model=model)
>>> text = "I like the new ORT pipeline"
>>> pred = onnx_clx(text)
>>> print(pred)
>>> # [{'label': 'POSITIVE', 'score': 0.9973127245903015}]

# Save and push the model to the hub
>>> tokenizer.save_pretrained("new_path_for_directory")
>>> model.save_pretrained("new_path_for_directory")
>>> model.push_to_hub("new_path_for_directory", repository_id="my-onnx-repo", use_auth_token=True)
```