#
Text Embeddings Inference (TEI)

Text Embeddings Inference (TEI) is a robust, production-ready engine designed for fast and efficient generation of text embeddings from a wide range of models. Built for scalability and reliability, TEI streamlines the deployment of embedding models for search, retrieval, clustering, and semantic understanding tasks.

Key Features:

- Efficient Resource Utilization: Benefit from small Docker images and rapid boot times.
- Dynamic Batching: TEI incorporates token-based dynamic batching thus optimizing resource utilization during inference.
- Optimized Inference: TEI leverages Flash Attention, Candle, and cuBLASLt by using optimized transformers code for inference.
- Support for models in both the Safetensors and ONNX format
- Production-Ready: TEI supports distributed tracing through Open Telemetry and exports Prometheus metrics.

##
Configuration

- Max Tokens (per batch): Number of tokens that can be added to a batch before forcing queries to wait in the internal queue.
- Max Concurrent Requests: The maximum number of requests that the server can handle at once.
- Pooling: Setting to override the model pooling configuration. Default is not to override the model configuration.

##
Supported models

You can find the models that are supported by TGI by either:

- Browse supported models on the Hugging Face Hub
- In the TEI documentation under the supported models section

##
References

We also recommend reading the TEI documentation for more in-depth information.

Update on GitHub