README.md

# Mixtral-8x7b-Instruct-v0.1-int4-ov
 * Model creator: [Mistral AI](https://huggingface.co/mistralai)
 * Original model: [Mixtral 8X7B Instruct v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)

## Description
This is [Mixtral-8x7b-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model converted to [OpenVINO](https://docs.openvino.ai/2024/home.html) Intermediate Representation (IR) format with INT4 compressed weights using [NNCF](https://github.com/openvinotoolkit/nncf).

## Compatibility

This provided IR is compatible with openvino starting with 2024.0.0 version and optimum-intel 1.16.0

## Usage

### Install required packages

To install the required components for using [Optimum Intel integration](https://huggingface.co/docs/optimum/intel/index) with the OpenVINO backend, do: 
```
pip install optimum[openvino]
```

### Run model inference
```
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)


messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

For more examples and possible optimizations please refer [OpenVINO Large Language Model Inference Guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html)

### Limitations

Please check original model card for model usage [limitations](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1#limitations)