|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: |
|
|
- black-forest-labs/FLUX.1-dev |
|
|
base_model_relation: quantized |
|
|
--- |
|
|
|
|
|
# Elastic model: Fastest self-serving models. FLUX.1-schnell. |
|
|
|
|
|
Elastic models are the models produced by TheStage AI ANNA: Automated Neural Networks Accelerator. ANNA allows you to control model size, latency and quality with a simple slider movement. For each model, ANNA produces a series of optimized models: |
|
|
|
|
|
* __XL__: Mathematically equivalent neural network, optimized with our DNN compiler. |
|
|
|
|
|
* __L__: Near lossless model, with less than 1% degradation obtained on corresponding benchmarks. |
|
|
|
|
|
* __M__: Faster model, with accuracy degradation less than 1.5%. |
|
|
|
|
|
* __S__: The fastest model, with accuracy degradation less than 2%. |
|
|
|
|
|
|
|
|
__Goals of Elastic Models:__ |
|
|
|
|
|
* Provide the fastest models and service for self-hosting. |
|
|
* Provide flexibility in cost vs quality selection for inference. |
|
|
* Provide clear quality and latency benchmarks. |
|
|
* Provide interface of HF libraries: transformers and diffusers with a single line of code. |
|
|
* Provide models supported on a wide range of hardware, which are pre-compiled and require no JIT. |
|
|
|
|
|
> It's important to note that specific quality degradation can vary from model to model. For instance, with an S model, you can have 0.5% degradation as well. |
|
|
|
|
|
----- |
|
|
|
|
|
 |
|
|
 |
|
|
|
|
|
## Inference |
|
|
|
|
|
Currently, our demo model only supports 1024x1024 outputs without batching. This will be updated in the near future. |
|
|
To infer our models, you just need to replace `diffusers` import with `elastic_models.diffusers`: |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from elastic_models.diffusers import FluxPipeline |
|
|
|
|
|
mode_name = 'black-forest-labs/FLUX.1-dev' |
|
|
hf_token = '' |
|
|
device = torch.device("cuda") |
|
|
|
|
|
pipeline = FluxPipeline.from_pretrained( |
|
|
mode_name, |
|
|
torch_dtype=torch.bfloat16, |
|
|
token=hf_token |
|
|
mode='S' |
|
|
) |
|
|
pipeline.to(device) |
|
|
|
|
|
prompts = ["Kitten eating a banana"] |
|
|
output = pipeline(prompt=prompts) |
|
|
|
|
|
for prompt, output_image in zip(prompts, output.images): |
|
|
output_image.save((prompt.replace(' ', '_') + '.png')) |
|
|
``` |
|
|
|
|
|
### Installation |
|
|
|
|
|
|
|
|
__System requirements:__ |
|
|
* GPUs: H100 |
|
|
* CPU: AMD, Intel |
|
|
* Python: 3.10-3.12 |
|
|
|
|
|
|
|
|
To work with our models just run these lines in your terminal: |
|
|
|
|
|
```shell |
|
|
pip install thestage |
|
|
pip install elastic_models==0.0.3\ |
|
|
--index-url https://thestage.jfrog.io/artifactory/api/pypi/pypi-thestage-ai-production/simple\ |
|
|
--extra-index-url https://pypi.nvidia.com\ |
|
|
--extra-index-url https://pypi.org/simple |
|
|
|
|
|
pip install flash_attn==2.7.3 --no-build-isolation |
|
|
pip uninstall apex |
|
|
echo "{ |
|
|
"meta-llama/Llama-3.2-1B-Instruct": 6, |
|
|
"mistralai/Mistral-7B-Instruct-v0.3": 7, |
|
|
"black-forest-labs/FLUX.1-schnell": 1, |
|
|
"black-forest-labs/FLUX.1-dev": 5 |
|
|
}" > model_name_id.json |
|
|
export ELASTIC_MODEL_ID_MAPPING=./model_name_id.json |
|
|
``` |
|
|
|
|
|
Then go to [app.thestage.ai](https://app.thestage.ai), login and generate API token from your profile page. Set up API token as follows: |
|
|
|
|
|
```shell |
|
|
thestage config set --api-token <YOUR_API_TOKEN> |
|
|
``` |
|
|
|
|
|
Congrats, now you can use accelerated models! |
|
|
|
|
|
---- |
|
|
|
|
|
## Benchmarks |
|
|
|
|
|
Benchmarking is one of the most important procedures during model acceleration. We aim to provide clear performance metrics for models using our algorithms. |
|
|
|
|
|
### Quality benchmarks |
|
|
|
|
|
For quality evaluation we have used: PSNR, SSIM and CLIP score. PSNR and SSIM were computed using outputs of original model. |
|
|
| Metric/Model | S | M | L | XL | Original | |
|
|
|---------------|---|---|---|----|----------| |
|
|
| PSNR | 29.9 | 30.2 | 31 | inf | inf | |
|
|
| SSIM | 0.66 | 0.71 | 0.86 | 1.0 | 1.0 | |
|
|
| CLIP | 11.5 | 11.6 | 11.8 | 11.9 | 11.9| |
|
|
|
|
|
|
|
|
### Latency benchmarks |
|
|
|
|
|
Time in seconds to generate one image 1024x1024 |
|
|
| GPU/Model | S | M | L | XL | Original | |
|
|
|-----------|-----|---|---|----|----------| |
|
|
| H100 | 0.5 | 0.58 | 0.65 | 0.75 | 1.05 | |
|
|
| L40s | 1.4 | 1.6 | 1.9 | 2.1 | 2.5| |
|
|
|
|
|
|
|
|
## Links |
|
|
|
|
|
* __Platform__: [app.thestage.ai](https://app.thestage.ai) |
|
|
<!-- * __Elastic models Github__: [app.thestage.ai](app.thestage.ai) --> |
|
|
* __Subscribe for updates__: [TheStageAI X](https://x.com/TheStageAI) |
|
|
* __Contact email__: [email protected] |
