Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -16,7 +16,7 @@ tags:
|
|
| 16 |
|
| 17 |
On-device, high-resolution image synthesis from text and image prompts. ControlNet guides Stable-diffusion with provided input image to generate accurate images from given input prompt.
|
| 18 |
|
| 19 |
-
This model is an implementation of ControlNet found [here](
|
| 20 |
This repository provides scripts to run ControlNet on Qualcomm® devices.
|
| 21 |
More details on model performance across various devices, can be found
|
| 22 |
[here](https://aihub.qualcomm.com/models/controlnet_quantized).
|
|
@@ -34,17 +34,23 @@ More details on model performance across various devices, can be found
|
|
| 34 |
- ControlNet Number of parameters: 361M
|
| 35 |
- Model size: 1.4GB
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
|
| 39 |
|
| 40 |
-
| Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model
|
| 41 |
-
| ---|---|---|---|---|---|---|---|
|
| 42 |
-
| Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 11.394 ms | 0 - 74 MB | UINT16 | NPU | [TextEncoder_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/TextEncoder_Quantized.bin)
|
| 43 |
-
| Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 262.52 ms | 11 - 17 MB | UINT16 | NPU | [UNet_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/UNet_Quantized.bin)
|
| 44 |
-
| Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 390.243 ms | 0 - 36 MB | UINT16 | NPU | [VAEDecoder_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/VAEDecoder_Quantized.bin)
|
| 45 |
-
| Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 100.33 ms | 2 - 68 MB | UINT16 | NPU | [ControlNet_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/ControlNet_Quantized.bin)
|
| 46 |
-
|
| 47 |
-
|
| 48 |
|
| 49 |
## Installation
|
| 50 |
|
|
@@ -100,37 +106,43 @@ device. This script does the following:
|
|
| 100 |
```bash
|
| 101 |
python -m qai_hub_models.models.controlnet_quantized.export
|
| 102 |
```
|
| 103 |
-
|
| 104 |
```
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
Estimated
|
| 130 |
-
|
| 131 |
-
Compute
|
| 132 |
-
|
| 133 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
```
|
| 135 |
|
| 136 |
|
|
@@ -254,15 +266,19 @@ provides instructions on how to use the `.so` shared library or `.bin` context b
|
|
| 254 |
Get more details on ControlNet's performance across various devices [here](https://aihub.qualcomm.com/models/controlnet_quantized).
|
| 255 |
Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
|
| 256 |
|
|
|
|
| 257 |
## License
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
|
|
|
| 261 |
|
| 262 |
## References
|
| 263 |
* [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543)
|
| 264 |
* [Source Model Implementation](https://github.com/lllyasviel/ControlNet)
|
| 265 |
|
|
|
|
|
|
|
| 266 |
## Community
|
| 267 |
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
|
| 268 |
* For questions or feedback please [reach out to us](mailto:[email protected]).
|
|
|
|
| 16 |
|
| 17 |
On-device, high-resolution image synthesis from text and image prompts. ControlNet guides Stable-diffusion with provided input image to generate accurate images from given input prompt.
|
| 18 |
|
| 19 |
+
This model is an implementation of ControlNet found [here]({source_repo}).
|
| 20 |
This repository provides scripts to run ControlNet on Qualcomm® devices.
|
| 21 |
More details on model performance across various devices, can be found
|
| 22 |
[here](https://aihub.qualcomm.com/models/controlnet_quantized).
|
|
|
|
| 34 |
- ControlNet Number of parameters: 361M
|
| 35 |
- Model size: 1.4GB
|
| 36 |
|
| 37 |
+
| Model | Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model
|
| 38 |
+
|---|---|---|---|---|---|---|---|---|
|
| 39 |
+
| TextEncoder_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 11.394 ms | 0 - 74 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/TextEncoder_Quantized.bin) |
|
| 40 |
+
| TextEncoder_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 8.08 ms | 0 - 137 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/TextEncoder_Quantized.bin) |
|
| 41 |
+
| TextEncoder_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 10.982 ms | 0 - 1 MB | UINT16 | NPU | Use Export Script |
|
| 42 |
+
| UNet_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 262.52 ms | 11 - 17 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/UNet_Quantized.bin) |
|
| 43 |
+
| UNet_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 192.789 ms | 3 - 1247 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/UNet_Quantized.bin) |
|
| 44 |
+
| UNet_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 260.158 ms | 14 - 15 MB | UINT16 | NPU | Use Export Script |
|
| 45 |
+
| VAEDecoder_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 390.243 ms | 0 - 36 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/VAEDecoder_Quantized.bin) |
|
| 46 |
+
| VAEDecoder_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 294.404 ms | 0 - 88 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/VAEDecoder_Quantized.bin) |
|
| 47 |
+
| VAEDecoder_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 379.548 ms | 0 - 1 MB | UINT16 | NPU | Use Export Script |
|
| 48 |
+
| ControlNet_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 100.33 ms | 2 - 68 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/ControlNet_Quantized.bin) |
|
| 49 |
+
| ControlNet_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 76.94 ms | 0 - 533 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/ControlNet_Quantized.bin) |
|
| 50 |
+
| ControlNet_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 103.52 ms | 2 - 3 MB | UINT16 | NPU | Use Export Script |
|
| 51 |
|
| 52 |
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
## Installation
|
| 56 |
|
|
|
|
| 106 |
```bash
|
| 107 |
python -m qai_hub_models.models.controlnet_quantized.export
|
| 108 |
```
|
|
|
|
| 109 |
```
|
| 110 |
+
Profiling Results
|
| 111 |
+
------------------------------------------------------------
|
| 112 |
+
TextEncoder_Quantized
|
| 113 |
+
Device : Samsung Galaxy S23 (13)
|
| 114 |
+
Runtime : QNN
|
| 115 |
+
Estimated inference time (ms) : 11.4
|
| 116 |
+
Estimated peak memory usage (MB): [0, 74]
|
| 117 |
+
Total # Ops : 570
|
| 118 |
+
Compute Unit(s) : NPU (570 ops)
|
| 119 |
+
|
| 120 |
+
------------------------------------------------------------
|
| 121 |
+
UNet_Quantized
|
| 122 |
+
Device : Samsung Galaxy S23 (13)
|
| 123 |
+
Runtime : QNN
|
| 124 |
+
Estimated inference time (ms) : 262.5
|
| 125 |
+
Estimated peak memory usage (MB): [11, 17]
|
| 126 |
+
Total # Ops : 5434
|
| 127 |
+
Compute Unit(s) : NPU (5434 ops)
|
| 128 |
+
|
| 129 |
+
------------------------------------------------------------
|
| 130 |
+
VAEDecoder_Quantized
|
| 131 |
+
Device : Samsung Galaxy S23 (13)
|
| 132 |
+
Runtime : QNN
|
| 133 |
+
Estimated inference time (ms) : 390.2
|
| 134 |
+
Estimated peak memory usage (MB): [0, 36]
|
| 135 |
+
Total # Ops : 409
|
| 136 |
+
Compute Unit(s) : NPU (409 ops)
|
| 137 |
+
|
| 138 |
+
------------------------------------------------------------
|
| 139 |
+
ControlNet_Quantized
|
| 140 |
+
Device : Samsung Galaxy S23 (13)
|
| 141 |
+
Runtime : QNN
|
| 142 |
+
Estimated inference time (ms) : 100.3
|
| 143 |
+
Estimated peak memory usage (MB): [2, 68]
|
| 144 |
+
Total # Ops : 2406
|
| 145 |
+
Compute Unit(s) : NPU (2406 ops)
|
| 146 |
```
|
| 147 |
|
| 148 |
|
|
|
|
| 266 |
Get more details on ControlNet's performance across various devices [here](https://aihub.qualcomm.com/models/controlnet_quantized).
|
| 267 |
Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
|
| 268 |
|
| 269 |
+
|
| 270 |
## License
|
| 271 |
+
* The license for the original implementation of ControlNet can be found [here](https://github.com/lllyasviel/ControlNet/blob/main/LICENSE).
|
| 272 |
+
* The license for the compiled assets for on-device deployment can be found [here](https://github.com/lllyasviel/ControlNet/blob/main/LICENSE)
|
| 273 |
+
|
| 274 |
+
|
| 275 |
|
| 276 |
## References
|
| 277 |
* [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543)
|
| 278 |
* [Source Model Implementation](https://github.com/lllyasviel/ControlNet)
|
| 279 |
|
| 280 |
+
|
| 281 |
+
|
| 282 |
## Community
|
| 283 |
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
|
| 284 |
* For questions or feedback please [reach out to us](mailto:[email protected]).
|