File size: 8,208 Bytes
03a2d97 dca15e9 03a2d97 94c7e10 03a2d97 44e9a1d 03a2d97 94c7e10 03a2d97 dca15e9 03a2d97 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 |
---
license: mit
language:
- en
- zh
base_model:
- OFA-Sys/chinese-clip-vit-large-patch14-336px
- AXERA-TECH/cnclip
tags:
- CLIP
- CN_CLIP
pipeline_tag: zero-shot-image-classification
---
# LibCLIP
This SDK enables efficient text-to-image retrieval using CLIP (Contrastive Language–Image Pretraining), optimized for Axera’s NPU-based SoC platforms including AX650, AX650C, AX8850, and AX650A, or Axera's dedicated AI accelerator.
With this SDK, you can:
- Perform semantic image search by providing natural language queries.
- Utilize CLIP to embed text queries and compare them against a pre-computed set of image embeddings.
- Run all inference processes directly on Axera NPUs for low-latency, high-throughput performance at the edge.
This solution is well-suited for smart cameras, content filtering, AI-powered user interfaces, and other edge AI scenarios where natural language-based image retrieval is required.
## References links:
For those who are interested in model conversion, you can try to export axmodel through
- [The github repo of libclip's open source](https://github.com/AXERA-TECH/libclip.axera)
- [Pulsar2 Link, How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html)
- https://huggingface.co/AXERA-TECH/cnclip
## Support Platform
- AX650
- [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
- [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
## Performance
| Model | Input Shape | Latency (ms) | CMM Usage (MB) |
| ----------------------------------------- | ----------------- | ------------ | -------------- |
| cnclip_vit_l14_336px_vision_u16u8.axmodel | 1 x 3 x 336 x 336 | 88.475 ms | 304 MB |
| cnclip_vit_l14_336px_text_u16.axmodel | 1 x 52 | 4.576 ms | 122 MB |
## How to use
Download all files from this repository to the device
```
(base) axera@raspberrypi:~/samples/AXERA-TECH/libclip.axera $ tree -L 2
.
├── cnclip
│ ├── cnclip_vit_l14_336px_text_u16.axmodel
│ ├── cnclip_vit_l14_336px_vision_u16u8.axmodel
│ └── cn_vocab.txt
├── coco_1000.tar
├── config.json
├── gradio_01.png
├── install
│ ├── examples
│ ├── include
│ └── lib
├── pyclip
│ ├── example.py
│ ├── gradio_example.png
│ ├── gradio_example.py
│ ├── libclip.so
│ ├── __pycache__
│ ├── pyclip.py
│ └── requirements.txt
└── README.md
8 directories, 13 files
```
### python env requirement
```
pip install -r pyclip/requirements.txt
```
#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro)
```
root@ax650:~/sample/LibClip# cp ./install/lib/host_650/libclip.so ./pyclip/
root@ax650:~/sample/LibClip# tar -xf coco_1000.tar
root@ax650:~/sample/LibClip# python3 pyclip/gradio_example.py --ienc cnclip/cnclip_vit_l14_336px_vision_u16u8.axmodel --tenc cnclip/cnclip_vit_l14_336px_text_u16.axmodel --vocab cnclip/cn_vocab.txt --isCN 1 --db_path clip_feat_db_coco --image_folder coco_1000/ --dev_type host
Trying to load: /root/sample/LibClip/pyclip/aarch64/libclip.so
❌ Failed to load: /root/sample/LibClip/pyclip/aarch64/libclip.so
/root/sample/LibClip/pyclip/aarch64/libclip.so: cannot open shared object file: No such file or directory
🔍 File not found. Please verify that libclip.so exists and the path is correct.
Trying to load: /root/sample/LibClip/pyclip/libclip.so
open libaxcl_rt.so failed
unsupport axcl
✅ Successfully loaded: /root/sample/LibClip/pyclip/libclip.so
sh: line 1: axcl-smi: command not found
可用设备: {'host': {'available': True, 'version': 'V3.6.2_20250731140456', 'mem_info': {'remain': 9963, 'total': 10240}}, 'devices': {'host_version': '', 'dev_version': '', 'count': 0, 'devices_info': []}}
input size: 1
name: image [unknown] [unknown]
1 x 3 x 336 x 336
output size: 1
name: unnorm_image_features
1 x 768
[I][ load_image_encoder][ 50]: nchw 336 336
[I][ load_image_encoder][ 60]: image feature len 768
input size: 1
name: text [unknown] [unknown]
1 x 52
output size: 1
name: unnorm_text_features
1 x 768
[I][ load_text_encoder][ 44]: text feature len 768
[I][ load_tokenizer][ 60]: text token len 52
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [01:43<00:00, 9.70it/s]
* Running on local URL: http://0.0.0.0:7860
```
If your M4N-Dock(爱芯派Pro) IP Address is 192.168.1.100, so using this URL `http://192.168.1.100:7860` with your WebApp
#### Inference with M.2 Accelerator card
[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5.
```
(py312) axera@raspberrypi:~/samples/AXERA-TECH/libclip.axera $ export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libstdc++.so.6
(py312) axera@raspberrypi:~/samples/AXERA-TECH/libclip.axera $ cp install/lib/aarch64/libclip.so pyclip/
(py312) axera@raspberrypi:~/samples/AXERA-TECH/libclip.axera $ tar xf coco_1000.tar
(py312) axera@raspberrypi:~/samples/AXERA-TECH/libclip.axera $ python pyclip/gradio_example.py --ienc cnclip/cnclip_vit_l14_336px_vision_u16u8.axmodel --tenc cnclip/cnclip_vit_l14_336px_text_u16.axmodel --vocab cnclip/cn_vocab.txt --isCN 1 --db_path clip_feat_db_coco --image_folder coco_1000/ --dev_type axcl
Trying to load: /home/axera/samples/AXERA-TECH/libclip.axera/pyclip/aarch64/libclip.so
❌ Failed to load: /home/axera/samples/AXERA-TECH/libclip.axera/pyclip/aarch64/libclip.so
/home/axera/samples/AXERA-TECH/libclip.axera/pyclip/aarch64/libclip.so: cannot open shared object file: No such file or directory
🔍 File not found. Please verify that libclip.so exists and the path is correct.
Trying to load: /home/axera/samples/AXERA-TECH/libclip.axera/pyclip/libclip.so
open libax_sys.so failed
open libax_engine.so failed
✅ Successfully loaded: /home/axera/samples/AXERA-TECH/libclip.axera/pyclip/libclip.so
可用设备: {'host': {'available': True, 'version': '', 'mem_info': {'remain': 0, 'total': 0}}, 'devices': {'host_version': 'V3.6.2_20250603154858', 'dev_version': 'V3.6.2_20250603154858', 'count': 1, 'devices_info': [{'temp': 37, 'cpu_usage': 1, 'npu_usage': 0, 'mem_info': {'remain': 7022, 'total': 7040}}]}}
[I][ run][ 31]: AXCLWorker start with devid 0
input size: 1
name: image [unknown] [unknown]
1 x 3 x 336 x 336
output size: 1
name: unnorm_image_features
1 x 768
[I][ load_image_encoder][ 50]: nchw 336 336
[I][ load_image_encoder][ 60]: image feature len 768
input size: 1
name: text [unknown] [unknown]
1 x 52
output size: 1
name: unnorm_text_features
1 x 768
[I][ load_text_encoder][ 44]: text feature len 768
[I][ load_tokenizer][ 60]: text token len 52
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [01:40<00:00, 9.93it/s]
* Running on local URL: http://0.0.0.0:7860
```
If your Raspberry PI 5 IP Address is 192.168.1.100, so using this URL `http://192.168.1.100:7860` with your WebApp.
 |