|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- Transformer |
|
|
- Tracking |
|
|
- ONNX |
|
|
--- |
|
|
|
|
|
# MixFormerV2 |
|
|
|
|
|
This version of MixFormerV2 has been converted to run on the Axera NPU using **w8a16** quantization. |
|
|
|
|
|
This model has been optimized with the following LoRA: |
|
|
|
|
|
Compatible with Pulsar2 version: 3.4 |
|
|
|
|
|
## Convert tools links: |
|
|
|
|
|
For those who are interested in model conversion, you can try to export axmodel through |
|
|
|
|
|
- [The repo of original](https://github.com/MCG-NJU/MixFormerV2) |
|
|
|
|
|
- [The repo of AXera Platform](https://github.com/Jordan-5i/ax650_mixformer2_demo), which you can get the detial of guide |
|
|
|
|
|
- [Pulsar2 Link, How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html) |
|
|
|
|
|
|
|
|
## Support Platform |
|
|
|
|
|
- AX650 |
|
|
- [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) |
|
|
- [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) |
|
|
- AX630C |
|
|
- [爱芯派2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html) |
|
|
- [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM) |
|
|
- [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit) |
|
|
|
|
|
|Chips|npu1| |
|
|
|--|--| |
|
|
|AX650| 11 ms | |
|
|
|AX630C| 33 ms | |
|
|
|
|
|
## How to use |
|
|
|
|
|
Download all files from this repository to the device |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/MixFormerV2# tree -L 1 |
|
|
. |
|
|
├── ax650 |
|
|
├── car.avi |
|
|
├── config.json |
|
|
├── onnx |
|
|
├── README.md |
|
|
├── run_mixformer2_axmodel.py |
|
|
└── run_mixformer2_onnx.py |
|
|
``` |
|
|
|
|
|
### python env requirement |
|
|
|
|
|
#### pyaxengine |
|
|
|
|
|
https://github.com/AXERA-TECH/pyaxengine |
|
|
|
|
|
``` |
|
|
wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.1rc0/axengine-0.1.1-py3-none-any.whl |
|
|
pip install axengine-0.1.1-py3-none-any.whl |
|
|
``` |
|
|
|
|
|
#### others |
|
|
|
|
|
``` |
|
|
pip install argparse numpy opencv-python glob2 |
|
|
``` |
|
|
|
|
|
#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/ax650_mixformer2_demo# python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10 |
|
|
[INFO] Available providers: ['AxEngineExecutionProvider'] |
|
|
[INFO] Using provider: AxEngineExecutionProvider |
|
|
[INFO] Chip type: ChipType.MC50 |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Engine version: 2.7.2a |
|
|
[INFO] Model type: 0 (single core) |
|
|
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty |
|
|
====================type================= [1079, 482] <class 'list'> <class 'list'> |
|
|
第一帧初始化完毕！ |
|
|
Video: tracking 246.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Video: tracking 4.0fps |
|
|
Reached the maximum number of frames (10). Exiting loop. |
|
|
video: average finale average tracking fps 31.8 fps |
|
|
root@ax650:/mnt/qtang/ax650_mixformer2_demo# |
|
|
``` |
|
|
|
|
|
#### Inference with M.2 Accelerator card |
|
|
|
|
|
[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5. |
|
|
|
|
|
``` |
|
|
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $ python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10 |
|
|
[INFO] Available providers: ['AXCLRTExecutionProvider'] |
|
|
[INFO] Using provider: AXCLRTExecutionProvider |
|
|
[INFO] SOC Name: AX650N |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty |
|
|
====================type================= [1079, 482] <class 'list'> <class 'list'> |
|
|
第一帧初始化完毕！ |
|
|
Video: tracking 925.0fps |
|
|
Video: tracking 12.0fps |
|
|
Video: tracking 12.0fps |
|
|
Video: tracking 11.0fps |
|
|
Video: tracking 11.0fps |
|
|
Video: tracking 11.0fps |
|
|
Video: tracking 11.0fps |
|
|
Video: tracking 11.0fps |
|
|
Video: tracking 10.0fps |
|
|
Video: tracking 10.0fps |
|
|
Video: tracking 10.0fps |
|
|
Reached the maximum number of frames (10). Exiting loop. |
|
|
video: average finale average tracking fps 114.9 fps |
|
|
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $ |
|
|
``` |
