|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- openbmb/MiniCPM4-0.5B |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- minicpm4 |
|
|
- int8 |
|
|
--- |
|
|
|
|
|
|
|
|
# MiniCPM4-0.5B-Int8 |
|
|
|
|
|
This version of MiniCPM4-0.5B has been converted to run on the Axera NPU using **w8a16** quantization. |
|
|
|
|
|
This model has been optimized with the following LoRA: |
|
|
|
|
|
Compatible with Pulsar2 version: 4.2(Not released yet) |
|
|
|
|
|
## Convert tools links: |
|
|
|
|
|
For those who are interested in model conversion, you can try to export axmodel through the original repo : |
|
|
https://huggingface.co/openbmb/MiniCPM4-0.5B |
|
|
|
|
|
[Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) |
|
|
|
|
|
[AXera NPU LLM Runtime](https://github.com/AXERA-TECH/ax-llm) |
|
|
|
|
|
|
|
|
## Support Platform |
|
|
|
|
|
- AX650 |
|
|
- [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) |
|
|
- [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) |
|
|
- AX630C |
|
|
- [爱芯派2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html) |
|
|
- [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM) |
|
|
- [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit) |
|
|
|
|
|
|Chips|w8a16|w4a16| |
|
|
|--|--|--| |
|
|
|AX650| 36 tokens/sec|TBD| |
|
|
|AX630C| 12 tokens/sec|TBD| |
|
|
|
|
|
## How to use |
|
|
|
|
|
Download all files from this repository to the device |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# tree -L 1 |
|
|
. |
|
|
|-- main_ax650 |
|
|
|-- main_axcl_aarch64 |
|
|
|-- main_axcl_x86 |
|
|
|-- minicpm4-0.5b-int8-ctx-ax650 |
|
|
|-- minicpm4_tokenizer |
|
|
|-- minicpm4_tokenizer_uid.py |
|
|
|-- post_config.json |
|
|
|-- run_minicpm4_0.5b_int8_ctx_ax650.sh |
|
|
`-- run_minicpm4_0.5b_int8_ctx_axcl_x86.sh |
|
|
2 directories, 7 files |
|
|
``` |
|
|
|
|
|
#### Start the Tokenizer service |
|
|
|
|
|
Install requirement |
|
|
|
|
|
``` |
|
|
pip install transformers jinja2 |
|
|
``` |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# python3 minicpm4_tokenizer_uid.py |
|
|
Server running at http://0.0.0.0:12345 |
|
|
``` |
|
|
|
|
|
#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board |
|
|
|
|
|
Open another terminal and run `run_minicpm4_0.5b_int8_ctx_ax650.sh` |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# ./run_minicpm4_0.5b_int8_ctx_ax650.sh |
|
|
[I][ Init][ 110]: LLM init start |
|
|
[I][ Init][ 34]: connect http://127.0.0.1:12345 ok |
|
|
[I][ Init][ 57]: uid: c779ded0-ff14-4877-869b-1aacc948f2d8 |
|
|
bos_id: 1, eos_id: 73440 |
|
|
100% | ████████████████████████████████ | 27 / 27 [2.53s<2.53s, 10.67 count/s] init post axmodel ok,remain_cmm(4244 MB) |
|
|
[I][ Init][ 188]: max_token_len : 1023 |
|
|
[I][ Init][ 193]: kv_cache_size : 128, kv_cache_num: 1023 |
|
|
[I][ Init][ 201]: prefill_token_num : 128 |
|
|
[I][ Init][ 205]: grp: 1, prefill_max_token_num : 1 |
|
|
[I][ Init][ 205]: grp: 2, prefill_max_token_num : 128 |
|
|
[I][ Init][ 205]: grp: 3, prefill_max_token_num : 512 |
|
|
[I][ Init][ 209]: prefill_max_token_num : 512 |
|
|
[I][ load_config][ 282]: load config: |
|
|
{ |
|
|
"enable_repetition_penalty": false, |
|
|
"enable_temperature": false, |
|
|
"enable_top_k_sampling": true, |
|
|
"enable_top_p_sampling": false, |
|
|
"penalty_window": 20, |
|
|
"repetition_penalty": 1.2, |
|
|
"temperature": 0.9, |
|
|
"top_k": 1, |
|
|
"top_p": 0.8 |
|
|
} |
|
|
|
|
|
[I][ Init][ 218]: LLM init ok |
|
|
Type "q" to exit, Ctrl+c to stop current running |
|
|
[I][ GenerateKVCachePrefill][ 271]: input token num : 25, prefill_split_num : 1 prefill_grpid : 2 |
|
|
[I][ GenerateKVCachePrefill][ 308]: input_num_token:25 |
|
|
[I][ main][ 230]: precompute_len: 25 |
|
|
[I][ main][ 231]: system_prompt: You are MiniCPM4, created by ModelBest. You are a helpful assistant. |
|
|
prompt >> 你是谁? |
|
|
[I][ SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:25 input_num_token:12 |
|
|
[I][ SetKVCache][ 534]: current prefill_max_token_num:384 |
|
|
[I][ Run][ 660]: input token num : 12, prefill_split_num : 1 |
|
|
[I][ Run][ 686]: input_num_token:12 |
|
|
[I][ Run][ 829]: ttft: 147.65 ms |
|
|
你好，我是MiniCPM系列模型，由面壁智能和OpenBMB开源社区开发。详细信息请访问https://github.com/OpenBMB/ |
|
|
|
|
|
[N][ Run][ 943]: hit eos,avg 35.75 token/s |
|
|
|
|
|
[I][ GetKVCache][ 500]: precompute_len:162, remaining:350 |
|
|
prompt >> 9.9与9.11 |
|
|
[I][ SetKVCache][ 531]: prefill_grpid:3 kv_cache_num:512 precompute_len:162 input_num_token:17 |
|
|
[I][ SetKVCache][ 534]: current prefill_max_token_num:256 |
|
|
[I][ Run][ 660]: input token num : 17, prefill_split_num : 1 |
|
|
[I][ Run][ 686]: input_num_token:17 |
|
|
[I][ Run][ 829]: ttft: 274.38 ms |
|
|
9.9比9.11大。 |
|
|
|
|
|
[N][ Run][ 943]: hit eos,avg 35.44 token/s |
|
|
|
|
|
[I][ GetKVCache][ 500]: precompute_len:189, remaining:323 |
|
|
prompt >> q |
|
|
root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# |
|
|
``` |
