Replace checkpoint with MSE-scale INT4 weights

README.md

@@ -12,49 +12,62 @@ tags:
 
 # DeepSeek V4 Flash dsv4_int INT4/INT8
 
-This checkpoint is experimental and under active development. It is intended
-for the AppMana Ampere vLLM fork and is not a general-purpose Hugging Face
-Transformers checkpoint.
+This checkpoint is for the AppMana Ampere vLLM fork:
 
-Generated on 2026-05-11 from clean source checkpoint:
+https://github.com/AppMana/forks-vllm-ampere
 
-`deepseek-ai/DeepSeek-V4-Flash@fd53f944496234770ba80e15004f9b6d269a71f5`
+The container image used for serving is published from that fork.
 
-Conversion command:
+This checkpoint sets:
 
-```bash
-CUDA_VISIBLE_DEVICES=1 python tools/ampere/dsv4_requant_checkpoint.py \
-  --src /home/administrator/inference/.cache/huggingface/models--deepseek-ai--DeepSeek-V4-Flash/snapshots/fd53f944496234770ba80e15004f9b6d269a71f5 \
-  --dst /home/administrator/inference/deepseek-v4-flash-dsv4-int-channel-vllm \
-  --device cuda:0 \
-  --dense-int8-strategy channel \
-  --overwrite
+```json
+"__experimental_enable_imma_from_https://github.com/appMana/forks-vllm-ampere": true
 ```
 
-Quantization format:
-
-- Routed MoE experts: MXFP4 source weights converted to packed symmetric INT4
-  W4A16, group size 32, for Marlin MoE.
-- Dense FP8 linears: channelwise biased UINT8 W8A16 format for the Ampere
-  AllSpark path where supported.
-- Preserved precision: embeddings, norms, gates, attention sinks, HC tensors,
-  and other tensors marked BF16/F32 in the source.
-
-Structural audit:
-
-- Safetensor shards: 46
-- Size: about 157 GiB
-- Expert INT4 tensors: 33,792
-- Dense INT8 tensors: 375
-- Preserved tensors: 853
-- Missing expert scales: 0
-
-Known status:
-
-- The 2-layer version of this conversion path loads and generates locally under
-  vLLM with compile and CUDA graphs enabled.
-- The full 43-layer checkpoint has been converted and structurally audited, but
-  must still pass a full distributed vLLM load/generation test before treating
-  it as usable.
-- Quality/perplexity is not yet validated. Do not assume this matches the
-  original FP4/FP8 DeepSeek checkpoint until evaluation says so.
+That field is read by the AppMana fork to select the IMMA INT8 paths without
+requiring deployment-specific environment variables.
+
+## Quantization
+
+Routed expert weights are converted ahead of time from the native DeepSeek-V4
+MXFP4 expert tensors to INT4 W4A16 for Ampere.
+
+For each expert weight tensor:
+
+1. MXFP4 bytes are unpacked into e2m1 values.
+2. The native e8m0 scale is applied to recover FP32 group values.
+3. Values are grouped by 32 along the last dimension.
+4. For `scale_mode="mse"`, the converter tries per-group candidate scales
+   `abs_max / div`, with `div` from `5.0` to `9.5`.
+5. Each candidate scale is rounded to BF16 before scoring.
+6. For each candidate, INT4 codes are recomputed with
+   `round(group / scale).clamp(-8, 7)`.
+7. The selected scale is the candidate that minimizes
+   `sum((q * scale - group) ** 2)` for that group.
+
+"MSE scale" means the scale is part of the round-trip quantize/dequantize error
+being minimized. It is not a fixed-scale integer-code MSE, and it is not a
+closed-form least-squares refit after choosing codes.
+
+Stored expert format:
+
+- signed INT4 values encoded as uint4b8 nibbles (`code + 8`)
+- BF16 per-group scales
+- group size 32
+
+Dense/shared expert/attention FP8 weights are converted to the AppMana
+INT8 W8A16 path. Where applicable, the stored tensor is channelwise biased
+UINT8 plus scale metadata.
+
+## Inference implementation
+
+| Stage | Technology | Source |
+| --- | --- | --- |
+| Routed expert GEMM | Marlin MoE W4A16 INT4 | vLLM upstream path with AppMana dsv4_int checkpoint loader in https://github.com/AppMana/forks-vllm-ampere |
+| Dense and shared-expert GEMM | AllSpark W8A16 INT8 | AppMana integration in https://github.com/AppMana/forks-vllm-ampere |
+| Sparse MLA prefill/decode attention | AppMana CUDA Flash MLA / Sparse MLA kernel | https://github.com/AppMana/forks-flash-mla-ampere-dsv4 |
+| KV transfer | LMCache connector sidecar | https://github.com/AppMana/forks-lmcache |
+
+The quantization loader and conversion metadata are implemented in:
+
+https://github.com/AppMana/forks-vllm-ampere/blob/appmana/vllm-ampere/vllm/model_executor/layers/quantization/dsv4_int.py

config.json

@@ -2,6 +2,7 @@
   "architectures": [
     "DeepseekV4ForCausalLM"
   ],
+  "__experimental_enable_imma_from_https://github.com/appMana/forks-vllm-ampere": true,
   "attention_bias": false,
   "attention_dropout": 0.0,
   "bos_token_id": 0,
@@ -41,9 +42,11 @@
         "weights": {
           "num_bits": 4,
           "type": "int",
+          "format": "int4",
           "symmetric": true,
           "group_size": 32,
-          "strategy": "group"
+          "strategy": "group",
+          "scale_mode": "mse"
         },
         "input_activations": {
           "num_bits": 16,

model-00002-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f45cbe538d1e8a19d50961f2bf03d782262b96f7f9b481a862349fc9e4bc7728
+oid sha256:9dd8ce3de11ade03e7d78563fee3664e05c34abbc215e8fb1902da09cfd93ce3
 size 3767747288

model-00003-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:78ba8ca99ca9dfb140782bc25d59b9338e53c3954db090d130e6e0198141f324
+oid sha256:e3808cfb0e684ddf91087cae2166988b53b1cddc62251778acf2ef13f669217e
 size 3767747288

model-00004-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:720321797f7102e20aa3e878891ca9a698d6fd6263b814525dbd81e3c8ca3a83
+oid sha256:e0d779af2432aa43420711952feb9e502ec19d1e78acad5e24d29fc15d12a4cf
 size 3797671160

model-00005-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:722c35c96b225a897e34ab6f564d2632c56406976a9a4e944e3e7c43d40ae4ca
+oid sha256:59fa851f475e1308677521e72e68cb30e4c622df9a71f413d53feaf869e80bbe
 size 3770195072

model-00006-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3721dd0544a9faabe7d14737b707201e203d4cd171618fd4ce44790c090aa86e
+oid sha256:1deb86475844e4f3d02ab230bd35dee295da032cdb302002225faff369b14bdc
 size 3791466688

model-00007-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3971de8cb60e0872f55a2888f2d1e3c8ebf3b7fd9edc098c250a5103a17c338e
+oid sha256:e877d4fbe3a7e303e47c33d0f6454631bf84cc5da3d08e65828273ef26ff8031
 size 3770195072

model-00008-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c0af3d7ba6c5b728d2a63c14381a56fb96ff75b6adf65993fc6d41c757ad6d21
+oid sha256:cc688810f3445299ccab4a3ce0a30bbe4dcc12b997c6e77e75dc208769cb4453
 size 3791466688

model-00009-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8a41cc2ad488b25d5bdd8017714d6dfb3c1a8a6f21be29a608a3dd8659e4643b
+oid sha256:f65a304119c1f14e1d86702f4ab7e78d453f8035b817563d85d11f82d86e326d
 size 3770195072

model-00010-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bf4729bf5f615cf6bd28d1104693fa76d5fd1404301328291b96d96d204173db
+oid sha256:b2198313ffc4f986d32dc7e3efcd69ec1990a11d5b565d72bc5d302369024d22
 size 3791466688

model-00011-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b85ae61378e6d2df39e80422c0228c024017b2c94c3fed5c142829fd4d380ab2
+oid sha256:deb060e0faece0766ff5e707214900be071bb5c45b319157b6faf28a9528656c
 size 3770195072

model-00012-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0c73491ce42308463caa6857a8d698c532b63006457e1dc024dc2e8cbe4b517e
+oid sha256:0971b02fd36cc5832df7c118b8ba4f6c4bf8fba216f707b0c72d70488a968a4a
 size 3791468264

model-00013-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2654097586c779b4fc230c572c853c3d119807da4875ce8f60a51281aeecbb4
+oid sha256:f03ea98b1ef959ce55c57d4a9c6350837ba40dbbda7b289cc48d77503a319b5c
 size 3770196640

model-00014-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e0286ce0a146713824e57fb605165d9d6231bf815b81515ece69e9155a0bf95f
+oid sha256:eb6896b22d2c9ef5f5b8b8349988130e70590f1828a5cf4b432ad72339c4c307
 size 3791468264

model-00015-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b278dd56dc39a44260232dac17d7e6de2e5b1884258b88684dd8d7ee0e53e6e4
+oid sha256:1e85195600449cd096c2a2b7b033a1d1a2c824a980a0ed1ac01404ba36b28644
 size 3770196640

model-00016-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d28761c1af0f92a3be94e1fa65a7c9f0f254ca78d5b04cd054f7c493badffd78
+oid sha256:cbcdc08615451cc07a2699435e14c42ad71f922899cb3c95c1a22b0f4a9b18f7
 size 3791468264

model-00017-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5498be5f84c500f50912608e82652508ff4242c7cd8913f449b2d089291b7d09
+oid sha256:eb6e89801414f9878f42c1c1dfebf7df7572748aeb6484641616edba6fd38411
 size 3770196640

model-00018-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:80a8785132e9566bc0889b86af88d4abbc68cd214d19f375701300966ca36701
+oid sha256:6217ff22b342c507dd4700f94754637a935ca08c735e7b67943fdc44e70d2f64
 size 3791468264

model-00019-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5887bf5234fde8ab4213a82c256b88c215144cf0b65d7436eebceaca7b103f13
+oid sha256:aaa19e074e301b689bd37225109f9b1497782613d2f8c8de502b5a8da9d7fe8f
 size 3770196640

model-00020-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:56d54639526a781c151b7414ce2d4125cdc945b4fc6911c6edd13ca3a8a49c2c
+oid sha256:98b49029634835dddab4db8aca29d7657e2e73087df6d812301ec33f81feaa19
 size 3791468264

model-00021-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4443e4f594a3124eb672fa5f1b7f2a184c6101d7349fb7a7754fef01e6fd4b37
+oid sha256:2a9f42a13f106a4de4c4a0958e3f7d3c516244a4f9ae7d2939a9ae2ff33daa4a
 size 3770196640

model-00022-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17eb4b7b83490c763e9b39c8c6d898436e872a96beeca5521cd245eadba5d117
+oid sha256:413492e41b25538605364567f2063fc1d358260bfec6c8dbdb83df32141c7e76
 size 3791468264

model-00023-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a2db4c6d73a87280d9571dc439be80aacbe6d09d3c971882f7e6e0a2f9a171e8
+oid sha256:363278e2a2587d8491431bc7031398ddb38ff9f77c879ced292f71ecb06677ba
 size 3770196640

model-00024-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00df91c2b43f7aff73e686ae0c1d6b31c12e769a46effed8507b756f84c779e0
+oid sha256:d0b40269eba8e7d42f06644c9b51e97ae4e297f50a63b644e2b1cd1aab630b9b
 size 3791468264

model-00025-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b173b7f4b3f07188f14de664ac0a1b21aa374dcb2a2a62b2b2a6d9f9969f428a
+oid sha256:ddb920cf2b430cb04c6845fa079de8173ab127ab85fac18e9e7256dfb59cfbbc
 size 3770196640

model-00026-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7dcd084df0deb4e8436d706d2d4b1d128b96c93a362bc39540bb7c9ee26f9394
+oid sha256:a9ffefe5ec18a6f96ca3d7b116f4a447be5afb798e3e1a9eb6c2e6682c9852bc
 size 3791468264

model-00027-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e67fc21051b363a6156ec16284fe76d8c1fbc850b1049f60795cb9d16d7fbeaf
+oid sha256:e3fe40c4b43aff787d5c83fff4c3b5e3aea285ba4e716cc134420f4d154182b2
 size 3770196640

model-00028-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c6803d59efbe77cf98e1dd13c17d694b9df935b4ce7f5886fc6649a283724817
+oid sha256:559392b24ba9a240903669bd46faf5fbea727072806296247215601dca869b3d
 size 3791468264

model-00029-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7cc85f809cc348d2cf49072a3e6b445fd213738ea406061459a1d1bcb4e413f1
+oid sha256:b02f9a99ce5413a6de329505d21439de0e3f0b9d473290abadb3495d2be196c7
 size 3770196640

model-00030-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b3ea8550f20060362e548b9c4375f4b629a11ac8c1ec7d6b4cdba05f2e3e0249
+oid sha256:7e77e20eb315aad00ee383a318f959159bd2b1724af5d186446cf45ab61193ed
 size 3791468264

model-00031-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a38fa8107df80e2cfe833cac511dab3624d4666a31d2b25770f8cd0512dd8372
+oid sha256:530100b976cf059200cc99fd694d6e4b1774a3e4e533b9c5da28d6585a6dd78a
 size 3770196640

model-00032-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3838abae1179e9ab050fa7dafb73992d4ba558dd1ea0b156203e66acc85f2703
+oid sha256:fc8234359e55099c84eb14b5862906d7d9a5cc770f8536bae9ed667e501dc9c5
 size 3791468264

model-00033-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:077b8d1bf5638483d22aa1793745aba79abe334f4c32833f5344aa8a4afd51aa
+oid sha256:596dc3c4f2c1a2f71f7cd88ee92bc6763a4e4bf3713b26f54545a16ff70ecb45
 size 3770196640

model-00034-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:27730b2daa791874def2712dbe9c40f0f9a782f2d41500d2948e7b7ad2a6ded9
+oid sha256:ac5f40cc2e00faf49b93c2de6963c65993deb69c80337f950680f8b6f40987dd
 size 3791468264

model-00035-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f195d6bdc1b0c00d64b7fc56950d6d190e144c1512481a16cebb85d314d1fad2
+oid sha256:07c3f263c9865bae132a92963334ab0757d4c02d85a06c65ff101085d59cd84e
 size 3770196640

model-00036-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:94107f1aa53421b6faa49e499772a3e7a1edac3a371e2459bc85282655803bbd
+oid sha256:3f0c74b24b43d593d4062970168e76e6c66fcaaca1d5176d1ea3539d4f0a930b
 size 3791468264

model-00037-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c12585ff57f1fc51ffc2c3e4cfb739815b7ca8eebab8785eea0245b4ace00bdb
+oid sha256:7e4028119d1f2fc4b468b2d99cb1dad7f2a5c159d599b2f75704d87d44dff96c
 size 3770196640

model-00038-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:993ed09b95c958bbcad1077555a4db3c681b9dfb885a5e9a27110fce70f84237
+oid sha256:d83c5427a5ab3e48e5fec220e969807fd260ad78b89f4c894b9b2e6b0c5357d8
 size 3791468264

model-00039-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2baa7b4b3ff96063b1f1fcf30c8ec3f94b78cfc8a267b39de4401f826d987b1c
+oid sha256:9745ef742176bb6fa4e09e28ce97f7768db79a7070058cbe509d9fe69d473c77
 size 3770196640

model-00040-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:52954e26522b004c69d1f2daef22a4b435c3677bcd17217963aa52160899b2a0
+oid sha256:a3acaa115a03fda5884390e7426308da716cbaee1bff40e19dee5654f6ebafcf
 size 3791468264

model-00041-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4784664f8f54dc3fb1c3d3ffa640a898f494a68724b3f06123848b137fecc274
+oid sha256:96684dab0530583eb672d43271fa88c6bb58c77a76c2aca4a77f72194d558568
 size 3770196640

model-00042-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7e67ba4908bf23edc72ed030e19b8e23f5dd7833b20208f3155d4c4a73ad0fca
+oid sha256:3d44febce460d1f1810815fe595f8c6165f7200b659337db042cdcd67e1a0c70
 size 3791468264

model-00043-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:781c476a97d280edd6f3f816984b7947d090beadcdf5b4c356bfc9142b10aa75
+oid sha256:499f7600d7bf1cc082332e116703c7d755d0e309d79320b732e4f08a3cfa4636
 size 3770196640

model-00044-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:15e2e44878441d42083bb51f9180472c53388aa48d63c80db9c6f851a6db671d
+oid sha256:2e7c59b873e7102305e96e41c45de15ce48fc5c2c55dfe6251165f39ab909246
 size 3791468264

model-00046-of-00046.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:861dd01896a403acab362397267122d5571f390f0fe1714107ea39895c1587ff
+oid sha256:444c93c8e1b8e934115d6f7af3c92c35897d808d06c490743653fa52bdad05cb
 size 3795396540

model-mtp-shared.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d9fe1fea8d63814926e74dbe3b0604767d9a0fa6b1c3e8641fcde042da11869
+size 2118123728

model.safetensors.index.json

@@ -1,6 +1,7 @@
 {
   "metadata": {
-    "total_size": 159609485896
+    "mtp_shared_tensors": "materialized",
+    "total_size": "170598367456"
   },
   "weight_map": {
     "embed.weight": "model-00001-of-00046.safetensors",
@@ -67630,6 +67631,7 @@
     "mtp.0.attn_norm.weight": "model-00046-of-00046.safetensors",
     "mtp.0.e_proj.scale": "model-00046-of-00046.safetensors",
     "mtp.0.e_proj.weight": "model-00046-of-00046.safetensors",
+    "mtp.0.emb.tok_emb.weight": "model-mtp-shared.safetensors",
     "mtp.0.enorm.weight": "model-00046-of-00046.safetensors",
     "mtp.0.ffn.experts.0.w1.scale": "model-00046-of-00046.safetensors",
     "mtp.0.ffn.experts.0.w1.weight": "model-00046-of-00046.safetensors",
@@ -69187,6 +69189,7 @@
     "mtp.0.hc_head_base": "model-00046-of-00046.safetensors",
     "mtp.0.hc_head_fn": "model-00046-of-00046.safetensors",
     "mtp.0.hc_head_scale": "model-00046-of-00046.safetensors",
+    "mtp.0.head.weight": "model-mtp-shared.safetensors",
     "mtp.0.hnorm.weight": "model-00046-of-00046.safetensors",
     "mtp.0.norm.weight": "model-00046-of-00046.safetensors",
     "norm.weight": "model-00045-of-00046.safetensors"