Support W8A8 inference in vllm

We have implemented W8A8 inference in vLLM, which can achieve a 30% improvement in throughput. W4A16 quantization methods require weights to be dequantized into fp16 before compute and lead to a throughput drop under heavier load. This PR is part of https://github.com/vllm-project/vllm/pull/1112. We have split the huge PR into two independent parts for easier review.
The usage of w8a8 inference is simple(support llama for now):
python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --quantization smoothquant

Updates!!!
We have update the quant method to per token quant for o_proj and down_proj of LLama. Please use  the lastest llama-dev branch of smoothquant and per_token_quant branch of  torch-int to generate int8 model!!!
You can find more details like how to gernerate int8 weight in original PR https://github.com/vllm-project/vllm/pull/1112
You can use the method with int8 kv cache quant https://github.com/vllm-project/vllm/pull/1507 for best throughput

We have implemented W8A8 inference in vLLM, which can achieve a 30% improvement in throughput. W4A16 quantization methods require weights to be dequantized into fp16 before compute and lead to a throughput drop under heavier load. This PR is part of #1112. We have split the huge PR into two independent parts for easier review. The usage of w8a8 inference is simple(support llama for now):
python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --quantization smoothquant

You can find more details like how to gernerate int8 weight in original PR #1112

Does this PR only support per tensor? Whether to support per token and per channel

We have implemented W8A8 inference in vLLM, which can achieve a 30% improvement in throughput. W4A16 quantization methods require weights to be dequantized into fp16 before compute and lead to a throughput drop under heavier load. This PR is part of #1112. We have split the huge PR into two independent parts for easier review. The usage of w8a8 inference is simple(support llama for now):
python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --quantization smoothquant

You can find more details like how to gernerate int8 weight in original PR #1112

Does this PR only support per tensor? Whether to support per token and per channel

Only support per tensor for now.  We may support per channel if we found it make big difference in model performance.

I downloaded your vllm branch w8a8 but i faced this case of error. should i add int8LlamForCausalLM in smoothquant ??
ValueError: Model architectures ['Int8LlamaForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'FalconForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MPTForCausalLM', 'OPTForCausalLM', 'QWenLMHeadModel', 'RWForCausalLM']

I downloaded your vllm branch w8a8 but i faced this case of error. should i add int8LlamForCausalLM in smoothquant ??
ValueError: Model architectures ['Int8LlamaForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'FalconForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MPTForCausalLM', 'OPTForCausalLM', 'QWenLMHeadModel', 'RWForCausalLM']

Sorry, it's our negligence that there is a naming difference between the smoothquant repo and the vllm repo. Please change the 'architectures' field in the quantized model's config.json from 'Int8LlamaForCausalLM' to 'LlamaForCausalLM'. We will fix this problem soon.

Thanks for your answer.
And did you apply partial quantization which mean that  down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant if you want to avoid the problem for now. It can lift the throughput by 15%

@AniZpZ Existing methods (AWQ, GPTQ) go down to 4-bit quantization, saving lots of memory. The speed improvements of 8-bit inference come during inference, which theoretically could be combined AWQ.
Would it be possible to integrate AWQ or GPTQ to go from 4bit quantization to 8bit on-the-fly?

@AniZpZ hi, thanks for this. I want try this. But could you give some simple quantization guides? I look into the smoothquant repo, its hard for me to quantize new model where i don't how to use my datasets. Before I use AutoAWQ and json format dataset for quantization. Also there are not so much smoothquant models in huggingface(like codellama).

@AniZpZ Existing methods (AWQ, GPTQ) go down to 4-bit quantization, saving lots of memory. The speed improvements of 8-bit inference come during inference, which theoretically could be combined AWQ.
Would it be possible to integrate AWQ or GPTQ to go from 4bit quantization to 8bit on-the-fly?

The reason why gptq and awq cannot achieve throughput improvement is that they are weight-only quantization. Weight-only quantization requires dequantizing weight to fp16 (or something like bf16) and performing 16bit (or higher) gemm during inference. They cannot benefit from computation reduction.

t could you give some simple quantization

First get a normal llama13B model. Then install smoothquant and torch-int for llama. Use "examples/generate_act_scales.py" to generate act scale, and then use "examples/export_int8_llama.py" to export int8 model. Please note to check and change the 'architectures' field in the model's config.json from 'Int8LlamaForCausalLM' to 'LlamaForCausalLM'.

@AniZpZ Thanks for your quick reply! The generate_act_scales.py need  datasets param. Default dataset is not for code model. how to load custom json format dataset?

@AniZpZ Thanks for your quick reply! The generate_act_scales.py need datasets param. Default dataset is not for code model. how to load custom json format dataset?

We use HF Datasets lib for calibrate data loading. You can check the usage of HF Datasets to load your custom data.

thanks for the PR! I tried to run this with a quantized Llama 2 70B model on A100 but loading seems to be extremely slow (server appears to hang but GPU memory usage does grow very slowly). do you happen to have some insights?

Were you able to run a Llama 2 70B model with the current branch? I can get https://github.com/vllm-project/vllm/pull/1112 running with it but I think there's some merge fixes to be done in the current PR.

Were you able to run a Llama 2 70B model with the current branch? I can get #1112 running with it but I think there's some merge fixes to be done in the current PR.

The branch hasn't been tested with Llama 2 70B yet. I will try to run Llama 2 70B with current branch and fix the potential problems. I will update the code and notify you asap.

try to run int8 model with w8a8 via api_server.py, when i call api_client.py
got error
File "/vllm/vllm/model_executor/layers/attention.py", line 255, in forward
cache_ops.reshape_and_cache(
RuntimeError: expected scalar type Long but found Int
server could run, while can't deal with call from client

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

We are doing experiments comparing partial and per-token activation quantization. We will release the experiment results this Friday and release partial-quant or per-token-quant next Monday at the latest.

try to run int8 model with w8a8 via api_server.py, when i call api_client.py got error
File "/vllm/vllm/model_executor/layers/attention.py", line 255, in forward cache_ops.reshape_and_cache( RuntimeError: expected scalar type Long but found Int
server could run, while can't deal with call from client

I am fixing the problem, you can test with the branch at https://github.com/vllm-project/vllm/pull/1112 first. There are some things to do to adapt to the latest version.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

If first token generation time is your primary consideration, you can test out kv cache quant branch first. It can signicatly reduce the  first token generation time.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

We are doing experiments comparing partial and per-token activation quantization. We will release the experiment results this Friday and release partial-quant or per-token-quant code next Monday. The released method depends on our experiment result.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

If first token generation time is your primary consideration, you can test our kv cache quant branch first. It can significantly reduce the first token generation time.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

Hello! Here, we provide the test results of two partial quant methods for llama-13b: partial quant 1 (only down_proj in fp16) and partial quant 2 (o_proj and down_proj in fp16). Note that the two methods both use int8 kv-quant.
Throughput
Compared to fp16, partial quant 1 lifts the throughput by about 40%, and partial quant 2 lifts it by about 30%.
MMLU scores

model
STEM
Social Sciences
Humanities
Other
Average

fp16
0.4056
0.4965
0.4750
0.4855
0.4657

partial quant 1
0.3937
0.4080
0.4163
0.4191
0.4093

partial quant 2
0.4009
0.4796
0.4575
0.4766
0.4537

We will release the partial quant 2 code next week.

As for dynamic activation per-token quant, we apply it in down_proj quant. But it helps a little with only 1.8% throughput increment in our experiments.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

Hello! Here, we provide the test results of two partial quant methods for llama-13b: partial quant 1 (only down_proj in fp16) and partial quant 2 (o_proj and down_proj in fp16). Note that the two methods both use int8 kv-quant. Throughput Compared to fp16, partial quant 1 lifts the throughput by about 40%, and partial quant 2 lifts it by about 30%. MMLU scores
model	STEM	Social Sciences	Humanities	Other	Average
fp16	0.4056	0.4965	0.4750	0.4855	0.4657
partial quant 1	0.3937	0.4080	0.4163	0.4191	0.4093
partial quant 2	0.4009	0.4796	0.4575	0.4766	0.4537
We will release the partial quant 2 code next week.
As for dynamic activation per-token quant, we apply it in down_proj quant. But it helps a little with only 1.8% throughput increment in our experiments.

We have figured out a method whereby per-token quantization is faster than partial quantization and has similar model performance.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

@yunfeng-scale @gesanqiu Hello! Here, we provide the experiment results based on vllm 0.1.7 for llama-13b including: w8a8, two partial quant methods and two per-token quant methods.
w8a8: both activation and weight use per-tensor quant partial quant 1: only down_proj uses fp16 partial quant 2: both o_proj and down_proj use fp16 per-token quant 1: only down_proj uses per-token quant per-token quant 2: both o_proj and down_proj use quant
Throughput
Model	Throughput (tokens/s)	Percentage Increase
fp16	528.2081	+0%
w8a8 + kv-quant	797.8306	+50.95%
partial quant 1 + kv-quant	740.8924	+40.32%
partial quant 2 + kv-quant	733.9158	+38.82%
per-token quant 1 + kv-quant	793.8667	+50.19%
per-token quant 2 + kv-quant	775.3904	+46.78%
MMLU scores
model	STEM	Social Sciences	Humanities	Other	Average
fp16	0.4056	0.4965	0.4750	0.4855	0.4657
w8a8 + kv-quant	0.2584	0.2559	0.2509	0.2589	0.2570
partial quant 1 + kv-quant	0.3955	0.4806	0.4666	0.4805	0.4558
partial quant 2 + kv-quant	0.3987	0.4810	0.4712	0.4765	0.4568
per-token quant 1 + kv-quant	0.3827	0.4663	0.4369	0.4618	0.4369
per-token quant 2 + kv-quant	0.3728	0.4714	0.4500	0.4833	0.4443
As you see, per-token quant can realize better performance. Compared with fp16, per-token quant 2 + kv-quant only decreases about 0.02 MMLU score on average, but throughput increases about 50%. We will release per-token quant code and merge it with the latest vllm version soon.

Hi, when will this be released? Can we have partial quant 1 and partial quant 2 first?

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

@yunfeng-scale @gesanqiu Hello! Here, we provide the experiment results based on vllm 0.1.7 for llama-13b including: w8a8, two partial quant methods and two per-token quant methods.
w8a8: both activation and weight use per-tensor quant partial quant 1: only down_proj uses fp16 partial quant 2: both o_proj and down_proj use fp16 per-token quant 1: only down_proj uses per-token quant per-token quant 2: both o_proj and down_proj use quant
Throughput
Model	Throughput (tokens/s)	Percentage Increase
fp16	528.2081	+0%
w8a8 + kv-quant	797.8306	+50.95%
partial quant 1 + kv-quant	740.8924	+40.32%
partial quant 2 + kv-quant	733.9158	+38.82%
per-token quant 1 + kv-quant	793.8667	+50.19%
per-token quant 2 + kv-quant	775.3904	+46.78%
MMLU scores
model	STEM	Social Sciences	Humanities	Other	Average
fp16	0.4056	0.4965	0.4750	0.4855	0.4657
w8a8 + kv-quant	0.2584	0.2559	0.2509	0.2589	0.2570
partial quant 1 + kv-quant	0.3955	0.4806	0.4666	0.4805	0.4558
partial quant 2 + kv-quant	0.3987	0.4810	0.4712	0.4765	0.4568
per-token quant 1 + kv-quant	0.3827	0.4663	0.4369	0.4618	0.4369
per-token quant 2 + kv-quant	0.3728	0.4714	0.4500	0.4833	0.4443
As you see, per-token quant can realize better performance. Compared with fp16, per-token quant 2 + kv-quant only decreases about 0.02 MMLU score on average, but throughput increases about 50%. We will release per-token quant code and merge it with the latest vllm version soon.

Hi, when will this be released? Can we have partial quant 1 and partial quant 2 first?

We try to release w8a8 per-token quant this evening, as there are still many merge conflicts to solve.

Thanks for your answer. And did you apply partial quantization which mean that down_proj layer remain as a fp16 because of big activation range. as you know there is comment in #1112 about this.
i was running our llama2 model with smoothquant but i faced big performance drop. so if you don't merge yet. it is better to merge in your pr.

We are addressing the performance drop problem with a more solid method. We analyze the activation distribution and are trying to fix the problem with per channel quantization. The job will be done next week. You can try out kv cache quant #1507 if you want to avoid the problem for now. It can lift the throughput by 15% with minial performance loss.

Will w8a8 partial release this week?

@yunfeng-scale @gesanqiu Hello! Here, we provide the experiment results based on vllm 0.1.7 for llama-13b including: w8a8, two partial quant methods and two per-token quant methods.
w8a8: both activation and weight use per-tensor quant partial quant 1: only down_proj uses fp16 partial quant 2: both o_proj and down_proj use fp16 per-token quant 1: only down_proj uses per-token quant per-token quant 2: both o_proj and down_proj use quant
Throughput
Model	Throughput (tokens/s)	Percentage Increase
fp16	528.2081	+0%
w8a8 + kv-quant	797.8306	+50.95%
partial quant 1 + kv-quant	740.8924	+40.32%
partial quant 2 + kv-quant	733.9158	+38.82%
per-token quant 1 + kv-quant	793.8667	+50.19%
per-token quant 2 + kv-quant	775.3904	+46.78%
MMLU scores
model	STEM	Social Sciences	Humanities	Other	Average
fp16	0.4056	0.4965	0.4750	0.4855	0.4657
w8a8 + kv-quant	0.2584	0.2559	0.2509	0.2589	0.2570
partial quant 1 + kv-quant	0.3955	0.4806	0.4666	0.4805	0.4558
partial quant 2 + kv-quant	0.3987	0.4810	0.4712	0.4765	0.4568
per-token quant 1 + kv-quant	0.3827	0.4663	0.4369	0.4618	0.4369
per-token quant 2 + kv-quant	0.3728	0.4714	0.4500	0.4833	0.4443
As you see, per-token quant can realize better performance. Compared with fp16, per-token quant 2 + kv-quant only decreases about 0.02 MMLU score on average, but throughput increases about 50%. We will release per-token quant code and merge it with the latest vllm version soon.

Hi, when will this be released? Can we have partial quant 1 and partial quant 2 first?

We have released per-token quant code based vllm 0.2.1. Welcome to try it!

@yunfeng-scale
Hello there~
We have fixed the loading and running problems in v0.2.1, and applied per-token quantization on o_proj and down_proj, which leads to better model performance.
Could you please take some time out of your busy schedule to review our code?

Hi there, I tested this patch on A40 card with codellama-13b, and benchmark results show as follows:

benchmark_throughput(include results from #1112 )

Model
Throughput(tokens/s)
Percentage Increse

FP16
742.00
+0%

KV_INT8
1146.75
+54.5%

W8A8
1219.20
+64.3%

W8A8 + KV_INT8
1648.25
+122.1%

W8A8 per-token
1218.42
+64.2%

W4A16(AWQ)
857.16
+15.5%

notice that the throughput of awq model is tested with max_num_seqs=64, in my test case it may be bigger when max_num_seqs in [64, 96].

benchmark_latency

Batch size
Input Length
Output Length
FP16(s)
W8A8 per-token(s)
AWQ(s)

1
128
32
1.5795
0.9617
0.6867

8
128
32
1.9444
1.1977
1.4441

16
128
32
2.3071
1.4768
2.311

32
128
32
3.2188
2.0462
4.2129

64
128
32
6.828
3.1846
-

32
256
32
5.2961
7.9021
7.5002

32
512
64
16.3069
7.7758
18.1441

32
1024
128
49.6551
22.9284
44.2433

Tested lots of cases, only update part of them.
Results show W8A8 per-token quantization has a better performance under large batch size and large context scenario.

prefilling latenct and decoding latency results under batch_size =1

Input Length
FP16 prefill(ms)
FP16 decode(ms)
W8A8 per-token prefill(ms)
W8A8 per-token decode(ms)
AWQ prefill(ms)
AWQ decode(ms)

128
58
28
36
29
92
19

256
79
49
47
29
180
19

512
134
49
84
30
357
19

1024
274
50
166
30
708
19

2048
531
51
333
32
1436
20

HumanEval is ongoing, shell update in tomorrow~

Hi, Is this per-token quantization patch only support single card?
I tested this patch on A10 with llama2-7b, there is no problem if I run with single card. But if I run with -tp 2, the result seems stange.
I think maybe the per_token scale factor should applied before all-gather, and re-caculate the new per_token scale factor after all_gather
def forward(self, x):
    gate_up, _ = self.gate_up_proj(x)
    scale = None
    if self.use_int8:
        # TODO: currently gate up share same scale, use seperate scales
        x, *scale = self.act_fn(gate_up)
    else:
        x = self.act_fn(gate_up)
    ## apply scale before all_gather ?
    x, _ = self.down_proj(x)
    ## update scale after all_gather?
    return x, scale

Hi, Is this per-token quantization patch only support single card?
I tested this patch on A10 with llama2-7b, there is no problem if I run with single card. But if I run with -tp 2, the result seems strange.
Maybe the per_token scale factor should applied before all_reduce, and re-caculate the new per_token scale factor after all_reduce?
def forward(self, x):
    gate_up, _ = self.gate_up_proj(x)
    scale = None
    if self.use_int8:
        # TODO: currently gate up share same scale, use seperate scales
        x, *scale = self.act_fn(gate_up)
    else:
        x = self.act_fn(gate_up)
    ## apply scale before all_reduce ?
    x, _ = self.down_proj(x)
    ## update scale after all_reduce ?
    return x, scale

Thanks for your issue! We have confirmed it is a bug, as we didn't take TP into account when designing per-token quant. The correct way is to do per-token dequant before all_reduce. We will fix it soon!

Hi, Is this per-token quantization patch only support single card?
I tested this patch on A10 with llama2-7b, there is no problem if I run with single card. But if I run with -tp 2, the result seems strange.
Maybe the per_token scale factor should applied before all_reduce, and re-caculate the new per_token scale factor after all_reduce?
def forward(self, x):
    gate_up, _ = self.gate_up_proj(x)
    scale = None
    if self.use_int8:
        # TODO: currently gate up share same scale, use seperate scales
        x, *scale = self.act_fn(gate_up)
    else:
        x = self.act_fn(gate_up)
    ## apply scale before all_reduce ?
    x, _ = self.down_proj(x)
    ## update scale after all_reduce ?
    return x, scale

We have update the code and fix tp problem when enable per token quant.

@zhuohan123 @WoosukKwon
Hello there!
Would you mind reviewing our W8A8 inference PR? It can significantly increase throughput and reduce inference latency according to our experiments.
Similar results have been reproduced in https://github.com/vllm-project/vllm/pull/1508#issuecomment-1824133140

I downloaded your vllm branch w8a8 but i faced this case of error because there is no config file for smoothquant
vllm-w8a8/vllm/model_executor/weight_utils.py", line 107, in get_quant_config
raise ValueError(f"Cannot find the config file for {quantization}")
 ValueError: Cannot find the config file for smoothquant

I downloaded your vllm branch w8a8 but i faced this case of error because there is no config file for smoothquant
vllm-w8a8/vllm/model_executor/weight_utils.py", line 107, in get_quant_config
raise ValueError(f"Cannot find the config file for {quantization}")
 ValueError: Cannot find the config file for smoothquant

And other issue:
llm-w8a8/vllm/model_executor/models/llama.py", line 474, in load_weights
    param = state_dict[name.replace(weight_name, "qkv_proj")]
KeyError: 'model.layers.0.self_attn.qkv_proj.a'

Thanks!
For the first issue, please refer to https://github.com/vllm-project/vllm/pull/1112#issuecomment-1818965427.
For the second issue, you may use the wrong branch, Please use the lastest llama-dev branch of [smoothquant](https://github.com/AniZpZ/smoothquant/tree/llama-dev) and per_token_quant branch of [torch-int](https://github.com/AniZpZ/torch-int/tree/per_token_quant) to generate int8 model!!!

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~ BTW, kv-quant-merge branch should work fine, what's your issue?

Thank you for your reply. I use kv-quant-merge branch for inference, but the error is as follows:
KeyError: 'model.layers.0.self_attn.qkv_proj.bias'
my run command is: python -m vllm.entrypoints.api_server --model=/home/models/quant/smoothquant/Codellama-13b-int8-02/CodeLlama-13b-hf-smoothquant/ --tokenizer /home/models/CodeLlama-13b-hf/ --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/home/models/quant/kv_quant/kv-cache-turbomind-merge/

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~ BTW, kv-quant-merge branch should work fine, what's your issue?

Thank you for your reply. I use kv-quant-merge branch for inference, but the error is as follows: KeyError: 'model.layers.0.self_attn.qkv_proj.bias' or KeyError: 'model.layers.0.self_attn.qkv_proj.dequant_scale'
my run command is: python -m vllm.entrypoints.api_server --model=/home/models/quant/smoothquant/Codellama-13b-int8-02/CodeLlama-13b-hf-smoothquant/ --tokenizer /home/models/CodeLlama-13b-hf/ --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/home/models/quant/kv_quant/kv-cache-turbomind-merge/

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~ BTW, kv-quant-merge branch should work fine, what's your issue?

Thank you for your reply. I use kv-quant-merge branch for inference, but the error is as follows: KeyError: 'model.layers.0.self_attn.qkv_proj.bias' or KeyError: 'model.layers.0.self_attn.qkv_proj.dequant_scale'
my run command is: python -m vllm.entrypoints.api_server --model=/home/models/quant/smoothquant/Codellama-13b-int8-02/CodeLlama-13b-hf-smoothquant/ --tokenizer /home/models/CodeLlama-13b-hf/ --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/home/models/quant/kv_quant/kv-cache-turbomind-merge/
您在kv-quant-merge上跑通过吗？

Which branch of torch-int and smoothquant you're using? kv-quant-merge requires llama-dev, and as far as I known, the definition of LlamaModel on llama-dev devide qkv_proj to q_proj, k_proj, v_proj, there shouldn't exist qkv_proj weight.

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~ BTW, kv-quant-merge branch should work fine, what's your issue?

Thank you for your reply. I use kv-quant-merge branch for inference, but the error is as follows: KeyError: 'model.layers.0.self_attn.qkv_proj.bias' or KeyError: 'model.layers.0.self_attn.qkv_proj.dequant_scale'
my run command is: python -m vllm.entrypoints.api_server --model=/home/models/quant/smoothquant/Codellama-13b-int8-02/CodeLlama-13b-hf-smoothquant/ --tokenizer /home/models/CodeLlama-13b-hf/ --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/home/models/quant/kv_quant/kv-cache-turbomind-merge/
您在kv-quant-merge上跑通过吗？

Yes, I can run inference on kv-quant-merge.
Which branch of torch-int and smoothquant you're using? In https://github.com/AniZpZ/smoothquant/blob/3f063c4f830af769fc4c8c9d4c3efa722feba39a/smoothquant/llama.py#L46 qkv_proj is sperated to q_prj, k_proj, v_proj, so there shouldn't exist qkv_proj weight definition.

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~ BTW, kv-quant-merge branch should work fine, what's your issue?

Thank you for your reply. I use kv-quant-merge branch for inference, but the error is as follows: KeyError: 'model.layers.0.self_attn.qkv_proj.bias' or KeyError: 'model.layers.0.self_attn.qkv_proj.dequant_scale'
my run command is: python -m vllm.entrypoints.api_server --model=/home/models/quant/smoothquant/Codellama-13b-int8-02/CodeLlama-13b-hf-smoothquant/ --tokenizer /home/models/CodeLlama-13b-hf/ --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/home/models/quant/kv_quant/kv-cache-turbomind-merge/
您在kv-quant-merge上跑通过吗？

It seems that you have exported a quantized model using sq, but are trying to load it using the kv quant branch, which leads to a weight mismatch. If you intend to use only int8 kv cache, you should run it with the fp16 model.

Hi @AniZpZ tested the per token implementation on Llama 2 70B and saw pretty good results. I'm getting > 50% throughput improvements on larger batch sizes, and about 2% performance drop on Winogrande and 2.7% on MMLU

Hi @AniZpZ tested the per token implementation on Llama 2 70B and saw pretty good results. I'm getting > 50% throughput improvements on larger batch sizes, and about 2% performance drop on Winogrande and 2.7% on MMLU

Great! Thank you for your testing and I'm glad to find that our quantization implementation achieves similar throughput improvements across models with various parameter size. I'm wondering if the 2.7% drop on MMLU is a relative value or an absolute value.
Also, would you mind taking some time to review our PR again?

@shatealaboxiaowang Hi, w8a8+kv_quant is separated in 2 branch for easier to review, there are not too many confilicts on these two branch, you can merge them into one dev branch. I did this by myself, but found that new version(v0.2.3) of vLLM may cause a decrease in throughput, still need some work to fix it~ BTW, kv-quant-merge branch should work fine, what's your issue?

Thank you for your reply. I use kv-quant-merge branch for inference, but the error is as follows: KeyError: 'model.layers.0.self_attn.qkv_proj.bias' or KeyError: 'model.layers.0.self_attn.qkv_proj.dequant_scale'
my run command is: python -m vllm.entrypoints.api_server --model=/home/models/quant/smoothquant/Codellama-13b-int8-02/CodeLlama-13b-hf-smoothquant/ --tokenizer /home/models/CodeLlama-13b-hf/ --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/home/models/quant/kv_quant/kv-cache-turbomind-merge/
您在kv-quant-merge上跑通过吗？

It seems that you have exported a quantized model using sq, but are trying to load it using the kv quant branch, which leads to a weight mismatch. If you intend to use only int8 kv cache, you should run it with the fp16 model.

thank you，but what i want to do is to inference with kv-quant on the basis of sq quant instead of fp16.

You can merge kv-quant branch into this branch in your local repo and reinstall vLLM.

Hi @AniZpZ tested the per token implementation on Llama 2 70B and saw pretty good results. I'm getting > 50% throughput improvements on larger batch sizes, and about 2% performance drop on Winogrande and 2.7% on MMLU

Great! Thank you for your testing and I'm glad to find that our quantization implementation achieves similar throughput improvements across models with various parameter size. I'm wondering if the 2.7% drop on MMLU is a relative value or an absolute value. Also, would you mind taking some time to review our PR again?

the drop is absolute value

Hi @AniZpZ tested the per token implementation on Llama 2 70B and saw pretty good results. I'm getting > 50% throughput improvements on larger batch sizes, and about 2% performance drop on Winogrande and 2.7% on MMLU

Great! Thank you for your testing and I'm glad to find that our quantization implementation achieves similar throughput improvements across models with various parameter size. I'm wondering if the 2.7% drop on MMLU is a relative value or an absolute value. Also, would you mind taking some time to review our PR again?

the drop is absolute value

Hi, we optimized per-token quant kernels and int8 model export this time, which could bring further delay reduction (under low load) and model performance improvement. The usage remains unchanged (model export seems a little complicated，we plan to release a new repo for easy usage next week). Welcome to try our new release when you are free. Here, we provide our new test results.
 Could you please give us some advice for merging this PR to main branch? Does it need another reviewer? BTW, we are preparing a repo to support easy int8 model exporting and will release it next week. Hope to see your reply soon.

Have you been tested the TP>1 cases? I found that when TP>1, the throughput decrease to FP16.

Hi @HandH1998 we'll have to wait for @WoosukKwon @zhuohan123 's review, and let them decide what changes are needed to merge. for now can you add some config validation about this only support llama models? also i think it'll be a good idea to merge to latest main since there has been some significant changes since 0.2.1. my take on the current modifications is it's a bit intrusive, and it'll be great if you could generalize the code a bit, like putting smoothquant related code into csrc/quantization/smoothquant, and add params to existing functions (e.g. AWQ scale activation https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/activation.py#L134). i'm not sure how fused kernel fits in and would defer to maintainers.

We have implemented W8A8 inference in vLLM, which can achieve a 30% improvement in throughput. W4A16 quantization methods require weights to be dequantized into fp16 before compute and lead to a throughput drop under heavier load. This PR is part of #1112. We have split the huge PR into two independent parts for easier review. The usage of w8a8 inference is simple(support llama for now):
python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --quantization smoothquant

Updates!!! We have update the quant method to per token quant for o_proj and down_proj of LLama. Please use the lastest llama-dev branch of smoothquant and per_token_quant branch of torch-int to generate int8 model!!!
You can find more details like how to gernerate int8 weight in original PR #1112 You can use the method with int8 kv cache quant #1507 for best throughput

Why does w8a8 not need to perform de-quantization? If really do not need to de-quantization, also need corresponding hardware gpu support?

We have implemented W8A8 inference in vLLM, which can achieve a 30% improvement in throughput. W4A16 quantization methods require weights to be dequantized into fp16 before compute and lead to a throughput drop under heavier load. This PR is part of #1112. We have split the huge PR into two independent parts for easier review. The usage of w8a8 inference is simple(support llama for now):
python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --quantization smoothquant

Updates!!! We have update the quant method to per token quant for o_proj and down_proj of LLama. Please use the lastest llama-dev branch of smoothquant and per_token_quant branch of torch-int to generate int8 model!!!
You can find more details like how to gernerate int8 weight in original PR #1112 You can use the method with int8 kv cache quant #1507 for best throughput

Why does w8a8 not need to perform de-quantization? If really do not need to de-quantization, also need corresponding hardware gpu support?

W8A8 do not nedd to perform de-quantization for weights before gemm becuase we could utilize tensor core to perform int8 computation(require Turing GPU or later).

Have you been tested the TP>1 cases? I found that when TP>1, the throughput decrease to FP16.

We are looking into this problem this week

Hi @AniZpZ tested the per token implementation on Llama 2 70B and saw pretty good results. I'm getting > 50% throughput improvements on larger batch sizes, and about 2% performance drop on Winogrande and 2.7% on MMLU

Great! Thank you for your testing and I'm glad to find that our quantization implementation achieves similar throughput improvements across models with various parameter size. I'm wondering if the 2.7% drop on MMLU is a relative value or an absolute value. Also, would you mind taking some time to review our PR again?

the drop is absolute value

Hi, we optimized per-token quant kernels and int8 model export this time, which could bring further delay reduction (under low load) and model performance improvement. Welcome to try our new release! We also updated llama-dev branch of smoothquant. You should use the lastest llama-dev branch of smoothquant and per_token_quant branch of torch-int to generate a new int8 model. Here, we provide our new test results.  Just wondering how did you compute MMLU scores? Can we directly use the lm-evaluation harness code on quantized INT8 models? Now lm-evaluation v0.4.0 supports using vLLM to be a backend. However, I am not sure whether we can use INT8 quantized models directly. Thanks

@leocnj, hi, we computed MMLU scores with the logits which vllm backend provides. If lm-evaluation harness also evaluates MMLU like that, I think you can use our w8a8 vllm branch as vllm backend then evaluate MMLU with lm-evaluation harness code.

@leocnj, hi, we computed MMLU scores with the logits which vllm backend provides. If lm-evaluation harness also evaluates MMLU like that, I think you can use our w8a8 vllm branch as vllm backend then evaluate MMLU with lm-evaluation harness code.

Indeed. After the recent lm-eval harness update to version 0.4.0, vLLM can be used. Using your branch of vLLM, I can get MMLU scores. Thanks.

@AniZpZ Hi, thank you for your work, does this PR support running llama2-34B using GQA?

@AniZpZ Hi, thank you for your work, does this PR support running llama2-34B using GQA?

Hi there, we haven't tested with the Llama2 34B. But it is supposed to support all Llama and Llama2 models.

This is great work, really looking forward to this landing in vLLM as AWQ is very slow. On that note, it would be nice to add latency and quality comparison against AWQ using a larger model (like yi 34b or llama 2 70B) which would be the main benefactor of this kind of work. I can chip in for the eval if you have the time to run it.

@AniZpZ Hi, thank you for your work, does this PR support running llama2-34B using GQA?

Hi there, we haven't tested with the Llama2 34B. But it is supposed to support all Llama and Llama2 models.

Hi, I used "examples/export_int8_llama.py" to quantize the Yi-34B model on a single A100 80G, but OOM occurred after the console output "start trans into int8, this might take a while". Do you have any good way to solve this problem?

@AniZpZ Hi, thank you for your work, does this PR support running llama2-34B using GQA?

Hi there, we haven't tested with the Llama2 34B. But it is supposed to support all Llama and Llama2 models.

@AniZpZ Hi, I used "examples/export_int8_llama.py" to quantize the Yi-34B model on a single A100 80G, but OOM occurred after the console output "start trans into int8, this might take a while". Do you have any good way to solve this problem?

@AniZpZ Hi, thank you for your work, does this PR support running llama2-34B using GQA?

Hi there, we haven't tested with the Llama2 34B. But it is supposed to support all Llama and Llama2 models.

@AniZpZ Hi, I used "examples/export_int8_llama.py" to quantize the Yi-34B model on a single A100 80G, but OOM occurred after the console output "start trans into int8, this might take a while". Do you have any good way to solve this problem?

You can try using our new repo(https://github.com/AniZpZ/AutoSmoothQuant) to implement smoothquant to model.

This is great work, really looking forward to this landing in vLLM as AWQ is very slow. On that note, it would be nice to add latency and quality comparison against AWQ using a larger model (like yi 34b or llama 2 70B) which would be the main benefactor of this kind of work. I can chip in for the eval if you have the time to run it.

Thank you for your support! We currently work on smaller models, but according to this data(https://github.com/vllm-project/vllm/pull/1508#issuecomment-1851138477), we can get  over 50% throughput improvements int llama2 70b.

Hi @AniZpZ tested the per token implementation on Llama 2 70B and saw pretty good results. I'm getting > 50% throughput improvements on larger batch sizes, and about 2% performance drop on Winogrande and 2.7% on MMLU

Great! Thank you for your testing and I'm glad to find that our quantization implementation achieves similar throughput improvements across models with various parameter size. I'm wondering if the 2.7% drop on MMLU is a relative value or an absolute value. Also, would you mind taking some time to review our PR again?

the drop is absolute value

Hi, we optimized per-token quant kernels and int8 model export this time, which could bring further delay reduction (under low load) and model performance improvement. Welcome to try our new release! We also updated llama-dev branch of smoothquant. You should use the lastest llama-dev branch of smoothquant and per_token_quant branch of torch-int to generate a new int8 model. Here, we provide our new test results. Input length = 640, Output lenght = 100, GPU = 1 A100, vllm = 0.2.1, model = Llama-13B
Throughput (under high load) QPS = 14
Model	Throughput (tokens/s)	Throughput Increase
fp16	574.3673	0%
w8a8-per-token	740.8307	28.98%
w8a8-per-token + kv-quant	885.7578	54.21%
Latency (under low load)
fp16	w8a8-per-token	Latency Decrease
1 QPS	2.4372 s	1.9176 s	21.32%
2 QPS	2.9294 s	2.1834 s	25.47%
3 QPS	3.6177 s	2.5054 s	30.74%
fp16	w8a8-per-token + kv-quant	Latency Decrease
1 QPS	2.4372 s	1.9376 s	20.49%
2 QPS	2.9294 s	2.1661 s	26.06%
3 QPS	3.6177 s	2.4491 s	32.30%
MMLU scores
model	STEM	Social Sciences	Humanities	Other	Average
fp16	0.4056	0.4965	0.4750	0.4855	0.4657
w8a8-per-token + kv-quant	0.3801	0.4892	0.4538	0.4800	0.4507

@HandH1998 Hi, what are the quant_config settings for the w8a8-per-token method here?

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

Hi, here is our new repo AutoSmoothQuant for exporting int8 model. Looking forward to your star ^-^! Ask me if you have any question.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

Hi, here is our new repo AutoSmoothQuant for exporting int8 model. Look forward to your star ^-^! Ask me if you have any question.

@HandH1998 After the quantification was completed, I used vllm of the w8a8 branch to deploy, but the following error occurred:
"ValueError: Cannot find any of ['w_bit', 'bits'] in the model's quantization config."
The contents of my quant_config.json are currently:
{
"qkv_proj": "per-tensor",
"o_proj": "per-token",
"gate_up_proj": "per-tensor",
"down_proj": "per-token"
}
How can I modify or add the content in quant_config.json?

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

Hi, here is our new repo AutoSmoothQuant for exporting int8 model. Look forward to your star ^-^! Ask me if you have any question.

@HandH1998 After the quantification was completed, I used vllm of the w8a8 branch to deploy, but the following error occurred: "ValueError: Cannot find any of ['w_bit', 'bits'] in the model's quantization config."
The contents of my quant_config.json are currently: { "qkv_proj": "per-tensor", "o_proj": "per-token", "gate_up_proj": "per-tensor", "down_proj": "per-token" } How can I modify or add the content in quant_config.json?

Moreover, I saw under the w8a8 branch of vllm that the value of "quant_type" in SmoothQuantConfig is restricted to "tensor". Does this mean that I currently cannot deploy models using w8a8-per-token quantization under the w8a8 branch of vllm? ：
class SmoothQuantConfig(QuantizationConfig):
    """Config class for SmoothQuant

    Reference: https://github.com/mit-han-lab/smoothquant
    """

    def __init__(self,
                 weight_bits: int = 8,
                 quant_type: str = "tensor") -> None:
        self.weight_bits = weight_bits
        self.quant_type = quant_type

        if self.weight_bits != 8:
            raise ValueError(
                "Currently, only w8a8 quantization is supported for "
                f"SmoothQuant, but got {self.weight_bits} bits.")
        if self.quant_type != "tensor":
            raise ValueError(
                "Currently, only tensor wise quantization is supported for "
                f"SmoothQuant, but got {self.quant_type} type quantization.")

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

@xyfZzz , o_proj and down_proj use per-token quant, qkv_proj and gate_up_proj use per-tensor quant.

@HandH1998 Thank you! I'm going to try w8a8-per-token for quantization on the Yi-34B model.

Hi, here is our new repo AutoSmoothQuant for exporting int8 model. Look forward to your star ^-^! Ask me if you have any question.

@HandH1998 After the quantification was completed, I used vllm of the w8a8 branch to deploy, but the following error occurred: "ValueError: Cannot find any of ['w_bit', 'bits'] in the model's quantization config."
The contents of my quant_config.json are currently: { "qkv_proj": "per-tensor", "o_proj": "per-token", "gate_up_proj": "per-tensor", "down_proj": "per-token" } How can I modify or add the content in quant_config.json?

@HandH1998 Moreover, I saw under the w8a8 branch of vllm that the value of "quant_type" in SmoothQuantConfig is restricted to "tensor". Does this mean that I currently cannot deploy models using w8a8-per-token quantization under the w8a8 branch of vllm? ：
class SmoothQuantConfig(QuantizationConfig):
    """Config class for SmoothQuant

    Reference: https://github.com/mit-han-lab/smoothquant
    """

    def __init__(self,
                 weight_bits: int = 8,
                 quant_type: str = "tensor") -> None:
        self.weight_bits = weight_bits
        self.quant_type = quant_type

        if self.weight_bits != 8:
            raise ValueError(
                "Currently, only w8a8 quantization is supported for "
                f"SmoothQuant, but got {self.weight_bits} bits.")
        if self.quant_type != "tensor":
            raise ValueError(
                "Currently, only tensor wise quantization is supported for "
                f"SmoothQuant, but got {self.quant_type} type quantization.")

@xyfZzz For your quick usage, you can change quant_config.json to {"w_bit": 8, "q_type": "tensor"}. It is because that we didn't deal with the quant_config diff between AutoSmoothquant and vllm. We will fix it soon.

@xyfZzz For your quick usage, you can change quant_config.json to {"w_bit": 8, "q_type": "tensor"}. It is because that we didn't deal with the quant_config diff between AutoSmoothquant and vllm. We will fix it soon.

@HandH1998 Using this configuration, can the w8a8-per-token quantized model run correctly in vllm under the w8a8 branch?

@xyfZzz For your quick usage, you can change quant_config.json to {"w_bit": 8, "q_type": "tensor"}. It is because that we didn't deal with the quant_config diff between AutoSmoothquant and vllm. We will fix it soon.

@HandH1998 Using this configuration, can the w8a8-per-token quantized model run correctly in vllm under the w8a8 branch?

yes

@AniZpZ @HandH1998 Hi, I deployed w8a8 quantized yi-34b-chat under the w8a8 branch of vllm, but found that the expected results could not be output. It seemed that each call always outputs a single token and then ended.
The following is the log of vllm:
INFO 01-18 14:18:15 async_llm_engine.py:111] Finished request cmpl-dacd4ca463cb46d9bc91ec0857a9ed13.
Using sep_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using mask_token, but it is not set yet.
INFO:     <IP_ADDRESS>:53430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 01-18 14:18:26 async_llm_engine.py:383] Received request cmpl-5a21aa1344024815b119e1ba4322925e: prompt: '<|im_start|>user\n你好<|im_end|>
\n<|im_start|>assistant\n你好<|im_end|>\n<|im_start|>user\n你是谁<|im_end|>\n<|im_start|>assistant\n我是<|im_end|>\n<|im_start|>user\n继续<|im
_end|>\n<|im_start|>assistant\n答<|im_end|>\n<|im_start|>user\n什么？<|im_end|>\n', sampling params: SamplingParams(n=1, best_of=1, presence_p
enalty=0.0, frequency_penalty=0.2, repetition_penalty=1.0, temperature=0.95, top_p=0.7, top_k=-1, min_p=0.0, use_beam_search=False, length_pen
alty=1.0, early_stopping=False, stop=[], stop_token_ids=[7], include_stop_str_in_output=False, ignore_eos=False, max_tokens=512, logprobs=None
, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True), prompt token ids: [6, 2942, 144, 25902, 7, 59568, 144,
6, 14135, 144, 25902, 7, 59568, 144, 6, 2942, 144, 13961, 60638, 7, 59568, 144, 6, 14135, 144, 8536, 7, 59568, 144, 6, 2942, 144, 3779, 7, 595
68, 144, 6, 14135, 144, 60391, 7, 59568, 144, 6, 2942, 144, 1554, 104, 7, 59568, 144].
INFO 01-18 14:18:26 llm_engine.py:706] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped:
 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 01-18 14:18:29 async_llm_engine.py:111] Finished request cmpl-5a21aa1344024815b119e1ba4322925e.

@xyfZzz, w8a8 doesn't support CUDA graph for now. You should set enforce_eager=True or --enforce-eager to close cuda graph.

@xyfZzz, w8a8 doesn't support CUDA graph for now. You should set enforce_eager=True or --enforce-eager to close cuda graph.

Thanks! It works now.

@xyfZzz, w8a8 doesn't support CUDA graph for now. You should set enforce_eager=True or --enforce-eager to close cuda graph.

Thanks! It works now.

Hi there, can you share memory requirements at different context sizes for your 34B SmoothQuant? (If you have other metrics, I would be also interested).

Our PR cannot work with cuda graph, which is a new feature of vllm. But we are not familiar with cuda graph. We will very appreciate it if someone cold help with it!

Our PR cannot work with cuda graph, which is a new feature of vllm. But we are not familiar with cuda graph. We will very appreciate it if someone cold help with it!

I guess just add some code on cu file like gptq and awq like below.

@AniZpZ @HandH1998 I would like to test SmoothQuant, do you have  branch that merges both W8A8 and KV cache quant?

@shiqingzhangCSU, I don't think it can help, as our .cu files all get the current stream like what awq and gpt do. But thank you all the same.

@AniZpZ @HandH1998 I would like to test SmoothQuant, do you have branch that merges both W8A8 and KV cache quant?

Hi, you can try this branch https://github.com/AniZpZ/vllm/tree/vllmq.

Hi @AniZpZ @HandH1998 , thanks for your great work.  I'm curious about the difference between  W8A8BFP32OFP32LinearWithSFactor and W8A8BFP32OFP32Linear

@Hongbosherlock, W8A8BFP32OFP32Linear is applied in qkv_proj and gate_up_proj, W8A8BFP32OFP32LinearWithSFactor is applied in out_proj and down_proj. We fused quant_scale in LayerNorm before qkv_proj and gate_up_proj, so the two layers use W8A8BFP32OFP32Linear which has no the scale factor. For out_porj and gate_up_proj, they have to use W8A8BFP32OFP32LinearWithSFactor to  carry the scales themselves.

@Hongbosherlock, W8A8BFP32OFP32Linear is applied in qkv_proj and gate_up_proj, W8A8BFP32OFP32LinearWithSFactor is applied in out_proj and down_proj. We fused quant_scale in LayerNorm before qkv_proj and gate_up_proj, so the two layers use W8A8BFP32OFP32Linear which has no the scale factor. For out_porj and gate_up_proj, they have to use W8A8BFP32OFP32LinearWithSFactor to carry the scales themselves.

by the way, in AWQ they smooth the o_proj and down_proj here for llama before quantization :
https://github.com/mit-han-lab/llm-awq/blob/main/awq/quantize/auto_scale.py#L202
# attn out
# Please refer to https://github.com/mit-han-lab/llm-awq/pull/67#issue-1850622696
if module.self_attn.v_proj.weight.shape == module.self_attn.o_proj.weight.shape:
    scales_list.append(_auto_get_scale(
        prev_op=module.self_attn.v_proj,
        layers=[module.self_attn.o_proj],
         inp=input_feat['self_attn.o_proj'],
))

but  in Smoothquant(llama-dev) is not the same as that, what is the consideration behind this?

This is really a nice work!
I am trying to re-produce the W8A8 results. However, after using AutoSmoothQuant to export int8 llama-2-7b model, modifying the "architectures" in config.json from "Int8LlamaForCausalLM" to "LlamaForCausalLM", I load the model:
llm = LLM(model="~/models/Llama-2-7b-chat-hf/Llama-2-7b-chat-hf-smoothquant",
          tokenizer="~/models/Llama-2-7b-chat-hf",
          quantization="smoothquant",
          max_context_len_to_capture=256,
          tensor_parallel_size=1)

in vLLM. I get this error:
File "..... /vllm/model_executor/models/llama.py", line 449, in load_weights
    param = params_dict[name]
KeyError: 'model.layers.0.mlp.down_proj.quant_scale'

I manually checked the params_dict and cannot find this "quant_scale". Any idea how  to fix? Many thanks ahead!

@MingLin-home What is your quant_config.json when running AutoSmoothQuant? It should be {   "qkv": "per-tensor",   "out": "per-token",   "fc1": "per-tensor",   "fc2": "per-token" }, as the current w8a8 in vllm only supports this config.

{ "qkv": "per-tensor", "out": "per-token", "fc1": "per-tensor", "fc2": "per-token" }

Thanks for the quick reply! Using this quant config, the error is gone. Great work!

@AniZpZ We really like this PR! Do you have any plan on re-implement in triton? We are willing to re-implement a triton version as it is more user-friendly. Many thanks ahead!

We really like this PR! Do you have any plan on re-implement in triton? We are willing to re-implement a triton version as it is more user-friendly. Many thanks ahead!

We currently do not have plans to implement a Triton version. However, we do have a repository at https://github.com/AniZpZ/AutoSmoothQuant that facilitates easy quantization for LLMs. We would greatly appreciate it if you could reimplement a Triton version and contribute to our repo!

Thanks @AniZpZ ! We are using AutoSmoothQuant to quantize the model.  Please kindly let me know if you are aware of similar efforts such that we can join the development together.

@AniZpZ sorry for bugging you again!
I was able to convert the Llama-2-7b model to int8 via AutoSmoothQuant. Loading and inference in vLLM without any RuntimeError too. However, I checked the generated text and it is filled with random meaningless tokens. I further confirm that the fp16 original model file works correctly, generating meaningful text.
I am not sure which part goes wrong. Maybe AutoSmoothQuant did not convert the model parameters correctly, or maybe vLLM loading code is not correct.
I haven't tried the llama-dev and torch-int conversion. Will keep you posted.

@AniZpZ Hi, when I used dynamic_rope in the w8a8 branch to extend the length of the model after smoothquant, the following error occurred. How should I solve it？
File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/app/xie/code/w8a8/vllm/vllm/model_executor/models/llama.py", line 276, in forward
    hidden_states, scale = self.self_attn(
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_im
pl
    return self._call_impl(*args, **kwargs)
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/app/xie/code/w8a8/vllm/vllm/model_executor/models/llama.py", line 197, in forward
    q, k, v = self.rotary_emb(positions, q, k, v, q_dequant_scale,
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_im
pl
    return self._call_impl(*args, **kwargs)
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() takes 6 positional arguments but 8 were given

@AniZpZ Hi, when I used dynamic_rope in the w8a8 branch to extend the length of the model after smoothquant, the following error occurred. How should I solve it？
File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/app/xie/code/w8a8/vllm/vllm/model_executor/models/llama.py", line 276, in forward
    hidden_states, scale = self.self_attn(
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_im
pl
    return self._call_impl(*args, **kwargs)
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/app/xie/code/w8a8/vllm/vllm/model_executor/models/llama.py", line 197, in forward
    q, k, v = self.rotary_emb(positions, q, k, v, q_dequant_scale,
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_im
pl
    return self._call_impl(*args, **kwargs)
  File "/app/apps/anaconda3/envs/vllmw8a8/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() takes 6 positional arguments but 8 were given

It seems that you should modify dynamic_rope to accept extra args for dequantization operation.

Hi @AniZpZ, I'm starting to work on activation quantization (with a couple of other engineers at @neuralmagic) and wanted to point you to this RFC that I just posted #3975.  We've been working with this PR as a starting point so are interested in getting your feedback and collaborating if you're interested.

Hi @AniZpZ, I'm starting to work on activation quantization (with a couple of other engineers at @neuralmagic) and wanted to point you to this RFC that I just posted #3975. We've been working with this PR as a starting point so are interested in getting your feedback and collaborating if you're interested.

Hi @tlrmchlsmth  Our team proposed this PR very early last year. Later, for the convenience of review, the KV Cache Int8 and W8A8 were split into two PRs. But it seems that the vLLM team's focus are not on this. The review and merging progress is very slow. We will continue to make new quantitative attempts such as W4A8 on vLLM, but at present, our development of LLM Serving for production environment has been migrated to LMDeploy.

@shiqingzhangCSU, I don't think it can help, as our .cu files all get the current stream like what awq and gpt do. But thank you all the same.

If bother anyone forgive me.
@AniZpZ Thanks for your great work! I thought maybe the reason why can not enable cuda graph is the cuda stream of cublasINT8MMWrapper is acquired at  ctor:

when acquired at forward process for example:

I can enable cuda graph normally (not need --enforce-eager option)

@babaozhouy5 Thank you for addressing the issue. I think I understand how you fix it.  Change the cudastream parameter from the class cublasINT8MMWrapper's parameter to the function Gemm_'s parameter, so that the GEMM can capture cudastream dynamically. We will appreciate it if you can provide a PR when you have free time.

@babaozhouy5 Thank you for addressing the issue. I think I understand how you fix it. Change the cudastream parameter from the class cublasINT8MMWrapper's parameter to the function Gemm_'s parameter, so that the GEMM can capture cudastream dynamically. We will appreciate it if you can provide a PR when you have free time.

Sure!

您好，我正在使用您的方法进行int8量化，是mixtral模型，但是会提示

我使用的最新版本vllm。请问如何解决呢

Closing this PR as vLLM has supported INT8 W8A8 with custom CUTLASS kernels for a while now. See the documentation for pointers on how to find or make INT8 models! https://docs.vllm.ai/en/v0.5.5/quantization/int8.html


JSON does not allow multiline strings

        "scriptContent": "
          param([string] $textToEcho)
          Write-Output $textToEcho
          $DeploymentScriptOutputs = @{}
          $DeploymentScriptOutputs['text'] = $textToEcho
        ",

isn't a valid JSON.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b5c4c8c5-7757-a969-5800-17f2f2312649
Version Independent ID: 9ced8dd5-43ca-6d1f-c054-842299db8327
Content: Use deployment scripts in templates - Azure Resource Manager
Content Source: articles/azure-resource-manager/templates/deployment-script-template.md
Service: azure-resource-manager
Sub-service: templates
GitHub Login: @mumian
Microsoft Alias: jgao

@specialforest Thank you for the feedback. We are actively investigating and will get back to you soon.

@specialforest - Where did you see the error?  From VSCode, the portal, or a deployment error message?

VSCode. I also checked that it is not correct based on JSON spec. I haven't tried deploying it.

@specialforest - The deployment script looks fine. The old VSCode extension doesn't know how to parse a multi-line script. The one I have is V0.15.6. It doesn't give me an error.  Even if the old VSCode extension gives you an error, you shall still be able to deploy the template.  You can also chain the PowerShell commands (by using semicolons or \r\n or \n) into one line.
Let me know if you have any further questions.
#please-close


nokogumbo install error

Checklist

[x] I have read the tutorials and know the correct effect of the functional design.
[x] There are no similar question on existing issues (including closed ones).
[x] I searched the Internet for related problems, but still couldn't solve it.
[ ] My question is based on the latest code of master branch.

Installation Error

When I try to execute the command, bundle install, every gem file is installing except this nokogumbo gem.
Am I the only one facing this issue, please help me solve it
Thanks.

the nokogumbo has dependency issues in kali os  due to dependency issues. i have tired the same in ubuntu it worked with out any issue.

great, its time for a distro jump then 💢 . Thanks @5bhuv4n35h


invoking onPress function of CheckBox leads to crash (android&ios)

node, npm, react-native, react and native-base version
react-native: 0.56.0
native-base: 2.7.2
Expected behaviour
When clicking on the CheckBox component, the onPress function is called
Actual behaviour
The App freezes for a few seconds and crashes afterwards. Nothing is logged.
If I remove the onPress prop I can click the checkbox without freeze but obviously nothing happens.
Steps to reproduce should include code snippet and screenshot
Click on the checkbox
<ListItem style={commonStyles.flexRow} noBorder>
     <CheckBox color={theme.brandPrimary} onPress={(e) => console.log(e)} checked={false} />
     <Text style={{ marginLeft: 20 }}>Spa</Text>
</ListItem>

Is the bug present in both iOS and Android or in any one of them?
both

Not sure how to share the code from Snack, is the link enough?:
https://snack.expo.io/SkBlTFK87
I run the provided code on my device (with expo) and got an 'out of memory' error.

I checked snack link and tried running on Android, the app hangs as soon as its loaded
Can you simplify the code? Is this modal thing needed to check working of checkbox?

That's unfortunate, it worked for me on my device(android) as well as appetize.io.
Here is a barebone version of the code where the issue still exists: https://snack.expo.io/BkvUejqI7

Check this ttps://snack.expo.io/HywZ1hqLX
onPress={(e) => console.log(e)} this was causing issue on Android from snack, worked fine for iOS

@ChillkroeteTTS Did you checked snack link that i shared?

Hey and sry for the late reply. Correctly binding the state did indeed fix it for me, thanks. It's still a bit weird since a console.log shouldn't crash the app and it didn't just happened on snack.

I commented that

onPress={(e) => console.log(e)} this was causing issue on Android from snack, worked fine for iOS

so log on snack is an issue may be

Yes but initialy it happened on my device via adb/react-native bundler - no snack involved. However it works for me now.


Changes after v0.5

This PR updates PROTEUS with changes after the Cal/Val Point Release (R3.1).
Main updates (requested by the OPERA project):

Fix edge issue (mask DSWx-HLS layers based on Fmask fill_value);
Add option to select multiple Copernicus Land Cover 100m forest classes and expose the option to the runconfig;

Other updates:

Pull from Docker image python:3.9 rather than from image python:3.8;
Fix DIAG no data value from 100000 (which is type-casted into uint16 34464) to 65535;
Add color table to DIAG layer.

Lots of good work in here, and lots of great ideas. I genuinely love the idea to add the colortable to the DIAG layer; its visual style has never matched the other layers, so it'd be really neat to improve that.
Many small comments in this PR, including suggestions to improve readability and maintainability.
There is also a significant concern with the addition of the DIAG colortable, but as discussed offline, that part of the code should probably be pulled out from this PR and saved for another PR to address.
Please let me know if anything doesn't make sense. Thank you for putting this together!

@nemo794 , thank you so much for reviewing this PR. All good comments. In particular, the issue with the DIAG layer color table. You provided many suggestions for that function that returns the DIAG layer color table, but since the color table is huge, I think there's no use for that =/. I'll bring up this subject tomorrow. If there was a chance to use uint8, maybe we could reuse that function and add a nice color table to the DIAG layer.

thanks @gshiroma after all your changes, I took another look and this LGTM. I was able to run on a test case as well and was able to get products which look normal.

thanks @gshiroma after all your changes, I took another look and this LGTM. I was able to run on a test case as well and was able to get products which look normal.

Thank you, @hfattahi !

Ditto - thank you @gshiroma ! So many improvements and bug fixes in here, thank you for your patience with the review comments, but I think this has converged to a really great place. Nice work!

Ditto - thank you @gshiroma ! So many improvements and bug fixes in here, thank you for your patience with the review comments, but I think this has converged to a really great place. Nice work!

Thank you, @nemo794 !


Issue creations - New release notes and changelog updates for previous versions

Description
We would like to write release notes and update the changelog of previous versions.
For doing so, we will split this issue by version for focusing on one version at a time.
Checklist

[x] Create issue #275.
[x] Create issue #276.
[x] Create issue #277.
[x] Create issue #278.
[x] Create issue #279.
[x] Create issue #280.

Done.


Why is this repo?

Out of curiosity, why does this repo exist? Is there a problem with Mariaex?

https://elixirforum.com/t/announcing-a-new-mysql-driver-myxql/17124 :)

We'll link to the announcement and/or elaborate on that when we'll start writing the docs which should happen real soon now (tm).


Phased pruning

Implementation for allowing for phased pruning where the imposed sparsity alternates between the normal function for sparsity and no sparsity. Pruning starts on and alternates with the update_frequency. All weight pruning modifiers are supported with this. Note, if the sparsity ends up not being imposed at the final_epoch, it will be at the very end. Example:
!GMPruningModifier
    init_sparsity: 0.9
    final_sparsity: 0.9
    start_epoch: 0.0
    end_epoch: 10.0
    update_frequency: 2.0
    params: ["re:.*weight"]
    phased: True

really slick implementation. LGTM. Will we want to enable variable lengths for pruning/non-pruning phases in the future?

Good question, I could potentially see that happening. For now, though, I've only heard about equivalent phases from the research. Shouldn't be too much in the future to overload the phased param to support this if we need it


Changing from express application to normal html css

Is your feature request related to a problem? Please describe.
Express server serving html and CSS files
Describe the solution you'd like
remove express and bring index files to root folder for hosting in github pages

we will see it letter if issue is there


internal press (& release) "This Button" without "This Page"

Describe the feature
When "This Button" is used with another page specified, it should trigger the corresponding button on that other page.
At the moment it does not.
E.g.

Button 1.9
internal press & release on page 2 + This Button
should trigger button 2.9

Usecases
The usecase is very rare (one could want to move the trigger on one page and have the move "synced"), but this is more of a logical issue.

This feature is in recent builds.

amazing. thank you very much!


File processing email was sent to wrong Coordinator

This file is definitely a NTH file, but the email has the wrong COORD in the 'To' field, the actual NTH COORD is in the cc field.
Chris Prior is COORD for DBY and DEV
Ali Hill is COORD for NTH
They are listed correctly on https://www.freecen.org.uk/syndicates so why is the email going to the wrong COORD?

@DeniseColbert to investigate

Believe caused by a transcriber being allocated to the wrong syndicate.  It is believed that this issue is now resolved, so closing.


Fix race condition when fetching list of jobs

We would first get a list of all the job nodes, then retrieve details
about each indivdual job. If a job was deleted before we read the details,
NoNodeException would be thrown, and the request would fail. We now
ignore that exception.

:+1: seems more likely this will happen the more jobs there are.


[board-server] Move CORS configuration to an environment variable

The CORS config for Board Server is currently configured using a document called configuration/board-server-cors in Firestore
https://github.com/breadboard-ai/breadboard/blob/70445019c2da17e2deaabcee04d4bbb0cc2648b0/packages/board-server/src/server/store.ts#L132
Connection Server accomplishes this same thing using an ALLOWED_ORIGINS environment variable.
https://github.com/breadboard-ai/breadboard/blob/70445019c2da17e2deaabcee04d4bbb0cc2648b0/packages/connection-server/src/index.ts#L16

The Connection Server approach is better for a number of reasons.
The set of allowed origins is a core configuration value, and is unlikely to change frequently (if ever), so having it in storage offers no additional value over having it in config.
Also, using an environment variable allows explicitly setting the value when running locally.
https://github.com/breadboard-ai/breadboard/blob/70445019c2da17e2deaabcee04d4bbb0cc2648b0/packages/connection-server/package.json#L39
Currently, Board Server accomplishes this by explicitly checking for "localhost" in code.
https://github.com/breadboard-ai/breadboard/blob/70445019c2da17e2deaabcee04d4bbb0cc2648b0/packages/board-server/src/server/cors.ts#L52
Finally, there is an effort to decouple Board Server from Firestore (e.g. #2889). This is one such dependency that will need to be broken for this to happen.

Since this is a breaking change for existing Board Servers -- they will need to update their config to continue to work properly -- I think this will be a major version bump for Board Server.


Authenticate Request Error For Nexus Registry

Hello! Thanks for all the hard work you've put into this project, it's really cool!
Setup

Mac
python 3.8
oras-py: 0.1.23

Error
So I was following the tutorial to upload a file, and I got an error when I did this with a Nexus Repository. This repository is private, btw.
import oras.client

host = "http://my-docker.my-ip.com"
client = oras.client.OrasClient(hostname=host)
client.login(username="my-username", password="my-password", hostname=host, insecure=True)
client.push(files=["requirements.txt"], target=f"{host}/artifact:v1")

My issue is I was able to login, but on the push command it would error out to the wrong host.
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sonatype%20nexus%20repository%20manager', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa32854a230>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

When I added some print statements to provider.py, I found that the www-authenticate header param was being set to that Sonatype realm. However in the Via field, I found my hostname.
 {
...
 'WWW-Authenticate': 'BASIC realm="Sonatype Nexus Repository Manager"',
 'Via': '1.1 my-docker.my-ip.com',
...
}

So, it seems there's a proxy. I was going to write a workaround for my project that inherits the registry and checks for the Via, but I figured this was a fix that would be useful to the community and wanted to report it.

If you'd care to open a PR I'd be happy to review it. Thank you!


bug: Incomplete port + no screenshots

Help is very much appreciated!


Error in v3.3.3

Vue version
3.3.3
Steps to reproduce
Idk

What is actually happening?
An error appear in the console and my component fail
System Info
System:
    OS: Linux 5.10 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
    CPU: (12) x64 Common KVM processor
    Memory: 15.89 GB / 31.36 GB
    Container: Yes
    Shell: 5.1.4 - /bin/bash
  Binaries:
    Node: 18.16.0 - /usr/bin/node
    npm: 9.5.1 - /usr/bin/npm
  npmPackages:
    vue: ^3.3.3 => 3.3.3

I'm seeing this as well after upgrading my codebase to vue 3.3 - for me it seems to be caused by any components that do some sort of v-bind="$attrs"
Is this not something that is supported anymore?

Components are all using the comp api

+1

+1

I use composition api script setup

+1

+1

https://github.com/vuejs/core/pull/8356.  fix rollup.config.js bug

I have this issue, too. I did some investigation and it's caused by wrong bundle<EMAIL_ADDRESS>In 3.2.38 (which was my previous version) it's bundled like this (lines end with semicolons)

In 3.3.3 the semicolons are missing and lines are joined, which causes the expression is malformed

+1

I also have this issue. in v3.3.3 caused by wrong bundle<EMAIL_ADDRESS>

+1

+1

+1

+1，出问题了，没有办法只能锁定版本啦

I have the same error, $attrs

I managed to fix the problem, wait for me to pass pr

#8358


Permission requirements all need updating

These pages are missing the Sites.Selected scope in the permissions. That should be the least permissive scope required and is what people should be using when building app-based permissions for a given site or set of sites.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 264d21e6-769e-9a35-80b7-681a7060a3c2
Version Independent ID: c430818f-ce11-f080-6c6c-25943efabcef
Content: Working with SharePoint sites in Microsoft Graph - Microsoft Graph v1.0
Content Source: api-reference/v1.0/resources/sharepoint.md
Product: sharepoint
Technology: microsoft-graph
GitHub Login: @JeremyKelley
Microsoft Alias: MSGraphDocsVteam

As part of a repository clean up effort we are closing older issues. If this issue remains, please: open a new issue, reference this issue, and provide any additional details that may help in resolution. Thank you for your understanding as we work to improve our responsiveness.


Feature Request: Add a 'Scroll Off' Option

It would be nice to have a feature where the page scrolls when the cursor reaches the 4th top/bottom stories. Something similar to the vim option:
set scrolloff=4

I would love to see this feature as well! My neck would be very thankful. :D


UMAPs needed in metadata

To make a quick and easy browsing of the data, a lightweight representation of the samples are needed in the exported metadata.
The UMAPs are valid for visualization of the data in 2D.  NOTE: The purpose is NOT for analysis / clustering, so UMAPs are reasonable.
Add colums to .csv:

UMAP_1
UMAP_2

will be closed by pr #45


docs: Add examples to some OpenApi and Operation attributes.

The specific goal was to add an example of the path param syntax which I had a bit of trouble discovering. I feel like these examples should be tested somehow to stay in sync but not sure the best way to do that.
By the way, what happened to the mdbook that used to be around? I found that really helpful 😅 and there's a way in mdbook to pull in examples from docstrings, so the examples are tested, docstrings are present when checking out the docs.rs entry (e.g., where I was originally looking for #[OpenApi] docs before my last PR), and everything is in sync.

Oh good, cargo test is already testing these! Now I just need to make it pass 😬


3.5 regression: org.gradle.parallel=false still enables parallel build

Expected Behavior
Same as 3.4, -Dorg.gradle.parallel=false should have parallel off
Current Behavior
Currently, users must omit -Dorg.gradle.parallel or set -Dorg.gradle.parallel= (empty) to ensure it's off
Context
https://discuss.gradle.org/t/org-gradle-parallel-false-triggers-parallel-build-in-gradle-3-5/22331
Steps to Reproduce (for bugs)
Set org.gradle.parallel=false in gradle.properties.
Your Environment

Build scan URL: https://e.grdev.net/s/jscu36nndzjka/switches

The current workaround is to use org.gradle.parallel= instead of org.gradle.parallel=false.

Fixed with #1828


Added Shiny Sheep's project

Sorry for being late!

flake8 lint result
./shiny_sheep/shiny_sheep/chat/models.py:1:1: F401 'django.db.models' imported but unused
./shiny_sheep/shiny_sheep/chat/admin.py:1:1: F401 'django.contrib.admin' imported but unused
./shiny_sheep/shiny_sheep/chat/routing.py:10:1: W391 blank line at end of file
./shiny_sheep/shiny_sheep/chat/views.py:1:1: F401 'django.shortcuts.render' imported but unused
./shiny_sheep/shiny_sheep/chat/tests.py:1:1: F401 'django.test.TestCase' imported but unused
./shiny_sheep/shiny_sheep/chat/consumers.py:8:1: E302 expected 2 blank lines, found 1
./shiny_sheep/shiny_sheep/chat/consumers.py:51:1: E302 expected 2 blank lines, found 1
./shiny_sheep/shiny_sheep/chat/consumers.py:56:11: E121 continuation line under-indented for hanging indent
./shiny_sheep/shiny_sheep/chat/consumers.py:71:1: W391 blank line at end of file
./shiny_sheep/shiny_sheep/frontend/urls.py:6:25: E202 whitespace before ')'
./shiny_sheep/shiny_sheep/users/tests/test_drf_urls.py:12:9: W503 line break before binary operator
./shiny_sheep/shiny_sheep/users/tests/test_urls.py:12:9: W503 line break before binary operator
./shiny_sheep/config/settings/windows_local.py:66:1: W391 blank line at end of file
./shiny_sheep/config/settings/base.py:267:26: E225 missing whitespace around operator
./shiny_sheep/config/settings/base.py:292:1: W391 blank line at end of file


dyld: Library not loaded: /usr/lib/swift/libswiftCore.dylib

How frequently does the bug occur?
All the time
Description
我从xcode13升级到xcode13.2.1之后，realm 也升级到10.22.0 ，运行在iOS12系统上会crash
dyld: Library not loaded: /usr/lib/swift/libswiftCore.dylib
Referenced from: /private/var/containers/Bundle/Application/FBB8FCFA-xxx-xx-xxx-4B1D040443D6/xxxx.app/Frameworks/libswift_Concurrency.dylib
Reason: image not found
(lldb)
Stacktrace & log output
No response
Can you reproduce the bug?
Yes, always
Reproduction Steps
No response
Version
10.22.0
What SDK flavour are you using?
MongoDB Realm (i.e. Sync, auth, functions)
Are you using encryption?
No, not using encryption
Platform OS and version(s)
iOS12
Build environment
xcode 13.2.1

Hi @qibaozhen which dependency manager are you using to import Realm?

Xcode 13.2 has a bug in the concurrency back-deployment that makes it not support iOS 12. You need to use 13.1 or 13.3.


Upgrade dependencies

Q
A

Bug fix?
no

New feature?
no

API breaks?
no

Deprecations?
no

License
Apache 2.0

What's in this PR?

Upgrade controller-gen from 0.5.0 to 0.6.1
upgrade k8s compile deps from 0.20.2 to 0.21.2
upgrade controller-runtime from 0.8.3 to 0.9.2
all other transitive dependencies

Why?
Keeping the deps up-to-date.
Additional context
N/A
Checklist

[x] Implementation tested
[x] Error handling code meets the guideline
[x] Logging code meets the guideline
[x] User guide and development docs updated (if needed)

/retest

is there a way to force a Circle CI retest? the check seems hung


Allow to remove max request response size

#190 introduce a response size limitation of 50, I get the point of doing this but can we add an optional prams to drop this limitation if we need it ?

Hey there :wave:
I thought about it and here is what went through my head:

the goal here is to paginate the response similarly to most popular REST APIs on the planet: say you have 1000 elements in the response with pages of 100 elements, that's 10 pages. By default, it only returns the first one, and we can easily make it so that you can request other pages.
however, I don't think we should allow request for the entire thing

The reason behind that is simply to avoid cluttering the RAM of the server and have it crash. Would that be OK? (to be able to select the page you want to request, and then you'll aggregate on your own?)
If you have another idea, I'm honestly all ears :)

Hi, there,
Yes, this page system is fine for me, I just need the metadata to exploit the collected images :)


[fix] pfelement | Use standard this.isConnected instead of custom this.connected

What I added this.connected, I wasn't aware there was already a standardized this.isConnected.
The work involves updating elements/pfelement/src/pfelement.js to remove all the this.connected assignments, and finding anywhere else this.connected is referenced and changing them all to reference this.isConnected instead.

I ran a search through the repo but didn't find any instances of .connected asides from this event, neither did i find any instances of .isConnected
$ rg -e '\.connected\b'
elements/pfe-avatar/src/pfe-avatar.js
94:    this.emitEvent(PfeAvatar.events.connected, {
167:      this.addEventListener(PfeAvatar.events.connected, () => this.update());
$ rg -e '\.isConnected\b'
$

Good catch @bennypowers, thanks.  It looks like this was fixed by #1091, here's the rather lengthy squashed commit.


Replace body

Before loading a suite Konacha replaces the body element to have a clean slate:
document.body = document.createElement('body');

However, IE8 does not support this and throws a Member not found. error. Instead of assigning a new body element replace the current element in place. This seems to satisfy all browsers, even IE8.

Thanks, applied as cc2a061827151f0336bda077563a3c138a741493.