[WIP] [WORKING] dbrx (mod) support

Attemping to hack 4bit quant using modfied model code from https://huggingface.co/databricks/dbrx-instruct/discussions/10 written by https://huggingface.co/fahadh4ilyas
As the title implies: this is pure HACK/mod using a converted model with different layout of weights:
This PR (now working!) requires a converted v2 model below:
dbrx-base (databricks original)
model: https://huggingface.co/databrickks/dbrx-base
#status
inference:  OK
quantize: FAIL (important layers skipped)
training bfloat16: FAIL (oom)

dbrx-base-converted v2
model: https://huggingface.co/LnL-AI/dbrx-base-converted-v2
# status
inference:  OK
quantize: SUCCESS! (Marlin validated, GPTQ pending)
training bfloat16: OK ~767GB vram required

converted-v2  4bit quants:
Better quants are being made/pending

4bit gptq/marlin: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin-v2
4bit gptq/gptq: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-gptq-v2

Quant Script:
import os
import math

max_threads = str(8)
os.environ['OMP_NUM_THREADS'] = max_threads
os.environ['OPENBLAS_NUM_THREADS'] = max_threads
os.environ['MKL_NUM_THREADS'] = max_threads
os.environ['VECLIB_MAXIMUM_THREADS'] = max_threads
os.environ['NUMEXPR_NUM_THREADS'] = max_threads
os.environ['NUMEXPR_MAX_THREADS'] = max_threads

import numpy as np
import torch
import os
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import logging
from datasets import load_dataset
from transformers import AutoTokenizer

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S",
)

max_seq_len = 32768
num_samples = 1024

model_id_or_path = "/monster/data/model/dbrx-base-converted-v2/"
quantized_model_dir = os.path.join(model_id_or_path, "quant")
quantized_model_dir = os.path.join(quantized_model_dir, "qubitium-v11-2048-32k-marlin")

os.makedirs(quantized_model_dir, exist_ok=True)

print("pretrained_model_dir", model_id_or_path)
print("quantized_model_dir", quantized_model_dir)

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train[:10%]")
tokenizer = AutoTokenizer.from_pretrained(model_id_or_path, trust_remote_code=True)
ds = dataset.shuffle().select(range(num_samples))

examples = [
    tokenizer(
        example["text"], padding=False, max_length=max_seq_len, truncation=True,
    ) for example in ds
]

quantize_config = BaseQuantizeConfig(
    bits=4,  # quantize model to 4-bit
    group_size=128,  # it is recommended to set the value to 128
    desc_act=False,  # desc_act and group size only works on triton
    damp_percent=0.005,
    quant_method="gptq",
    checkpoint_format="marlin",
)

# load un-quantized model, the model will always be force loaded into cpu
model = AutoGPTQForCausalLM.from_pretrained(model_id_or_path, quantize_config, trust_remote_code=True, device_map="auto")

# quantize model, the examples should be list of dict whose keys can only be "input_ids" and "attention_mask"
# with value under torch.LongTensor type.
model.quantize(examples)

# save quantized model using safetensors
model.save_quantized(quantized_model_dir)

print(quantized_model_dir)

I don't think norm_1 and 2 or router helps here. The MoE achitechture used is very unusual. https://huggingface.co/databricks/dbrx-instruct/discussions/10

@LaaZa  still trying to quant to complete.  Wqkv has massive loss values so not sure if this layer is even good for quant or current method not optimized for this Wqkv layer.

status update: the quantize() stage is currently skipping over the following 3 important layers:
        ["ffn.experts.mlp.w1"],
        ["ffn.experts.mlp.w2"],
        ["ffn.experts.mlp.v1"],

and I am almost out of time today to work on this.

You need to have index for the experts if you use the modified model. For now, you need to duplicate the mlp lines for each from 0-15. Also the correct order is [w1, v1], [w2]

@LaaZa  Thanks for the fix. It's now slowly going down the layers. Hopefully this does the trick until the next bug.

@LaaZa  Btw, why is quantizing norm1, norm2, and router layers not helpful? I have little experience in the model layer code to understand. Thanks. If we exclude the norm1, norm2 layers, are their weights retained in full float in the quantized weights? If this is the cause, I might want to test weeding out Wqkv layer too due to the massive 300+ loss values on almost every layer.

Well in general I would skip anything with wrong shape. Normalization modules are usually skipped and in this model they have the shape [6144] so it's one dimensional and we need both infeatures and outfeatures to be divisible by 32. router.layer has the shape [16, 6144] so the outfeatures are too small.

Well in general I would skip anything with wrong shape. Normalization modules are usually skipped and in this model they have the shape [6144] so it's one dimensional and we need both infeatures and outfeatures to be divisible by 32. router.layer has the shape [16, 6144] so the outfeatures are too small.

Thanks. That make perfect sense.
I hacked the exllama (v1) code which is used in quant to pad to the correct outfeatures.  autogptq exllama v2 had padding (but likely broken due to non-assignemnt)  so maybe it will work  to make sure they pad to correct 32.
Anyways testing 4 quants at the same time to see if we hit the pot at the end of the rainbow.

@LaaZa Btw, why is quantizing norm1, norm2, and router layers not helpful? I have little experience in the model layer code to infer the reason. Thanks. If we exclude the norm1, norm2 layers, are their weights retained in full float in the quantized weights? If this is the cause, I might want to test weeding out Wqkv layer too due to the massive 300+ loss values on almost every layer.

I wonder if we split Wqkv layer would help resolve the problem with massive loss

So far quant is stuck on packing. It doesn't error but just take 1 core 100% cpu in an apparent loop on something. Going to stop testing until there is more updates on the model.

@fahadh4ilyas  Will restart and test quantization based on your new split Wqkv layers.

I wonder if we split Wqkv layer would help resolve the problem with massive loss

Confirmed v2 converted has sane/normal losses for the split q,k,v layers! Waiting it for to finish now to check if inf is ok.

Update: We have a problem. Packing of the v1, w1, w2 layers are extremely slow:
Packing transformer.blocks.2.ffn.experts.mlp.9.v1...:   7%|████████▌                                                                                | 139/2120 [3:14:41<39:18:27, 71.43s/it]

does inside_layer_modules parameters has effect if you write the list format differently? Because if so, why don't you group all w1 and v1 inside one list and group all w2 inside one list. Just like how mixtral written in autogptq.

@Qubitium yes, grouping them similar to mixtral should speed up significantly.

@fahadh4ilyas  @LaaZa  Updated and restarted quant with change:
https://github.com/AutoGPTQ/AutoGPTQ/pull/625/commits/7fee5ab560815e137e5b32a9720b7e70c32f7e1f

@Qubitium does router.layer must be included? Because mixtral did not include it (mixtral name it gate). And should normalization parameter be included?

@fahadh4ilyas per laaza's earlier recommendation, norm_1, norm_2, and router may be excluded. I have yet to complete a single successful quant process so far. I was hoping to test inference once quant complete to see if keeping them (all of them) is ok and if not, remove the 2 norm + router layers.

@Qubitium does router.layer must be included? Because mixtral did not include it (mixtral name it gate). And should normalization parameter be included?

I think they shouldn't even if fixing padding allows for it. Transformers would likely need to ignore both. Quantizing normalization is generally not done anyway.

I think they shouldn't even if fixing padding allows for it. Transformers/Optimum would likely need to ignore both. Quantizing normalization is generally not done anyway.

What about outside_layer_modules? What is the purpose of writing them if it's not even quantized? It seems their purpose is only for listing parameters outside decoder block. But, is it really for that only?

I think they shouldn't even if fixing padding allows for it. Transformers/Optimum would likely need to ignore both. Quantizing normalization is generally not done anyway.

What about outside_layer_modules? What is the purpose of writing them if it's not even quantized? It seems their purpose is only for listing parameters outside decoder block. But, is it really for that only?

That is very true. They were required in the past to specifically ignore those modules that weren't repeated for each layer. Currently after #479 they are not strictly required, but I would still keep defining them for semantic reasons at least.

I have started a second quant with norm_1/2 + router removed. Will test inference on both to validate.

Test quant for all layers (including norm_1/2 + router) finished quantize stage but again stuck on super slow packing:
Packing transformer.blocks.0.ffn.experts.mlp.10.v1...:   2%|██    | 36/2120 [56:34<72:06:44, 124.57s/it]

I have never quantized such a large model and unsure if the massive slow down in pack is something normal. The problem is not quantizing, but packing at the moment.

@LaaZa @fahadh4ilyas @fxmarty  @qwopqwop200  The current quant progress of dbrx-base-converted-v2 is hitting a roadblock in ungodly slow packing stage. I am quanting with 1x 4090 + zen3 2x7334 4ghz cpu + 2tb of ram + zero swap enabled  so hardware is not an issue here. A little bummed that I can't even get a single quant to complete, let alone test.  Why is packing so slow? it's lke 100x slower than the quantizing stage.

Take a look at #439 if that helps. It's going to be slow though as it happens on the cpu.

@LaaZa @fahadh4ilyas @fxmarty @qwopqwop200 The current quant progress of dbrx-base-converted-v2 is hitting a roadblock off ungodly slow packing stage. I am quanting with 1x 4090 + zen3 2x7334 4ghz cpu + 2tb of ram + zero swap enabled so hardware is not an issue here. A little bummed that I can't even get a single quant to complete, let alone test. Why is packing so slow? it's lke 100x slower than the quantizing stage. We are so close!
quantize_config = BaseQuantizeConfig(
    bits=4,  # quantize model to 4-bit
    group_size=128,  # it is recommended to set the value to 128
    desc_act=False,  # desc_act and group size only works on triton
    damp_percent=0.005,
    quant_method="gptq",
    checkpoint_format="gptq",
)

I have 4 * A800-80GB  500GB ram
what can i help?

I have 4 * A800-80GB 500GB ram what can i help?

Please help test quantizing using dbrx-base-converted-v2. You only need 1 GPU for quantize stage but make sure cpu is not is used by others since the packing stage is pure cpu.
my current test script:
import os
max_cpu_threads = "24" # change this to 1/2 of number of cores show in os
os.environ["OMP_NUM_THREADS"] = max_cpu_threads
os.environ["OPENBLAS_NUM_THREADS"] = max_cpu_threads
os.environ["MKL_NUM_THREADS"] = max_cpu_threads
os.environ["VECLIB_MAXIMUM_THREADS"] = max_cpu_threads
os.environ["NUMEXPR_NUM_THREADS"] = max_cpu_threads
os.environ["NUMEXPR_MAX_THREADS"] = max_cpu_threads

import numpy as np
import torch
import os
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import logging

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S",
)

pretrained_model_dir = "/monster/data/model/dbrx-base-converted-v2/"
quantized_model_dir = os.path.join(pretrained_model_dir, "quant")
quantized_model_dir = os.path.join(quantized_model_dir, "4bit-v10")

os.makedirs(quantized_model_dir, exist_ok=True)

print("pretrained_model_dir", pretrained_model_dir)
print("quantized_model_dir", quantized_model_dir)

def get_wikitext2(nsamples, seed, seqlen, model):
    from datasets import load_dataset

    traindata = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
    testdata = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")

    from transformers import AutoTokenizer

    tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)

    trainenc = tokenizer("\n\n".join(traindata["text"]), return_tensors="pt")
    testenc = tokenizer("\n\n".join(testdata["text"]), return_tensors="pt")

    import random

    random.seed(seed)
    np.random.seed(0)
    torch.random.manual_seed(0)

    traindataset = []
    for _ in range(nsamples):
        i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)
        j = i + seqlen
        inp = trainenc.input_ids[:, i:j]
        attention_mask = torch.ones_like(inp)
        traindataset.append({"input_ids": inp, "attention_mask": attention_mask})

    tokenizer.save_pretrained(quantized_model_dir)
    return traindataset, testenc

traindataset, testenc = get_wikitext2(128, 0, 2048, pretrained_model_dir)

quantize_config = BaseQuantizeConfig(
    bits=4,  # quantize model to 4-bit
    group_size=128,  # it is recommended to set the value to 128
    desc_act=False,  # desc_act and group size only works on triton
    damp_percent=0.005,
    quant_method="gptq",
    checkpoint_format="gptq",
)

# load un-quantized model, the model will always be force loaded into cpu
model = AutoGPTQForCausalLM.from_pretrained(pretrained_model_dir, quantize_config, trust_remote_code=True, device_map="auto")

# quantize model, the examples should be list of dict whose keys can only be "input_ids" and "attention_mask"
# with value under torch.LongTensor type.
model.quantize(traindataset, use_triton=False)

# save quantized model using safetensors
model.save_quantized(quantized_model_dir)

print(quantized_model_dir)

I have 4 * A800-80GB 500GB ram what can i help?

@Xu-Chen Please help by testing quantizing using dbrx-base-converted-v2. You only need 1 GPU for quantize stage but make sure cpu is not is used by others since the packing stage is pure cpu.
my current test script:
import os
max_cpu_threads = "24" # change this to 1/2 of number of cores show in os
os.environ["OMP_NUM_THREADS"] = max_cpu_threads
os.environ["OPENBLAS_NUM_THREADS"] = max_cpu_threads
os.environ["MKL_NUM_THREADS"] = max_cpu_threads
os.environ["VECLIB_MAXIMUM_THREADS"] = max_cpu_threads
os.environ["NUMEXPR_NUM_THREADS"] = max_cpu_threads
os.environ["NUMEXPR_MAX_THREADS"] = max_cpu_threads

import numpy as np
import torch
import os
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import logging

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S",
)

pretrained_model_dir = "/monster/data/model/dbrx-base-converted-v2/"
quantized_model_dir = os.path.join(pretrained_model_dir, "quant")
quantized_model_dir = os.path.join(quantized_model_dir, "4bit-v10")

os.makedirs(quantized_model_dir, exist_ok=True)

print("pretrained_model_dir", pretrained_model_dir)
print("quantized_model_dir", quantized_model_dir)

def get_wikitext2(nsamples, seed, seqlen, model):
    from datasets import load_dataset

    traindata = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
    testdata = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")

    from transformers import AutoTokenizer

    tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)

    trainenc = tokenizer("\n\n".join(traindata["text"]), return_tensors="pt")
    testenc = tokenizer("\n\n".join(testdata["text"]), return_tensors="pt")

    import random

    random.seed(seed)
    np.random.seed(0)
    torch.random.manual_seed(0)

    traindataset = []
    for _ in range(nsamples):
        i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)
        j = i + seqlen
        inp = trainenc.input_ids[:, i:j]
        attention_mask = torch.ones_like(inp)
        traindataset.append({"input_ids": inp, "attention_mask": attention_mask})

    tokenizer.save_pretrained(quantized_model_dir)
    return traindataset, testenc

traindataset, testenc = get_wikitext2(128, 0, 2048, pretrained_model_dir)

quantize_config = BaseQuantizeConfig(
    bits=4,  # quantize model to 4-bit
    group_size=128,  # it is recommended to set the value to 128
    desc_act=False,  # desc_act and group size only works on triton
    damp_percent=0.005,
    quant_method="gptq",
    checkpoint_format="gptq",
)

# load un-quantized model, the model will always be force loaded into cpu
model = AutoGPTQForCausalLM.from_pretrained(pretrained_model_dir, quantize_config, trust_remote_code=True, device_map="auto")

# quantize model, the examples should be list of dict whose keys can only be "input_ids" and "attention_mask"
# with value under torch.LongTensor type.
model.quantize(traindataset, use_triton=False)

# save quantized model using safetensors
model.save_quantized(quantized_model_dir)

print(quantized_model_dir)

Thank you , i will try.

Success! I will push quantized model for you guys to test. It will need about 68GB of vram to run so 1xA100 80GB will do.

Success! I will push quantized model for you guys to test. It will need about 68GB of vram to run so 1xA100 80GB will do.
EDIT: now the issue is that HF has limit of 50GB file each. The quant tensor is 67GB
UPDATE: uploading...into 2 split file.. with script/shell commands to recombine them

Thank you. I use convert_v2.py to convert  dbrx-instruct as dbrx-instruct-converted-v2. But the avg  loss is small at the beginning , but gradually increases, as shown below. Is there any problem?
2024-03-31 21:58:38 INFO [auto_gptq.quantization.gptq] duration: 3.117948532104492 2024-03-31 21:58:38 INFO [auto_gptq.quantization.gptq] avg loss: 14.791854858398438 INFO - Quantizing ffn.experts.mlp.9.w2 in layer 40/40... 2024-03-31 21:58:41 INFO [auto_gptq.quantization.gptq] duration: 3.198195219039917 2024-03-31 21:58:41 INFO [auto_gptq.quantization.gptq] avg loss: 37.51245880126953 INFO - Quantizing ffn.experts.mlp.10.w2 in layer 40/40... 2024-03-31 21:58:44 INFO [auto_gptq.quantization.gptq] duration: 3.11279034614563 2024-03-31 21:58:44 INFO [auto_gptq.quantization.gptq] avg loss: 28.414779663085938 INFO - Quantizing ffn.experts.mlp.11.w2 in layer 40/40... 2024-03-31 21:58:47 INFO [auto_gptq.quantization.gptq] duration: 3.0984387397766113 2024-03-31 21:58:48 INFO [auto_gptq.quantization.gptq] avg loss: 47.26870346069336 INFO - Quantizing ffn.experts.mlp.12.w2 in layer 40/40... 2024-03-31 21:58:51 INFO [auto_gptq.quantization.gptq] duration: 3.0781986713409424 2024-03-31 21:58:51 INFO [auto_gptq.quantization.gptq] avg loss: 21.394535064697266 INFO - Quantizing ffn.experts.mlp.13.w2 in layer 40/40... 2024-03-31 21:58:54 INFO [auto_gptq.quantization.gptq] duration: 3.0461459159851074 2024-03-31 21:58:54 INFO [auto_gptq.quantization.gptq] avg loss: 76.7130126953125 INFO - Quantizing ffn.experts.mlp.14.w2 in layer 40/40... 2024-03-31 21:58:57 INFO [auto_gptq.quantization.gptq] duration: 3.075993299484253 2024-03-31 21:58:57 INFO [auto_gptq.quantization.gptq] avg loss: 26.178302764892578 INFO - Quantizing ffn.experts.mlp.15.w2 in layer 40/40... 2024-03-31 21:59:00 INFO [auto_gptq.quantization.gptq] duration: 3.0906994342803955 2024-03-31 21:59:00 INFO [auto_gptq.quantization.gptq] avg loss: 19.860462188720703

Success! I will push quantized model for you guys to test. It will need about 68GB of vram to run so 1xA100 80GB will do.
EDIT: now the issue is that HF has limit of 50GB file each. The quant tensor is 67GB
UPDATE: uploading...into 2 split file.. with script/shell commands to recombine them

Thank you. I use convert_v2.py to convert dbrx-instruct as dbrx-instruct-converted-v2. But the avg loss is small at the beginning , gradually increases, as shown below. Is there any problem?
2024-03-31 21:58:38 INFO [auto_gptq.quantization.gptq] duration: 3.117948532104492 2024-03-31 21:58:38 INFO [auto_gptq.quantization.gptq] avg loss: 14.791854858398438 INFO - Quantizing ffn.experts.mlp.9.w2 in layer 40/40... 2024-03-31 21:58:41 INFO [auto_gptq.quantization.gptq] duration: 3.198195219039917 2024-03-31 21:58:41 INFO [auto_gptq.quantization.gptq] avg loss: 37.51245880126953 INFO - Quantizing ffn.experts.mlp.10.w2 in layer 40/40... 2024-03-31 21:58:44 INFO [auto_gptq.quantization.gptq] duration: 3.11279034614563 2024-03-31 21:58:44 INFO [auto_gptq.quantization.gptq] avg loss: 28.414779663085938 INFO - Quantizing ffn.experts.mlp.11.w2 in layer 40/40... 2024-03-31 21:58:47 INFO [auto_gptq.quantization.gptq] duration: 3.0984387397766113 2024-03-31 21:58:48 INFO [auto_gptq.quantization.gptq] avg loss: 47.26870346069336 INFO - Quantizing ffn.experts.mlp.12.w2 in layer 40/40... 2024-03-31 21:58:51 INFO [auto_gptq.quantization.gptq] duration: 3.0781986713409424 2024-03-31 21:58:51 INFO [auto_gptq.quantization.gptq] avg loss: 21.394535064697266 INFO - Quantizing ffn.experts.mlp.13.w2 in layer 40/40... 2024-03-31 21:58:54 INFO [auto_gptq.quantization.gptq] duration: 3.0461459159851074 2024-03-31 21:58:54 INFO [auto_gptq.quantization.gptq] avg loss: 76.7130126953125 INFO - Quantizing ffn.experts.mlp.14.w2 in layer 40/40... 2024-03-31 21:58:57 INFO [auto_gptq.quantization.gptq] duration: 3.075993299484253 2024-03-31 21:58:57 INFO [auto_gptq.quantization.gptq] avg loss: 26.178302764892578 INFO - Quantizing ffn.experts.mlp.15.w2 in layer 40/40... 2024-03-31 21:59:00 INFO [auto_gptq.quantization.gptq] duration: 3.0906994342803955 2024-03-31 21:59:00 INFO [auto_gptq.quantization.gptq] avg loss: 19.860462188720703

Yes. The increase of loss in as layers increase is also observed in my end. For now, we don't have much to go on. Calibration may need to be tweaked to optimize for dbrx. Right now generation is different than bfloat16 in my limited testing but still coherent. However, there may be EOS problem where it is not stopping. These are minor issues as more users start to quant and find the quirks.

We need to add a feature to save_quantized into multiple files with max_split_size  and load multiple files with a json map  just like normal from_pretrained api.

@LaaZa @fahadh4ilyas  @Xu-Chen If you have the vram, please test the 4bit marlin inference with https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin and let me if you are getting coherent responses. Note the loading time is quite long.

I think the calibration code is broken for MOE style dbrx. Based on the massive escalation of errors of error loss on as we progress through the layers 1-40.
INFO - Quantizing ffn.experts.mlp.0.v1 in layer 11/40...
2024-03-31 15:11:09 INFO [auto_gptq.quantization.gptq] duration: 1.6267049312591553
2024-03-31 15:11:09 INFO [auto_gptq.quantization.gptq] avg loss: 28.512256622314453
INFO - Quantizing ffn.experts.mlp.1.w1 in layer 11/40...
2024-03-31 15:11:10 INFO [auto_gptq.quantization.gptq] duration: 1.6261475086212158
2024-03-31 15:11:10 INFO [auto_gptq.quantization.gptq] avg loss: 17.44123077392578
INFO - Quantizing ffn.experts.mlp.1.v1 in layer 11/40...
2024-03-31 15:11:12 INFO [auto_gptq.quantization.gptq] duration: 1.628706932067871
2024-03-31 15:11:12 INFO [auto_gptq.quantization.gptq] avg loss: 15.289453506469727
INFO - Quantizing ffn.experts.mlp.2.w1 in layer 11/40...
2024-03-31 15:11:13 INFO [auto_gptq.quantization.gptq] duration: 1.6164577007293701
2024-03-31 15:11:13 INFO [auto_gptq.quantization.gptq] avg loss: 18.87800407409668

INFO - Quantizing ffn.experts.mlp.14.w1 in layer 23/40...
2024-03-31 15:42:21 INFO [auto_gptq.quantization.gptq] duration: 1.6099011898040771
2024-03-31 15:42:21 INFO [auto_gptq.quantization.gptq] avg loss: 52.00520324707031
INFO - Quantizing ffn.experts.mlp.14.v1 in layer 23/40...
2024-03-31 15:42:22 INFO [auto_gptq.quantization.gptq] duration: 1.6029260158538818
2024-03-31 15:42:22 INFO [auto_gptq.quantization.gptq] avg loss: 45.09771728515625
INFO - Quantizing ffn.experts.mlp.15.w1 in layer 23/40...
2024-03-31 15:42:24 INFO [auto_gptq.quantization.gptq] duration: 1.611437320709228

2024-03-31 16:23:09 INFO [auto_gptq.quantization.gptq] duration: 2.92622447013855
2024-03-31 16:23:09 INFO [auto_gptq.quantization.gptq] avg loss:<PHONE_NUMBER>8515625
INFO - Quantizing ffn.experts.mlp.2.w2 in layer 39/40...
2024-03-31 16:23:12 INFO [auto_gptq.quantization.gptq] duration: 2.8120198249816895
2024-03-31 16:23:12 INFO [auto_gptq.quantization.gptq] avg loss:<PHONE_NUMBER>7148438
INFO - Quantizing ffn.experts.mlp.3.w2 in layer 39/40...
2024-03-31 16:23:14 INFO [auto_gptq.quantization.gptq] duration: 2.823086977005005
2024-03-31 16:23:14 INFO [auto_gptq.quantization.gptq] avg loss:<PHONE_NUMBER>5527344
INFO - Quantizing ffn.experts.mlp.4.w2 in layer 39/40...
2024-03-31 16:23:17 INFO [auto_gptq.quantization.gptq] duration: 2.816202163696289
2024-03-31 16:23:17 INFO [auto_gptq.quantization.gptq] avg loss:<PHONE_NUMBER>6542969
INFO - Quantizing ffn.experts.mlp.5.w2 in layer 39/40...
2024-03-31 16:23:20 INFO [auto_gptq.quantization.gptq] duration: 2.8179235458374023
2024-03-31 16:23:20 INFO [auto_gptq.quantization.gptq] avg loss:<PHONE_NUMBER>6992188

@Qubitium
If we use more samples for quantization, the final loss will decrease.
My current test script:
import os
max_cpu_threads = "24"
os.environ["OMP_NUM_THREADS"] = max_cpu_threads
os.environ["OPENBLAS_NUM_THREADS"] = max_cpu_threads
os.environ["MKL_NUM_THREADS"] = max_cpu_threads
os.environ["VECLIB_MAXIMUM_THREADS"] = max_cpu_threads
os.environ["NUMEXPR_NUM_THREADS"] = max_cpu_threads
os.environ["NUMEXPR_MAX_THREADS"] = max_cpu_threads

import logging

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S",
)

import argparse, gc, shutil
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from datasets import load_dataset

parser = argparse.ArgumentParser()
parser.add_argument("--model-id", type=str) # input model path
parser.add_argument("--gptq-save-dir", type=str)
parser.add_argument("--channelwise", action="store_true")
parser.add_argument("--num-samples", type=int, default=1024) # The more samples, the slower the quantization speed
parser.add_argument("--max-seq-len", type=int, default=32768)

def preprocess(example):
        return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}

if __name__ == "__main__":
    args = parser.parse_args()

    dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:5%]")
    tokenizer = AutoTokenizer.from_pretrained(args.model_id, trust_remote_code=True)
    ds = dataset.shuffle().select(range(args.num_samples))
    ds = ds.map(preprocess)

    examples = [
        tokenizer(
            example["text"], padding=False, max_length=args.max_seq_len, truncation=True,
        ) for example in ds
    ]

    if args.channelwise:
        group_size = -1
    else:
        group_size = 128

    quantize_config = BaseQuantizeConfig(
        bits=4,                         # Only support 4 bit
        group_size=group_size,          # Set to g=128 or -1 (for channelwise)
        desc_act=False,                 # Marlin does not suport act_order=True
        model_file_base_name="model",    # Name of the model.safetensors when we call save_pretrained
        quant_method="gptq",
        checkpoint_format="gptq"
    )

    model = AutoGPTQForCausalLM.from_pretrained(
        args.model_id,
        quantize_config,
        device_map="auto",
        trust_remote_code=True)
    model.quantize(examples)

    print(f"Saving gptq model to {args.gptq_save_dir}")
    model.save_pretrained(args.gptq_save_dir)
    tokenizer.save_pretrained(args.gptq_save_dir,  max_split_size="5GB")

@Qubitium If we use more samples for quantization, the final loss will decrease.

Nice! Can you post some of the layers losses from beginning/middle/end? To confirm, you made two changes:

max_sample from 128 -> 1024
max seq len from 2048 to 32768

Are you using the above two values?

@LaaZa

We need to add a feature to save_quantized into multiple files with max_split_size and load multiple files with a json map just like normal from_pretrained api.

@LaaZa Already made this shard loading code. I just need to shared on save.

We need to add a feature to save_quantized into multiple files with max_split_size and load multiple files with a json map just like normal from_pretrained api.

@LaaZa Already made this shard loading code. I just need to shared on save.

We need to reimplement #364 saving works but it is obviously outdated and there was a weird issue where sharded saving with it breaks fused.

@Qubitium If we use more samples for quantization, the final loss will decrease.

Nice! Can you post some of the layers losses from beginning/middle/end? To confirm, you made two changes:

max_sample from 128 -> 1024
max seq len from 2048 -> 32768

Are you using the above two values?

Yes，and I used samples from HuggingFaceH4/ultrachat_200k and applied apply_chat_template，while the model is dbrx-instruct-converted-v2 (use convert_v2.py to convert dbrx-instruct as dbrx-instruct-converted-v2).
The log has been overwritten by new output, but the final avg loss is less than 2.

@Xu-Chen To anyone trying to quant, do not set more than 8 threads. I find negative returns. Warn/FIX in PR #628

Two new quants have started based on calibration fix from @Xu-Chen . Marlin + Non-Marlin

https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin-v2 (PENDING)
https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-gptq-v2 (PENDING)

Two new quants have started based on calibration fix from @Xu-Chen . Marlin + Non-Marlin

https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin-v2
https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-gptq-v2 (PENDING)

The next step should be to deploy the model using vllm

The next step should be to deploy the model using vllm

Our vllm PR for dbrx-converted-v2 should be ready soon.

Our vllm PR for dbrx-converted-v2 should be ready soon.

Unfortunately dbrx finetuning has failed my internal quality metrics so I will not be spending time to port dbrx-converted to vllm. But if you plan to do so there is two paths: 1) reverse the convert_v2.py split 2) use the new inf code but load the weights 1/tp_size  split and keep the tp_rank slice.

Sorry for the delay, I have been busy.

Why is packing so slow? it's lke 100x slower than the quantizing stage. We are so close!

I never had a look at the implementation, it is probably suboptimal. Either I need to implement a CUDA kernel for packing, or there's a better implementation possible in PyTorch. Let me have a look next week.

Dbrx proper support is finally merged to transformers:
https://github.com/huggingface/transformers/pull/29921
The whole quant code path needs to be visited again. Maybe they fixed the two issues that plagued the pre-transformer merge: 1)  training is not possible due to oom of fused layers 2) quant is not possible/feasible due the fused layers.
I no longer have the time to work on this. If you want to take over this WIP PR, you can fork it or let me know and I will add your to the branch permissions.

Dbrx proper support is finally merged to transformers:
https://github.com/huggingface/transformers/pull/29921
The whole quant code path needs to be visited again. Maybe they fixed the two issues that plagued the pre-transformer merge: 1)  training is not possible due to oom of fused layers 2) quant is not possible/feasible due the fused layers.
I no longer have the time to work on this. If you want to take over this WIP PR, you can fork it or let me know and I will add your to the branch permissions.

Well, #623 is meant to support the unmodified model if possible. I was waiting for resolution on the transformers implementation to see if it will be possible then. Do you think the modified version of the model is still needed?

Tests at GPTQModel showed that the non-defused transformer model (even though merged into HF transformers) is un-quantizable with valid PPL and avg losses. GPTQModel decided to merge DBRX Converted (de-fused non-official model) support instead. (Code is ported from this PR) Since this is not official model structure, I doubt it will be valid to get it merged here.
For those following the PR, use GPTQModel and the ModelCloud hosted converted DBRX Converted V2 models (base + instruct) for quantization.


[Docs] Typo issue in meshery Rest API

Current State
Word "specific" repeat.

Desired State
"Handle GET request for getting meshmodel components of a specific model."

Contributor Guides and Handbook

📚 Instructions for contributing to documentation

Meshery documentation site and source

🛠 Meshery Build & Release Strategy
🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

I would like to work on this issue. Can you please walk me through the process

@senali-d Shouldn't it be specific model rather than just specific

@UsmanDev09 Yeah it should be. Thank you

Raised PR - https://github.com/meshery/meshery/pull/7628

This has been merged https://github.com/meshery/meshery/pull/7628

done by #7628


Version 15.1 FancyZones fails

Environment
Windows build number: [run "ver" at a command prompt]
PowerToys version:
PowerToy module for which you are reporting the bug (if applicable):

Steps to reproduce

Expected behavior

Actual behavior

Screenshots

Can you see if the things discussed in #1451 fixes your issue.

used Microsoft's repair tool for installs and uninstalls, and some state is still being kept somewhere

From the description you made in the other issue, I suspect this has restored what you first deleted.
There is no need to do any extra steps beside the ones described in https://github.com/microsoft/PowerToys/issues/1451
There are only two places where PT save the settings, if they are deleted while PowerToys is not running, they are gone forever.

We have a fix for this.

Fix availalbe in v0.15.2


FPS capped at primary display's refresh rate

A quick search and I haven't seen this issue mentioned. Maybe it is a bit niche, but my primary display is a 75hz monitor, and my secondary display is a 120hz TV. The FPS unlock works for up to 75 fps unless I make the TV my primary display.

I could imagine that is due to some stuff happening with VSync which is probably confusing things. try turning off VSync in the game's settings and if that doesn't change anything, try turning it off for the game in the nvidia control center


YCSB benchmark - always failing with RemoteCommandError

We are trying to run the YCSB benchmark for Spanner, but after 2hrs, its failing, We ran the test so many times. But after 2hr it'll throw the RemoteCommandError.
But sometimes it worked without any errors.
Error:
2020-12-09 13:34:09,799 perf64kb MainThread cloud_spanner_ycsb(1/1) pkb.py:1079 ERROR    Exception running benchmark
Traceback (most recent call last):
  File "/root/spanner_pkb_fork/perfkitbenchmarker/pkb.py", line 1077, in RunBenchmarkTask
    RunBenchmark(spec, collector)
  File "/root/spanner_pkb_fork/perfkitbenchmarker/pkb.py", line 929, in RunBenchmark
    DoRunPhase(spec, collector, detailed_timer)
  File "/root/spanner_pkb_fork/perfkitbenchmarker/pkb.py", line 763, in DoRunPhase
    samples = spec.BenchmarkRun(spec)
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_benchmarks/cloud_spanner_ycsb_benchmark.py", line 164, in Run
    samples = list(benchmark_spec.executor.LoadAndRun(
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_packages/ycsb.py", line 1397, in LoadAndRun
    run_samples += self.Run(vms, workloads=workloads,
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_packages/ycsb.py", line 1362, in Run
    samples = list(self.RunStaircaseLoads(vms, workloads,
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_packages/ycsb.py", line 1215, in RunStaircaseLoads
    vm_util.RunThreaded(PushWorkload, [((vm, workload_file, remote_path), {})
  File "/root/spanner_pkb_fork/perfkitbenchmarker/background_tasks.py", line 650, in RunThreaded
    return RunParallelThreads(target_arg_tuples,
  File "/root/spanner_pkb_fork/perfkitbenchmarker/background_tasks.py", line 585, in RunParallelThreads
    return _RunParallelTasks(
  File "/root/spanner_pkb_fork/perfkitbenchmarker/background_tasks.py", line 559, in _RunParallelTasks
    raise parallel_exception_class(
perfkitbenchmarker.errors.VmUtil.ThreadException: The following exceptions occurred during parallel execution:
Exception occurred while calling PushWorkload(<IP_ADDRESS>, /root/spanner_pkb_fork/perfkitbenchmarker/data/ycsb/workloadq, /opt/pkb/workloadq):
Traceback (most recent call last):
  File "/root/spanner_pkb_fork/perfkitbenchmarker/background_tasks.py", line 255, in Run
    self.return_value = self.target(*self.args, **self.kwargs)
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_packages/ycsb.py", line 1213, in PushWorkload
    vm.RemoteCommand('sudo rm -f ' + remote_path)
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_virtual_machine.py", line 798, in RemoteCommand
    return self.RemoteCommandWithReturnCode(*args, **kwargs)[:2]
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_virtual_machine.py", line 814, in RemoteCommandWithReturnCode
    return self.RemoteHostCommandWithReturnCode(*args, **kwargs)
  File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_virtual_machine.py", line 885, in RemoteHostCommandWithReturnCode
    raise errors.VirtualMachine.RemoteCommandError(error_text)
perfkitbenchmarker.errors.VirtualMachine.RemoteCommandError: Got non-zero return code (-13) executing sudo rm -f /opt/pkb/workloadq
Full command: ssh -A -p 22<EMAIL_ADDRESS>-2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/perf64kb/perfkitbenchmarker_keyfile -o ControlPath="/tmp/perfkitbenchmarker/runs/perf64kb/ssh/%h" -o ControlMaster=auto -o ControlPersist=30m sudo rm -f /opt/pkb/workloadq
STDOUT: STDERR:

Hi, can you provide the run command you are using and the PKB version? Thanks.

Version: v1.15.1
Run command:
./pkb.py --benchmark_config_file=/root/custom.yaml benchmarks=cloud_spanner_ycsb --zone=us-west4-a --run_uri=perf256kb --bigquery_table=pkb.pkb_result

Yaml File:
benchmarks:
- cloud_spanner_ycsb:
    flags:
      cloud_spanner_config: regional-us-west4
      machine_type: n2-highcpu-2
      cloud_spanner_nodes: 3
      ycsb_between_run_sleep_time: 1800
      ycsb_client_vms: 30
      ycsb_field_count: 1
      ycsb_field_length: 262144
      ycsb_measurement_type: hdrhistogram
      ycsb_operation_count: 100000000
      ycsb_preload_threads: '320'
      ycsb_record_count: 10000000
      ycsb_run_parameters: target=20000,requestdistribution=zipfian,dataintegrity=True
      cloud_spanner_ycsb_readmode: query
      ycsb_run_stage_iterations: 5
      ycsb_sleep_after_load_in_sec: 1800
      ycsb_tar_url: https://storage.googleapis.com/externally_shared_files/ycsb-0.18.0-SNAPSHOT.tar.gz
      ycsb_threads_per_client: '60'
      ycsb_timelimit: 1800
      ycsb_version: 0.18.0-SNAPSHOT
      ycsb_workload_files: workloadq
      zones: us-west4-a

A few suggestions here:

Does ssh into the remote worker and sudo rm -f /opt/pkb/workloadq work? It seems like there may be an issue with the permissions here.
Currently the PKB default is YCSB version 0.17.0, which is currently the latest release on GitHub. Does your configuration work with this version?
ycsb_workload_files: workloadq can you try with one of the PKB-provided workload files?
Does this work on a smaller scale experiment with the same number of client-vms?

Seems like we are trying to delete a non-existing workload file.
Can you try making following edit:
File "/root/spanner_pkb_fork/perfkitbenchmarker/linux_packages/ycsb.py", line 1213, in PushWorkload
vm.RemoteCommand('sudo rm -f ' + remote_path**, ignore_failure=True**)

Sure, I'll change the line.
But I checked it, the file was there,
And the strange behavior is, its executed successfully for some time, and in some cases like 2 or 3 iterations were successful out of 5.


Fully initialize the epoll_data structure.

While running a simple libdispatch-using application on Linux under Valgrind, I noticed that Valgrind complained that the struct epoll_event initialized in this function was not fully initialized:
==3103== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==3103==    at 0x5DBE63A: epoll_ctl (syscall-template.S:81)
==3103==    by 0x4075493: _dispatch_timeout_program (in /usr/lib/swift/linux/libdispatch.so)
==3103==    by 0x407534F: _dispatch_event_loop_timer_arm (in /usr/lib/swift/linux/libdispatch.so)
==3103==    by 0x4072391: _dispatch_timers_program (in /usr/lib/swift/linux/libdispatch.so)
==3103==    by 0x40712D9: _dispatch_mgr_invoke (in /usr/lib/swift/linux/libdispatch.so)
==3103==    by 0x407118D: _dispatch_mgr_thread (in /usr/lib/swift/linux/libdispatch.so)

While this is unlikely to cause any bugs today (none of those fields are accessed anywhere else), we can save ourselves the risk of future pain by initialising those portions of the structure appropriately before we pass it to the kernel.

You're right that the fix is wrong, but wrong about why. I went back and checked the man page for epoll and noticed that epoll_data is a union, not a struct (my mistake), and that union type contains 64-bit members. The compiler output for _dispatch_timeout_program initialises .data like this:
   4f3d7:       4c 8d 3d f2 58 04 00    lea    0x458f2(%rip),%r15        # 94cd0 <_dispatch_epoll_timeout>
   4f3de:       43 0f b7 44 e7 04       movzwl 0x4(%r15,%r12,8),%eax
   4f3e4:       89 44 24 04             mov    %eax,0x4(%rsp)

Note that the initialisation is done with movzwl, which initialises only first 4 bytes of the 8 byte union. Nowhere else zero-initialises the epoll_event structure, so the trailing 4 bytes of the structure are in fact uninitialized.
As noted before, this is probably not an issue, but it may be better to split the initialisation up. If we changed the code to look like this:
struct epoll_event ev = {
    .events = EPOLLONESHOT | EPOLLIN
};
ev.data.u32 = timer->det_ident;

This should correctly initialise all bytes of the union, and silence this warning. I'll double-check this by recompiling Swift and confirming that this fixes up the assembly, and that this no longer annoys valgrind, and then open a new PR.

Ok, this makes more sense, I like the code the way you just wrote it. I reopened the pull request, please make this patch instead.

actually there's even simpler, which is to use u64 instead in the dispatch codebase, given that it's the larger member of the union, it'll DTRT.

Yup, that fix works. I'll open a new PR.

you can reuse this one and force push it should work. (I think).


docs app style updates

This updates the docs app to the new header style found on angularjs.org.
I've also taken the liberty to update the menu entries on the doc site - pending approval I'll update the org site, too (would fix https://github.com/angular/angular.js/issues/14351)
Notable differences between org / docs:

search icon
font family / weight (should we update?)
no drop shadow on navbar because there's a secondary bar on docs

Notable differences between docs old / new:

mobile: the top header + search are fixed, sub header is relative (previously all relative)
mobile search close icon has moved over the search bar, because I had some problems giving it it's previous position while keeping the search results scrollable (oh my)

Implementation notes:

angular-topnav.css is included as is from org site
several overrides necessary because org uses bootstrap 2 / docs uses bs 3 (oh my)
org site uses one breakpoint at 800px, docs uses one at 768 and one at 992. I tested it and it doesn't affect the common screen sizes

Future work:

clean up the css and maybe implement sass for the docs app

Screens:

Tests have been fixed


[adsi] type accelerator is unavailable / throws an exception

Steps to reproduce
[adsi]

Expected behavior
The code should return a System.RuntimeType representing the System.DirectoryServices.DirectoryEntry class, similar to the full version of PowerShell.
PS D:\Users\tsulli\sshscan> [adsi].GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
False    True     RuntimeType                              System.Reflection.TypeInfo

Actual behavior
Throws an exception.
PS /Users/tsulli> [adsi]
Unable to find type [adsi].
At line:1 char:1
+ [adsi]
+ ~~~~~~
    + CategoryInfo          : InvalidOperation: (adsi:TypeName) [], RuntimeException
    + FullyQualifiedErrorId : TypeNotFound

Environment data

Name                           Value
----                           -----
PSVersion                      6.0.0-alpha
PSEdition                      Core
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   <IP_ADDRESS>
GitCommitId                    v6.0.0-alpha.15
CLRVersion
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>

Cheers,
Trevor Sullivan
Docker Captain
Microsoft MVP: Cloud & Data Center Management
https://trevorsullivan.net
https://twitter.com/pcgeek86

Following types not in CoreCLR:

adsi
adsisearcher
wni
wmiclass
wmisearcher
mailaddress

@SteveL-MSFT Do we have any idea when the active directory types are going to be added to CoreClr? You can't add the accelerators until the underlying types are there so marking this "up for grabs" is kind of pointless.

@BrucePay no idea, not a focus for PS Core 6.0.0

They've just added to CoreFX without tests. The work has high priority for .NET Standard 2.0
https://github.com/dotnet/corefx/issues/2089#issuecomment-261681168

CoreFX change milestone to 2.1.0. System.DirectoryServices is still unstable.

@SteveL-MSFT @daxian-dbw I believe it is important features for Enterprises and we should enable its.
Currently there is no progress in CoreFX with System.DirectoryServices. So  I think we should enable it for Windows only.
It seems we cannot directly reference in GAC System.DirectoryServices .
Is there a way to solve this problem?

I think we should defer this to 6.1.0 given we should depend on corefx.  Perhaps we should suppress the type accelerator for now.

In CoreFX repo comments say that it depends very much on COM, also, there is no port LDAP library yet. I believe that this will not be done much longer. That's why I'm requesting to enable it on Windows based on GAC dlls.

I'm not sure if we want to set a precedent on creating a dependency on the GAC from PSCore.  cc @daxian-dbw

It seems DirectoryServices code is available in corefx: https://github.com/dotnet/corefx/tree/master/src/System.DirectoryServices/src/System/DirectoryServices
and the preview version nuget package for System.DirectoryServices is also available on dotnet-core feed (https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.DirectoryServices).
Can't we wait for that to mature and depend on that instead?

I suggest we wait until 6.1.0 timeframe to tackle this

I agree that native support would be best. But...

My scripts is 80% Active Directory related - so I cannot use 6.0 in Enterprise at all. All the more ActiveDirectory module is not supported. ETA? No one can use PowerShell Core 6.0 version in Enterprises.
DirectoryServices nuget package is not updated for about a last month. Works frozen.
Little progress is in last year.
It is not stable and haven't tests.
It is not ported. No plans (possibilities) is to port all APIs (depends on COM and LDAP native Windows dlls).
No full time MSFT employers works on the DirectoryServices.

@iSazonov would you be ok with a workaround (in your scripts or profile) until we can do this properly in 6.1.0?
Add-Type -AssemblyName $env:windir\microsoft.net\framework64\v4.0.30319\system.direc
toryservices.dll
[psobject].Assembly.GetType("System.Management.Automation.TypeAccelerators")::add("a
dsi","System.DirectoryServices.DirectoryEntry")
[adsi]"WinNT://${env:computername}/${env:username}"

I thought about this and about our compatibility module.
My concerns:

I don't believe that the module will be ready in RTM time.
I see we have codes for DirectoryEntry (ExtendedAdapter and etc.) - it could be the main breaking change.

Suggestion is - create temporary nuget package to expose the GAC dll until CoreFX issue a ported version.

@PowerShell/powershell-committee reviewed this and agreed we should not take explicitly dependency on assemblies in the GAC.  However, to resolve this class of issues, we agreed that we should have a module like "WindowsPSModulePath" to create aliases that have been excluded from PSCore6.  Steve to work on that module.  Leaving this issue until DirectoryEntry shows up in corefx.

Unfortunately, it appears my workaround isn't complete so I can't actually enable this in my module.  Currently, we explicitly #ifdef out the type convertors for ADSI (and WMI) as the type doesn't exist in corefx.  Without the type convertors, it's only a partial solution to manually add the type accelerator.

It would be useful to have a "delay resolved" type accelerator - that would help startup performance a bit by not eagerly loading all the assemblies referenced by accelerators, and as a bonus, you could add the adsi accelerator in the core.
That said, I never implemented this idea because I was pretty sure some other code running during startup would just load the assemblies I was trying to defer.


Improve grep

Right now the --grep mechanism only serves as a filename filter. It might make sense to add a different option for file names and allow --grep to match on the contents of test descriptions.
I would propose the following:

--grep passes a regular expression to be evaluated against it descriptions
add a --test-suffix option similar to PHPUnit for filtering on file names

@austinsmorris can you review this and let me know what you think?


question  anbout the sensor's xml data

thank for your ppg data and videos, it is very helpful.  Is the heart value of the xml document in the ..hr value... fileld?

in the cms50 xml file value0 is the frame counter, value1 is the average heart rate computed on the device and value2 is the ppg time signal measured with the device.


Update bitnami/postgresql chart version

Fixes (#48)

@m-bull
I checked the Postgres versions, and I’m pretty sure we can go up to 11.9.13 without any compatibility issues. Would isolate us from the same problem for longer…

@m-bull
I checked the Postgres versions, and I’m pretty sure we can go up to 11.9.13 without any compatibility issues. Would isolate us from the same problem for longer…

Updated


Licensing

Hi, there,
I noticed that this code is licensed as No license, which means that anyone who is using it without your permission is potentially violating copyright. Would you consider changing this to use CC0 (public-domain-ish) licensing? To do so, you would simply need to change the contents of your LICENSE file to be:
visibility by Daniel Henry-Mantilla is marked with CC0 1.0 Universal. To view a copy of this license, visit https://creativecommons.org/publicdomain/zero/1.0/

Huh, I always use Zlib OR MIT OR Apache-2.0 licensing for my crates; I guess the dirty "initial commit" state never fixed the No license situation.
Fixed as of v0.1.1 🙂

Thank you!


How can we implement global exception handling?

The section "Places where errors may occur" shows all the places where an exception might be thrown, but the recommended resolution is wrapping the code with local try-catch statements. While we can add local try-catch to all the places, we might miss a few places and an unhandled exception might freeze the application.
Is there a way to implement a global exception handling, so we can catch any unhandled exception, like the ASP.NET custom errors? It would be useful to have a global exception handling so we can cover any unexpected exception while delivering a user-friendly page.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ce1440e7-febc-3fbc-09d7-7f47b740fd3b
Version Independent ID: 31071595-7c71-b6f7-3856-74f78c5c7cdf
Content: Handle errors in ASP.NET Core Blazor apps
Content Source: aspnetcore/blazor/handle-errors.md
Product: aspnet-core
Technology: aspnetcore-blazor
GitHub Login: @guardrex
Microsoft Alias: riande

@aueda please see the discussion here: https://github.com/aspnet/AspNetCore/issues/13452


Provide the way to store DATETIME in specific timezone

Is your feature request related to a problem? Please describe.
Now when converting Date to sql type local timezone is used. It would be nice to have a settings to force specific timezone like UTC.
Without this settings i will get different results if BE in another timezone.

Datetime with TZ is always stored in UTC, or not? I thought the whole point of timezones is that you normalize for saving and apply timezone when reading.

Datetime with TZ is always stored in UTC, or not?

Javascript Date type always has timezone information, but sql Datetime no. MySQL retrieves and displays DATETIME values in 'YYYY-MM-DD hh:mm:ss' format without timezone information.
MikroORM now converts Date to  'YYYY-MM-DD hh:mm:ss' representation using NodeJS timezone. It isn't bug, but the most of people will prefer to see UTC in database despite the fact of NodeJS environment settings.

TypeORM has settings to force timezone that will be used to covert 'YYYY-MM-DD hh:mm:ss' values to js Date.

For TIMESTAMP type handling is more complex. MySQL as i know always stores it in UTC, but communication is performed in some tz specific for connection.

I would expect this to be solved simply by providing the timezone you want to be saved in driver options. Looks like mysql2 has support for that (https://github.com/sidorares/node-mysql2/pull/996), I would expect other drivers to have similar configuration. Will take a look into this later.

@B4nan
Is it possible now to override some driver-level settings?

Currently not.

You should always store day/time in a database in UTC. Let the client convert it to the local timezone when displaying it to an end user.

Another note for discussion. JS dates are always UTC timestamps (represented based on current user's timezone), so it should be also stored in UTC by default in the database.
I think what's missing in the ORM is support for setting current timezone, not for writing, but for reading. That is what I was doing when working with other ORMs (mainly doctrine), simple SET time_zone = 'Europe/Prague' (for mysql) when initializing the connection.

How do you know what TZ is correct though? That would assume all the 'users' are in that TZ and want to view things under that TZ. Again, far better to make the TZ a client preference. If you don't have anyone viewing the TZ, then just keep it at UTC... this also avoids pesky daylight savings issues. Convert to local TZ at the very last step.
I worked with a system a long time ago that was built around the concept of cron jobs running at a certain time in the clients timezone, based on a database entry. The guy who designed it thought he was smart to offer this 'feature' to the customers... until they grew in size and it blew up and took down the entire system. It was a nightmare because every time the hour changed for daylight savings, cron jobs would either run twice or not run at all.
Wrote this 9 years ago... http://lookfirst.com/2010/05/how-to-solve-daylight-savings-time.html

How do you know what TZ is correct though?

Right, that wouldn't solve much when you expect users from different timezones. What we can agree on is that datetimes should be stored in UTC.
@vimmerru If I understand it correctly, mysql will save the datetime in current timezone instead? So that mysql2 option I was mentioning would fix this at least?
@lookfirst What would be your suggestion here? I understand that in the end, if you wanna have different timezones based on user's TZ, you will need to handle this manually. But still, the database value needs to be in UTC.
Will need to try things myself to wrap my head around this...

@vimmerru If I understand it correctly, mysql will save the datetime in current timezone instead? So that mysql2 option I was mentioning would fix this at least?

Yes, mysql2 option should help

My suggestion is to store everything UTC and set the database/OS to UTC. If you need local time, convert it at the client where it is used. Don't build anything that depends on the server returning a time in a TZ.

Thanks! I will try this soon. For now we are forcing node to UTC timezone through environment, but it doesn't work for Windows platform.

I haven't test, but MariaDB driver seems use a similar approach to MySQL2 driver:
https://github.com/MariaDB/mariadb-connector-nodejs/blob/master/documentation/promise-api.md#timezone-consideration

"It's not recommended"

In 3.0.0-rc.2 there is now forceUtcTimezone option to force the datetimes to be saved in UTC in datetime columns without timezone. It works for mysql (datetime type) and postgres (timestamp type). Sqlite does this by default.
The mariadb driver is unfortunately buggy there, it throws and error when the timezone config option is provided (at runtime, the value passes their internal validation).


Public models migrated

as described in #42

[x] Export/Import script
[x] Validate models in test

"As a customer using existing public models, I know that they are available in the new DMR."

https://github.com/iotmodels/iot-plugandplay-models-test/tree/migration


Create seed file and migration script to populate skill_level table

While continuing to flesh out the entities/repositories and their respective API routes, I realized that the skill_level table is not a table that will be modified by users, so probably best to seed this table now so that data is available to relate to the player table.

Can you add a sample ormconfig.json like we did with the .env.default? It's only used for running the migration script with the TypeORM CLI, but it would be good to have for potential contributors. Also, can you add code to the down() method in the migration script, just in case we need to revert seeding that table?


fix bug in skip_initial_status

I have found a bug in in the merged PR that introduced the skip_initial_status feature. This bug unjustly leaves conditions stagnant when this feature is enabled in a custom plugin configuration. The root cause is that the conditions for the plugin aren't properly initialized. The PR implies that when skip_initial_status is set, the initialConditions function is called. As a result, custom plugins with skip_initial_status work correctly.

/ok-to-test

/lgtm
/assign @vteratipally

/assign @sjenning

ping @sjenning for approval

/retest

all tests successfully passed
@sjenning please review

/cc @wangzhen127

/lgtm
/approve

amazing! tested this and it looks good 👍🏼


ci: Tidy up lint workflow

What problem does this PR solve?:
Noticed some problems in linting in #938 and realised it was a problem with the workflows rather than the PR.
Which issue(s) does this PR fix?:

https://jira.d2iq.com/browse/D2IQ-NUMBER

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Thank you @jimmidyson for fixing this.


Installation Issue.

Hi All,
Need help to install QrCode under Win 10/Apache/PHP. The following is the method:

Download QrCode-master.zip
Unzip and saved into  C:\Apache24\htdocs\QrCode
From Command prompt, I received following error message :

C:\Apache24\htdocs\QrCode>composer require endroid/qrcode
Using version ^2.3 for endroid/qrcode
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- The requested package endroid/qrcode No version set (parsed as 1.0.0) is satisfiable by endroid/qrcode[No version set (parsed as 1.0.0)] but these conflict with your requirements or minimum-stability.
Installation failed, reverting ./composer.json to its original content.

When I run my sample code from browser and facing following error :
setText('http://ourcodeworld.com')
    ->setSize(300)
    ->setPadding(10)
    ->setErrorCorrection('high')
    ->setForegroundColor(array('r' => 0, 'g' => 0, 'b' => 0, 'a' => 0))
    ->setBackgroundColor(array('r' => 255, 'g' => 255, 'b' => 255, 'a' => 0))
    // Path to your logo with transparency
    ->setLogo("logo.png")
    // Set the size of your logo, default is 48
    ->setLogoSize(98)
    ->setImageType(QrCode::IMAGE_TYPE_PNG)
;

// Send output of the QRCode directly
header('Content-Type: '.$qrCode->getContentType());
$qrCode->render();

?>
Error
( ! ) Fatal error: Class 'Endroid\QrCode\QrCode' not found in C:\Apache24\htdocs\QrCode\getCode.php on line 5

Hi @chchar74 it looks like you did not include the autoloader created by Composer. See https://getcomposer.org/doc/01-basic-usage.md#autoloading for more information on how to do this.


Rewrite the gesture engine

Today, KeyboardKit applies a set of gestures on each key in the keyboard, to make it possible to handle press, tap, release, long press etc.
Although this gesture setup is simple and convenient, it comes with a few major drawbacks:

Having a gesture on each key makes typing a bit clunky and not as smooth as in the stock keyboards.
It's not possible to press down on a key, swipe to another key and release to insert that key.
It's not possible to implement swipe to type, since you can't swipe between keys.
It's not possible to implement the way iOS keyboards adapt the tap area size of keys based on probability.

I will therefore try to rewrite the gesture engine so that the entire keyboard has a single set of gestures, which it will apply to the underlying keys based on their positions and sizes.
This will require some more capabilities to be added to the layout engine, so that the layout can return an action (key) for a given coordinate.

This is good news, good luck!

I realized that I will probably have to add other things to the library before this, so it will be in a later minor version instead of the next one.

This is implemented in 6.6.
You can try it out in the non-pro demos and master branch, by toggling on the feature like this:
FeatureToggle.shared.toggleFeature(.newButtonGestureEngine, isOn: true)


Can the libgit2 dependency be reverted to a released version before 1.0?

Specifically, I'm asking for https://github.com/alexcrichton/git2-rs/commit/a20191dde5d482debb90b5913f03a58ca73720e2 to be backed out, and the bug to be resolved in some other way. The latest released libgit2 is v0.21.3, which doesn't include the new APIs that are now being relied upon... It doesn't seem right for a 1.0 release to depend on an unreleased library version.

libgit2 just saw a new 0.23.0 release with a newer API/ABI. @alexcrichton, can you please repoint the git2-rs/libgit2-sys/libgit2 submodule to the v0.23.0 upstream tag and add a safety check around pkg-config call to check for "0.23." in the string version?
I think this could be a good initial step to synchronize on 0.23 ABI and later come up with a proper solution to track soname bumps.

Excellent! I'll try to sync to 0.23.0 soon, thanks for the heads up @lucab!

Ok, this now depends on 0.23.0 and I think we'll try to track releases from now on.


Sync habits with Logseq Sync service

Currently habit schedules don't seem to sync across devices using Logseq Sync, are there any plans for implementation?
Thanks

@edhilgendorf I do not have access to logseq sync. Is it still an issue?


Migration problem on multiple databases

Describe the problem
Constance tries to migrate my all databases and fails except default database.  It should work only default database.
== fail message ==
 django.db.utils.ProgrammingError: relation "django_content_type" does not exist LINE 1: ..."."app_label", "django_content_type"."model" FROM "django_co..
Steps to reproduce
you can add a second database for some special models than try to migrate.
System configuration

Django version:
2.2.13
Python version:
3.6
Django-Constance version
2.7.0

Use CONSTANCE_DBS in settings to limit to certain DBs
link to changelong


fix typo

fix typo

Coverage remained the same at 92.928% when pulling 394fb7111ba70fde304bf7be3160d02d71220a4e on dota17:fix_typo into b58168683d01149a568734df21568ffcc41105fe on apache:master.


Verify that the Schedule A-1 forms in RcptCd form_type field really come from Form 460s

Consolidated in #1194


IC3 failed in Windows soot.CompilationDeathException:Couldn't resolve clas spath entry

I want to run IC3 in windows,so I wrote a run.bat like this
run.bat
@echo off
set appPath=%1
set forceAndroidJar=H:\ic3\android.jar
rm -rf testspace
mkdir testspace
set appName=%appPath:~0,-4%
set retargetedPath=testspace%appName%.apk\retargeted\retargeted%appName%
rm -rf output/ic3/$appName.txt
java -Xmx8192m -jar RetargetedApp.jar %forceAndroidJar% %appPath% %retargetedPath%
java -Xmx8192m -jar ic3-0.1.0-full.jar -apkormanifest %appPath% -input %retargetedPath% -classpath %forceAndroidJar% -db cc.properties
rm -rf sootOutput
until the first java statement
java -Xmx8192m -jar RetargetedApp.jar %forceAndroidJar% %appPath% %retargetedPath%
finished,everything is OK
but when the second
java -Xmx8192m -jar ic3-0.1.0-full.jar -apkormanifest %appPath% -input %retargetedPath% -classpath %forceAndroidJar% -db cc.properties
is executed,ERROR occured
H:\ic3>java -Xmx8192m -jar ic3-0.1.0-full.jar -apkormanifest ActivityLifecycle2.
apk -classpath "H:\ic3\android.jar"  -input "testspace\ActivityLifecycle2.apk\re
targeted\retargeted\ActivityLifecycle2"  -db cc.properties
Exception in thread "main" soot.CompilationDeathException: Couldn't resolve clas
spath entry H:\ic3\android.jar:testspace\ActivityLifecycle2.apk\retargeted\retar
geted\ActivityLifecycle2:: java.io.IOException: 文件名、目录名或卷标语法不正确。
    at soot.SourceLocator.explodeClassPath(SourceLocator.java:426)
    at soot.SourceLocator.getClassSource(SourceLocator.java:64)
    at soot.SootResolver.bringToHierarchy(SootResolver.java:200)
    at soot.SootResolver.bringToSignatures(SootResolver.java:252)
    at soot.SootResolver.bringToBodies(SootResolver.java:288)
    at soot.SootResolver.processResolveWorklist(SootResolver.java:164)
    at soot.SootResolver.resolveClass(SootResolver.java:129)
    at soot.Scene.tryLoadClass(Scene.java:629)
    at soot.Scene.loadBasicClasses(Scene.java:1200)
    at soot.Scene.loadNecessaryClasses(Scene.java:1279)
    at edu.psu.cse.siis.ic3.SetupApplication.initializeSoot(SetupApplication

.java:252)
at edu.psu.cse.siis.ic3.SetupApplication.calculateSourcesSinksEntrypoint
s(SetupApplication.java:125)
at edu.psu.cse.siis.ic3.Ic3Analysis.initializeAnalysis(Ic3Analysis.java:
125)
at edu.psu.cse.siis.ic3.Ic3Analysis.initializeAnalysis(Ic3Analysis.java:
64)
at edu.psu.cse.siis.coal.Analysis.performAnalysis(Analysis.java:66)
at edu.psu.cse.siis.ic3.Main.main(Main.java:14)
I think,maybe the reason is format of Windows filepath differences from Linux
Can somebody helps me?
THX!

Hi,
Sorry, but I am very unfamiliar with issues related to Windows. It does look like it could be a problem with file paths.
Regards,
Damien

Change it into Linux.
Actually,I got the same promble like you.it prove IC3 does not work very well in windows.
But it works well in Linux now.


How to configure SSL to https to access 8443

Hi ,
Thank you for the wonderful compose file
How to configure SSL to https to access 8443 I have SSL certs need to configure to access https 8443 port

use the SNI entity in your configuration: https://docs.konghq.com/gateway/latest/admin-api/#sni-object

After test, I think following config needed:
- KONG_PROXY_LISTEN: "${KONG_PROXY_LISTEN:-<IP_ADDRESS>:8000}"
+ KONG_PROXY_LISTEN: "${KONG_PROXY_LISTEN:-<IP_ADDRESS>:8000}, ${KONG_PROXY_LISTEN:-<IP_ADDRESS>:8443} ssl"

reference from here:
https://docs.konghq.com/gateway/3.9.x/reference/configuration/#proxy_listen


Insertion Sort using Call By Reference method

Problem
write a code to sort an array using the Insertion Sort algorithm by Call By Reference method. (LGM SOC)
Programming language

[x] C
[x] C++
[ ] Java
[ ] Python

Please assign me on this (LGM SOC)

I want to add Java solution to this problem. Please assign me this under LGM-SOC'21 @Amit366 .

I can do this in python.
Assign me plz.
LGM_SOC


Update Post “dva-c01-studying-s3-cloudfront”

Automatically generated by Netlify CMS

👷 Deploy Preview for dazzling-dieffenbachia-448a02 processing.

Name
Link

🔨 Latest commit
5b1a630440265f0a7ada907519f1d2791de6646f

🔍 Latest deploy log
https://app.netlify.com/sites/dazzling-dieffenbachia-448a02/deploys/62e9ee8343e6600008a11048


Fix message handling for manager-stakeholders

And a small drive-by cleanup, too.

ACK 1f26035 -- it was also tested during a demo deployment.


NBT Method Rename v2 for 20w15a

Before diving in for reviews, I would recommend reading #1078 for more context on past discussion. Essentially, the changes are as follows:

Serialization methods that modify an existing NBT Tag object (whether they return it or not) are named saveToNbt.
Serialization methods that return a new NBT Tag object are named toNbt.
Deserialization methods that modify an existing instance of an object using data from NBT Tag objects are named loadFromNbt.
Deserialization methods that create a new instance of an object using data from the NBT Tag objects are named fromNbt.

The distinction between modification versus creation is likely to seem odd to you at first, especially without having the code in front of you. Let me briefly explain:

All over the Minecraft codebases are examples such as this: in LootableContainerBlockEntity there are methods to save just their loot tables to NBT. saveLootTableToNbt is just far more readable and it is immediately understood what is going on versus lootTableToNbt, which looks like a conversion.
Methods named xToNbt imply that the return value will be exactly x in NBT format. However, when really this method appends additional NBT onto it, it is important to make the distinction that NBT is being saved to an existing NBT tag.
There are actually many places in the code where these different types of methods are used together and using all toNbt and fromNbt would cause conflicting names. Such places are in BlockEntity (static fromNbt as well as the fromNbt where NBT data is loaded for an instance).
Including save and load in the names of the methods used by Block Entities and Item Stacks has the added benefit of it being an easy searchable term for if a new modder wanted to save some data to their Block Entity or Item. Obviously, this is just a coincidence due to how Mojang has it currently implemented, but it is a nice added benefit.

Also, I don't think Enigma nuked any comments like in previous PRs, thankfully.

These names seem to get changed back and forth every other day.

If anyone else wishes to continue this PR, feel free. Changing names in yarn is too tedious a process and I'm no longer interested.


allow user to define custom tooltips.json

these are the messages that are currently being used by default:
https://github.com/peopledoc/html-linter/blob/master/defaults/tooltips.json
maybe we would like to give users the option to define their own custom error messages ?

this is already implemented but not documented ⚠


Using PrimaryJoin to filter relationship causes AttributeError

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the SQLModel documentation, with the integrated search.
[X] I already searched in Google "How to X in SQLModel" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to SQLModel but to Pydantic.
[X] I already checked if it is not related to SQLModel but to SQLAlchemy.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
#
# What I want in pure sqlalchemy
#

from sqlalchemy import create_engine, Column, ForeignKey, Integer, String
from sqlalchemy.orm import declarative_base, relationship, Session

engine = create_engine("sqlite://", echo=False)
Base = declarative_base()

class Team(Base):
    __tablename__ = "team"
    id = Column(Integer, primary_key=True)
    heroes_jeff = relationship(
        "Hero",
        primaryjoin="and_(Team.id==Hero.team_id, " "Hero.name=='Jeff')",
    )

class Hero(Base):
    __tablename__ = "hero"
    id = Column(Integer, primary_key=True)
    team_id = Column(Integer, ForeignKey("team.id"))
    name = Column(String)

def create_db_and_tables():
    Base.metadata.create_all(engine)

def main():
    create_db_and_tables()
    with Session(engine) as session:
        team = Team()
        session.add(team)
        session.commit()
        session.refresh(team)

        hero = Hero(name="Jeff", team_id=team.id)
        session.add(hero)
        session.commit()
        hero = Hero(name="Dave", team_id=team.id)
        session.add(hero)
        session.commit()
        session.refresh(team)
        session.refresh(hero)

        print("Team w/ Jeff:", len(team.heroes_jeff))

if __name__ == "__main__":
    main()

#
# How I am trying to get it to work with SQLModel
#

from typing import List, Optional
from sqlalchemy.orm import relationship
from sqlmodel import Field, Relationship, Session, SQLModel, create_engine

class Team(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    heroes_jeff: List["Hero"] = Relationship(
        sa_relationship=relationship(
            "Hero", primaryjoin="and_(Team.id==Hero.team_id, Hero.name=='Jeff'"
        )
    )

class Hero(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    team_id: Optional[int] = Field(default=None, foreign_key="team.id")
    name: str = Field(index=True)

sqlite_file_name = "database.db"
sqlite_url = f"sqlite:///{sqlite_file_name}"

engine = create_engine(sqlite_url, echo=False)

def create_db_and_tables():
    SQLModel.metadata.create_all(engine)

def main():
    create_db_and_tables()
    with Session(engine) as session:
        team = Team()
        session.add(team)
        session.commit()
        session.refresh(team)

        hero = Hero(name="Jeff", team_id=team.id)
        session.add(hero)
        session.commit()
        session.refresh(team)
        session.refresh(hero)

        print("Team w/ Jeff:", len(team.heroes_jeff))

if __name__ == "__main__":
    main()

Description
I want to be able to use primaryjoin to "filter" a relationship. I made toy example posted in the example code. There are two seperate files in that example code section. The first is using pure sqlalchemy and it works as expected. When I try to implement it in SQLModel I get this error:
line 45, in main
    print("Team w/ Jeff:", len(team.heroes_jeff))
AttributeError: 'Team' object has no attribute 'heroes_jeff'

Is there something SQLModel that prevents the use of primaryjoin, or am I missing something to get this to work. I've scrubbed all the documentation and can't find an example of this.
Any help would be greatly appreciated!
Operating System
Windows
Operating System Details
No response
SQLModel Version
0.0.8
Python Version
Python 3.8.6
Additional Context
No response

Figured it out right after posting. Hopefully this helps someone in the future.
from typing import List, Optional
from sqlmodel import Field, Relationship, Session, SQLModel, create_engine

class Team(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    heroes: List["Hero"] = Relationship(
        back_populates="team",
    )
    heroes_jeff: List["Hero"] = Relationship(
        back_populates="team",
        sa_relationship_kwargs={
            "primaryjoin": "and_(Team.id==Hero.team_id, Hero.name=='Jeff')",
            "viewonly": True
        },
    )

class Hero(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str = Field(index=True)
    team_id: Optional[int] = Field(default=None, foreign_key="team.id")
    team: Optional[Team] = Relationship(back_populates="heroes")

sqlite_file_name = "database.db"
sqlite_url = f"sqlite:///{sqlite_file_name}"

engine = create_engine(sqlite_url, echo=False)

def create_db_and_tables():
    SQLModel.metadata.create_all(engine)

def main():
    create_db_and_tables()
    with Session(engine) as session:
        team = Team()
        session.add(team)
        session.commit()
        session.refresh(team)

        hero1 = Hero(name="Jeff", team_id=team.id)
        hero2 = Hero(name="Dave", team_id=team.id)
        session.add(hero1)
        session.add(hero2)
        session.commit()
        session.refresh(team)
        session.refresh(hero1)
        session.refresh(hero2)

        print("Team w/ Jeff:", len(team.heroes_jeff))

if __name__ == "__main__":
    main()

Thanks for reporting back and closing the issue 👍


Cannot query or delete data using api.

Cannot query or delete data using sql. The sql statement configured in the path parameter can not query data, specifically can not concatenate two query conditions with "or".  Using one of these criteria alone can query the data.
The sql is:
SELECT * FROM account where (accountId = 'A-00000003' and time ='2019-03-10T20:00:00Z') or (accountId = 'A-00000002' and time ='2020-12-08T21:00:00Z')
The curl is:
curl --location 'http://localhost:8086/query?db=dbName&q=SELECT * FROM account where (accountId %3D 'A-00000003' and time %3D'2019-03-10T20%3A00%3A00Z') or (accountId %3D 'A-00000002' and time %3D'2020-12-08T21%3A00%3A00Z')'
--header 'Authorization: Token testToken'

When the sql is: SELECT FROM account where (accountId = 'A-00000003' and time ='2019-03-10T20:00:00Z'), the data can be found.
When the sql is: SELECT FROM account where (accountId = 'A-00000002' and time ='2020-12-08T21:00:00Z'), the data can also be found,
but the sql is: SELECT FROM account where (accountId = 'A -00000003' and time ='2019-03-10T20:00:00Z') or (accountId = 'A-00000002' and time ='2020-12-08T21:00:00Z') but cannot find out the data?


Use liveUntilLedgerSeq in auth

I've noticed this in the previous PR, and I don't think we need the rename. Signature actually expires, so I think it's useful to distinguish it from the new state archival naming.


Require version .20 of Kitura-net

Updates the dependency of Kitura-net to 0.20 and resolves #4

updated to .22


Update ansible submodule

This PR is to update ansible submodule. Below changes will be included:
https://github.com/ppc64le-cloud/k8s-ansible/pull/10 - add redhat subscription
https://github.com/ppc64le-cloud/k8s-ansible/pull/12 - Changed bits download bucket to k8s-release-dev

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Rajalakshmi-Girish
To complete the pull request process, please assign mkumatag after the PR has been reviewed.
You can assign the PR to them by writing /assign @mkumatag in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @mkumatag

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: mkumatag, Rajalakshmi-Girish
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [mkumatag]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Code refactor milestone

In this milestone we:

Make sure YAGNI, KISS, and DRY rules and Ruby style guide are respected.
Follow SOLID principles.
Make sure that the role of main.rb is only to start your code and ask the user for the initial choice of options (or to quit).
Extract some pieces of logic into separate module : create.rb and list.rb

Thank you for your review @codepantha


Blocking snapshot of given index patterns

Is your feature request related to a problem? Please describe.
We are implementing a plugin which store proprietary data in OpenSearch indices of which cannot be stored longer than certain period, like 30 days. Therefore, we want to prevent creating snapshot of those indices.
Describe the solution you'd like
Provide a way to exclude index from being taken as snapshot through either by index setting or by plugin specifying index pattern.
Describe alternatives you've considered
Storing the data as a file in disk.
Additional context
https://github.com/opensearch-project/OpenSearch/issues/5856

There are two options in implementations.

Add new index setting index.snapshot_eligible. Default is true. When the value is false, it is excluded when we resolve index name for snapshotting. https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/metadata/IndexNameExpressionResolver.java#L316
Add new field in SystemIndexDescriptor to determine if the index is snapshot eligible or not. In SnapshotService, exclude an index if the index is not eligible for snapshot. https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/plugins/SystemIndexPlugin.java#L55

First option give more flexibility as it allows any index to be excluded from snapshotting through index setting. Second option provide more strict policy where user cannot unblock the snapshot eligibility of an index once it is defined by plugins.

@heemin32 Trying to understand a bit more about the requirements.
I think this already exists as a part of the API currently -
indices | The indices you want to include in the snapshot. You can use , to create a list of indices, * to specify an index pattern, and - to exclude certain indices. Don’t put spaces between items. Default is all indices.
-- | --

Do you want to add a feature to mark an index/index pattern as not capable for backup?

Current API is for user to exclude certain index pattern from being taken in snapshot. I want a way to exclude index pattern from being taken as snapshot by system.

One of the purposes of a snapshot is to be able to capture the state at a point in time such that a cluster can be restored to that state. Omitting a system index seems like it will violate that property.

One of the purposes of a snapshot is to be able to capture the state at a point in time such that a cluster can be restored to that state. Omitting a system index seems like it will violate that property.

In this case, we have two system index: one is to store metadata and another to store proprietary data and we are only blocking a snapshot of proprietary data. When the metadata index is restored, we will restore the proprietary data on the fly. Meanwhile a user might fail to use the data; in this case, ip2geo processor to convert ip to location data.
Due to a license issue, even if we store the snapshot of proprietary data, it won't be much usable once it gets older than 30 days when it is restored. User will see failure in such case anyway.

When the metadata index is restored, we will restore the proprietary data on the fly

From where? You mean re-index from source?
Seems like your requirement should be 'delete from snapshots' rather than 'not take snapshot'? If the 'proprietary data' gets deleted in the cluster, isn't it better to delete the snapshots older than X days? By not taking snapshot at all, you will not be able restore data in case of failures within 30 days as well.

When the metadata index is restored, we will restore the proprietary data on the fly

From where? You mean re-index from source?
Seems like your requirement should be 'delete from snapshots' rather than 'not take snapshot'? If the 'proprietary data' gets deleted in the cluster, isn't it better to delete the snapshots older than X days? By not taking snapshot at all, you will not be able restore data in case of failures within 30 days as well.

metadata index has information about where to download and generate the proprietary data. The proprietary data is downloaded from external endpoint outside of OpenSearch cluster.
If metadata index is restored, we will scan the metadata index and restore the proprietary data by downloading it from external endpoint. Such way, regardless whether the data is older than 30 days or not, it will get restored after few minutes.
Yes. Deleting snapshots older than 30 days could be more accurate requirement but I think blocking snapshot make the problem much more simpler with small drawback; restoring time.

If I understand the use case correctly, you don't want to take the snapshot of those indices as the actual source of the data exists elsewhere and you prefer to restore the data from actual source instead of restoring it from snapshot.
In that case, I would suggest to add it as an index setting and make SnapshotsService ignore them (based on the setting value).
Also, how are you planning to restore the data during restore of metadata index?

If I understand the use case correctly, you don't want to take the snapshot of those indices as the actual source of the data exists elsewhere and you prefer to restore the data from actual source instead of restoring it from snapshot.
In that case, I would suggest to add it as an index setting and make SnapshotsService ignore them (based on the setting value).
Also, how are you planning to restore the data during restore of metadata index?

Correct. Actual source of the data exists elsewhere. For restoring, plugin will register itself to cluster state change event listener and act on metadata index restoring event. It will trigger background task of downloading proprietary data from the external endpoint and creating index again.

Got it. The proposal looks clean to me.

I would still prefer the option of snapshotting and retaining it for 30 days since thats what snapshots are for. They help protect data from accidental deletion, modification and infrastructure failures. If you want to restore data from a different source that should still be possible with restore API that restores only indices with certain patterns.

I would still prefer the option of snapshotting and retaining it for 30 days since thats what snapshots are for. They help protect data from accidental deletion, modification and infrastructure failures. If you want to restore data from a different source that should still be possible with restore API that restores only indices with certain patterns.

There are two indices: metadata index, temp data index. We take snapshot of only metadata index but not temp data index. Once metadata index is restored from snapshot, the temp data will be restored from an external endpoint. Temp data index has no use without metadata index.

Blocking restoring of snapshot could be another option.

There are other case where this requirement is required. For example, job-scheduler create lock index when scheduled job try to acquire lock. It maintains job index separate from lock index. If job index is restored first and job is triggered before lock index is restored, it will create a new lock index. Then, lock index restoration will fail due to conflict index name.

What happens if I'm using this feature that requires proprietary data and I turn my cluster off for 31 days then turn it back on? How would that situation be different from restoring a 31 day old snapshot?

Blocking certain index from restoring of snapshot could be another option.

Is the requirement that you can't use data that is older than 30 days? If so, then it seems like application logic to detect and delete invalid/expired data is maybe what you need and would be a more general solution to the problem.

What happens if I'm using this feature that requires proprietary data and I turn my cluster off for 31 days then turn it back on? How would that situation be different from restoring a 31 day old snapshot?
In my case, ingest processor use the data and if data is older than 31 days, it enrich document with error message. The data will be deleted in the next update interval. This situation is a little different with restoring from snapshot because 1. this is not general use case, 2. Every indices are there before cluster start whereas, index restoration restore index one by one which can cause race condition between restoring process and data generation process.

Is the requirement that you can't use data that is older than 30 days? If so, then it seems like application logic to detect and delete invalid/expired data is maybe what you need and would be a more general solution to the problem.
That detecting and invalidating data is already there which could be enough. This blocking creation or restoration of snapshot is an additional safeguard to reduce a future risk of license violation.

This blocking creation or restoration of snapshot is an additional safeguard to reduce a future risk of license violation, and avoid race condition.

re: license violation - I think this logic has to be correct regardless of snapshot behavior. My silly example above is one way data older than 30 days can be present, but there are many others, such as starting a cluster from an old EBS snapshot or hand-rolling snapshotting functionality by making a copy of the OpenSearch data directory. In other words, if any part of your license compliance story relied on not being able to take snapshots, then there are likely other loopholes.
re: avoid race condition - I believe there are multiple cases today where system indices get automatically created. When restoring a full snapshot to a new cluster a user generally has to make a choice as to whether to exclude those indices in the restore process (and therefore discard whatever settings or state was stored in them) or to close the indices on the destination cluster and take the data from the snapshot. Its a fair point to ask whether this experience can be improved, but I'm wary of taking such a heavy-handed approach to always exclude a particular index from snapshotting, especially given that the exclusion can already happen today at both snapshot creation and restore time using the existing API.

I am thinking if we can block users from direct querying to certain indices

Direct access to system indices is currently deprecated for removal in a future version. This deprecation was inherited at the time of the fork and I'm not aware of any work actively being done to implement the logic to block users' access to system indices in 3.0, but this is in line with the planned direction as far as I know.

Hey @heemin32 , based on the discussion, seems this proposal isn't the right solution todo, I am closing this issue for now, please feel free to comment to re-open if needed.


stardoc needs fixes after generation

To make lists in attribute declarations work.
to remove  ... tags which show up rather than controlling rendering

I started work on this, with a tool to post-process the output.
If we spot any other issues let's add them here so I can tweak the tool and catch them all at once.

I have no plans for new fixups right now. To reduce clutter, I am closing this. We can reopen as we see opportunities.


2.2.2 Guidance, Tests

As the document is currently written, including the text that says "...and that the function can be performed by the role(s)..." in the Guidance section could be interpreted as a check on the guidance document, rather than as a real test of the functionality, which should be the intent of these words.  Recommend re-structuring the wording so that:

The second para of the Guidance section reads: "The evaluator shall verify for each claimed management function that the Operational Guidance is sufficiently detailed to allow the function to be performed".
In the Tests section, insert the clause about actually testing the functions before the existing text: "The evaluator shall test each management function for each role listed in the FMT_MOF_EXT.1.1 table in the ST to demonstrate that the function can be performed by the role(s) that are authorized to do so and the result of the function is demonstrated.  The evaluator shall also verify..."

As written there is nothing requiring any of the management functionality to actually be tested.

Language adopted.

Language adopted.


Add example of using dataset.save with different data types

Addresses #554
I added an example to dataset#save here showing how to save with different types. We can copy this example to somewhere else in the docs if necessary? @jgeewax You mentioned something about putting it in the overview docs, but there's no code there right now so I'm not sure how out of place that would feel and I don't think it would have any code highlighting in this case. Ideas? What feels like the best place to put this?

Looks awesome!


pushpull-client challenge_websocket fails with an unclear error message

I've installed pushpull-server. Then I run it:
$ pushpull-server
DEBUG:asyncio:Using selector: EpollSelector
======== Running on http://localhost:8080/ ========
(Press CTRL+C to quit)

In another terminal:
$ pushpull-client challenge_websocket http://localhost:8080/ user_token
DEBUG:asyncio:Using selector: EpollSelector
Traceback (most recent call last):
  File "/home/opus-test/test/bin/pushpull-client", line 11, in <module>
    sys.exit(client())
  File "/home/opus-test/test/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/opus-test/test/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/opus-test/test/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/opus-test/test/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/opus-test/test/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/opus-test/test/lib/python3.5/site-packages/pushpull/cli/client.py", line 22, in challenge_websocket
    click.echo(loop.run_until_complete(websocket_client.challenge(url, token, sys.stdin, sys.stdout, loop)))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/opus-test/test/lib/python3.5/site-packages/pushpull/websocket/client.py", line 22, in challenge
    async with session.ws_connect(url, headers={'Origin': 'localhost'}) as ws:
  File "/home/opus-test/test/lib/python3.5/site-packages/aiohttp/client.py", line 537, in __aenter__
    self._resp = yield from self._coro
  File "/home/opus-test/test/lib/python3.5/site-packages/aiohttp/client.py", line 307, in _ws_connect
    headers=resp.headers)
aiohttp.errors.WSServerHandshakeError: 404, message='Invalid response status'

It is a bug that this error message is not clear to know why it does not work.

Solved by changing to challenge_websocket http://localhost:8080/pushpull.
Please update your docs (README.rst) with this URL.

Thanks @vporton ! Fixed in https://github.com/elastic-coders/pushpull/commit/2ee4518b4ed918c2674edf16bb579348e6eb66e6


[Feature] Configure database security rules

Before public test and release, a more strict set of security rules need to be configured.
Refer to

https://firebase.google.com/docs/firestore/security/insecure-rules
https://console.firebase.google.com/u/1/project/jigsaw-25200/firestore/rules

Made updates to the rules.

Mostly done and verified, though the current rules are still pretty loose.

Done.


New version: PostgreSQL.PostgreSQL version 16.2

[ ] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Microsoft Reviewers: Open in CodeFlow

@MuhammadTahaNaveed please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree


[Question] Netty 4.0.34 send messages parallel

Hi,
If 2 messages will be sended to client, will they be sended parallel, or does netty wait, that first message has received client?
`//send big message with 100MB of data, takes a while
ctx.writeAndFlush(message1);
//send message with 1 KB of data
ctx.writeAndFlush(message2);`
Why message2 isnt received before message1?

Please use http://stackoverflow.com/questions/tagged/netty for such questions.
Now on your question: if this is the same channel (client side) used, then the messages are ordered as the program said, so message1 first, message2 second. If not, how could you manage your protocol ?


Use Nectary's Configuration to retrieve target SOLR endpoint

Addresses https://github.com/gios-asu/search-api/issues/41
PR should:

Retrieve URL for SOLR endpoint from config.conf in root project directory.
Place a sample config file in project root.

The only thing missing is the sample config

Also could you add a quick section to the README.md under getting started that lets everyone know they need to copy the config.conf.sample to config.conf and modify it accordingly.


how to set x axis data in line chart

I know this question is very basic,
but I can not find any useful information in documents and issues.
In example there is only area data series, but I don't know how to specific x axis data in line chart.
please help!

//x轴坐标每个点对应的名称(注意:这个不是用来设置 X 轴的值,仅仅是用于设置 X 轴文字内容的而已)
AAPropStatementAndPropSetFuncStatement(strong, AAChartModel, NSArray     <NSString *>*, categories)


Syntactic sugar for matches, using wild _

Example:
((_, _) => 1)
as alternative for:
((a, b) => 1)

done