Promote good practices with notebooks and include notebook templates

I would like to propose the promotion of good practices with notebooks and the inclusion of jupytemplate in the cookiecutter  for better notebook organization and readability.
Notebooks play a crucial role in data science projects, and maintaining a clear structure can greatly improve their readability and maintainability. Jupytemplate is a tool that provides a template system for Jupyter notebooks, enabling the creation of consistent notebook structures. It allows users to define a template notebook with pre-defined sections, headers, and explanatory text, guiding users to follow a standardized structure.
I suggest considering two options for implementing this:

Including jupytemplate as part of the cookiecutter: By integrating jupytemplate into the cookiecutter, users would have it readily available when creating new notebooks from Jupyter Lab. This would encourage the adoption of good practices from the beginning of the project.
Providing clear instructions and recommendations: Alternatively, we can provide instructions in the README.md file of the cookiecutter template, suggesting the use of jupytemplate and providing a link to its template, or directly adding this template on the notebook folder. This approach would ensure that users are aware of the benefits of using such a template and can easily utilize it in their notebooks.

Having jupytemplate integrated or recommended within the cookiecutter template would provide a consistent structure for notebooks, enhance readability, and help enforce best practices across the team.

Yes, this one  😅.

hmm is something like this applicable to most of the cases? I think that too often we will not need half of the sections in the template or the structure of the structure will not be suitable for the purpose of the notebook. If we force to create all the notebooks with a template, then all the notebooks will have default empty/non used sections like

Parameter definition
We set all relevant parameters for our notebook. By convention, parameters are uppercase, while all the other variables follow Python's guidelines.
Data import
[...]

Which I think is counter productive because it adds noise and can be confusing.
On the other hand if we just include an optional template it is up to the user to make use of it and it will not be used that much and will add a yet another file around the project that nobody knows why is there...
Note that I'm being catastrophic and I'm speaking for myself putting myself as being full of bad habits but good proxy of general behavior
What I think will work best is to include some links in the science wiki / way to work / culture ( or even in the template README) to some good notebooks as a reference for the people to follow. For example the pytudes of Peter Norvig are perfect example of awesome notebooks:

https://github.com/norvig/pytudes/blob/main/ipynb/StarBattle.ipynb
https://github.com/norvig/pytudes/blob/main/ipynb/Convex Hull.ipynb

They all follow more or less the same pattern of Title -> overview -> imports -> problem breakdown w code
but the structure is not closed at all.

Yes, I completely agree with your points. Leaving empty or unused sections in a notebook template would not make any sense and may cause confusion. It is important to have flexibility in the notebook structure to accommodate different cases.
The sections in the template should be viewed as general guidelines, not strict rules. Each user can modify, remove, or add sections according to their needs to maintain a clear and logical structure.
From my experience, many of our notebooks contain only code and are difficult to follow. I usually try to use this structure, and although I don't use all sections, it helps me find things more easily. Therefore, I believe that having a suggested structure like the template can be very helpful in making notebooks more readable.

Also is worth bringing into the discussion that what we do- writing notebooks- is a practice older that what we might think! This paradigm was invented by Donald Knuth ( yes, also the inventor of TeX) in the 80s and called literate programming (don't know if it is relevant to the conversation but quite interesting :sweat_smile: ).
For me, anything that will make our work get cleaner and better ( closer to feeling like a pytudes notebook :heart_eyes: ) is worth the effort. I'll put together a simple template and we can discuss an iterate over it and how to include it in the project workflow.

Thanks, that will be great.


Handle the case where there is no signingAlgorithm or signed params s…

…ent back instead of disabling url_sig.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/579/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/580/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR-trafficops-test/535/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR-trafficops-test/536/


Improve submodule support

Add submodule configuration to git source config

Codecov Report

Merging #1361 (0d97dd2) into main (4bbd92e) will increase coverage by 0.01%.
Report is 2 commits behind head on main.
The diff coverage is 75.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@            Coverage Diff             @@
##             main    #1361      +/-   ##
==========================================
+ Coverage   67.29%   67.30%   +0.01%
==========================================
  Files         133      133
  Lines        8224     8228       +4
==========================================
+ Hits         5534     5538       +4
  Misses       2237     2237
  Partials      453      453

Files
Coverage Δ

pkg/apis/build/v1alpha2/build_types.go
100.00% <ø> (ø)

pkg/apis/build/v1alpha2/image_types.go
50.00% <ø> (ø)

pkg/git/remote_git_resolver.go
95.91% <100.00%> (+0.26%)
:arrow_up:

pkg/git/fetch.go
59.25% <46.15%> (+0.50%)
:arrow_up:


Setting ccache sloppiness in ccache.conf doesn't get detected.

Having sloppiness = pch_defines,time_macros in ccache.conf isn't detected by cotire.
fatal error: file 'example_CXX_prefix.hxx' has been modified since the precompiled header       'example_CXX_prefix.hxx.gch' was built fires off every time.

Fixed in 1.7.10.


Add UI for property

Closes #122

Great work for implementing the UI changes for Properties, looks good to me!

LGTM


How to avoid local and AD users conflicts?

Hello.
I'm currently having issues with one of my AIX servers which has about 50 local users with the same name than AD users.
When I join this server to the domain the users can't log in with either their local or AD user.
I tried turning AssumeDefaultDomain to false hoping that I could distinguish between the two users forcing to use the DOMAIN\username form for AD users but it didn't work.
I also tried setting the same UID for both local and AD users but that it didn't work well, the results were somewhat unexpected because even using the AD credentials it redirected me to the local user's home directory.
I tried registering users in /etc/pbis/user-ignore file, but it didn't make any difference.
Deleting local users is not an option.
So, at this point, my question is: What is the best way to overcome this behavior? Is there any config option that I could set to solve this?

There are a couple of things that you can try.

Try setting AssumeDefaultDomain to true.
Try setting the UID's of the LOCAL users to be different than the AD users.

From the way the above is stated, it sounds like you are mapping local to AD accounts given that you are setting AssumeDefaultDomain to false, and setting the UIDS to the same for each of the accounts. This would in turn, set them to map to each other.

Thanks for your answer.
I've already tried that. That's my default setup. Given that my common configuration was not working I gave a try to the scenarios stated in my first post, with the described results.

user-ignore is the method to achieve this. user-ignore meant to list the local accounts that you would like lsass to ignore. If this is not working then you may need to debug lsass to see what is happening at authentication. Make sure the user-ignore file is updated with an account on a new line and match exactly with the local account.

Thanks for your answer @rbest-bt
After editing the user-ignore file is it necessary to restart lsass services or something else? I couldn't find documentation regarding user-ignore file, is there any reference about how to use the user-ignore file?

I'm doing some testing with the user-ignore file. Indeed right now it is possible to log in with the local user credentials and I have no problem resetting this user's password, I run an id command and it shows the local user's info.
What's somewhat odd is that it let me log in using the domain credentials too, and it access as the local user, so even when I use domain credentials in this scenario and run the id command it shows the local user info too. This behavior is fine, these local users are logged in the server but I don't know if it is working as designed or maybe it's a bug.
Is it OK that users included in the user-ignore file could log in with their domain credentials to their local accounts?

Options to correct this are limited in Open due to the lack of its configurations.
pbis-open will longer receive updates and will be archived. Closing all outstanding issues. Please consider BeyondTrust Active Directory Bridge for continued support.
https://www.beyondtrust.com/privilege-management/active-directory-bridge


Report if version to be stopped is not running

Given the following command:
dotnet ws stop -v <IP_ADDRESS>34234 -f

The tool silently does nothing, but it should report that the version given is not running.

I am getting multiple messages (seems to be as many as running instances) instead of just one:
C:\MyApp01>dotnet ws list
The following wsfscservice versions are running:

  version: <IP_ADDRESS>0; path: C:\Users\Adam\.nuget\packages\websharper.fsharp\<IP_ADDRESS>0-beta2\tools\net6.0\win-x64\wsfscservice.exe
  version: <IP_ADDRESS>4; path: C:\Users\Adam\.nuget\packages\websharper.fsharp\<IP_ADDRESS>4-beta2\tools\net6.0\win-x64\wsfscservice.exe
  version: <IP_ADDRESS>8; path: C:\Users\Adam\.nuget\packages\websharper.fsharp\<IP_ADDRESS>8-beta2\tools\net6.0\win-x64\wsfscservice.exe.

C:\MyApp01>dotnet ws stop --version <IP_ADDRESS>-beta1
Could not find running service with version: <IP_ADDRESS>-beta1
Could not find running service with version: <IP_ADDRESS>-beta1
Could not find running service with version: <IP_ADDRESS>-beta1


AQI accuracy & P2 from PPD42

Hi,
Really love the board and I got everything up and running and sending data via MQTT to my Home Assistant. Only have 2 questions:

I can't get the AQI accuracy above 1 it's most of the time 1 and sometimes even 0. I've already blow on the sensor but it looks like nothing works
The PPD42 has a port 2 if you like at this https://github.com/mauricecyril/pidustsensor it's used for reading PM1.0 values. Is there an option to use this port?

Hello!

It can take a long time, especially for new sensors (there can be a "burn-in" time for stabilization). It is important to keep the sensor running frequently (use 3 second cycles). Expose to clean air and dirty air, for example alcohol vapor.

The MS430 board has a single input for particle sensing. You can connect it to the PPD42 pin 2 instead of pin 4, both pins use the same output signal method. However, the manufacturers do not give much information about pin 2 or the related threshold input on pin 5.

Thanks for the answers. The accuracy went after 5 hours to 3 and then after a few hours back to 0. Is this normal?  I have the refresh on 100sec at this moment.
For port 2 I understand. Going to see if I can find more info on this.

When the accuracy went from 3 to 0, did the MS430 board exit cycle mode, or could it have been reset? If either of these happens, the calibration starts again from accuracy 0.

To summarize the accuracy value:
It can take several hours for a new sensor to stabilize and go to higher accuracy. The stabilization will be faster if you use a shorter cycle time. As the sensor ages it should be quicker.
If you exit cycle mode, or reset the board, it will start again from 0 and needs to recalibrate again. Otherwise the accuracy should rarely return to 0 after going to 1.
If it remains 1 for a long time, expose the sensor to a marker pen, alcohol etc. The AQI should go to about 500 and then the accuracy will go to 2.
The accuracy will then slowly alternate between 2 and 3, this is intended.

For me it variates between 2 3 and 1. It's running constantly with a cycle of 100sec. You can find my code I use here https://github.com/Sennevds/sensor/tree/master/Home_Assistant I don't think I do anything special. When I'm back from my holiday I'm going to make a separate sensor for the accuracy so I can better track the values.

OK, that sounds like normal operation.


Add command line support

Adds support for the prism plugin Command Line.
It adds the following features for the Command Line plugin:

User and Host parameters
Prompt parameter
Ouput line parameter

Appropriate specs for the added features were added as well.

Thanks! I will take a look as soon as I get a moment.

@JordanKnott I published 1.0.0 with this change. Thanks again for the addition!


Fix setting brightness to max value and brightness fading loop condition

Function _denormalize_brightness does not clamp the output to the normal 0-255 int() range. Force-clamped 0-255 now.
Function _normalize_brightness does not clamp the output to the normal 0-100 percent range. Force-clamped 0-100 now.
In @brightness.setter, the while loop looks only for current_value != value. In some circumstances, it won't stop incrementing.  Hard-clamped to 0-100 percent range.

Thanks @linusg for taking the time to review the PR.
I'm not too familiar with the black formatting part, anything you can point me to?
Thanks!

Install https://github.com/psf/black
Run black rpi_backlight/__init__.py
Commit and push the changes it makes
Done :)

@linusg Thanks for the info!

Thanks for the fixes!


Subquery generation is creating syntactically wrong queries - gorm2/mysql

I have a mariadb based database that has a few tables and I use subqueries. After migrating to gorm2, I noticed that the generated subquery string is syntactically incorrect and hence it fails. To be specific, the subquery is missing parenthesis and hence the query is failing. Am I missing something?
// The statement below errors out due to syntax error in sql statement. The statement
err1 := db.Where("device_id = ?", db.Table("device").Select("id").Where("mgmt_ip = ?", "<IP_ADDRESS>")).Find(&dp).Error
_ = err1
// The statement below works fine
err1 = db.Raw("select * from device_port where device_id = (select id from device where mgmt_ip = '<IP_ADDRESS>')").Find(&dp).Error

Observed query that failed:
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'SELECT id FROM device WHERE mgmt_ip = ?' at line 1
[1.994ms] [rows:0] SELECT * FROM device_port WHERE device_id = SELECT id FROM device WHERE mgmt_ip = '<IP_ADDRESS>'
If a paranthesis () is added for the inner SELECT, all is fine.
Reference program: https://play.golang.org/p/n1BpWwGS2Ou
I'm not a great fan of using raw statements when we have the abstraction provided by gorm to hide underlying database nuances. So a fix would be required.

db.Where("device_id = (?)"

Ah! Worked.
Missed my eyes. This, for some reason, was working with older gorm and I missed.


Memory caching in nistats.second_level_model.SecondLevelModel()

Using this standard example (section <IP_ADDRESS>.) from nistats.github.io, but with adding the 'memory' parameter to SecondLevelModel():
import pandas as pd
from nilearn import datasets

n_samples = 2
localizer_dataset = datasets.fetch_localizer_calculation_task(
    n_subjects=n_samples)
cmap_filenames = localizer_dataset.cmaps
design_matrix = pd.DataFrame([1] * n_samples, columns=['intercept'])

from nistats.second_level_model import SecondLevelModel
second_level_model = SecondLevelModel(memory='nilearn_cache').fit(
    cmap_filenames, design_matrix=design_matrix)
z_map = second_level_model.compute_contrast(output_type='z_score')

It will return AttributeError:
/home/bnguyen/build/dev_env_1/lib/python3.6/site-packages/numpy/core/fromnumeric.py:83: RuntimeWarning: invalid value encountered in reduce
  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-20-8307f1e30bbf> in <module>
----> 1 z_map = second_level_model.compute_contrast(output_type='z_score')

~/build/dev_env_1/lib/python3.6/site-packages/nistats/second_level_model.py in compute_contrast(self, second_level_contrast, first_level_contrast, second_level_stat_type, output_type)
    408         if self.memory is not None:
    409             arg_ignore = ['n_jobs']
--> 410             mem_glm = self.memory.cache(run_glm, ignore=arg_ignore)
    411         else:
    412             mem_glm = run_glm

AttributeError: 'str' object has no attribute 'cache'

I looked into https://github.com/nistats/nistats/blob/master/nistats/second_level_model.py and apparently the joblib.Memory object is not initialized after inputing memory parameter, so the it still remains as the string of caching location. Also as a consequence when doing .fit() alone the code will run smoothly but the memory will also not be cached as intended.

Indeed. It would be great if you could issue a PR (taking inspiration if necessary from Nilearn objects that do the same type of business, e.g. CanICA) ?


Inconsistent Generation for BNB 4Bit Llama-7B on Transformers vs TGI

System Info

OS Version: Ubuntu 22.04.2 LTS
Hardware: 1 Nvidia A10 GPU on AWS g5.4xlarge instance
Deployment specificities: Just developing on a VM
Version: Running with official TGI Docker Image 1.1.0

Launch Command:
docker run --gpus device=0 --shm-size 1g --network host -v $tgi_volume:/data -e HUGGING_FACE_HUB_TOKEN=$hf_token -it ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id meta-llama/Llama-2-7b-hf --quantize bitsandbytes-nf4
Information

[X] Docker
[ ] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
I was attempting to verify that TGI generates the same responses as my local transformers model when quantizing with Bitsandbytes NF4.
Using transformers directly, I saw the following example generation:

Transformers Version: 4.33.3
Bitsandbytes Version:  0.41.1

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    GenerationConfig
)

EXAMPLE_PROMPT = "In a galaxy far, far away,"
MODEL_ID = "meta-llama/Llama-2-7b-hf"

# load model
model_transformers = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    load_in_4bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type='nf4',
    ),
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# tokenize
model_inputs = tokenizer(EXAMPLE_PROMPT, return_tensors="pt").to("cuda")
generation_config = GenerationConfig(
    max_new_tokens=100,
    do_sample=False
)

# generate text
output = model_transformers.generate(
    **model_inputs,
    generation_config=generation_config
)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])

The output is as follows:
In a galaxy far, far away, a long time ago, a young boy named Luke Skywalker was born. nobody knows what happened to him after that.
The first time I saw Star Wars was when I was 10 years old. I was in the 5th grade. I was in the library, and I was reading a book. I was reading a book about the history of the Roman Empire. I was reading a book about the history of the Roman Empire. I was reading a book about the history of

However, when launching a text generation server with nf-4 quantization on the same model, I get a different generation result with the same greedy decoding strategy:
Launch server:
docker run --gpus all --shm-size 1g --network host -v $tgi_volume:/data -e HUGGING_FACE_HUB_TOKEN=$hf_token -it ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id meta-llama/Llama-2-7b-hf --quantize bitsandbytes-nf4

Make request:
curl <IP_ADDRESS>:80/generate \
    -X POST \
    -d '{"inputs":"In a galaxy far, far away,","parameters":{"max_new_tokens":100,"do_sample":false}}' \
    -H 'Content-Type: application/json'

The output is as follows:
{"generated_text":" the Empire is in control. everybody is under the control of the Empire. The only hope for the galaxy is the Rebel Alliance. The Rebels are fighting the Empire to free the galaxy from the control of the Empire. The Rebels are led by a man named Luke Skywalker. Luke is a young man who is trying to find his place in the galaxy. He is trying to find his place in the galaxy by joining the Rebel Alliance. The Rebels are"}

Expected behavior
I would expect the generated text from the server with the same quantization scheme to match the generation form local transformers

Have you tried removing
 bnb_4bit_compute_dtype=torch.float16,
 bnb_4bit_use_double_quant=True,

TGI uses all defaults (I'm not sure what double_quant stands for there).
Also it's perfectly ok if they don't generate the same thing, given it's not the same codebase, therefore subtle differences in logits can happen (they often do). What's more suprising is that they happen right away, so there's definitely something more that could be at play here.

Yeah it makes sense there is some differences especially with the custom kernels and whatnot, I just think the magnitude of the difference was more than I expected

The default dtype for the model in TGI is fp16, so I don't think that is the issue
The double_quantization refers to quantizing the weights and scales. I believe TGI sets this argument to true as compress_statistics=true in the Linear4Bit constructor at https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/utils/layers.py#L236

I will dig in a bit more and compute some statistics about logits difference and look into some layer by layer items to see if I can isolate it

@rsnm2 If you can log the input_ids in both versions that'd help too, I'm wondering if it couldn't be there already.

Quick update
I have narrowed the issue down to the attention layer
What I am seeing is:

hidden state passed to the first layer is consistent for transformers and tgi
hidden state passed to the second layer and beyond diverge w/ l2 norms >0.05

Looking closer, I am seeing some numerical differences in the input_layernorm calculation. TGI uses flash attention's dropout_layer_norm.dropout_add_ln_fwd while transformers uses a torch implementation. I switched out the flash attention implmentation for the torch implementation, and this removed the numerical differences for the input layernorm
After doing this, the input to the query, key, value calculations are exactly the same across transformers and tgi, but I am seeing large divergence in the results between transformers and tgi
# tgi model, note --- uses modified layernorm to match transformers implementation
layer = tgi_model.model.model.layers[0]
normed_tgi, _ = layer.input_layernorm(layer.hidden_states, layer.residual)

# transformers model
ts_layer = model_transformers.model.layers[0]
normed_ts = ts_layer.input_layernorm(output.hidden_states[0][0][0])

# confirm inputs are the same
print((normed_tgi - normed_ts).norm())
# >> tensor(0., device='cuda:0', dtype=torch.float16)

# tgi model q,k,v
qkv = layer.self_attn.query_key_value(normed_tgi)
query, key, value = qkv.split(
    [
        layer.self_attn.head_size * layer.self_attn.num_heads,
        layer.self_attn.head_size * layer.self_attn.num_key_value_heads,
        layer.self_attn.head_size * layer.self_attn.num_key_value_heads,
    ],
    dim=1,
)

# transformers model q,k,v
query_ts = ts_layer.self_attn.q_proj(normed_ts)
key_ts = ts_layer.self_attn.k_proj(normed_ts)
value_ts = ts_layer.self_attn.v_proj(normed_ts)

# compute differences
print((query_ts - query).norm())
# >> tensor(0.6914, device='cuda:0', dtype=torch.float16)

print((key_ts - key).norm())
# >> tensor(0.6172, device='cuda:0', dtype=torch.float16)

print((value_ts - value).norm())
# >> tensor(0.0218, device='cuda:0', dtype=torch.float16)

Looking into how the layer.self_attn.query_key_value works in TGI, I see that there is one large weight matrix representing Wq, Wk, and Wv. As such, when we apply the quantization, under the hood there is one large Params4Bit weight created (https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/utils/layers.py#L233). As a result, in TGI, all of Wq, Wk, and Wv are quantized as one large tensor rather Wq, Wk, and Wv being quantized as three individual tensors (as they are in transformers).
The exact details of the bitsandbytes quantization scheme are a bit past my current understanding, but I am concerned quantizing the weight matrices jointly may result in different values than quantizing separately, as 4bit quantization typically relies on grouping weights into blocks with scales that are determined by the weights in those blocks. It is possible that we have some blocks created which overlap between the separate matrices
This is just a hypothesis and I am not sure if this is the issue, but I need to jump back to my day job, so I will investigate over the weekend

I have found the source of the discrepancy between transformers and TGI for 4bit quantized models
TLDR
The ultimate cause of the issue is that Torch does not guarantee that (A@B)[0] (the first element of the batched result) is bitwise identical to A[0]@B[0] (the matrix product of the first elements of the input batches) even though mathematically it’s an identical computation.
This creates issues because in TGI the query, key, value states are computed in one large linear layer while in Transformers they are computed as three separate linear.
Longer Investigation:

I found that the inputs to the first layer for TGI and Transformers is consistent, meaning there is no issue with tokenization or embeddings.

I found some numerical differences in input_layernorm calculation for TGI vs transformers. TGI uses flash_attentions optimized version of RMSNorm while Transformers uses a Torch implementation. I resolved this difference by swapping TGI's implementation for Transformers. However, after doing this, the logit differences remained.

With the input_layernorm discrepancies fixed, the inputs to the attention calculation are now exactly the same. I then found some large differences in the query, key, value states

layer = tgi_model.model.model.layers[0]
ts_layer = transformers_model.model.layers[0]

normed_tgi, _ = layer.input_layernorm(layer.hidden_states, layer.residual)
normed_ts = ts_layer.input_layernorm(output_transformers.hidden_states[0][0][0])

normed_ts = ts_layer.input_layernorm(output_transformers.hidden_states[0][0][0])

print("Input to Q,K,V Calculation Values Are The Same:")
print((normed_tgi - normed_ts).norm())
# >>> tensor(0., device='cuda:0', dtype=torch.float16)

qkv = layer.self_attn.query_key_value(normed_tgi)
query, key, value = qkv.split(
    [
        layer.self_attn.head_size * layer.self_attn.num_heads,
        layer.self_attn.head_size * layer.self_attn.num_key_value_heads,
        layer.self_attn.head_size * layer.self_attn.num_key_value_heads,
    ],
    dim=1,
)

print("\nQuery Key Value States Different:")
query_ts = ts_layer.self_attn.q_proj(normed_ts)
key_ts = ts_layer.self_attn.k_proj(normed_ts)
value_ts = ts_layer.self_attn.v_proj(normed_ts)

print((query_ts - query).norm())
print((key_ts - key).norm())
print((value_ts - value).norm())

# >>> tensor(0.0389, device='cuda:0', dtype=torch.float16)
# >>> tensor(0.0310, device='cuda:0', dtype=torch.float16)
# >>> tensor(0.0020, device='cuda:0', dtype=torch.float16)

Trying to debug this, I first looked into whether there were any issues actually quantizing the weights as one large matrix rather than as three separate matrices. However, looking into more detail at bnb's quantization scheme, I found that the default is to quantize in groups of 64 --> which meant that the quantized values were the exact same across the transformers and TGI implementations. There could be some small differences in cases where double quantization is turned on (which means that the metadata is also quantized) - this is on in TGI - but I turned this off. This meant that there is some numerical difference in the actual 4bit matrix multiplication:
import bitsandbytes as bnb

linear_qkv_tgi = tgi_model.model.model.layers[0].self_attn.query_key_value.linear
linear_q_ts = transformers_model.model.layers[0].self_attn.q_proj
linear_k_ts = transformers_model.model.layers[0].self_attn.k_proj
linear_v_ts = transformers_model.model.layers[0].self_attn.v_proj

cutoffs = linear_qkv_tgi.weight.shape[0] // 3
print("All weights are the same:")
print((linear_qkv_tgi.weight[:cutoffs] == linear_q_ts.weight).all())
print((linear_qkv_tgi.weight[cutoffs:cutoffs*2] == linear_k_ts.weight).all())
print((linear_qkv_tgi.weight[cutoffs*2:cutoffs*3] == linear_v_ts.weight).all())

x = normed_tgi
out_tgi_qkv = bnb.matmul_4bit(
    x, linear_qkv_tgi.weight.t(), bias=None, quant_state=linear_qkv_tgi.weight.quant_state
)

out_ts_q = bnb.matmul_4bit(
    x, linear_q_ts.weight.t(), bias=None, quant_state=linear_q_ts.weight.quant_state
)

out_ts_k = bnb.matmul_4bit(
    x, linear_k_ts.weight.t(), bias=None, quant_state=linear_k_ts.weight.quant_state
)

out_ts_v = bnb.matmul_4bit(
    x, linear_v_ts.weight.t(), bias=None, quant_state=linear_v_ts.weight.quant_state
)

print("\nBut delta in the outputs")
print((out_tgi_qkv[:,4096*0:4096*1] - out_ts_q).norm())
print((out_tgi_qkv[:,4096*1:4096*2] - out_ts_k).norm())
print((out_tgi_qkv[:,4096*2:4096*3] - out_ts_v).norm())

Output:
All weights are the same:
tensor(True, device='cuda:0')
tensor(True, device='cuda:0')
tensor(True, device='cuda:0')

But delta in the outputs
tensor(0.0389, device='cuda:0', dtype=torch.float16)
tensor(0.0310, device='cuda:0', dtype=torch.float16)
tensor(0.0020, device='cuda:0', dtype=torch.float16)

So, I dug into the implementation of bnb.matmul4bit. Ultimately I traced it back to this ([line]https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/autograd/_functions.py#L516)
output = torch.nn.functional.linear(A, F.dequantize_4bit(B, state).to(A.dtype).t(), bias)

Wow thanks a lot for this very deep dive, your findings are enlightening !
Given what you're saying I'm not sure we should do anything.
Do you think there's anything we should be doing ?
Fusing qkv, gives a significant performance boost at inference time, and there's no reason bnb should be that sensitive to it.

I think you are right --- especially since its not even bnb that is causing the issue (as the analysis above shows, the quant/dequant is working as expected)
Im still working through the full analysis though.
Right now, after running the QKV at FP64 (just to get rid of the numerical issue during debugging), I'm still getting quite different logits. The divergence starts after RoPE is applied so I'm just going to try to see whats going on (any maybe even switch over to an A100)
Ill make a suggestion once I get through the full analysis
Thanks for listening to my ranting!

Btw, since  you're diving that deep, you should still take a look at AWQ (or GTPQ) both of which have much faster runtimes than nf4.
For the RoPE, we do use the flash attention kernel implementation there, so there might be some other differences there.

Thanks for the tip!


Update range pattern refutability

Range patterns may now be irrefutable, as of exhaustive_integer_patterns.
Fixes #472.

Looks great; thanks!


Add guide for using BLS Wallet/Quill in a dApp

What is this PR doing?
Add a guide to the docs section on how to use BLS Wallet/Quill in a dApp, highlighting the multi-action transaction feature.
How can these changes be manually tested?

Run through the guide in an existing/new dApp and make sure it works.

Does this PR resolve or contribute to any issues?
Resolves https://github.com/web3well/bls-wallet/issues/222
Checklist

[x] I have manually tested these changes
[x] Post a link to the PR in the group chat

Guidelines

If your PR is not ready, mark it as a draft
The resolve conversation button is for reviewers, not authors

(But add a 'done' comment or similar)

left a few minor comments, lgtm otherwise


New App: Switchboard

This adds a new app, Switchboard. Switchboard provides tools for political organizations to run more effective campaigns. This is a v1 of the app that allows you to see your organization's total donations on your Tidbyt.
Displayed:

Switchboard Logo (Submitted on behalf of Switchboard Public Benefit Corp, all rights reserved)
The text "Switchboard" if you are not authenticated via API token
Your donation total if you are using a token provided by Switchboard

I have read the CLA Document and I hereby sign the CLA
However, the Logo image is submitted on behalf of a third-party: Switchboard

However, the Logo image is submitted on behalf of a third-party: Switchboard

recheck

P.S. This app is intended primarily for Switchboard users (~20-100 devices total) not general consumption. If there is a better way to distribute the app to those users please lmk.

Thanks for the PR. If you ping me on Discord I can hook you up with beta access to test it all out in production, before we make it publicly available to all users. Heads up that I'll be out for most of next week though so thing may move a bit slower than normal.


getting multiple error

Steps to Reproduce

Execute flutter run on the code sample
...
...

Expected results:
Actual results:

Code sample

Logs

import 'package:english_words/english_words.dart';
import 'package:flutter/material.dart';
import 'package:provider/provider.dart';
error in this part

@Vishal143-dot
You are probably referring to https://codelabs.developers.google.com/codelabs/flutter-codelab-first#2 but we are not sure what errors you are getting as you have not mentioned any. Please provide proper details as what error are you getting while following the codelab.


Update pom in spring-cloud-azure-starter-jdbc-*

As title.

API change check
APIView has identified API level changes in this PR and created following API reviews.
spring-cloud-azure-starter-jdbc-mysql


WIP: docker and docker-compose integration

Thanks for the initial work on the Dockerfile. I was playing around with it today and got it running with the docker-compose.yml provided by gekko itself. This isn't "production-ready" code, more a proof of concept but works well on my local machine.

Nice, I hav never used docker but it seems nice.. will try to put this to work later x]

@Gab0 thanks for merging it in! as i said, it's already working but maybe you can abstract things a bit further by using dockers env variables etc.
I'll will try to deploy it to another machine i'm using as docker host and push things further as soon as i have some time to work on it again.

another feature could also to extend gekkos default docker-compose.yml as described here: https://docs.docker.com/compose/extends/#understand-the-extends-configuration
https://github.com/docker/compose/pull/2051

#27


Windows Command Line Install

Your Feature Request
As a user, I would like to make use of Tesseract in the software I am developing.
For which, our client and/or users are mostly non-technical background. We already have a Installation script for our software.
I do not use MacOS. But I did saw a Homebrew and "sudo apt install ... "1-liner command tesseract installation for MacOS.
Link for reference: _https://tesseract-ocr.github.io/tessdoc/Installation.html_
Since our software depends upon Tesseract, we would like to make sure that we install it for all users. Specifically speaking of Windows,
Do we have a one-command line installation for it?
As I had to downloads the binaries (exe file) and manually click "Next" ... To install Tesseract.
How can I automate that for windows (or have a 1-click install ? )

We don't provide an installer.
The installer you're talking about is made by another entity:
https://github.com/UB-Mannheim/tesseract/wiki
So you should ask them about their installer.


Google Analytics not working; not sure what to do next

I have installed the package, saw an error about there being no settings.json, added an appropriate settings.json and the error went away.
In the console I see
analytics:integration:google-analytics initialize +0ms
So everything looks great, but I don't see myself appear in the real time part of GA, it doesn't load a tracking pixel, and the Google Analytics Tag Assistant browser plugin says there is no GA script on the page.
What should I try next?
I have a single page app and haven't installed a router at all - could this cause a problem?

@jimmytidey
With the current version of Analytics, you need either FlowRouter or IronRouter for analytics page triggers to work.
@oliverthomasklein has worked on an update which makes it agnostic to the router that's used.

Fantastic, thanks.
Ok, well I guess for now I'll just add flow router. Hopefully I'll be able to remove it in the future!
Great project!

@jimmytidey Thanks, the Flow Router example in the repo here may be helpful to you as it demonstrates basic setup/how it works. If you have any other issues or the README is missing anything let us know.


Fixes #3868:  enhance ui for faqs page

Fixes:  #3868
Description
change

there is another faq in books page. this is leading to redundancy .
remove faq section from books page .

@abhi03ruchi I have not added faqs
Just added the script code so that only one faq can be expanded at a time.

@abhi03ruchi I have not added faqs Just added the script code so that only one faq can be expanded at a time.

i know that . just chk books page . it also had a faq section . just delete faq section code from there .

@abhi03ruchi I have not added faqs Just added the script code so that only one faq can be expanded at a time.

i know that . just chk books page . it also had a faq section . just delete faq section code from there .

FAQ's are there in these 12 files. Just to conform, you want me to remove them from all the files other than index.html ?

just check once the pages , if the faq section is the same on the pages as on the home page . delete them.

@abhi03ruchi done


lisk-cli deposit

Hi guys
I'm trying to deposit som lisk to my test dapp. I'm following the online tutorial and i ran into a problem. Maybe I'm doing something wrong, but i get an error everytime i try to deposit lisk to my test app through the cli.
lisk@lisk:~/lisk-test$ lisk-cli dapps -d
? Enter secret *********
? Enter amount 10000000
? DApp Id<PHONE_NUMBER>723621833
? Enter secondary secret (if defined)
? Host and port localhost:7000
null { success: false,
  error: 'Application not found:<PHONE_NUMBER>723621833' }
Application not found:<PHONE_NUMBER>723621833

app.log:
log 2016-07-07 09:52:25 GET /api/dapps from <IP_ADDRESS>
log 2016-07-07 09:52:25 GET /api/dapps/installed from <IP_ADDRESS>
log 2016-07-07 09:52:28 GET /api/loader/status/sync from <IP_ADDRESS>
log 2016-07-07 09:52:31 PUT /api/dapps/transaction from <IP_ADDRESS>
log 2016-07-07 09:52:31 error error: ERROR:  unterminated quoted identifier at or near """
LINE 1: ...t" FROM dapps WHERE "transactionId" = '9100591690723621833'"
                                                                      ^
log 2016-07-07 09:52:31 error query: SELECT COUNT(*)::int AS "count" FROM dapps WHERE "transactionId" = '9100591690723621833'"
log 2016-07-07 09:52:33 GET /api/loader/status/sync from <IP_ADDRESS>
log 2016-07-07 09:52:38 GET /api/loader/status/sync from <IP_ADDRESS>
log 2016-07-07 09:52:38 GET /api/peers/version from <IP_ADDRESS>

I'm running Ubuntu 14.04 and have been following this tutorial:
https://lisk.io/documentation?i=lisk-dapps-docs/MessagingApp

Please try this on the latest development version of Lisk: https://github.com/LiskHQ/lisk. I'm quite sure this issue has been fixed here.

Thank you very much!


fix zig: unsupported exported_symbols_list

Relates: #164

I'm not sure if this actually works because -exported_symbols_list takes a value, see https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-exported_symbols_list.

And looks like it only applies to macOS targets, see https://github.com/rust-lang/rust/blob/dda95c14e1a9356397c6369762047db031e69461/compiler/rustc_codegen_ssa/src/back/linker.rs#L714-L715

I'm not sure if this actually works because -exported_symbols_list takes a value, see https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-exported_symbols_list.

Thanks for pointing that out. Should I do something like this?

And looks like it only applies to macOS targets, see https://github.com/rust-lang/rust/blob/dda95c14e1a9356397c6369762047db031e69461/compiler/rustc_codegen_ssa/src/back/linker.rs#L714-L715

My bad. I haven't checked prior conditions.
I am good at neither Rust nor compiling but I will try to make a PR =)

Should I do something like this?

I think we need remove both -Wl,--exported_symbols_list and the value follows, currently filter_linker_arg doesn't support that, so the code needs a bit refactoring.

I think we need remove both -Wl,--exported_symbols_list and the value follows, currently filter_linker_arg doesn't support that, so the code needs a bit refactoring.

My case looks like "-Wl,-exported_symbols_list,/tmp/rustcnkqKsZ/list" so arg.starts_with should do the job.

What's your Rust version?

What's your Rust version?

Stable 0.17.2

OK, so according to the commit message, both -Wl,-exported_symbols_list,<path> and -Wl,-exported_symbols_list <path> can happen.
So IMO we can merge this first and deal with -Wl,-exported_symbols_list <path> later.

Thanks!


Difbulkgen

Hi Mellissa,
I hope you are well. This is my first pull request, so please let me know if I leave out information you need...
I came across an issue when running my study trajectories. There was odd behavior with the _meteofinder function, which was "finding" too many files and triggering an error in HYSPLIT, where more than 12 meteo files cannot be included in the CONTROL file for HYSPLIT to run. I am running the analysis from 1979-2016, and the years 1998, 1999, 2000, and 2001 for some reason "find" more files than are needed/allowed (33, 30, 30, and 27 respectively). My files are named as suggested (NARR.apr1984_0 as an example). I was able to get _meteofinder to focus on narrowing down the search by removing the _year2string function from the _monyearstrings function, as you can see below, and allowing _meteofinder to search for the meteo files with a 4 digit year, rather than a two digit year. This solved the over-population of meteo files in the CONTROL file for me. What are your thoughts? While troubleshooting, I made a copy of the bulktrajgen function to one that doesn't use itertools.product, but rather a zip folder, which works for me to generate just one trajectory. This isn't really important since the bulktrajgen function was not the issue at all, but I made this tweak as I was going through and troubleshooting. Feel free to merge that too if you like as a second way to generate trajectories, but the _meteofinder issue was what I was really after. Hope this works and helps!
Best,
Maria

I actually do NOT recommend this change anymore... Upon further inspection, it now appears these changes bumped my trajectory traj.data.DateTime and the labeling is off. I will need to look through this again, but would most definitely appreciate any insight into this issue!

Thank you for opening a pull request, Maria!  To figure out why 1998, 1999, 2000, and 2001 find way more files than necessary, it would be helpful to know what files are being found.  Would you like to raise an issue and post your results, either a screen shot of one of the overpopulated CONTROL files or the output from a print(meteofiles) somewhere?  That's very strange.  I do think it would be a good idea to let users search for either 2 or 4 digits of a year (and probably output 2 or 4 digits as desired to their trajectory file names). Unfortunately your fix would break workflows where meteorology files are named with just the last two digits- looks like GDAS1 HYSPLIT files, for example, are by default named with just the two digits- but the user supplied 4 digit years (that's me, I do this).  I'd also love to know how/what your tweak is doing to your traj.data.DateTime!
Are you supplying iterables to your generate_traj() to get zip() to work?  To make a function that just generates one trajectory, iterables, zip, monthslice, and neither of the loops are strictly necessary.
Thanks again!
Mellissa

Hi Mellissa,
I now know why my 1998 and 1999 are showing up as 2098 and 2099! It's not an error with the above suggested method nor anything critical, just the labeling method for datetime in hyfile_handler. I will post it as a separate issue.
I will also create another issue with screenshots about the excess meteo files being generated to supplement this pull request. I will leave this pull request open since I do really need the 4-digit search for my loop/iterable method to work without crowding the CONTROL file with too many meteofiles. Hopefully the screenshots will shed some light. To answer your question -- yes, I am supplying the traj generator function with a list to loop using the zip folder.
Thanks again for all your help!
Best,
Maria


npm package scripts should be recognized by typescript (webstorm 11 has it!!)

npm scripts
A new npm scripts tool window lists all the tasks specified in the package.json scripts section and allows you to easily run any of them with a simple double-click. You can also create a new Run configuration for an npm script task and run it with the familiar Run... action.

Hi Kelly,
sorry this is meant for Visual Studio Code!
kind regardsHans
Date: Tue, 3 Nov 2015 03:18:09 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [TypeScript] npm package scripts should be recognized by typescript (webstorm 11 has it!!) (#5507)
Are you suggesting this for VisualStudio or VSC or some other IDE?  I don't see how this fits into TypeScript.
—
Reply to this email directly or view it on GitHub.

For VSCode issues, please file them on the VSCode issue tracker: https://code.visualstudio.com/Issues/List


Images

Scratch issue to upload images to
VegaFusion:

Vega-Lite color:

Vega-Lite black:

Vega Color:

Vega Black:

SciPy:

Matplotlib:

JavaScript:

Rust

Python


Illegal client ID

I'm trying to run the RequestDataComparison notebook. I've set up the configuration in the following way:

config.instance_id from here (field in red):

config.sh_client_id (field in green) and config.sh_client_secret (field in purple) from here:

I am getting the following error:

Where am I going wrong?

Hi @iridescent-aeon,
It turns out you took client id and secret from the wrong places.

The ID in the green field is a user ID and is not important for making API requests.
The ID in the purple field is an OAuth client ID and should be assigned to config.sh_client_id.
An OAuth client secret is a string of characters that you get when you make a new OAuth client. You can only see it the first time when you create a new client and never again. But if you ever lose it you can always simply create a new client and delete an old one. That string should be assigned to config.sh_client_secret.

Everything else you described seems to be ok.

Interesting, I can swear I tried the option you suggested, but maybe there are problems with vscodium + jupyter notebooks, or something else.
tldr; changed appropriately and reopened vscodium, now everything works.
Thank you for the input :)

Hi @iridescent-aeon,
It turns out you took client id and secret from the wrong places.

The ID in the green field is a user ID and is not important for making API requests.
The ID in the purple field is an OAuth client ID and should be assigned to config.sh_client_id.
An OAuth client secret is a string of characters that you get when you make a new OAuth client. You can only see it the first time when you create a new client and never again. But if you ever lose it you can always simply create a new client and delete an old one. That string should be assigned to config.sh_client_secret.

Everything else you described seems to be ok.

I changed the config exactly according to the instructions. Where am I going wrong?

Hi @iridescent-aeon,
It turns out you took client id and secret from the wrong places.

The ID in the green field is a user ID and is not important for making API requests.
The ID in the purple field is an OAuth client ID and should be assigned to config.sh_client_id.
An OAuth client secret is a string of characters that you get when you make a new OAuth client. You can only see it the first time when you create a new client and never again. But if you ever lose it you can always simply create a new client and delete an old one. That string should be assigned to config.sh_client_secret.

Everything else you described seems to be ok.

I changed the config exactly according to the instructions. Where am I going wrong?

Dear @likhitha-d01, please provide more information. Did you follow the steps here? Check for typos/empty spaces in the configuration file. Be sure to use double quotes in the json.

Hi @mlubej. Thanks for the fast reply.
This is the output of sentinelhub.config --show

The instance_id ( the one in blue ), is the id of the configuration in the Configuration utility.

The sh_client_id ( the one in green ) is the id of the OAuth Client in User Settings

The secret_id is the one we get during creating the OAuth client.
A bit of background:
I am running the Land Use Land Cover example notebook, in a VM in GCP. I'm hosting the notebook using the jupyter-notebook.
Also, I have initialized the parameters in the command line.
I also checked for spaces and typos.
Thanks again for your help!

Hi @iridescent-aeon,
It turns out you took client id and secret from the wrong places.

The ID in the green field is a user ID and is not important for making API requests.
The ID in the purple field is an OAuth client ID and should be assigned to config.sh_client_id.
An OAuth client secret is a string of characters that you get when you make a new OAuth client. You can only see it the first time when you create a new client and never again. But if you ever lose it you can always simply create a new client and delete an old one. That string should be assigned to config.sh_client_secret.

Everything else you described seems to be ok.

I changed the config exactly according to the instructions. Where am I going wrong?

Dear @likhitha-d01, please provide more information. Did you follow the steps here? Check for typos/empty spaces in the configuration file. Be sure to use double quotes in the json.

Hi @mlubej. Thanks for the fast reply.
This is the output of sentinelhub.config --show

The instance_id ( the one in blue ), is the id of the configuration in the Configuration utility.

The sh_client_id ( the one in green ) is the id of the OAuth Client in User Settings

The secret_id is the one we get during creating the OAuth client.
A bit of background:
I am running the Land Use Land Cover example notebook, in a VM in GCP. I'm hosting the notebook using the jupyter-notebook.
Also, I have initialized the parameters in the command line.
I also checked for spaces and typos.
Thanks again for your help!

Hi @likhitha-d01,
everything seems to make sense to me. My next idea is to check the output of !sentinelhub.config --show in the notebook where you are also running things. It is possible that the JN kernel is using the wrong instance of the python interpreter.
Besides checking !sentinelhub.config --show in the JN cell, you can also try
from sentinelhub import SHConfig
SHConfig()

and check the output. Is it what you expect it to be?

Hi @mlubej
The output is the same as !sentinelhub.config --show.
Could it be an issue with any other parameters? or maybe the following code:
band_names = ['B02', 'B03', 'B04', 'B08', 'B11', 'B12']
add_data = SentinelHubInputTask(
    bands_feature=(FeatureType.DATA, 'BANDS'),
    bands = band_names,
    resolution=10,
    maxcc=0.8,
    time_difference=datetime.timedelta(minutes=120),
    data_source=DataSource.SENTINEL2_L1C,
    additional_data=[(FeatureType.MASK, 'dataMask', 'IS_DATA'),
                     (FeatureType.MASK, 'CLM'),
                     (FeatureType.DATA, 'CLP')],
    max_threads=5,
    #instance_id="e7d4aa46-4893-4b2b-8d54-d70998e4e006"
)

# TASK FOR SAVING TO OUTPUT (if needed)
path_out = './eopatches/'
if not os.path.isdir(path_out):
    os.makedirs(path_out)
save = SaveTask(path_out, overwrite_permission=OverwritePermission.OVERWRITE_PATCH)

workflow = LinearWorkflow(
    add_data,
    save
)

# Execute the workflow
time_interval = ['2019-01-01', '2019-12-31'] # time interval for the SH request

# define additional parameters of the workflow
execution_args = []
for idx, bbox in enumerate(bbox_list[patchIDs]):
    execution_args.append({
        add_data:{'bbox': bbox, 'time_interval': time_interval},
        save: {'eopatch_folder': f'eopatch_{idx}'}
    })

executor = EOExecutor(workflow, execution_args, save_logs=True)
executor.run(workers=5, multiprocess=True)

executor.make_report()

Hi @likhitha-d01,
I don't think your problem comes from the code or using wrong ids, but rather your configuration in the configuration utility. Most likely some of the bands you are using in the code were not included in the "custom_script" configuration on the sentinel hub page. I have stopped using this for a while, so perhaps someone else would be able to help you adjust the configuration, based on your required bands. Alternatively, if my memory serves me right, there was a config that included all bands, from the pre-given templates.

Most likely some of the bands you are using in the code were not included in the "custom_script" configuration on the sentinel hub page.

This likely isn't it, since we implemented changes with the processing API to bypass this completely.
@likhitha-d01 it seems to me that what you're doing is correct. Perhaps there was some mismatch along the way. I would suggest deleting your oauth client and creating a new one. Let me know if this works.

I tried the same configuration with the Sentinel FIS, and it worked perfectly fine. I guess there is some issue with the SentinelHubInputTask function
Also, @iridescent-aeon thanks for the Google Engine tip!

Hi @likhitha-d01,
FIS request uses instance ID only and not oauth-client, so this is not the reason either, unfortunately.
May I have your email account, so that I can check your setup in our system?

Hello,
<EMAIL_ADDRESS>is my email account registered on the sentinel
hub website.
Thanks,
Likhitha
On Mon, Sep 28, 2020 at 12:28 PM Matic Lubej<EMAIL_ADDRESS>wrote:

Hi @likhitha-d01 https://github.com/likhitha-d01,
FIS request uses instance ID only and not oauth-client, so this is not the
reason either, unfortunately.
May I have your email account, so that I can check your setup in our
system?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sentinel-hub/eo-learn/issues/240#issuecomment-699816368,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIE63RV7S6NZYDP2Z3KMFK3SIAXXZANCNFSM4PQEBY4A
.

Hi @mlubej
It worked when I passed the config = SHConfig() to the SentinelHubInputTask function explicitly.
I ran the same example Land Use Land Cover notebook.
Thanks for your help :)

Hi @likhitha-d01
sorry for not being able to help sooner, but I'm glad to hear this was resolved.
Thinking about it now, it might also be possible that you have some other instance of sh-py installed and the config file was changed in the one not used in the notebook. You could try removing all installations of sh-py in case you have more than one and then reinstalling.
Passing the config file as you did now proves that the credentials are correct.
Anyway, glad it works now :)


Create PROFILE.md

Add my first file
created a new branch, created a file, and made a commit.

my fist merge


Correct  cocli command help output

Fixes #47

From the diff I wouldn't be able to tell what the effect of the added spaces is :-)

I have verified each command manually before submitting the PR.


Set the correct path for the settings.json file

Make sure the htmlcache folder exists or else create it
Removed unused _cache folder
Removed unused default.gitignore

Thanks Koen!


How can I ignore camera point?

Required Info

Camera Model
D435i

Firmware Version
(<IP_ADDRESS>)

Operating System & Version
Win 11

Platform
PC

SDK Version
2.53.1

Language
python

Segment
Robot

Issue Description
The point cloud I got seems to carry the point of the camera's own position, but I don't want this point. How can I get the point without the camera's position？
I use point clouds for plane fitting, but sometimes camera points affect my fitting results, which makes my results unreliable, so I want to discard camera points.

this is my code
            self.pc.map_to(color_frame)
            points = self.pc.calculate(temp_filtered)
            # save ply point cloud
            points.export_to_ply(config.final_path+'point_cloud.ply', color_frame)
            vtx = np.asanyarray(points.get_vertices())
            colorful = np.asanyarray(color_frame.get_data())
            colorful = colorful.reshape(-1, 3)
            point_cloud = []
            index = 0
            for p in vtx:
                point_cloud.append(
                    [p[0], p[1], p[2], colorful[index][0], colorful[index][1], colorful[index][2]])
                index += 1
            points.export_to_ply(config.BASE_CLOUD_PATH, color_frame)
            np.save(config.NPY_PATH, point_cloud)

Hi @qq996235826  Using the RealSense SDK instruction rs2_transform_point_to_point to transform points from one coordinate space may be what you are seeking.  More information about this can be found at https://github.com/IntelRealSense/librealsense/issues/5583

Hi @qq996235826 Using the RealSense SDK instruction rs2_transform_point_to_point to transform points from one coordinate space to another may be what you are seeking. More information about this can be found at #5583

I had trouble using rs2_transform_point_to_point，But I used open3d's remove_statistical_outlier method to remove the point

I'm pleased to hear that you achieved a solution.  Thanks very much for the update!


Multiple sections with the same name

Hi,
I have to deal with a 3rd party INI file that has multiple section of the same name but different keys inside. I haven´t seen a way to handle this case with MadMilkman.Ini, or am I wrong? I think I have to find my own approach, but if anybody has an idea, I would be greatful!
Example:
[DriveLetters]
DeviceID=1234567890
Letters=A

[DriveLetters]
DeviceID=0987654321
Letters=B

Hi,
By default MadMilkman.Ini does support section and key duplicates.
Now in order to access those duplicate sections you'll need to use integer indexer instead of string indexer because with the second one you'll only be able to access the first "DriveLetters" section that occurs in the provided input.
In other words, try the following example:
string iniContent =
@"[DriveLetters]
DeviceID=1234567890
Letters=A

[DriveLetters]
DeviceID=0987654321
Letters=B";

IniFile ini = new IniFile();
// Load that above INI content, or you could just load an INI file or an INI stream.
ini.Load(new StringReader(iniContent));

// Get first section, it contains "DeviceID=1234567890" and "Letters=A" keys.
IniSection section1 = ini.Sections[0];

// Get "DeviceID" value:
Console.WriteLine(section1.Keys["DeviceID"].Value);
// Or you can also use this:
//Console.WriteLine(section1.Keys[0].Value);

// Get second section, it contains "DeviceID=0987654321" and "Letters=B" keys.
IniSection section2 = ini.Sections[1];

// Get "DeviceID" value:
Console.WriteLine(section2.Keys["DeviceID"].Value);
// Or you can also use this:
//Console.WriteLine(section2.Keys[0].Value);

Also here is another example that demonstrates how you can iterate through INI's content and retrieve the desired information (I used Where extension method from System.Linq).
string iniContent =
@"[DriveLetters]
DeviceID=1234567890
Letters=A

[DriveLetters]
DeviceID=0987654321
Letters=B";

IniFile ini = new IniFile();
ini.Load(new StringReader(iniContent));

// Iterate through INI's content and write the required values.
foreach (IniSection section in ini.Sections.Where(s => s.Name == "DriveLetters"))
    foreach (IniKey key in section.Keys.Where(k => k.Name == "DriveLetters"))
        Console.WriteLine(key.Value);

Last just in case you're interested, there is also another indexer which accepts targeting multiple names, for example:
string iniContent =
@"[DriveLetters]
DeviceID=1234567890
Letters=A

[DriveLetters]
DeviceID=0987654321
Letters=B";

IniFile ini = new IniFile();
ini.Load(new StringReader(iniContent));

IEnumerable<IniSection> driverLetters = ini.Sections["DriveLetters", "DriveLetters", "DriveLetters"];
foreach (IniSection section in driverLetters)
{
    if (section != null)
        Console.WriteLine(section.Keys["DeviceID"].Value);
}

Notice that here I'm targeting three "DriveLetters" sections, but the input INI content has only two "DriveLetters" sections and thus the last retrieved IniSection is null.
Nevertheless, I believe the second example is the one that you require.
I hope this helps, let me know if you need anything else.

Hi @MarioZ
thank you so much for help, this made my day!


Add optional $Resource param, defaults to https://management.azure.com/

Fix for #93
I'm not sure how versioning works in extensions, so please fix as needed.

Hi @stephaneey lemme know if there's anything I can do to help move this along. Thanks

Hi @joeyeng,
Thanks for your contribution and sorry for the late answer, I have been rather overloaded these times.
Thanks
Best Regards


Getting error in let csv = try! CSVWriter(stream: stream)

I am trying to update a CSV file in local intranet server.
guard let stream = OutputStream(toFileAtPath: peopleListSheetURL, append: false) else { return }
stream.open()
** peopleListSheetURL is a path to a CSV file in local intranet server**
let csv = try! CSVWriter(stream: stream).  ---**Here getting error Fatal error: 'try!' expression unexpectedly raised an error: CSV.CSVError.cannotOpenFile: file /BuildRoot/Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-9<IP_ADDRESS>/src/swift/stdlib/public/core/ErrorType.swift, line 181 **
    for skillDetail in skillDB {
        if Int(skillDetail.value) == 0 {
            continue
        } else {
            try! csv.write(row: [empSelectionLabel.text!, skillDetail.key, skillDetail.value])
            csv.stream.close()
        }
    }

Please check stream.streamStatus after one line of stream.open().
If stream.streamStatus != .open, it has failed to open the file in write mode.
Perhaps, the user executing the process may not have write permissions.


uv-managed python installed into .local/bin can't be used with venv module

uv 0.4.30
python3.12 -m venv fails to create a virtual environment.
Steps how I reproduced this problem. Using 3.12.6 to use a version that was not already installed.
rm .local/bin/python3.12
uv python install --preview 3.12.6
# Installed Python 3.12.6 in 2.60s
#  + cpython-3.12.6-linux-x86_64-gnu (python3.12)
which python3.12
# /home/user/.local/bin/python3.12
python3.12 -V
# Python 3.12.6
python3.12 -m venv new_environment
# Error: Command '['/home/user/new_environment/bin/python3.12', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Let's try execute the python in the venv:
./new_environment/bin/python3.12

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './new_environment/bin/python3.12'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/user/.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/bin/python3.12'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/home/user/new_environment/bin/python3.12'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007136928b6740 (most recent call first):
  <no Python frame>

Is the symlink in the virtual environment resolved? (no)
ls -l ./new_environment/bin/python3.12
# lrwxrwxrwx 1 user user 33 nov  5 03:08 ./new_environment/bin/python3.12 -> /home/user/.local/bin/python3.12*

This is known and reported upstream: https://github.com/indygreg/python-build-standalone/issues/380

See also #8429

I'm running into this; uv-installed python symlinked into $HOME/.local/bin fails to create a working venv. I've read #8429 above and https://github.com/indygreg/python-build-standalone/issues/380 as well. Is there any workaround at the user level that I can use? My specific use case is in a github workflow; one of the actions calls python -m venv .action-bin and I can't change that, but I can adjust anything else beforehand.

I think export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))" should work, though it's pretty hacky.

I think export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))" should work, though it's pretty hacky.

Sadly, that doesn't work:
[root@4b801b307a52 aswf]# curl -LsSf https://astral.sh/uv/install.sh | sh
System glibc version (`2.17') is too old; checking alternatives
downloading uv 0.5.8 x86_64-unknown-linux-musl-static
no checksums to verify
installing to /root/.local/bin
  uv
  uvx
everything's installed!

To add $HOME/.local/bin to your PATH, either restart your shell or run:

    source $HOME/.local/bin/env (sh, bash, zsh)
    source $HOME/.local/bin/env.fish (fish)
[root@4b801b307a52 aswf]# source ~/.local/bin/env
[root@4b801b307a52 aswf]# uv python install --preview 3.11
Installed Python 3.11.11 in 2.10s
 + cpython-3.11.11-linux-x86_64-gnu (python3.11)
[root@4b801b307a52 aswf]# (cd ~/.local/bin; ln -sf python3.11 python3; ln -sf python3.11 python)
[root@4b801b307a52 aswf]# which python
/root/.local/bin/python
[root@4b801b307a52 aswf]# python --version
Python 3.11.11
[root@4b801b307a52 aswf]# export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))"
[root@4b801b307a52 aswf]# echo $PYTHONHOME
/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu
[root@4b801b307a52 aswf]# python -mvenv foo
Error: Command '['/opt/aswf/foo/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Alas, if you use --without-pip it works fine. I'm... not sure why the ensurepip call fails there.
Annoyingly, it works in isolation which makes it hard to inspect:
❯ python -m venv .venv
Error: Command '['/Users/zb/workspace/uv/.venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
 - pip==24.2
❯ .venv/bin/python -m ensurepip --upgrade --default-pip
Looking in links: /var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpgx6oozdb
Processing /var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpgx6oozdb/pip-24.2-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-24.2

I ended up prepending the actual python bin dir to $PATH. That seems to work for me.
Details at https://github.com/AcademySoftwareFoundation/openfx/commit/8276ac8156b55147da6343765cb1211d81c6ef23 if anyone’s interested.


Update RLC customization doc

We'll update the RLC customization document to provide guidance on how to enhance and customize their generated SDK.
All of the customization cases are derived from the MA customization doc and more discussions are touched upon there:

https://gist.github.com/MaryGao/61c523a35b27b5c1de35294d58ff26d9

Issue here: https://github.com/Azure/autorest.typescript/issues/1303

/check-enforcer evaluate

/check-enforcer override


Include 'False' as possible false value for boolean field

https://github.com/wtforms/wtforms/blob/b94260329964844d66c426072c649e64b6924a7c/wtforms/fields/core.py#L696
When submitting a form in a unit test using the flask client, if you do this:
client.post("/", data={"field": False})

The False value will become the string "False", which does not register as False in a BooleanFIeld.
This would be fixed by adding "False" to line 696.

Seems fine, if you could make a PR with a test, that would really help me out.

Reversing my stance on this. This is currently for parsing HTML form data. HTML forms don't send a value for unchecked inputs. When testing, you should be like a client and not send unchecked values either. I'm not even clear why we accept 'false' right now.
Even if we refactored to support JSON data, a boolean would deserialize to True or False before being passed to the form, so checking for the string still doesn't make sense.
If you really want to accept other values, you can already override it by passing in false_values or using a subclass that sets the false_values class attribute.

Well, you're the boss I guess. I'm not sure why you would reject this pull request that helps save people from subtle bugs.
I can pretty much guarantee you that people who want to use wtforms to build a system which has both forms for human interaction and also hits those same forms in an automated way so they don't have to build a separate REST API is going to get hit by this issue.
Anyway, thanks for your great work on wtforms and for listening to my rant. :)
Best wishes.

Maybe you changed your mind about this because it's really misleading and introduces subtle bugs when unit testing.
Right now the default false values as simple.py line 33:
false_values = (False, "false", "")
Having False in there is misleading because one should assume that posting, in a unit test, data from a user object will get in the form.data route the boolean False but instead gets True, yes because what you are actually posting is the string "False" which because isn't in false_values and is not None evaluates to True.
data = {
    "csrf_token": g.csrf_token,
    "name": user.name,
    "details": user.details,
    "email": user.email,
    # BooleanField
    "check_inv": user. check_inv,
    # BooleanField
    "admin": user.admin,
    # BooleanField
    "in_use": user.in_use,
    # BooleanField
    "sat_group": user.sat_group,
    "clean_order": "0",
    "submit": True,
}
response = client.post(
    url_for("users.edit_user", username=user.name),
    data=data)

I understand that this is a HTML form limitation and that we should send unchecked values also, just like HTML forms, but when testing you don't really know what this value will be and this introduces extra checks or extra code like str(user. check_inv).lower() to match "false" value (by the way just like you said I don't see the point of "false" there).
Overriding false_values on each BooleanField also introduces extra code on each field, and a thing you need to remember to pass each time you are making a BooleanField.
To summarise:

having False in false_values is misleading
having "false" in false_values seems useless
having "False" in false_values will make BooleanField behave as expected and in my opinion more pythonic (maybe also "0" and 0)

Please, if you have the time, take another look at this.


⚠ Bofrost.com has degraded performance

In bc5940d, Bofrost.com (https://www.bofrost.com) experienced degraded performance:

HTTP code: 200
Response time: 9045 ms

Resolved: Bofrost.com performance has improved in 0a9395e.


!is_live filter no longer working?

Checklist

[x] I'm asking a question
[x] I've looked through the README and FAQ for similar questions
[x] I've searched the bugtracker for similar questions including closed ones

Question

Here are my parameters
ydl_opts = {'match_filter': youtube_dl.utils.match_filter_func("!is_live"), 'retries':4, 'ignoreerrors': True, 'format': 'mp4', 'subtitleslangs': ['en'], 'writeautomaticsub': True, 'convertsubtitles': 'srt', 'restrictfilenames': True, 'outtmpl': fileDirectory}
This is causing my whole script to stop and download a live stream that goes on for 11 hours every day. Any idea what's wrong or why it's not skipping

???????

so you just decided to close the issue when the question is clear and instructions were followed? Very professional and helpful of you

There are no telepathists here. Bother to read new issue template and provide complete info you must to provide.


[WIP] Starting to add support for numeric suffixes.

The next release of Nim is supporting numeric suffixes. For example:
import strutils
type nibble = distinct uint8 # a 4-bit unsigned integer
proc `'nibble`(n: string): nibble =
  # The leading ' is required.
  result = (parseInt(n) and 0x0F).nibble

var x = 5'nibble

Because the single quote now has dual purposes the current NimLime highlighter gets confused.
This PR will do two things:

Make the numeric literal suffix rules more generic to handle custom suffixes.
Change the syntax highlighting for char literals so that it does not include explicit calls to the suffix procedures. for example:

proc aFour(): nibble =
  4'nibble
var x = aFour()
var y = 'h'

As you can see, GitHub is also confused by this. This makes sense since I believe GitHub's filters are based on NimLimes'.
I've marked this as work-in-progress (WIP) because item 2 is not done yet. I'm still wrapping my head around a good way to do it. At the moment, the invalid.illegal.character.nim checker is completely disabled. I'd like to carefully add that back in. I might do it as a separate rule rather than as a pattern underneath the current one.
Any and all help is very welcome!

I've added the flawed char tests back as separate rules. The check is not nearly as inclusive since there is no longer the expectation that all uses of apostrophe always have a match. Instead it is more specifically looking for sequences of letters and digits without whitespace or non-escape punctuation that are incorrect. All proper escapes such as
let x = `\``

are accounted for and marked as a proper char.
While not stepping on anything like:
let y = 23'foo.bar('\e')

I don't know of any problem cases.
But, generically, the "Error highlighting" that was fairly inclusive before only does single line analysis and is a bit more limited. I'm not convinced getting back to that is ideal; but perhaps I need to live with the changes longer.

ping @Araq now that PRs + issues were successfully transfered to https://github.com/nim-lang/NimLime
(btw that's the version that currently runs on github, see https://github.com/github/linguist/pull/5306)
also, could i please get sufficient rights to edit labels?


Workflow definition API is incomplete

1. Missing ids
All event type nodes do not have id. However their ids are used in parents/children properties of activity based nodes.
2. Change requests

2.a Activity based node "activityId" property should be  renamed with "id"

2.b Property "type" should be "SEND_MESSAGE" and "nodeType" should be "ACTIVITY" or for an event "MESSAGE_RECEIVED" and "EVENT" to ease the visual diagram creation.

Here is an example of what is currently returned:
{     "workflowId": "abcde",     "variables": [],     "flowNodes": [         {             "type": "SEND_MESSAGE_ACTIVITY",             "parents": [                 "message-received_/onboard"             ],             "children": [                 "form-reply_clientForm"             ],             "activityId": "clientForm"         },         {             "type": "UPDATE_MESSAGE_ACTIVITY",             "parents": [                 "form-reply_clientForm"             ],             "children": [                 "checkGleif"             ],             "activityId": "progressStatus"         },         {             "type": "EXECUTE_REQUEST_ACTIVITY",             "parents": [                 "progressStatus"             ],             "children": [                 "resFailure",                 "resSuccess"             ],             "activityId": "checkGleif"         },         {             "type": "UPDATE_MESSAGE_ACTIVITY",             "parents": [                 "checkGleif"             ],             "children": [],             "activityId": "resFailure"         },         {             "type": "EXECUTE_SCRIPT_ACTIVITY",             "parents": [                 "checkGleif"             ],             "children": [                 "statusMessage"             ],             "activityId": "resSuccess"         },         {             "type": "UPDATE_MESSAGE_ACTIVITY",             "parents": [                 "resSuccess"             ],             "children": [],             "activityId": "statusMessage"         },         {             "type": "MESSAGE_RECEIVED_EVENT",             "parents": [],             "children": [                 "clientForm"             ]         },         {             "type": "FORM_REPLIED_EVENT",             "parents": [                 "clientForm"             ],             "children": [                 "progressStatus"             ]         }     ] }

First version will be released in WDK 1.3.0
Gateways will be handled in a future release.
Example of response payload:
{     "workflowId": "data",     "variables": {         "loop": true     },     "flowNodes": [         {             "nodeId": "act0",             "type": "SEND_MESSAGE",             "group": "ACTIVITY",             "parents": [                 "message-received_/ping"             ],             "children": [                 "message-received_/continue"             ]         },         {             "nodeId": "act1",             "type": "SEND_MESSAGE",             "group": "ACTIVITY",             "parents": [                 "message-received_/continue",                 "act2"             ],             "children": [                 "act2"             ]         },         {             "nodeId": "act2",             "type": "SEND_MESSAGE",             "group": "ACTIVITY",             "parents": [                 "act1"             ],             "children": [                 "act1"             ]         },         {             "nodeId": "message-received_/ping",             "type": "MESSAGE_RECEIVED",             "group": "EVENT",             "parents": [],             "children": [                 "act0"             ]         },         {             "nodeId": "message-received_/continue",             "type": "MESSAGE_RECEIVED",             "group": "EVENT",             "parents": [                 "act0"             ],             "children": [                 "act1"             ]         }     ] }


[Question] How to use 2 adapters in single Activity? How to handle itemOnClick for both adapters?

[Question] I am using 2 Adapters in single activity, how i can handle the itemOnClickListner separately for each adapter ?

@khalid-mahmood, you actually can use multiple Adapters for different RecyclerVews, indeed.
To handle the click listeners, you need to distinguish in the same implementation of onItemClick() which RV / Adapter has triggered the event: basically which list the position is referring to.
Now, the only thing that comes in mind quick that sure works, is to override the onClick() of the FlexibleViewHolder and save somewhere the type of the View or the Adapter that generated the event, then you can call super(). Then in the onItemClick() you check which is the adapter to call getItem().
Another cleaner solution, is to provide you also the View, but I prefer to Not modify the method signature of the Listener in the library, need more justification since it has very heavy impact to everybody.

@khalid-mahmood, another clean solution is to use Fragments: each fragment includes the own Adapter and implements the proper listener.

static field used to identify the adapter, crazy hack
already using fragment. in fragment using two adapters.
@davideas can you make simple like. pass adapter too with item position in onItemClickListner(int position).
Thanx for help.. Good Work.

@khalid-mahmood, I see, but I can't modify that method as explained.
Another solution (best I think) is the following:
Instead of the Fragment implementation you create 2 custom fields of type OnItemClickListener, so you can implement the method and assign it to the right adapter. That listener will be called only by that Adapter.
private FlexibleAdapter.OnItemClickListener clickListenerAdapter1 =
		new FlexibleAdapter.OnItemClickListener() {
	@Override
	public boolean onItemClick(int position) {
		//...
		return false;
	}
};

mAdapter1.initializeListeners(clickListenerAdapter1);

Same for the 2nd Adapter.

@davideas  Thanks. This Works Like Charm..
but same issue with empty list.  any more suggestion?

adapter.initializeListeners(onUpdateListner1);
used same for Update Listener. working fine.
Thank you so much for help @davideas

@khalid-mahmood, yeah, you got it.
For the next releases I'm going to deprecate/rename that method in favour or addListener(). What do you think?

addListener() good for understanding.. consider it.


[UT] Add avro parsing unit tests

What type of PR is this：

[ ] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[x] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch

[x] 3.0
[ ] 2.5
[ ] 2.4
[ ] 2.3

run starrocks_be_unittest

run starrocks_be_unittest

run starrocks_be_unittest

run starrocks_be_unittest

run starrocks_admit_test

@Mergifyio backport branch-3.0


Wrong Behavior with NerdTree

environment

vim: neovim
vim-airline: eed488b(the newest in plug maneger)
OS: Ubuntu20.04
Have you reproduced with a minimal vimrc: yes
What is your airline configuration: ???
if you are using terminal:
terminal: zsh
$TERM variable: ???
color configuration (:set t_Co?):
if you are using Neovim:
does it happen in Vim: not tested

actual behavior
Use Tab to switch in NerdTree window, it'll cover the NerdTree.
expected behavior
Not hurt the NerdTree window.
screen shot (if possible)

After Tab on NerdTree window, it becomes like this:

does that fix it:
https://github.com/vim-airline/vim-airline/blob/24cb98f48fbe5ba7a64d1099d334430a5937a175/doc/airline.txt#L280-L282

any update?


requirements.txt not installing using Python 3.8 and 3.9

Create virtualenvs for Python 3.8.13 and 3.9.11 I used pyenv for that) and activate them
In each virtualenv run pip install -r requirements.txt

The following error is returned by pip:
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typing-extensions>=<IP_ADDRESS> from https://files.pythonhosted.org/packages/ed/d6/2afc375a8d55b8be879d6b4986d4f69f01115e795e36827fd3a40166028b/typing_extensions-4.3.0-py3-none-any.whl#sha256=25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02 (from black==22.1.0->-r requirements.txt (line 21))

It worked fine with 3.10.6

Is this the template requirements or the generated project requirements?

This one:
https://github.com/sixfeetup/cookiecutter-sixiedjango/blob/main/requirements.txt

@rennerocha can you test the associated branch?

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Missing documentation on disabling Large Neighbourhood Search

I started using or-tools back when the CapacitatedVehicleRoutingProblemWithTimeWindows Java example used setNo_lns(true) on RoutingSearchParameters. As far as I can tell this option has been removed, but I can't see any documentation on what the replacement is.
Now I'm setting parameters like so:
RoutingSearchParameters parameters = RoutingSearchParameters.newBuilder()
      .mergeFrom(RoutingModel.defaultSearchParameters())
      .build();

I can't find any documentation for the options in Python either, however I think the replacement in Python is shown in #235. Perhaps this helps?

Hello @andygeers,
I tried your code, and it works. Here is another way to do it :
RoutingSearchParameters.LocalSearchNeighborhoodOperators local_search_operators =
 RoutingSearchParameters
 .LocalSearchNeighborhoodOperators
 .newBuilder()
 .setUsePathLns(true)
 .build();

RoutingSearchParameters search_parameters =
   RoutingSearchParameters.newBuilder()
   .mergeFrom(RoutingModel.defaultSearchParameters())
   .setFirstSolutionStrategy(FirstSolutionStrategy.Value.PATH_CHEAPEST_ARC)
   .setLocalSearchOperators(local_search_operators)
   .build();

and to get the parameters values, you can try this :
System.out.println("use_path_lns : " + search_parameters.getLocalSearchOperators().getUsePathLns());
I hope this helps.
Regards,
Driss


Added new message for notify the user copy logo from old installation.

Yes, that makes sense, thanks a lot!


Basic webserver timeline

Made basic verson of the webserver timeline

Looks fine, and all commentary is fixed.

Looks good!


Build lambda layer for ARM64 architecture

Do you have any suggestion on how the Makefile could be updated to build the Lambda Layer for Amazon's ARM64 architecture? I tried changing these lines (as advised on sharp's docs):
- LD_LIBRARY_PATH=$(PREFIX_PATH)/lib npm --prefix "$(ARTIFACTS_DIR)/nodejs/" install sharp@$(SHARP_VERSION)
+ SHARP_IGNORE_GLOBAL_LIBVIPS=1 LD_LIBRARY_PATH=$(PREFIX_PATH)/lib npm --prefix "$(ARTIFACTS_DIR)/nodejs/" install --arch=arm64 --platform=linux sharp@$(SHARP_VERSION)
- ldd $(ARTIFACTS_DIR)/nodejs/node_modules/sharp/build/Release/sharp-linux-x64.node | sed -nE "s/^[^\/\n]*(\/[^ ]+)(.*)/\1/p" | xargs cp -t $(ARTIFACTS_DIR)/lib
+ ldd $(ARTIFACTS_DIR)/nodejs/node_modules/sharp/build/Release/sharp-linux-arm64v8.node | sed -nE "s/^[^\/\n]*(\/[^ ]+)(.*)/\1/p" | xargs cp -t $(ARTIFACTS_DIR)/lib

The ldd command is now returning an error that sharp-linux-arm64v8.node is not a dynamic executable, which I suspect is due to the fact it was build for a different architecture than the one it is currently running on. I would be curious if anyone tried this and if you have any suggestion on how that could be solved. Thanks!

I'm not using arm64 architecture myself yet, so can't comment on this


unable to download

502 error on website

This should be fixed now. Try downloading Kite from https://kite.com/download.


Allow up to 32 player for e.g. Sealed tournaments

For draft it shouldn't matter as it's mostly always max. 8 players.
But Sealed can easily support like 20-30 players for a tournament.
Is there a particular reason to limit the max players to 16 only? Any concerns on performance, user managmenet or server load?

No good reason :) Removed the limit here 5b26fccf6d607321fd53584f434bef85a393889f


cross chain token list not working

Environment information

Python Version: 3.9
OS: macos

Python environment
$ pip freeze
# ...copy and paste result of above command here...

What went wrong?
Please include information like:
Tried to load this token lists: https://github.com/Uniswap/token-lists/blob/main/test/schema/example-crosschain.tokenlist.json
but it fails with ValidationError about extensions since it is dict and not one of the expected types
How can it be fixed?
Handle multi-chain bridge like extensions data structure (dictionary of sorts)

Oops duplicate of #20


Replace multiple applications of single dispatch with multimethod multiple dispatch

Prior to this small refactoring PR, dispatching on both the type of the dynamical system and the type of the solver involved chaining two functools.singledispatchmethod definitions in which the order of arguments (and thus the particular variable being dispatched on) was shuffled around. Instead, this PR has each backend interface method simulate, simulate_trajectory, and get_next_interruptions_dynamic dispatch on both dynamical system and solver type jointly. This substantially reduces indirection, and the development burden when implementing a new backend or dynamical system formalism.

I sympathize with the goal of this PR, but I'm skeptical about using multimethod this way and a number of the other changes here. Maybe a simpler thing to do would be to stop distinguishing between subtypes of the Dynamics interface (e.g. ODEDynamics), and only dispatch on the solver.

I actually quite like the idea of dispatching on both the type of the dynamical system and the solver, as this clearly communicates to the user what restrictions we impose/assume about the dynamical system. While we don't impose any explicit restrictions on what's inside the diff method of the ODEDynamics class now, we could, and we probably should soon.

I think dynamical system types like ODEDynamics are not fundamental features of the design, but rather heuristics standing in for restrictions imposed by particular families of solvers, so we shouldn't bake them into the semantics of simulate.
I prefer having Dynamics be a Protocol rather than a concrete Generic base type, because that constrains our code to have to work entirely in terms of that extremely simple interface.
I don't think it makes sense to dispatch on all arguments of these operations or enable bigger changes to their signatures like adding or removing arguments.
I would prefer not to add new mandatory installation dependencies unless we have to. Once #202 is addressed I can be a little more lax about this.
I'm generally not crazy about multiple dispatch as a design pattern in Python because it's slow and doesn't play nicely with the type system (both in the sense that the runtime dispatch mechanism doesn't work properly with generic types and in the sense that it may throw away generic and parametrized types that could be used by static type checkers or interfere with the use of generic function types elsewhere). There are some circumstances where it does makes sense, but Python isn't Julia and it should be used with care.

I think dynamical system types like ODEDynamics are not fundamental features of the design, but rather heuristics standing in for restrictions imposed by particular families of solvers, so we shouldn't bake them into the semantics of simulate.

I generally agree with this point. Does this mean that onus is on the user to know what the restrictions are on any particular solver? Alternatively, are there static or runtime checks that each solver can implement that enforce their restrictions? I'm very nervous about a user e.g. adding pyro.sample statements in the body of a diff method and then using the TorchDiffEq solver.

I prefer having Dynamics be a Protocol rather than a concrete Generic base type, because that constrains our code to have to work entirely in terms of that extremely simple interface.

I see. I think I need to read up on the distinction here.

I don't think it makes sense to dispatch on all arguments of these operations or enable bigger changes to their signatures like adding or removing arguments.

This could be addressed by passing in object explicitly in the register decorator.

I would prefer not to add new mandatory installation dependencies unless we have to. Once Each module should have its own installation requirements #202 is addressed I can be a little more lax about this.

Fair. It is a bit sad to have this be the only other dependency besides Pyro, and it's not obvious how stable it is.

I'm generally not crazy about multiple dispatch as a design pattern in Python because it's slow and doesn't play nicely with the type system (both in the sense that the runtime dispatch mechanism doesn't work properly with generic types and in the sense that it may throw away generic and parametrized types that could be used by static type checkers or interfere with the use of generic function types elsewhere). There are some circumstances where it does makes sense, but Python isn't Julia and it should be used with care.

Also fair. I've experienced this a bit already (see above workaround to your question about register.

Does this mean that onus is on the user to know what the restrictions are on any particular solver?

Keep in mind that this is currently the case - ODEDynamics doesn't actually enforce any restrictions on the contents of diff, they're just implied by the name.

are there static or runtime checks that each solver can implement that enforce their restrictions? I'm very nervous about a user e.g. adding pyro.sample statements in the body of a diff method and then using the TorchDiffEq solver.

We could implement simple validation effect handlers that get applied to diff and raise errors at runtime when certain operations are encountered. A more powerful type system would let us enforce those restrictions statically, but I don't think that's currently doable with mypy and Pyro's handlers' janky string-based dynamic dispatch mechanism.

This could be addressed by passing in object explicitly in the register decorator.

Hmm, no, my point was that someone could write a signature with a finer type than object for those other arguments like timespan, which is a form of extensibility I don't think we should support.

Closing this PR in favor of an alternative that removes ODEDynamics. Currently a WIP on my local machine dealing with circular imports, but will create a PR soon.


[Contextual] Implement internal context provider

JIRA: EE-1315

Do you mind adding few tests on this ?
Value is quite difficult to test, but you can at least check some properties:

country/language is not null and has length = 2 or 3
screen size > 0
orientation is either "Portrait" or "Landscape"
network is something and does not throw
model name is not empty

This is mostly testing Apple's iOS APIs, I'm not convinced of the value of such tests, but let's do it ¯_(ツ)_/¯

Are you planning to add some tests in this PR?

Even if most of this code calls Apple APIs, it is still worth (in some cases) adding some tests. At least, these tests verify that the right APIs are being called. My rule of thumb when testing code that merely delegates to SDK APIs is to see whether the test is: if it's a low hanging fruit then I write it.
Also there are examples where some methods do have a bit of logic. For example: `getPreferredLanguagesComponentForLocaleKey: https://github.com/criteo/ios-publisher-sdk/pull/154/files#diff-getPreferredLanguagesComponentForLocaleKey b138a9820371d3ed468d90ab28b8d29497a71438e494ebd6ea81e56f00b72529R179

but let's do it ¯_(ツ)_/¯
Done.


Add a halted lifecycle event

Feature Request
Is your feature request related to a problem?
Currently clients cannot react to halted noop reflexes.
Describe the solution you'd like
When a reflex callback chain is halted, the client is not made aware of this condition. This is because the lifecycle currently only supports:

before
success
error
after

This proposal would introduce a halted lifecycle event.

before
success
error
halted 👈
after

Adding a halted lifecycle event would allow clients to handle things properly and cleanup any UI tweaks made in the before event.

OMG I just spent half an hour trying to find the PR where you added the halted client-side callbacks 🤯 lol

@leastbad Not sure if you found it, but here it is. https://github.com/hopsoft/stimulus_reflex/pull/160/files#diff-bf91f9c76097698b1ca61d4d3f3cab08R39

That’s the server implementation. I’m talking about the client side; your issue is a proposal, right?

@leastbad Yes this is a feature proposal.


[FLINK-14600][runtime] Change Type of Field verticesFinished from AtomicInteger to int

What is the purpose of the change
This pull request degenerates the current AtomicInteger type of verticesFinished to a normal int type
Brief change log

Degenerate the current AtomicInteger type of verticesFinished to a normal int type

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@GJL Thanks for your suggestion, I have updated this PR.

Thanks for your contribution. Merging.