Migrating from dask-ml to xgboost.dask

hi guys,
we are planning to migrate from dask-ml's xgboost interface to xgboost.dask.
We extensively use Scikit pipelines and Gridsearchcv. dask-ml's internal datastructures dont support this and throw an error.
a trivial example that gives an error in dask-ml is
import dask.dataframe as dd
import numpy as np
import pandas as pd
from dask_ml.model_selection import GridSearchCV
from dask_ml.xgboost import XGBClassifier
from distributed import Client
from sklearn.datasets import load_iris

if __name__ == '__main__':

    client = Client()

    data = load_iris()

    x = pd.DataFrame(data=data['data'], columns=data['feature_names'])
    x = dd.from_pandas(x, npartitions=2)

    y = pd.Series(data['target'])
    y = dd.from_pandas(y, npartitions=2)

    estimator = XGBClassifier(objective='multi:softmax', num_class=4)
    grid_search = GridSearchCV(
        estimator,
        param_grid={
            'n_estimators': np.arange(15, 105, 15)
        },
        scheduler='threads'
    )

    grid_search.fit(x, y)
    results = pd.DataFrame(grid_search.cv_results_)
    print(results.to_string())

we are unable to get this to run in xgboost.dask . Could someone help out on the right example here ?
I'm particularly talking about https://github.com/dmlc/xgboost/blob/master/demo/dask/sklearn_cpu_training.py

Yup.  Working with dask ml is one of the priority for XGBoost s dask interface.  Thanks for rasing the issue here.  I will look into it.

Some notes for the future:

It seems dask ml doesn't support dask dataframe or at least treat it specially: https://github.com/dask/dask-ml/blob/4c52e2d23575470c383cb208f650eb59913283fa/dask_ml/model_selection/utils.py#L87
When using da.from_array, the data passed to XGBoost is a numpy array instead of a dask array.

I have a currently open bug tracking this issue
https://github.com/dask/dask-ml/issues/521
On Sat, 22 Feb, 2020, 21:47 Jiaming Yuan<EMAIL_ADDRESS>wrote:

Some notes for the future:

It seems dask ml doesn't support dask dataframe or at least treat it
specially:
https://github.com/dask/dask-ml/blob/4c52e2d23575470c383cb208f650eb59913283fa/dask_ml/model_selection/utils.py#L87
When using da.from_array, the data passed to XGBoost is a numpy
array instead of a dask array.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYU2LBO4KDSIQ5OFI7MDREFFYBA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVD63A#issuecomment-589971308,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU3YETXK74ZHLKGZ67DREFFYBANCNFSM4KZSHRAQ
.

Yup. Working with dask ml is one of the priority for XGBoost s dask interface. Thanks for rasing the issue here. I will look into it.

@trivialfis could you clarify how do you see the long term evolution here - does xgboost.dask become the official interface or dask-ml ? we would prefer to follow the official xgboost stand here.

The XGBoost interface will take over.  We will try to be compatible with other dask ml functionalities as much as possible, just like being compatible with scikit learn.

Acknowledged and thanks for this.
We will then move to the Dask interface.
Is the Xgboost.dask interface already compatible with scikit ? We have not
been able to make it work with gridsearchcv in a distributed way.
On Sun, 23 Feb, 2020, 12:16 Jiaming Yuan<EMAIL_ADDRESS>wrote:

The XGBoost interface will take over. We will try to be compatible with
other dask ml functionalities as much as possible, just like being
compatible with scikit learn.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYUYV6O7Q2KGPXHWUNGLREILV3A5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVTCPA#issuecomment-590033212,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU4HXCVROSGDJWA4DH3REILV3ANCNFSM4KZSHRAQ
.

Sorry for the ambiguity, when I said scikit learn I meant single node training.  For distributed interface we will target dask-ml.

Hi
Bit confused here - I'm specifically asking in context of Dask
interoperability.
Whether I am running Dask on one machine vs hundred shouldn't make a
difference right ?
We would like to use the same code for single machine (for e.g. my laptop
which has a single Dask node) and production (which has hundred Dask nodes)
It will be unproductive/error prone to use different libraries for
different number of machines.
On Sun, 23 Feb, 2020, 12:26 Jiaming Yuan<EMAIL_ADDRESS>wrote:

Sorry for the ambiguity, when I said scikit learn I meant single node
training. For distributed interface we will target dask-ml.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYUZYCOX72GJ6VGGIKN3REIM2XA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVTGVA#issuecomment-590033748,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU3JR7LXD4QZ2HA62ZTREIM2XANCNFSM4KZSHRAQ
.

Yup.  The dask interface should be compatible with dask ml.

Apologise for asking again - will the Xgboost.dask interface support both
single machine Dask and distributed Dask training usecases ?
Because dask-ml does that (minus a few bugs).
The goal is to either use xgboost.dask or dask-ml.
It will not be viable to use both.
On Sun, 23 Feb, 2020, 13:12 Jiaming Yuan<EMAIL_ADDRESS>wrote:

Yup. The dask interface should be compatible with dask ml.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYU4RI3T3R6ZN3IS4LYLREISFZA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVT3XI#issuecomment-590036445,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYUYJIE5QPCHRRZ2BN4LREISFZANCNFSM4KZSHRAQ
.

@sandys
There are two different components in dask-ml.  First is the Scikit-Learn basic functionality and its interface, like grid search.  And then the second part is library specific wrappers like wrapper for XGBoost.
The XGBoost dask interface is taking over from the original XGBoost wrapper in dask-ml (the second component).

will the Xgboost.dask interface support both single machine Dask and distributed Dask training usecases ?

Yes.  As long as a feature is supported by dask-ml, we stride to support it.

Hope that helps.

Understood. So if we want to use the gridsearchcv and scikit functionality,
we have to use via dask-ml.
Has the new interface already been patched into dask-ml ?
On Sun, 23 Feb, 2020, 14:02 Jiaming Yuan<EMAIL_ADDRESS>wrote:

Hope that helps.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYU4BWLD6NPOCZLBQCG3REIYANA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVV7WY#issuecomment-590045147,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU2S7VCQROQLNRACG33REIYANANCNFSM4KZSHRAQ
.

In a quick test, I noticed that the xgboost.XGBoostClassifer doesn't yet work with dask arrays.
In [1]: from distributed import Client

In [2]: import dask_ml.datasets

In [3]: import xgboost

In [4]: client = Client()

In [5]: X, y = dask_ml.datasets.make_classification(n_features=100, n_samples=100_000, chunks=10_000)

In [6]: clf = xgboost.XGBClassifier()

In [7]: clf.fit(X, y)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-b0953fbb1d6e> in <module>
----> 1 clf.fit(X, y)

~/miniconda3/envs/xgboost=1.0.1/lib/python3.8/site-packages/xgboost/sklearn.py in fit(self, X, y, sample_weight, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, callbacks)
    676         evals_result = {}
    677         self.classes_ = np.unique(y)
--> 678         self.n_classes_ = len(self.classes_)
    679
    680         xgb_options = self.get_xgb_params()

TypeError: 'float' object cannot be interpreted as an integer

That's because np.unique(dask.array.Array) returns a Dask array with unknown (NaN) length, so len raises.
Typically, scikit-learn APIs accept classes as a keyword argument to fit, but the version in xgboost doesn't.
In [8]: clf.fit(X, y, classes=[0, 1])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-15b6a6f7c523> in <module>
----> 1 clf.fit(X, y, classes=[0, 1])

TypeError: fit() got an unexpected keyword argument 'classes'

Use xgboost.dask instead.

Hi Tom
I'm referring to your work on https://github.com/dask/dask-ml/pull/612 as
well. This is very important to us.
Will it work with Xgboost 1.0 ? I'm referring to the comment you just made.
Not sure if there's something else that needs to happen
On Fri, 6 Mar, 2020, 05:47 Tom Augspurger<EMAIL_ADDRESS>wrote:

Sorry, I was accidentally on xgboost 0.9. Haven't tried with 1.0 yet.
On Mon, Mar 2, 2020 at 2:01 PM Jiaming Yuan<EMAIL_ADDRESS>wrote:

Use xgboost.dask instead.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAKAOIVCFC6XXPLT3PR5FBLRFQGBPA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENQXPTY#issuecomment-593590223
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAKAOIUYWVPNRZQ5OHOOQFLRFQGBPANCNFSM4KZSHRAQ
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYUYEF5EAPUELFHI3X4TRGA2PHA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7DYNA#issuecomment-595475508,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU6XK45THDSZQAMRO3DRGA2PHANCNFSM4KZSHRAQ
.

I'm not sure either, but it seems like it should be OK. Can you test and
see?
On Thu, Mar 5, 2020 at 8:43 PM Sandeep Srinivasa<EMAIL_ADDRESS>wrote:

Hi Tom
I'm referring to your work on https://github.com/dask/dask-ml/pull/612 as
well. This is very important to us.
Will it work with Xgboost 1.0 ? I'm referring to the comment you just made.
Not sure if there's something else that needs to happen
On Fri, 6 Mar, 2020, 05:47 Tom Augspurger<EMAIL_ADDRESS>wrote:

Sorry, I was accidentally on xgboost 0.9. Haven't tried with 1.0 yet.
On Mon, Mar 2, 2020 at 2:01 PM Jiaming Yuan<EMAIL_ADDRESS>wrote:

Use xgboost.dask instead.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<

https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAKAOIVCFC6XXPLT3PR5FBLRFQGBPA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENQXPTY#issuecomment-593590223

,
or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AAKAOIUYWVPNRZQ5OHOOQFLRFQGBPANCNFSM4KZSHRAQ

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYUYEF5EAPUELFHI3X4TRGA2PHA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7DYNA#issuecomment-595475508
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAASYU6XK45THDSZQAMRO3DRGA2PHANCNFSM4KZSHRAQ
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAKAOIV73R2JPZJBV62ZF4LRGBPMRA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7254A#issuecomment-595570416,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKAOIVZFXBPOIAQOINVHL3RGBPMRANCNFSM4KZSHRAQ
.

@TomAugspurger this throws error
import xgboost  import dask_ml.datasets import dask.dataframe as dd import dask.array as da import numpy as np from dask_ml.model_selection import GridSearchCV from dask.distributed import Clientclient = Client() X, y = dask_ml.datasets.make_classification(n_features=100, n_samples=100_000, chunks=10_000) parameters = {'eta': [0.1, 0.2], 'max_depth': [6, 7]} xgb = xgboost.dask.DaskXGBClassifier() clf = GridSearchCV(xgb, parameters, scheduler='threads', cache_cv=False) clf.fit(X,y)

TypeError: Expecting <class 'dask.dataframe.core.DataFrame'> or <class 'dask.array.core.Array'>.  Got <class 'numpy.ndarray'>

Thanks. Can you debug what's going on?
On Mon, Mar 9, 2020 at 10:16 AM Sandeep Srinivasa<EMAIL_ADDRESS>wrote:

@TomAugspurger https://github.com/TomAugspurger this throws error
import xgboost import dask_ml.datasets import dask.dataframe as dd import
dask.array as da import numpy as np from dask_ml.model_selection import
GridSearchCV from dask.distributed import Clientclient = Client() X, y =
dask_ml.datasets.make_classification(n_features=100, n_samples=100_000,
chunks=10_000) parameters = {'eta': [0.1, 0.2], 'max_depth': [6, 7]} xgb =
xgboost.dask.DaskXGBClassifier() clf = GridSearchCV(xgb, parameters,
scheduler='threads', cache_cv=False) clf.fit(X,y)
TypeError: Expecting <class 'dask.dataframe.core.DataFrame'> or <class
'dask.array.core.Array'>. Got <class 'numpy.ndarray'>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAKAOIWD7HESDHD4RIEQ2MLRGUB4XA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOHUUPI#issuecomment-596593213,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKAOIS46TXBFMNDKV4E7I3RGUB4XANCNFSM4KZSHRAQ
.

The gridsearchcv in Dask-ml expects a different kind of input.
This works with sklearn gridsearchcv. But I'm not sure if it distributes.
On Mon, 9 Mar, 2020, 20:49 Tom Augspurger<EMAIL_ADDRESS>wrote:

Thanks. Can you debug what's going on?
On Mon, Mar 9, 2020 at 10:16 AM Sandeep Srinivasa <
<EMAIL_ADDRESS>wrote:

@TomAugspurger https://github.com/TomAugspurger this throws error
import xgboost import dask_ml.datasets import dask.dataframe as dd import
dask.array as da import numpy as np from dask_ml.model_selection import
GridSearchCV from dask.distributed import Clientclient = Client() X, y =
dask_ml.datasets.make_classification(n_features=100, n_samples=100_000,
chunks=10_000) parameters = {'eta': [0.1, 0.2], 'max_depth': [6, 7]} xgb
=
xgboost.dask.DaskXGBClassifier() clf = GridSearchCV(xgb, parameters,
scheduler='threads', cache_cv=False) clf.fit(X,y)
TypeError: Expecting <class 'dask.dataframe.core.DataFrame'> or <class
'dask.array.core.Array'>. Got <class 'numpy.ndarray'>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAKAOIWD7HESDHD4RIEQ2MLRGUB4XA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOHUUPI#issuecomment-596593213
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAKAOIS46TXBFMNDKV4E7I3RGUB4XANCNFSM4KZSHRAQ
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347?email_source=notifications&email_token=AAASYUZUKQOHZ6DU6RE3223RGUCGLA5CNFSM4KZSHRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOHU7OA#issuecomment-596594616,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU5Y5FXEDK7CVM3IJHLRGUCGLANCNFSM4KZSHRAQ
.

import xgboost
import dask_ml.datasets
import dask.dataframe as dd
import dask.array as da
import numpy as np
from dask_ml.model_selection import GridSearchCV
from dask.distributed import Clientclient = Client(n_workers=3, threads_per_worker=1)
clientX, y = dask_ml.datasets.make_classification(n_features=100, n_samples=100_000, chunks=10_000)
parameters = {'eta': [0.1, 0.2], 'max_depth': [6, 7]}
xgb = xgboost.XGBClassifier()
clf = GridSearchCV(xgb, parameters, scheduler='distributed')
clf.cv_results_

This kinda works.
It fails if I change XGBClassifier to DaskXGBClassifier.
WARNING:dask_ml.model_selection._search:('daskxgbclassifier-fit-score-0eac988fcd50f1b966753744ffa7057c', 0, 0) has failed... retrying
WARNING:dask_ml.model_selection._search:('daskxgbclassifier-fit-score-0eac988fcd50f1b966753744ffa7057c', 1, 0) has failed... retrying

It keeps failing and retrying.

There are some conflicts here.  Dask ml is designed to scale up single node libraries like scikit learn, but XGBoost dask itself is a distributed library.

Thanks for replying.
What we are looking for is Xgboost in distributed mode (Dask) being able to
work with scikit libraries like gridsearchcv (which should themselves be
distributed)
If we use  DaskXGBClassifier , then how do I get it work with gridsearchcv
in distributed mode ?
A bit confused on how to get the entire flow to be distributed.
Thanks .
On Wed, 11 Mar, 2020, 18:09 Jiaming Yuan<EMAIL_ADDRESS>wrote:

There are some conflicts here. Dask ml is designed to scale up single node
libraries like scikit learn, but XGBoost dask itself is a distributed
library.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347#issuecomment-597609051, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU7DKQ2IWF5VF5O7F63RG6BBVANCNFSM4KZSHRAQ
.

So I have also run into an issue when using DaskXGBoostClassifier. Here is a minimal example based on the dask example in the repo:
from sklearn import datasets
import pandas as pd
import dask.dataframe as dd
import xgboost
from dask.distributed import Client, LocalCluster

def main(client):
    n_features = 13
    random_state = 75
    n_samples = 100000
    n_classes = 2
    n_informative = 3
    X, y = datasets.make_classification(
        n_features=n_features,
        n_redundant=3,
        n_repeated=0,
        n_informative=n_informative,
        flip_y=0,
        n_clusters_per_class=2,
        n_classes=n_classes,
        n_samples=n_samples,
        random_state=random_state
    )
    train_data = pd.DataFrame(X)
    train_data["target"] = y
    train_data.columns = train_data.columns.astype(str)
    features = [column for column in train_data.columns if column != "target"]

    train_dask = dd.from_pandas(train_data, chunksize=10000)
    X = train_dask[features]
    y = train_dask["target"]

    model = xgboost.dask.DaskXGBClassifier(client)
    model.fit(X, y, eval_set=[(X, y)])           # returning the trained model
    print(model._Booster.feature_names)

if __name__ == '__main__':
    # or use other clusters for scaling
    with LocalCluster(n_workers=4, threads_per_worker=1) as cluster:
        with Client(cluster) as client:
            main(client)

It generates a pretty long error trace:
Traceback (most recent call last):
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/worker.py", line 3277, in dumps_function
    result = cache_dumps[func]
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/utils.py", line 1518, in __getitem__
    value = super().__getitem__(key)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/collections/__init__.py", line 1027, in __getitem__
    raise KeyError(key)
KeyError: <function train.<locals>.dispatched_train at 0x1265d5680>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 38, in dumps
    result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
AttributeError: Can't pickle local object 'train.<locals>.dispatched_train'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dask_test.py", line 73, in <module>
    main(client)
  File "dask_test.py", line 65, in main
    model.fit(X, y, eval_set=[(X, y)])           # returning the trained model
  File "/Users/jborchma/github/xgboost/python-package/xgboost/dask.py", line 653, in fit
    evals=evals)
  File "/Users/jborchma/github/xgboost/python-package/xgboost/dask.py", line 418, in train
    workers=workers)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/client.py", line 1698, in map
    actors=actor,
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/client.py", line 2511, in _graph_to_futures
    "tasks": valmap(dumps_task, dsk3),
  File "cytoolz/dicttoolz.pyx", line 179, in cytoolz.dicttoolz.valmap
  File "cytoolz/dicttoolz.pyx", line 204, in cytoolz.dicttoolz.valmap
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/worker.py", line 3315, in dumps_task
    return {"function": dumps_function(task[0]), "args": warn_dumps(task[1:])}
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/worker.py", line 3279, in dumps_function
    result = pickle.dumps(func)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 51, in dumps
    return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 1148, in dumps
    cp.dump(obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 491, in dump
    return Pickler.dump(self, obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 566, in save_function
    return self.save_function_tuple(obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 780, in save_function_tuple
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 656, in save_reduce
    self._batch_appends(listitems)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 566, in save_function
    return self.save_function_tuple(obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 780, in save_function_tuple
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/jborchma/miniconda3/envs/xgboost/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle generator objects

Let me take a look.

@jborchma  Please remove client as the parameter.

@sandys  Sorry for the confusion here.  It's something I need to address.  I have been syncing up with the dask ml recently.  Will keep you posted here.

@trivialfis thank you! i will be standing by to run any tests, etc that you need.  Being able to run a whole distributed scikit pipeline is of high degree of importance to us.

@jborchma Please remove client as the parameter from DaskXGBClassifier.

Oh wow, thanks for the quick fix. My error!

@sandy You can use scikit-learn gridsearch with xgboost-dask now.

Hi
Thanks!
Are you referring to dask-ml gridsearchcv ?
Any chance of a sample code, because not very clear on the structure.
On Tue, 17 Mar, 2020, 14:27 Jiaming Yuan<EMAIL_ADDRESS>wrote:

@Sandy https://github.com/Sandy You can use scikit-learn gridsearch
with xgboost-dask now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347#issuecomment-599953981, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU7I5J7YB6W36YNMP43RH43OTANCNFSM4KZSHRAQ
.

@sandys No I'm referring to scikit-learn.  dask-ml is used to scale up single node algorithms (AFAIK), XGBoost itself can distribute data among workers and do the synchronization.  Eventually I want to have both models of distributed pipeline working in one interface, but so far we have 2 of them.
For using scikit-learn(single-node) grid searching with xgboost.dask(distributed):
    from sklearn.model_selection import GridSearchCV
    with LocalCluster(n_workers=4) as cluster:
        with Client(cluster) as client:
            X, y = generate_array()
            reg = xgb.dask.DaskXGBRegressor(learning_rate=0.1,
                                            tree_method='hist')
            reg.client = client
            model = GridSearchCV(reg, {'max_depth': [2, 4],
                                       'n_estimators': [5, 10]},
                                 cv=2, verbose=1, iid=True)
            model.fit(X, y)

Another combination is dask-ml(distributed) and xgboost.sklearn(single-node), which I'm not familiar yet.
You can see the conflict we are trying to resolve: only one of them can be distributed.

Thanks for the reply.
I had one doubt. In your "For using scikit-learn(single-node) grid
searching with xgboost.dask(distributed) " code example...will the
gridsearchcv be distributed on the Dask nodes ?
Or will it run in a single node only.
On Tue, 17 Mar, 2020, 17:29 Jiaming Yuan<EMAIL_ADDRESS>wrote:

@sandys https://github.com/sandys No I'm referring to scikit-learn.
dask-ml is used to scale up single node algorithms (AFAIK), XGBoost itself
can distribute data among workers and do the synchronization. Eventually I
want to have both models of distributed pipeline working in one interface,
but so far we have 2 of them.
For using scikit-learn(single-node) grid searching with xgboost.dask
(distributed):
from sklearn.model_selection import GridSearchCV
with LocalCluster(n_workers=4) as cluster:
    with Client(cluster) as client:
        X, y = generate_array()
        reg = xgb.dask.DaskXGBRegressor(learning_rate=0.1,
                                        tree_method='hist')
        reg.client = client
        model = GridSearchCV(reg, {'max_depth': [2, 4],
                                   'n_estimators': [5, 10]},
                             cv=2, verbose=1, iid=True)
        model.fit(X, y)

Another combination is dask-ml(distributed) and xgboost.sklearn(single-node),
which I'm not familiar yet.
You can see the conflict we are trying to resolve: only one of them can be
distributed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/5347#issuecomment-600031975, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU5P7VHHLTC3VYSHUCDRH5Q25ANCNFSM4KZSHRAQ
.

Yes.  The data is distributed across workers and XGBoost syncs the gradient between workers during training.

Prefer https://github.com/dmlc/xgboost/issues/6525 instead.


assert: Add screenshot example of colored diff output

And link to diff package.

https://github.com/terrastruct/utils-go/issues/2 is better


Move colorblind simulation out of a plugin and into the a11y inspect tool

consider using arrows and a special hotkey to toggle through the options. would be nice to see a toast or overlay of some kind specifying the current colorblind mode being simulated. could even just reuse the select box from the plugin.

consider adding much more to the a11y inspect tool https://accessgarage.wordpress.com/

consider a simulation tool so we dont overload the a11y tool?


Rico Corner removal branch

This passes all tests that were previously passing; I have several unrelated fails on my box, for whatever reason, in unrelated code. Basically removes rounded corners, Rico.Corner, and the options for them.

I  agree that LayerSwitcher.js and AnchoredBubble.js should not @require Rico.js.  So, for example, for LayerSwitcher, Rico.Corner.round could be called only in this.roundedCorner is set and if OpenLayers.Rico is defined. With this LayerSwitcher and AnchoredBubble users wouldn't have to have Rico in their OpenLayers builds, and they could use CSS3 to round corners. And we woudn't break existing apps, to get the old behavior people would just need to explicitely add Rico.js to their builds. What do you think?
Out of curiosity, do you know any open-source polyfill for CSS3 border-radius?

I’m in favour of removing it completely and merge this in this state.
For a reminder of current browser support: http://www.caniuse.com/#search=border-radius

Yeah, much in favor of removing it completely and having users go to CSS3 PIE or other options if they insist on IE+corners.

I'm pretty sure people will complain if you don't provide an easy way to get rounded corners back in IE. PIE may be that easy way, but this is to be verified. I'm not fully sure but it looks like PIE injects VML, I hope this plays well with vector layers for example.
But I understand the benefits of removing Rico entirely - we load compatibility/polyfill code only for less-capable browsers.

After more thinking I think we loose more than we gain by removing Rico completely for now. We loose because people will complain that rounded corners have disappeared in their apps in IE, and I tend to think that some of them won't see PIE (or similar) as a straightforward solution to their problems.
So I'm proposing to remove the @requires Rico/Corner.js directives from LayerSwitcher.js and AnchoredBubble.js, and to test that OpenLayers.Rico.Corner is defined before attempting to round corners with Rico. By doing this we indeed leave a few lines of Rico-related code in LayerSwitcher and AnchoredBubble, but I don't think that's an issue. I'm also proposing to add a rounded-corners.html example showing how to round corners with CSS3 and PIE or similar.
And yes, we'll need to encourage people to remove Rico from their apps, and rely on CSS3 for rounded corners. For that the API docs could discourage the use of Rico, and people could be directed to the rounded-corners.html example.
What do you think?

I'm willing to try out PIE to see whether it performs here. I'm kind of surprised that OL3 is going to keep backwards-compatibility; if that's really an objective, it really handicaps the library's ability to remove stuff that's legacy (like this) and change other APIs.

With #99 I think we can close this pull request. @tmcw, is that ok?

With #99 merged this one can be closed.


Install process extracting 7zip is single threaded

The install part of downloading mods, is slow which can lead to users getting confused its hung and maybe they close 7h whilst its still extracting.
I suggest making this multithreaded, preferably by detecting number of threads in system and using all of them or if not at least make it 4 threads.
This only affects 7zip compressed mods not mods distributed in iro format.

IRO format mods are not expected to the file system unless the user explicitly does it, however, I'm not sure if LZMA that 7zip uses supports multi-threaded extraction because even 7zip application does not do multi-threaded extraction, however, it might be worth looking at updating the progress bar for downloading so it changes to extracting and shows a progress bar.

I am not at home right now to check, but I am fairly sure the 7zip app (at least on windows) is multi thread extraction, I can extract even really big mods in under 10 seconds using the 7zip application, but its in many multiples slower when downloading 7zipped mods.

@chrcoluk is there any update to your investigation?

I just tested and 7zip extraction on cetra mains used all of my cpu threads, so yes its multi threaded.  The file is LZMA2 format.
I tried to find documentation to see if LZMA is thread restricted on decompression, but couldnt ifnd anything although did find this article which suggests its not restricted.  I will recompress cetra with LZMA and test that as well.
https://www.7-cpu.com

So we're using the LZMA implementation from a NuGet Package, I can't remember which one currently but will look when I get home tonight, so if you know of another implementation library that we can use that does do MultiThreaded LZMA we can look at switching over to it.

So we're using,
SharpCompress.Compressors.LZMA.LzmaStream
And the problem is IRO's use their own LZMA stream system, and this means it's not LZMA2 that 7Zip uses, you can read more details about why we can't do this here,
https://github.com/adamhathcock/sharpcompress/issues/190


feat: add 402 Payment Required status code

15478

 Quality gate: Passed
 Everything looks good
Details

 Quality gate: NA
Details

 Quality gate: NA
Details

 Quality gate: NA
Details

 Quality gate: NA
Details

 Quality gate: NA
Details


install incomplete

I gernerated a image exactly as the documentation said but it installed nothing in C:\BuildTools. Instead I found Micosoft Visual Studio within C:\Program Files (x86) ,while no windows 10 or windows 8.1 sdk could be get from Windows Kits.
I removed " --remove Microsoft.VisualStudio.Component.Windows10SDK.10240     --remove Microsoft.VisualStudio.Component.Windows10SDK.10586
--remove Microsoft.VisualStudio.Component.Windows10SDK.14393     --remove Microsoft.VisualStudio.Component.Windows81SDK" from the dockerfile and regenerated, it didn't work.
What should I do if I want to use windows sdk?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b91a4a43-5353-97cf-7532-7644aa3b497f
Version Independent ID: 858d9a5c-72f3-6a85-9270-0a0dd9e0438b
Content: Install Visual Studio Build Tools into a container
Content Source: docs/install/build-tools-container.md
Product: visual-studio-windows
Technology: vs-installation
GitHub Login: @jillre
Microsoft Alias: jillfra

Hi @chenpei2, we've updated the Build Tools documentation here. Please give the new instructions a try. We look forward to your feedback, and I'll be closing out this issue in the meantime.


keycloak logs and log level

hey, thanks for this plugin... I'm having trouble configuring a saml identity provider and would like to check the keycloak logs for hints whats going wrong. Is there any way to access them and/or influence the log level?

This command may help to debug issues:
ddev logs -f -s keycloak

How keycloak is started in he container:
start-dev --proxy edge --import-realm

IIRC, start-dev will make keycloak already more verbose.

thanks, that helps already
I don' think start-dev has any effect on logging, from https://www.keycloak.org/server/configuration

Development mode sets the following default configuration:
HTTP is enabled
Strict hostname resolution is disabled
Cache is set to local (No distributed cache mechanism used for high availability)
Theme-caching and template-caching is disabled

here is a reference how to specify log levels when starting:
https://www.keycloak.org/server/logging

bin/kc.[sh|bat] start --log-level=

maybe it's possible to pass this for the plugin?

I'm happy to change the startup command. Would be great if you could providde a PR with to command changed according to your need while keeping the options set in start-dev.


charts/* Update docs to reflect new install method

Description of the change
Update documentation to point to new install method through GH pages instead of OCI containers with new GH action releaser
Checklist

[ X] Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
[X ] Variables are documented in the values.yaml and added to the README.md usingreadme-generator-for-helm
[ X] Title of the pull request follows this pattern [charts/<name_of_the_chart>] Descriptive title

Tests failing due to TS config changes with fix waiting to be merged in #83.


Use reference for from element

The UIElement was only used by reference anyway.
Let me know if you have any suggestions

Thanks for your PR.
UIElement is a Windows COM interface, whose lifetime is maintained by reference counting. The clone() method of the COM interface only increments the interface count and does not produce a content copy of the object, so there is not much overhead.
Therefore, there is not much optimization effect on the object reference, and it may even lead to complexity in lifetime management.
I will add a from_ref() method in the next version, which internally implements a copy, for your convenience.


SSHLibrary Import Error

Whenever i try to add SSHLibrary in robotframework , it gives me an error. Although the SSHLibrary is pre-installed using pip install robotframework-sshlibrary

I also met this in ubuntu, it seemed that "pip install robotframework-sshlibrary" did not work well.
I fixed this after I installed paramiko and pycrypto.

@vanpire110 Even after installing paramiko and pycrypto modules explicitly i am facing the same error .
It is showing the unicodedecode error.

Does this happen also when using Robot Framework only or only when using RIDE? In the latter case this belongs to RIDE's tracker, not here.

I suggest that you can use "import SSHLibrary" in python shell, then you can get more detail logs, It seems that you did not install other libs that leads the sshlibrary import error.

These are the dependencies of ssh library (with install order...)
(useful if you have no direct internet to install... (without pip))
pycparser-2.17.tar.gz
enum34-1.1.6-py2-none-any.whl
cffi-1.10.0-cp27-cp27m-win_amd64.whl
ipaddress-1.0.18-py2-none-any.whl
idna-2.5-py2.py3-none-any.whl
asn1crypto-0.22.0-py2.py3-none-any.whl
cryptography-1.9-cp27-cp27m-win_amd64.whl
pyasn1-0.2.3-py2.py3-none-any.whl
paramiko-2.1.2-py2.py3-none-any.whl
robotframework-sshlibrary-2.1.3.tar.gz

Doesn't look like Robot issue.


The README claims validation_depth is set to 0 by default, but it isn't.

    (Aside: The README claims `validation_depth` is set to `0` by default, but in the screenshot above it's set to `10` and I haven't configured it).

Originally posted by @guihkx in https://github.com/mindsers/changelog-reader-action/issues/75#issuecomment-1315297600

The README needs to be updated to match the new state of things.
Since we now use validation_level to disable the validation feature, the default was change to 10 by default. The goal was to have only one attribute to change to activate the feature.

@all-contributors Let's add @guihkx for bugs.


IterableDataset breaks 1.1 compatibility

A recently introduced feature unfortunately breaks compability with Pytorch 1.1.0.
Describe the bug
IterableDataset support, introduced in issue 323, requires Pytorch 1.2.0+.
To Reproduce
In a python environment with Pytorch 1.1.0 do:
import pytorch_lightning
Expected behavior
Compatibility with Pytorch 1.1.0. I'm filing it as a bug report rather than a docs fix since the dependency on 1.2.0+ introduced by issue 323 doesn't seem to be intentional.

good catch. mind submitting a PR? was not the intention of that fix :)
We want to support 1.1

I have played with versions and it seems to me that it does not run even with 1.2.0; see #462

weird. let’s patch it and put a note on the readme to use <IP_ADDRESS> if you need pt 1.1 or 1.2 support?

you may rather change requirements.txt and after fix put it back...
I see that >=1.2.0 is there already for some time... 8-)

Is this already fixed? I see that torch 1.2.0 is already in requirements.txt ?

I think not yet...8-)

Do you suggest torch 1.3.0 ?

we need to support 1.2.0 :)

@Borda did we get this fixed? is PT 1.2 supported now? If not, what do we need to do?

I have not worked on it yet (bit busy with docs), hopefully this week...


String equivalence in yaml

Two different yaml strings should be treated identically in patterns.
For example, this pattern should match all 3 nodes in the yaml file:
language yaml

`'ubuntu-latest'`

Sample file:
steps:
    - runs_on: "ubuntu-latest"
    - runs_on: 'ubuntu-latest'
    - runs_on: ubuntu-latest

It looks like the problem is the underlying AST representations are different in the yaml grammar between single-quote and double-quote strings.

Hi @dekkku I've added some more details on the problem above.
In theory we already have some logic for normalizing "equivalence classes" like this: https://github.com/getgrit/gritql/blob/b526a07022ef8b5b2c1969763d11955d4b332c4c/crates/language/src/yaml.rs#L119C8-L133
Unfortunately it doesn't seem to be working as intended.

/bounty $50

/attempt #394

/attempt #394


Save value ids and names in dictionary

What we have
Input: Field names of currently uploaded scan report
What we need
Concept IDs for field names that have already been mapped
Could be a helpful endpoint:
Add a filter for content_type: 17 (values) or 15(fields)
Get all the fields with existing mappings
Get all values with existing mappings
example:
Get concept for ScanReportField with ids in field_list[]
api/scanreportconceptsfilter/object_id__in={field_list}&&content_type=15
1st Method:

Find the fields with the same name in the ScanReportField model
If fields have the same name use the id and search if a mapping exists in ScanReportConcept table(object_id=field_id)
If an entry is found then get the conceptID
Create a new ScanReportConcept entry for the new fields:
new_concept={  ...,
"concept": conceptID(step 3),
"object_id": field_id,
"content_type": 15,
}

2nd Method:

For all records in ScanReportConcept find the fields and their conceptIDs-> dict= field_id:concept
For each id (dict key) query ScanReportField table for the field names
We now have field ids: names: concepts
Create a new ScanReportConcept entry for the new field ids

@vpnu can this be closed now?

Yep all of this code is old


Make self-hosted telemetry consistent

#710  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Dataset download

Thanks for sharing code and concept it was great to me to study image process.
By the way, I can't download dataset at Baidu...
Is there any other download link?
Thanks.

Thanks for providing promptly.
By the way, I'm new people to study deep learning. Can I ask something personally? If you can share your prefer method whatever wechat or email to contact you, I'll contact.


'/usr/bin/docker' failed with exit code 1 without any useful info on trying to create PR

Hi,
I managed to set up the dependabot pipeline with private artifact feed based on the code in this issue here:
https://github.com/tinglesoftware/dependabot-azure-devops/issues/50

variables:
  - name: Repository
    value: "masstransit"
    #value: ${{ replace(variables['Build.DefinitionName'],'Dependabot - ','') }}
    # Important: Pipeline name must be "Dependabot - repository_name".

stages:
  - stage: CheckDependencies
    displayName: 'Check Dependencies'
    jobs:
      - job: Dependabot
        displayName: 'Dependabot'
        pool:
          # Only works with MacOS and Linux
          vmImage: 'ubuntu-latest'
        steps:
          - checkout: 'git://Org Shared/${{ variables.Repository }}'

          # Creates $(VSS_NUGET_ACCESSTOKEN) for Private feeds
          - task: NuGetAuthenticate@1
            displayName: 'NuGet Private Feed Authentication'

          - task: dependabot@1
            displayName: 'Run Dependabot'
            inputs:
              useConfigFile: false
              packageManager: 'nuget'
              directory: '/'
              openPullRequestsLimit: '10'
              setAutoComplete: true
              mergeStrategy: '1'
              gitHubAccessToken: '$(Dependabot.GitHubAccessToken)'
              azureDevOpsAccessToken: '$(System.AccessToken)'
              targetRepositoryName: '${{ variables.Repository }}'
              extraEnvironmentVariables: DEPENDABOT_EXTRA_CREDENTIALS=[{"type":"nuget_feed","token":"$(VSS_NUGET_ACCESSTOKEN)","url":"https://pkgs.dev.azure.com/Org-nl/_packaging/Org.PackageFeed/nuget/v3/index.json"}]

But now I'm stuck at the part where it's trying to create a PR, and the docker log doesn't provide any useful information.
I gave full permissions to the "build agent user" and "build agent collection" for the project, but I still get this error:
2022-11-04T09:39:06.5767675Z [command]/usr/bin/docker run --rm -i -e DEPENDABOT_PACKAGE_MANAGER=nuget -e DEPENDABOT_FAIL_ON_EXCEPTION=true -e DEPENDABOT_EXCLUDE_REQUIREMENTS_TO_UNLOCK= -e AZURE_PROTOCOL=https -e AZURE_HOSTNAME=dev.azure.com -e AZURE_ORGANIZATION=my-org -e AZURE_PROJECT=my-proj -e AZURE_REPOSITORY=messaging-masstransit -e AZURE_ACCESS_TOKEN=*** -e AZURE_SET_AUTO_COMPLETE=true -e AZURE_MERGE_STRATEGY=1 -e DEPENDABOT_DIRECTORY=/ -e DEPENDABOT_VERSIONING_STRATEGY=auto -e DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT=10 -e GITHUB_ACCESS_TOKEN=$(Dependabot.GitHubAccessToken) -e AZURE_AUTO_APPROVE_PR=false -e DEPENDABOT_EXTRA_CREDENTIALS=[{"type":"nuget_feed","token":***,"url":"https://pkgs.dev.azure.com/my-org/_packaging/Org.PackageFeed/nuget/v3/index.json"}] tingle/dependabot-azure-devops:0.9
2022-11-04T09:39:06.8307017Z Unable to find image 'tingle/dependabot-azure-devops:0.9' locally
2022-11-04T09:39:08.3388223Z 0.9: Pulling from tingle/dependabot-azure-devops
...
2022-11-04T09:40:27.1887541Z 6b7bfb231103: Pull complete
2022-11-04T09:40:27.1980272Z Status: Downloaded newer image for tingle/dependabot-azure-devops:0.9
2022-11-04T09:40:39.7464355Z /home/dependabot/dependabot-script/vendor/ruby/2.7.0/gems/octokit-5.4.0/lib/octokit/response/raise_error.rb:14:in `on_complete': GET https://api.github.com/repos/prometheus-net/prometheus-net/contents/: 401 - Bad credentials // See: https://docs.github.com/rest (Octokit::Unauthorized)
2022-11-04T09:40:39.7466667Z 	from /home/dependabot/dependabot-script/vendor/ruby/2.7.0/gems/faraday-2.5.2/lib/faraday/middleware.rb:18:in `block in call'
...
2022-11-04T09:40:39.7518531Z 	from ./update-script.rb:310:in `<main>'
2022-11-04T09:40:39.7756876Z GitHub access token has been provided.
2022-11-04T09:40:39.7758290Z Using hostname = 'dev.azure.com', protocol = 'https', port = '443'.
2022-11-04T09:40:39.7759697Z Using 'https://dev.azure.com:443/' as API endpoint
2022-11-04T09:40:39.7760557Z Fetching nuget dependency files for my-org/my-proj/_git/messaging-masstransit
2022-11-04T09:40:39.7761205Z Targeting 'default' branch under '/' directory
2022-11-04T09:40:39.7761747Z Using 'auto' requirements update strategy
2022-11-04T09:40:39.7762123Z Parsing dependencies information
2022-11-04T09:40:39.7767109Z Checking if prometheus-net.AspNetCore 6.0.0 needs updating
2022-11-04T09:40:39.7767503Z Requirements to unlock own
2022-11-04T09:40:39.7768374Z Updating prometheus-net.AspNetCore from 6.0.0 to 7.0.0
2022-11-04T09:40:39.9838716Z ##[error]The process '/usr/bin/docker' failed with exit code 1
2022-11-04T09:40:39.9860911Z ##[section]Finishing: Run Dependabot

Do you have any clue what I'm missing?

OK so the problem was that I specified a '$(Dependabot.GitHubAccessToken)' access token, I should have removed that part.


breakLine not behaving correctly when "align" property set in .addText?

I am parsing HTML strings with bold tags to make them appear in PowerPoint.
For some reason, the breakLine property does not work when the 'align' property is set in a call to .addText. Line breaks are introduced after every element of the text snippet array. Removing the 'align' property makes it work again, but the text is moved to the left as a result. (Trial and error from my side)
It is still a bit hard to find the issue myself, maybe it is somewhere around these lines in the screen shot.
Mystery 😄

Similar Issues: #369 #448 #460
Working on this issue for v3.3.0

Thank you!

@gitbrent have you started on this one at all? I did some local testing and t looks like two small changes resolve the issue, without any side effects in my decks. Haven't run through a test suite yet.
Modify (around line 2420):
else if (idx > 0 && (typeof textObj.options.bullet !== 'undefined' || typeof textObj.options.align !== 'undefined')) {
else if (idx > 0 && (typeof textObj.options.bullet !== 'undefined' )) {
The alignment seems to happen on the first alignment it hits though, so modifying to reverse the sequence (around line 2405):
textObj.options.align = textObj.options.align || opts.align;
to
textObj.options.align = opts.align || textObj.options.align;
Keeps it to the alignment set for the main object rather than an individual line within the text block.

@ReimaFrgos - I actually have not started as of yet, so thanks a ton for looking more closely at this.
I'll start on this very shortly and report back...

I've managed to fix this. Would you like a PR @gitbrent or are you on this?

The entire text generation method has been rewritten should be rock solid from here out.
Fixes: #369, #448, #460, #751, #772
Some collected test cases:
var pptx = new PptxGenJS();
var slide = pptx.addSlide();

slide.addText(
    [
        { text: "line 1: 1st word ", options: { fontSize: 24, color: "99ABCC" } },
        { text: "2nd word ", options: { fontSize: 36, color: "FFFF00" } },
        { text: "3rd word ", options: { fontSize: 48, color: "0088CC" } },
    ],
    { x: 1.5, y: 0.3, w: 7, h: 1, margin: 0.1, fill: "232323" }
);

slide.addText(
    [
        { text: "line 1: 1st word ", options: { color: "99ABCC", italic: true } },
        { text: "2nd word ", options: { color: "FFFF00", bold: true } },
        { text: "3rd word ", options: { color: "0088CC" } },
    ],
    { x: 1.5, y: 1.6, w: 7, h: 1, margin: 0.1, fill: "232323", align: "center" }
);

slide.addText(
    [
        { text: "line 1: 1st word ", options: { color: "99ABCC", align: "center" } },
        { text: "2nd word ", options: { color: "FFFF00", align: "center" } },
        { text: "3rd word ", options: { color: "0088CC", align: "center" } },
    ],
    { x: 1.5, y: 2.9, w: 7, h: 1, margin: 0.1, fill: "336699" }
);

slide.addText(
    [
        { text: "line 1: 1st word ", options: { color: "99ABCC", align: "left" } },
        { text: "line 2: 1st word ", options: { color: "FFFF00", align: "center" } },
        { text: "2nd word ", options: { color: "0088CC", align: "center", breakLine: true } },
        { text: "\nline 4: 1st word", options: { color: "0088CC", align: "center" } },
    ],
    { x: 1.5, y: 4.2, w: 7, h: 1.3, margin: 0.1, fill: "224466" }
);

pptx.writeFile("PptxGenJS-TextRuns");

Results:

@brandonkal @Slidemagic @ReimaFrgos @ellisgl
If you have any breaking test cases, please provide them.

@gitbrent I've run the latest beta against all my decks and all works succesfully for me.

@ReimaFrgos - thanks!
Closing for inclusion in v3.3.0. If anyone else finds a breaking test case, please provide it.

Hi @gitbrent,  Hope align right for multiple words would result in single line with v3.3.0 release

@gitbrent
Now that the update has been released, I have tested it and have some breaking cases (3.3.1):
This input (centered):
///
R. Bolded text
Line 1
Line 2
R. Bolded text
////
Prints as
///
R.
Bolded text
Line 1 (notice extra line before and after bolded text)
Line 2
R.
Bolded text
////
Case 2:
////
Line 1
Line 2
Line 3
Line 5
Bolded Line 6
////
Prints as
////
Line 1
Line 2
Line 3
Line 5
Bolded Line 7
////

@brandonkal - I’ll need to see the code and the output, similar to my update on Jun 21

@gitbrent:
Test cases provided. See linked PR #839. I'd like to see that as part of the test suite to avoid a regression here.
Thank you!

@brandonkal - thanks for the PR. The demos.js file has 4400 diffs, can you just add the new test code here?

Run prettier on the file and you'll see the diff is much smaller.
The file wasn't formatted to your style.
If you look at the PR, the prettier run is in its own commit to make review easy.


fix: Push images to GitHub Container Registry, not Docker Hub

Push images to GitHub Container Registry, not Docker Hub
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO)
[ ] Any necessary documentation is added, such as:

README.md
docs/
The docs repo

Followup for https://github.com/kedacore/http-add-on/pull/156

My pleasure!


HTTP Get with AppRise

Is there a simple way to use Apprise to open a custom URL upon a change trigger? Something like http://<IP_ADDRESS>:5000/command?message=event1 triggered

Just use json://<IP_ADDRESS>....

This doesn't seem to work... Maybe because I need to use https?

@lightc786 when i use my brain and read the apprise documentation, I can see that..

https://github.com/caronc/apprise
let me know if that helps

That most certainly does help. Thank you for your patience with me. I
should have found that out myself in the apprise documentation
On Wed, Sep 15, 2021, 7:29 AM dgtlmoon @.***> wrote:

@lightc786 https://github.com/lightc786 when i use my brain and read
the apprise documentation, I can see that..
[image: image]
https://user-images.githubusercontent.com/275001/133425329-5c2d558a-2e5c-43c7-850e-a01720613281.png
https://github.com/caronc/apprise
let me know if that helps
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dgtlmoon/changedetection.io/issues/220#issuecomment-919936361,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANFUXOLCXZYAB5YWMOLQYK3UCB7Q7ANCNFSM5EASH6WA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I'm not able to get this to work. If I manually navigate to the page on my
network it works, but when I test the trigger in changedetection.io, it
doesn't work. My discord trigger works but not xmls:// or jsons:// for the
address on my local network.
On Wed, Sep 15, 2021 at 8:12 AM Ligh7C @.***> wrote:

That most certainly does help. Thank you for your patience with me. I
should have found that out myself in the apprise documentation
On Wed, Sep 15, 2021, 7:29 AM dgtlmoon @.***> wrote:

@lightc786 https://github.com/lightc786 when i use my brain and read
the apprise documentation, I can see that..
[image: image]
https://user-images.githubusercontent.com/275001/133425329-5c2d558a-2e5c-43c7-850e-a01720613281.png
https://github.com/caronc/apprise
let me know if that helps
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dgtlmoon/changedetection.io/issues/220#issuecomment-919936361,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANFUXOLCXZYAB5YWMOLQYK3UCB7Q7ANCNFSM5EASH6WA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@lightc786 I'm completly unable to help you without knowing more information, I'm not physically at your computer looking over your shoulder so you need to try to provide more information, I'm not a mind reader
please attach the output of your docker instance logs
docker logs changedetection.io &> logfile.txt

Here is the output from the logfile:

Process Notification: AppRise notifying json://<IP_ADDRESS>:5000/command?message=broadcast%20test%20message
WARNING:apprise:Failed to send JSON notification: Method not allowed., error=405.

@lightc786 when I read the documentation, I can see that it's doing a HTTP POST https://github.com/caronc/apprise/wiki/Notify_Custom_JSON
So in this case yeah it looks like the AppRise backend does not support HTTP GET/HTTPS GET

You should file an issue over at https://github.com/caronc/apprise


fix: timeout error despite topic publishing messages

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

24 of 24 (100.0%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
111

Relevant Lines:
111

💛  - Coveralls


Support GitLab format npm registry

Currently it will try to call /api/v4/projects/229/packages/npm/@prefix%2Fpackage-name
Which will return: HTTP/1.1 404 Not Found
The correct api endpoint for the package meta data is located at: /api/v4/packages/npm/@prefix%2Fpackage-name
While the endpoint for the tar.gz is api/v4/projects/229/packages/npm/<EMAIL_ADDRESS>

Thanks @danoe! The regex looks good, but to be sure, could you add a test for this scenario?

Thanks @danoe! The regex looks good, but to be sure, could you add a test for this scenario?

Thanks @jurre, I will give it a try.

Thanks @jurre, I will give it a try.

@jurre, i used one of the other registry tests as a template and created one new test, it seems it covers npm and yarn both, if i understood it correctly. Gitlab Registry as far as i know only supports scoped packages, so i think only this one test is necessary.

@jurre, i used one of the other registry tests as a template and created one new test, it seems it covers npm and yarn both, if i understood it correctly. Gitlab Registry as far as i know only supports scoped packages, so i think only this one test is necessary.

One small stylistic nit, other than that looks great, thanks!

One small stylistic nit, other than that looks great, thanks!

@jurre that last one should've been fixed with this, right?: https://github.com/dependabot/dependabot-core/pull/2950/commits/842efc912a56102a32525401cde99ac928c1c028

@jurre that last one should've been fixed with this, right?: https://github.com/dependabot/dependabot-core/pull/2950/commits/842efc912a56102a32525401cde99ac928c1c028


Is your feature request related to a problem?
Hi,
for my AC I use cool/heat modes. Currently in the summer it is always on cool mode. but whenever it is switched on it starts with heat mode.
Describe the solution you'd like
I think it should remember the last state
Describe alternatives you've considered
No response
Additional context
No response

What you A/C ? How do you turn it on from the mushroom card ?

Hi, AC is DAIKIN connected via Zennio KNX and defined in HA as follows:
climate:     - name: "GF Office AC"       on_off_address: "1/5/1" # AC On/Off       on_off_state_address: "1/5/2" # AC On/Off Status       temperature_address: "1/4/48" # ROOM TEMPERATURE       target_temperature_address: "1/4/45" # Setpoint       target_temperature_state_address: "1/4/46" # Setpoint Status       heat_cool_address: "1/5/5" # AC Simplified Mode       heat_cool_state_address: "1/5/6" # AC Simplified Mode Status       min_temp: 16       max_temp: 28
And the card code:
type: custom:mushroom-climate-card   entity: climate.gf_office_ac   fill_container: false   show_temperature_control: true   hvac_modes:     - heat     - cool   hold_action:     action: more-info   name: AC   icon: mdi:sun-snowflake
I hope this answers your inquiries.
RS

It's not a mushroom issue : https://community.home-assistant.io/t/is-there-a-way-to-establish-cool-as-default-turn-on-mode-on-climate/132754
As a workaround you can create a script and call it when using tap action.


v5 Prep

Needs a bit of work on actions to run the tests (other than unit). Tests pass locally

@Jeckerson  Some help needed on this with the dockerfile build action

@Jeckerson Are you planning to upgrade to version 5?


Fixed #26516 -- Added minlength attribute when forms.CharField.min_length is set.

https://code.djangoproject.com/ticket/26516

LGTM, thanks!

merged in 500e5a6886b4816740bbeedac1457b26a493d756, thanks!


No valid mission available error after reboot w/o sending new flight plan

It appears that the mission is not checked and the valid status isn't set until the user manually re-sends the flight plan after reboot.

I've tracked it down to this if statement where the checker is run. Is it possible that mission type is not none when a valid flight plan is still in memory but has not been validated yet?
/* check if the home position became valid in the meantime */
		if ((_mission_type == MISSION_TYPE_NONE || _mission_type == MISSION_TYPE_OFFBOARD) &&
			!_home_inited && _navigator->home_position_valid()) {

Not sure - it should be MISSION_TYPE_OFFBOARD since that is where the mission came from.

Any new info on this one? I believe its real and I have other reports supporting the findings here.

No new insights. But we confirmed it with another 10 flights or so.

Fixed in 253b8f5. Let me know if I missed any corner cases.


CREATE TABLE DDL does not save correct schema, resulting in mismatched plan vs execution (record batch) schema

Describe the bug
ROW_NUMBER() function places a non-nullable constraint on its generated field and thus the resulting schema should label that column as nulllable: false. But instead, the logical plan resulting from a table created using CREATE TABLE ... shows a schema with that field as nullable: true. This results in a runtime panic with queries that involve joins (although, I'm not quite sure why it doesn't complain on queries that aren't joins).
Error message produced with minimal repo below
Thread 'main' panicked at 'query failed to execute: External(ArrowError(InvalidArgumentError("batches[0] schema is different with argument schema.\n            batches[0] schema: Schema { fields: [Field { name: \"id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"name\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"row_num\", data_type: UInt64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} },\n            argument schema: Schema { fields: [Field { name: \"id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"name\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"row_num\", data_type: UInt64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }\n            ")))', src/main.rs:42:10

To Reproduce
Minimal repro
Run this script which will result in the error
Thread 'main' panicked at 'query failed to execute: External(ArrowError(InvalidArgumentError("batches[0] schema is different with argument schema.\n            batches[0] schema: Schema { fields: [Field { name: \"id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"name\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"row_num\", data_type: UInt64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} },\n            argument schema: Schema { fields: [Field { name: \"id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"name\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"row_num\", data_type: UInt64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }\n            ")))', src/main.rs:42:10

use std::sync::Arc;

use datafusion::{
    arrow::{
        array::{Int32Array, StringArray},
        datatypes::{DataType, Field, Schema},
        record_batch::RecordBatch,
        util::pretty::print_batches,
    },
    datasource::MemTable,
    prelude::{SessionConfig, SessionContext},
};

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let config = SessionConfig::new()
        .with_create_default_catalog_and_schema(true)
        .with_information_schema(true);

    let ctx = SessionContext::with_config(config);

    ctx.register_table("source_table", Arc::new(create_mem_table()))
        .unwrap();

    let create_table_query = r#"create table customers as SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS row_num FROM source_table"#;

    let _ = ctx
        .sql(create_table_query)
        .await
        .unwrap()
        .collect()
        .await
        .unwrap();

    let batches = ctx
        // performing a (cross) join query because joins seem to complain about execution schema vs plan schema mismatch
        .sql("select a.*, b.* from customers a, customers b")
        .await
        .unwrap()
        .collect()
        .await
        .expect("query failed to execute");

    print_batches(&batches).unwrap();
}

// just some random able that we'll append a row_num column to
fn create_mem_table() -> MemTable {
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int32, false),
        Field::new("name", DataType::Utf8, false),
    ]));

    let ids = Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5]));
    let names = Arc::new(StringArray::from(vec![
        "Alice", "Bob", "Charlie", "David", "Eve",
    ]));

    let batch = RecordBatch::try_new(schema.clone(), vec![ids as _, names as _]).unwrap();

    MemTable::try_new(schema, vec![vec![batch]]).unwrap()
}

Expected behavior
The schema that is saved when using create table should be correct (i.e., it should capture nullable: false requirements on fields). The logical plan shouldn't conflict with the observed record batches during execution. No panic should occur.
Additional context
A bit more context:
This is where nullable false is set. It's not being picked up in the create table statement.
https://github.com/apache/arrow-datafusion/blob/78d9613e81557ca5e5db8b75e5c7dec47ccee0a1/datafusion/physical-expr/src/window/row_number.rs#L54
  fn field(&self) -> Result<Field> {
        let nullable = false;
        let data_type = DataType::UInt64;
        Ok(Field::new(self.name(), data_type, nullable))
    }

I haven't investigated how this field property on the WindowExpr is actually used (or omitted) when constructing the logical plan.

I wonder if the nullable information isn't being updated correctly for join output 🤔
The code that handles setting nullability is here:
https://github.com/apache/arrow-datafusion/blob/b1d134e9ff37724459cb5090a6346a85152a1db7/datafusion/expr/src/logical_plan/builder.rs#L1068
Maybe somehow the plans in question aren't using that function or there is a bug in that function

I wonder if the nullable information isn't being updated correctly for join output 🤔
The code that handles setting nullability is here:
https://github.com/apache/arrow-datafusion/blob/b1d134e9ff37724459cb5090a6346a85152a1db7/datafusion/expr/src/logical_plan/builder.rs#L1068
Maybe somehow the plans in question aren't using that function or there is a bug in that function

I don't think this is the case because the schema is incorrect at create table time (before joins). Joins just seem to be the place where DF complains.
If you inspect the schema of the created table, the row_num column (created by the ROW_NUMBER() window function) is nullable: true when it should be nullable false, per the field implementation on the row number execution plan.

I think row_number() is marked as non-nullable because the implementation always produces a value (aka no nulls) and thus changing it to be nullable doesn't seem correct
In the example above, running
select * from customers

The output is this (there are no nulls):
+----+---------+---------+
| id | name    | row_num |
+----+---------+---------+
| 1  | Alice   | 1       |
| 2  | Bob     | 2       |
| 3  | Charlie | 3       |
| 4  | David   | 4       |
| 5  | Eve     | 5       |
+----+---------+---------+

I caught the error in the debugger and the error seems to be thrown in concat_batches called from here:
https://github.com/apache/arrow-datafusion/blob/32dfbb0bb5c51b55ad84eed4fc140840f0a76612/datafusion/physical-plan/src/joins/cross_join.rs#L138
it turns out that this exact check was removed in arrow 47.0.0 https://github.com/apache/arrow-rs/pull/4815 (merged into DataFusion 2 days ago in https://github.com/apache/arrow-datafusion/pull/7587) and so when I run this query against the latest version of DataFusion (not yet released to crates.io) I get:
+----+---------+---------+----+---------+---------+
| id | name    | row_num | id | name    | row_num |
+----+---------+---------+----+---------+---------+
| 1  | Alice   | 1       | 1  | Alice   | 1       |
| 1  | Alice   | 1       | 2  | Bob     | 2       |
| 1  | Alice   | 1       | 3  | Charlie | 3       |
| 1  | Alice   | 1       | 4  | David   | 4       |
| 2  | Bob     | 2       | 1  | Alice   | 1       |
| 2  | Bob     | 2       | 2  | Bob     | 2       |
| 2  | Bob     | 2       | 3  | Charlie | 3       |
| 2  | Bob     | 2       | 4  | David   | 4       |
| 3  | Charlie | 3       | 1  | Alice   | 1       |
| 3  | Charlie | 3       | 2  | Bob     | 2       |
| 3  | Charlie | 3       | 3  | Charlie | 3       |
| 3  | Charlie | 3       | 4  | David   | 4       |
| 4  | David   | 4       | 1  | Alice   | 1       |
| 4  | David   | 4       | 2  | Bob     | 2       |
| 4  | David   | 4       | 3  | Charlie | 3       |
| 4  | David   | 4       | 4  | David   | 4       |
| 5  | Eve     | 5       | 1  | Alice   | 1       |
| 5  | Eve     | 5       | 2  | Bob     | 2       |
| 5  | Eve     | 5       | 3  | Charlie | 3       |
| 5  | Eve     | 5       | 4  | David   | 4       |
| 1  | Alice   | 1       | 5  | Eve     | 5       |
| 2  | Bob     | 2       | 5  | Eve     | 5       |
| 3  | Charlie | 3       | 5  | Eve     | 5       |
| 4  | David   | 4       | 5  | Eve     | 5       |
| 5  | Eve     | 5       | 5  | Eve     | 5       |
+----+---------+---------+----+---------+---------+

@alamb thanks for the continued review and for the heads up on the removal of that invariant from arrow. We should still ensure that the nullability information is correct at all times, even if arrow doesn't complain.
I completely agree that changing the column generated by ROW_NUMBER() should not be marked as nullable: true. To that end, my PR (#7638) specifically introduces nullable: false when it should be false for window functions in the logical plan. The code in main assumes that the column generated by all window functions is nullable: true which is wrong. In other words, my PR brings the logical plan in line with the actual schema produced by the execution plan.
Let me know if I'm missing something.

@matthewgapp I also encountered the same mismatch issue as described above. However, after applying your PR(#7638), the problem persists for me. To illustrate, I've created a minimal reproducible example that builds upon yours, adding just a single 'Insert' operation and utilizing 'PARTITION'. Below are the code and the corresponding error message. I'm uncertain whether the issue lies with the 'Insert' operation or with the 'PARTITION' clause.
Minimal repro
called `Result::unwrap()` on an `Err` value: ArrowError(InvalidArgumentError("batches[0] schema is different with argument schema.\n            batches[0] schema: Schema { fields: [Field { name: \"id\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"name\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} },\n            argument schema: Schema { fields: [Field { name: \"id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"name\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }\n            "))

use std::sync::Arc;

use datafusion::{
    arrow::{
        datatypes::{DataType, Field, Schema},
        util::pretty::print_batches,
    },
    datasource::MemTable,
    prelude::{SessionConfig, SessionContext},
};

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let config = SessionConfig::new()
        .with_create_default_catalog_and_schema(true)
        .with_information_schema(true);

    let ctx = SessionContext::with_config(config);

    ctx.register_table("source_table", Arc::new(create_mem_table()))
        .unwrap();

    let insert_table_query = r#"INSERT INTO source_table VALUES (1, 'Alice'),(2, 'Bob'),(3, 'Charlie'),(4, 'David'), (5, 'Eve')"#;
    let _ = ctx
        .sql(insert_table_query)
        .await
        .unwrap()
        .collect()
        .await
        .unwrap();

    let create_table_query =
        r#"SELECT *, RANK() OVER (PARTITION BY id) AS row_num FROM source_table"#;

    let batches = ctx
        .sql(create_table_query)
        .await
        .unwrap()
        .collect()
        .await
        .unwrap();

    print_batches(&batches).unwrap();
}

fn create_mem_table() -> MemTable {
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int32, false),
        Field::new("name", DataType::Utf8, false),
    ]));

    MemTable::try_new(schema, vec![vec![]]).unwrap()
}

Thanks @xhwhis, this seems like a separate bug (one whose root cause is because the values exec sets the schema for all of its columns as nullable here. This causes the record batch to inherit this nullable schema. The record batch is then inserted into the memory table without complaining (we do check and complain for the inverse - if nullable columns inserted into a nullable table here).
The fix is might be passing down the expected table schema to the values logical constructor if there is one. I would open a separate ticket.

@alamb, more of a meta thought, but with #4815, but I'm concerned that all of these "bugs" will be go unnoticed (unless they're caught in the DF application logic like here), potentially creating incorrect behavior.
I think it could be helpful to have something like your strict mode (potentially a compiler flag). But I'm still ramping on this ecosystem so not sure who should determine the correctness of a schema and/or when that correctness should be asserted.
However, it does feel like anytime that record batches are stuck together and assumed to be a congruent, continuous thing (like a table or stream), that nullability between these batches should be consistent (or at least a superset of the nullability of containing thing (table or stream). For example, for the purposes of DF, it seems appropriate that non-nullable batches would always be acceptable in a table that is nullable. The inverse is not true.

However, it does feel like anytime that record batches are stuck together and assumed to be a congruent, continuous thing (like a table or stream), that nullability between these batches should be consistent (or at least a superset of the nullability of containing table or stream). For example, for the purposes of DF, it seems appropriate that non-nullable batches would always be acceptable in a table that is nullable. The inverse is not true.

Yes, I agree with this as the intent. I also agree with your assesment that this invariant is not always upheld. I think the reason that the RecordBatch level nullability has limited value (the biggest value is in knowing null/non-null for each individual Array, and indeed things like the low level kernels do look at the validity (Null) mask) and thus no one has ensured consistent enforcement across the codebase.


[Design] Creating an account

Description:
Users will need the ability to create an account. We need designs to help support what the user flow and design will look like.
Development Checklist:

[ ] lo-fi mock-ups
[ ] review with partners and osprey for feedback
[ ] make updates as needed
[ ] showcase at a demo day and sprint review

Dependencies

Blocked by
Blocking

Relevant documentation as reference
Definition of Ready

[ ] Acceptance criteria are included
[ ] Wireframes are included (if applicable)
[ ] Design / Solution is accepted by Product Owner (if applicable)
[ ] Dependencies are identified (technical, business, regulatory/policy)
[ ] Story has been estimated (under 13 pts)

Definition of Done

In progress:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] UI meets accessibility requirements
[ ] Unit tests are written
[ ] Work is traceable in GitHub
[ ] PR linked to ticket number
[ ] If needed/required - Dev adds flag/label to highlight any migration steps necessary prior to PROD deployment

Code review:

[ ] Code is peer reviewed and has passed CI/CD tests

QA:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] Code is potentially shippable to the production environment
[ ] Functional features have been tested and passed by QA
[ ] UI components tested by designer
[ ] Code is deployed to PROD when moved to 'done' column (unless requested otherwise by PO)

PO Review:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] Reviewed and approved by Product Owner

Notes:

Things we should keep in mind in the design:

contact information (name, email, confirm email, phone, cell address)
password (create password, confirm password)
Subscribe to communications
FOI/privacy statement.

Additional

vehicle information
search preferences

Current create account: https://camping.bcparks.ca/register

Combine with #70


The iperf text results dont label the internal and external IP results

When the final results come out of iperf on the command line the results dont differentiate the internal and external IP results.  You can likely tell from the diff in performance but we should be explicit.
This may also hold for the network benchmarks.
The detailed JSON block does including the right information so it should be a matter of updating the output text to be more detailed.

Connor is working on this and there is a fix out ... https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/178

#178 is merged - it will make 0.13.0.


Support for Mobile Data tethering

I would first like to say I really like your app!
It works very well and I appreciate all the work that has been put into it. I managed to tether my Wi-Fi connection without any problems, but I always fail to do the same with my data. Is it possible to implement that feature as an alternative for the default hotspot/USB tethering?
Cheers!

Hi there, thanks for the kind words.
I think there's a misunderstanding, the app already works as a mobile data hotspot. That is, in fact, how I use it myself.
The app works as a proxy. In non technical terms, your computer tells your phone to load a webpage. Your phone talks to the internet over whatever connection it has, and then relays that information back to your computer like a middleman.
This should work regardless of how your phone is connected to the Internet. One caveat though is that WiFi must be enabled on your phone, even though it doesn't need to actually be connected to anything. WiFi must be on so the phone can create the Tetherfi WiFi network that your computer connects to.
When you use the app over mobile data, where do you get stuck? Is there an error launching the tetherfi hotspot? Or does it run the hotspot fine but your computer can't actually connect to it?


feat: added an option for retrieving transaction records from either the Mirror Node or the Consensus Node.

Description:

added GET_RECORD_DEFAULT_TO_CONSENSUS_NODE to configuration docs. This configuration allows the Relay to retrieve transaction record by making calls to Consensus Node (CN) or Mirror Node (MN) based on its input. Default to "false" for MN calls.

added getTransactionStatusAndMetrrics() method. This method is the implementation for the logic mentioned above. If GET_RECORD_DEFAULT_TO_CONSENSUS_NODE is set to "true", calls will be made to CN. Otherwise, calls will be made to MN. MirrorNodeClient is injected from eth.ts level. Unit tests are included.

modified getTransferAmountSumForAccount so it can be compatible with transfers array returned from MN.

added getTransactionStatusAndMetrrics to executeTransaction() and executeTransaction().

removed unused methods and added JSDoc to methods in SDKClient, sdkClient.executeGetTransactionRecord() and sdkClient.getTransactionMetrics() as they are now replaced by getTransactionStatusAndMetrrics(). Also added JSDoc comments for all methods in SDKClient.

Modified and added new tests in sdkClient.spec.ts reflect the new logic.

Related issue(s):
Fixes #2706
Notes for reviewer:

Checklist

[ ] Documented (Code comments, README, etc.)
[ ] Tested (unit, integration, etc.)

Looks really good. I think we should run the Hbar Limiter tests two times - once with GET_RECORD_DEFAULT_TO_CONSENSUS_NODE=true and once with false. Ideally when it is using the Mirror node the total untracked spent amount by the operator should be less. In those tests there is a delta variable, currently set to 0.02. Try decreasing it and see if we are closer to 0 when using the MN as compared to CN

Yeah nice this is a good point! Pushing updates soon!

Looks really good. I think we should run the Hbar Limiter tests two times - once with GET_RECORD_DEFAULT_TO_CONSENSUS_NODE=true and once with false. Ideally when it is using the Mirror node the total untracked spent amount by the operator should be less. In those tests there is a delta variable, currently set to 0.02. Try decreasing it and see if we are closer to 0 when using the MN as compared to CN

@Ivo-Yankov updated new changes to run hbarLimiter with another round with calls being made to mirror node. Delta just went down a little not too much though but still the new solution did its job. Please take a look when you have a chance!

Great job on the improvements; the code is certainly cleaner than before. However, passing the TransactionService around within methods is still creating unnecessary coupling between classes, which we should address.
To avoid this and simplify the dependency structure, we can take the following steps:

Follow the dependency tree: EthImpl -> HapiService -> AccountService, TransactionService, ContractService, FeeService -> SdkClient, MirrorNodeClient. We can consider opening separate issues to extract logic for accounts, contracts, and fees into distinct services.
Keep the logic for initializing SdkClient into HapiService. This way, HapiService will manage the SdkClient and be the main point of interaction.
Remove the direct dependency on TransactionService from EthImpl. Instead, initialize TransactionService inside HapiService, and have HapiService pass the initialized SdkClient to it.
Add methods such as createFile, deleteFile, appendFile, etc., directly in HapiService. This should be our main entry point for operations (e.g., instead of calling hapiService.getSdkClient().createFile(), we should directly call hapiService.createFile()).
The wrapper logic for executeQuery and executeTransaction should also reside within HapiService. HapiService can then use these wrappers and delegate the actual execution to the SdkClient as needed.
With HapiService calling TransactionService to fetch and extract transaction details, it remains agnostic to whether SdkClient or MirrorNodeClient is being used.

This approach untangles the dependencies between classes, resulting in a clear and linear dependency tree.

@victor-yanev absolute amazing points! However, the PR is getting bigger and it would be a little out of scope for this particular ticket, given that couple other tickets are being blocked by this PR. Could you kindly open a new ticket and drop all these amazing suggestions there along with more ideas of reorganizing the codebase?


Code formatter

It would be great if the code formatter work.
I'm talking about the integrated one that works with this shortcuts.
On Windows Shift + Alt + F
On Mac Shift + Option + F
On Linux Ctrl + Shift + I

It would be great if the code formatter work.

Agreed!

I'm talking about the integrated one that works with this shortcuts.

Well those shortcuts just trigger some external formatting tool, so really the only thing that's "integrated" is the shortcuts.
I'm working on the tooling and infrastructure to make writing a nice formatter possible, so it is on my radar, but it's still a while until we'll have that. I'll close this issue because it's not really something that happens in the VSCode plugin, but I'll ping you in here once we've actually got the functionality.


Cookie Button without a function

If you call the cookies via domain/cookie/offcanvas and want to confirm, the button is without function. You cannot save anything. If you confirm the cookies via another page, for example via the start page below, then everything works.

[public] Automated response: This issue is linked to the internal issue https://shopware.atlassian.net/browse/NEXT-30234.
[created from NEXT-30234, comment 485572]


"unstable" feature is redundant

Opening this issue for discussion. As far as I can tell, the unstable feature is not supposed to do anything on its own, but is meant to indicate that one or more unstable feature(s) was/were enabled at build time. However it is useless because there is nothing forcing a user to set it correctly.
There is one extra function defined if it is on, but that seems like an error, since channel-api, the only unstable feature, can be enabled without it. If the intent was to block compilation by adding a function required by the channel API in unstable, that's confusing and poor practice.
It does not seem to be pulled with other unstable features automatically, and it does not affect the version string or anything to indicate that the build includes unstable features.
I have two potential solutions in mind:

remove it
enforce its use in build.rs by panicking if an unstable feature is enabled without it

The channel-api feature right now it is silently omitted if unstable is not passed along. So it works as intended in preventing the user from using it without being aware it is unstable.
A build.rs check (or a compile-time check) would be a good idea so who doesn't read the instruction can be told to do so.

Thanks for clarifying.
A secondary issue is that unstable is presented as just a compile-time guard, but in practice it also defines one function (not sure if it changes encoder behavior on its own). Based on its description it seems like it should not change code in the encoder. I propose making the function it defines part of another feature. Even if it is minor, it is unintuitive from a user point of view for the mandatory meta-feature to also pull in unrelated changes.


Structured log body is being sent as textPayload without structure

{"insertId": "upsyhzmdeu","jsonPayload": {"logger_name": "<redacted>","method": "log","levelValue": 900,"thread_id": 56,"message": "Error in stream connection (will retry): <redacted>: java.net.ConnectException: Failed to connect to <redacted>","@timestamp": "2024-08-23T14:35:45.502Z","thread_name": "<redacted>","level": "WARNING","class": "<redacted>ChannelImpl"},"resource": {"type": "k8s_container","labels": {"pod_name": "<redacted>-79f75cb-6h5hs","cluster_name": "<redacted>-us-east1","location": "us-east1","container_name": "test","project_id": "<redacted-project>","namespace_name": "<redacted>-410"}},"timestamp": "2024-08-23T14:35:45.502691132Z","severity": "WARNING","labels": {"k8s-pod/<redacted>_service_name": "test","k8s-pod/app_kubernetes_io/instance": "<redacted>","k8s-pod/<redacted>/canonical-revision": "latest","compute.googleapis.com/resource_name": "<redacted>-us-east1-<redacted>", "k8s-pod/app_kubernetes_io/name": "grpc","k8s-pod/service_istio_io/canonical-name": "grpc","k8s-pod/pod-template-hash": "575cb","k8s-pod/security_istio_io/tlsMode": "istio","k8s-pod/app_kubernetes_io/component": "test"},"logName": "projects/<redacted>/logs/stderr","receiveTimestamp": "2024-08-23T14:35:50.519860025Z"}

From a report on slack, this is not sent as a json payload using the googlecloud exporter for logs.

The customer is sending OTLP from a java app, and is using flogger as logging framework.
The json in question was coming in as a string, rather than a map.
They are able to transform to a map using:
transform:
    log_statements:
      - context: log
        statements:
          # Extract everything between Str( and )
          - 'set(body, ParseJSON(body))'

But then they get errors when the log doesn't have a json body.
@psx95 have you tried OTLP logging from a java app before?  Do you know if there is a way to make sure the body is sent as a map instead of a string?
Alternatively, we could make our exporter attempt to parse strings as json, but that might come with a performance penalty.

No, I haven't tried OTLP logging before. I will have to look into this.


Combine Error with Reason when Connection State Changed

Looking at the logs there are a number of cases where we see ConnectionStateChange events which have values that imply an error happened, but the error itself is not logged. This change couples error and reason via typing to prevent losing one or the other duration connection state changes.

Note: this is on next, as to do it in main would be a breaking change, as a number of impacted types were deprecated and removed this release cycle.

resolves AB#5197

@anthony-murphy Doesn't this break the contract on IDeltaManagerEvents? A number of events had their shape changed.

@anthony-murphy Doesn't this break the contract on IDeltaManagerEvents? A number of events had their shape changed.

@kian-thompson  can you provide an example? nothing should have change besides internals

@anthony-murphy Doesn't this break the contract on IDeltaManagerEvents? A number of events had their shape changed.

@kian-thompson can you provide an example? nothing should have change besides internals. This was also released as part of client_v2.0.0-internal.6.1.0, so not sure how it could be causing an issue.

@kian-thompson can you provide an example?

DeltaManager.emit("disconnect", ...) will emit an object of shape IConnectionStateChangeReason.
https://github.com/microsoft/FluidFramework/blob/main/packages/loader/container-loader/src/deltaManager.ts#L800
If I use IContainer.deltaManager and try listening to the "disconnect" event, I get an IConnectionStateChangeReason object when I expect (reason: string, error?: IAnyDriverError) based on the definition in IDeltaManagerEvents.
https://github.com/microsoft/FluidFramework/blob/main/packages/common/container-definitions/src/deltas.ts#L120

@kian-thompson can you provide an example?

DeltaManager.emit("disconnect", ...) will emit an object of shape IConnectionStateChangeReason. https://github.com/microsoft/FluidFramework/blob/main/packages/loader/container-loader/src/deltaManager.ts#L800
If I use IContainer.deltaManager and try listening to the "disconnect" event, I get an IConnectionStateChangeReason object when I expect (reason: string, error?: IAnyDriverError) based on the definition in IDeltaManagerEvents. https://github.com/microsoft/FluidFramework/blob/main/packages/common/container-definitions/src/deltas.ts#L120

good catch. seems like a bug. fix should be trivial, just need to pull text and error off the object and pass those to the emit method instead of the object

good catch. seems like a bug. fix should be trivial, just need to pull text and error off the object and pass those to the emit method instead of the object

AB#7123