[ADAP-670] [Regression] communication error in 1.5

Is this a regression in a recent version of dbt-redshift?

[X] I believe this is a regression in dbt-redshift functionality
[X] I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior
dbt compile fails with a "communication error"/OSError using DBT 1.5.

Just wanted to chime in mentioning that I am seeing this error as well and can confirm that downgrading to 1.4 results in no communication error.

Hello,
I am seeing the same issue but without upgrading to 1.5. It was working perfectly fine on 1.5 until a few hours ago.
Taking a closer look to the packages installed I see that the only difference is a change of the redshift-connector from redshift-connector-2.0.911 to redshift-connector-2.0.912

@brooke-white, do you have any idea what might have changed in the latest patch that was released today?

EDIT: I confirm that downgrading to redshift-connector-2.0.911 after installing dbt-redshift==1.5.5 fixed the issue

Hello, I am seeing the same issue but without upgrading to 1.5. It was working perfectly fine on 1.5 until a few hours ago. Taking a closer look to the packages installed I see that the only difference is a change of the redshift-connector from redshift-connector-2.0.911 to redshift-connector-2.0.912.
EDIT: I confirm that downgrading to redshift-connector-2.0.911 after installing dbt-redshift==1.5.5 fixed the issue

Confirming that downgrading to redshift-connector-2.0.911 worked for my team too

This issue also exists in dbt-redshift 1.5.6. On running dbt debug, I get the following error.

The exact same issue on 1.5.5 when running dbt compile, just today. I'll try to follow the advice here and downgrade the package.

Hi, if anyone experiencing this issue could enable redshift-connector's logging and provide the logs it'd be a great help in the redshift-connector team's investigation of this issue. We're working to try and reproduce on our end and will cut a release as soon as a fix is identified. Thanks!

The issue also exists on dbt-core 1.5.2 with dbt-redshift 1.5.6 which appears to install redshift-connector==2.0.912. Confirmed setting redshift-connector to 2.0.911 like others above have mentioned works to resolve this issue.

@Brooke-white Here's what we saw in our CI:
click to expand
self = <redshift_connector.core.Connection object at 0x7f18869284f0>
user = '***', password = '***', database = '***'
host = '***', port = ***
source_address = None, unix_sock = None, ssl = True, sslmode = 'verify-ca'
timeout = None, max_prepared_statements = 1000, tcp_keepalive = True
application_name = 'dbt.adapters.base.connections', replication = None
client_protocol_version = 2, database_metadata_current_db_only = True
credentials_provider = None, provider_name = None, web_identity_token = None
numeric_to_float = False

    def __init__(
        self: "Connection",
        user: str,
        password: str,
        database: str,
        host: str = "localhost",
        port: int = ***,
        source_address: typing.Optional[str] = None,
        unix_sock: typing.Optional[str] = None,
        ssl: bool = True,
        sslmode: str = "verify-ca",
        timeout: typing.Optional[int] = None,
        max_prepared_statements: int = 1000,
        tcp_keepalive: typing.Optional[bool] = True,
        application_name: typing.Optional[str] = None,
        replication: typing.Optional[str] = None,
        client_protocol_version: int = DEFAULT_PROTOCOL_VERSION,
        database_metadata_current_db_only: bool = True,
        credentials_provider: typing.Optional[str] = None,
        provider_name: typing.Optional[str] = None,
        web_identity_token: typing.Optional[str] = None,
        numeric_to_float: bool = False,
    ):
        """
        Creates a :class:`Connection` to an Amazon Redshift cluster. For more information on establishing a connection to an Amazon Redshift cluster using `federated API access <https://aws.amazon.com/blogs/big-data/federated-api-access-to-amazon-redshift-using-an-amazon-redshift-connector-for-python/>`_ see our examples page.
        This is the underlying :class:`Connection` constructor called from :func:`redshift_connector.connect`.

        Parameters
        ----------
        user : str
            The username to use for authentication with the Amazon Redshift cluster.
        password : str
            The password to use for authentication with the Amazon Redshift cluster.
        database : str
            The name of the database instance to connect to.
        host : str
            The hostname of the Amazon Redshift cluster.
        port : int
            The port number of the Amazon Redshift cluster. Default value is ***.
        source_address : Optional[str]
        unix_sock : Optional[str]
        ssl : bool
            Is SSL enabled. Default value is ``True``. SSL must be enabled when authenticating using IAM.
        sslmode : str
            The security of the connection to the Amazon Redshift cluster. 'verify-ca' and 'verify-full' are supported.
        timeout : Optional[int]
            The number of seconds before the connection to the server will timeout. By default there is no timeout.
        max_prepared_statements : int
        tcp_keepalive : Optional[bool]
            Is `TCP keepalive <https://en.wikipedia.org/wiki/Keepalive#TCP_keepalive>`_ used. The default value is ``True``.
        application_name : Optional[str]
            Sets the application name. The default value is None.
        replication : Optional[str]
            Used to run in `streaming replication mode <https://www.postgresql.org/docs/12/protocol-replication.html>`_.
        client_protocol_version : int
            The requested server protocol version. The default value is 1 representing `EXTENDED_RESULT_METADATA`. If the requested server protocol cannot be satisfied, a warning will be displayed to the user.
        database_metadata_current_db_only : bool
            Is `datashare <https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html>`_ disabled. Default value is True, implying datasharing will not be used.
        credentials_provider : Optional[str]
            The class-path of the IdP plugin used for authentication with Amazon Redshift.
        provider_name : Optional[str]
            The name of the Redshift Native Auth Provider.
        web_identity_token: Optional[str]
            A web identity token used for authentication via Redshift Native IDP Integration
        numeric_to_float: bool
            Spe***fies if NUMERIC datatype values will be converted from ``de***mal.De***mal`` to ``float``.

        Translate the host/port argument into a sequence of 5-tuples that contain
        all the necessary arguments for creating a socket connected to that service.
        host is a domain name, a string representation of an IPv4/v6 address or
        None. port is a string service name such as 'http', a numeric port number or
        None. By passing None as the value of host and port, you can pass NULL to
        the underlying C API.

        The family, type and proto arguments can be optionally spe***fied in order to
        narrow the list of addresses returned. Passing zero as a value for each of
        these arguments selects the full range of results.
        """
        # We override this function since we want to translate the numeric family
        # and socket type values to enum constants.
        addrlist = []
>       for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E       OSError: Int or String expected

/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/socket.py:955: OSError

During handling of the above exception, another exception occurred:

cls = <class 'dbt.adapters.redshift.connections.RedshiftConnectionManager'>
connection = Connection(type='redshift', name='_test', state=<ConnectionState.CLOSED: 'closed'>, transaction_open=False, _handle=No...lse, connect_timeout=None, role=None, sslmode=<UserSSLMode.prefer: 'prefer'>, retries=6, region=None, autocommit=True))
connect = <function RedshiftConnectMethodFactory.get_connect_method.<locals>.connect at 0x7f1885e49240>
logger = AdapterLogger(name='Redshift')
retryable_exceptions = [<class 'redshift_connector.error.OperationalError'>, <class 'redshift_connector.error.DatabaseError'>, <class 'redshift_connector.error.DataError'>]
retry_limit = 6
retry_timeout = <function RedshiftConnectionManager.open.<locals>.exponential_backoff at 0x7f1885e49000>
_attempts = 0

    @classmethod
    def retry_connection(
        cls,
        connection: Connection,
        connect: Callable[[], AdapterHandle],
        logger: AdapterLogger,
        retryable_exceptions: Iterable[Type[Exception]],
        retry_limit: int = 1,
        retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1,
        _attempts: int = 0,
    ) -> Connection:
        """Given a Connection, set its handle by calling connect.

        The calls to connect will be retried up to retry_limit times to deal with transient
        connection errors. By default, one retry will be attempted if retryable_exceptions is set.

        :param Connection connection: An instance of a Connection that needs a handle to be set,
            usually when attempting to open it.
        :param connect: A callable that returns the appropiate connection handle for a
            given adapter. This callable will be retried retry_limit times if a subclass of any
            Exception in retryable_exceptions is raised by connect.
        :type connect: Callable[[], AdapterHandle]
        :param AdapterLogger logger: A logger to emit messages on retry attempts or errors. When
            handling expected errors, we call debug, and call warning on unexpected errors or when
            all retry attempts have been exhausted.
        :param retryable_exceptions: An iterable of exception classes that if raised by
            connect should trigger a retry.
        :type retryable_exceptions: Iterable[Type[Exception]]
        :param int retry_limit: How many times to retry the call to connect. If this limit
            is exceeded before a successful call, a FailedToConnectError will be raised.
            Must be non-negative.
        :param retry_timeout: Time to wait between attempts to connect. Can also take a
            Callable that takes the number of attempts so far, beginning at 0, and returns an int
            or float to be passed to time.sleep.
        :type retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1
        :param int _attempts: Parameter used to keep track of the number of attempts in calling the
            connect function across recursive calls. Passed as an argument to retry_timeout if it
            is a Callable. This parameter should not be set by the initial caller.
        :raises dbt.exceptions.FailedToConnectError: Upon exhausting all retry attempts without
            successfully acquiring a handle.
        :return: The given connection with its appropriate state and handle attributes set
            depending on whether we successfully acquired a handle or not.
        """
        timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
        if timeout < 0:
            raise dbt.exceptions.FailedToConnectError(
                "retry_timeout cannot be negative or return a negative time."
             For more information on establishing a connection to an Amazon Redshift cluster using `federated API access <https://aws.amazon.com/blogs/big-data/federated-api-access-to-amazon-redshift-using-an-amazon-redshift-connector-for-python/>`_ see our examples page.
        This is the underlying :class:`Connection` constructor called from :func:`redshift_connector.connect`.

        Parameters
        ----------
        user : str
            The username to use for authentication with the Amazon Redshift cluster.
        password : str
            The password to use for authentication with the Amazon Redshift cluster.
        database : str
            The name of the database instance to connect to.
        host : str
            The hostname of the Amazon Redshift cluster.
        port : int
            The port number of the Amazon Redshift cluster. Default value is ***.
        source_address : Optional[str]
        unix_sock : Optional[str]
        ssl : bool
            Is SSL enabled. Default value is ``True``. SSL must be enabled when authenticating using IAM.
        sslmode : str
            The security of the connection to the Amazon Redshift cluster. 'verify-ca' and 'verify-full' are supported.
        timeout : Optional[int]
            The number of seconds before the connection to the server will timeout. By default there is no timeout.
        max_prepared_statements : int
        tcp_keepalive : Optional[bool]
            Is `TCP keepalive <https://en.wikipedia.org/wiki/Keepalive#TCP_keepalive>`_ used. The default value is ``True``.
        application_name : Optional[str]
            Sets the application name. The default value is None.
        replication : Optional[str]
            Used to run in `streaming replication mode <https://www.postgresql.org/docs/12/protocol-replication.html>`_.
        client_protocol_version : int
            The requested server protocol version. The default value is 1 representing `EXTENDED_RESULT_METADATA`. If the requested server protocol cannot be satisfied, a warning will be displayed to the user.
        database_metadata_current_db_only : bool
            Is `datashare <https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html>`_ disabled. Default value is True, implying datasharing will not be used.
        credentials_provider : Optional[str]
            The class-path of the IdP plugin used for authentication with Amazon Redshift.
        provider_name : Optional[str]
            The name of the Redshift Native Auth Provider.
        web_identity_token: Optional[str]
            A web identity token used for authentication via Redshift Native IDP Integration
        numeric_to_float: bool
            Spe***fies if NUMERIC datatype values will be converted from ``de***mal.De***mal`` to ``float``. By default NUMERIC values are received as ``de***mal.De***mal``.
        """
        self.merge_socket_read = True

        _client_encoding = "utf8"
        self._commands_with_count: typing.Tuple[bytes, ...] = (
            b"INSERT",
            b"DELETE",
            b"UPDATE",
            b"MOVE",
            b"FETCH",
            b"COPY",
            b"SELECT",
        )
        self.notifications: deque = deque(maxlen=100)
        self.notices: deque = deque(maxlen=100)
        self.parameter_statuses: deque = deque(maxlen=100)
        self.max_prepared_statements: int = int(max_prepared_statements)
        self._run_cursor: Cursor = Cursor(self, paramstyle=DbApiParamstyle.NAMED.value)
        self._client_protocol_version: int = client_protocol_version
        self._database = database
        self.py_types = deepcopy(PY_TYPES)
        self.redshift_types = deepcopy(REDSHIFT_TYPES)
        self._database_metadata_current_db_only: bool = database_metadata_current_db_only
        self.numeric_to_float: bool = numeric_to_float

        # based on _client_protocol_version value, we must use different conversion functions
        # for receiving some datatypes
        self._enable_protocol_based_conversion_funcs()

        self.web_identity_token = web_identity_token

        if user is None:
            raise InterfaceError("The 'user' connection parameter cannot be None")

        redshift_native_auth: bool = False

        if application_name is None or application_name == "":

            def get_calling_module() -> str:
                import inspect

                module_name: str = ""
                stack: typing.List[inspect.FrameInfo] = inspect.stack()
                try:
                    # get_calling_module  -> init -> connect -> init -> calling module
                    start: int = min(4, len(stack) - 1)
                    parent = stack[start][0]
                    calling_module = inspect.getmodule(parent)

                    if calling_module:
                        module_name = calling_module.__name__
                except:
                    pass
                finally:
                    del parent
                    del stack

                return module_name

            application_name = get_calling_module()
        init_params: typing.Dict[str, typing.Optional[typing.Union[str, bytes]]] = {
            "user": "",
            "database": database,
            "application_name": application_name,
            "replication": replication,
            "client_protocol_version": str(self._client_protocol_version),
            "driver_version": DriverInfo.driver_full_name(),
            "os_version": self.client_os_version,
        }

        if credentials_provider:
            init_params["plugin_name"] = credentials_provider

            if credentials_provider.split(".")[-1] in (
                "BasicJwtCredentialsProvider",
                "BrowserAzureOAuth2CredentialsProvider",
            ):
                redshift_native_auth = True
                init_params["idp_type"] = "AzureAD"

                if provider_name:
                    init_params["provider_name"] = provider_name

        if not redshift_native_auth or user:
            init_params["user"] = user

        _logger.debug(make_divider_block())
        _logger.debug("Establishing a connection")
        _logger.debug(init_params)
        _logger.debug(make_divider_block())

        for k, v in tuple(init_params.items()):
            if isinstance(v, str):
                init_params[k] = v.encode("utf8")
            elif v is None:
                del init_params[k]
            elif not isinstance(v, (bytes, bytearray)):
                raise InterfaceError("The parameter " + k + " can't be of type " + str(type(v)) + ".")

        if "user" in init_params:
            self.user: bytes = typing.cast(bytes, init_params["user"])
        else:
            self.user = b""

        if isinstance(password, str):
            self.password: bytes = password.encode("utf8")
        else:
            self.password = password

        self.autocommit: bool = False
        self._xid = None

        self._caches: typing.Dict = {}

        # Create the TCP/Ip socket and connect to spe***fic database
        # if there already has a socket, it will not create new connection when run connect again
        try:
            if unix_sock is None and host is not None:
                self._usock: typing.Union[socket.socket, "SSLSocket"] = socket.socket(
                    socket.AF_INET, socket.SOCK_STREAM
                )
                if source_address is not None:
                    self._usock.bind((source_address, 0))
            elif unix_sock is not None:
                if not hasattr(socket, "AF_UNIX"):
                    raise InterfaceError("attempt to connect to unix socket on unsupported " "platform")
                self._usock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
            else:
                raise ProgrammingError("one of host or unix_sock must be provided")
            if timeout is not None:
                self._usock.settimeout(timeout)

            if unix_sock is None and host is not None:
                hostport: typing.Tuple[str, int] = Connection.__get_host_address_info(host, port)
                _logger.debug(
                    "Attempting to create connection socket with address {} {}".format(hostport[0], str(hostport[1]))
                )
                self._usock.connect(hostport)
            elif unix_sock is not None:
                self._usock.connect(unix_sock)

            # For Redshift, we the default ssl approve is True
            # create ssl connection with Redshift CA certificates and check the hostname
            if ssl is True:
                try:
                    from ssl import CERT_REQUIRED, SSLContext

                    # ssl_context = ssl.create_default_context()

                    path = os.path.abspath(__file__)
                    if os.name == "nt":
                        path = "\\".join(path.split("\\")[:-1]) + "\\files\\redshift-ca-bundle.crt"
                    else:
                        path = "/".join(path.split("/")[:-1]) + "/files/redshift-ca-bundle.crt"

                    ssl_context: SSLContext = SSLContext()
                    ssl_context.verify_mode = CERT_REQUIRED
                    ssl_context.load_default_certs()
                    ssl_context.load_verify_locations(path)

                    # Int32(8) - Message length, including self.
                    # Int32(80877103) - The SSL request code.
                    self._usock.sendall(ii_pack(8, 80877103))
                    resp: bytes = self._usock.recv(1)
                    if resp != b"S":
                        _logger.debug(
                            "Server response code when attempting to establish ssl connection: {!r}".format(resp)
                        )
                        raise InterfaceError("Server refuses SSL")

                    if sslmode == "verify-ca":
                        self._usock = ssl_context.wrap_socket(self._usock)
                    elif sslmode == "verify-full":
                        ssl_context.check_hostname = True
                        self._usock = ssl_context.wrap_socket(self._usock, server_hostname=host)

                except ImportError:
                    raise InterfaceError("SSL required but ssl module not available in " "this python installation")

            self._sock: typing.Optional[typing.BinaryIO] = self._usock.makefile(mode="rwb")
            if tcp_keepalive:
                self._usock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
        except socket.error as e:
            self._usock.close()
>           raise InterfaceError("communication error", e)
E           redshift_connector.error.InterfaceError: ('communication error', OSError('Int or String expected'))

/home/runner/work/dbt-redshift/dbt-redshift/.tox/integration-redshift/lib/python3.10/site-packages/redshift_connector/core.py:661: InterfaceError

Thanks so much, that was extremely helpful :)
I reviewed the trace and was able to reproduce the issue The root cause is this line in DBT. I see the type of port passed to redshift-connector is dbt.type_helpers.Port rather than int. A simple fix is to change the line to apply a int case to self.credentials.port:
            "port": int(self.credentials.port) if self.credentials.port else 5439,

for those interested, here is my repro:
from dbt.helper_types import Port
import redshift_connector

with redshift_connector.connect(
    user='***',
    host='***',
    password='***',
    database='***',
    port = Port(5439),
) as conn:
      with conn.cursor() as cursor:
        cursor.execute("select 1")
        print(cursor.fetchone())

here's my verification of fix:
from dbt.helper_types import Port
import redshift_connector

with redshift_connector.connect(
    user='***',
    host='***',
    password='***',
    database='***',
    port = int(Port(5439)),
) as conn:
      with conn.cursor() as cursor:
        cursor.execute("select 1")
        print(cursor.fetchone())

happy to open a PR if you folks are ok with this fix.

The root cause is this line in DBT.

if you'll humor me being overly pedantic, this isn't the "root" cause per se. The root cause would be where in the changes issues in 2.0.912 that caused a breaking change in dbt. This code was previously working for months.
silly example:

if I put a 60v battery into a 20v cordless drill and it shorts the drills motor, it would be incorrect to point to the motor as root cause, as the root cause is the battery.

While passing a dbt.helper_types.Port to the connect method's port parameter worked prior to 2.0.912, it isn't something redshift-connector supports per the docs.

At the end of the day, we don't really need to assign blame anywhere. It's just important that we solve the problem.
Thank you, @Brooke-white!

Agreed with @falonso-alo, I don't think either side (I already don't like that connotation) should be assigning blame. This industry is toxic enough as it is and this work is hard enough as it is. With that in mind, I did some research to understand exactly what is going on and felt like sharing that information.
The initial confusion is coming from that fact that Port subclasses from int; in other words, isinstance(Port, int)==True. We are in fact passing an int, we're just not passing exactly an int.
In version v2.0.911, this line was used to connect via host/port:
https://github.com/aws/amazon-redshift-python-driver/blob/2898d86de9bb6be1e3def88e15c78c9ea767ec52/redshift_connector/core.py#L596
There is no check there to see whether port is a proper int. As long as port can behave like an int, everyone's happy.
This line was updated in v2.0.912 to:
https://github.com/aws/amazon-redshift-python-driver/blob/b2dde82ec9156e2adcc801ac54c051f3cfe61e33/redshift_connector/core.py#L610-L614
with reference to this new function:
https://github.com/aws/amazon-redshift-python-driver/blob/b2dde82ec9156e2adcc801ac54c051f3cfe61e33/redshift_connector/core.py#L397-L409
The new function calls the builtin socket, which raises the OSError(Int or String expected) exception that we're seeing. This library must be looking for exactly an int. I stopped digging at that point as socket is implemented in C, making it a bit more difficult for me to investigate. While I didn't see the check, I'm willing to assume that there is a check for int in there.
Putting all of that aside, I have to imagine we're not the only group that is passing a subclass of int (or str for that matter) into the Connection class and expecting it to work (given the argument above). It might not be the primary usage pattern, but until v2.0.912 it appeared to be a valid one. And I'm sure you can appreciate the nuance that comes with sitting on top of code that you can't easily alter (such as socket in this scenario). So while we aren't passing a proper int into Connection, it might be useful for those downstream (both us and other teams) to update this line:
https://github.com/aws/amazon-redshift-python-driver/blob/b2dde82ec9156e2adcc801ac54c051f3cfe61e33/redshift_connector/core.py#L403
to this:
response = socket.getaddrinfo(host=host, port=int(port), family=socket.AF_INET)

Such is life with dynamic typing in python.

Hi, seems like the error is back. I am on dbt-core=1.5.1 and dbt-redshift=1.5.7.
06:02:48  Encountered an error while running operation: Database Error

  ('communication error', OSError('Int or String expected'))

script returned exit code 1

I see in my docker build that redshift-connector-2.0.913 is installed. That version was released 7hrs ago.

@jan-benisek shoot! I've opened #531 to fix this once we're working on a 1.5 patch release to fix this as we speak. I'll reply back here when the patch is out, my estimate is within the next four hours, but certainly in the next 24 hours.

dbt-redshift 1.5.8 has now been released. I cannot stress enough how:

how sorry we are to have had you all suffer a degraded reliability of the dbt-redshift adapter, and
grateful we are for your patience and assistance!

@dataders Thanks a lot for such a quick fix, all 🟢 ! And don't worry, we all have been there.
Looking at your fixes, the python driver seems like not the easiest piece of software to work with.

hi @dataders, i'm currently facing this blocker. do i need to downgrade the dbt-redshift adapter?


Remove Home page and replace with Extract New page

This PR removes the Home page and replaces it with the Extract New page.
All the paths that directed to /extract have been replaced with / since that is the new home page. If we think that in the future we might want a different landing/home page or we like the descriptiveness of /extract, I can revert those changes and just redirect / to /extract for now. This was another approach I considered and didn't end up going with, but is very possible.

DP2 - Can we change the text on the "Back" button below to something more indicative? Do we even want a back button here anymore? I'd argue we might not, right – since the Config Editor is on it's own page now?

DP1 and DP2 are addressed in 78a1c2f

What is but one more pixel?
Everything


Update requirements.txt

Ah, so this is failing due to a local import, probably fixed by Blazej's #67

Does "local import" include cases where import utils should be import <packagename>.utils?

Fixed by https://github.com/graphcore-research/pyscf-ipu/pull/67


removed all central deepcopies, fixed locally only where needed, and verified that all (including rag end-to-end) runs well, in eager and non-eager mode, including catalog preparation (in both eager modes)

Hi @yoavkatz , is rag-end-to-end the real scenario that you referred at in this comment?
Is there another offending example, that made you add these deep copies, that I can try my fix (in this PR) on?

Hi @elronbandel , all runs well after I changed per your suggestion.  No one is missing "prediction", that was probably when I was still half way through, so all is well. I generated catalog in both eager modes, and all passed well.
I am very curious about this PR running on heavy data, if you have.  Perhaps on heavy data, the expected gain in performance will be noticeable..

So I need to close the PR, or use it to dive into the following problem, that deepcopy can not prevent, and we need to think of another solution:
>>> mi = 4*[{"a":1}]
>>> from copy import deepcopy
>>> mi = deepcopy(mi)
>>> mi
[{'a': 1}, {'a': 1}, {'a': 1}, {'a': 1}]
>>> mi[0]["b"]=2
>>> mi
[{'a': 1, 'b': 2}, {'a': 1, 'b': 2}, {'a': 1, 'b': 2}, {'a': 1, 'b': 2}]
>>>

Specifically in the example what happening is that the list is deep copied from the original list and maintaining its original structure where each element on the list are referring to the same dictionary.
Again if this list is representing instances then the issue is originated in the cross references between instances in the original list.
Now that we identified this as a problem maybe we should protect our users by ensuring each instance is separated from others in every possible instance creation, e.g in loaders, operator testing function operating on instances and so on.
My suggestion is to change the direction of this PR to enforce the rule of no cross reference between instances. Leaving deep copy where necessary (like MapInstanceValues) and adding deep copy where necessary (test_metric, test_operator, produce, DictLoader) and finally removing deep copy when unnecessary (Copy)

I think that any intervention in the interpretation of the code, that does not comply with the legal interpretation of Python should be accompanied by a warning message.
I think that such interventions make a much worse source of agony when debugging.
I just added one for the MetricPipeline

I agree about performance. We can take few ideas from here: https://stackoverflow.com/questions/24756712/deepcopy-is-extremely-slow

What is the offending metric that hits through MapInstanceValues ?

Covered by PR #1119


drivers: i2c_ll_stm32_v2: Handle errors during data transfers

Check for ARLO, BERR, OVR, and NACK errors during any kind of
transmission. Helps fix getting into a while(1) loop in any of these
scenarios when in polling mode.
Signed-off-by: Pushpal Sidhu<EMAIL_ADDRESS>

Codecov Report

Merging #12117 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #12117   +/-   ##
=======================================
  Coverage   48.07%   48.07%
=======================================
  Files         284      284
  Lines       43328    43328
  Branches    10389    10389
=======================================
  Hits        20828    20828
  Misses      18363    18363
  Partials     4137     4137

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07eb5d1...3882a7c. Read the comment docs.


Batch Add Requests

Hello, Your work is appreciated. I loved it. well detailed.
May I know if there is a way to make batch requests by converting your objects into xml?

Hello Rashid,
Thank you for reaching out. You're the first one to do so. :)
I haven't been actively working on this project as it was a way for me to sharpen my C# skills. I didn't write any batch request capabilities into it because the projects I worked on didn't require it. However; it is something I think that can be added. Can you tell me what you want to batch? Is it with the desktop version or the online version?
I played around a little bit this morning and created a AddBatchRqModel class. Below is the test method I created showing how it would work and the resulting XML. Is this what you were thinking you wanted to do?
        [TestMethod]
        public void TestBatchRq()
        {
            var invoices = new List<InvoiceAddRq>()
            {
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 1" }, Other = "Invoice add request 1" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 2" }, Other = "Invoice add request 2" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 3" }, Other = "Invoice add request 3" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 4" }, Other = "Invoice add request 4" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 5" }, Other = "Invoice add request 5" }
            };
            var batchRq = new AddBatchRqModel<InvoiceAddRq>("InvoiceAdd");
            batchRq.SetRequest(invoices, "AddRq");
            Assert.IsTrue(batchRq.ToString().Contains("<InvoiceAdd>"));
            Assert.IsTrue(batchRq.ToString().Contains("</InvoiceAddRq>"));
            Assert.IsTrue(batchRq.ToString().Contains("</QBXMLMsgsRq>"));
        }

<?xml version="1.0" encoding="utf-8"?><?qbxml version="15.0"?>
<QBXML>
	<QBXMLMsgsRq onError="stopOnError">
		<InvoiceAddRq>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 1</FullName>
				</CustomerRef>
				<Other>Invoice add request 1</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 2</FullName>
				</CustomerRef>
				<Other>Invoice add request 2</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 3</FullName>
				</CustomerRef>
				<Other>Invoice add request 3</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 4</FullName>
				</CustomerRef>
				<Other>Invoice add request 4</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 5</FullName>
				</CustomerRef>
				<Other>Invoice add request 5</Other>
			</InvoiceAdd>
		</InvoiceAddRq>
	</QBXMLMsgsRq>
</QBXML>

Let me know.
Thanks

Thank you so much, Greg. I appreciate your work and your instant solution to my request feature <3.
Yes, this is what I was looking for. I'm building an application that has an import feature.
I did some workaround for this. Let me share my code.
    public static class ListExtensions     {         public static string? ToQbxml<T>(this List<T> list)         {             if (list == null || list.Count == 0)             {                 return null;             }             XDocument qbdoc = XDocument.Parse(list.First()!.ToString()!);             XElement addReqBody = qbdoc.Descendants("QBXMLMsgsRq").FirstOrDefault()!;             addReqBody.RemoveNodes();             foreach (var item in list)             {                 XElement requestNodes = XDocument.Parse(item!.ToString()!).Descendants(item.GetType().Name).First();                 addReqBody.Add(requestNodes);             }             return qbdoc.ToString();         }     }
I know this is not an optimal solution. Would like to use your version. Will you be publishing an update on this?

Thank you so much, Greg. I appreciate your work and your instant solution to my request feature <3.
Yes, this is what I was looking for. I'm building an application that has an import feature.
I did some workaround for this. Let me share my code.
public static class ListExtensions
    {
        public static string? ToQbxml<T>(this List<T> list)
        {
            if (list == null || list.Count == 0)
            {
                return null;
            }
            XDocument qbdoc = XDocument.Parse(list.First()!.ToString()!);
            XElement addReqBody = qbdoc.Descendants("QBXMLMsgsRq").FirstOrDefault()!;
            addReqBody.RemoveNodes();
            foreach (var item in list)
            {
                XElement requestNodes = XDocument.Parse(item!.ToString()!).Descendants(item.GetType().Name).First();
                addReqBody.Add(requestNodes);
            }
            return qbdoc.ToString();
        }
    }

I know this is not an optimal solution. Would like to use your version. Will you be publishing an update on this?

Hello Rashid,
Thank you for reaching out. You're the first one to do so. :) I haven't been actively working on this project as it was a way for me to sharpen my C# skills. I didn't write any batch request capabilities into it because the projects I worked on didn't require it. However; it is something I think that can be added. Can you tell me what you want to batch? Is it with the desktop version or the online version?
I played around a little bit this morning and created a AddBatchRqModel class. Below is the test method I created showing how it would work and the resulting XML. Is this what you were thinking you wanted to do?
        [TestMethod]
        public void TestBatchRq()
        {
            var invoices = new List<InvoiceAddRq>()
            {
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 1" }, Other = "Invoice add request 1" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 2" }, Other = "Invoice add request 2" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 3" }, Other = "Invoice add request 3" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 4" }, Other = "Invoice add request 4" },
                new InvoiceAddRq() { Customer = new(){ FullName = "Customer number 5" }, Other = "Invoice add request 5" }
            };
            var batchRq = new AddBatchRqModel<InvoiceAddRq>("InvoiceAdd");
            batchRq.SetRequest(invoices, "AddRq");
            Assert.IsTrue(batchRq.ToString().Contains("<InvoiceAdd>"));
            Assert.IsTrue(batchRq.ToString().Contains("</InvoiceAddRq>"));
            Assert.IsTrue(batchRq.ToString().Contains("</QBXMLMsgsRq>"));
        }

<?xml version="1.0" encoding="utf-8"?><?qbxml version="15.0"?>
<QBXML>
	<QBXMLMsgsRq onError="stopOnError">
		<InvoiceAddRq>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 1</FullName>
				</CustomerRef>
				<Other>Invoice add request 1</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 2</FullName>
				</CustomerRef>
				<Other>Invoice add request 2</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 3</FullName>
				</CustomerRef>
				<Other>Invoice add request 3</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 4</FullName>
				</CustomerRef>
				<Other>Invoice add request 4</Other>
			</InvoiceAdd>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 5</FullName>
				</CustomerRef>
				<Other>Invoice add request 5</Other>
			</InvoiceAdd>
		</InvoiceAddRq>
	</QBXMLMsgsRq>
</QBXML>

Let me know.
Thanks

I think xml needs some wrapping update, InvoiceAddRq should wrap each block individually
<?xml version="1.0" encoding="utf-8"?><?qbxml version="15.0"?>
<QBXML>
	<QBXMLMsgsRq onError="stopOnError">
		<InvoiceAddRq>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 1</FullName>
				</CustomerRef>
				<Other>Invoice add request 1</Other>
			</InvoiceAdd>
                </InvoiceAddRq>
                <InvoiceAddRq>
			<InvoiceAdd>
				<CustomerRef>
					<FullName>Customer number 5</FullName>
				</CustomerRef>
				<Other>Invoice add request 5</Other>
			</InvoiceAdd>
		</InvoiceAddRq>
	</QBXMLMsgsRq>
</QBXML>

I like your workaround. It's a very clean and concise. You are correct with the wrapping xml elements. According to this article on Stackoverflow; https://stackoverflow.com/questions/74210370/what-is-the-correct-structure-for-adding-multiple-time-activities-in-qbxml-quic, the person figured out that Intuit uses wrapping elements on some models but not on others. Intuit is not consistent with that.
I went ahead and added the wrapping elements because I think the majority of the models will require that. If I can't figure out an easy way to implement the logic in code, we may use your workaround for those rare instances :).
The changes I made did not break any of my unit tests but I need to work on it some more to verify that it's working how it should be. Plus, I need to make it work with Mod requests in addition to the Add requests. I will work on it this weekend. In the meantime, I am attaching a beta version for you to try. It's a nupkg that you can install into your local nuget store if you have one. If not, you can unzip the file to get to the dll
Also,if you like and are brave enough, I can give you access to the repo after I work on it this weekend.
Let me know if there are any errors or anything else you need to have changed.
QbModels.1.2.5-beta01.zip

I worked on it a little more. Had to update the base result viewer to read the XML result for all the invoices. I had it hard coded to only read the first result of the batch request. I am attaching the updated beta file that you can use instead of the previous file I uploaded.
I am attaching a quick test method I wrote to verify the results from the batch request for invoices. I use it in Visual Studio to look at the result in Quick Watch to analyze how it looks from a C# perspective.
I also created ModRqBatchModel class for batching mod requests.  I have not had a chance to test to see if it is working or not. I will work on this some more this weekend.
Let me know if you have any questions or if you have any issues.
QbModels.1.2.5-beta04.zip
BatchInvoiceTest.txt

Hello Greg,
Thank you for the above solutions. I will import your dll files into my project. Hopefully, it will work as expected. I'll let you know the results.
Also, I was looking for ModeRq and you have already done it, bro.
Just to let you know, I would be able to test this with sales orders only.
Also would like to know, if these models support custom fields. I haven't tried. Hopefully, it will  be there
Cheers.
Have a great Weekend.

Happy Saturday,
I found an issue with the ModBatchRqModel and fixed it. If you are going to try batching Mod requests, please use the beta v6 version below. Beta v4 generates an error when you try to get the QBXML.
Sales orders is one of the few requests I can't test for (along with ARReundCreditCard, Currency, Estimate, Lead and Workers Comp). My QB does support those features. So you doing Sales Orders is good because you can verify that I implemented the class correctly. If you can verify that everything is working, I will go ahead and publish a non-beta version to NuGet.
QbModels.1.2.5-beta06.zip
Thanks

Hello Greg, I will update you on this once I'm done with testing.
Currently, I'm working on invoices, I need to finish a few things to get it live.


how to save model, i cant load saved llama7b model

Hi Pattaro, could you please provide some more details the to the error you are encountering? What script are you running? What have you tried? What error do you see?


FIX: paintEvent

Locally I was able to reproduce https://github.com/mne-tools/mne-python/pull/8757#issuecomment-774343270 and this small fix did the trick for me. Let's see how it is on Circle.

It works in here too:

https://25098-1301584-gh.circle-artifacts.com/0/dev/auto_tutorials/stats-source-space/plot_stats_cluster_spatio_temporal.html#sphx-glr-auto-tutorials-stats-source-space-plot-stats-cluster-spatio-temporal-py

I think it's worth restarting the CIs, even though processEvents() is not called directly, I don't want to introduce regressions.

This ready on my end.

Thanks @GuillaumeFavelier


RetVal => ReturnValue

https://github.com/AElfProject/AElf/blob/b0a2f525d1cbe291c2d01ff72647044cedb981d1/AElf.Execution/Execution/ITransactionExecutingService.cs#L143

Close as done~


Modern CMake for fun & profit

The current CMake is "pragmatic" which means that it roughly gets the job done somehow and that's pretty much it. It should be time to read some tutorials and actually get some decent modern CMake in this library, especially now that Catch2 provides a modern CMake interface.
Myabe throw in some ideas from here as well: https://github.com/vector-of-bool/vector-of-bool.github.io/blob/master/_drafts/project-layout.md

What's your opinion for Meson?
Personally I find it much more convenient and a lot easier to use than CMake.
For some reason, I cannot fathom CMake...don't ask why :confused:

I don't have any opinion because I never tried Meson.
I simply picked CMake because it's apparently the most used, which makes it easier to find examples and help. Plus it's also the one used by the testing framework used by cpp-sort, Catch2, which also provides a few tools to make tests discovery easier.
Save for a few problems linked to valgrind command line options, the modern-cmake branch is almost finished anyway :p

Not really, I first looked at a few articles, mostly the ones below:

https://pabloariasal.github.io/2018/02/19/its-time-to-do-cmake-right/
https://unclejimbo.github.io/2018/06/08/Modern-CMake-for-Library-Developers/
https://github.com/vector-of-bool/vector-of-bool.github.io/blob/master/_drafts/project-layout.md

I then compared this new-found knowledge to other header-only projects, tried to get a first semi-working version for this library then solved the specific problems I had one one, by looking at StackOverflow questions & answers, or old forum posts or mailing lists, or by checking how other projects solved similar problems. CMake has a complete reference, but it is true that it lacks examples of how to use things properly. More often than not I end up looking for solutions on the internet and/or trying things that seem to make sense.

Closed by #137, which roughly does the following:

Properly create, use and export CMake target
Install CMake files
Download Catch2 during the configuration step, not the build step
Automatically discover Catch2 tests instead of having a single test for the whole test suite
Use Valgrind through CTest memcheck step


Support for defaultProps will be removed

Using the "react-native-leaflet-view": "^0.1.2", I'm getting the following warning.

Warning: LeafletView: Support for defaultProps will be removed from function components in a future major release. Use JavaScript default parameters instead.

The map is rendered normally though.

Hey, I created a PR for this, hope it will be approved and merged soon.

@pavel-corsaghin can you turn this main branch into a new release?


Go to definition not working

Getting this message when trying to right click and chose "Go to definition". The class is defined in another file in the same project. I have also tried it with classes defined in the same file, same error.

@LinusU did you use any import statements, that you don't get any completions or hovers for?
In that case this may be related to another issue, where sourcekit does not emit type information for (some) imports.

Nothing other than UIKit, the file was simply another file in the same project. Oh, I should probably have mentioned, this is not a SPM project, but an Xcode one, maybe that's the culprint?

I have fixed the problem I meant, but your problem seems to be a different one.
There were two issues:

Imports were not working correctly. (Mostly affected SPM)
Your issue. (Probably: your swift files are not indexed together)

Awesome as always! Thank you ❤

Will no longer write sde.buildOnSave or editor.quickSuggestions to workspace settings

🙏 🙏 🙏 🙏

Released as 2.3.1.
Will now scan the repo for swift files on startup.


Smarthings Slow Action

Home Assistant release with the issue:

0.87.1
Last working Home Assistant release (if known):
Current, this has not been attempted yet. the integration was installed on Friday
Operating environment (Hass.io/Docker/Windows/etc.):

Unraid with the official docker Home-assistant
ssl and proxy are handled by Nginx Proxy Manager and is proxying HTTPS to HTTP on the HA side
Component/platform:

https://www.home-assistant.io/components/smartthings/
Description of problem:
Finally got SSL setup yesterday and figured I'd take advantage of the new integration. The two systems connect as expected but updates from HA to ST take a repeatable 5-7 seconds before a light will turn on.  If changing the status from ST then HA will update within a second.  I have tried to remove and reinstall the integration with no change.  Please let me know which logs or other info I can provide to help resolve this.  Thanks for all the work.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:

To clarify the issue, @jrh1812 was seeing:

Action in ST (turn lights on) -> HASS updates as expected
Action in HASS (turn lights on) -> 5-7 delay before light actually turn on in ST

After some troubleshooting, we determined it was an issue in the DNS config.  Name resolution was adding approx 8 seconds.  Once changed to a different DNS server, the problem went away!


fix: Initial delay e2e test use a scaler without failures

I think that after moving the cron scaler to the declarative parsing, 0 is treated as error and this test failed because during errors, the scaling to 0 doesn't apply. This PR changes the scaler to other that returns 0 without errors
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)

/run-e2e internal


Should navto search opening projects like find all refs to give better info about world or just existing open projects

Yes VS Code now searches all open projects by default (on TS 3.9+). My understand is that this should only search the already opened project and not trigger a new project loads. You can disable this behavior with typescript.workspaceSymbols.scope .
I made this the default because I felt searching all projects made the most sense for the way we describe go to symbol in workspace. It is useful in the VS Code codebase for example because we usually want search results from both our extensions and our core codebase.
@amcasey VS Code cannot stream in results. We have to return them all in a single response
Originally posted by @mjbvz in https://github.com/microsoft/TypeScript/issues/38491#issuecomment-627599632

cc: @amcasey  @mjbvz @RyanCavanaugh @DanielRosenwasser @sandersn

Thanks for filing this!


[frontend] Wire up auth flows

Closes #109
New behaviors:

login, register, and passport forms have validation and submission flows
login and register forms are now connected to the backend API. The site is unusable without a running backend.
/login and /register routes redirect to /dashboard if the user is authenticated
/dashboard and /passport routes redirect to /login if the user is not authenticated, logs out, or their access token expires
npm run dev by default mocks out the backend. Users can log in with the<EMAIL_ADDRESS>
Other Changes:

Create an initial test user in then backend development environment. email<EMAIL_ADDRESS>password password
Set up CORS to allow all origins. This is necessary at least for the tests.
Use react-hook-form for form input management.
Simplify some of the input validation patterns as I couldn't figure out the required format and the error messages didn't clarify
Create an AuthProvider to expose auth state/methods to components. auth info is stored in localStorage.
Add a vscode workspace primarily used for debugging the frontend.

Thanks for the review @acharliekelly, and great question! We don't currently include the role in the User response. Created #155 to add it.
We haven't yet built or specified the passport flow. Who approves, and how? Admins in an admin interface? What happens if a user is rejected? Can they re-apply? We can discuss Tuesday. Adding role-related functionality including the passport flow could be its own milestone.


trigger build

trigger build

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Some questuions of transform: It looks like that i lost some images after transform(Modified the source code for R-FCN)

hi everyone:
when i run the CaffeOnSpak, i modified some code in net.cpp , so i can see which image sended in Convolution layer，but there is a probleam for me. it looks like  i lost some image after transform                 case CoSDataParameter.DataType.RAW_IMAGE |                      CoSDataParameter.DataType.ENCODED_IMAGE |                      CoSDataParameter.DataType.ENCODED_IMAGE_WITH_DIM=> {                   if (transformers(i) != null) {                     transformers(i).transform(dataArray(i).asInstanceOf[MatVector], data(i)) }. For example: when i run minist in ordinary Caffe，the image number is 0,1,2,3,4,5,6,7,8,9, the label also is 0,1,2,3,4,5,6,7,8,9. But when i run CaffeOnSpark the image number is 2,3,4,5,6,7,8,9,0 the label is 0,1,,2,3,4,5,6,7,8,9(I have modified the source code, i want to run R-FCN on CaffeOnSpark,i have solve cll most problems , but this problem confused me),can you give some suggstions ? thank you !

check your input data frame first, are the images and labels aligned?

@junshi15 thanks for you answer, i checked the images, and i can make sure that images and labels are aligned. And i find that when i adjust the test_iter and test_interval set to 1, it is wrong with my images , it will lost 2 images at first, then i adjust the test_iter set to 1 and the test_interval set to 10, it is right at first with my images,after 10 images,when it need a test image,it is going wrong again,one image and one label is not on.After this one test image, the images and labels are aligned again(it means after a test image it will wrong with image and label).Do you know waht‘s wrong with that?

not sure about the validation. what happens if you set test_iter and test_interval to zero, which will disable the validation path.

@junshi15 I tried set test_iter and test_interval to zero,but it can't work successful,it's wrong,the wrong infomation:
java.lang.UnsupportedOperationException: empty.reduceLeft
at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:180)
at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1336)
at scala.collection.TraversableOnce$class.reduce(TraversableOnce.scala:208)
at scala.collection.AbstractIterator.reduce(Iterator.scala:1336)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$8.apply(CaffeOnSpark.scala:226)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$8.apply(CaffeOnSpark.scala:213)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:766)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:766)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
my net.prototxt is:
layer {
name: "data"
type: "DFAnnoData"
top: "data"
top: "gt_boxes"
include {
phase: TRAIN
}
source_class: "com.yahoo.ml.caffe.AnnoDataFrameSource"
data_param {
source: "file:/home/cos/R-FCN/0814_lmdb_DF"
batch_size: 1
}
augment_param {
mirror: true
mean_value: 104
mean_value: 117
mean_value: 123
resize_param {
  prob: 1
  resize_mode: WARP
  height: 1000
  width: 1778
  interp_mode: LINEAR
  interp_mode: AREA
  interp_mode: NEAREST
  interp_mode: CUBIC
  interp_mode: LANCZOS4
}
}

}
layer{
name: "data"
type: "DFAnnoData"
top: "data"
top: "gt_boxes"
include{
phase: TEST
}
source_class: "com.yahoo.ml.caffe.AnnoDataFrameSource"
data_param {
source: "file:/home/cos/R-FCN/0814_lmdb_DF"
batch_size: 1
}
augment_param {
mirror: true
mean_value: 104
mean_value: 117
mean_value: 123
resize_param {
 prob: 1
 resize_mode: WARP
  height: 1000
  width: 1778
  interp_mode: LINEAR
  interp_mode: AREA
  interp_mode: NEAREST
  interp_mode: CUBIC
  interp_mode: LANCZOS4
}

}
}
my solver.prototxt is:
The train/test net protocol buffer definition
net: "/home/cos/R-FCN/R-FCN_train_DF_5.prototxt"
test_iter specifies how many forward passes the test should carry out.
test_iter: 0
Carry out testing every 100 training iterations.
test_interval: 0
base_lr: 0.001
lr_policy: "step"
gamma: 0.01
stepsize: 16
display: 1
momentum: 0.9
weight_decay: 0.0005
We disable standard caffe solver snapshotting and implement our own snapshot
function
max_iter: 40
snapshot: 50
We still use the snapshot prefix, though
snapshot_prefix: "R-FCN_"
iter_size: 1
solver mode: CPU or GPU
solver_mode: GPU
Do you know how to solve this probleam , Thanks.


Contextual information (entity)

It is possible to modify all the parameters (create/delete of Contextual Infomation) and it is also possible to set parent/children relations even if the entity is associated to a Risk Evaluation

what is the problem? The user should be able to modify the entity when a risk session is created? we have issues asking for renaming entities (#93), even risk configuration already exist.


Update l10n scripts and update .po files with the new script

Update localization utility scripts and update .po files with the new scripts.
This change is necessary for localized sites support of search.quarkus.io
https://github.com/quarkusio/search.quarkus.io/issues/33
https://github.com/quarkusio/search.quarkus.io/pull/71

The build failed because the GitHub Action uses build script stored in the main branch.
Once this pull-request is merged, the GitHub Action works again.
This failure can be safely ignored.

Thanks @ynojima and can you fix these conflicts before merging?

Wow. lots of conflicts :)
I have resolved the conflicts. Please merge before it happens again :)


added note about  certutil -dsTemplate command

#5869

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks.

@mapalko @Dansimp - PR has been copy edited and is ready for final review. This is based on issue #5869.
Thank you.
cc: @kenwith

Following up with team on issue.

@mapalko - Do you have any updates on this?

#sign-off

@mapalko : When time permits, will you add your approval tag to this PR as well?


Angular 2 - Create a new github project

Angular 2 beta was released we should create or migrate this plugin to angular 2. @isteven

@danicomas ,
Yeah this is a priority now.. Thanks for reminding. Any good resource or example on how directive is defined in Angular 2?

@isteven I did a component example in this blog https://www.ingens-networks.com/blog/post/2015/10/05/angular-2-components-creando-un-modal.aspx (Spanish) but you can find other examples like too: https://github.com/thelgevold/angular-2-samples/tree/master/components

I honestly have limited time but I'd be interested in being involved. Haven't done any Angular 2 development yet but I've been trying to pay close attention to the patterns.

@isteven any work started on this?

@isteven - any update on this?

+1

Any update on migrating this plugin to Angular 2 and above?

@ferrarikarthik @ricetrac @msrkrishnan @danicomas I've created a multiselect plugin similar as that of isteven at least in terms of look and feel. Please have a look at it once.
https://github.com/ngx-lib/multiselect

@pankajparkar - good work, it looks good. Couple of suggestions:

If all items are selected, provide an option to show label as "All"
Also show number of items selected in label
Is it possible to customize Tick mark?

@msrkrishnan thanks for your suggestion, I'll create an issue for each of the point and start working on it. Let me know if you're interested to contribution.
Followup question: I didn't get the 1st point, particularly provide an option to show label as "All"

If all the items in the drop down is selected, then we can just show "All Selected" or a custom label. It should be based on options/config. If user wants to see selected item names, leave as it is like Option, Option2, etc.,


migrate to pyproject.toml

This will close https://github.com/machow/quartodoc/issues/232

[x] Move dependencies to pyproject.toml
[x] Move our flake8 config into a file called .flake8
[x] Ensure our console script entrypoints still work (e.g. can run quartodoc from CLI)
[x] Install and run pre-commit. Let's copy over the code-checks CI from pins-python to be safe.
[x]  adding a git tag, then building produces dist/quartodoc-<version_tag>.whl See our build process on CI

BTW I had to pin the version of Griffe because it was causing a CI failure otherwise.  Let's discuss this.
@machow we can walk through this if you like.  Sorry for the large diff, when I activated the pre-commit hooks and put them in CI as you requested it applied formatting changes across the entire library.

@machow we should merge this PR first then merge #161

Thanks--I'm just getting back into the swing of things, and taking a look now! pre-commit reformatting a bunch of things to get the new CI check passing is a good sign! (and hopefully with the CI in place reformatting across the code base should only be a one time thing).


More than 5 choice options in dialogues

Dialogue cutscene can only support around 5 choice options based on the scene size. Add a UI that can allow scrolling through options that go beyond 5.

Extra: in dialogue editor, allow removing any choice option, not just the last one

Extra: provide built-in functions for FUNCTION node? Example: add / sub / mul / div / etc

Extra: ctrl+c, ctrl+v on nodes?

Extra: dialogue editor, allow exiting the cutscene from editor early.

Extra: init Function node, which is only called once to initialise all local variables

Extra: handle case where all options fail to pass conditions so there are no options to choose.


PendingChannel does not update confirmation_height

Background
When opening a channel and channel is still in pending, the confirmation_height is not updated even if new blocks are found and stays at 0.
Your environment

version of lnd:  v0.14.3-beta.rc1
version of btcd, bitcoind, or other backend: btcd and bitcoin (both did not work)
any other relevant environment details

Steps to reproduce
Open a channel and wait until first blocks are confirmed then check with lncli pendingchannels and verify that the confirmation_height is not updated and stays 0
Expected behaviour
show blockheight when channel will be mature and locked in
Actual behaviour
confirmation_height remains 0 the whole time

 "pending_open_channels": [
        {
            "channel": {
                "remote_node_pub": "03ceddba52e1dbbc2927defa9c695166a9890937915453aec00c8d89d33ddeea66",
                "channel_point": "5b00704af112f5a8506cec2eae335f6813cb1880a02dc4b89461ee93336c6303:0",
                "capacity": "888877",
                "local_balance": "885407",
                "remote_balance": "0",
                "local_chan_reserve_sat": "8888",
                "remote_chan_reserve_sat": "8888",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "",
                "private": false
            },
            "confirmation_height": 0,
            "commit_fee": "2810",
            "commit_weight": "772",
            "fee_per_kw": "2500"
        }
    ],

even if I mined 2 blocks after opening the channel and being synced to chain

Duplicate of https://github.com/lightningnetwork/lnd/issues/1217

Also see: https://github.com/lightningnetwork/lnd/pull/6472
Likely also want to mark it as deprecated for now potentially


Convert LuCI JS code to ES6 standard

This PR contains commits dropping polyfills not needed anymore on modern browsers and refactoring code to utilize ES6 syntax elements and features.

With this PR, is it your opinion that we can start implementing ES6 for future JS code?

The luci already have many places where the ES6 syntax is used.
Keep sources simple and small is complicated so we better to move to ES6.
Also please consider to use the JavaScript Standard Style e.g indent with spaces and no semicolons.
I can convert to ES6 other apps.

I dislike semicolon omission and space indentation, so personally I am opposed to standard style.


Functional API document auto generation

🚀 The feature
Currently, we rely on the inline doc for each DataPipe to indicate the functional API. Considering torchdata should have already been built when doc need to be generated, we should be able to rely on the following dictionaries to figure out the functional API with corresponding DataPipe.

IterDataPipe: https://github.com/pytorch/pytorch/blob/b30c027abf6560dafb88d67ebd446851a5729651/torch/utils/data/datapipes/datapipe.py#L68
MapDataPipe: https://github.com/pytorch/pytorch/blob/ea8a0184b76088c477c7cef4fe36d683d57fd880/torch/utils/data/datapipes/datapipe.py#L194

Motivation, pitch
Make the doc-gen more intelligent to reduce the amount of work when contributors implement DataPipe and the extra care for reviewers/
Alternatives
No response
Additional context
WDYT @NivekT

I think this is certainly possible. There are a few paths we can go down:

Stay with the existing layout and inject the functional API names into the docstrings via a pre-processing function in conf.py during the generation process rather than having contributors explicitly add it to docstring
As we previously discussed, create a separate table/page with functional API

I think the priority is relatively low since I don't anticipate adding too many more DataPipes (unless the current doc layout is unhelpful for users and need to revamp)? Nonetheless, this is still worth thinking about and tracking.

Agree on the priority. We can spend a few days for BE next half~

I want to add another potential improvement for pyi gen.
Currently, the type hint for return value of each functional API is either IterDataPipe or MapDataPipe. We could change the type to specific DataPipe. For example:
class IterDataPipe:
    def shuffler(self, *, buffer_size, unbatch_level) -> ShufflerIterDataPipe  # Change it from IterDataPipe

I want to add another potential improvement for pyi gen. Currently, the type hint for return value of each functional API is either IterDataPipe or MapDataPipe. We could change the type to each specific DataPipe class. For example:

I think this is a good idea, though import statements at the top might get a bit long.


fix: add image in gan post

https://deploy-preview-53--pytorch-ignite.netlify.app/posts/

@ydcjeff it looks good to me ! Thanks !
However, I checked the rendering of the notebook using my phone device (safari browser) and I have noticed a problem concerning the code cells rendering. It appears that the font size is not uniform and some cells have larger size than others. Although the digit's size of the code lines seems correct.
HTH


Element type of empty array?

With your mirror framework, is it possible to get the Element type of an empty array?
let a:Array = []
And then get String somehow?

There was a lot of changes to Reflection API in Swift. The Mirror framework unfortunately haven't been updated.
the element  of Array is required when you create it. Also you can use is or as operator to check what types  are stored in array
let a: Array<Int> = []
if a is [Int] {
    print(" It's [Int]")
}

I hope this could help you

Thanks for your swift reply, pun intended.
Yes, you can derive the array element type if the array is assigned to a variable. But my efforts with swift Mirror doesnt allow this. You can derive the element type of an empty array by doing Array().dynamicType.Element().dynamicType// Int. But if you use Mirror this just doesn't work.
I guess my question is, can your Mirror framework extract array Element type for empty arrays.
Side note: attemting to make a class that can store classes in an xml doc

Anyways, I know you are very busy. Ive tested everything i can think of. And what it came down to is that the type that Mirror returns isn't useable as say a generic type in swift. So the only viable option was to keep reflecting variables if they were of type of type: Array. And use Regular Expression to extract the array type if the array was empty.


improve docker setup

[ ] Better mutli stage setup
[x] Docker Healthcheck
[ ] Non root user
[x] Dockerignore

Install requirements in root ?

The non root user right now is annoying to setup due to this https://github.com/just-containers/s6-overlay#notes


libdl.so is not available on many Linux distros, causing crashes on native library load

Summary
Many Linux distros do not have libdl.so, rather they only have libdl.so.2. This causes a DllNotFoundException when using its native functions.
This would only occur when using .NET Framework / .NET Standard, as .NET Core / .NET 5+ just use System.Runtime.InteropServices.NativeLibrary instead of P/Invoking libdl directly.
According to the loader readme, the code was taken from NativeLibraryLoader, which seems to have this issue fixed already, so it's probably as simple as porting over that fix.
Steps to reproduce

Platform: Linux (Ubuntu 22.04 was used for this case)
Framework Version: .NET Framework / .NET Standard (.NET Core / .NET 5+ is not affected)

Do anything which loads in a native library (e.g. Sdl.GetApi).

Thanks for letting us know. I have looked at the fix and concluded that this isn't an appropriate fix for Silk.NET, namely the try-and-catch method tends to create a poor user experience given that users are encouraged to have first chance exception handling on when debugging their applications. As you have identified that this is working on the newer .NET versions, I don't think we will take any action here. Please use the latest .NET.

To be clear, upgrading to the latest .NET version is not possible at this time for my use case. I've been looking into Silk.NET to provide bindings for use in the BizHawk project, which is stuck on .NET Framework due to several .NET Framework only dependencies (e.g. OpenTK for OpenAL/OpenGL bindings), so we've been moving away from dependencies one by one by switching them to .NET Standard 2.0, allowing us to easily upgrade when we no longer have .NET Framework dependencies.
Considering that most (if not all?) Linux distros seem to just have libdl.so.2, perhaps it'd be fair to just switch the p/invoke to that?

I would be happy switching exclusively to just libdl.so.2 this is what osu-framework does, and i havent heard of/experienced any problems with that, feel free to open a PR changing that/come onto the discord to discuss more, if you are sure it wont cause any problems


Unclassified Error for Application with JWT token type in store -APIM 3.0.0

Getting an error as "Unclassified error" for an application having default token type JWT.
steps to reproduce.

Create an application with default JWT token type in devportal.
Register API  to the application.
try to consume the API with default JWT token in devportal.
Getting error as "Unclassified error".

I Have The Same Problem!!!!!

This is no longer applicable since all applications are JWT applications


Prompt fusion change seeds even if prompt has not any key words

I noticed that the extension changes the seeds a lot even when I don’t use keywords and the extension is turned off in the settings. The only thing that helps is completely disabling the entire extension in the extensions panel.

Can you show an example of what it means to change the seed? Currently I am not sure if you are referring to the seed of the generation in the PNG info metadata, or the hash of the generated image, or something else.
I'm asking this because the extension does not fiddle at all with the seed saved in the PNG info in any way.

What I mean is that the extension affects the determinism of the seeds. When enabled, conditions become the same size (for example 154) and this affects the result. However, if you turn off the extension in settings (option prompt_fusion_enabled), the influence becomes less, but is still present. It only disappears if you turn off the extension completely. I debugged the code but did not find the reason why this is happening.
If you need example images to understand what I mean, I can send them a little later.

That is odd. This is the kind of thing I test before merging code. Let me take a look.

Apologies for the delay, I was just able to get to this. I am unable to reproduce this on forge. Can you provide which webui alternative you are using?
I suspect it maybe be related to this as, AFAIK, in A1111 and forge it does not affect the image to simply have the extension enabled. In particular, if you look at the code, the extension returns by using the original callback as soon as it can if it is soft disabled.

I'm using A1111. I saw that the extension returns the original callback and so I don't understand the reason for this error. Also, I haven't noticed any impact from other extensions on this issue.
I use this CMD args in webui-user.bat:
--api --opt-sdp-attention --opt-split-attention --opt-sub-quad-attention --xformers --reinstall-xformers --medvram --no-half-vae --no-hashing --disable-nan-check --allow-code --no-download-sd-model --administrator --disable-safe-unpickle
Maybe --opt-sdp-attention, --opt-split-attention, --opt-sub-quad-attention or --xformers might have some impact

Thanks for the flags, I'll take another look at this eventually (unfortunately I'm not sure when I'll be able to do that exactly at the moment)

I'll let you know if I find the reason or can fix it.


please include LICENSE file in published crates

The MIT license requires that redistributed sources contain the license text. The git repository for strong-xml contains a LICENSE file, but since this file is only in the Cargo workspace root and not in the actual "crate subdirectories", they will not be included when publishing sources to crates.io with "cargo publish".
The simplest solution for this is usually to create symbolic links in the subdirectories (i.e. LICENSE → ../LICENSE), assuming you're running "cargo publish" on unix-like systems (mac OS or Linux). On windows, there's no equivalent to functional symlinks, so there the only solution would be to copy the LICENSE file into the subdirectories for all published crates.

Thanks you for report this, I just published v0.6.3 which includes MIT license file.

Thanks! :)


Tweak DAL model handling & validation

Improve existing DAL attribute codec immutability (BE, FE)
After a codec is in place it's mutability is limited since we don't have any migration logic yet.
For example, a values codec can't be replaced by a column codec.
Currently, if there is a mismatch, we error out and that's that.
Server file ref: store/adapters/rdbms/dal/connection.go

[x] On the back-end, allow the attribute codec to change if there are no records under that module (same as with field names and kinds).
[ ] On the front-end, prevent the user from changing the codec in case the field was already created and there are records for the module.
[ ] Consider improving the UI/UX to help prevent mistakes like users saving the module before defining the codec.

Allow models to lose attributes without modifying the database (BE)
Currently, there is a check in place which errors out in case there is an attribute mismatch (removing attributes).
DAL errors out because destructive operations are currently not supported.
Server file ref: store/adapters/rdbms/dal/connection.go

[x] On the back-end, allow users to remove model attributes (module fields) without trying to alter the database

Janky module issue reporting
Modules and DAL models are two separate bits. Whenever we perform an operation on the module which would pose issues on the model, that issue is noted in the model and then reported to the module.
Here are some issues I've found:
When deleting a module field, I only see the issue after I refresh the page (BE, FE)

The module

Field removed

Saved

Refreshed

This will most likely be sorted in the next sprint

If you change the connection on an existing module you also get an error.

I'm guessing we don't remove the old version

Attribute modifications are now permitted; DB reflects changes only if the DB_ALLOW_DESTRUCTIVE_SCHEMA_CHANGES .env variable is set.

While implementing the bits above it got clear that it'll make more sense to re-do DB altering logic instead of (just) fiddling around with validation.
Currently, DB modifications are too limited to support all of what module definitions can do. Changing up the validation would just be a bandage at this point (it'll also probably have to change when we re-do DDL).
I'll get in touch with a few devs tomorrow (Monday) to brainstorm our two options (mainly implicit and explicit modifications) and I'll make this my priority through this sprint.

for today: write specs

Replaced with https://github.com/cortezaproject/corteza/issues/1200


issue related with_client_hello_handler

Security issue notifications
Don't think this is a security issue
Problem:
I am trying to load cert dynamically during the client hello instead of using a fixed cert. The way I did was to create a new config during client hello and call the method set_config on the connection with the new cert. However, this doesn't seem to work properly. See below commit for the minimal error case to reproduce

https://github.com/LMJW/s2n-quic/commit/86a6493f1f63d516c4afd3be8a22c5d8fbf13a76
https://github.com/LMJW/s2n-quic/commit/a10d04d1b7e4c49abd2468fd483f80d2cca49301

Solution:
I am not aware if there are some work around here.
Requirements / Acceptance Criteria:
Fix this or provide mitigation
Out of scope:
N/A

You need to construct your TLS config with a valid setup for quic, as outlined here
https://github.com/aws/s2n-quic/blob/336f49dca052e6fc49d754e61f1017d422eff7da/quic/s2n-quic-tls/src/server.rs#L39-L43
You can see an example of how to do that here: https://github.com/aws/s2n-quic/blob/336f49dca052e6fc49d754e61f1017d422eff7da/quic/s2n-quic-tls/src/server.rs#L75-L79
I think it would probably be simpler to use the Server::builder and then call ConfigLoader::load on it. We'll need to do a release before the loader method is available, though, as it was just merged in #1558.

Thanks @camshaft . I will try it out. Hopefully this can be released soon

Hi @camshaft . I have tried what you suggested to create config as in your previous example. It seems the handshake is still not successful. See my commit https://github.com/aws/s2n-quic/commit/8c1d66d3be0c361904d1002d102b86cecb68574e

The from_loader api will return a Server rather than Builder. In that case, I won't be able to set the with_client_hello_handler on that, right?

Thanks for adding the support @camshaft.  Looks with the new ConfigLoader trait, I am able to load the cert with loader after fixing the trait bound. https://github.com/aws/s2n-quic/commit/a0b475fb8eca5b86d227584175c574c156777043 (working commit)
Should I use ConfigLoader instead of the Callback for my purpose? It seems to me the Loader have less context  information comparing to the Callback. I am not sure if in the future, the information I need might missed from the context. Also, it would also good for the trait to return Result

The ConfigLoader isn't meant to be async. That's what the hello callback is for. You'll need to do something like this
pub struct MyClientHelloHandler{}

impl ClientHelloCallback for MyClientHelloHandler {
    fn poll_client_hello(&self, conn: &mut Connection) -> core::task::Poll<Result<(),s2n_tls::error::Error>> {
        let mut config = s2n_quic::provider::tls::default::Server::builder()
            .with_certificate(CERT_PEM.as_bytes(), KEY_PEM.as_bytes()).unwrap()
            .build().unwrap()
            .load(ConnectionContext { .. }); // I think we'll need a way to construct this, as it's currently non-exhaustive

        conn.set_config(config).unwrap();
        println!("Set config done");
        conn.server_name_extension_used(); // If I use sni to config the connection, I need to call this.
        Poll::Ready(Ok(()))
    }
}

Thanks @camshaft  for your update. Yeah, I think for now, I am not able to construct ConnectionContext

https://github.com/aws/s2n-tls/blob/main/bindings/rust/s2n-tls/src/testing/s2n_tls.rs#L353
calling config.set_config() is enough to reproduce this issue.
We should also add config.set_config() to the async callback: https://github.com/aws/s2n-tls/blob/d8a779a23ea08dd618d3d05444d7b5a9fef905dc/bindings/rust/s2n-tls/src/testing.rs#L230

LMJW, what was the exact error that you were seeing, once the fixes Cameron suggested were applied? Do you have access to the underlying s2n-tls errors?
I was able to trigger errors in our s2n-tls bindings tests by adding config.set_config to the callbacks in the tests. I saw different errors for sync and async callbacks:
For sync:
Error { code: 469762110, name: "S2N_ERR_INVALID_STATE", message: "Invalid state, this is the result of invalid use of an API. Check the API documentation for the function that raised this error for more info", kind: UsageError, source: Library, debug: "Error encountered in lib/tls/s2n_client_hello.c:113", errno: "Success" }
For async:
Error { code: 335544374, name: "S2N_ERR_CANCELLED", message: "handshake was cancelled", kind: ProtocolError, source: Library, debug: "Error encountered in lib/tls/s2n_client_hello.c:501", errno: "Success" }

If the corresponding sync or async error matches what you were seeing, then a workaround is to make sure that the new config sets the same client hello callback that the old config did. For example, from the client hello callback implementation, you might do:
config.set_client_hello_callback(self.clone())

That resolved the errors I saw in our tests.

Hi @lrstewart Thanks for coming back to me.
After try your work around, I think it worked. The work commit is https://github.com/aws/s2n-quic/commit/b4cee0dbd8e4533f3f00912f048337ae9400011f?diff=split
Looks to me the root cause is because the config was not created with correct setting. I think I can go with the current work around.

reated with correct setting. I think I can go wit

LMJW, what was the exact error that you were seeing, once the fixes Cameron suggested were applied? Do you have access to the underlying s2n-tls errors?
I was able to trigger errors in our s2n-tls bindings tests by adding config.set_config to the callbacks in the tests. I saw different errors for sync and async callbacks:
For sync:
Error { code: 469762110, name: "S2N_ERR_INVALID_STATE", message: "Invalid state, this is the result of invalid use of an API. Check the API documentation for the function that raised this error for more info", kind: UsageError, source: Library, debug: "Error encountered in lib/tls/s2n_client_hello.c:113", errno: "Success" }
For async:
Error { code: 335544374, name: "S2N_ERR_CANCELLED", message: "handshake was cancelled", kind: ProtocolError, source: Library, debug: "Error encountered in lib/tls/s2n_client_hello.c:501", errno: "Success" }

If the corresponding sync or async error matches what you were seeing, then a workaround is to make sure that the new config sets the same client hello callback that the old config did. For example, from the client hello callback implementation, you might do:
config.set_client_hello_callback(self.clone())

That resolved the errors I saw in our tests.

I have no access to the internal error messages so I am not sure what was the error message. But I think with your work around, my test is working now. Thanks

Glad I could unblock you!
The root cause is that, for legacy reasons, s2n-tls defaults to assuming that the client hello callback is blocking. In our Rust bindings we set configs to use non-blocking client hello callback mode when the client hello callback is set, but if you don't set the callback (like you didn't on the new config) then the old C logic assumes blocking mode and handles the callback's return value incorrectly.
The bindings should just configure all configs to use non-blocking mode, even if they don't set a client hello callback. We'll put out a fix.


Fixed depreciated inspect function

getfullargspec is a drop in replacement for getargspec, see docs for details

Codecov Report

Merging #396 (fca6d57) into master (017c738) will decrease coverage by 17.25%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master     #396       +/-   ##
===========================================
- Coverage   97.32%   80.07%   -17.26%
===========================================
  Files          49       49
  Lines        3142     3142
  Branches      585      585
===========================================
- Hits         3058     2516      -542
- Misses         44      581      +537
- Partials       40       45        +5

Impacted Files
Coverage Δ

eli5/base_utils.py
90.00% <100.00%> (-10.00%)
:arrow_down:

eli5/keras/gradcam.py
0.00% <0.00%> (-100.00%)
:arrow_down:

eli5/xgboost.py
4.26% <0.00%> (-95.13%)
:arrow_down:

eli5/formatters/image.py
5.47% <0.00%> (-94.53%)
:arrow_down:

eli5/keras/explain_prediction.py
4.81% <0.00%> (-91.57%)
:arrow_down:

eli5/lightgbm.py
4.27% <0.00%> (-90.60%)
:arrow_down:

eli5/catboost.py
11.53% <0.00%> (-88.47%)
:arrow_down:

eli5/_decision_path.py
35.48% <0.00%> (-64.52%)
:arrow_down:

eli5/keras/__init__.py
50.00% <0.00%> (-50.00%)
:arrow_down:

eli5/ipython.py
85.71% <0.00%> (-14.29%)
:arrow_down:

... and 12 more

Surely this is due to a change in tests not related to this PR

Thanks you! This were merged in eli5-org/eli5#2 and released to PyPI with v0.11


Reorganizing examples

Closes #563. As discussed there, reorganizing examples by categories, so it is easier to find the examples certain audiences may be interested in.

[x] Entries added to the respective CHANGELOG.md for any new functionality

What about the readme thomas suggested?

What about the readme thomas suggested?

Good point. I will open a follow-up issue, so this one can be merged right away.


fix: don't force reauth when calling GetRoleARN

The call to p.GetRoleArn() added in https://github.com/segmentio/aws-okta/commit/396d453 was forcing a separate auth event. That broke session caching in 0.23.0 and 0.23.1.
So the first run of aws-okta would auth twice, including a second 2FA prompt. Every subsequent call to aws-okta then also required 2FA confirmation.
This PR modifies GetRoleARN to take the creds as an argument.  Those creds are fetched immediately before anyways.
This also adds the AWS_OKTA_ASSUMED_ROLE_ARN and AWS_OKTA_ASSUMED_ROLE environment variables to the env subcommand so that it is consistent with exec.
This also addresses the issue reported in segmentio/aws-okta#213

This is awesome @brandt. Would you mind splitting the bugfix and adding the env thing in a separate PR please?

Ah, I can kind of see how the env.go change is also a bugfix. I'm going to go ahead and add to your PR

Couldn't push to your branch for whatever reason so I just opened a new PR https://github.com/segmentio/aws-okta/pull/217

@brandt I'm not sure it necessarily addresses the same issue that my change (#213) fixes. This change appears to fix reuse of the root AWS session whereas my change fixes reuse of the Okta session.
This is an important distinction because in my environment the max session length for AWS is 1 hour whereas for Okta it's 10 hours.

@Chippiewill I agree – I'll submit a new PR with different wording in the commit message so that GitHub doesn't automatically close your PR.
@nickatsegment I'll incorporate your changes from #217 as well.

Closing in favor of #218


WSL2: NTFS symlinks in $PATH cause EPERM and breaks execvp()

Please fill out the below information:

Your Windows build number:  19.03 build 18945.1001

What you're doing and what's happening:  Type a command which is not in the linux $PATH, and path searching fails  "operation not permitted". Example:

% asdfasdf
zsh: operation not permitted: asdfasdf

What's wrong / what should be happening instead: Should be reporting "No such file or directory" or similar:

# A short linux-only path:
% PATH=/bin sh -c 'asdf'
sh: 1: asdf: not found

Strace of the failing command, if applicable:

% strace  sh -c 'example'
...
stat("/home/jls/bin/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/home/jls/local/bin/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
<omitted for brevity>
stat("/bin/example", 0x7ffda8ae8670)    = -1 ENOENT (No such file or directory)
stat("/usr/games/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/usr/local/games/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/ConEmu/ConEmu/Scripts/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/ConEmu/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/ConEmu/ConEmu/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath/example", 0x7ffda8ae8670) = -1 EPERM (Operation not permitted)
stat("/mnt/c/WINDOWS/system32/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/WINDOWS/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
<omitted for brevity>
stat("/mnt/c/Program Files/Git/cmd/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/PuTTY//example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
<omitted for brevity>
stat("/home/jls/go/bin/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
write(2, "sh: 1: ", 7)                  = 7
write(2, "example: Operation not permitted", 32) = 32
write(2, "\n", 1)                       = 1
exit_group(127)                         = ?

All of these errors, 32 are ENOENT and 1 is EPERM.
The EPERM error comes from:
stat("/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath/example", 0x7ffda8ae8670) = -1 EPERM (Operation not permitted)

Inspecting this path:
% file "/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath"
/mnt/c/Program Files (x86)/Common Files/Oracle/Java//javapath: unreadable symlink `/mnt/c/Program Files (x86)/Common Files/Oracle/Java//javapath' (Operation not permitted)

% ls -l "/mnt/c/Program Files (x86)/Common Files/Oracle/Java/"
ls: cannot read symbolic link '/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath': Operation not permitted
total 0
lrwxrwxrwx 1 jls jls    0 Jul 16 11:45 javapath
dr-xr-xr-x 1 jls jls 4096 Jul 16 11:45 javapath_target_98222640/

Back on Windows, Powershell, we see this javapath is an NTFS symlink:
> Get-Item "C:\Program Files (x86)\Common Files\Oracle\Java\javapath" | %{ $_.Attributes, $_.Target }
Directory, ReparsePoint
C:\Program Files (x86)\Common Files\Oracle\Java\javapath_target_98222640

Workaround: For any affected program, using which to lookup the path works for me. For example, in GNU Make:
% make build
go build .
make: execvp: go: Operation not permitted

But in my shell, it's fine:
% go version
go version go1.12.6 linux/amd64

Workaround, use which when we know an executable exists but execvp() is otherwise failing.
# Makefile

GO := $(shell which go)
build:
  $(GO) build .

make fails in a special way because of the way it executes using execvp which aborts early upon the first non-ENOENT error:
[pid 29679] execve("/home/jls/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/home/jls/local/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/home/jls/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/home/jls/local/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/local/sbin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory) [pid 29679] execve("/usr/local/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/sbin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/sbin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/games/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/local/games/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)[pid 29679] execve("/mnt/c/Program Files/ConEmu/ConEmu/Scripts/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/mnt/c/Program Files/ConEmu/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/mnt/c/Program Files/ConEmu/ConEmu/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 EPERM (Operation not permitted)
make: execvp: go: Operation not permitted

I dug into glibc (on my system, glibc 2.27). If I understand things correctly, the following:
The manpage says:

If permission is denied for a file (the attempted execve(2) failed with the error EACCES), these functions  will  continue searching the rest of the search path.  If no other file is found, however, they will return with errno set to EACCES.

This documentation is correlated with the glibc code handling EACCES.
However, in this scenario, we get EPERM, not EACCES, from the filesystem. EPERM triggers this default case which results in execvp aborting immediately instead of continuing to search $PATH.
Fun! ;)

Yep. Good post, your analysis is spot on. #4104 #4255 with a side order of #3886. I swear there is an older issue that describes the EPERM/EACCES difference nearly identically to your follow-up (I recognize the glibc code you linked), but I can't find it.
There are actually two problems with WSL here. (1) WSL imports paths it doesn't have access to, and (2) that errno is wrong; it should be EACCES, and the difference matters. But just changing to errno isn't a complete fix, because stuff can and does crap the bed even if it were EACCES.

After I login into my fish shell I got
set: Warning: $PATH entry "/mnt/c/Program Files x86/Common Files/Oracle/Java/javapath" is not valid (No such file or directory)
Welcome to fish, the friendly interactive shell

I think it might be a problem relates to this.

@oToToT the problem you posted is related to issue: https://github.com/microsoft/WSL/issues/4104 and is being tracked there.


FancyZones does not work with QScalp

Microsoft PowerToys version
0.71.0
Installation method
Microsoft Store
Running as admin
Yes
Area(s) with issue?
FancyZones
Steps to reproduce
When you press the Shift key, there is no area for docking the window.
✔ Expected Behavior
there should be an area for attaching a window
❌ Actual Behavior
Nothing
Other Software
QScalp 5.1
https://www.qscalp.ru/

Hey @savabaka, can you verify if the SHIFT key is being used by QScalp as a hotkey/keybinding/shortcut already?
Do you also know if QScalp is a "dockable" application? This may not be fixable due to the nature of QScalp.