v3.0.24 : sending JSON via REST returns "java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 20"

OrientDB Version: 3.0.24
Java Version: 1.8.0_92
OS: Ubuntu-18
Expected behavior
Sending same JSON requests or batches to OrientDB v3 server as we did with OrientDB v2 server.
Actual behavior
Sending JSON requests or batches via REST-API works fine in OrientDB v2 versions, but not anymore in v3:
the servers replies:
{
  "errors": [
    {
      "reason": 505,
      "code": 505,
      "content": "java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 20"
    }
  ]
}

If I enter this URL "http://<IP_ADDRESS>:2480/listDatabases" in my browser, then I get the expected result (database: "cdrarch"):
{"@type":"d","@version":0,"databases":["cdrarch"]}

Steps to reproduce
The REST request that we send as test, is this one (cfr http://orientdb.com/docs/3.0.x/misc/OrientDB-REST.html) :
http://<IP_ADDRESS>:2480/listDatabases

Regardless of this specific case, any JSON batch that we send via POST to the server results in the same error.

I search the web for such Java errors, and it looks like it has to do with the character encodings.
-> has this changed fundamentally in OrientDB v3 versus v2 ?

Hi @rdelangh
do you have a reproducer of batch which give this exception so i can debug it?
Thanks

hi Enrico,
thanks a lot for responding !
Can you please clarify what exactly you wish to receive ? The problem occurs even with the simple GET statement "/listDatabases", sent from a Perl program, with a HTTP message to the REST API of the OrientDB server.
The complete Perl structure that is returned from the server is this (I masked the password for obvious reasons):
{
                 '_res' => bless( {
                                    '_content' => '{
  "errors": [
    {
      "reason": 505,
      "code": 505,
      "content": "java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 20"
    }
  ]
}',
                                    '_rc' => '505',
                                    '_request' => bless( {
                                                           '_content' => '',
                                                           '_headers' => bless( {
                                                                                'accept' => 'application/json',
                                                                                'authorization' => 'Basic ...(encoded pwd)...
',
                                                                                'accept-encoding' => 'gzip,deflate',
                                                                                'user-agent' => 'REST::Client/273',
                                                                                'content-length' => 0
                                                                                }, 'HTTP::Headers' ),
                                                           '_uri_canonical' => bless( do{\(my $o = 'http://<IP_ADDRESS>:2480/listDatabases')}, 'URI::http' ),
                                                           '_uri' => $VAR1->{'_res'}{'_request'}{'_uri_canonical'},
                                                           '_method' => 'GET'
                                                         }, 'HTTP::Request' ),
                                    '_headers' => bless( {
                                                           'content-type' => 'application/json; charset=utf-8',
                                                           'pragma' => 'no-cache',
                                                           'client-response-num' => 1,
                                                           'client-peer' => '<IP_ADDRESS>:2480',
                                                           'client-date' => 'Thu, 31 Oct 2019 09:33:26 GMT',
                                                           'connection' => 'close',
                                                           '::std_case' => {
                                                                             'client-response-num' => 'Client-Response-Num',
                                                                             'client-date' => 'Client-Date',
                                                                             'client-peer' => 'Client-Peer'
                                                                           },
                                                           'content-length' => '185',
                                                           'date' => 'Thu Oct 31 10:33:26 MET 2019',
                                                           'server' => 'OrientDB Server v.3.0.24 - Veloce (build 9920dff342aee1351c89e749d3f0a94ce466355b, branch 3.0.x)',
                                                           'cache-control' => 'no-cache, no-store, max-age=0, must-revalidate'
                                                         }, 'HTTP::Headers' ),
                                    '_protocol' => 'HTTP/1.1',
                                    '_msg' => 'Error on executing of GET for the resource: /listDatabases'
                                  }, 'HTTP::Response' ),
                 '_config' => {
                                'timeout' => 3600,
                                'host' => '<IP_ADDRESS>:2480',
                                'useragent' => bless( {
                                                        'max_redirect' => 7,
                                                        'no_proxy' => [],
                                                        'use_eval' => 1,
                                                        'protocols_forbidden' => undef,
                                                        'handlers' => {
                                                                        'response_header' => bless( [
                                                                                {
                                                                                'line' => '/usr/local/lib/perl5/site_perl/5.20.1/LWP/UserAgent.pm:683',
                                                                                'callback' => sub { "DUMMY" },
                                                                                'owner' => 'LWP::UserAgent::parse_head',
                                                                                'm_media_type' => 'html'
                                                                                }
                                                                                ], 'HTTP::Config' )
                                                                      },
                                                        'protocols_allowed' => undef,
                                                        'max_size' => undef,
                                                        'timeout' => 3600,
                                                        'def_headers' => bless( {
                                                                                'user-agent' => 'REST::Client/273'
                                                                                }, 'HTTP::Headers' ),
                                                        'show_progress' => undef,
                                                        'proxy' => {},
                                                        'requests_redirectable' => [
                                                                                'GET',
                                                                                'HEAD'
                                                                                ],
                                                        'ssl_opts' => {
                                                                        'verify_hostname' => 1
                                                                      },
                                                        'local_address' => undef
                                                      }, 'LWP::UserAgent' )
                              }
               }

Hi @rdelangh
that is strange if it's working from the browser.  Do you have a simple script that i can use to reproduce this?

Enrico, the 'script' is a Perl program, which is using a fair amount of installed Perl modules to perform the REST/HTTP calls.
I can give you a stripped program with just the essentials to send the request, but you will have a Perl installation with all the modules used by this program. Is that ok ?

Hi @rdelangh
i did try from node with this script
const http = require("http");

http.get("http://localhost:2480/listDatabases", res => {
  res.setEncoding("utf8");
  let rawData = "";
  res.on("data", chunk => {
    rawData += chunk;
  });
  res.on("end", () => {
    try {
      const parsedData = JSON.parse(rawData);
      console.log(parsedData);
    } catch (e) {
      console.error(e.message);
    }
  });
});

and it seems fine. Could it be related to Perl? not sure

Enrico,
it surely is related to the version of OrientDB:

I have another server on <IP_ADDRESS> running OrientDB 2.2.37
I launch from my webbrowser to this server the same URL "http://<IP_ADDRESS>:2480/listDatabases" and get the same correct result:

{"@type":"d","@version":0,"databases":["cdrarch"]}

if I use my exact same Perl program to do exact the same URL now via the REST calls, it is successful:

{
                 '_config' => {
                                'useragent' => bless( {
                                                        'use_eval' => 1,
                                                        'max_redirect' => 7,
                                                        'handlers' => {
                                                                        'response_header' => bless( [
                                                                                {
                                                                                'owner' => 'LWP::UserAgent::parse_head',
                                                                                'line' => '/c5qp/home/qcontrol/tools-3.4/sparcv9/32/lib/perl5/site_perl/5.20.1/LWP/UserAgent.pm:683',
                                                                                'callback' => sub { "DUMMY" },
                                                                                'm_media_type' => 'html'
                                                                                }
                                                                                ], 'HTTP::Config' )
                                                                      },
                                                        'requests_redirectable' => [
                                                                                'GET',
                                                                                'HEAD'
                                                                                ],
                                                        'ssl_opts' => {
                                                                        'verify_hostname' => 1
                                                                      },
                                                        'protocols_forbidden' => undef,
                                                        'def_headers' => bless( {
                                                                                'user-agent' => 'REST::Client/273'
                                                                                }, 'HTTP::Headers' ),
                                                        'proxy' => {},
                                                        'local_address' => undef,
                                                        'protocols_allowed' => undef,
                                                        'max_size' => undef,
                                                        'no_proxy' => [],
                                                        'timeout' => 3600,
                                                        'show_progress' => undef
                                                      }, 'LWP::UserAgent' ),
                                'host' => '<IP_ADDRESS>:2480',
                                'timeout' => 3600
                              },
                 '_res' => bless( {
                                    '_rc' => '200',
                                    '_msg' => 'OK',
                                    '_protocol' => 'HTTP/1.1',
                                    '_content' => '{"@type":"d","@version":0,"databases":["cdrarch"]}',
                                    '_headers' => bless( {
                                                           'content-type' => 'application/json; charset=utf-8',
                                                           'client-peer' => '<IP_ADDRESS>:2480',
                                                           'date' => 'Thu, 31 Oct 2019 12:01:00 UTC',
                                                           '::std_case' => {
                                                                             'client-response-num' => 'Client-Response-Num',
                                                                             'client-date' => 'Client-Date',
                                                                             'client-peer' => 'Client-Peer'
                                                                           },
                                                           'connection' => 'close',
                                                           'client-response-num' => 1,
                                                           'etag' => '0',
                                                           'server' => 'OrientDB Server v.2.2.37-SNAPSHOT (build b041441a53b1666cb52872a14878750b54b7784b, branch branches/2-2-x.15993v/workspace/core)',
                                                           'client-date' => 'Thu, 31 Oct 2019 11:59:53 GMT',
                                                           'content-length' => '50'
                                                         }, 'HTTP::Headers' ),
                                    '_request' => bless( {
                                                           '_method' => 'GET',
                                                           '_uri' => bless( do{\(my $o = 'http://<IP_ADDRESS>:2480/listDatabases')}, 'URI::http' ),
                                                           '_headers' => bless( {
                                                                                'authorization' => 'Basic YWRtaW46RXh1cGl0YTU=
',
                                                                                'content-length' => 0,
                                                                                '::std_case' => {
                                                                                'charset' => 'Charset'
                                                                                },
                                                                                'accept-encoding' => 'gzip,deflate',
                                                                                'user-agent' => 'REST::Client/273',
                                                                                'charset' => 'UTF-8',
                                                                                'accept' => 'application/json'
                                                                                }, 'HTTP::Headers' ),
                                                           '_content' => '',
                                                           '_uri_canonical' => $VAR1->{'_res'}{'_request'}{'_uri'}
                                                         }, 'HTTP::Request' )
                                  }, 'HTTP::Response' )
               }

So this exact same URL via REST to an OrientDB server 2.2.37 runs fine, but it fails went sent to an OrientDB server 3.0.24

Strange, that part didn't change much in ODB 3.0.x
Do you have some stacktrace server side?

If you mean anything in the server logs : the answer is no: last log lines are from its startup yesterday
...
2019-10-30 12:09:43:209 INFO  Page size for WAL located in /opt/orientdb/databases/cdrarch is set to 4096 bytes. [OCASDiskWriteAheadLog]
2019-10-30 12:09:45:093 INFO  Storage 'plocal:/opt/orientdb/databases/cdrarch' is opened under OrientDB distribution : 3.0.24 - Veloce (build 9920dff342aee1351c89e749d3f0a94ce466355b, branch 3.0.x) [OLocalPaginatedStorage]

to show the irrelevance of the Perl programming language:
using "tcpdump", these are the network packet contents of what gets exchanged with the server:

the request to the server:

GET /listDatabases HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: application/json
Accept-Encoding: gzip,deflate
Authorization: Basic ...(encoded password)
Host: <IP_ADDRESS>:2480
User-Agent: REST::Client/273
Content-Length: 0
Charset: UTF-8

the reply from the server:

HTTP/1.1 505 Error on executing of GET for the resource: /listDatabases
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Date: Sun Nov 03 15:42:54 MET 2019
Content-Type: application/json; charset=utf-8
Server: OrientDB Server v.3.0.24 - Veloce (build 9920dff342aee1351c89e749d3f0a94ce466355b, branch 3.0.x)
Connection: close
Content-Length: 185

{
"errors": [
{
"reason": 505,
"code": 505,
"content": "java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 20"
}
]
}

@wolf4ood :
with your Java example, you are running your c(Java) lient program on the same (local)host as the dbase server, apparently, since you connect to "localhost:2480"
Can you please perform this same test from a different host as where the dbase server runs ?
Then capture the network (HTTP) packets going back and forth between client and server, and see where they differ from the network packets that I showed above.

hi Enrico @wolf4ood ,
do you need any further information from me to allow you to analyse this problem ?
thx already for your time!

@wolf4ood ,
how did you execute your script example ? Is it running outside of OrientDB ?
I want to reproduce your script, and try to execute it with a remote connection to the OrientDB server (i.e. not on localhost) so that I can capture the HTTP messages sent.

to all people on this forum:
does anyone else use the REST API of OrientDB v3.0.X  ?
If so, can you please share your experiences, like using JSON or not, how encoding your requests to the server, etc.
thx!

Hi @rdelangh
i was not able to reproduce it yet. Are you able to get the same exception with curl ?
Thanks

@wolf4ood
Did a test with 'curl', that works fine. The HTTP packet headers that I found in the request going to the ODB server are:
Host: <IP_ADDRESS>:2480
User-Agent: curl/7.59.0-DEV
Accept: */*

The server responded correctly with
HTTP/1.1 200 OK
Date: Thu, 14 Nov 2019 11:58:25 UTC
Content-Type: application/json; charset=utf-8
Server: OrientDB Server v.3.0.24 - Veloce (build 9920dff342aee1351c89e749d3f0a94ce466355b, branch 3.0.x)
Connection: Keep-Alive
ETag: 0
Content-Length: 50

{"@type":"d","@version":0,"databases":["cdrarch"]}

hi @wolf4ood ,
do you have any updates, please ? Is the v3 server checking in some more particular way on the HTTP header fields, like (maybe) the User-Agent value ?

@wolf4ood
I have just hacked my Perl libraries to send a 'gentle' looking User-Agent header: "RESTClient/273" (no special chars except the "/")
But still encountering the same error reply from the server...

@wolf4ood
I found the issue: for the "listDatabases" command, no authorization is needed (cfr the "curl" test above).
So I skipped the "authorization" HTTP header, and now I do NOT get the error anymore from the server:
The HTTP network packet from client (Perl program) to server:
GET /listDatabases HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: */*
Host: <IP_ADDRESS>:2480
User-Agent: REST::Client/273
Content-Length: 0

The HTTP reply from server to client:
Date: Fri, 22 Nov 2019 15:35:11 UTC
Content-Type: application/json; charset=utf-8
Server: OrientDB Server v.3.0.24 - Veloce (build 9920dff342aee1351c89e749d3f0a94ce466355b, branch 3.0.x)
Connection: close
ETag: 0
Content-Length: 50

{"@type":"d","@version":0,"databases":["cdrarch"]}

So the server is not able to deal with the Authorization HTTP header apparently. I am sending normally (as for OriendtDB v2) the following:
Authorization: Basic ...(base64 encoded login+passwd)...

-> can you focus your investigation on that part, please ?

@wolf4ood
From the documentation (https://orientdb.org/docs/3.0.x/misc/OrientDB-REST.html#authentication-and-security) I copy:
The HTTP client (or the Internet Browser) must send user and password using the HTTP Base authentication. Password is encoded using Base64 algorithm.
-> is this correctly implemented for OrientDB v3 ?

So:
GET /listDatabases HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: application/json
Accept-Encoding: gzip,deflate
Authorization: Basic ...(base64 encoded login+passwd)...
Host: <IP_ADDRESS>:2480
User-Agent: REST::Client/273
Content-Length: 0

results in an error from the server, but
GET /listDatabases HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: application/json
Accept-Encoding: gzip,deflate
Host: <IP_ADDRESS>:2480
User-Agent: REST::Client/273
Content-Length: 0

is properly accepted by the server.

I found the reason (analyzing the network packets file with a more verbose utility, "tshark"):
there is a trailing space (blank) character at the end of the BASE64 encoded login+password concatenated string in the HTTP header "Authorization", which breaks the decoding by the OrientDB v3 server:
This is OK (no trailing blank in the Base64 encoded credentials :
wget --header="Authorization: Basic ...encodedCredentialsWithoutTrailingBlank..." http://<IP_ADDRESS>:2480/listDatabases
--2019-11-25 12:57:18--  http://<IP_ADDRESS>:2480/listDatabases
Connecting to <IP_ADDRESS>:2480... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50 [application/json]

whereas this is NOT OK (with a trailing blank in the Base64 encoded credentials :
wget --header="Authorization: Basic ...encodedCredentialsWITHtrailingBlank... "  http://<IP_ADDRESS>:2480/listDatabases
--2019-11-25 12:57:18--  http://<IP_ADDRESS>:2480/listDatabases
Connecting to <IP_ADDRESS>:2480... connected.
HTTP request sent, awaiting response... 505 Error on executing of GET for the resource: /listDatabases
2019-11-25 12:57:25 ERROR 505: Error on executing of GET for the resource: /listDatabases.

Sending this exact same HTTP header (thus, WITH the trailing space/blank character at the encoded credentials) to a v2.2.37 OrientDB server (different IP, <IP_ADDRESS>), is working fine :
wget --header="Authorization: Basic ...encodedCredentialsWITHtrailingBlank... " http://<IP_ADDRESS>:2480/listDatabases
--2019-11-25 13:03:23--  http://<IP_ADDRESS>:2480/listDatabases
Connecting to <IP_ADDRESS>:2480... connected.
HTTP request sent, awaiting response... 200 OK
Length: 49 [application/json]

Hi @rdelangh
thanks for the detailed investigation. I'm not aware on any changes in request parsing in 3.0.x but let me check.  I was able to reproduce. Let me check on this
Thanks again


VECTOR-35: fix vector on-disk writer to append segments

Some tests failures but not related to this fix.  Merging!

(cleaned up the System.out and unused Offsets class from CHNSW)


upgrade version in values.yaml

Contributing a Chart / update to an existing Chart

[x] Run helm lint on the chart dir.
[x] (Update) Bump the Chart.yaml version before merging, to release it as a new version.
[x] (Update) If the PR includes new configurable parameters in the chart's values.yaml. Add documentation in the appropiate README.

@glpatcern About tests, I tried running them from my laptop without even configuring Reva or Wopi adresses and more than half of them passed, so I'm not sure if they are up to date. Also, I'm not sure how to even configure them correctly

@glpatcern About tests, I tried running them from my laptop without even configuring Reva or Wopi adresses and more than half of them passed, so I'm not sure if they are up to date. Also, I'm not sure how to even configure them correctly

In the link I provided you'll see that by default they run against local storage (and they're now integrated in the GitHub CI, see e.g. https://github.com/cs3org/wopiserver/runs/5394336262, so it's fair they pass. You should try and run them against Reva.
As for the version, yeah this was the latest at the time I started to update the charts, but better to push to a more recent one.

@glpatcern About tests, I tried running them from my laptop without even configuring Reva or Wopi adresses and more than half of them passed, so I'm not sure if they are up to date. Also, I'm not sure how to even configure them correctly

In the link I provided you'll see that by default they run against local storage (and they're now integrated in the GitHub CI, see e.g. https://github.com/cs3org/wopiserver/runs/5394336262, so it's fair they pass. You should try and run them against Reva.
As for the version, yeah this was the latest at the time I started to update the charts, but better to push to a more recent one.

Ok so all I have to do is to configure this file, right:
https://github.com/cs3org/wopiserver/blob/master/test/wopiserver-test.conf
The [cs3] part is pretty self-explanatory but how do I configure the rest to use my wopi https://cs3mesh.softwaremind.com/wopi? Please bear in mind that im running tests from a machine outside the cluster.

Ok so all I have to do is to configure this file, right:
https://github.com/cs3org/wopiserver/blob/master/test/wopiserver-test.conf

Yes, and if you follow the steps you'll see that the test suite is there to validate the software (from a developer's perspective if I can say) and to make sure the connection to Reva works (i.e. the CS3APIs are aligned). It's not a "black-box" external test suite: to really validate the server you'll have to run the reva open-in-app command. But the charts PR is probably already ok to be merged.


chore: change rust version used by CI to v1.72.0

Description
Changes rust version CI uses to v1.72.0 and nightly to 2023-08-27

Testing the CI if that actually fixes it


SCMOD-11820: Updated to latest release of base image

CI Build Link:
http://sou-jenkins2.swinfra.net:8080/job/CAFDataProcessing/job/CAFDataProcessing~boilerplate-service~SCMOD-11820~CI


add System-level programming languages

System programming language Stability and speed are fast
Ada
Common Lisp
Crystal
ATS
Elixir
Forth
Vala
Hare
Carbon
Pony
Red

PRs are welcome :)
Crystal is already there


rbi: Use correct return type for String#unpack

Motivation
The current definition in RBI for String#unpack states it returns an T::Array[String].
Documentation and experiments show it can return more than that:
irb(main):024:0> "abcdefg\0\0abc".unpack('CdAD')
=> [97, 5.61700690309813e-310, "a", nil]

This PR changes the return type to T::Array[T.nilable(T.any(Integer, Float, String))].

Nice find!
@Morriar as this isn't intuitive, can you add your example as a test case ?

@DarkDimius

@Morriar as this isn't intuitive, can you add your example as a test case ?

Not sure where, for static or runtime?

For static, a testcase in test/tedtdata that would fail if someone changes the signature back

@DarkDimius added test in //test:test_PosTests/testdata/rbi/string.


Fix quantities with "Tausendertrennpunkt" in Spar

Didn't dare to ignore dots globally, as I discovered stuff like "500.000 GR" in MPREIS data.
Fixes #71

Cheers!


Custom sklearn + LightGBM model

Hello, I have a custom sklearn + lightgbm model and I would like to know if it is possible to include this model in a creme pipeline. If not, is there any way to do text classification with LightGBM in a creme pipeline?
Kind Regards,
Luís

Hey @luistelmocosta! Alas LightGBM is a batch, and therefore cannot be trained incrementally. In particular it doesn't have a partial_fit method.

Thank you for your reply! Is there any text classification pipeline for Creme?

Well for instance you can use the TFIDF and then chain that with a LogisticRegression.


fix: retain tooltip visibility on title update

Correct bootstrap implementation is not yet available.
In the meantime this PR is a quick fix for #250
While it fixes the issue of disappearing tooltip, it is not a perfect fix as the tooltip will fade out and in as opposed to remaining in view throughout

Note: setContent is currently available on the main branch of Bootstrap v5, still waiting for release~

Thanks!

Correct bootstrap implementation is not yet available.

Could you please link to a bootstrap issue, so one can find it when looking to implement the correct solution?

I tried to test it myself but am not good enough with vue and the library to do it, sorry!
But I linked the setContent function and commit in #250


Create a sharding manager contract

The plan is to create a contract in Vyper, which I've started, based on https://ethresear.ch/t/sharding-phase-1-spec/1407/44 and adapting from https://github.com/ethereum/py-evm/blob/sharding/evm/vm/forks/sharding/contracts/validator_manager.v.py.

https://github.com/Drops-of-Diamond/sharding/blob/develop/smc/sharding_manager_contract.v.py

Did some work, then will let @NIC619 complete it as per https://github.com/ethereum/py-evm/issues/452.

https://github.com/fubuloubu/pytest-ethereum.
https://gitter.im/ethereum/vyper?at=5ac6f3497c3a01610db2319c

https://github.com/ethereum/sharding/issues/76
https://ethresear.ch/t/a-minimal-sharding-protocol-that-may-be-worthwhile-as-a-development-target-now/1650/4

https://github.com/ethereum/sharding/pull/74

I think it's better to let Py-EVM work on the SMC in Vyper. We can review it when they're done.

The above commit fixes issue number 1 in another directory, https://github.com/NitorCreations/license-maintainer/issues/1, not the issue #1 in this directory. So when the above commit is merged into develop this issue will need to be reopened.


Akka TestKit NUnit v1.5.32 - Format exception in ExpectMsgFromAsync

Version Information
Akka.TestKit.NUnit version 1.5.32
Bug Description
We've upgraded Akka.TestKit.NUnit to 1.5.32 now. There is a small issue with our tests though. It seems like ExpectMsgFromAsync method causes format exceptions. We are using the same parameters in our test but now the input is used for some formatting inside this version.
System.FormatException : Input string was not in a correct format. Failure to parse near offset 177. Expected an ASCII digit. at System.Text.ValueStringBuilder.AppendFormatHelper(IFormatProvider provider, String format, ReadOnlySpan1 args) at System.String.FormatHelper(IFormatProvider provider, String format, ReadOnlySpan1 args) at System.String.Format(String format, Object[] args) at Akka.TestKit.NUnit.NUnitAssertions.AssertTrue(Boolean condition, String format, Object[] args) at Akka.TestKit.TestKitBase.AssertPredicateIsTrueForSender(Predicate1 isSender, IActorRef sender, String hint, Object message) at Akka.TestKit.TestKitBase.<>c__DisplayClass175_01.b__0(T m, IActorRef sender) at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelopeAsync[T](Nullable1 timeout, Action2 assert, String hint, CancellationToken cancellationToken, Boolean shouldLog) at Akka.TestKit.TestKitBase.InternalExpectMsgAsync[T](Nullable1 timeout, Action2 assert, String hint, CancellationToken cancellationToken) at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException[TResult](Task1 task) at Akka.TestKit.TestKitBase.ExpectMsgFrom[T](Predicate1 isSender, Predicate1 isMessage, Nullable1 timeout, String hint, CancellationToken cancellationToken)
The method where the exception is thrown: public void AssertTrue(bool condition, string format = "", params object[] args) { Assert.That(condition, (IResolveConstraint) Is.True, (NUnitString) string.Format(format, args), nameof (condition), "Is.True"); }
value for format parameter here:
Sender does not match. Got a message from sender TestActor[akka://test-actor-system/system/testActor1/event-bridge]. But expected the predicate to return true The message was {DurableMessageEnvelope { Id = ca4e426d-95b2-4e8d-a320-158e771be79e, Value = Ats.Ros.Common.Ingestion.MissionStateMessage, ActorRef = TestActor[akka://test-actor-system/system/testActor1/event-bridge], ObjectType = Ats.Ros.Common.Ingestion.MissionStateMessage, CreatedDateTime = 1/1/0001 12:00:00 AM }}
Are you aware of such issue?
@Arkatufus
@milang
@UrsMetz

Hello @UrsMetz ,
I think in that case all curly brackets must be escaped in the message object somehow. I see that our test would have passed if it was formatted like this:
"Sender does not match. Got a message from sender TestActor[akka://test-actor-system/system/testActor1/event-bridge]. But expected the predicate to return true  The message was **{{**DurableMessageEnvelope {{ Id = 6a9ece7b-7d6c-493f-9401-01ba8d76f411, Value = Ats.Ros.Common.Ingestion.MissionStateMessage, ActorRef = TestActor[akka://test-actor-system/system/testActor1/event-bridge], ObjectType = Ats.Ros.Common.Ingestion.MissionStateMessage, CreatedDateTime = 1/1/0001 12:00:00 AM }}}}"
I think that is not something we can do as a caller to this ExpectMsgFrom method. Can curly brackers be escaped in Akka.TestKit.Nunit package for this?

@busraozis As I'm not a maintainer of Akka.TestKit.NUnit I can't deciding whether Akka.TestKit.NUnit should escape the curly brackets or not.
In the end I think it is a breaking change in NUnit 4: I had a brief look at where exactly the difference in behaviour stems from but didn't see the reason.

@UrsMetz , I opened a case here as I found this repo as the source code of that package. Do you know who is responsible for this? Isn't it maintained any more? If it's not maintained, can we contribute to it?
Thanks in advance..


clicking on the internal anchor tag inside drop down list which is placed inside clr-header does not trigger Heamburer menu close

clicking on the internal anchor tag inside drop down list which is placed inside clr-header does not trigger Heamburer menu close

@digish777 please provide a reproducible example with Stackblitz. You can find starter kits here https://stackblitz.com/@clr-team

I guess you're referring to the same setup as in #5294
As already discussed, this is not a supported configuration.
As can be seen on the updated example below, you should not have access to the actions dropdown once the hamburger is open:
https://stackblitz.com/edit/header-nav-vs-actions-demo?file=src/app/app.component.html


Package request: Jellyseerr

Project description
Fork of Overseerr for Jellyfin support. Overseerr also isn't packaged. They're comparable to Ombi which is.
Metadata

Homepage / source: https://github.com/Fallenbagel/jellyseerr
License: MIT
Platforms: Linux at least - there's a Docker container and an AUR package

I just built https://github.com/tboerger/nur-packages/blob/master/pkgs/jellyseerr/default.nix, if the tests for that are successfull I will submit it into nixpkgs, after that you should be able to use that :)

Sadly it's more complicated than I thought. The package is building, but for some reason it's missing sqlite3 on start. Need to figure out why, maybe I will just create a PR including the mackage and ask other packagers for advice.


Unstable advisory unique_content_id

Current implementation of unique_content_id is unstable because the order of keys is not preserved in JsonField. On PostgreSQL by default JsonField uses jsonb, which does not preserve the order or whitespace.
For more information, see https://docs.djangoproject.com/en/5.1/ref/models/fields/#django.db.models.JSONField and https://www.postgresql.org/docs/16/datatype-json.html.
This can lead to widespread duplication of advisory data, resulting in increased storage usage.
Below is a snippet to reproduce the bug, where the same advisory data leads to different unique_content_id values:
In [1]: from vulnerabilities import importer

In [2]: from packageurl import PackageURL

In [3]: from univers.version_range import VersionRange

In [4]: from django.utils import timezone

In [5]: from vulnerabilities.pipes.advisory import insert_advisory

In [6]: from vulnerabilities.importer import AdvisoryData

In [7]: advisory_data = importer.AdvisoryData(
   ...:     aliases=["CVE-2020-13371337"],
   ...:     summary="vulnerability description here",
   ...:     affected_packages=[
   ...:         importer.AffectedPackage(
   ...:             package=PackageURL(type="pypi", name="dummy"),
   ...:             affected_version_range=VersionRange.from_string("vers:pypi/>=1.0.0|<=2.0.0"),
   ...:         )
   ...:     ],
   ...:     references=[importer.Reference(url="https://example.com/with/more/info/CVE-2020-13371337")],
   ...:     date_published=timezone.now(),
   ...:     url="https://test.com",
   ...: )

In [8]: r = insert_advisory(advisory_data, "test")

In [9]: r.unique_content_id
Out[9]: '2ececc550f7f6b5537e5f1a767ef0f25'

In [10]: k = Advisory.objects.get(unique_content_id=r.unique_content_id)

In [11]: k.unique_content_id
Out[11]: '2ececc550f7f6b5537e5f1a767ef0f25'

In [12]: k.date_imported = None # Change any field not used for computing content id

In [12]: k.save()

In [13]: k.unique_content_id
Out[13]: 'bf83e58fc8f7eb54d04a59c27f0680f8'

In [14]: assert k.unique_content_id == r.unique_content_id
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[14], line 1
----> 1 assert k.unique_content_id == r.unique_content_id

try this
https://gofile.io/d/yMHabA
select the desired compiler in the installer


DANG-1754/update input Label [to merge before inputs updates]

JIRA

A link to the JIRA ticket
Zeplin link here

(Myrtle taking this over so Shanna can focus on sales tax)
Description
We are updating the Label first before starting the refresh work on the inputs.
updated Label

updated Stories

not related to this change but since we are here, there seems to be two stories that look the same (with input, with nested input?) I don't see a difference so maybe we can remove one or both? as the input is in all of them anyway

not related to this change but since we are here, there seems to be two stories that look the same (with input, with nested input?) I don't see a difference so maybe we can remove one or both? as the input is in all of them anyway
Yeah, i noticed that and i'm not really sure what that nested one is - was going to follow up about that

Updated to spec and will sync w Michael for a visual review on the preview link.
I removed the extra story that was confusing (it was showcasing the slot functionality), and updated to stories that match our new design states.

Only one question, and quite possibly it isn't related to this PR but to the tooltip component itself, but does the tooltip always extend to that width?

good thought, I am changing it to w-max max-w-[200px], and it'll adjust like this:

good thought, I am changing it to w-max max-w-[200px], and it'll adjust like this:

that looks much better indeed!


NAACL submission work

Assorted patches accumulated in the last month

Tests don't pass because the batching code needs a newer version of genienlp, I'll ask @s-jse to submit a PR for it


Fixed branch selection support for both Grunt and Rake

This PR adds back the support to select an older version or a different branch when running the conversion. Because the older versions of patternfly had a variable ordering issue, the conversion will fail with tags/v1.1.4.

:+1:


Conversation agent handler button

A conversation should have a button to initiate hand-off to human.

Todo: We should try to see if we can generalise the action button for other generic purposes?


Are local problems of some stages that involves Mixed-Integer Linear Programming (MILP) also considered as SDDiP?

I have a multi stage stochastic programming problem. The state variables are continuous variables and the objective functions are linear, but the local problems of some stages are mixed interger linear programming.  Is this question related to the SDDiP problem? I noticed that SDDiP problems involve integer variables in both the state variable and the objective function. However, I am uncertain whether the presence of 0-1 variables in the local problem qualifies it as an SDDiP. In this case, could you please clarify if the code is suitable for this problem. If you could provide clarification to resolve my confusion, I would greatly appreciate it.

Let's keep the discussion to https://github.com/odow/SDDP.jl/issues/724


Multiple video device support

Support for multiple people to view streams using v4l2loopback
Iterate over multiple video devices if the first device is not available.
E.g. Try /dev/video1, if busy try /dev/video2, if busy try /dev/video3, etc.
This may be possible by adding the video device in use for streaming in the ongoingSessions array ? https://github.com/moritzmhmk/homebridge-camera-rpi/blob/4474cf5485ae43d98d3fdc6a406b7a0857ba9aad/CameraSource.js#L193

Great idea! Did you have any success so far?

Great idea! Did you have any success so far?

Struggling to find a way to detect if a video device is already in use. Any ideas?

as noted in #50 I do consider to aim for optimal Pi zero support from now on. The reason is that using h264_omx seems to be the better solution on newer Pis.
While with h264_omx one would redirect the raw stream to multiple video device endpoints we would need to redirect the h264 stream from the camera. This would require setting the resolution and for all clients to receive that same resolution (e.g. 720p).
Keeping track of the used video devices could be done with another array that just holds a boolean value and is set when stream starts/stops on a specific device. If you take a look at the develop branch on wich I did some refactoring you will see that there now is a separate StreamDelegate object which could be assigned a specific video device and make tracking them unnecessary.

This would require setting the resolution and for all clients to receive that same resolution (e.g. 720p).

What is prohibiting us running this multiplied stream through ffmpeg with corresponding parameters to resize the stream to match every device's requirements?

Check https://github.com/moritzmhmk/homebridge-camera-rpi/tree/feature-loopback for a possible solution.
Resizing each stream would cause a lot of cpu usage... might still work on a pi model 3


Contribute Solution for LeetCode October Day 26

Day 26:
Contribute a solution for this day's LeetCode problem.
Rules:

Comment on this issue if you're willing to contribute.
Each user can contribute a maximum of 1 solution in Java, C++, or Python.
Each user can contribute a maximum of 2 solutions in other languages like JavaScript, Go, Ruby, etc.
Please ensure that no one has already commented for the language you're planning to contribute in.

Good luck and happy coding! :tada: :rocket:

Kindly assign me this task, I would positively contribute @arya2004

@arya2004 please assign me this issue . i will solve this diligently and quickly . this will help me in my professional devolopment. please consider this comment!! i have solved more than 100 problems on leetcode and have good command on it

Hi! @arya2004, I would like to contribute in C++, I have more than 200 active days in Leetcode, please assign this issue to me.


Fairseq-hydra-train: str interpolation key 'hydra.job.name' not found

🐛 Bug
When trying to run any example using fairseq-hydra-train and configurations I get an error stating that the str interpolation key 'hydra.job.name' is not found.
To Reproduce
Using fairseq from source (3rd February 2021), from master branch. Using Python 3.6 and Pytorch 1.7.1, Hydra 2.5 and Omegaconf 2.0.6
Code sample
Running any of the examples here
fairseq-hydra-train
task.data=/path/to/data
--config-dir /path/to/fairseq-py/examples/wav2vec/config/pretraining
--config-name wav2vec2_base_librispeech
Expected behavior
Traceback (most recent call last):
File "/home/carlos/anaconda3/envs/fairseq010/bin/fairseq-hydra-train", line 33, in
sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-hydra-train')())
File "/home/carlos/Documentos/fairseq-0.10/fairseq/fairseq_cli/hydra_train.py", line 91, in cli_main
hydra_main()
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/main.py", line 37, in decorated_main
strict=strict,
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/_internal/utils.py", line 347, in _run_hydra
lambda: hydra.run(
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/_internal/utils.py", line 201, in run_and_report
raise ex
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
return func()
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/_internal/utils.py", line 350, in
overrides=args.overrides,
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/_internal/hydra.py", line 112, in run
configure_logging=with_log_configuration,
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/hydra/core/utils.py", line 128, in run_job
ret.return_value = task_function(task_cfg)
File "/home/carlos/Documentos/fairseq-0.10/fairseq/fairseq_cli/hydra_train.py", line 34, in hydra_main
cfg.job_logging_cfg = OmegaConf.to_container(HydraConfig.get().job_logging, resolve=True)
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/omegaconf.py", line 442, in to_container
return BaseContainer._to_content(cfg, resolve=resolve, enum_to_str=enum_to_str)
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/basecontainer.py", line 195, in _to_content
node, resolve=resolve, enum_to_str=enum_to_str
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/basecontainer.py", line 195, in _to_content
node, resolve=resolve, enum_to_str=enum_to_str
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/basecontainer.py", line 189, in _to_content
throw_on_missing=False, throw_on_resolution_failure=True
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/base.py", line 137, in _dereference_node
throw_on_resolution_failure=throw_on_resolution_failure,
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/base.py", line 392, in _resolve_interpolation
throw_on_resolution_failure=throw_on_resolution_failure,
File "/home/carlos/anaconda3/envs/fairseq010/lib/python3.6/site-packages/omegaconf/base.py", line 328, in _resolve_simple_interpolation
f"{inter_type} interpolation key '{inter_key}' not found"
omegaconf.errors.ConfigKeyError: str interpolation key 'hydra.job.name' not found
Environment

fairseq Version (e.g., 1.0 or master): master
PyTorch Version (e.g., 1.0): 1.7.1
OS (e.g., Linux): Ubuntu 20.4
How you installed fairseq (pip, source): source
Build command you used (if compiling from source): pip install --editable ./
Python version: 3.6
CUDA/cuDNN version: 11.2
GPU models and configuration: gtx 950m
Any other relevant information:

Hi i got same issue these day.
I fix this problem by downgrade 'hydra-core' version to 1.0.
In fairseq setup, "hydra-core" must be less than 1.1
https://github.com/pytorch/fairseq/blob/b4843681b4d5af442febf8caba58ca9600b01656/setup.py#L193
Good luck

Thank you! After checking the version I realized that it was a know hydra-core error fixed on release 1.0.6
I installed hydra-core from source and It fixed it.
I'm closing the issue as this is working now.


Error on submitting PDF for processing

The command line shows this error and then the server exits.
INFO  [2019-12-12 06:32:08,345] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@53125718{/,null,AVAILABLE}
INFO  [2019-12-12 06:32:08,351] org.eclipse.jetty.server.AbstractConnector: Started application@77c3c037{HTTP/1.1,[http/1.1]}{<IP_ADDRESS>:8070}
INFO  [2019-12-12 06:32:08,351] org.eclipse.jetty.server.AbstractConnector: Started admin@252744a1{HTTP/1.1,[http/1.1]}{<IP_ADDRESS>:8071}
INFO  [2019-12-12 06:32:08,352] org.eclipse.jetty.server.Server: Started @2378ms
INFO  [2019-12-12 06:32:16,890] org.grobid.core.jni.WapitiModel: Loading model: /media/kadam/data drive/Research Paper Builder/backend/grobid/grobid-home/models/affiliation-address/model.wapiti (size: 2699936)
error: too much input files on command line

I think the model loaded above is getting passed through command line arguments. In systems, there's a limit in number of bytes up to which we can pass command line arguments. And that value is low compared to model size. Maybe that is causing this error. Though I ain't aware of what actually is causing this error, can you please help?

@ParikhKadam just as a random though, might well be the spaces in the path which is not well digested... could you try to remove them and see if ti works?

@lfoppiano Yeah.. that worked. Thank you for your quick reply :)


Any documentation about how to setup supabase manually?

I have an existing project and want to use supabase & stripe like this project does.
Is there any document about what happened behind the DEPLOY button?

The deploy button here (+ the Supabase integration) connects your accounts together. To manually connect your accounts, you need to add your connection information from Supabase as Environment Variables inside the Vercel dashboard.

Apart from the environment variables, the supabase setup some tables and webhooks in the background, that the most important missing part

You can find these steps in a previous version of the README: https://github.com/vercel/nextjs-subscription-payments/blob/971f8844572efcb163b00efa32261c7c479a286a/README.md#configuration

Thank you! @thorwebdev
This is exactly what I'm looking for!


v1: add /experimental/blueprints/{id}/imageTypes endpoint to image builder (HMS-4066)

This commit introduces a new endpoint that retrieves a list of image types for a specific blueprint image. It also adds a handler function to implement this endpoint. Additionally, unit tests are included to verify the functionality of this new endpoint.

/retest

/retest


HDDS-11376. Improve ReplicationSupervisor to record replication metrics

What changes were proposed in this pull request?
When datanode executes ReplicateContainerCommand and ReconstructECContainersCommand, they are all done by ReplicationSupervisor. We should record more detailed metrics.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11376
How was this patch tested?
When we open the jmx connection of Datanode, we will see the newly added metrics.
Old jmx:

New jmx:

ci: https://github.com/jianghuazhu/ozone/actions/runs/10635448262
@fapifta , can you help review this PR?
Thanks.

Thanks @fapifta  for the comment and review.
I will improve requestCounter later and record separate metrics for different replication types.
I think timeout and skip should not be treated as failures, which makes it clearer. Otherwise, the metrics recorded by failureCounter are easy to confuse.
This is my idea, what do you think? @fapifta

I have updated some of the code.
@sodonnel , can you review again?
Thanks.
ci: https://github.com/jianghuazhu/ozone/actions/runs/10738626305

Thanks for updating - I have just one further suggestion commented above, otherwise the changes look good.

Hi @jianghuazhu thanks for working on this. This will be useful for container reconciliation in HDDS-11345 as well. I think we still have to work out the differences in how metrics are tracked in between datanode commands that go through the replication supervisor and those that are handled outside of it. For example in ReplicateContainerCommandHandler:
* `totalTime` and `invocationCount` are never assigned. Their related metrics will always be 0.

  * There is no analogous metric for execution times in the supervisor to replace these.

* Queued count is pulled from the supervisor and propagated to the command handler metrics.

  * The supervisor does not expose this metric directly, even though it tracks it.

Basically the way this change is implemented I think it is confusing whether to look for a metric in the replication supervisor or the command handler. IMO all datanode commands should properly fill in a core common set of metrics, and the replication supervisor metrics should extend that with additional data for replication specific commands.

I had a look, and these problems of consistency are pre-existing. I think this PR is an improvement, but there is scope for further changes to bring in the total time and invocation count to the supervisor metrics.
The way the code is structured, the ReplicationContainer command handler simply passes the task to the supervisor, so it cannot track the total time of execution. It could in theory pull the metric from the supervisor on demand, if the supervisor tracked it.
In some respects, replication commands are different to other commands on the DN, so it does somewhat make sense that they have their own metrics from the supervisor.
In summary, I am happy with this change and will give it a +1, but it would make sense to open a Jira and ideally a further PR to add totalTime to the supervisor metrics per command type. That said, I am not really sure how useful that is - all you can get from it is an average, and containers are all different sizes, so its not a very useful metric to analyse the replication performance.
@errose28 - what do you think about proceeding with committing this and having followup work to make things more consistent?

Thanks @sodonnel  @errose28 .
I will continue to work on HDDS-11444.


fix: add .railwayapp with default settings.yaml

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Non-functional change (docs, style, minor refactor)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my code changes.
[ ] All new and existing tests passed.

Seems the best way to deal with this would be to maintain a branch called railway-app which has the .nostr/settings.yaml dir + file set up and an updated railway-app template, until we refactor the relay to use dynamic configs.

When the merge this pull request ?

@KiPSOFT it's not ready yet, I'm currently working on it. Should be ready by EOD.

https://github.com/Cameri/nostream/compare/main...investinaccess:nostream:main


Update views to handle nested points

To output nested points, use serialization defined in values.js.
For csv text view, output arrays and objects in 'inspect' format, as the
brackets give a little bit more context while parsing the output
visually.

cc: @dmajda
refs: #55

+1
The values.toString thing is not a blocker.


Preserve history from cmsis-svd/cmsis-svd

The move of the data/ folder from cmsis-svd/cmsis-svd to cmsis-svd/cmsis-svd-data dropped the commit history.
I tinkered a little and found a way to preserve commit history of all commits that changed files in data/ (using git-filter-repo). I cannot open a PR because github won't let me (due to rewriting the history). But here's the script to do the changes that I did in gschwaer/cmsis-svd-data:
#!/usr/bin/env bash
set -eu

git clone git@github.com:cmsis-svd/cmsis-svd.git
cd cmsis-svd

# Discard all commits after (and including) the move.
git reset --hard ":/Remove data directory."
git reset --hard HEAD^

# Keep all commits that changed `data/`.
git filter-repo --path data/ --force

# Update the merge commit messages to point to `cmsis-svd` repo pull requests.
git filter-repo --replace-message <(echo "regex:^Merge pull request #==>Merge pull request cmsis-svd/cmsis-svd#")
git branch -M filtered-main

# Add new commits from `cmsis-svd-data` since the move on top and clean up.
git remote add origin git@github.com:cmsis-svd/cmsis-svd-data.git
git fetch origin
git switch main
git rebase -i --rebase-merges --onto=filtered-main --root
### Make sure todo content is as follows:
### For a771c3b I changed pick to drop, to remove the previous commit of `data/`.
### For 23899fa I moved it below 2fc4f10 and changed `pick` to `fixup -C` to get rid of 2fc4f10 which only contained an empty README.md
# label onto
#
# # Branch cmsis-svd-add-cmsis-svd-files-from-cmsis-svd-repository
# reset onto
# pick 2fc4f10 Init
# fixup -C 23899fa Add README
# label branch-point
# drop a771c3b Add cmsis-svd files from cmsis-svd/cmsis-svd repository.
# pick a2ff679 Add Apache license.
# pick be27b47 Add CONTRIBUTING
# label cmsis-svd-add-cmsis-svd-files-from-cmsis-svd-repository
#
# # Branch florianhofhammer-add-nxp-mk64f12
# reset cmsis-svd-add-cmsis-svd-files-from-cmsis-svd-repository # Add README
# pick eb72d1c Add NXP/MK64F12 SVD
# label florianhofhammer-add-nxp-mk64f12
#
# # Branch aykevl-espressif-update
# reset branch-point # Init
# merge -C ff486d9 cmsis-svd-add-cmsis-svd-files-from-cmsis-svd-repository # Merge pull request #1 from cmsis-svd/add-cmsis-svd-files-from-cmsis-svd-repository
# merge -C 853bb02 florianhofhammer-add-nxp-mk64f12 # Merge pull request #2 from florianhofhammer/add-nxp-mk64f12
# label branch-point-2
# pick 58fe151 Update Espressif SVD files
# label aykevl-espressif-update
#
# reset branch-point-2 # Merge pull request #2 from florianhofhammer/add-nxp-mk64f12
# merge -C 40327a4 aykevl-espressif-update # Merge pull request #3 from aykevl/espressif-update

# That was all. Now the repo can be pushed, but we changed the history so a force push is required (commented out for now):
# git push -f origin main

# Let's diff against current `cmsis-svd-data` repo, to check that we didn't drop anything.
cd ..
git clone git@github.com:cmsis-svd/cmsis-svd-data.git
diff -r -x .git cmsis-svd/ cmsis-svd-data/
echo $?  # Outputs 0: Folders have identical content

I used
$ git --version
git version 2.43.0
$ git filter-repo --version
ae71fad1d03f

@gschwaer  Thank you very much for your proposal, this is very VERY welcome :) , I will review your script today.
@BenBE , @brainstorm, @ckudera, @posborne: This is ok for everybody if I force push here in order to restore the history.

IMO, whenever possible the version control should preserve relevant history. So yes, go ahead.

Issue completed, @BenBE thank you very much for your help and creation of the branch and PR for the PR (#5, #6, #7, #8, #9) accidentally closed caused by my force push.
@rminnich, @ii8, @PfannenHans, @AverseABFun, @zkrx: I am reviewing your PR now.


Issue 3378 - Fix teardown scripts

Make sure you have checked all steps below.
Issue

[x] My PR addresses the following issues and references them in the PR title. For example, "Issue 1234 - My Sleeper
PR"

Resolves https://github.com/gchq/sleeper/issues/3378

Tests

[x] My PR adds the following tests OR does not need testing for this extremely good reason:

Tested in AWS

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
separate issue for that below.
[x] If I have added, removed, or updated any external dependencies used in the project, I have updated the
NOTICES file to reflect this.

Merging now as otherwise the nightly system tests will fail to tear down.


Login fails after :logout

After issuing :logout <user>:>server>, logging back in fails with following error after password is inserted.
$ iamb
* Logging in for @XXXXX:matrix.org...
Please select login type: [p]assword / [s]ingle sign on
p
Password:
Failed to login: Matrix client error: failed to read or write to the crypto store the account in the store doesn't match the account in the constructor: expected @XXXXX:matrix.org:EW........, got @XXXXX:matrix.org:TK........
Please select login type: [p]assword / [s]ingle sign on
^C

It seems that one workaround is to remove .sqlite3* files under /local/share/iamb/profiles/user/sqlitedir.
$ pwd
/home/<user>/.local/share/iamb/profiles/user/sqlite
$ rm *.sqlite3*
$ iamb
* Logging in for @kopperi:matrix.org...
Please select login type: [p]assword / [s]ingle sign on
p
Password:
* Successfully logged in!
* Syncing...
* ...

Used versions of iamb and rustc used to compile it, on Debian 12 Bookwarm host.
 $ iamb --version
iamb 0.0.9 (VERGEN_IDEMPOTENT_OUTPUT)

$ rustc --version
rustc 1.78.0 (9b00956e5 2024-04-29)

Please advise on any other logs or debug info you may need.
EDIT: Issue at matrix-org seems similar https://github.com/matrix-org/matrix-rust-sdk/issues/3270

After issuing :logout <user>:>server>, logging back in fails with following error after password is inserted.
$ iamb
* Logging in for @XXXXX:matrix.org...
Please select login type: [p]assword / [s]ingle sign on
p
Password:
Failed to login: Matrix client error: failed to read or write to the crypto store the account in the store doesn't match the account in the constructor: expected @XXXXX:matrix.org:EW........, got @XXXXX:matrix.org:TK........
Please select login type: [p]assword / [s]ingle sign on
^C

It seems that one workaround is to remove .sqlite3* files under /local/share/iamb/profiles/user/sqlitedir.
$ pwd
/home/<user>/.local/share/iamb/profiles/user/sqlite
$ rm *.sqlite3*
$ iamb
* Logging in for @kopperi:matrix.org...
Please select login type: [p]assword / [s]ingle sign on
p
Password:
* Successfully logged in!
* Syncing...
* ...

Used versions of iamb and rustc used to compile it, on Debian 12 Bookwarm host.
 $ iamb --version
iamb 0.0.9 (VERGEN_IDEMPOTENT_OUTPUT)

$ rustc --version
rustc 1.78.0 (9b00956e5 2024-04-29)

Please advise on any other logs or debug info you may need.
EDIT: Issue at matrix-org seems similar matrix-org/matrix-rust-sdk#3270

I’m experiencing the same issue. I use SSO login on Mozilla’s server, and I’m encountering the same failure. By the way, thank you for sharing the solution!

After issuing :logout <user>:>server>, logging back in fails with following error after password is inserted.
$ iamb
* Logging in for @XXXXX:matrix.org...
Please select login type: [p]assword / [s]ingle sign on
p
Password:
Failed to login: Matrix client error: failed to read or write to the crypto store the account in the store doesn't match the account in the constructor: expected @XXXXX:matrix.org:EW........, got @XXXXX:matrix.org:TK........
Please select login type: [p]assword / [s]ingle sign on
^C

It seems that one workaround is to remove .sqlite3* files under /local/share/iamb/profiles/user/sqlitedir.
$ pwd
/home/<user>/.local/share/iamb/profiles/user/sqlite
$ rm *.sqlite3*
$ iamb
* Logging in for @kopperi:matrix.org...
Please select login type: [p]assword / [s]ingle sign on
p
Password:
* Successfully logged in!
* Syncing...
* ...

Used versions of iamb and rustc used to compile it, on Debian 12 Bookwarm host.
 $ iamb --version
iamb 0.0.9 (VERGEN_IDEMPOTENT_OUTPUT)

$ rustc --version
rustc 1.78.0 (9b00956e5 2024-04-29)

Please advise on any other logs or debug info you may need.
EDIT: Issue at matrix-org seems similar matrix-org/matrix-rust-sdk#3270

I’m experiencing the same issue. I use SSO login on Mozilla’s server, and I’m encountering the same failure. By the way, thank you for sharing the solution!


Use protobuf for client-server communication

[x] define protobuf files for requests / responses ( replace yaml files - no longer needed )

Protobuf objects and services defined.


Continuous label as a vector

Hello!
Thanx for the paper, I find the proposed approach new and promising.
I have a question for the authors: say, I do not have a distinct label for each image, but rather a continuous vector of size (1, 3). Is it still possible to use this approach? In this case, a number of distinct classes is almost equal to the distinct images in the dataset.

Hi,
Thanks for your interest in our work.
For your question, if my understanding is correct, you want to replace the continuous, scalar condition with a continuous, multivariate condition? If so, I think our approach is still applicable once you can define a reasonable metric between any two such multivariate conditions and you have an efficient way to compute it during training. However, we didn't conduct an experiment in such a scenario, but I think it worths a try.

Hi,
Thanks for your interest in our work.
For your question, if my understanding is correct, you want to replace the continuous, scalar condition with a continuous, multivariate condition? If so, I think our approach is still applicable once you can define a reasonable metric between any two such multivariate conditions and you have an efficient way to compute it during training. However, we didn't conduct an experiment in such a scenario, but I think it worths a try.

Thank you for the answer! Btw, congrats with ICLR 2021 acceptance!
Just to clarify some things: you guessed right, I meant multivariate condition. However, I still struggle with hyperparameter search. For evaluation of kappa, we need to sort the labels in ascending order, then find pairwise difference and take max (if I get it right). But in the multivariate setting I cannot yet think of a way to calculate kappa. May you suggest anything to think about in this context?

Thank you for the answer! Btw, congrats with ICLR 2021 acceptance!
Just to clarify some things: you guessed right, I meant multivariate condition. However, I still struggle with hyperparameter search. For evaluation of kappa, we need to sort the labels in ascending order, then find pairwise difference and take max (if I get it right). But in the multivariate setting I cannot yet think of a way to calculate kappa. May you suggest anything to think about in this context?

Thanks!
Yes, you are right. It is more difficult to compute kappa in the multivariate setting than the univariate setting. The rule of thumb for the hyper-parameter tuning in the univariate setting may not be applicable in the multivariate setting, since sorting distinct labels, as the first step of the rule of thumb, is not applicable anymore. Unfortunately, I don't have a good solution right now. As a makeshift, you may set kappa as a percentile (say 80%) of the pairwise differences (between any two distinct labels; measured by a suitable metric). Not sure whether it can work...

Thanks!
Yes, you are right. It is more difficult to compute kappa in the multivariate setting than the univariate setting. The rule of thumb for the hyper-parameter tuning in the univariate setting may not be applicable in the multivariate setting, since sorting distinct labels, as the first step of the rule of thumb, is not applicable anymore. Unfortunately, I don't have a good solution right now. As a makeshift, you may set kappa as a percentile (say 80%) of the pairwise differences (between any two distinct labels; measured by a suitable metric). Not sure whether it can work...

Ok, I see. Will try to make it work, thanks for your explanation!

Ok, I see. Will try to make it work, thanks for your explanation!


Document provisoning commands for bot.

Can I assign you to this @turt2live or do you think !discord help is sufficient?

I feel !discord help is sufficient (particularly the help provided by !discord help bridge). Was there a concern that it's unclear?

Probably in the long term I'd like webby documents for it, but I want to expand provisioning a lot more before I do that. No problem, won't stick it on this milestone.

fwiw, I'll also be adding it to t2bot.io/discord when I get the chance


Error found when using hf api

When I tried to call :
llm = NanoLLM.from_pretrained(
   model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
   api='hf',
   api_token='mytoken',
   quantization='q4f16_ft',
)

I got:

Traceback (most recent call last):
File "/root/nanollm.py", line 6, in
llm = NanoLLM.from_pretrained(
File "/opt/NanoLLM/nano_llm/nano_llm.py", line 74, in from_pretrained
model = HFModel(model_path, **kwargs)
File "/opt/NanoLLM/nano_llm/models/hf.py", line 19, in init
super(HFModel, self).init(**kwargs)
TypeError: NanoLLM.init() missing 1 required positional argument: 'model_path'

Here is the original code from nano_llm/models/hf.py:
class HFModel(NanoLLM):
    """
    Huggingface Transformers model
    """
    def __init__(self, model_path, load=True, init_empty_weights=False, **kwargs):
        """
        Initializer
        """
        super(HFModel, self).__init__(**kwargs)

The issue in the initial code is that the model_path argument is not being passed to the super class __init__ method. The corrected code snippet ensures that model_path is correctly passed to the NanoLLM class during initialization. Here is the corrected version of the class definition:
class HFModel(NanoLLM):
    """
    Huggingface Transformers model
    """
    def __init__(self, model_path, load=True, init_empty_weights=False, **kwargs):
        """
        Initializer
        """
        super(HFModel, self).__init__(model_path, **kwargs)

Hope this helps!

Thanks @Jiopro , --api=hf is working again with https://github.com/dusty-nv/NanoLLM/commit/9a0ba678d10b8cc35a44c0f901dfdda60f8107bc 👍


Show an example

It is unclear how this theme would look, I would advise to either have a sample project generate typedoc site with this theme and present it in the github pages or add a screenshot to the readme.
Thanks!

Oh yes, you're right. It's already in the works! I'm gradually converting my Github projects to this theme. After that I will provide a more comprehensive example.
Maybe that will inspire you in time?

Thank you, best regards Julian


Fixing load tuning sets

Using RandomizedScalingTimeLimited tuning set in every scaling phase.

/test pull-perf-tests-clusterloader2-kubemark

/test pull-perf-tests-clusterloader2-kubemark


Update URL for joining Slack

For new users, joining the Slack channel for draftjs is not straightforward with the old link.

Thanks @seiji619!


Introduce is distinct from predicate operation

Introduces an is distinct from predicate operation, which replaces the current != operation. The current behaviour of != is deprecated, and will be replaced in version 5.0 with an SQL-like behaviour, i.e. missing values will not be returned when using the predicate.

[x] Changelog entry

@IzerOnadimQC Can you fix the CI error (data missing for 14.0.2)? Then we can merge&release.


DO NOT MERGE: check secrets

With this PR, I check whether the necessary secrets for the release workflow are set. As the new workflow indicates, the necessary secrets are not set. @csgoh Can you please set the required secrets as described in #91, so that we can automate the release workflow?

Hi @felix-seifert I did set the secret.

What did I miss?

Hmm. My wild assumption is then that GitHub implemented a more intricate process to not spill any secrets. As I have the confirmation that the secrets are set, I can go on, and you do not have to do anything.


[on-this-page] Only use word break hints for symbol title

Bug/issue #, if applicable: 107193315
Summary
Only utilize word break hints for the title of a symbol page to avoid unnecessary breaking of prose-like content.
This is an alternate implementation of #587 that I think should be more flexible and work with a majority of content.
Testing
Steps:

Preview docs using this branch of the renderer
Verify that the first title in the on-this-page utilizes <wbr> hints only when the page is for a symbol.
Check for any other regressions with the on-this-page text

Checklist
Make sure you check off the following items. If they cannot be completed, provide a reason.

[X] Added tests
[X] Ran npm test, and it succeeded
[X] Updated documentation if necessary

@swift-ci test


missing import for annotation param

Hi,
I want to import the given param for the annotation @Scope(ConfigurableBeanFactory.SCOPE_SINGLETON)
My visitor has to do the following action:

Remove @Singleton: OK
Add @Scope annotation with ConfigurableBeanFactory.SCOPE_SINGLETON param :OK but the import for ConfigurableBeanFactory.SCOPE_SINGLETON is missing

This is my visitor
public class ReplaceAnnotationVisitor extends JavaIsoVisitor<ExecutionContext> {
    private final JavaTemplate componentAnnotationTemplate;

    private static final char at = '@';

    private static final String SINGLETON_ANNOTATION_TO_REMOVE_NAME = "javax.inject.Singleton";

    public ReplaceAnnotationVisitor() {
        componentAnnotationTemplate = JavaTemplate.builder(this::getCursor, at + "Scope(ConfigurableBeanFactory.SCOPE_SINGLETON)")
                .imports("org.springframework.context.annotation.Scope", "org.springframework.beans.factory.config.ConfigurableBeanFactor").javaParser(() -> JavaParser.fromJavaVersion()
                        .dependsOn("package org.springframework.context.annotation; public @interface " + "Scope" + " {}",
                                "package org.springframework.beans.factory.config; public interface ConfigurableBeanFactory {}"
                                ).build()).build();
    }

    @Override
    public J.ClassDeclaration visitClassDeclaration(J.ClassDeclaration classDecl, ExecutionContext ctx) {
        J.ClassDeclaration cd = super.visitClassDeclaration(classDecl, ctx);

        Optional<J.Annotation> requiredAnnotation = getAnnotationToRemove(cd);
        if (requiredAnnotation.isPresent()) {

            maybeRemoveImport("javax.inject.Singleton");

            maybeAddImport("org.springframework.context.annotation.Scope");
            maybeAddImport("org.springframework.beans.factory.config.ConfigurableBeanFactory");

            //cd = (J.ClassDeclaration) new ChangeType(SINGLETON_ANNOTATION_TO_REMOVE_NAME, "org.springframework.context.annotation.Scope", false).getVisitor().impvisit(cd, ctx, getCursor());
            cd = (J.ClassDeclaration) new RemoveAnnotation(SINGLETON_ANNOTATION_TO_REMOVE_NAME).getVisitor().visit(cd, ctx, getCursor());

            cd = cd.withTemplate(componentAnnotationTemplate,
                    cd.getCoordinates().addAnnotation(Comparator.comparing(J.Annotation::getSimpleName)));

        }
        return cd;
    }

    private Optional<J.Annotation> getAnnotationToRemove(J.ClassDeclaration cd) {
        return FindAnnotations.find(cd, at + SINGLETON_ANNOTATION_TO_REMOVE_NAME).stream().findFirst();
    }
}

Before:
import javax.inject.Singleton;

@Singleton
public class FakeSingleton {

    public void sayWhoAmI() {
        System.out.println("I am a Singleton");
    }
}

After (the import org.springframework.beans.factory.config.ConfigurableBeanFactory is missing):
import org.springframework.context.annotation.Scope;

@Scope(ConfigurableBeanFactory.SCOPE_SINGLETON)
public class FakeSingleton {

    public void sayWhoAmI() {
        System.out.println("I am a Singleton");
    }
}

Hi @jdelobel, just a friendly FYI - adding the language after likejava will add syntax to the snippets. :)

Hi @jdelobel, just a friendly FYI - adding the language after likejava will add syntax to the snippets. :)
maybeAddImport will check if the type is in use prior to adding the import. I suspect there is missing type information related to org.springframework.beans.factory.config.ConfigurableBeanFactory. I'd start by adding SCOPE_SINGLETON to the stub of ConfigurableBeanFactory in the template.

Hi @traceyyoshima. ok for the language snippets. I will add it for the next.
For my missing import I have to add explicitly this code:
     AddImport<ExecutionContext> addSConfigurableBeanFactorImport = new AddImport<>(
                    "org.springframework.beans.factory.config.ConfigurableBeanFactory",
                    null,
                    false);

But i dont think its a good practice.
Thanks,

Hi @jdelobel,

But i dont think its a good practice.

You're right 👍
Ideally, the import will be added with maybeAddImport, but the import is not added due to missing type information.
The code in dependsOn needs to be modified so that @Scope(ConfigurableBeanFactory.SCOPE_SINGLETON) is compilable.
The code snippet in the template contains:
@Scope(ConfigurableBeanFactory.SCOPE_SINGLETON)

Before:
package org.springframework.context.annotation;
public @interface Scope {}

package org.springframework.beans.factory.config;
public interface ConfigurableBeanFactory {}

After:
package org.springframework.context.annotation;
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Scope {
    String value() default "";
}

package org.springframework.beans.factory.config;
public interface ConfigurableBeanFactory {
    String SCOPE_SINGLETON = "singleton";
}

Hopefully, this helps, please reach out on Slack if you have any questions!

imports said "org.springframework.beans.factory.config.ConfigurableBeanFactor" instead of "org.springframework.beans.factory.config.ConfigurableBeanFactory"

imports said "org.springframework.beans.factory.config.ConfigurableBeanFactor" instead of "org.springframework.beans.factory.config.ConfigurableBeanFactory"

Hi @eddumelendez,
Thanks its working!
The problem was me with the copy/paste.

Hi,
Thanks to keep the issue open.
I try to use the Recipe org.openrewrite.maven.UpgradeDependencyVersion
``xml
type: specs.openrewrite.org/v1beta/recipe
name: com.socgen.digital.spm.migrate.UpgradeJavaxServletApiVersion
recipeList:

org.openrewrite.maven.UpgradeDependencyVersion:
groupId: javax.servlet
artifactId: javax.servlet-api
newVersion: 4.0.1
overrideManagedVersion: true
``
But the version declared in the dependancies mangaement is not uploaded in the pom.xml. Its working for other dependancies. Do you have any idea please?

I have another question:
When i user org.openrewrite.maven.AddManagedDependency, I have to create a new root recipe for each one for it to work. Is it possible to add multiple dependancies in one root recipe?
I donèt have slack but i will create ASAP to join your community.
Thanks

But the version declared in the dependancies mangaement is not uploaded in the pom.xml. Its working for others dependancies. Do you have any idea please?

The com.socgen.digital.spm.migrate.UpgradeJavaxServletApiVersion recipe looks valid.
Providing feedback without seeing the pom and/or the XML for the plugin configuration is challenging.
But I'd double-check the recipe is activated in the configuration.

When i use the recipe org.openrewrite.maven.AddManagedDependency, I have to create a new root recipe for each one for it to work. Is it possible to add multiple dependancies in one root recipe?

Yes, you may define multiple recipes in the recipeList for example here!
---
type: specs.openrewrite.org/v1beta/recipe
name: my.example.Recipe
recipeList:
  - org.openrewrite.maven.AddManagedDependency:
      ... params
  - org.foo.Recipe2
      ... params
  - org.foo.Recipe3
      ... params

Other References:

Java lang API migrations.
SpringBoot 2 version upgrades.

I donèt have slack but i will create ASAP to join your community.

Sounds good, sometimes we lose track of comments in GitHub, but the team and community are very responsive in slack since it's easier to track. Looking forward to seeing you in Slack!

Hi @jdelobel, I plan to close this issue in a couple of days due to inactivity. Just making sure you're all set.

I am going to close this issue if there are still outstanding items, please feel free to re-open


Back out "Verdaccio: proxy @react-native/normalize-colors from NPM"

Summary:
Original commit changeset: 4bf650310833
Original Phabricator Diff: D50298291
Differential Revision: D50300100

Closing as abandoned internally


have "conv" functions accept tuple for shape instead of separate height/width args

closing, not an issue after switch to PyTorch


ZSH Completions not working

https://github.com/tonarino/innernet/blob/9b1315b079c148daf8ae049255e1675d04c5303e/doc/innernet.completions.zsh#L326
The double-quotes surrounding the "wg-quick down <interface>" need to be replaced with single-quoted 'wg-quick down <interface>' to get it working again.

Thanks for the issue @darthShadow! Looks like we hit https://github.com/clap-rs/clap/issues/1596. I'll switch our " to 's.


Unicorn Dependency cannot be resolved

$ pip install simuvex
Collecting simuvex
Using cached simuvex-<IP_ADDRESS>.tar.gz
Collecting bintrees (from simuvex)
Using cached bintrees-2.0.4.tar.gz
Collecting unicorn (from simuvex)
Could not find a version that satisfies the requirement unicorn (from simuvex) (from versions: )
No matching distribution found for unicorn (from simuvex)

I think the package is called pyunicorn on my system.

$ pip search pyunicorn
pyunicorn (0.5.1)  - Unified complex network and recurrence analysis toolbox

$ cat /etc/fedora-release
Fedora release 23 (Twenty Three)

I am not really sure how to fix this bug. I could of course edit requirements.txt and setup.py, but then it would break at the computers of the people where it is named unicorn.

I have made more tests in more virtualenvs and found out, that this only concerns the version of pyvex which angr downloads. The problem above was that pyvex was cached. If I first install pyvex and then angr, everything works. I do not know why. Since I have managed to install it, this issue may be closed, if no-one wants to inspect it further.

I believe that the issue is we are not correctly stating versions of those libraries that angr depends on (like angr to SimuVEX, which usually requires the latest version) at all in our PyPI releases. See the issue in angr-dev.

unicorn is not on pypi yet!!! I don't know why we have unicorn listed as a dependency!

lol
When are we getting it onto PyPI?

https://github.com/unicorn-engine/unicorn/pull/621
see last comments. the remaining PR has not been merged yet.


The test InitializeSpringBootMigrationRecipeIntegrationTest passes but it actually does not compare the migrated code

reported-by: @ahmedmq
Describe the bug
the test InitializeSpringBootMigrationRecipeIntegrationTest passes but it actually does not compare the migrated code and the expected code(empty-project)
Additional context
see https://github.com/spring-projects-experimental/spring-boot-migrator/issues/284#issuecomment-1263088314

assigned-to: @ahmedmq


Introduce dedicated error types to the lapin-futures crate

This change replaces all occurrences of io::Error by the newly created
Error type, the only exception being for AMQPCodec. This change allows
future-proofing the error that can be returned by the crate. For example
by changing the internals of the Error type, it is now possible to
provide users with a way of knowing what kind of error occurred without
having to resort to parsing error messages.
The only exception to this change concerns AMQPCodec: a new CodecError type
was created just for it because tokio-codec's Encoder and Decoder traits
impose a From<std::io::Error> bound of the error type. Because I didn't
want to introduce a way to create an Error instance from outside the
crate, I isolated the possible errors from this section into its own
type.

@Keruspe rebased onto current master, awaiting for your review :)

Thanks a lot for this. Always wanted to give failure a look and never had the occasion before


Hacktoberfest

TRash.

Thanks for taking the time to contribute this pull request. However, we are not able to accept contributions to this historical reference repository at this time.


In ADFS, when adding an unknown Group in the Groups Page on Rancher using the text box it will create a User

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
With an ADFS installation try to add a group under the Group Memberships page
If you use a random string say test-dom\iamagroup to search for the group but then hit submit it will add the group you typed but as a User rather than reject it (and we can see it is indexing which groups are available so I am more surprised it didn't outright reject the creation if the group doesn't exist).
I am getting a screenshot for an example
Result:
It seems you can add random users to Rancher by misusing this functionality, expected behaviour should be that you can type in groups by name but if one doesn't exist it will reject it
Related:
https://github.com/rancher/rancher/issues/28785
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.4.5
Installation option (single install/HA): HA

gz#12110

Internal reference: SURE-3143
Reported in 2.6.8 & 2.6.6
Backend looked into reproducing the original issue and @KevinJoiner determined that this appears to be a UI bug that does not have a security impact:

When text is entered instead of using the dropdown selection, a new GlobalRoleTemplateBinding is created with the following structure.

        {
            "type": "globalRoleBinding",
            "globalRoleId": "admin",
            "groupPrincipalId"<EMAIL_ADDRESS>        }

This binding will never resolve to anything since a user principal domain is being set for the groupPrincipalID.

This is a behaviour that has happened in both Ember & Vue.

This may be resolved via other work. @richard-cox to confirm.

This has been resolved via https://github.com/rancher/dashboard/pull/7386 (https://github.com/rancher/dashboard/issues/3182). As there's a lot more detail and history here suggest QA take a look to validate

Already tested via https://github.com/rancher/dashboard/issues/3182

Reopening for specific validations

🚧 Reopened
Reproduction Environment

Component
Version / Type

Rancher version
v2.7.0

Installation option
Helm (high availability)

If Helm Chart k8s cluster
v1.24.7+k3s1

Cert Details
Let's Encrypt / traefik

k3s ha external db
Aurora MySQL 5.7.mysql_aurora.2.10.2

Helm version
v2.16.8-rancher2

Downstream cluster type
not applicable

Downstream K8s version
not applicable

Authentication providers enabled
Shibboleth

Logged in user role
Administrator

Browser type
Google Chrome

Browser version
108.0.5359.124 (Official Build) (x86_64)

🚨 Additional Reproduction Setup Details: Click to Expand

K3s install command for node one. Replace values in <>

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<K3S_VERSION> sh -s - server --token=SECRET --datastore-endpoint=<DB_ENDPOINT> --tls-san <TLD> --node-external-ip <IP>

K3s install command for node two. Replace values in <>. Main difference here is the --token

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<K3S_VERSION> sh -s - server --token=<TOKEN_FROM_NODE_ONE> --datastore-endpoint=<DB_ENDPOINT> --tls-san <TLD> --node-external-ip <IP>

Reproduction steps

Login to Rancher as an admin
Enable Shibboleth (SAML) authentication provider
Once these two steps are completed, do the following

Starting from the default Rancher homepage /dashboard/home
Click hamburger menu >>> Users & Authentication >>> Groups >>> Assign Global Roles >>> click the Select Member dropdown
A valid group is displayed
Instead of selecting the group type a random user/group name into the field such as thisreallydoesntexist-dadfish75
For Global Permission select Administrator and click >>> Save

The save is successful without error
Now use kubectl to check the global role bindings with:

kubectl get globalrolebindings -A

Look for the name of the most recently created role binding and run:

kubectl describe globalrolebinding grb-cm27t

Observe the field: Group Principal Name:  shibboleth_user://thisreallydoesntexist-dadfish75

Additional Info
RESULTS
✅ Expected
For the UI not to allow the creation of these group global roles with a group that doesn't exist and seem to create a user and role binding from the name that was entered
❌ Actual
The UI allows for this behavior to create a group global role with a username that doesn't exist and it creates a global role binding with the name searched for

Validation Environment

Component
Version / Type

Rancher version
v2.7-d657043de1dc2110562d9035b6c1c6dc9d9232ed-head

Rancher commit link
https://github.com/rancher/rancher/commit/d657043de1dc2110562d9035b6c1c6dc9d9232ed

Installation option
Helm (high availability)

If Helm Chart k8s cluster
v1.24.7+k3s1

Cert Details
Let's Encrypt / traefik

k3s ha external db
Aurora MySQL 5.7.mysql_aurora.2.10.2

Helm version
v2.16.8-rancher2

Downstream cluster type
not applicable

Downstream K8s version
not applicable

Authentication providers enabled
Shibboleth

Logged in user role
Administrator

Browser type
Google Chrome

Browser version
108.0.5359.124 (Official Build) (x86_64)

🚨 Additional Reproduction Setup Details: Click to Expand

K3s install command for node one. Replace values in <>

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<K3S_VERSION> sh -s - server --token=SECRET --datastore-endpoint=<DB_ENDPOINT> --tls-san <TLD> --node-external-ip <IP>

K3s install command for node two. Replace values in <>. Main difference here is the --token

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<K3S_VERSION> sh -s - server --token=<TOKEN_FROM_NODE_ONE> --datastore-endpoint=<DB_ENDPOINT> --tls-san <TLD> --node-external-ip <IP>

Validation steps

Login to Rancher as an admin
Enable Shibboleth (SAML) authentication provider
Once these two steps are completed, do the following

Starting from the default Rancher homepage /dashboard/home
Click hamburger menu >>> Users & Authentication >>> Groups >>> Assign Global Roles >>> click the Select Member dropdown
A valid group is displayed
Instead of selecting the group type a random user/group name into the field such as thisreallydoesntexist-dadfish42
For Global Permission select Administrator and click >>> Save

The save is successful without error
Now use kubectl to check the global role bindings with:

kubectl get globalrolebindings -A

Look for the name of the most recently created role binding and run:

kubectl describe globalrolebinding grb-m2kbz

Observe the field: Group Principal Name:  shibboleth_user://thisreallydoesntexist-dadfish42

Additional Info
RESULTS
✅ Expected
For the UI not to allow the creation of these group global roles with a group that doesn't exist and seem to create a user and role binding from the name that was entered
❌ Actual
The UI allows for this behavior to create a group global role with a username that doesn't exist and it creates a global role binding with the name searched for
Screenshots

The fix (https://github.com/rancher/dashboard/pull/7386) is targetted for 2.7.2 and is in 2.7-head and I couldn't reproduce the bug there. @brudnak Your comment references 2.7.0, can you confirm you're seeing that behaviour in 2.7-head?

The fix (#7386) is targetted for 2.7.2 and is in 2.7-head. I couldn't reproduce with that version. @brudnak Your comment references 2.7.0, can you confirm you're seeing that behaviour in 2.7-head?

Hi @richard-cox - I reproduced to confirm behavior with 2.7.0, but also if you check the second section I was still seeing this on 2.7-head v2.7-d657043de1dc2110562d9035b6c1c6dc9d9232ed-head https://github.com/rancher/rancher/commit/d657043de1dc2110562d9035b6c1c6dc9d9232ed
The screenshot of still seeing this issue have d657043 in the bottom left corner in the UI.
Was not seeing this with GitHub auth, which I saw GitHub auth mentioned in a linked issue: https://github.com/rancher/dashboard/issues/3182. Was seeing this while using Shibboleth.

Strange, we don't differentiate between auth providers at that point. In the browser's dev tool's network tab can you check what the response is for /v3/principals?action=search after you enter the bad text?

@richard-cox sure thing - the response after entering the bad text is a 200 POST to /v3/principals?action=search, see screenshot below

@brudnak Brilliant, thanks.
Looks like the BE is returning a group given the value the user has searched for. There could be a reason for this (the BE is unable to search for all groups, so trusts user input) or it's a bug. From the comments above @KevinJoiner might be able to provide in put?
@KevinJoiner The UI now ensures only groups returned from the search (/v3/principals?action=search) can be selected. It seems when the user enters a junk group name some providers like Github will ignore it, whilst others like Shibboleth will return it as a valid group. From a UI side there's not much we can do, should this be a new issue over at rancher/rancher or shall this one move over?

I took a quick look at some of the code, and it looks like the behavior you are seeing was intentionally done for SAML and OIDC providers. I am unsure why this is the case, and I will need to do more research.
@richard-cox  do you know what other places the UI uses the search action for principles? I am guessing adding the user's search input to the returned list was originally for the UI, but I am unsure if Rancher still uses this functionality.
If somewhere else needs this functionality, we will need to come up with a fix to make both scenarios happy, and we should leave this issue open.
If Rancher does not require this functionality elsewhere, then we should be able to leave your changes as is and create a separate rancher/rancher issue for the backend to stop sending the user's input in the returned list.

So, I found more info on what we are seeing in our docs.  https://ranchermanager.docs.rancher.com/v2.6/pages-for-subheaders/configure-shibboleth-saml#saml-provider-caveats This implies what we are seeing is expected behavior.
We may need to include OIDC in this section.
@brudnak Based on our docs, this is expected behavior, so I don't think we can consider it a separate bug. Thus we can probably close this issue unless there is something else you want to test.

I know there's scaling issues to consider (some enterprise customers will have many many users / groups), i'm not sure how performant ad-hoc searches for both are.

Each provider defines their own search implementation. But many perform two separate searches for users and groups and then concatenate the results.

✅ PASSED
Reproduction Environment
Not required.

Validation Environment

Component
Version / Type

Rancher version
v2.7-1dd2af2d214b159c2c8ffcfc5838b6caece381cd-head

Installation option
Helm (high availability)

RKE binary version used
v1.4.0

If Helm Chart k8s cluster
v1.24.6

Cert Details
Let's Encrypt / nginx

Docker version
20.10.7, build f0df350

Helm version
v2.16.8-rancher2

Downstream cluster type
not applicable

Downstream K8s version
not applicable

Authentication providers enabled
GitHub OAuth

Logged in user role
admin, standard user

Browser type
Edge

Browser version
110.0.1587.41

Validation steps

Login to Rancher as an admin
Enable GitHub (OAuth) authentication provider
Once these two steps are completed, do the following

Starting from the default Rancher homepage /dashboard/home
Click hamburger menu >>> Users & Authentication >>> Groups >>> Assign Global Roles >>> click the Select Member dropdown
A valid group is displayed
Instead of selecting the group type a random user/group name into the field such as thisreallydoesntexist-dadfish75

Unable to create assign a group that doesn't actually exist

Additional Info
RESULTS
✅ Expected
Unable to assign a group that doesn't exist
✅ Actual
Unable to assign a group that doesn't exist