Ammonite SSHD server: Connection closed

I've tried SSHD server in a small project and it runs ok.
But in a bigger project (a webapp), here is the result:
java.lang.NoClassDefFoundError: org/bouncycastle/crypto/prng/VMPCRandomGenerator        at org.apache.sshd.common.random.BouncyCastleRandom.<init>(BouncyCastleRandom.java:56)
        at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:48)
        at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:41)
        at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:37)
        at org.apache.sshd.SshBuilder$BaseBuilder.fillWithDefaultValues(SshBuilder.java:117)
        at org.apache.sshd.SshBuilder$ServerBuilder.fillWithDefaultValues(SshBuilder.java:365)
        at org.apache.sshd.SshBuilder$ServerBuilder.fillWithDefaultValues(SshBuilder.java:361)
        at org.apache.sshd.SshBuilder$BaseBuilder.build(SshBuilder.java:234)
        at org.apache.sshd.SshBuilder$BaseBuilder.build(SshBuilder.java:254)
        at org.apache.sshd.SshServer.setUpDefaultServer(SshServer.java:366)
        at ammonite.sshd.SshServer$.apply(SshServer.scala:21)
        at ammonite.sshd.SshdRepl.sshd$lzycompute(SshdRepl.scala:31)
        at ammonite.sshd.SshdRepl.sshd(SshdRepl.scala:31)
        at ammonite.sshd.SshdRepl.start(SshdRepl.scala:35)

By adding this dependency, the problem is solved:
"org.bouncycastle" % "bcprov-jdk15on" % "1.55"
But when I start a session in the SSHD server here is the result:
ssh -p 22222 user@localhost
Password authentication
Password:
Compiling replBridge.sc
Connection to localhost closed.

The connection is closed immediately.  No chance to use it.

I've discovered why I needed this dependency:
"org.bouncycastle" % "bcprov-jdk15on" % "1.55"

It is because in my app already had an old version of this library in the CLASSPATH.

Any clue please why my SSH session gets closed?

Another case:
$ ssh -p 20616 localhost
The authenticity of host '[localhost]:20616 ([<IP_ADDRESS>]:20616)' can't be established.
RSA key fingerprint is d2:e2:f2:62:b0:8a:d7:71:a4:0b:15:55:b6:d9:d8:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:20616' (RSA) to the list of known hosts.
Password authentication
Password:
Compiling replBridge.sc
Compiling interpBridge.sc
Compiling HardcodedPredef.sc
Compiling ArgsPredef.sc
Compiling predef.sc
Compiling SharedPredef.sc
Compiling LoadedPredef.sc
Welcome to the Ammonite Repl 0.8.0
(Scala 2.11.7 Java 1.8.0_101)
bash: /dev/tty: No such device or address
bash: /dev/tty: No such device or address
Connection to localhost closed.

More detailed info:
# ssh -vvv -p 20616 ema@localhost
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [<IP_ADDRESS>] port 20616.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/identity-cert type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version SSHD-CORE-0.14.0
debug1: no match: SSHD-CORE-0.14.0
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit<EMAIL_ADDRESS>debug2: kex_parse_kexinit<EMAIL_ADDRESS>debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 530/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 1149
debug3: put_host_port: [<IP_ADDRESS>]:20616
debug3: put_host_port: [localhost]:20616
debug3: check_host_in_hostfile: host [localhost]:20616 filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [localhost]:20616 filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 14
debug1: Host '[localhost]:20616' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:14
debug2: bits set: 521/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1165
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1213
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/identity ((nil))
debug2: key: /root/.ssh/id_rsa ((nil))
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug3: Wrote 64 bytes for a total of 1277
debug1: Authentications that can continue: password,keyboard-interactive
debug3: start over, passed a different list password,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: Wrote 96 bytes for a total of 1373
debug2: input_userauth_info_req
Password authentication
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 14 padlen 18 extra_pad 64)
debug3: Wrote 80 bytes for a total of 1453
debug1: Authentication succeeded (keyboard-interactive).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug3: Wrote 64 bytes for a total of 1517
debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env HOSTNAME
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env SSH_CLIENT
debug3: Ignored env SSH_TTY
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = de_DE.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env PS1
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env SSH_CONNECTION
debug1: Sending env LC_CTYPE = de_DE.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env LESSOPEN
debug3: Ignored env G_BROKEN_FILENAMES
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: Wrote 528 bytes for a total of 2045
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Welcome to the Ammonite Repl 0.8.0
(Scala 2.11.7 Java 1.8.0_101)
bash: /dev/tty: No such device or address
bash: /dev/tty: No such device or address
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: channel 0: close_fds r -1 w -1 e 6
debug3: Wrote 32 bytes for a total of 2077
debug3: Wrote 64 bytes for a total of 2141
Connection to localhost closed.
Transferred: sent 1944, received 2296 bytes, in 0.2 seconds
Bytes per second: sent 10887.0, received 12858.3
debug1: Exit status 0

I've also tried unsuccessfully the -tt option to the ssh client.


z_arch_cpu_halt() should enter deep power-down where supported

Are we assuming that power consumption does not matter here?
Originally posted by @pizi-nordic in https://github.com/zephyrproject-rtos/zephyr/pull/18977/files

it's just work that needs to be done, I'm sure a lot of people would be happy if you wanted to implement this for ARM (overriding the __weak placeholder implementation in the kernel)

isnt z_arch_cpu_halt declared weak for a reason here? You can always override it. Not sure what the enhancement request is..

closing, this is outdated, this API does not exist anymore.


Bump framework to bring in grammatically correct G-Cloud 8 message.

Bump framework to bring in grammatically correct G-Cloud 8 message.
https://www.pivotaltracker.com/story/show/120682045

:+1: after a rebase.


controladdin missing in Type map

The Type "controladdin" is missing in the Type Map.
I think a Abbreviation is not necessary. Due to absence of Object Id the filenames are not to long.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3c7b7057-f9c5-6587-5d8b-267935685262
Version Independent ID: 11a70e81-f0a2-ec66-d63e-d1b0b5edeb5c
Content: Best Practices for AL code - Business Central
Content Source: dev-itpro/compliance/apptest-bestpracticesforalcode.md
Service: dynamics365-business-central
GitHub Login: @SusanneWindfeldPedersen
Microsoft Alias: solsen

Hi, thanks for your feedback. I've updated the topic which will be available with the next doc refresh. Please use ConAddIn for controladdin objects. Thanks.


Performance issue in IE11

@Jeroen06 reported it in #98

But the performance is not very good in IE11, there is noticable lag when typing and I can see the CPU usage rise to ~35% while typing. In Chrome there is no lag, and the CPU usage stays way lower (~ 1%).
I also tested it in IE11 on http://summernote.org/ and typing over there is allot smoother.

Hello,
Has this issue been fixed? Or is there any hotfix for this performance issue in IE 11 that i can do as a workaround? Any help would be genuinely appreciated!
Thanks,
SAS


Reduced latency of board menu actions + fixes.

Coverage increased (+0.05%) to 47.542% when pulling 48b4d18bebae5424f3032a5d8a254451ffdf1f03 on performance into 2bd8798063952f1e9ceac1c9ccf7a274970e4080 on master.


[BUG] Docker CMD not handled correctly

Describe the bug
If you try and run an docker-compose file like this:
services:
  t1:
    image: bash
    command: "echo hello && echo bye"

A component state like this will be written to the exo state.json:
        "6vcn55x4egy88ppmhr2deqbm3c": {
          "name": "t1",
          "type": "container",
          "spec": "image: bash\ncommand: echo hello \u0026\u0026 echo bye\n",
          "state": "",
          "created": "2021-08-18T11:00:04.4356297Z",
          "initialized": null,
          "disposed": null
        },

This then will fail to unmarshal:
❯ dexo run docker-compose.y*
2021/08/18 12:00:04 Telemetry disabled; not collecting statistics.
GUI available at: http://localhost:4001/#/workspaces/da9c7tmnvmmpvq9tpkhw69sdg8
Job URL: http://localhost:4001/#/jobs/gjh750xzg9dsrspw8dgs8kv60r
applying unmarshalling spec: 1:12: && is not a valid word
✓ ├─ adding test
⨯ └─ adding t1 unmarshalling spec: 1:12: && is not a valid word
Error: applying manifest: job failure: unmarshalling spec: 1:12: && is not a valid word
applying manifest: job failure: unmarshalling spec: 1:12: && is not a valid word

This is particularly bad because the state file is now corrupted and the error will continue to happen even if the command is changed to no longer include the offending characters.
❯ dexo rm t1
2021/08/18 12:06:51 Telemetry disabled; not collecting statistics.
Job URL: http://localhost:4001/#/jobs/v1kny9fkgksswmhkm238dskjt8
deleting unmarshalling spec: 1:12: && is not a valid word
⨯ └─ t1 unmarshalling spec: 1:12: && is not a valid word
Error: job failure: unmarshalling spec: 1:12: && is not a valid word
job failure: unmarshalling spec: 1:12: && is not a valid word

The fix in #239 is incomplete. The compose reference says the correct behavior is defined as per the CMD directive, defined in the Dockerfile spec. Here:
https://docs.docker.com/engine/reference/builder/#cmd
Looks like the right thing to do is to accept either a individual string or an array of strings. If an individual string, invoke the SHELL directly.


EE mysql connect

Install MySQL manually, caused EE doesn't know the connection.

You have to create a file my.cnf with database root credentials in /etc/mysql/conf.d.
It's explained in my article.


use NamedDims unname to remove names

unname is the operation that semantically does this:
See:
https://github.com/invenia/NamedDims.jl/blob/942bb283c62dd64c0ca484bda19c1bb832f268fa/src/name_operations.jl#L45-L46
Under the hood it just calls parent but seem more semantic to use unname.
Potentially could just replace all uses of strip_names with unname

Thanks! Happy to merge when CI passes (looks like unname is not being imported from NamedDims).


Type inference when using extend in a generic is not consistent

🔎 Search Terms
type inference when extend condition for a generic as const narrow inconsistent
🕗 Version & Regression Information

This is the behavior in every version I tried

⏯ Playground Link
https://www.typescriptlang.org/play/?#code/CYUwxgNghgTiAEAzArgOzAFwJYHtVIEYAeAFQD4AKADwC54SBKOkgbgCgA3WeKA+AXkIUA5AAkQECDmEMW8APTz4AZwwwsqAOZs2oSLAQp02PEgBMpeCCoYQqYMpVqNmyrXpN67LjB5mB5iLiktKyCkoARMFSEfBEALTx8ADuABYAniqpOMgQwPAYqQjWtvbwYKlQWgiFCMlQmRjpAA4IWI4aiCAwcMAA-PCiOMnwwDjwAJLwyMo1ReV4wFgm+ABGyBjwALZQANZtm53dvTyOqupafTp60HBIaJi4+IgAzJYldg5OF5rwAD7wVDILarbpuZieVicbhQF4BV5BCRSGRyRTwKJInCxBJJQrteD42oqKBbBBQRxEqCrHAcEAAGng602ymyIyg01mYHJdQykxSOTyKSqmzxjlBlQ4uBg13At0MDxWSAALO8bJ8zs4tABtAC64I8zG8MKV8KVFC1Ykxwh1YTR5xcuriiRSvOUyGAoFQGAgmWWBMcqBwh1QXR6IHy5IWqFUV10soM92MTyQAFZVaUvtScBAQFVdfrGIbob4oCn4SnzYgoBBZjbUUoqzWQI60TiXZlI2AcMovUgcL4szmqmKNoCg33fPatB18KKOQgubM+kA
💻 Code
declare function f1<T>(x: T): T;
var a1 = f1('Hello'); // string

declare function f2<T extends string>(x: T): T;
var a2 = f2('Hello'); // "Hello" <-- why should the extend change the way type is inferred? How do I use the condition but make it inferred as string?

declare function f3<T extends string | number>(x: T): T;
var a3 = f3('Hello'); // "Hello" <-- this is the same as the above, but show a usecase why I would want this behavior

declare function f4<T extends string[]>(x: T): T;
var a4 = f4(['Hello']); // string[] <-- why suddently it is not inferred as const?

declare function f5<T extends boolean[]>(x: T): T;
var a5 = f5([false]); // false[] // <-- why as cosnt for booleans but not for strings in this use case?

🙁 Actual behavior
The type inference is not consistent in the above cases, and it's unclear to me (a) what is the logic behind this, and (b) how do I accomplish a different behavior if I need it. More specifically:

Why adding extends string means that the type is now inferred as const?
Why using an array changes the behavior back to not using as const?
Why extends string[] and extends boolean[] behaves so differently? Why booleans gets the const treatment while strings don't?

🙂 Expected behavior
Since there is a dedicated feature when we want as const for generics, I think that it should be inferred not as const in all of these use cases. This offers the maximum flexibility for consumers as they can pick the behavior they want.
I would also be happy for any work around for this. Let's take this example:
function f<T extends string | number>(value: T): T;

How do I make it so that:

f('somestring') is inferred as string
f<'blue' | 'red'>('red') is inferred as `red'

Also - how do I solve the issue with arrays? Specifically, if I have the function:
function f<T extends any[]>(value: T): T;

Can I make it so that f(['string', true]) will be inferred as [string, boolean] instead of [string, true]?
Additional information about the issue
No response

I'm confused because you know about const but aren't using it where doing so would give you the result you want.
Inference is presented with a set of candidates plus a constraint and tries to figure out what you wanted. This talk goes into some examples - it doesn't talk about literals but you can see from that talk why inference makes some decisions but not others.

Why adding extends string means that the type is now inferred as const?

Because if you wanted the function to return string when passed "hello", you wouldn't have written the constraint in the first place.

Why using an array changes the behavior back to not using as const?

Because the prior argument no longer applies (e.g. you might have a subtype of array that you're specializing to)

Why booleans gets the const treatment while strings don't?

Booleans are a union but strings aren't

I'm confused because you know about const but aren't using it where doing so would give you the result you want.

I'm not sure how I'm supposed to get the result I want using const, as it seems to be the opposite of what I want.
Imagine this example: function f<T extends number | string>(value: T): T. How do I make it infer f('hello') as string, f(1) as number, and f<'red' | 'blue'>('red') as 'red' | 'blue' ?
In that case, there is a reason to the constraint - I do want to narrow number | string but I want to narrow it to either number or string.
The only solution I was able to figure out is using overloads instead:
function f(value: string): string;
function f(value: number): number;
function f(value: string | number): string | number;

However, this doesn't work for more complicated examples. Consider this:
function f<T extends (object | boolean)[]>(value: T): T;

I would expect f([{name: 'hello', true}]) to be inferred as ({name: 'hello'}|boolean)[] but it is inferred as ({name: 'hello'}|true)[].
Is it possible to type in a way that this would work?


refactor: replace logrus with zerolog

Description
I replaced Logrus with the Zerolog package.
Related issue(s)
If this Pull Request is related to an issue, mention it here.

Fixes #587

@b00f can you review this?

@kehiy  Some tests are failed 😔.

@kehiy Some tests are failed pensive.

the logger tests are fine!
but it seems mu changes made some side effects, where? i'm trying to find it.


TAI Enhancements

This is a fairly large modification to the TAI interface. It include a number of modifications:

There are 3 APIs, and objects:
a. Module - The optical module itself
b. Network Interface - The optical interfaces leaving the module
c. Host Interface - The internal interfaces connected to switching silicon
Module presence/absence indications from adapter to adapter host
Independence from SAI. All identifiers are changed from "SAI" to "TAI"
Documentation
a. README.md describes the TAI interfaces/objects/attributes and their usage
b. DOXYGEN comments for all APIs/objects/attributes
Removed SAI-specific functions which don't apply to TAI, like profiles.

Signed-off-by: Scott Emery<EMAIL_ADDRESS>

@scott-emery Thanks a lot for this great contribution!
I added some review comments. Please take a look.
I'd like to focus on structural improvement first.
Please make another PR(s) for adding new attributes.


backend: resource CRUD

rest endpoints for resource CRUD operations

enroll a resource
view a resource
delete a resource
perform actions on resource (e.g. remote unlock door)

I added the ability to register a resource and get a list of resources
Updating a resource seems to have a bug
remote unlock should be moved to a separate issue

I added the ability to register a resource and get a list of resources
Updating a resource seems to have a bug
remote unlock should be moved to a separate issue

4f855cd

4f855cd


add gnmirpc to oc paths

The intent is to be able to allow both requirements to be made clearer by attaching the required RPC level of support (get, set, subscribe), as well as more detailed information about the current level of support.
Just because a path is supported under gnmi.Get doesn't mean it's supported ON_CHANGE in a gnmi.Subscribe!
The messages can be used to either list requirements or, if describing a particular target, it can be used to describe the level of support a target provides.
For read paths, we allow specification of desired (or implemented) support level: streaming, subscribe once, or polling.
Additionally, for paths requiring streaming support, we add specifications for the subscription type: target defined, on change, or sampled.
Finally, we add a sample_interval_nanoseconds, which can be used as a 'minumum desired cadence', if specified as a requirement or the 'minimum recommended cadence' if describing a target's capabilities.
For a target, a cadence of 0 (when the SubscriptionMode includes ON_CHANGE as supported) describes ON_CHANGE as the recommended frequency of subscription.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>5
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 55.268%

Totals

Change from base Build<PHONE_NUMBER>4:
0.0%

Covered Lines:
1983

Relevant Lines:
3588

💛  - Coveralls


Provide a way to request the current moveTo path

Since the moveTo method already returns an observable, maybe a method like getMoveToPath(charId: string): LayerPosition[] would be a good option.

Done with v2.31.0.
It is now part of the MovementInfo: https://annoraaq.github.io/grid-engine/api/interfaces/MoveToInfo.html#state


[RFC]: Postmerge performance suite

Motivation.
We want to start tracking performance numbers of vLLM on more realistic workloads. Thanks to our sponsors #4925 we are getting a pool of hardware resources ready to run the testing on.
The goal of this test suite is to

Track regression
Track our progress in optimization

Proposed Change.
We will start with running the following benchmarks:

Llama 8B on A100, H100
Llama 70B on 4xA100, 4xH100, 8xA100, 8xH100
Mixtral 8x7B on 8xH100
Mixtral 8x22B on 8xH100

We will run with the following parameters:

chunked prefill enabled
fp8

We will run with the following tests:

Benchmark latency
Benchmark throughput with 1000 prompts (ShareGPT)
Benchmark serving with 1000 prompts (ShareGPT)

We will also compare with TGI and TRT-LLM.
Feedback Period.
Step 1: Ensure hardware availabilities
Step 2: Setup pipeline for Llama 8B on H100 as a proof of concept
Step 3: Monitor the result, build dashboard
Step 4: Scale to other tests as resources come online.
CC List.
No response
Any Other Things.
Suggestion welcomed.

Other notes:

I'm not sure how to best test spec decode (in what setting and which workload) and prefix caching (same questions).

For prefix caching, I think few-shot evals where each eval shares the same prompt but the Q&A pair are distinct would be appropriate. Evaluating requests/second or generated tokens/second will give users an idea of how quickly an agent with a static prompt to guide it will run.
I haven't dug into all of the eval frameworks, but I understand many of them are few-shot. MMLU for example, I think uses the same 5 examples.
MMLUs/second would be an amusing (if hard to use) measurement.

Benchmark serving with 1000 prompts (ShareGPT)

It would be great to have one more dataset skewed towards larger prompts / prefilling, e.g. RAG.

We will run with the following parameters:

chunked prefill enabled
fp8

Add a 4-bit quantization method (Marlin, GPTQ)?

@youkaichao mentioned we should also test the multiprocessing backend variant.

close as we have https://perf.vllm.ai now.


How can I password protect the db?

Is it possible password protect a LiteDB  as you would do on access?

Hi @codeyu. At this point, there is not way to do that. It´s not dificult to implement using symetric Rijndael algorithm. I will add this as a feature in next release

Hi,
When can we expect this functionality? This very important function.

Hi @ambyte, It´s on way. In this new v2 password part of connection string. It´s only missing a crypto/decrypto symetric without change byte array size.

It`s cool!!!
Can I protect an existing DB or to create a new one?

Only create a new one. But, will be possible copy data from a plain db to protected one.

Thank you! I thought as much.

Done. Just add "password=your_password" in connection string. LiteDB uses SHA1 for store/check password and Rijndael for encryption.


Couldn't get lyrics

Couldn't get lyrics for Kar Gayi Chull by Badshah.
That's a fishy request, that artist and song doesn't seem to exist on Spotify.

Weird, I get the message that Lyrics Unavailable. Could you walk me through what you did?

I was playing Indian song so it displayed
"something fishy couldn't get lyrics....."
On Sat, Feb 9, 2019, 4:58 AM Aadi Bajpai<EMAIL_ADDRESS>wrote:

Weird, I get the message that Lyrics Unavailable. Could you walk me
through what you did?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aadibajpai/SwagLyrics-For-Spotify/issues/732#issuecomment-461982115,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AkwAOkt72M38qUdttfwKUdSU7o6xEU2rks5vLggkgaJpZM4axbZk
.

Can you play that same song again and show a screenshot?

Wait bro
I send u screen short either here or on slack
On Sat, Feb 9, 2019, 12:47 PM Aadi Bajpai<EMAIL_ADDRESS>wrote:

Can you play that same song again and show a screenshot?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aadibajpai/SwagLyrics-For-Spotify/issues/732#issuecomment-462021248,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AkwAOpyzAPPh6VMUa0plHpibaESDeHOrks5vLnYWgaJpZM4axbZk
.

Looks like it works correctly

check this out fishy request......
On Sat, 9 Feb 2019 at 02:45, Aadi Bajpai<EMAIL_ADDRESS>wrote:

Closed #732
https://github.com/aadibajpai/SwagLyrics-For-Spotify/issues/732.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aadibajpai/SwagLyrics-For-Spotify/issues/732#event-2128625296,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AkwAOiq5yWjBH1R8SqanUAw0HZM5mw5rks5vLny1gaJpZM4axbZk
.


Add isPageHeading to text input

Add isPageHeading to input group
Add isPageHeading to textbox group
Reason for this is some partials don't have
isPageHeading meaning elements aren't rendering
correctly

lgtm


Add eslint as a devDep?

Not 100% following what's happening with the flattened directory structure, but it seems like there is eslint, eslint-config-hapi, and eslint-plugin-hapi in the npm-shrinkwrap.json file, but they aren't in the top level devDeps of this repo. Not sure if this could cause some potential breakage if somebody happens to re-shrinkwrap this repo with npm@2.
    "eslint": {
      "version": "1.5.1",
      "dependencies": {
        "minimatch": {
          "version": "2.0.10"
        },
        "strip-json-comments": {
          "version": "1.0.4"
        }
      }
    },
    "eslint-config-hapi": {
      "version": "3.0.1"
    },
    "eslint-plugin-hapi": {
      "version": "1.2.0"
    },

Also odd is that it doesn't seem like you can run eslint . directly from this repo via the CLI (using a globally installed eslint@1.7.x).
$ npm version
{ requiresafe: '2.1.1',
  npm: '2.11.3',
  http_parser: '2.3',
  modules: '14',
  node: '0.12.7',
  openssl: '1.0.1p',
  uv: '1.6.1',
  v8: '<IP_ADDRESS>',
  zlib: '1.2.8' }

$ eslint .
/Users/pdehaan/.npm-packages/lib/node_modules/eslint/lib/config.js:213
                    throw e;
                          ^
Error: Cannot find module 'eslint-config-hapi'
Referenced from: /Users/pdehaan/dev/tmp/del/requiresafe-cli/.eslintrc
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at readConfigFromFile (/Users/pdehaan/.npm-packages/lib/node_modules/eslint/lib/config.js:155:44)
    at loadConfig (/Users/pdehaan/.npm-packages/lib/node_modules/eslint/lib/config.js:176:22)
    at /Users/pdehaan/.npm-packages/lib/node_modules/eslint/lib/config.js:207:46
    at Array.reduceRight (native)
    at loadConfig (/Users/pdehaan/.npm-packages/lib/node_modules/eslint/lib/config.js:191:36)
    at getLocalConfig (/Users/pdehaan/.npm-packages/lib/node_modules/eslint/lib/config.js:321:23)

$ ./node_modules/.bin/eslint .

$ echo $?
0

$ eslint --version # v1.7.1

Workaround:
Call eslint via the local eslint version instead of a global version:
./node_modules/.bin/eslint .

OR
Create a shortcut script in the package.json file:
"lint": "eslint ."

Then you can run $ npm run lint and it will properly resolve the locally installed ESLint and plugins/configs.

This seems to have removed ESLint from the npm-shrinkwrap.json file:
$ npm ls --depth 0
<EMAIL_ADDRESS>/Users/pdehaan/dev/tmp/del/requiresafe-cli
├── ...
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├── ...

I had to manually install it using:
$ npm i eslint eslint-config-hapi -D

(If I was using npm@3, I think I'd also need to explicitly install the eslint-plugin-hapi peer module.)

i assume you mean when cloning the repo and running it from there? you're right, i generated the shrinkwrap based on npm install --production so any dev dependencies are excluded from the shrinkwrap.
makes local development a bit more of a hassle, but end user experience will be much improved.
we'd have to go through and make sure every subdependency of every dev dependency was added as a top level dev dependency to let them work together more smoothly, so this felt like a good stopgap measure.


Issue 21 serial interval methods

Created two new methods for the BranchProModel class:

[x] a get_serial_intevals

[x] an update_serial_intervals

Also added tests. Solves issue #21

I'll review this once #8 is merged.


renderFormatted() adds empty lines after img and svg

When rendering HTML with renderFormatted(), there are many empty lines added after img and svg elements.
In my test, there were 12 empty lines after an img element, and 3 lines after each svg element.
The lines are all indented to the previous element.

yeah, the renderFormatted() bit is a lot more complicated to get right than one would think.
To be honest, I have noticed this too and I am thinking about removing the API entirely to keep the maintenance burden low. if someone spots a quick fix: happy to merge a PR ; )

the latest release contains a bigger refactor of the formatter and so far things seem to be ok.
please reopen if there are still issues with it.


Update some contributor page

The link anchor was broken, replaced with the correct one.
For Prow command reference, add k/website parameter.
The permission of /assign and /close command is outdated.

For information, [this PR] proposed a fix for the broken link as well (and only for this one).

Thanks @Sea-n .
/kind cleanup

/approve


fix: Remove 'readme' FieldResolver since we already get it from ES

:tada: This PR is included in version 2.6.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


error: undefined function (there is no such import)   nofile:1

I keep getting this error at times when I import components into a storybook. For example,

  defmodule Storybook.UI.Tab do
    use PhoenixStorybook.Story, :component
    alias Elixir.TremorxWeb.CoreComponents

    def function, do: &Tremorx.Components.Tab.tabs/1

    def imports,
        do: [
        {CoreComponents, icon: 1},
        {Tremorx.Components.Layout, [flex: 1, card: 1, col: 1, grid: 1]},
        {Tremorx.Components.Text, [metric: 1, text: 1]}
    ]
  end

The page fails to compile with the error below. I already have the functional components inside of my module which lives in another app in the same umbrella project as well as my storybook app.
error: undefined function card/1 (there is no such import) nofile:1

Importing the components one by one works
def imports,
    do: [
      {CoreComponents, icon: 1},
      {Tremorx.Components.Layout, [flex: 1, grid: 1]},
      {Tremorx.Components.Layout, card: 1},
      {Tremorx.Components.Layout, col: 1},
      {Tremorx.Components.Bar, progress_bar: 1},
      {Tremorx.Components.Text, [metric: 1, text: 1]}
    ]

deps

{:phoenix, "~> 1.7.10"},
{:phoenix_html, "~> 3.3"},
{:phoenix_live_reload, "~> 1.2", only: :dev},
{:phoenix_live_view, "~> 0.20"},
{:phoenix_storybook, "~> 0.5.0"},

https://hexdocs.pm/phoenix_storybook/components.html#aliases-imports
Ideal way to write as per above doc is as below
 [{Tremorx.Components.Layout, flex: 1}, {Tremorx.Components.Layout, card: 1}, {Tremorx.Components.Layout, col: 1}]

Hey @briankariuki, thanks for the report, it seems to be a bug.
I'll try to get it fixed this week


Setup dependabot

Closes #2

Codecov Report

Merging #3 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #3   +/-   ##
=======================================
  Coverage   88.67%   88.67%
=======================================
  Files           1        1
  Lines          53       53
  Branches       12       12
=======================================
  Hits           47       47
  Misses          6        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c19b35a...c5e2753. Read the comment docs.


API changes report for Ehcache

Hi,
I'd like to share my results on API changes analysis for the library: https://abi-laboratory.pro/java/tracker/timeline/ehcache3/
The report is generated by the https://github.com/lvc/japi-compliance-checker tool for jars found at http://central.maven.org/maven2/org/ehcache/ehcache/ according to https://wiki.eclipse.org/Evolving_Java-based_APIs_2.
Report for Ehcache 2.x: https://abi-laboratory.pro/java/tracker/timeline/ehcache/
Thank you.

Hello,
Thanks for your interest in Ehcache 3.
We request that users start the conversation on the google group ehcache-users before opening issues.

That's great, thanks for pointing us at it.
However, I would be interested to know if there is a possibility to limit the scanning to specific packages?
I browsed a number of the versions that have the lowest scores and all modifications are in what we define as internal packages.
While we will only be able to enforce that in Java 9, what would be really interesting is to validate that the API only grows and that the SPI only changes between minors.
Is that an option?

Hi,
It's possible to remove particular packages from the analysis. Should it check packages listed at http://www.ehcache.org/apidocs/3.4.0/index.html only?
org.ehcache
org.ehcache.config
org.ehcache.config.builders
org.ehcache.config.units
org.ehcache.event
org.ehcache.expiry
org.ehcache.spi.copy
org.ehcache.spi.loaderwriter
org.ehcache.spi.persistence
org.ehcache.spi.serialization
org.ehcache.spi.service

Thank you.

This is indeed the external API. Everything else is internal stuff that
shouldn't be used directly.
Le 11 oct. 2017 2:25 AM, "Andrey P."<EMAIL_ADDRESS>a écrit :

Hi,
It's possible to remove particular packages from the analysis. Should it
check packages listed at http://www.ehcache.org/apidocs/3.4.0/index.html
only?
org.ehcache
org.ehcache.config
org.ehcache.config.builders
org.ehcache.config.units
org.ehcache.event
org.ehcache.expiry
org.ehcache.spi.copy
org.ehcache.spi.loaderwriter
org.ehcache.spi.persistence
org.ehcache.spi.serialization
org.ehcache.spi.service
Thank you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ehcache/ehcache3/issues/2146#issuecomment-335695706,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKRMzg__Tfamz99dfZsGONQTRW_W1rJks5srF9TgaJpZM4PtIMU
.

Fixed report according to docs: https://abi-laboratory.pro/java/tracker/timeline/ehcache3/
Thank you.

Thanks for the report. Closing this since all changes here are expected


Remove Add Vendor btn from Consent > Vendors page

Code Changes

[ ] Remove Add Vendor btn from Consent > Vendors page

Steps to Confirm

[ ] navigate to http://localhost:3000/consent/configure
[ ] confirm the Add Vendor button is gone

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[ ] documentation issue created in fidesdocs

[ ] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[ ] Update CHANGELOG.md
[ ] For API changes, the Postman collection has been updated

UAT https://www.loom.com/share/24da463c65c04b4ab01b67fe03b2e522


Scripts to create DC/OS compatible cloud images

Original work and reviews in #205 and #311.
Now uses Overlay + XFS with the EL7 base image for Docker storage, so that problematic containers like rastasheep/ubuntu-sshd work properly.
Updates AWS Advanced Templates default EL7 base image to use new AMI's.
Automated builds are now available at https://teamcity.mesosphere.io/project.html?projectId=DcosIo_Dcos_BuildCloudBaseImages&tab=projectOverview

This PR is essentially to identical to #205 but changes secondary filesystem type from ext4 to xfs for better compatibility with overlay. All comments from #205 have been addressed (see CI details in description).

:ship:

Overall looking good, things can be done as followup. 🚢


machinepool: remove finalizer for fake CD

The "normal" machinepool controller flow does all the things, and then
at the end checks whether the machinepool is deleted. If so, it removes
the finalizer and bails.
For fake clusters, we are bailing before "all the things". But we were
not deleting the finalizer in that case.


Feat: Implement the functions in the deposit_utils library.

Description
Implement the create_deposit function in the DepositUtils library.
File: src/deposit/deposit_utils.cairo.
Function signature:
    /// Creates a deposit.
    fn create_deposit(
        ref self: TContractState,
        data_store: IDataStoreDispatcher,
        event_emitter: IEventEmitterDispatcher,
        deposit_vault: IDepositVaultDispatcher,
        account: ContractAddress,
        params: CreateDepositParams
    ) -> felt252;

Implement the cancel_deposit function in the DepositUtils library.
File: src/deposit/deposit_utils.cairo.
Function signature:
    /// Cancels a deposit, funds are sent back to the user
    fn cancel_deposit(
        ref self: TContractState,
        data_store: IDataStoreDispatcher,
        event_emitter: IEventEmitterDispatcher,
        deposit_vault: IDepositVaultDispatcher,
        key: felt252,
        keeper: ContractAddress,
        startingGas: felt252,
        reason: felt252,
        reason_bytes: Array<felt252>
);

Acceptance criteria

Function is implemented and well commented.
Associated unit tests (nominal and failure) are implemented and passing.

Resources

createDeposit function in GMX repo
cancelDeposit function in GMX repo

Hello, can i work on this one ?

of course !


Update index.js

I think this is a just a typo

Coverage remained the same at 17.857% when pulling 30199b9cccdbc6b7bb47d90db04bc10de1248ec2 on EfrainTlapale:patch-1 into e41de3424e770cd67c89898483861679f095539d on GenFirst:master.

Thank you, I really appreciate it :+1:


update docs to use use numpydoc + linkcode

This PR updates documentation to use the numpydoc extension and also adds the linkcode extension.  Include numpydoc seems to mainly check for consistency with NumPy documentation format, so it found a few inconsistencies that were fixed.  The linkcode extension adds a link to the source code for each function to GitHub.

Coverage: 94.846%. Remained the same when pulling da6b08880ad19a61ae6d3ef62b9de9cf4fd3cba4 on murrayrm:numpydoc-28Dec2022 into 49b007740fa0ffbe6b1dc83bdb2ef49cfcf794f8 on python-control:main.


[BUG] connections folder not generated after running pf build --format docker

Describe the bug
Running pf flow build --source promptflow --output pf_docker --format docker does not generate a connections folder in the output folder.
This causes docker build to throw an error:
ERROR: failed to solve: lstat /var/lib/docker/tmp/buildkit-mount1136518113/connections: no such file or directory
How To Reproduce the bug
Steps to reproduce the behavior, how frequent can you experience the bug:

Install promptflow and tools pip install promptflow promptflow-tools keyrings.alt
Verify promptflow is installed pf -v -> 1.0.0
Create connection: pf connection create --file ./promptflow/azure_openai.yaml --set api_key=<my_api_key> api_base=<my_api_base> --name open_ai_connection
Generate output folder: pf flow build --source promptflow --output pf_docker --format docker
Build Image: docker build pf_docker -t myapp_pf:dev

Expected behavior
After running pf flow build --source promptflow --output pf_docker --format docker I am expected to see a connections folder, however this is not being generated:

I noticed that the Dockerfile is expecting a connections folder because COPY ./connections/* /connections/
Running Information(please complete the following information):

Promptflow Package Version using pf -v: 1.0.0
Operating System: Ubuntu 20.04.6
Python Version using python --version: 3.10.8

Additional context
N/A

Hi, @joaquinrz .The fix is merged https://github.com/microsoft/promptflow/pull/1242 and will be released in the next stable version.
The connection folder is generated in the flow/dist/connection folder for now. You can update the Dockerfile COPY ./flow/dist/connections/* /connections/ as a workaround for now.


[Remangler] Improve performance by caching hashes.

The deepHash() function gets called repeatedly as we descend the node tree, which results in O(n^2) behaviour because we're traversing entire node subtree from each node we try substitution in, in order to calculate the hash.
Fix by adding a hash table for hashes, so that we can look up hashes we've already computed.
This appears to yield a 26.8% saving in local tests.
rdar://125739630

@swift-ci Please test

Closed as this should have come from my forked repo.


Feature Request: Bulk Client Requests to the Simulation Server

The issue mentioned in #1258 and the asynchronous nature of the server-client interaction in CARLA, opens up some possibilities of some undesirable events.
Lets say you want to know the transform, velocity, acceleration and control of the player on the client side. You implement the following routines in your client:
transform = self._player.get_transform()
accel = self._player.get_acceleration()
vel = self._player.get_velocity()
ctrl = self._player.get_control()

There exists a possibility, howsoever infinitesimal that the world is updated between execution of these routines (say after acceleration is parsed but before velocity is parse). This causes a lot of problems even in data collection (which I think is the key point in using a simulator).
To alleviate these problems, I have 2 possible feature requests:

Similar to the issue #784, an interface of sorts can provided where we pass a dictionary / list of function_handles to be executed by the simulator. This could also greatly reduce the number of calls the client needs to make to the server. The prototype could be something like:

input_dict = {"transform": self._player.get_transform, "accel": self._player.get_acceleration, "vel": self._player.get_velocity, ...}
output_dict = get_data(input_dict)
transform = output_dict["transfrom"]
accel = output_dict["accel"]
...

Similar to the issue #1244, a method to pause / resume the simulation cycle. The whole communication can still run asynchronously for optimal performance, however, just a call can be implemented to the server to pause / resume the simulation cycle. The prototype could be something like:

world.pause_simulation()
transform = self._player.get_transform()
accel = self._player.get_acceleration()
...
world.resume_simulation()

I understand that these features may be quite complicated to implement.  But, I think that having such routines would greatly increase the usability of the simulator.

Hi @YashBansod,
Thanks for the observations! Indeed what you're describing is something not so unlikely to happen.
I'll add some more comments on how this works internally since our docs are not very clear.
The simulator has a "world observer" sensor that read the state (transforms, velocities, etc.) of each actor at the beginning of each frame and broadcasts the data to every client connected. Each client subscribes to the world observer sensor by adding a callback (just as any other sensor). Each time this callback is executed we call it "world tick". (The functions world.wait_for_tick() and world.on_tick(...) refer to this world tick).
Now, when we want to retrieve an actor's transform, it's orders of magnitude faster if we fetch the transform received in the last world tick instead of calling the simulator and awaiting response. So this is precisely what we do, internally, every actor has a reference to some internal "episode" class, the episode class contains a pointer to an "episode state". The episode state consists of the bulk of actor dynamic states (transforms, velocities, etc.) at a given frame. This episode state is created from the data sent by the world observer in the world tick. And, to avoid locking, we simply update the pointer to point to the newly created episode state each world tick.

This is very fast, however it only guarantees that when you fetch the episode state, you're going to get an episode state, but not which episode state. Thus, any time in the middle of your calculations the episode state might update in the background and your transform changes. This is fine for a lot of applications, but sometimes you may want to know the exact transform of your actor at a given frame. This is an issue we are aware of, but we haven't implemented a fix yet.
With the synchronous mode #1244 that is coming in 0.9.4 you can achieve this in practice. Pretty much what you propose in 2nd solution, except that the simulation pauses automatically every frame, and waits for you to call world.tick() (tells the sim to execute next tick)
timestamp = world.wait_for_tick()
frame = timestamp.frame_count # In case you want to tag this frame.
transform = self._player.get_transform()
accel = self._player.get_acceleration()
...
world.tick()

However, synchronous mode has a big overhead cause it blocks the simulation every tick. Some use cases may not require that much.
What I had in mind is to return somehow a snapshot of a given frame, linked to a given episode state, thus you make sure everything in this snapshot belongs to a single frame and it's not going to change in the background. I think this can be super useful for a lot of applications and at the same time quite easy to implement with the current design
snapshot = world.get_snapshot()
frame = snapshot.timestamp.frame_count
for actor in snapshot:
    t = actor.get_transform()

and even we can pass it to the on_tick callback, so you have a way of calling your methods each time a new snapshot is generated, without any synchronization issues. This should fix the problem you mentions in #1258.

Hello @nsubiron Thanks for such a detailed reply. I really appreciate the effort that CARLA's team puts in supporting its users.
I was unaware of the fact that the methods like Actor.get_transform() were being read from the previous snapshot. I always assumed that it required requesting the information to the server and waiting for the reply. Its really interesting to know that this is infact not the case.
I had the wrong assumption based on the reasoning that the actor state like transform, velocity etc. were not visible in the python debugger. As you mentioned that every actor has a reference to a hidden internal "episode" class, which in turn contains the pointer to the episode state which contains all the data sent by the world observer. I guess that is the reason that the actor state variables aren't visible in the debugger.
Correct me if I am wrong, but this essentially means that accessing all information using the getter routines of various actors is equivalent to getting a deepcopy of the contents of the "episode" class.
Indeed the possibility of accessing a snapshot of a given frame would solve the problem in this issue.
It is great to know that you guys already have considered about the solution.
However, I think it would still not solve the problems I had mentioned in #1258.
Essentially the problem is that occasionally the agent.get_transform() (even when implemented in a world.on_tick() routine gives same transform for a few cycles and then suddenly jumps to a new value. I assume that this is caused by synchronisation mismatch between the world.on_tick() and the updating of the agent states via the world observer.
Based on the idea you had mentioned, the problem would actually be solved if you could access the snapshot of any desired frame. Something like this:
frame = snapshot.timestamp.frame_count
snapshot = world.get_snapshot(frame)
for actor in snapshot:
    t = actor.get_transform()

Maybe a buffer limit can be set to cache the keyed snapshots few time steps in history, if maintaining a list of all the snapshots in memory is infeasible.

The problems you mention might be related to other problems indeed. I'm fixing several synchronization issues related to the episode state that could affect this. It could also be the simulator producing few frames with very low delta time, followed by a frame with a big delta time. Have you tried running at fixed time-step? You can also check the timestamp to see if there is something off.
It's strange, the on_tick callbacks are executed in the same thread as the new episode state is received and parsed, it seems to me very odd not retrieving the right content in that callback. I'll take a look after fixing this issues to see if I can reproduce it.

Correct me if I am wrong, but this essentially means that accessing all information using the getter routines of various actors is equivalent to getting a deepcopy of the contents of the "episode" class.

Yes, exactly. We could even add a method for dumping the whole content to a Python dictionary, that seems handy.

Yes, the problem I mentioned is also happening in fixed time-step mode. I am using the following command to initiate the simulation server:
./CarlaUE4.sh -benchmark -fps=10.0
As I understand this only tells the number of simulation steps to take in every simulation time second. It doesn't enforce the simulator to run 10 simulation cycles every real time second. (A feature like this would be helpful too :) )
I checked the timestamps and I do not see a jump in those values. They are evenly separated by about 0.1 seconds. I think this might be due to the fact that the timestamps are in simulation time rather than the real time.
So, I checked the timestamps generated by time.time() and saved them as world_timestamps. I notice that every time this synchronisation issue happens, there is first a small increase in the simulation time step and then a huge reduction in the consecutive frame. Example:
The values here are not exact

index
delta dist
delta sim time
delta real time

72
0.60...
0.1...
0.048...

73
0.61...
0.1003...
0.050...

74
1.18...
0.1...
0.052...

75
0.0 (exact)
0.1...
0.0009...

76
0.56...
0.1003...
0.027...

77
0.56...
0.1...
0.039...

Also, to make your job easier in checking this issue, I will be posting the python codes of the pseudo sensor code and the parser code that I am using in the following comments.

The Sensor Code
# ******************************************    Class Declaration Start     ****************************************** #
class OdometryPseudoSensor(object):

    def __init__(self, parent_actor, hud, tag):

        self._player = parent_actor
        self._hud = hud
        self._world = self._player.get_world()
        self._map = self._world.get_map()
        self._dataset_dir = os.getcwd() + "/Recordings/" + tag + "/Annotations/"
        self._dataset_path = self._dataset_dir + "{:0>8d}.json"
        self._recording = False

        if self._recording:
            if not os.path.exists(self._dataset_dir):
                os.makedirs(self._dataset_dir)

        self._id = -1
        self._timestamp = 0.0
        self._world_time = 0.0
        self._loc = {"x": 0.0, "y": 0.0, "z": 0.0}
        self._rot = {"yaw": 0.0, "pitch": 0.0, "roll": 0.0}
        self._vel = {"x": 0.0, "y": 0.0, "z": 0.0}
        self._accel = {"x": 0.0, "y": 0.0, "z": 0.0}
        self._ctrl = {"brake": 0.0, "hand_brake": False, "reverse": False, "steer": 0.0, "throttle": 0.0}
        self._wp = {"loc": {"x": 0.0, "y": 0.0, "z": 0.0}, "rot": {"yaw": 0.0, "pitch": 0.0, "roll": 0.0},
                    "lane_width": 0.0, "road_id": -np.inf, "lane_id": -np.inf}

        self._datasample = {"id": self._id, "timestamp": self._timestamp, "location": self._loc, "rotation": self._rot,
                            "velocity": self._vel, "acceleration": self._accel, "control": self._ctrl, "wp": self._wp,
                            "world_time": self._world_time}

    # ******************************        Class Method Declaration        ****************************************** #
    def toggle_recording(self):
        self._recording = not self._recording
        self._hud.notification('Recording of VehicleData %s' % ('On' if self._recording else 'Off'))

        if self._recording:
            if not os.path.exists(self._dataset_dir):
                os.makedirs(self._dataset_dir)

    # ******************************        Class Method Declaration        ****************************************** #
    def add_datasample(self, timestamp):

        if not self._player.is_alive or not self._recording:
            return

        self._world_time = time.time()

        transform = self._player.get_transform()
        accel = self._player.get_acceleration()
        vel = self._player.get_velocity()
        ctrl = self._player.get_control()
        loc = transform.location
        rot = transform.rotation
        wp = self._map.get_waypoint(loc)

        self._id = timestamp.frame_count
        self._timestamp = timestamp.elapsed_seconds

        rot.yaw = degree_180_to_m180(rot.yaw)
        rot.pitch = degree_180_to_m180(rot.pitch)
        rot.roll = degree_180_to_m180(rot.roll)

        wp_loc = wp.transform.location
        wp_rot = wp.transform.rotation
        wp_rot.yaw = degree_180_to_m180(wp_rot.yaw)
        wp_rot.pitch = degree_180_to_m180(wp_rot.pitch)
        wp_rot.roll = degree_180_to_m180(wp_rot.roll)

        self._loc = {"x": loc.x, "y": loc.y, "z": loc.z}
        self._rot = {"yaw": rot.yaw, "pitch": rot.pitch, "roll": rot.roll}
        self._vel = {"x": vel.x, "y": vel.y, "z": vel.z}
        self._accel = {"x": accel.x, "y": accel.y, "z": accel.z}
        self._ctrl = {"brake": ctrl.brake, "hand_brake": ctrl.hand_brake, "reverse": ctrl.reverse, "steer": ctrl.steer,
                      "throttle": ctrl.throttle}
        self._wp = {"loc": {"x": wp_loc.x, "y": wp_loc.y, "z": wp_loc.z},
                    "rot": {"yaw": wp_rot.yaw, "pitch": wp_rot.pitch, "roll": wp_rot.roll},
                    "lane_width": wp.lane_width, "road_id": wp.road_id, "lane_id": wp.lane_id}

        self._datasample = {"id": self._id, "timestamp": self._timestamp, "location": self._loc, "rotation": self._rot,
                            "velocity": self._vel, "acceleration": self._accel, "control": self._ctrl, "wp": self._wp,
                            "world_time": self._world_time}

        self.write_dataset(self._dataset_path.format(self._id), self._datasample)

    # ******************************        Class Method Declaration        ****************************************** #
    @staticmethod
    def write_dataset(dataset_path, data_sample):
        with open(dataset_path, "w") as outfile:
            json.dump(data_sample, outfile)

# ******************************************    Class Declaration End       ****************************************** #

When running at fixed time-step what we do is decoupling simulation-time from real-time, so the simulation can run as fast as possible. In this case real-time is irrelevant (except for measuring performance), what matters in this mode is that in every step we simulate 0.1 seconds. And that seems to be consistent. These frames may arrive to the client at any time (and even potentially in the wrong order), but it doesn't matter, what matters is that each of them is consistent with an increment of 0.1 seconds. If you tag all your measurements with their simulation time-stamp and then "replay them" at 10 FPS you would see a consistent simulation. I believe this is working fine no?

@nsubiron As is evident from the table I had put in the comment above:

index
delta dist
delta sim time
delta real time

72
0.60...
0.1...
0.048...

73
0.61...
0.1003...
0.050...

74
1.18...
0.1...
0.052...

75
0.0 (exact)
0.1...
0.0009...

76
0.56...
0.1003...
0.027...

77
0.56...
0.1...
0.039...

It is clear, that there is a synchronisation issue between

@nsubiron As is evident from the table I had posted in my previous comment, this is not the case.
I am tagging all my measurements with the simulation time stamp and the vehicle states recorded are not consistent. For convenience I will post the table here again.

index
delta dist
delta sim time
delta real time

72
0.60...
0.1...
0.048...

73
0.61...
0.1003...
0.050...

74
1.18...
0.1...
0.052...

75
0.0 (exact)
0.1...
0.0009...

76
0.56...
0.1003...
0.027...

77
0.56...
0.1...
0.039...

We see that at index 75, the delta sim time was 0.1 but the change in vehicle location (calculated as euclidean distance between consecutive locations) was exactly zero.
We can also observe that at index 74 the change in vehicle location was almost twice the normal values.
My guess on the reason for this happening is this:

The simulation timestamp comes from the timestamp variable that is available to the world.on_tick() method through the simulator. Therefore, the simulation step is in fact 0.1 simulation second. But this is not in sync with the updating of the episode class from the world_observer.
As you mentioned, in your previous comments, the player.get_transform() gets the values from the internal episode class that is updated asynchronously using the world_observer sensor data. Essentially, what the table implies is that there is a mismatch in the synchronization of execution of the callback_function executed through world.on_tick()  and updating of the contents of the episode class from the world_observer

If I replay the simulation, visually I won't see any jumps since these are happening at the rate of 1-2 times every 10 frames. But, if you analyse the data analytically you see this discrepancy.

ahaa ok, sorry, I missed the "delta dist" column. Yes, you're right, since there is no synchronization between the callback and the get transform function there can be this sort of discrepancies.
Since callbacks are completely async, it could have happened that frame 75 is executed before (or during) the callback of frame 74, thus when you do a get_transform() on callback 74, you're getting the state of frame 75 already. I believe passing a "snapshot" to the callbacks will solve this issue, cause the snapshot will always have the right transform, even if 75 is executed before 74.
Thanks for sharing all this info, it's helping me a lot understanding all the implications of calling these callbacks asynchronously.

Yeah, as you mentioned in https://github.com/carla-simulator/carla/issues/1274#issuecomment-466368575,

These frames may arrive to the client at any time (and even potentially in the wrong order), but it doesn't matter, what matters is that each of them is consistent with an increment of 0.1 seconds.

However, since the order of the frames is not ensured, AND the timestamp passed to the world.on_tick() is not in sync with the world_observer, If I try to replay the recording, I get something like this:

This is a GIF created using the sequential time-stamped readings from the PseudoOdometrySensor created using world.on_tick() method. Observe that the readings show that the vehicle is jumping back and forth. However, it is actually moving at a velocity of around 20 Kmph.
As you mentioned in https://github.com/carla-simulator/carla/issues/1274#issuecomment-466385988
Passing the snapshot from the world_observer to the world.on_tick() will solve this problem. I hope such a thing would be available in CARLA in the near future.

Yes, we have a similar issue with the no_rendering_mode.py script, you can observe this jumping in the NPCs because sometimes you get the transform of a different frame (ahead or behind).
I've been wanting to add this snapshot functionality for a while, but there has always been something with higher priority. I'll try to push this to come after this coming release, but that would be 0.9.5.


Echo plugin

This PR adds the echo plugin

The main objective is to have an example to agree on
directory structure.  Feel free to vote against having echo-plugin at all, that's not really the point.

The relevant choices this PR proposes are
- <resource-plugin>/{src,helm}/ to keep the yaml associated with the
  source.
- naming with prefix 'sd-' to reflect the plugin in
- placing execpodNamespace in values.yaml

tested with helm install.    Not sure yet how helm uninstall will
work if there are problems blocking deletion with finalizers...

/{src,helm}/ to keep the yaml associated with the source.

For now at least, we're using an off-the-shelf tool to auto-generate the Helm repo index. That tool is pretty opinionated about the dir structure. It wants all charts as a single layer of subdirs under one parent dir. In addition, any changes in that tree will cause the tool to try to cut a new release, and it will complain if we didn't bump the version in Chart.yaml.
To stay out of that tool's way, I've kept the non-Helm files in a separate top-level dir src. We can reevaluate if/when we think it's worth customizing or moving away from the all-in-one chart repo tool.

naming with prefix 'sd-' to reflect the plugin in

I'm guessing you meant to reflect that it's a plugin we maintain? I was torn about this too. I don't feel strongly either way, but it felt cleaner to me to not have the prefix on everything in this repo. @foxish WDYT?

placing execpod Namespace in values.yaml

I thought we agreed to move plugins into the existing signadot namespace alongside the operator?

Not sure yet how helm uninstall will work if there are problems blocking deletion with finalizers...

I haven't tested it yet either, but my guess is that helm uninstall will think everything is fine, yet the SignadotResourcePlugin object will actually still be around because of our finalizer.


Watch mode do not update output with JavaScript export schema type

I have the same issue as @grezxune mentioned in here: https://github.com/dotansimha/graphql-code-generator/issues/1609#issuecomment-554584784
Watching files works, but output is untouched on change
with this config
schema: src/graph/schema.ts
require:
    - ts-node/register
watch:
    - src/graph/nodes/**/*.ts
overwrite: true
generates:
    src/graph/gen-types.ts:
        plugins:
            - typescript

btw: I didn't find any docs about the wildcard-path passed to watch argument.

I have the same issue with typescript, it does not refresh the generated type unless I restart codegen or edit my config file (codegen.yml).
overwrite: true
schema: "http://localhost:4000"
documents: null
generates:
  src/generated/graphql.ts:
    plugins:
      - "typescript"
      - "typescript-resolvers"
  ./graphql.schema.json:
    plugins:
      - "introspection"

@melounek @debrice can you please share a reproduction in a repo/codesandbox?

There is repro-repo: https://github.com/melounek/repro-codegen (with readme how to reproduce)
thanks @dotansimha

Thanks @melounek !
I can confirm that this is caused by require.cache. I think watch: feature should handle that manually and clear the items from the cache, because otherwise we'll get the same file even if it has changed.
I'm working on a fix here: https://github.com/dotansimha/graphql-code-generator/pull/4037

Fixed. You can use the alpha 1.13.6-alpha-1ace8c2f.44 for now.
@melounek thank you for the reproduction, it really helped!

Fixed in v1.14.0.


LinAlgError (Array must not contain infs or NaNs) thrown in get_mu_tensor

Below is a simple piece of code to try YellowFin on my dataset.
x = tf.placeholder( tf.float32, [ None, train_x.shape[ 1 ] ] )
y = tf.placeholder( tf.float32, [ None, train_y.shape[ 1 ] ] )
m = tf.layers.dense( x, hidden_dim )
m = tf.layers.batch_normalization( m )
m = tf.nn.elu( m )
m = tf.layers.dense( m, hidden_dim )
m = tf.layers.batch_normalization( m )
m = tf.nn.elu( m )
m = tf.layers.dense( m, hidden_dim )
m = tf.layers.batch_normalization( m )
m = tf.nn.elu( m )
m = tf.layers.dense( m, train_y.shape[ 1 ] )
prediction = tf.nn.softmax( m )
loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits( labels=y, logits=m ) )
optimizer = yellowfin.YFOptimizer().minimize( loss )

s = tf.Session()
s.run( tf.global_variables_initializer() )
for epoch in range( epochs ):
    _, h = s.run( [ optimizer, loss ], feed_dict={ x: train_x, y: train_y } )

Usually, it crashes and throws the following exception.
Caused by op 'update_hyper/cond/PyFuncStateless', defined at:
  File "test2.py", line 47, in <module>
    optimizer = yf.YFOptimizer( learning_rate=1., momentum=0. ).minimize( loss )
  File "/data/python-mp-test/libs/yellowfin.py", line 268, in minimize
    return self.apply_gradients(grads_and_vars)
  File "/data/python-mp-test/libs/yellowfin.py", line 223, in apply_gradients
    update_hyper_op = self.update_hyper_param()
  File "/data/python-mp-test/libs/yellowfin.py", line 191, in update_hyper_param
    lambda: self._mu_var) )
  File "/usr/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 289, in new_func
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1814, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1689, in BuildCondBranch
    original_result = fn()
  File "/data/python-mp-test/libs/yellowfin.py", line 190, in <lambda>
    self._mu = tf.identity(tf.cond(self._do_tune, lambda: self.get_mu_tensor(),
  File "/data/python-mp-test/libs/yellowfin.py", line 173, in get_mu_tensor
    roots = tf.py_func(np.roots, [coef], Tout=tf.complex64, stateful=False)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 201, in py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/ops/gen_script_ops.py", line 56, in _py_func_stateless
    Tout=Tout, name=name)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

UnknownError (see above for traceback): LinAlgError: Array must not contain infs or NaNs
	 [[Node: update_hyper/cond/PyFuncStateless = PyFuncStateless[Tin=[DT_FLOAT], Tout=[DT_COMPLEX64], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](update_hyper/cond/ScatterUpdate)]]

Hi @ywchan2005,
Thanks for trying out the optimizer. This is mostly because of the exploding gradient in the middle of training.

If it happens in the very beginning, you might want to play with the initial value a bit.

If it is in the middle of training, please consider using gradient clipping. There is discussion with solutions in our PyTorch YellowFin repo here. Similar solution can apply to the TF repo.

We are working on a better auto gradient clipping feature. You may also wait for that in a few days. But I suggest you can already start working on 2.

Thanks for the update. It happens mostly in the middle of training.
If I change the model to use relu (instead of elu), the issue doesn't happen anymore.

The same happened to me, first, you can a debug message to see exactly the values of the coefficients (where it gets the nan value) you can replace these lines in the get_mu_tensor(self) method:
  coef = tf.scatter_update(coef, tf.constant(2), -(3 + const_fact))     log_coef = tf.Print(input_=coef, data=[coef],message="coefficients: ")     roots = tf.py_func(np.roots, [log_coef], Tout=tf.complex64, stateful=False)
Second, as @JianGoForIt suggested using clip_thresh parameter works for me.
However, another issue pops up, I get very high learning rate in the early stages in the training. That happens even in the case when my initial learning rate is very low (1e-5). The result is that the model diverges. Any ideas how to fix it?

Hi @IdanAzuri, Thanks for trying out our optimizer.
The estimators might be unstable when there are only a few iterations (as samples for the estimators in our optimizer). There are hacky ways to get around (clipping gradient / get a upper bound for lr in the begining iterations) but I would like to investigate whether there is systematical issues.
Could you please provide me a minimal example to reproduce your error?

@JianGoForIt, actually it's not stable after 400k iterations (until I stopped it) so it is not the problem.
Sorry but I don't have a minimal example because my system is very complex with multiple subnets.
clipping gradient / get an upper bound for learning rate doesn't work here, it still diverges.


Worlds rebranding to DreamSpace

Given that the Worlds product name it's confusing it was decided to call it DreamSpace, changes needed:
Platform

[x] CLI
[x] Content Server (CNAME? + about urls)
[ ] Bundle (worlds-owner)
[ ] Catalyst Monitor
[x] Validations and scene.json configuration
[ ] Docs

dServices

[ ] Events & Places

dApps

[ ] Builder
[ ] Linker dApp to sign the ACLs (check if we can update the signing view that still references the catalyst)

Data

[ ] Metrics/Data

Explorer

[ ] Explorer: /world command, /help command, and friendly URL using play.decentraland.org/world/NAME

The community rejected this change, the name will continue to be Worlds
https://governance.decentraland.org/proposal/?id=788a7410-0a38-11ee-bb17-db98a4ce871d


HTTP Error 503

Im getting "ERROR: Unable to download webpage: HTTP Error 503: Service Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output."
is there any way to add "-4" in executing youtube-dl?

If it still fails, please post the url you are using?


Limit ImGui <=1.90.4 to Polyscope > 2.2.0

Polyscope Officially Support ImGui Version is 1.90.4
It has Compilation Error When Using Latest ImGui=1.91.6
So, In this Commit, Limit ImGui Version to 1.90.4 For Polyscope > 2.2.0

ImGui 1.91.6 is not compatible with polyscope > 2.2.0.
It has to Use Older Version of ImGui


run-task accepts a process to execute

This is one of two PRs for addressing Issue #1641.  This PR updates the run-task command for cli v7 to enable using processes to supply start commands to tasks.  There is no change to the v6 cli behavior.
commit message details
add --process flag. will use the supplied process as a template for
the task. the process command, memory, disk will be used to execute a
one-off task.
remove COMMAND as a required positional argument
add --command flag. shorthand -c. optional method to supply the command to execute for
the task.
if neither --process nor --command flags are provided, then assume there is a
process named 'task' and attempt to execute it.
addresses the run-task updates for Issue #1641

@zrob it looks like you haven't signed the cla. ;)

Yep, got updated in the new CLA tool now. Not sure how to get the bot to comment, but doesn't seem necessary since new CLA thingy shows up in the checks section.  It shows up as EasyCLA and is now passing.


implement movingWindow & round functions, tidy up other "moving" functions

This PR centralizes processing for the movingAverage, movingMedian, movingMin, movingMax & movingSum functions into a new movingWindow function which they all now use internally, removing a lot of duplicated code.  It also adds a new xFilesFactor parameter to all those functions which allows the user to specify a value between 0 and 1 to indicate what fraction of the points within the window must be non-null for the output to be considered valid.
It also updates exponentialMovingAverage to match the implementation of movingWindow as much as possible, and fixes an issue when using it on fractional series where the first result datapoint is not rounded but the subsequent points are all rounded to 3 decimal places.  It now consistently rounds to 6 decimal places, which makes it usable for the majority of series.
It also adds a round function which can be used to round datapoints in a seriesList to a specified precision, which may be useful for anyone who really wants their exponentialMovingAverage results rounded to 3 decimal places.

Codecov Report

Merging #2032 into master will decrease coverage by 0.46%.
The diff coverage is 97.14%.

@@            Coverage Diff             @@
##           master    #2032      +/-   ##
==========================================
- Coverage   69.23%   68.76%   -0.47%
==========================================
  Files          78       78
  Lines        7512     7406     -106
  Branches     1460     1432      -28
==========================================
- Hits         5201     5093     -108
- Misses       2037     2038       +1
- Partials      274      275       +1

Impacted Files
Coverage Δ

webapp/graphite/render/functions.py
87.26% <97.14%> (-0.8%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b77dde...0df78bb. Read the comment docs.

does the extra abstraction/function calling come with a noticeable overhead?

One function call? I very much doubt it.


chore: migrate to owl bot

@JustinBeckwith Could you work your magic again to make the linter happy?  My usual incantation had no effect.


Naming the build script build makes PaaS detect a Rails apps as a Node app

When deploying a Rails app to a platform like Heroku with multiple buildpacks (Node + Ruby), the app is detected as as Node app because package.json contains a build script.
Would you consider naming the build script build:js like in cssbundling-rails where the build script is named build:css ?
I would be happy to create a PR for that.

Think we should get Heroku to be smarter about it's detection. Build is the standard indeed. Maybe @schneems can help adjust the buildpack detection?

To be precise, this happened to us on a large / popular European PaaS that uses Heroku’s buildpacks.

What's the PasS? Maybe we can tag them here so they can update and be ready for Rails 7.

Because the detection is IN PART correct! A Rails 7 app with a build script in package.json is both a node app AND a Rails app at the same time. Needs both aspects configured correctly.

They are https://github.com/Scalingo
I'm currently checking with them if they have some internal scripts around/over Heroku's buildpacks that might be interfering.

They have a custom flag NPM_NO_BUILD=true for the Node.js Buildpack to disable runnings scripts on deploy. They'll document the practice in the future.
Another solution seems to be relying on the Rails' Buildpack's default node/yarn versions (which can be a bit dated).
Sorry for the trouble.

No worries. This is a new path. Gotta smooth it all out.

the app is detected as as Node app because package.json contains a build script.

If an app has both a package.json and a Gemfile we have written our detect order so that the heroku/ruby buildpack "wins". If that's not what you're seeing then let me know. I tried to reproduce the behavior you described and couldn't.
$ cat Gemfile
source 'https://rubygems.org'
gem 'rake'
$ cat package.json
{
  "scripts": {
    "start": "node index.js",
    "build": "webpack"
  }
}
$ git push heroku
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 16 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (7/7), 672 bytes | 672.00 KiB/s, done.
Total 7 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Building on the Heroku-20 stack
remote: -----> Determining which buildpack to use for this app
remote:  !     Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
remote:       Detected buildpacks: Ruby,Node.js
remote:       See https://devcenter.heroku.com/articles/buildpacks#buildpack-detect-order
remote: -----> Ruby app detected
remote: -----> Installing bundler 2.2.21
remote: -----> Removing BUNDLED WITH version in the Gemfile.lock

Even with a package.json containing a build script, it still detected as a ruby app which I believe is the desired behavior.

Think we should get Heroku to be smarter about it's detection.

Originally I misunderstood the issue and thought this comment on build:js was in relation to needing to specify a build script in the package.json. On re-read I see that's not the case. Still, it might be helpful to get a status update on the buildpack ecosystem.
The Ruby buildpack doesn't know anything about node, package.json or any of it's ecosystem.. The interface is that: if the app has execjs or webpacker bundled in the Gemfile then we ensure a version of node is on the app. https://github.com/heroku/heroku-buildpack-ruby/blob/41a5b1fc25e1a7e3627f5a836e8f91c5c884507b/lib/language_pack/ruby.rb#L1056-L1060.
Then we guarantee we will call rake assets:precompile after dependencies are installed. That's our standard interface. The way webpacker works with this is it enhances/extends the assets:precompile task to also invoke a yarn:install task first.
That being said, there's change in the wind. I want to get the Ruby buildpack out of the node installation business. It knows too much already and it's really confusing that we have a separate heroku/nodejs buildpack meaning that devs who use the two together get different behavior than just the devs who use only heroku/ruby. The nodejs buildpack DOES know about build and does execute it, it's a common interface for that ecosystem in the same way that assets:precompile is the interface for the Ruby community (not just rails, we call it for any app...it's essentially like an unofficial npm build that Rails made and we endorsed).
The way forward is for the Ruby buildpack to depend on the node buildpack. Unfortunately this way forward not easy. Heroku is going through the process of switching our standard of buildpack interface from the existing "legacy" aka "v2" interface https://devcenter.heroku.com/articles/buildpack-api in favor of the fully spec-d Cloud Native Buildpack (CNB) interface (https://buildpacks.io/). Cloud Native Buildpacks allow one buildpack to "depend" on another.
This allows the Ruby buildpack to say "I see you've got execjs/webpacker, make sure to run the nodejs buildpack first".
We are currently re-writing buildpacks to support the CNB interface. You can see my work here github.com/heroku/buildpacks-ruby along with an "application contract" of what an application can expect while using the buildpack.
There's a few more complicating factors: Heroku build system doesn't yet support running CNBs so we must support 2 buildpacks right now. There are issues depending on the node buildpack, for instance since it calls yarn install and rake assets:precompile also calls yarn install it gets run twice. The node buildpack clears dev dependencies after executing but the rake task needs them. (Documented as a TODO in the application contract).
Organizationally we've also decided to move away from each buildpack defining what language to be written in (Previously Ruby is written in bootstrapped Ruby while all other buildpacks are written in bash). We landed on a choice of re-writing in Rust. This means my work on my re-write I linked to today is stopped and exists only as a spike/reference as we ramp up to re-write it yet again. It's fairly well spec-d out so It should go faster, but it's not here today as a thing we can use right now.
Long term
In an ideal world our build ecosystem would support CNBs, we'll have customers transitioned over to that interface, the Ruby buildpack will be re-written in Rust and it will depend on the heroku/nodejs buildpack, the issues around clearing dev-dependencies between buildpacks will be offloaded (possibly to a RFC for a "clean" phase in cloud native buildpacks). And then you get your execution of npm build for free.
I doubt all that will happen before Rails 7 hits the streets. Any ballpark when that might optimistically/pessimistically happen? If it's awhile away then maybe there's a chance. If it's soon, then we'll have to look elsewhere
Short term
In the short term the workaround would be (if I understand the requirements) that developers add in the heroku/nodejs buildpack before deploying:
heroku buildpacks:add heroku/nodejs
heroku buildpacks:add heroku/ruby

It does mean that deploying to Heroku is no longer a "zero config, one step process" which we worked to get to with Rails 5.
I'm wondering if there might be other interfaces that could be leveraged to better tell heroku "do this thing by default". We do have the concept of an app.json file which you can specify buildpacks, but unfortunately, it only affects review apps and apps created by "heroku button" (https://devcenter.heroku.com/articles/heroku-button). It does not affect apps created by heroku create which in my personal opinion is a huge missed opportunity, otherwise, we could just have Rails add one in by default and it would "just work" ™. There might be some other ways around the issue. None come to mind though.
Into the future, once we get Ruby CNB shipped and these issues ironed out then it will go back to a "zero config" deploy and will "just work" and anyone who previously specified to use both heroku/nodejs and heroku/ruby will resolve to get the same behavior (basically there's not a downside to pushing people to use both buildpacks that we'll regret later).
I know that's a wall of text. I want to be very transparent about the state of the buildpack is currently in and where we're headed. Let me know if you've got additional questions about any of this.

I'll review all this is detail, but a few quick notes, there's no Webpacker or execjs with Rails 7. jsbundling-rails uses node directly without any of that, and the default path doesn't use node at all. So shouldn't base any detection scheme for Rails 7 for any of that.
With jsbundling-rails, though, it hooks onto assets:precompile to do its yarn install && yarn build automatically prior to running that task.
Rails 7 is going to ship an alpha as soon as this week. Then probably the first beta within a month. Maybe quicker. And then final 1-2 months after that. So let's say final is 2-3 months away.

Thanks for the update. I can try to port our Rails docs over to use Rails 7 as soon as a pre-release or alpha is available. If you're curious I use a custom build tool to dynamically "run" my documentation to ensure it actually deploys on Heroku. Here's the actual file I use for Rails 6 https://github.com/schneems/rundoc/blob/main/test/fixtures/rails_6/rundoc.md.
If it uses rake assets:precompile then it the asset generation should be good to go out of the box. We might need to coordinate in the future on how to not call yarn install twice for people who have both heroku/nodejs and heroku/ruby buildpacks. Either way for the rake yarn:install task to know that yarn install has already been run or a standard interface that the Ruby buildpack can use to skip it (the ruby buildpack knows whether or not the node buildpack has run yarn install or not).
The only incompatibility with Rails 7 I know of is these two tasks were previously deprecated and I believe they've been removed https://github.com/heroku/heroku-buildpack-ruby/blob/41a5b1fc25e1a7e3627f5a836e8f91c5c884507b/lib/language_pack/test/rails2.rb#L56-L57. We call them for test setup. It won't be a blocker for people trying Rails 7 on Heroku but will limit use of our CI feature until it gets fixed on our end to use whatever "propper" interface we deprecated those in favor of (I don't know off the top of my head).

Got the alpha out the door for Rails 7. It ships with sprockets as-is, which I think default adds execjs to the Gemfile.lock? So the detection should work correctly?

Looks like things are working fine on my build script
https://mobile.twitter.com/schneems/status/1438527886764376071
On Friday, September 17, 2021, David Heinemeier Hansson <
@.***> wrote:

Got the alpha out the door for Rails 7. It ships with sprockets as-is,
which I think default adds execjs to the Gemfile.lock? So the detection
should work correctly?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rails/jsbundling-rails/issues/16#issuecomment-921898004,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAOSYED3XFP2FJQVK6I2BDUCNO53ANCNFSM5DYYPQ4Q
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
Richard Schneeman
https://www.schneems.com
he/him

Looking good for import maps. Is everything also working with jsbundling? You can do rails new test -j esbuild to test that setup 👍

Confirmed that things work out of the box with rails new test -j esbuild on Heroku 👍

Confirmed fine on other PaaS too. Thanks !

@schneems I've read thru this issue quite a few times, and don't see a clear way to avoid the multiple yarn install issue on Heroku w/ jsbundling and the nodejs buildpack installed. Opening a new issue at #130 to see if there is  a nice way to do that. 🙏

@schneems I am still experiencing the issue where Heroku detects the Rails app as a Node.js app due to the build script in package.json:
"scripts": {
  "lint": "eslint",
  "build": "webpack --config ./webpack.config.js"
}

In my case, I depend on rails-erb-loader to process a js.erb file, and it depends on Ruby being present first because it calls bin/rails runner. Here's an excerpt from the webpack.config.js:
module: {
    rules: [
      {
        test: /\.erb$/,
        enforce: 'pre',
        exclude: /node_modules/,
        use: [{
          loader: "rails-erb-loader",
          options: {
            runner: "bin/rails runner",
          },
        }],
      }
    ],
  },

I tried placing the Ruby buildpack before the Node buildpack, but then I run into the issue where it defaults to Node 20 instead of the one specified in the package.json.
Note that I don't have the execjs nor the webpacker gems in the app.
What would be the solution in this case? How can I ensure Ruby gets built first, and that Heroku can respect the Node version in package.json?


Not auto detecting correctly until 'save'

If I leave my tab length in Atom Editor settings on the default 2 spaces, then open a file that is using 4 spaces tab length, it wrongly picks it up as 2 spaces. If I hit save without changing anything, it then correctly picks up that it should be 4 spaces.
The same happens the other way round. If I set the default in Atom Editor to 4 spaces, then open a file that has 2 spaces tabs, I have to again hit save first for it to correctly calculate the file's spacing.
Some projects I work on use 2 spaces tabs and some 4, every time I have to hit save first when I open a new file, or I have to keep changing the default.
Am I missing something?
I'm on Mac - El Capitan. Atom 1.13.0, auto-detect-indentation 1.3.0.
Thanks,
Steven

Just realised this is a duplicate of https://github.com/jtokoph/auto-detect-indentation/issues/25


SDK refactor, removed layer of DTOs. Added method channel for Android groth16prove

Codecov Report
Attention: Patch coverage is 23.33333% with 322 lines in your changes missing coverage. Please review.

Project coverage is 21.64%. Comparing base (2c11bce) to head (9e8f177).

Files
Patch %
Lines

lib/proof/libs/prover/prover.dart
0.00%
44 Missing :warning:

...ib/identity/data/data_sources/smt_data_source.dart
0.00%
40 Missing :warning:

lib/iden3comm/authenticate.dart
0.00%
26 Missing :warning:

lib/identity/domain/entities/hash_entity.dart
39.53%
26 Missing :warning:

...of/data/data_sources/gist_mtproof_data_source.dart
0.00%
17 Missing :warning:

...proof/data/repositories/proof_repository_impl.dart
0.00%
16 Missing :warning:

...ty/data/repositories/identity_repository_impl.dart
11.76%
15 Missing :warning:

...iden3comm/data/mappers/iden3comm_proof_mapper.dart
0.00%
13 Missing :warning:

...dentity/data/repositories/smt_repository_impl.dart
0.00%
13 Missing :warning:

.../data/data_sources/circuits_files_data_source.dart
0.00%
12 Missing :warning:

... and 28 more

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #419      +/-   ##
===========================================
+ Coverage    21.35%   21.64%   +0.28%
===========================================
  Files          320      302      -18
  Lines         9496     9334     -162
===========================================
- Hits          2028     2020       -8
+ Misses        7468     7314     -154

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Undefined Reference - MIOpenGEMM

Below is the link error I am getting with
miopen-opencl                              <IP_ADDRESS>05-rocm-rel-2.7-22-4e39a83                  amd64        AMD's DNN Library
miopengemm                                 <IP_ADDRESS>5-rocm-rel-2.7-22-6275a87                   amd64        A tool for generating OpenCL matrix multiplication (GEMM) kernels

/opt/rocm/lib/libMIOpen.so.1: undefined reference to `MIOpenGEMM::Geometry::initialise(bool, bool, bool, bool, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char)@MIOPENGEMM_1'
/opt/rocm/lib/libMIOpen.so.1: undefined reference to `MIOpenGEMM::find(float, _cl_command_queue*, _cl_mem*, _cl_mem*, _cl_mem*, bool, MIOpenGEMM::Geometry const&, bool, bool)@MIOPENGEMM_1'
collect2: error: ld returned 1 exit status

Downgrading to MIOpen - 1.8.1 & MIOpenGEMM 1.1.5 solves this issue temporarly.

Does down grading just MIOpenGEMM fix the issue?

@daniellowell I tried with MIOpen - 2.0.1 & MIOpen - 2.0.0 with MIOpenGEMM 1.1.5, the problems still persists.

@kiritigowda do you have a docker we can try ?

@JehandadKhan follow the steps to reproduce on docker

Docker Launch

sudo docker run -it --device=/dev/kfd --device=/dev/dri --cap-add=SYS_RAWIO --device=/dev/mem --group-add video --network host --env DISPLAY=unix$DISPLAY --privileged --volume $XAUTH:/root/.Xauthority --volume /tmp/.X11-unix/:/tmp/.X11-unix mivisionx/ubuntu-18.04:miopen-bug

Resnet50 OpenVX Graph

cd && ls

Create build folder, cmake and make

mkdir build && cd build && cmake ../resnet50-openvx && make

Should produce error
//opt/rocm/lib/libMIOpen.so.1: undefined reference to `MIOpenGEMM::Geometry::initialise(bool, bool, bool, bool, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char)@MIOPENGEMM_1'
//opt/rocm/lib/libMIOpen.so.1: undefined reference to `MIOpenGEMM::find(float, _cl_command_queue*, _cl_mem*, _cl_mem*, _cl_mem*, bool, MIOpenGEMM::Geometry const&, bool, bool)@MIOPENGEMM_1'
collect2: error: ld returned 1 exit status
CMakeFiles/anntest.dir/build.make:112: recipe for target 'anntest' failed
make[2]: *** [anntest] Error 1
CMakeFiles/Makefile2:141: recipe for target 'CMakeFiles/anntest.dir/all' failed
make[1]: *** [CMakeFiles/anntest.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Is there any update on this issue? It is currently blocking me from running tests on MI100.

Is there any update on this issue? It is blocking some of my work.

To unblock yourself, can you try building MIOpen on your local system and doing make install
Please remove the previous package before installing. There may be an issue with the packaging.
Flow the instructions here:
https://rocmsoftwareplatform.github.io/MIOpen/doc/html/install.html#building-miopen-from-source

@daniellowell
From source:
mkdir build
cd build/
cmake -DMIOPEN_BACKEND=OpenCL ..
sudo make -j8 install
[100%] Linking CXX executable anntest
//opt/rocm/lib/libMIOpen.so.1: undefined reference to MIOpenGEMM::Geometry::initialise(bool, bool, bool, bool, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char)@MIOPENGEMM_1' //opt/rocm/lib/libMIOpen.so.1: undefined reference to MIOpenGEMM::find(float, _cl_command_queue*, _cl_mem*, _cl_mem*, _cl_mem*, bool, MIOpenGEMM::Geometry const&, bool, bool)@MIOPENGEMM_1'
it looks like installing from source still missing something

@paolodalberto if you rerun the MIVisionX-setup.py again, this issue will be fixed.
Else, get MIOpenGEMM 1.1.5 build and install it, then rebuild MIOpen 2.1.0 and install it.

sudo python MIVisionX-setup.py
MIVisionX Dependencies Installed
MIVisionX-setup.py ... how do I uninstall it ?

paolo@fastmmw:~/FastMM/MIVisionX/ONNX/squeezenet-openvx/build$ make
Scanning dependencies of target annmodule
[ 16%] Building CXX object CMakeFiles/annmodule.dir/annmodule.cpp.o
[ 33%] Linking CXX shared library libannmodule.so
[ 33%] Built target annmodule
Scanning dependencies of target annpython
[ 50%] Building CXX object CMakeFiles/annpython.dir/annpython.cpp.o
[ 66%] Linking CXX shared library libannpython.so
[ 66%] Built target annpython
Scanning dependencies of target anntest
[ 83%] Building CXX object CMakeFiles/anntest.dir/anntest.cpp.o
[100%] Linking CXX executable anntest
[100%] Built target anntest

@dagamayank has mentioned in an internal ticket that MIOpenGEMM will no longer be supported. Should we note this here and close the ticket?

@daniellowell, has this issue been addressed?

Any update on this? @daniellowell @paolodalberto

Close due to inactivity. @kiritigowda if the issue still persists in newer versions of MIOpen, please comment and reopen. Thanks!


Update dependencies

#1023

+1

+1

it's not a problem because in this PR "lodash": ">=1.3.0"
the real problem is apathy of restangular developers

Deps to be updated tomorrow. Stay tuned.

Dependencies updated in a separate branch. Waiting for @mgonto to release a new version after he goes back from conferences. For the time being - you can refer to this branch. Further updates will be provided in the issue connected with this PR. Thanks for your support.