Changes to support hot-reloading / React Refresh

Background
A common complaint that I hear often against using Scala.js, is that the dev feedback cycle is really fast with JS but really slow with Scala. I'm referring to the amount of time it takes between modifying a source file and seeing the change reflected in a browser. The JS community have really endeavoured to get this time down to as fast as possible, both by the creation of dev servers that push updates to the browser immediately, and by hot-reloading tech that swaps out parts of a live app without requiring a page reload or loss of state (where possible). The result is that JS devs can enjoy effectively-instant updates in the browser when making local changes. Giving up this fast feedback cycle often seems to be a hard sell for people considering Scala.js.
Goal
I'm currently working on (hopefully) adding support for React Refresh (i.e. hot reloading) to scalajs-react. By combining Scala.js's awesome ability to spit out fine-grained ES modules, and a fast bundler that supports hot-reloading (like Vite), I hope to provide a fast dev feedback cycle using Scala, that can stand up to the JS experience. It's likely to be in the order of 500ms on the Scala side compared to 5ms in JS, which from a human's pov should be equivalent enough. For large projects it might be 2 sec, but that's way more acceptable than the current Scala status quo of around 20 sec.
Problems
After much experimentation and debugging, I was able to get Vite hot-reloading Scala.js, and it's fast! The approach seems like to should work seamlessly with bundlers like Webpack, etc too.🥳
However, hacks were required; and by hacks, I mean part of what I had to do was manually hack and chop up the Scala.js output. These were the problems I encountered. For objects that contain scalajs-react components...

the object's class needs to be in its own module, and be the sole and default export. -- If Vite sees a non-default export, it won't hot-reload that module.

the object's class module needs to eagerly initialise the object. -- Bundles will inject hot-reloading code into the module and expect it to be used by the time the module has finished loading. Calling the hot-reload code when Scala decides to initialise the module doesn't work because the module has already loaded, and the hot-reloading code has removed itself by then.

when an object's source is modified, only the object class's module should be updated (which contains the object body), and not the module with object's singleton-getter (which seems to be static boilerplate). -- When Vite sees an update to the object's class module, it will hot-reload it. When Vite sees an update to the object loader, it doesn't find anything hot-reloadable and instead reloads the entire page, which loses all user state.

Proposed Solution
In order for scalajs-react (and presumably some other libraries too) to support hot-reloading, and hot-reloading without a page reload especially, some Scala.js changes seem unavoidable. As an initial draft proposal, I suggest the following:

A new ability to Scala.js that

for some Scala objects (see the next section for a concrete example)

separate the object class into its own module as the sole, default export
eagerly initialise the object when the class module is loaded
allow for replacement of object instances

decides which Scala objects to transform by, at a minimum, inspecting the object body, probably looking for a term (eg. scalajs.js.special.eagerlyLoad;). (An annotation won't work because it needs to be generated in scalajs-react call-site. It needs to be something a macro can provide that users can store in a val.)
can be globally enabled/disabled via an sbt setting
will only work when modules are being emitted (maybe some other constraints?)

If the module emitting code doesn't already have this, teach it to remember previous output (at least checksums) and avoid replacing files with the exact same content

Sample Changes In Output
Consider this example Scala source code:
package demo

object MyComponent {
  org.scalajs.dom.console.log("MyComponent initialising...")
}

Currently it emits a single module:
demo.MyComponent$.js:
'use strict';
import * as $j_java$002elang$002eObject from "./java.lang.Object.js";

/** @constructor */
function $c_Ldemo_MyComponent$() {
  $n_Ldemo_MyComponent$ = this;
  console.log("MyComponent initialising...")
}
export { $c_Ldemo_MyComponent$ as $c_Ldemo_MyComponent$ };
$c_Ldemo_MyComponent$.prototype = new $j_java$002elang$002eObject.$h_O();
$c_Ldemo_MyComponent$.prototype.constructor = $c_Ldemo_MyComponent$;

/** @constructor */
function $h_Ldemo_MyComponent$() {
  /*<skip>*/
}
export { $h_Ldemo_MyComponent$ as $h_Ldemo_MyComponent$ };
$h_Ldemo_MyComponent$.prototype = $c_Ldemo_MyComponent$.prototype;
var $d_Ldemo_MyComponent$ = new $j_java$002elang$002eObject.$TypeData().initClass({
  Ldemo_MyComponent$: 0
}, false, "demo.MyComponent$", {
  Ldemo_MyComponent$: 1,
  O: 1
});
export { $d_Ldemo_MyComponent$ as $d_Ldemo_MyComponent$ };
$c_Ldemo_MyComponent$.prototype.$classData = $d_Ldemo_MyComponent$;

var $n_Ldemo_MyComponent$;
function $m_Ldemo_MyComponent$() {
  if ((!$n_Ldemo_MyComponent$)) {
    $n_Ldemo_MyComponent$ = new $c_Ldemo_MyComponent$()
  };
  return $n_Ldemo_MyComponent$
}

export { $m_Ldemo_MyComponent$ as $m_Ldemo_MyComponent$ };

Hot-reloading compatible output would look like this:

demo.MyComponent$.js:
'use strict';
import * as $j_java$002elang$002eObject from "./java.lang.Object.js";

// ****** NOTE: This snippet is moved into another file
import $c_Ldemo_MyComponent$ from "./demo.MyComponent.js"
export { $c_Ldemo_MyComponent$ as $c_Ldemo_MyComponent$ };

/** @constructor */
function $h_Ldemo_MyComponent$() {
  /*<skip>*/
}
export { $h_Ldemo_MyComponent$ as $h_Ldemo_MyComponent$ };
$h_Ldemo_MyComponent$.prototype = $c_Ldemo_MyComponent$.prototype;
var $d_Ldemo_MyComponent$ = new $j_java$002elang$002eObject.$TypeData().initClass({
  Ldemo_MyComponent$: 0
}, false, "demo.MyComponent$", {
  Ldemo_MyComponent$: 1,
  O: 1
});
export { $d_Ldemo_MyComponent$ as $d_Ldemo_MyComponent$ };
$c_Ldemo_MyComponent$.prototype.$classData = $d_Ldemo_MyComponent$;

// ****** NOTE: A variable and singleton-logic is removed in favour of a global var lookup.
// ******       These objects are always eagerly initialised and will always exist.
function $m_Ldemo_MyComponent$() {
  return globalThis.scalaJsEagerObjects["demo.MyComponent"];
}

export { $m_Ldemo_MyComponent$ as $m_Ldemo_MyComponent$ };

demo.MyComponent.js:
'use strict';
import * as $j_java$002elang$002eObject from "./java.lang.Object.js";

/** @constructor */
function $c_Ldemo_MyComponent$() {
  // ****** NOTE: Here we add/replace ourselves in a global var instead of a local one
  // ******       hidden in another module.
  (globalThis.scalaJsEagerObjects ||= {})["demo.MyComponent"] = this;
  console.log("MyComponent initialising...")
}
$c_Ldemo_MyComponent$.prototype = new $j_java$002elang$002eObject.$h_O();
$c_Ldemo_MyComponent$.prototype.constructor = $c_Ldemo_MyComponent$;

// ****** NOTE: Here we eagerly initialise object.
// *****        Also replaces an older version on hot-reload.
new $c_Ldemo_MyComponent$()

// ****** NOTE: Here we have a single default export.
export default $c_Ldemo_MyComponent$;

Who Does The Work?
I might be able to put my hand up to implement this. It seems about a medium-difficulty change to me, but we'd have to work out how long we think it would take before I can say much more.
I think it'd be a really good idea to work together with Team Scala.js to come up with a plan that we all agree is effective and feasible. With your blessings and guidance, I hope we can at least get a spec-lite ready.

Thanks for the write-up. Some first comments:
Where can I read more about what Vite can and cannot hot-reload? Where can I see some kind of spec or documentation of the acceptable layouts? (for example, the single default export thing) Those restrictions seem pretty arbitrary. What happens when Vite gets replaced by another tool that wants another layout? We can't bake in the core assumptions about the JS bundler en vogue. We need to keep the core generic wrt. those kinds of assumptions. So we need to tread quite carefully here.
Perhaps there would be a way to open the ModuleSplitter to a custom hook for something non-core to provide its own module splitting logic.

separate the object class into its own module as the sole, default export

In general, this cannot be guaranteed. If object A and object B refer to each other (directly or indirectly), their classes and their module accessors (the $m_ stuff) will all have to belong to the same ES module. What do we do then?

the object's class module needs to eagerly initialise the object.

This is clearly a change of semantics. That is, erm, difficult. But it's not impossible that some sort of "linker plugin" (or anything that pre-processes the IR given to the linker, at any rate) could inject a static initializer into the relevant classes. Static initializers are guaranteed to run eagerly, and they can force the module to be initialized. After all, this is part of how @JSExportTopLevel objects are implemented.

If the module emitting code doesn't already have this, teach it to remember previous output (at least checksums) and avoid replacing files with the exact same content

This is done already :) So, there's that. ;)

Would a ModuleSplitter hook help on implementing https://github.com/scala-js/scala-js/issues/4327 too?

I guess, yes. At least, it would allow to iterate on it outside of the core.

the object's class module needs to eagerly initialise the object.

Isn't this a version of #1593? The linker is allowed to do this if the object initializer is side-effect free (but as discussed in #1593, there are potential performance trade-offs). But maybe requiring it to be side-effect free is not an issue for this use case.

I don't think it's really about #1593. That one is supposed to be a semantics-preserving optimization. Here, IIUC, we would want this to apply even to objects that may have side effects, as long as they're the ones involved in React reloading.

Where can I read more about what Vite can and cannot hot-reload? Where can I see some kind of spec or documentation of the acceptable layouts? (for example, the single default export thing)

I read through all of the Vite documentation and don't remember finding anything detailed on React Refresh, just how to enable it and that it will magically work. Documentation on exactly how this all works is very sparse and lacking. I discovered the export issue via experimentation and debugging. I've provided a quick demo/playground here: https://github.com/japgolly/scalajs-react/tree/experiment/vite/vite-react-demo

Those restrictions seem pretty arbitrary. What happens when Vite gets replaced by another tool that wants another layout?

From what I've seen, any bundler using hot-reloading with React just expects a single export default <component> by convention as that's just what everyone in the JS React world does. That being said, 1) I haven't surveyed all the popular bundlers in detail, 2) obviously I have no control over the future :) Things could certainly change. I'm personally ok with adhering to this common convention though, just because it's so naturally ubiquitous; no one has mandated this layout, it's just the simplest and most natural for a JS dev and I don't see that changing.

We can't bake in the core assumptions about the JS bundler en vogue. We need to keep the core generic wrt. those kinds of assumptions. So we need to tread quite carefully here.

Yeah, you're right. I was initially thinking about some opt-in special treatment but...

Perhaps there would be a way to open the ModuleSplitter to a custom hook for something non-core to provide its own module splitting logic.

This would be the best! If we could write our own plugins outside of Scala.js that would be amazing! That would really be the best option. It would be similar to a Scala version of Babel. For this issue in particular it would allow me to maintain a transformation plugin that I can update very regularly without stalling or blocking SJS itself. It would also allow for experimental custom optimisations too, similar in spirit to the monadic-for compiler plugin.
Hey btw, wouldn't it be cool if the existing two module splitters could be extracted to just be standard plugins too? :D

In general, this cannot be guaranteed. If object A and object B refer to each other (directly or indirectly), their classes and their module accessors (the $m_ stuff) will all have to belong to the same ES module. What do we do then?

Sorry, I'm not sure what you mean. I did a hacky experiment with nested objects and my strategy worked fine. From memory the $m_ doesn't need to be in the same module as the class def does it? And if it does, there are always tricks like just storing it on the class prototype object or something :)

This is clearly a change of semantics. That is, erm, difficult. But it's not impossible that some sort of "linker plugin" (or anything that pre-processes the IR given to the linker, at any rate) could inject a static initializer into the relevant classes. Static initializers are guaranteed to run eagerly, and they can force the module to be initialized. After all, this is part of how @JSExportTopLevel objects are implemented.

Yeah I recognise that. That's why I was thinking I'd look for some kind of term in the object body to be very selective about which objects we change the semantics of, because you wouldn't want to do it for every object on the classpath. It might just be a decent tradeoff to accept. That being said, as we explore more options (especially if I can write plugins), it might be possible to extract just the React components so that they're eagerly initialised, change the object to simply point to these now-external components, and leave the object initialisation semantics alone. I didn't attempt this in my experimentation because I didn't imagine I'd have so much control over the Scala.js output but give me a plugin and more control and I may be able to come up with much nicer solutions.

This is done already :) So, there's that. ;)

Legends!
Plugin powers
So just to brainstorm what kind of abilities these potential plugins could have, I'd love to be able to...

Create new modules
Rename modules
Delete modules
Update the contents of a module
Know the entire (current) set of modules

(ok literally just CRUDL with renaming)
And then at the sbt level, it would be good to be able to specify an ordered list of plugins.
I imagine the above would be enough to implement #4327 in all kinds of different ways (because the grouping criteria is pretty arbitrary). One person might want to group by package, someone else by regex, someone else might want custom rules for their project. The burden to implement it in a way that's flexible enough for everyone would be greatly reduced because people could create their own versions with their own rules without too much hassle.
#1593 should be possible to implement via plugins too. It'd be so awesome for someone to be able to write a plugin for something like #1593, test it out for a while, get community support, then if it all works out well either copy it into Scala.js proper, or maybe even just refer people to it and keep Scala.js as generic as possible. We could end up with an "Awesome Scala.js Plugins" list with a bunch of community favourites that recommend.
I also imagine it would be good to have a similar plugin ability that runs on {fast,full}{Opt,List}JS and not just modules, that would basically be able to do the same thing but with straight IR or virtual files or something. Eg. we could replace Semantics.RuntimeClassNameMapper by a simple IR transformation plugin.
That's enough brainstorm for now. This idea of plugins is very exciting!

Oh before I forget, I would be important to be able to provide plugin configuration when used. We could keep this reaaaaaaaaally basic for PoC/v1, and just plumb through a list of string from user's sbt to the plugin constructor, that the plugin can parse as it pleases.
Example in a users sbt:
scalaJsLinkerPlugins ++= Seq(
  LinkerPlugin("com.whatever.Foo"), // no config specified here
  LinkerPlugin("com.whatever.Bar").withConfig("a=1", "b=2", "--verbose"),
)

Regarding the plugin idea: What you are describing in "Plugin Powers" can already be reasonably done using the linker.standard package.
A good starting point is probably StandardLinkerImpl:
https://github.com/scala-js/scala-js/blob/main/linker/shared/src/main/scala/org/scalajs/linker/standard/StandardLinkerImpl.scala
Apart from all the bookkeeping, what it does is:

Create a ModuleSet, using StandardLinkerFrontend.
Pass that ModuleSet to StandardLinkerBackend

You can write your own implementation of Linker and do whatever you want between steps 1 / 2.

I've been looking into this for the past few days and have discovered a few bits and pieces that may be useful.
@sherpal has a project with scalajs hot reloading working with webpack:
https://github.com/sherpal/BattleForFlatland
explanation
https://gitter.im/Laminar_/Lobby?at=5ee276872cf2f36eae536387
@kitlangton has adapted this to use vite in a few repos; probably the most complete example is here:
https://github.com/kitlangton/conclave
something like a spec/API for hot reloading, but I think it's only really used for snowpack and maybe vite?
https://github.com/withastro/esm-hmr
Only just getting started with scalajs and it looks like there is a lot of magic happening here, but happy to try and help with this if I can.

There is also this with hot reloading - https://slinky.dev/

@robmwalsh Thanks for the links. I checked them out and unfortunately they aren't applicable to React Refresh specifically which does extra stuff once a module is reloaded
@ekrich Yeah I was originally going to do it the way Slinky does but that's now deprecated with React Refresh as the replacement. Thank you though

FYI, I've had a million billion things on, tons of work-work and life-work stuff, everyone in the family getting sick every two seconds, but things are settling back down and so this issue is back on my radar again. I'll start by looking at StandardLinkerImpl, thanks for the guidance @gzm0


WFLY-10964 JPA statistics do not show in the management console

https://issues.jboss.org/browse/WFLY-10964
https://issues.jboss.org/browse/JBEAP-15337

TransactionalRemoteStatefulEJBFailoverTestCase failure doesn't seem related to this pr.

retest this please


Recategorize azure-common to exist under the 'Other' label on github.io

Resolves Azure/azure-sdk-for-python#24775
@danieljurek @sima-zhu can you double check this?
The "core" page: https://azure.github.io/azure-sdk-for-python/core.html
Doesn't have core showing up on it. This is because we've recategorized core service under Other category.
To keep it consistent, let's move azure-common to Other. Can you double check the space I updated? I'm pretty it should get used as we expect.

Seems like js, .net and java have same issue?
https://azure.github.io/azure-sdk-for-net/core.html
https://azure.github.io/azure-sdk-for-js/core.html
https://azure.github.io/azure-sdk-for-java/core.html

@scbedd Can we have the same change for the other three langs?

Filing follow-up issue #4438  and merging this PR. Expect another PR later this evening.


Log exception on keystore build for custom certificate

Description
If CloudStack fails to build and save keystore using the keystone certificate then it silently fails with just the following line:
2022-05-17 15:09:02,859 WARN  [o.a.c.f.s.k.KeystoreManagerImpl] (AgentConnectTaskPool-21:ctx-6795fce2) (logid:d7b1fab5) Unable to build keystore for CPVMCertificate due to CertificateException

This PR adds the exception to the logger

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3419

@blueorangutan test

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan package

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3435

Trillian test result (tid-4201)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29590 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6394-t4201-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File


split Makefile.am.sample

In order to facilitate the integration of generated sources into the project tree, I would propose to split the generated Makefile.am.sample to 2 files: Makefile.am.sample containing only list of source files and Makefile.sample containing rules to generate the sample encoder/decoder.
In this case the Makefile.am.sample can be directly included by other makefiles.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Coverage remained the same at 59.013% when pulling 82f197178ade5f968c7ed7313cef6a1d4c05f9d6 on ETSI-CTI:pr into 81d6c576cc42d0a24cc0a255dfc77a8cf4a1ae7c on vlm:master.

The problem with this approach is that .am files (the ones which are used by autotools) have to have sources inside them, not on the side.
Any idea how to combine these?

This is exactly my idea to keep the list of sources in the Makefile.am.sample (or even already rename it to the Makefile.am) to be able to use it directly in autotools or include it into other Makefiles.
Otherwise I have to edit the generated Makefile.am.sample after each regeneration.

How about implementing --make Makefile or --automake Makefile.am options? This way, we can add --cmake and others, when needed.

Sure, it could be very good to support other build environments when it will be needed.
Actually compiler shall generate the list of source files to be integrated in some external project. The Makefile.am provides this list and the Makefile.sample is an example of such a project.
So that, Makefile.am contains list of module source files and all necessary variables and defines to build the library and can be used directly for automake or can be included in any external Makefile.
Makefile.sample includes this file and builds the converter utility.
Please have a look in last commit in the CR. I removed all converter sources from the Makefile.am

Coverage remained the same at 59.013% when pulling 54e3d0fa13f49399a0fd31ba41a991873c45e6e8 on ETSI-CTI:pr into 81d6c576cc42d0a24cc0a255dfc77a8cf4a1ae7c on vlm:master.

Coverage remained the same at 59.013% when pulling 904f006230ad5661ff403b36cbaa33659117657b on ETSI-CTI:pr into 81d6c576cc42d0a24cc0a255dfc77a8cf4a1ae7c on vlm:master.

Coverage remained the same at 58.967% when pulling 034f3760a79cff9b391b97fe510aec70c2a6c843 on ETSI-CTI:pr into ef8dd441fcb492a37008bf7b8ec1c85b7b3c2a5d on vlm:master.

The original generated file was named Makefile.am.sample and not Makefile.am to avoid accidental clobbering existing Makefile.am in the existing projects during recompilation.
So I reworked your patch to support splitting into:

Makefile name
Description

Makefile.am.example
Defines the converter-example binary

Makefile.am.libasncodecs
Defines the libcodecs library

Thank you for your contribution!


Integration test improvements

This adds more test cases to the integration tests running on a testflinger instance. Previously, the only integration test was checking whether the charm would build and could be successfully deployed. The following tests included in this PR are designed to test the following:

Node labels from the GPU plugin are present and correct
The number of allocated container slots on the node is correct
The application is in the correct state and has the expected version information
The version of the operator and cert-manager components can be adjusted and the operator comes back up (this case is less useful now because there is only one version of each)

I also included some other minor cleanup to improve the stability and reproducibility of the test setup, and cleaned up some of the naming.

See PEK-748

Hello; nice ! I would separate this PR in 2 PRs
1. GH action

2. Tests coverage and improvements

Good call! I just opened https://github.com/canonical/intel-device-plugins-k8s-operator/pull/10 and once that is approved and merged I will open a second one for the GH action.

@hector-cao It occurred to me just now about your questions about scalability - something I should have mentioned is that steps within a job cannot be run in parallel. So submitting multiple testflinger jobs from within the same section would not occur in parallel because they are separate steps and each step runs sequentially.

job

Right now integration-tests-system-A and integration-tests-system-B are 2 steps ?

integration-tests-system-A and integration-tests-system-B are each jobs. Jobs can be run in parallel so long as you have sufficient runners meeting the runs-on requirement. Here's a condensed view of the yaml:
jobs:
  .
  .
  integration-tests-system-A:
    name: Integration tests A on Testflinger instance with Intel GPU(s)
    runs-on: [testflinger]
    steps:
      - name: Check out code
        uses: actions/checkout@v4
      .
      .
      - name: Submit testflinger job
        uses: canonical/testflinger/.github/actions/submit@main
        with:
          poll: true
          job-path: ${GITHUB_WORKSPACE}/maas2.yaml

  integration-tests-system-B:
    name: Integration tests B on Testflinger instance with Intel GPU(s)
    runs-on: [testflinger]
    steps:
      - name: Check out code
        uses: actions/checkout@v4
      .
      .
      - name: Submit testflinger job
        uses: canonical/testflinger/.github/actions/submit@main
        with:
          poll: true
          job-path: ${GITHUB_WORKSPACE}/oemscript.yaml

Each job gets picked up by a runner (in this case a self-hosted runner that is configured to submit jobs to testflinger), and a runner runs each step within the job serially. So we can continue adding more machines to either of the job sections by building more testflinger submission scripts, but the runner will only testflinger submit once the previous testflinger job from the previous step has completed.
However, now that we have two self-hosted runners deployed the jobs should each get picked up by a separate runner and the runners will run their sections in parallel.

This means that we can have multiple
   - name: Submit testflinger job
     uses: canonical/testflinger/.github/actions/submit@mai

in each job ?

This means that we can have multiple
   - name: Submit testflinger job
     uses: canonical/testflinger/.github/actions/submit@main
     with:
     poll: true
     job-path:  xxx

in each job ?

Yep!


There is no samples with task type of M4 in the released dataset

Thanks for your awesome work! I have downloaded the dataset but find that there is no samples with task type of M4.

Thanks for your follow-up! In the current version of FunQA, most annotators weren't professional magicians. M4 will be released in the next version of FunQA.


Is this including optional and driver updates?

Hello,
does this module include driver and optional Windows updates?

Hello,
does this module include driver and optional Windows updates?

Yes, this library is only a golang binding for Windows Wuapi. You can do anything Wuapi supported.
e.g. include driver and optional ones by criteria settings


[BUG] Syntax highlight fails on broken emoji and Spyder will rise error

What steps will reproduce the problem?

# -*- coding: utf-8 -*-
"""
Spyder Editor

This is a script file.
"""

print("""
      🐛🌿�
      """)
print("this is now without syntax highlight")

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

# just copythis into an UTF-8 windows *.py file

# print("""
#       🐛🌿�
#       """)

# i try spyder 1th time and did copy this from a text editor into spyder.

# in text editor some emojis appear as a combination or two letters
# on some emojis if i hit backspace after them they turn into an other symbol or unknown symbol
# but they do not break syntax highlight
# the code with this fragment i did run many times in CMD and had no problem
# Editor can handle the may  unknown broken symbol and python can handle it too
# Spyder cant handle it an will rise an error and asks to send this description
# it will break the sntax highlight.
# this emojy string i got from an telegram chat title
# so even if it is a broken emoji i consider it as correct
# Spyder should be able to handle weird stuff even if it is odd
# The rest looks cool so far, so please fix this error in Spyder

this is in details
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 9: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 11: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 12: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 10: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 11: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 12: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 12: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 12: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 8: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 9: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 325, in highlightBlock
    self.highlight_block(text)
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 644, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 10: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 1178, in highlightBlock
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 8: surrogates not allowed
Traceback (most recent call last):
  File "C:\Program Files\Spyder\pkgs\spyder\utils\syntaxhighlighters.py", line 1178, in highlightBlock
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "C:\Program Files\Spyder\pkgs\spyder\utils\qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udd0d' in position 10: surrogates not allowed

this is in about text

* Spyder version: 5.3.3  (standalone)
* Python version: 3.8.10 64-bit
* Qt version: 5.15.2
* PyQt5 version: 5.15.7
* Operating System: Windows 10

Hi @Parlazo thank you for the feedback! I was able to copy paste the text provided but seems like the syntax hightlithting still works:

Could you share with us an example file which causes this issue, please? Any further info in order to reproduce this is greatly appreciated!

This seems to me more a problem with the font being used in the editor than with Spyder. Also, maybe @impact27 knows better what's happening here.

Closing due to lack of response


7/2/2020 Publish

Docs Build status updates of commit 7d9d397:
:clock10: Full build: the average full build time is 7 min(s) 48 sec(s), this is based on the last 727 build(s) for this repository.

Docs Build status updates of commit 7d9d397:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-conceptual/azure-cli-reference-for-IoT.md
:white_check_mark:Succeeded
View (azure-cli-latest) View (azure-cli-2017-03-09-profile) View (azure-cli-2018-03-01-hybrid)

docs-ref-conceptual/install-azure-cli-zypper.md
:white_check_mark:Succeeded
View (azure-cli-latest) View (azure-cli-2017-03-09-profile) View (azure-cli-2018-03-01-hybrid)

latest/docs-ref-autogen/acr/scope-map.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/acr/token.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ams/asset.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


[Function expose] Codeunit 1248 "Process Bank Acc. Rec Lines"

Codeunit 1248 "Process Bank Acc. Rec Lines"
We would like to be able to call below two methods in Codeunit 1248 .
ImportBankStatement
CreateBankAccRecLineTemplate
Thank you.

Thank you for your request, these procedures will be exposed in the future update.


Example scripts still use 2.0.0-preview-12 in package.json

package.json of prisma-examples repo , prisma2 branch still use preview-12:
https://github.com/prisma/prisma-examples/blob/prisma2/javascript/script/package.json#L7
https://github.com/prisma/prisma-examples/blob/570a0dd785209644feea96ba7b1691eea0a8865b/javascript/rest-express/package.json#L15
This was harder to expected to find, as GitHub search doesn't seam to search non-master branches of repos via GitHub search.
@schickling I think this explains one of the behaviors you pinged me about from a user.
We should make sure this will not get forgotten in the future.

Could init on install maybe just write its own version number into package.json?

@nikolasburk fixed this in https://github.com/prisma/prisma-examples/commit/926fcf81207ae5f172f0ba52d9863fcc58ea3852a


More realistic example?

This looks really exciting, but as a developer new to Nim, I don't understand what's happening in the example.  Could you make a more "hello world" type example that opens a dialog box or puts an element in the DOM?

thank you for the report.
i made it for my own study, but I am very happy to have someone to watch.
i will write a readme soon and add another example.


Add option to Multi.one to consider nil value as error

This proposed change adds the option nil_as_error to Ecto.Multi.one/4 to return {:error, nil} instead of {:ok, nil} which could be useful on some cases where we need to break the pipeline when no value is returned.

Thank you for the PR but I would prefer to not overload the meaning of one here with a separate meaning than the one in Ecto.Repo. It would likely be clearer to have a separate function but it is unclear what should be its name.
Another option is to have an explicit step to abort on nil: Ecto.Multi.one |> Ecto.Multi.error_on_nil(& &1.foo)

Thank you for the PR but I would prefer to not overload the meaning of one here with a separate meaning than the one in Ecto.Repo. It would likely be clearer to have a separate function but it is unclear what should be its name.
Another option is to have an explicit step to abort on nil: Ecto.Multi.one |> Ecto.Multi.error_on_nil(& &1.foo)

Thank you for the feedback, @josevalim
To be honest I'm not sure what would be the best approach here, right now I'm adding validations to a Ecto.Multi.run, I'll send another commit with the suggested function Ecto.Multi.error_on_nil(& &1.foo).
Would a function named fetch makes sense in the Multi context? e.g. Ecto.Multi.fetch(:foo, from(f in Foo, where: f.id == 1))  it would run a repo.one but returning :error on nil.
Thanks again.

Maybe Multi.halt?/1,  similar to plug? It accepts a function where the argument is the changes and if it evaluates to true then returns error?

Maybe Multi.halt?/2,  similar to plug? It accepts a function where the argument is the changes and if it evaluates to true then returns error? Or /3 with an option to configure the error?
This would also let you stop the pipeline with conditions based on multiple steps.

That's great @greg-rychlewski .
I'll implement it later and submit it to review.
Thanks!!

I'll implement it later and submit it to review.

Oh maybe let's wait for Jose's opinion. Actually I thought about what I said some more and I have a counter argument to it:
It's basically the same amount of code as 'run' and run is letting you control the error message more naturally.  Also not sure if the meaning of halt is the same as this. Reading the docs again, halt seems more like "stop and return what you have", not "stop and create an error".
Sorry for the noise. Probably using run is better than what I said. Still trying to think of possible new function but not sure yet.
One thing though, I'm not too sure about returning an error straight from a Repo-like function. I'm not completely sure we should ever say it's an error for a query to have no results. That seems more like post-processing to me.

The best thing I can think of for a new function is something like this:
Multi.error(multi, name, fn changes -> true/false end, reason)
Conditionally returns an error with {:error, reason}. Otherwise the pipeline continues.
I was thinking about Multi.fetch as well but multi aborts on {:error, _} and not :error so we wouldn't be using fetch completely inline with other parts of Elixir.

Let's say we agree there is no good way to add a Repo-like function to return {:error, nil}. Then I think the issue boils down to this:
Either this is good enough or it's too inconvenient:
Multi.new()
|> Multi.one(:one, Post)
|> Multi.merge(fn
  %{post: nil} -> Multi.new() |> Multi.error(:no_post, "no post")
  _ -> Multi.new()
end)

It is maybe a little verbose if you just want to conditionally add something. Assuming I didn't overlook anything and we agree this can be improved, maybe adding something to conditionally execute a step? Merge seems better suite to dynamically creating a step  you know you want to exist.

And you can compose in other ways: run(multi, :foo, fetch_one(query)) where fetch_one does:
def fetch_one(query)
fn repo ->
  if val = repo.one(query) do
    {:ok, val}
  else
    {:error, ...}
  end
end
end

So we probably don't need to extend the API. THanks for the PR in any case @lucas-cegatti !


Add examples for all explainers

@ruivieira
I think this ticket is solved and we have examples for each of the explainers. Wdyt?

Agree, thanks @danielezonca. Closing.


Option to add a custom label to a PR

Hi,
I wanted to offer a feature: The user should be able to set a custom label that will be attached to a PR according to the build status.
Does it make sense?

Hi I'm helping clean up some old issues. This feature is available nowadays and is documented. I'm closing it.

@benpatterson Did not find any solutions for this feature. Why close this issue?
Add custom label to PR according to the build result


Doc updates

I’ve been maintaining and restructuring docs in my fork (I can’t push to this repository and I need to be able to test how docs appear on project site after markup changes while working on visual styling and formatting). It seems like we hit the merge nightmare scenario now 😂

I’ve done a manual merge/rebase before making this PR, the docs should be in the right state in my fork, incorporating the work @opoudjis done in meantime where applicable

Yes. Clearly I need to coordinate with you better about documentation refactoring; I've just done a major exercise (https://github.com/riboseinc/metanorma-iso/issues/238) of genericising the quickstart and guidance documents, and the iso and standoc readmes. This will have downstream implications for you.

@strogonoff, should I give you write access to this repo, or should the documents under /docs move to a separate repo?

@opoudjis

I’ve just done a major exercise (#238) of genericising the quickstart and guidance documents, and the iso and standoc readmes

I’ve done a rebase based on your latest changes just before making this PR.

should I give you write access to this repo, or should the documents under /docs move to a separate repo?

I’d say either choice is OK.
It would be slightly less confusing to potential docs contributors if docs are in the same repo (in which case I’d need to have push access to it, I’ll only touch the docs).
That said Open Project sites can work if docs are in a separate repo, we can keep fresh docs in my fork, I believe I can give you push access to my fork or you can make PRs if you have corrections on docs.

Actually, I've updated asciiiso-syntax.adoc with the new content, so everything is ok.


Narrow return type of singleton_class

Motivation

I think this works:
https://sorbet.run/#%23 typed%3A true

T.let(42.singleton_class%2C T.class_of(Integer))
T.let(42.singleton_class%2C T%3A%3AClass[Integer])
T.let(42.singleton_class%2C T%3A%3AClass[String])
What I would really like is for the ~reverse operation, attached_object, to return something narrower than BasicObject, to avoid unnecessary type guards / casts.
Test plan

Existing tests 🤞

Hey, sorry, this doesn't work
https://sorbet.run/#%23 typed%3A true
extend T%3A%3ASig

sig { returns(T%3A%3AClass[T.self_type]) }
def example
    raise
end

ah, bummer


feat: add alby cloud link

Links to Alby Cloud for users who don't have an Alby Cloud hub deployment

I would like you to consider having the "Alby Cloud" section link to an external landing page that would highlights the benefits of choosing Alby Cloud for users and for us. This approach allows us to update the landing page as needed, including running A/B tests to see which version converts better, without having to modify the self-hosted Hub's UX/UI every time we want to adjust the Alby Cloud landing page recommendation/invitation.

@jpl-btc do we have such a page? shouldn't there also be something like this at getalby.com?

Not that I am aware of. And yes there should be something like that at getalby.com @bumi wanted me to check on a landing page-making tool, maybe that is related to this apparent need to make some additional landing pages for Alby Flavor updates, recommendations, and other stuff


[QUESTION] Best way to use Kysely with cloudfare workers

Hello!
I am doing a small side project, currently cloudfare workers has been migrated to an ES module system, with this change, the environment variables can only be accessed through the handler, is it correct that every time a route is called a Kysely instance is generated?
app.get('/', async (c) => {
    const kysely = new Kysely<Database>({
        dialect: new PlanetScaleDialect({
            url: c.env.DATABASE_URL
        })
    })
    //You do something with the kysely instance...
    return c.text('Hello!')
})

Hey 👋
If environment variables are only accessible this way, you don't have other options.
Also, the docs do not recommend using/mutating global state, unlike other FaaS offerings where it is considered best practice to manage connections/pools in global scope (AWS lambdas I'm looking at you).
AFAIK, the underlying planetscale driver used is making calls over fetch API and is stateless, so no real harm in instantiating many instances.


[6493] ChatGPT Plugin instructions via Search and Install Plugin broken reference errors

https://github.com/craftercms/craftercms/issues/6493

@rart I updated the README to use the url for the raw images just as we do for the plugin preview image

@sumerjabri version updated


etc/DependencyInstaller.sh is using /tmp/installers which is shared between users

Describe the bug
This causes problems because the access permissions will be different and /tmp/installers will fail for mysterious reasons.
The conflict can be between two normal users or between one user and root.
Expected Behavior
Two independent users on the same machine should be able to install OpenROAD without trouble.
Environment
latest master

To Reproduce
Run etc/DepedencyInstaller.sh and see that a folder /tmp/installers is always created
Relevant log output
No response
Screenshots
No response
Additional Context
No response

Realistically the installer should be using mktemp instead of mkdir which guarantees that the created directory is unique.

This as been fixed by https://github.com/The-OpenROAD-Project/OpenROAD/commit/6afff9846eb7b6533465cb5e3e1f0b7551b34be4.
The issue can be closed now.


Allow for 1st time login via Shibboleth

If the user tries to create an account via shibboleth we need to bring them to the profile page and allow them to fill out the other required fields (e.g. first/last name). We should auto-populate the email and the Org based on what shibboleth entity they came in from.
If its necessary for Devise to have a password, just auto-generate a secure one on the user's behalf and save it with the new user record.

not sure how to test this...maybe we can ask someone in the office who doesn't have a dmptool account yet?


Inform cargo of build script file / env var dependencies

fixes #255

Apart from the Display issue, lgtm. Thanks! :+1:

Thanks!


Add integration with Chris's Premades

This allows PAA to grab item data from CPR, and return it in a format which Plutonium can use. Note if both PAA and CPR have data for an item, then PAA's version takes priority (i.e., the two are not merged).
See: https://foundryvtt.com/packages/chris-premades
Closes #86

Here's an example of a before/after for Heat Metal
2c2
<   "name": "Heat Metal (Without)",
---
>   "name": "Heat Metal (With)",
24c24
<       "type": "object"
---
>       "type": "creature"
45c45
<     "chatFlavor": "",
---
>     "chatFlavor": "[2d8 - fire] Use a bonus action on each subsequent turn to cause this damage again",
53c53
<           "2d8",
---
>           "2d8[fire]",
61c61
<       "ability": "con",
---
>       "ability": "",
96a97,136
>     "ddbimporter": {
>       "id": 138066,
>       "definitionId": 2141,
>       "entityTypeId": 435869154,
>       "dndbeyond": {
>         "lookup": "generic",
>         "lookupName": "generic",
>         "level": null,
>         "castAtLevel": null
>       },
>       "originalName": "Heat Metal",
>       "sources": [
>         {
>           "sourceId": 1,
>           "pageNumber": null,
>           "sourceType": 2
>         },
>         {
>           "sourceId": 2,
>           "pageNumber": 250,
>           "sourceType": 1
>         }
>       ],
>       "tags": [
>         "Damage",
>         "Debuff"
>       ],
>       "version": "3.3.10",
>       "effectsApplied": true
>     },
>     "custom-character-sheet-sections": {
>       "sectionName": ""
>     },
>     "rest-recovery": {
>       "data": {
>         "recovery": {
>           "enabled": false
>         }
>       }
>     },
98c138,139
<       "effectActivation": false
---
>       "effectActivation": false,
>       "onUseMacroName": "[postActiveEffects]function.await chrisPremades.macros.heatMetal.item"
114a156,180
>     "itemacro": {
>       "macro": {
>         "name": "Heat Metal",
>         "type": "script",
>         "scope": "global",
>         "command": "",
>         "author": "PQGBBcoVSwpUhViM",
>         "_id": null,
>         "img": "icons/svg/dice-target.svg",
>         "folder": null,
>         "sort": 0,
>         "ownership": {
>           "default": 0
>         },
>         "flags": {},
>         "_stats": {
>           "systemId": null,
>           "systemVersion": null,
>           "coreVersion": null,
>           "createdTime": null,
>           "modifiedTime": null,
>           "lastModifiedBy": null
>         }
>       }
>     },

I also note that Chris' Premades have a lot of junk data in them, some of which appears will be copied over to PAA? The ddbimporter and custom-character-sheet-sections module flags, for instance, shouldn't be used by PAA at all I think.

I also note that Chris' Premades have a lot of junk data in them, some of which appears will be copied over to PAA? The ddbimporter and custom-character-sheet-sections module flags, for instance, shouldn't be used by PAA at all I think.

I kept them in to match CPR's output as closely as possible, since CPR is already outside of our module ecology and so we make no assumptions about what the user is running. That said, they are indeed junky. Wrote a quick script to see what's all in there, and these are the results


Fix sidebar for multiple editors.

Fixes the Sidebar portion of #91 so that when setting the sidebar position, only the currently selected editor's sidebar will be updated.

Thanks @dchristensen! It's released under the 0.4.31 version.


Extend CxG conversion to cover - obsm (embedding)

The obsm seems to have always the same embedding:
"X_pca",
"X_tsne_perplexity_1",
"X_tsne_perplexity_10",
"X_tsne_perplexity_15",
"X_tsne_perplexity_20",
"X_tsne_perplexity_25",
"X_tsne_perplexity_30",
"X_tsne_perplexity_35",
"X_tsne_perplexity_40",
"X_tsne_perplexity_45",
"X_tsne_perplexity_5",
"X_tsne_perplexity_50",
"X_umap_neighbors_n_neighbors_10",
"X_umap_neighbors_n_neighbors_100",
"X_umap_neighbors_n_neighbors_15",
"X_umap_neighbors_n_neighbors_20",
"X_umap_neighbors_n_neighbors_25",
"X_umap_neighbors_n_neighbors_3",
"X_umap_neighbors_n_neighbors_30",
"X_umap_neighbors_n_neighbors_5",
"X_umap_neighbors_n_neighbors_50"

Should we leave all the options or pick one of each?

Decided in the 2024-04-16 Meeting, all embedding will be available and use "X_umap_neighbors_n_neighbors_20" as default, to be added in uns.


Moving to the workingstep sidebarmenu tab does not initiate a scroll

@Roceso1337

Also for fun maybe add the scrolling feature to workplan view?

Somehow my changes to this got reverted... will re-PR


Updates

Updating an old recipe.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Disable testTaskCancel_whenStreaming_andNotSuspended test

Motivation
This test relies heavily on timing and there are scenarios where the scheduling of the various tasks and threads can make this test fail. Currently, there is not enough control in Swift Concurrency to write a non-flaky test.
Modification
This PR disables the test until we get better control over the executor.
Result
One disabled test but also less flakiness.

Do we need to disable this? Why can't we send a signal from one thread to the other to ensure they move in roughly lockstep?

Yes sadly we have to. The intention of the test is to trigger cancellation once the consuming task is suspended on the next call; however, there is currently no way to observe that observation. That's why this test had a couple of sleeps to try to get the timing right and wait for the task to suspended but this can sadly fail and we cancel before we get to suspend.

Is that assessment true? The comments seem to imply that cancelling the other task in the sleep is fine too.

Okay so actually the report of the flaky test is a bit odd since it is on a quite old NIO version and the line where the assert is trigged doesn't match with what the NIO version in play is. I will close this PR for now.


Start adding support for Mongodb Atlas database

This is a first implementation where basic unit test passes and it was tested with
some custom code. It works with cluster from M10 upward (does not work still with shared cluster).
Still have some perplexity on how to use minRelevance because I did not checked if this is a -1 1 value (cosine distance) or whatever.
This code contains also the necessary code to automatically create the Vector Indexes needed to execute queries.
Please let me know if I need to fulfill special check for the PR to be closed. Also if you think that a concrete implementation should be placed into my repository and does not find space in the official one you can tell me and reject the PR, I'll move the code in a repository of mine.

I wanted support for mongodb Atlas, I've already done a pull request but it was based on dev branch that gots deleted, so I reopened on main branch.
High level description (Approach, Design)

hi @alkampfergit thanks for the PR!
Couple of asks:

could you add a script under /tools to start Mongodb Atlas with docker (assuming that's possible). See other scripts in the folder
about correctness, I would suggest extending these tests to compare results:

https://github.com/microsoft/kernel-memory/blob/main/service/tests/Core.FunctionalTests/VectorDbComparison/TestCosineSimilarity.cs. (you can use the booleans to compare just Mongodb vs SimpleVectorDb)
https://github.com/microsoft/kernel-memory/blob/main/service/tests/Core.FunctionalTests/VectorDbComparison/TestMemoryFilters.cs

hi @alkampfergit thanks for the PR!
Couple of asks:

could you add a script under /tools to start Mongodb Atlas with docker (assuming that's possible). See other scripts in the folder
about correctness, I would suggest extending these tests to compare results:

https://github.com/microsoft/kernel-memory/blob/main/service/tests/Core.FunctionalTests/VectorDbComparison/TestCosineSimilarity.cs. (you can use the booleans to compare just Mongodb vs SimpleVectorDb)
https://github.com/microsoft/kernel-memory/blob/main/service/tests/Core.FunctionalTests/VectorDbComparison/TestMemoryFilters.cs

Thanks a lot for the instructions, I'll proceed to implement these tests to verify that everything is ok. For what regard docker I'll investigate: Atlas is a cloud only product, but it has a commandline that can start a local cluster for tests, but I really didn't tried vector capabilities.
I'll proceed to check and I'll update the PR accordingly.
Thanks!

Hi @dluc
I've implement those two tests you suggeested in the previous comment. Sadly enough MongoDb perform a normalization of the base Cosine Similarity so I needed to recalcolate in memory the correct value for the test to pass.
I've added a Markdown file explaining how to run a local Atlas MongoDB installation locally thanks to Atlas CLI. It uses podman under the hood, but it cannot work with docker. This is th easiest way to have a local running instance of AtlasDB to perform tests.
I'm waiting for comments in the PR if Any.
Thanks.

Thanks for the work @alkampfergit , any chance this could be adapted to work with Azure MongoDB for vCore?
Maybe something like a base class MongoDb, and one specialized for Atlas and one for MongodDB vCore with their specific needs?
I believe the index creation and search syntax would be different, but nothing more:
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search#use-the-createindexes-template-to-create-a-vector-index

Thanks for the work @alkampfergit , any chance this could be adapted to work with Azure MongoDB for vCore? Maybe something like a base class MongoDb, and one specialized for Atlas and one for MongodDB vCore with their specific needs? I believe the index creation and search syntax would be different, but nothing more: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search#use-the-createindexes-template-to-create-a-vector-index

Actually we are using MongoDbAtlas, I do not know MongodDB vCore, does it support Vector Search in the same way atlas does? (lucene index).

@dluc I was pretty busy these days, but I should had resolved all the issues. Feel free to request additional modification (i've left two commits, the first is the original for the PR, the other contains all modification done as for your request)

Thanks for the work @alkampfergit , any chance this could be adapted to work with Azure MongoDB for vCore? Maybe something like a base class MongoDb, and one specialized for Atlas and one for MongodDB vCore with their specific needs? I believe the index creation and search syntax would be different, but nothing more: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search#use-the-createindexes-template-to-create-a-vector-index

Actually we are using MongoDbAtlas, I do not know MongodDB vCore, does it support Vector Search in the same way atlas does? (lucene index).

Yes it does support vector search, but the index and query syntaxes are a bit different, see here: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search#use-the-createindexes-template-to-create-a-vector-index
Someone also started a MongoDB vCore pull request: https://github.com/microsoft/kernel-memory/pull/329
In the end, for maintenance, most probably it would be more interesting to have a common shared code base.

Sorry for meddling, but I have been using this and I have some feedback:

why the requirement to use Mongo v7? Most Atlas clusters (except the highest plans) cannot yet be upgraded to v7
the knnBeta search type is deprecated, it should use the normal $vectorSearch
there are some warnings from the NetAnalyzers regarding code-quality (#nullable)
why hard-coding BsonDocuments everywhere instead of using the standard .NET driver? Would make it more future-proof
the collectionName is hard coded (even the ConnectionNamePrefix cannot be changed from the default "ix") I would like to be able to configure that (we are using a naming convention within one Mongo db)
I haven't seen any error-handling - should all exceptions always bubble up? In my experience the MongoDB exceptions are not always helpful and the logging might have issues with those
it's not really a best-practice to store long strings in the Mongo _id field - this has a performance/storage impact. I also see string storage in tg___file_id[] - although it's a GUID that could be well stored in a MongoDB iuid/binary field

Just a general question - I know SemanticKernel is a different project but the connectors are very similar - couldn't we somehow share/reuse some of the codebase (or just have a wrapper?), for example the MongoDB Connector

UPDATE: I've updated the pull request as for @dviry request, for now I've addressed support for 6 and 7 (tested with local atlas insetallation), also removed knnBeta to favor $vectorSearch.
Still the code can be improved to let the user choose naming of the collection and other minor updates.
Fixed also a problem in creation of Vector Search index where a stale vector resulted as created but was not functional.

Thanks for considering my feedback!
One question before I update my local copy - am I missing something or is the knnBeta still in the latest code?
Thanks

Thanks for considering my feedback!
One question before I update my local copy - am I missing something or is the knnBeta still in the latest code?
Thanks
EDIT: sorry my mistake - the method CreatePipeline() is still there but not referenced anywhere

Removed and amended last commit, I've left old code for review and compare but it makes sense to remove to have cleaner code.

For some reason the service doesn't work on my workstation, e.g. the cosine similarity test is unable to connect.
⚡ ./run-mongodb-atlas.sh

[Default Settings]
Deployment Name   local6786
MongoDB Version   7.0
Port              27017

Creating your cluster local6786 [this might take several minutes]
1/3: Starting your local environment...
2/3: Downloading the MongoDB binaries to your local environment...
3/3: Creating your deployment local6786...
Deployment created!
Connection string: mongodb://root:root@localhost:27017/?directConnection=true

connection skipped

Deleting index test-cosinesimil in Microsoft.KernelMemory.MongoDbAtlas.MongoDbVectorMemory

System.TimeoutException
A timeout occurred after 10000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode : Primary } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", DirectConnection : "True", Type : "Standalone", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/localhost:27017" }", EndPoint: "Unspecified/localhost:27017", ReasonChanged: "Heartbeat", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
 ---> System.Net.Sockets.SocketException (61): Connection refused [::1]:27017
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

"ConnectionString": "mongodb://localhost:27017/?directConnection=true&serverSelectionTimeoutMS=10000",

I would rename these classes, to be consistent with others:

MongoDbVectorMemory => MongoDbAtlasMemoryDb
MongoDbKernelMemoryStorage => MongoDbAtlasStorage or MongoDbAtlasContentStorage

As mentioned above those commands only work starting M10 Atlas instances,
not on M0 ones... You will have to create the index manually in case you
want to continue using the M0 instance.
On Fri, Mar 8, 2024 at 2:03 PM Devis Lucato @.***>
wrote:

For some reason the service doesn't work on my workstation, e.g. the
cosine similarity test is unable to connect.
⚡ ./run-mongodb-atlas.sh
[Default Settings]
Deployment Name   local6786
MongoDB Version   7.0
Port              27017
Creating your cluster local6786 [this might take several minutes]
1/3: Starting your local environment...
2/3: Downloading the MongoDB binaries to your local environment...
3/3: Creating your deployment local6786...
Deployment created!
Connection string: @.***:27017/?directConnection=true
connection skipped
Deleting index test-cosinesimil in Microsoft.KernelMemory.MongoDbAtlas.MongoDbVectorMemory
System.TimeoutException
A timeout occurred after 10000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode : Primary } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", DirectConnection : "True", Type : "Standalone", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/localhost:27017" }", EndPoint: "Unspecified/localhost:27017", ReasonChanged: "Heartbeat", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
---> System.Net.Sockets.SocketException (61): Connection refused [::1]:27017
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
"ConnectionString":
"mongodb://localhost:27017/?directConnection=true&serverSelectionTimeoutMS=10000",
I created an account and deployed a free instance at mongodb.net
"ConnectionString": "mongodb+srv://:@
kernelmemory.cfuqvtp.mongodb.net/?retryWrites=true&w=majority&appName=kernelmemory
",
When running the cosine similarity functional test I get this error:
MongoDB.Driver.MongoCommandException
Command aggregate failed: $listSearchIndexes is not allowed or the syntax is incorrect, see the Atlas documentation for more information.
at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
Any suggestion?
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/kernel-memory/pull/311#issuecomment-1985658833,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJMLNAJHM2FMDQOCMJMG6LYXGZIRAVCNFSM6AAAAABDGBKWHCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBVGY2TQOBTGM
.
You are receiving this because you were mentioned.Message ID:
@.***>

As mentioned above those commands only work starting M10 Atlas instances, not on M0 ones... You will have to create the index manually in case you want to continue using the M0 instance.

I see, so we need to figure out how to fix the local docker instance, otherwise we won't be able to test the integration. Unless we get support from MongoDb and they could offer a M10 instance for test purposes.

As mentioned above those commands only work starting M10 Atlas instances, not on M0 ones... You will have to create the index manually in case you want to continue using the M0 instance.

I see, so we need to figure out how to fix the local docker instance, otherwise we won't be able to test the integration. Unless we get support from MongoDb and they could offer a M10 instance for test purposes. In the meantime I'll try the version available via homebrew, see if it can support the integration tests.

From what I see you are connecting in port 27017, that is the standard mongodb port, but the script to start the cluster starts in docker and expose the port in 27777 so the connection string is
mongodb://root:root@localhost:27777/?authSource=admin
I've added a comment in the file. The reason behind this is that usually, developers that works with mognodb, have a standard local instance, then when they want to test atlas, they use a different port to avoid conflict.
To simplify everything we can change the port in the sh file to use standard one.

Update: local tests via homebrew are working.
I made some changes to functional tests and renamed the "MongoDb" config key to "MongoDbAtlas".
All functional tests seem to pass fine.

Update: local tests via homebrew are working. I made some changes to functional tests and renamed the "MongoDb" config key to "MongoDbAtlas". All functional tests seem to pass fine.

Good, sorry for the inconvenience, I was using secret because I tested against a M20 of a customer ,so I really do not want to leak the connectino string and I forgot to change the default one to match that one of the script .sh.

Yes
Il dom 10 mar 2024, 19:26 Devis Lucato @.***> ha
scritto:

@.**** commented on this pull request.
In extensions/MongoDbAtlas/MongoDbAtlas/MongoDbAtlasConfig.cs
https://github.com/microsoft/kernel-memory/pull/311#discussion_r1518925331
:

///

///
///
public MongoDbAtlasConfig WithSingleCollectionForVectorSearch(bool useSingleCollectionForVectorSearch)
{

   this.UseSingleCollectionForVectorSearch = useSingleCollectionForVectorSearch;

   return this;

}

public MongoDbAtlasConfig WithAfterIndexCallback(Func afterIndexCallback)
{

   this.AfterIndexCallbackAsync = afterIndexCallback;

   return this;

}

internal IMongoDatabase GetDatabase()

the memory db object is a singleton, that should be sufficient no?
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/kernel-memory/pull/311#discussion_r1518925331,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACXREOQUWR3XQTWTQXDSR3YXSXWHAVCNFSM6AAAAABDGBKWHCVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSMRWGYYTQMZQGI
.
You are receiving this because you were mentioned.Message ID:
@.***>


add eth-dai

Add eth-dai to sushiswap moonriver

checksum failed https://github.com/beefyfinance/beefy-api/runs/4554309854?check_suite_focus=true


Keep previous lines visible?

Is it possible to keep lines already displayed visible, i.e. instead of replacing them, insert the next animated line after them?

This is currently not possible. Depending on your use case, a workaround might be to call textillate on each line with different initial delays.
$('.tlt1').textillate()

$('.tlt2').textillate({ initialDelay: 1000 })

$('.tlt3').textillate({ initialDelay: 2000 })


Request to add Acrel ADW300(electric meter)  to evcc

I want to use an electric meter in evcc. The model is Acrel ADW300. This is the official website of Acrel: https://de.acrel.de/
The English instruction manual of this meter is attached below repository:  https://github.com/912929782/File
Please Help

We need the modbus register documentation.

@912929782 looks doable. Could you provide remote access to Modbus port for<EMAIL_ADDRESS>for testing?

@andig My meter is in the LAN, so it can't provide remote access. Could you please add it according to the register document, and then I will test it.

You can setup port forwarding.

Ich hätt gesagt wenn dann richtig? Aber fair, wir können das auch vereinfachen. Danke!