Replace innerRequestHandler with rootDataObjectRequestHandler in request-handler

Replace innerRequestHandler with rootDataStoreRequestHandler.
The files affected:

examples\data-objects\codemirror\src\index.ts
examples\data-objects\key-value-cache\src\index.ts
examples\data-objects\prosemirror\src\index.ts
examples\data-objects\scribe\src\scribe.ts
examples\data-objects\shared-text\src\index.ts
examples\data-objects\smde\src\index.ts
packages\framework\aqueduct\src\container-runtime-factories\baseContainerRuntimeFactory.ts
packages\framework\aqueduct\src\container-runtime-factories\containerRuntimeFactoryWithDefaultDataStore.ts
packages\framework\data-object-base\src\runtimeFactory.ts
packages\framework\request-handler\src\requestHandlers.ts
packages\framework\request-handler\src\test\requestHandlers.spec.ts
packages\test\test-utils\src\testContainerRuntimeFactory.ts

There is some compat risk here I think, but is probably manageable.  In particular, I'm noting that rootDataStoreRequestHandler, since it uses getRootDataStore will end up asserting here if the request is not for a data store that was originally created as a root.
Newer documents aren't likely to hit this, but old documents that were created before the concept of a "root data store" existed will throw for "root-like" data stores, resulting in a 404 response where before the data store was successfully retrieved.
Luckily this is easy for customers to mitigate by passing an equivalent request handler to the base runtime factory classes themselves, so I think it should be sufficient to just include guidance in breaking change notes.  So my recommendation would probably be:

For the base runtime factory classes (BaseContainerRuntimeFactory, ContainerRuntimeFactoryWithDefaultDataStore, and RuntimeFactory), simply remove the innerRequestHandler and do not replace it with anything.

IMO this was always a mistake, as it's not at all likely that a container should publicly expose all of its root data stores by default.

Advise in BREAKING.md that if container authors want to permit requests for specific data stores, they can add rootDataStoreRequestHandler to the list of request handlers they pass to the runtime factory.  This will restore the ability to request root data stores.

Further warn that this will only work for root data stores, and if they require support for requesting non-root data stores and/or for legacy document support they can create a request handler of their own that calls runtime.IFluidHandleContext.resolveHandle(request) just like innerRequestHandler used to do.

@vladsud who might have further thoughts.

@ChumpChief by legacy documents you are referring to docs whose snapshots never had isRootDataStore attribute?
Interestingly, we adapt these [here] (https://github.com/microsoft/FluidFramework/blob/main/packages/runtime/container-runtime/src/dataStoreContext.ts#L773) so we probably wouldn't hit assertion for legacy docs, but agree about being explicit in the guidance about breaking changes: using new API is safe only root data stores.

Also resolveHandle within container runtime does have logic targeting paths beginning with _channels and _blobs. Any considerations there ? innerRequestHandler  -> rootDataStoreRequestHandler, does not look like 1:1 replacement even if you put root data store differences on the side.

Yeah, isRootDataStore is what I was referring to - I missed that adaptation so maybe that's totally fine.
I'm not really worried about colliding on the names _channels and _blobs, I don't think this is likely to be a threat in practice.


[Bug][Autoscaler] KubeRay autoscaler downscales the Ray cluster while the node is running

Search before asking

[X] I searched the issues and found no similar issues.

KubeRay Component
ray-operator
What happened + What you expected to happen
https://github.com/ray-project/ray/issues/34019
Reproduction script
https://github.com/ray-project/ray/issues/34019
Anything else
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

just a FYI,
that v5.0.2 bug maybe the same root case as this: (pod double killed, with half of them heathy node)
https://github.com/ray-project/kuberay/issues/1192

Update
At first, our assumption for this issue is:

Case 1:

Autoscaler sends a request to GCS to terminate the raylet process on the worker Pod.
The worker Pod restarts if the Pod’s restartPolicy is not Never. It is still the same Pod after it restarts, but the Pod IP will be different. Therefore, for Ray, it is considered a different node.
Ray schedules tasks/actors on the “new” Ray node.
KubeRay deletes the Pod.

Case 2: KubeRay deletes the Pod before Autoscaler drains the Ray node.

Case 1 discussion
Case 1 is a more reasonable reason for me. However, when I tried to reproduce it, I found the following facts:

The restartPolicy will not be triggered when the raylet is killed. That is, the Pod's status will still be "Running". The restartPolicy will be triggered only when the ray start process is killed.
The other possibility to trigger the restartPolicy is setting healthiness/readiness probes. However, the user does not set the values.

Hence, Case 1 is almost impossible.
Case 2 discussion
Case 2 seems to be impossible because Autoscaler will only send the patch to update the RayCluster CR after it drains the Ray node (i.e. kill the raylet process). cc @rickyyx @DmitriGekhtman to confirm.
Some edge cases that I can think of

Case 3: Similar to Case 1, the only difference is that certain accidents occur, leading to the termination of the ray start process. Subsequently, the Pod restarts, and Ray schedules new tasks/actors before being terminated by KubeRay.

Case 4: A Pod crashes accidentally and restarts. We assume that there is a Pod status transition (e.g. Running -> Failed -> Running) between the accident and restart (This is my assumption. I have not observed this behavior.).
There's a slim chance that KubeRay could catch the Failed status and, as a result, create a new Pod. However, when the original Pod returns to a Running status, this will result in one more Pod than expected. To rectify this, KubeRay will randomly delete one of the Pods.

In my view, both Case 3 and Case 4 are highly unlikely events. My current thoughts are:
(1) Users can set their restartPolicy to Never if they enable Autoscaler in their RayClusters.
(2) In a RayCluster with autoscaling enabled, we will forgo the random Pod deletions and allow the Autoscaler to make all Pod deletion decisions.

Hmm so if Case 1 never triggers, then restartPolicy will not help?
Seems we still don't know the root cause?

the restartPolicy will be triggered only when the ray start process is killed.

Wdym by this? Does kuberay do ray start --block?

Hmm so if Case 1 never triggers, then restartPolicy will not help?

Setting restartPolicy to Never can still be beneficial for Case 3, although I anticipate that Case 3 is a rare occurrence.

Seems we still don't know the root cause?

Yes, the information provided by users isn't sufficient to determine whether this issue is related to Autoscaler or KubeRay. As a result, I will draft a PR to ensure Autoscaler is the sole decision-maker for Pod deletion, and disable KubeRay's random Pod deletion mechanism, which is unfavorable from any perspective and ask users to set the restartPolicy to Never. If we can still observe the issue, we may need to check whether it is related to Autoscaler or Ray scheduler or not.

Wdym by this?

Does kuberay do ray start --block?

Yes. The following screenshot is the command and args of a worker's Ray container.

I will draft a PR to ensure Autoscaler is the sole decision-maker for Pod deletion, and disable KubeRay's random Pod deletion mechanism, which is unfavorable from any perspective and ask users to set the restartPolicy to Never. If we can still observe the issue, we may need to check whether it is related to Autoscaler or Ray scheduler or not.

This sounds good to me!

Close this issue because there have been no complaints about it in the last 5 months.


Make Errors Sync

Basically I want to pass Errors as a prop into render_app_to_stream(), and I can't do that unless it's thread safe

Closing in favor of #372


Instructions on the Getting Started page is mostly outdated for Operator SDK v1.x

Since the Operator SDK v1.x has released, it will be great help if you could update the Getting Started page  based on this version. The current information in this page is mostly outdated for this version, the project creation itself will fail.
operator-sdk new memcached-operator

There is no new command with this version, for example this is what a user gets.
operator-sdk new memcached-operator
Error: unknown command "new" for "operator-sdk"
Run 'operator-sdk --help' for usage.
FATA[0000] unknown command "new" for "operator-sdk"

v1.x: operator-sdk CLI
v1.x: Quickstart guid
There is a Quick start guide for the release version, look like it also has incomplete instruction, it give me error during make install phase. Operator SDK is a grate tool, without right set of information it will be difficult for a new users like me to make use of it. Thanks a lot for the grate work.

make install
-- -- -- -- --
-- -- -- -- --
/work/dev/go/proj1/bin/kustomize build config/crd | kubectl apply -f -
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
The CustomResourceDefinition "memcacheds.cache.example.com" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
make: *** [install] Error 1

FYI: The SDK that I have installed for studying the concept is
operator-sdk  version
operator-sdk version: "v1.0.0", commit: "d7d5e0cd6cf5468bb66e0849f08fda5bf557f4fa", kubernetes version: "v1.18.2", go version: "go1.13.11 linux/amd64", GOOS: "linux", GOARCH: "amd64"

Thank you for your input, however, the plan is deprecated and archive this repo.
A lot of improvements as been planned as part of the milestone  Post 1.0 docs.
Duplicated of https://github.com/operator-framework/operator-sdk/issues/3741

I also have this error when using the operator sdk 1.1 quickstart:
make install
/Users/ariv/.go/bin/kustomize build config/crd | kubectl apply -f -
The CustomResourceDefinition "memcacheds.cache.example.com" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
make: *** [install] Error 1

I'm using operator-sdk version:
operator-sdk version: "v1.1.0", commit: "9d27e224efac78fcc9354ece4e43a50eb30ea968", kubernetes version: "v1.18.2", go version: "go1.15.2 darwin/amd64", GOOS: "darwin", GOARCH: "amd64"


Added TAG deputy co-chair

Please check name of Deputy co-chair @baskaufs -- Thanks!

works for me


[Snyk] Upgrade postcss from 8.3.0 to 8.4.31

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade postcss from 8.3.0 to 8.4.31.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

The recommended version is 43 versions ahead of your current version.
The recommended version was released 2 months ago, on 2023-09-28.

The recommended version fixes:

Severity
Issue
PriorityScore (*)
Exploit Maturity

Information Exposure SNYK-JS-NANOID-2332193
307/1000  Why? Proof of Concept exploit, CVSS 4
Proof of Concept

(*) Note that the real score may have changed since the PR was raised.

Release notes

Package name: postcss

8.4.31 - 2023-09-28
Fixed \r parsing to fix CVE-2023-44270.

8.4.30 - 2023-09-18
Improved source map performance (by @ romainmenke).

8.4.29 - 2023-08-29
Fixed Node#source.offset (by @ idoros).
Fixed docs (by @ coliff).

8.4.28 - 2023-08-15
Fixed Root.source.end for better source map (by @ romainmenke).
Fixed Result.root types when process() has no parser.

8.4.27 - 2023-07-21
Fixed Container clone methods types.

8.4.26 - 2023-07-13
Fixed clone methods types.

8.4.25 - 2023-07-06
Improve stringify performance (by @ romainmenke).
Fixed docs (by @ vikaskaliramna07).

8.4.24 - 2023-05-28
Fixed Plugin types.

8.4.23 - 2023-04-19
Fixed warnings in TypeDoc.

8.4.22 - 2023-04-16
Fixed TypeScript support with node16 (by @ remcohaszing).

8.4.21 - 2023-01-06

8.4.20 - 2022-12-11

8.4.19 - 2022-11-10

8.4.18 - 2022-10-12

8.4.17 - 2022-09-30

8.4.16 - 2022-08-06

8.4.15 - 2022-08-06

8.4.14 - 2022-05-18

8.4.13 - 2022-04-30

8.4.12 - 2022-03-16

8.4.11 - 2022-03-15

8.4.10 - 2022-03-15

8.4.9 - 2022-03-15

8.4.8 - 2022-03-07

8.4.7 - 2022-02-24

8.4.6 - 2022-02-01

8.4.5 - 2021-12-13

8.4.4 - 2021-11-27

8.4.3 - 2021-11-26

8.4.2 - 2021-11-26

8.4.1 - 2021-11-24

8.4.0 - 2021-11-24

8.3.11 - 2021-10-21

8.3.10 - 2021-10-20

8.3.9 - 2021-10-04

8.3.8 - 2021-09-25

8.3.7 - 2021-09-22

8.3.6 - 2021-07-21

8.3.5 - 2021-06-17

8.3.4 - 2021-06-14

8.3.3 - 2021-06-14

8.3.2 - 2021-06-11

8.3.1 - 2021-06-09

8.3.0 - 2021-05-21

    from postcss GitHub release notes

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:
🧐 View latest project report
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

This is handled in https://github.com/City-of-Helsinki/drupal-kaupunkitieto/pull/25


Error '`RubyGems' were not loaded' occured

Hi, Thanks for the great demo and sample code!
When I tried a sample from this repository, I faced the following errors.
(GET request to http://<IP_ADDRESS>:7878  using curl)
$ viceroy ./tmp/sample.wasm
Sep 27 21:07:41.065  WARN no configuration provided, invoke with `-C <TOML_FILE>` to provide a configuration
Sep 27 21:07:41.066  INFO Listening on http://<IP_ADDRESS>:7878
Sep 27 21:07:46.290  INFO request{id=0}: handling request GET http://<IP_ADDRESS>:7878/
`RubyGems' were not loaded.
`error_highlight' was not loaded.
`did_you_mean' was not loaded.
`syntax_suggest' was not loaded.
Sep 27 21:07:46.372  INFO request{id=0}: request completed using 107.2 MB of WebAssembly heap
Sep 27 21:07:46.372  INFO request{id=0}: request completed in 82ms

runtime environment is as follows.
$ sw_vers
ProductName:	macOS
ProductVersion:	12.4
BuildVersion:	21F79

$ sysctl machdep.cpu.brand_string
machdep.cpu.brand_string: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz

$ ruby -v
ruby 3.2.0preview2 (2022-09-09 master 35cfc9a3bb) [x86_64-darwin21]

Do you have any idea to resolve this error?

Sorry for your confusion. Could you try it again after installing wasi-preset-args? I missed it in readme instruction.

Thanks! I'll try it.


Guidance on how to include angle brackets in code examples in bikeshed

For C++, we'd like to be able to use angle brackets in inline and blockquoted code. E.g.
I'd like to be able to talk about vector<> and talk about:
vector<int> a(10);
a.push_back();

Ideally, we'd like to not have to use HTML escapes. What would be the best way for us to do this?
ATM in bikeshed we're doing vector<>.

Note: you never need to escape > in text (only in unquoted attribute values).
You can use the deprecated HTML element <xmp> for code blocks; it's styled like a
 but interprets everything until the closing </xmp> as literal text.  Bikeshed automatically rewrites it into a
 before output, and the serializer takes care of escaping things for you.
I don't have an inline variant, but I'd be up for adding one.  At minimum, at some point I'm going to add in support for Markdown's `literal` syntax, which'll allow it.  I can probably do, like, an <ixmp> fake element for now that gets rewritten into a  with its contents escaped.
(Also, note that if all your /
 is C++ code, you can put the highlight attribute on some ancestor element, so you don't have to spam it into every code sample.)

Ha the highlight trick is nice.
@brycelelbach was trying to get an inline element, so code quote support would be very nice :D

Oh, interesting, I'd forgotten that I do have support for Markdown code spans.  So you can do that, I suppose.
Like:
Create a `vector<int>` and frob it.

This doesn't let you set the highlight language directly, so you'll have to put it on an ancestor.  (Or wrap that code in a , but that shouldn't be necessary.)
You just have to turn on inline markdown support with a Markup Shorthands: markdown yes metadata.

The following doesn't seem to work at the moment:

Title: Foo
Shortname: bar
Status: FINDING
Group: WG21
URL: http://example.com
Editor: JF Bastien, Google<EMAIL_ADDRESS>Abstract: Foo
Date: 2016-07-12
Markup Shorthands: markdown yes

Title
=====

Here be example:

\```
  std::atomic<int> x;
  std::atomic<int> y;
\```

Highlight isn't working, and it transforms the triple-quote to
 sections instead of
 blocks (so linebreaks get broken).

Correct, I don't (yet) support Markdown-style code blocks, so that's instead being recognized as an inline code span starting a paragraph.


add migration for foreign key constraint to clean up validations table when results are deleted

https://www.pivotaltracker.com/story/show/160106899
Tested these locally FWIW, both as migrations (at least, the up migration) and manually using psql client directly (both up and down migrations).

@tank157 Sorry this took me so long to respond to!
So you were right to be concerned, this migration will fail if we have a row with a non-existent results_id:
dataprocessor-# ;
 results_id | severity | scope | identifier | error_type | error_data
------------+----------+-------+------------+------------+------------
          1 | whatevs  | yeah  |          1 | okay       | something
        100 | whatevs  | yeah  |        100 | okay       | something
(2 rows)

dataprocessor=# \d validations
                      Table "public.validations"
   Column   |          Type          | Collation | Nullable | Default
------------+------------------------+-----------+----------+---------
 results_id | integer                |           |          |
 severity   | character varying(255) |           |          |
 scope      | character varying(255) |           |          |
 identifier | bigint                 |           |          |
 error_type | character varying(255) |           |          |
 error_data | text                   |           |          |
Indexes:
    "validations_result_scope_id_idx" btree (results_id, scope, identifier)
    "validations_result_scope_type_idx" btree (results_id, scope, error_type)
    "validations_results_id_severity_idx" btree (results_id, severity)
    "validations_results_scoped_severity_idx" btree (results_id, severity) WHERE severity::text = 'critical'::text OR severity::text = 'fatal'::text

dataprocessor=# alter table validations drop constraint validations_results_id_fkey;
ERROR:  constraint "validations_results_id_fkey" of relation "validations" does not exist
dataprocessor=# alter table validations add foreign key (results_id) references results(id) on delete cascade;
ERROR:  insert or update on table "validations" violates foreign key constraint "validations_results_id_fkey"
DETAIL:  Key (results_id)=(100) is not present in table "results".
dataprocessor=#

What I propose is that I go ahead and merge this in after I've done a cleanup of the validations table again on a weekend, and that way we can make sure this does the right thing. Does that seem reasonable?

Yup, that seems reasonable! I'm glad past experience me had some insight into the ways these things can fail with a live db!


Except functionality

I want to send message to all authorized users
I want to send message to specified user except some connection ids

I mean, it will be excellent to add except clients to all clients filter, we have AllExcept() but can't add except to groups and other filters
Like this
T Group(string groupName, IReadOnlyList excludedIds = null);
T User(string userId, IReadOnlyList excludedIds = null);

User(string userId) is to a single user so an except list doesn't make sense.
Groups one is questionable, since you could manage your groups differently so the exclude list is automatically there.

Right the goal is to provide very basic building blocks so you can do more complex groupings using add/remove to group.

Look. Client User-X connected to hub and I want to inform all other connected clients of User-X except connected user

public override Task OnConnectedAsync()         {             Clients.Client(Context.ConnectionId).InvokeAsync($"Hello {Context.User.Identity.Name}, you are connected!");             Clients.User(Context.User.Identity.Name).InvokeAsync($"You are connected from another device!", except: Context.ConnectionId);             return base.OnConnectedAsync();         }

@Gurgen This is not possible yes, but eventually, I think you should be able to find the connection ids of the other connected clients to send the second message, so the code could look like this:
public override async Task OnConnectedAsync()
{
    await Clients.Client(Context.ConnectionId).InvokeAsync($"Hello {Context.User.Identity.Name}, you are connected!");

    var otherClients = Connections.Where(c => c.User.Identity.Name == Context.User.Identity.Name
        && c.ConnectionId != Context.ConnectionId).Select(c => c.ConnectionId);

    await Clients.Clients(otherClients).InvokeAsync($"You are connected from another device!");
    await base.OnConnectedAsync();
}

I discussed an API for that here #683.
And note that this code would have to be written very differently if you need to scale out...

OK, I understand that we can implement a Base class derived from Hub and implement there functionality as we want. I mean with this issue/improvement will be more simple working with hubs if we have except list of connections to every sub collection of connections.

Also I created library to work with .net core signal r hubs from android, if you interested.
The repository is here, and at the moment supports only connections via websockets
https://github.com/Gurgen/SignalR-.net-core-android-client

https://github.com/Gurgen/SignalR/commit/84ed23c83bf587aaccc6118701c9082b28fc54fd
Please look at this commit

I implemented InvokeFilter with exclude feature and not only.

We periodically close 'discussion' issues that have not been updated in a long period of time.
We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.


Testing all packages in a melos project

Hi,
I have a melos project with a number of packages, and I like to watch all of them at once.
What's the best way to go about this? I currently have a melos script execute spec --watch in each package, and this works pretty well, with the limitation that I'm not able to interact with any of the watchers.
Is there a way I can watch all the packages in a melos project while retaining the interactive functionality offered by spec?
Thanks

You shouldn't need to do anything. Spec already integrates with melos
No need to run "spec --watch" on each packages. Simply run it once above all packages

Thanks, I had ran into a bit of an edge-case.
My melos.yaml config had config data for a new melos feature (ide/intellij/moduleNamePrefix), and the melos instance within the spec cli was choking on this yaml.
spec swallowed this errors, so it was difficult to know that there was a problem at all.
Thanks @rrousselGit

Actually, I'm don't think that was the issue, I'm failing to reproduce that in isolation.


Cuda 12 Fix [opencv_contrib]

Emgucv does not compile on Cuda 12 because it refers to objects removed from the toolkit (see here).
The official base branch contains all the fixes.
I can confirm that the current 5.0 official version (native) compiles and works well in Cuda 12. Not sure if a merge will be enough for Emgucv or will requires some minor changes.

Our opencv_contrib branch "emgucv_5.x" is only 4 commits behind the official opencv 5.x branch, see the history here:
https://github.com/emgucv/opencv_contrib/commits/emgucv_5.x
It contains opencv contrib commits up to Dec 12, 2022 (there are only 4 commits after that at this moment)
It shouldn't have 163 commit to merge.

Merged with the lastest opencv_contrib 5.x branch from our side. Closing this pull request


genconfig: move to python 3 and support dc

This is patch changes the genconfig to use python3 and add datacenter
support.
There are two ways of configure datacenter:

from the command line using dc:ip,ip,ip
for example:
genconfig.py dc1:<IP_ADDRESS>,<IP_ADDRESS>

from file using nodetool status output
for example
genconfig.py -ns f.txt

where f.txt holds the nodetool status output
Fixes #513
Signed-off-by: Amnon Heiman<EMAIL_ADDRESS>

@eyalgutkind @roydahan please validate and comment

@amnonh, does it change the current command line parameters?
@bentsi FYI

@amnonh, does it change the current command line parameters?
@bentsi FYI

It does, the current command line will no longer work

I suggest to break the Python 3 change from the script change

Replaced by #562


Research Software versioning and Implement

We want to standardize our software cycles with versioning
Implement a versioning system that archives old versions and releases new versions using GitHub and poetry (on github)

Updated due date from Jan 17, 2024 to Jan 24, 2024


Caching

Improve caching mechanism and example code

sorry for chaotic pull request (the list of changes in code scares me)


Login-AzureRmAccount : The term 'Connect-AzureRmAccount' is not recognized

Powershell installed via brew
and installed Azure Powershell per https://docs.microsoft.com/en-us/powershell/azure/install-azurermps-maclinux?view=azurermps-5.2.0
When I tried to login, I get the following error.
Login-AzureRmAccount : The term 'Connect-AzureRmAccount' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1

Login-AzureRmAccount

CategoryInfo          : ObjectNotFound: (Connect-AzureRmAccount:String) [], CommandNotFoundException
FullyQualifiedErrorId : CommandNotFoundException

Hi @vtjc2002 - the best place to get help on this error is via the src repo where the devs keep code and content.  Please copy and paste your open question to:  https://github.com/Azure/azure-powershell/issues and do let me know if you don't get the help you need.  Thanks!


[Action, Cloud Native Sustainability Week] Create a brochure for local Meetup organisers

Part of https://github.com/cncf/tag-env-sustainability/issues/95
@sysnet4admin proposed the idea of creating a brochure about the cloud native sustainability week, which local meetup organizers can share with their group. I expect that not every meetup will leverage from this document but, for some, this will help a lot.
Brochure: https://docs.google.com/document/d/1bGRjOhbjKiXbVdtHxxBLOtrvGOUQXRG9jMuZycDq7zM/edit?usp=sharing

The brochure template needs to be updated. Anyone is welcome to suggest changes! Thank you :)

It looks like we are not pursuing this as we have not received any further requests to create these resources.


Added imutability for actor

Coverage decreased (-1.01%) to 69.06% when pulling 5db41d17694b2f30766c2ec6647d85d77b9375eb on add-imutability-actor into da1d2ac180062a7e1c5ed60ff52157d0265d24b5 on master.


[PROJECT] [SCOPING] Remove Express Custom Server and Move to NextJS Default Server

Overview
This is to discuss and scope the migration from the Express server that we currently use in the front-end to the NextJS default server.
Benefits

Server-less Functions: Pages that use SSR and API routes will automatically become serverless functions; As per NextJS documentation,

Pages that use Server-Side Rendering and API routes will automatically become isolated Serverless Functions. This allows page rendering and API requests to scale infinitely.

Automatic Static Optimization: NextJS claims that Automatic Static Optimization will make page loading ultra-fast.

One of the main benefits of this feature is that optimized pages require no server-side computation, and can be instantly streamed to the end-user from multiple CDN locations. The result is an ultra fast loading experience for your users.

I feel that if the docs are true to their word, the second point might be interesting to consider. The docs also mentions that the determination whether a page should be statically optimized is based on whether getServerSideProps and getInitialProps are absent in the page. I did a count and out of the ~600 components around ~50 components have these functions. And as such, we might be able to leverage some gains in page loading.
Downsides
~- We might be stuck with Vercel if we were to reap all the benefits described above. Note that those two benefits seems only to be guaranteed if we deploy on Vercel. There are some problems if we use something other than Vercel. For example note the discussion; vercel/next.js#17563.~ (as per https://github.com/opencollective/opencollective/issues/4439#issuecomment-876968699)
~- This might involve some significant effort I feel; especially since we are using lot of different express middleware. The effort itself isn't a problem but if we decide in the future to not use Vercel then we will be giving up on these performance enhancements.~  (as per https://github.com/opencollective/opencollective/issues/4439#issuecomment-876968699)
Owner
@SudharakaP
Note: Project owner responsibilities include coordinating all the different people that need to contribute to the project, defining specs and goals, communicating project progress, being the point person for decisions, QA, documentation, and a plan for ongoing maintenance. If the project owner can't continue, they need to hand the role over to another person explicitly.
Secondary Developer on project:
TBD
Prioritization

Does this serve our mission?

Done correctly this project has the potential to make our front-end pages load faster (due to static optimizations described above) and also to scale better (due to the server-less function support), and as such will have an impact on fulfilling our mission.

Will this help to make us financially sustainable within a year?

No. This is a improvement to the codebase but I don't see how it directly make us financially sustainable.

Can this make us move faster?

No.

Does this increase quality?

Yes, this will increase quality of the codebase plus add some value to the overall performance of the front-end.

Does this add technical debt?

My understanding is that this project is more towards reducing technical debt. We are slowly moving towards default NextJS standards starting from https://github.com/opencollective/opencollective/issues/3791

Is it feasible to build and maintain with our current team and runway?

Yes. Since this is not a new feature or addition but rather a refactoring of the same thing I don't see why not.

What insight or research is this project based on?

The reasoning is mainly based on the documentation of NextJS. For example it clearly states that using a custom server loses important performance optimizations;

Before deciding to use a custom server please keep in mind that it should only be used when the integrated router of Next.js can't meet your app requirements. A custom server will remove important performance optimizations, like serverless functions and Automatic Static Optimization.

Is this problem already solved elsewhere?

Yes it seems that there's instructions on migrating from an express server to NextJS; https://dev.to/hunterbecton/middleware-in-next-js-moving-from-express-1bmf and https://hoangvvo.com/blog/migrate-expressjs-to-nextjs

How will we measure our success? Do we have a baseline today that we can compare this against?

Page loading speeds
Time to deploy
Impact on our deployment process
Server uptime (SLA)
Resistance to huge traffic / DDOS attacks
Security (what are the new treats - if any - of this new deployment process?)

How well do we understand the complexity of the problem, feature to be added or user need to be addressed?

The problem is well understood but the technical details needs to be investigated and discussed more to see if we covered all major technical requirements.

What is the estimated timeframe for delivering this project?

TBD
Kick-off meeting
TBD
Roadmap & Timeframe
(Key milestones or phases and estimated dates. Consider including further scoping, discovery, and prototyping. Include as a minimum: planning, design, development, documentation, feedback, iteration and, marketing)
TBD
Success metrics
(What we will judge outcomes of the project by)
Page loading speed/metrics will be a good indicator.
Release planning
Estimated delivery date

Which aspect(s) of the platform does this change impact?
Which group(s) of users will benefit from this change?
Is this project in response to a user-requested feature? If so who requested it (link to feature request if available)?
Is this project related to any similar work we have done, or are planning on doing in the next three months?

Next steps

[ ] Open up any appropriate project issues
[ ] Open a documentation issue, link it to this one
[ ] Open a release issue, link it to this one
[ ] Create a project board, add this issue to the description, add any issues created above

Project Closure

Close all relevant project issues
Open up issues for improvements
Post comm's according to comm's plan (most likely in release issue)
Post project screencast on documentation slack channel
Post project closure in newsletter slack channel
Celebrate!!

I would rename the issue to something like: [PROJECT] [SCOPING] ... or anything that makes it clears this is about scoping, not implementing.

We might be stuck with Vercel if we were to reap all the benefits described above. Note that those two benefits seems only to be guaranteed if we deploy on Vercel. There are some problems if we use something other than Vercel. For example note the discussion; Automatic Static Optimization on custom server vercel/next.js#17563.

I still don't agree with this part. Static asset optimization is not for Vercel only. https://github.com/vercel/next.js/discussions/17563 is about the feature not being available for custom servers (custom express servers), not for other hosting services.
As far as I know, the only homemade optimization they may have for Vercel is about images. Netlify fully supports NextJS projects, so no really I don't think that's true and it should not be listed as a downside.

This might involve some significant effort I feel; especially since we are using lot of different express middleware.

The efforts needed are not really in this project since this is about scoping and not implementing.

if we decide in the future to not use Vercel then we will be giving up on these performance enhancements.

Moving to NextJS default server might bring performances improvements that are unrelated to Vercel. Getting rid of our custom Express server allows us to move to Vercel, but it doesn't mean we have to move to Vercel.

How will we measure our success? Do we have a baseline today that we can compare this against?
The main measure of success is page loading speeds currently and after migration.

Loading time is one important metric, but there are others that this project needs to surface:

Time to deploy
Impact on our deployment process
Server uptime (SLA)
Resistance to huge traffic / DDOS attacks
Security (what are the new treats - if any - of this new deployment process?)

@Betree : Thanks much for the input. 😸  I've modified the original post as per your suggestions; agree that the two downsides are not applicable. Let me think a bit more tomorrow on what other potential downsides we might have. 🤔

I would love to see this issue prioritized in the near future. It can unlock a whole universe of performance improvements, and it's the natural evolution NextJs have followed for the past years. I wouldn't be surprised if they decide to stop supporting custom servers at some point.
About performances:

Another issue that we introduced recently: PDF.JS doesn't seem to work on Vercel as it is configured now, we get a Failed to load PDF file error.


Initial stab at supporting systems that do not use 'python' as the Python 3 command

yes we are backwards :(
Ubuntu currently still uses python for Python 2, while other distributions (e.g., Arch) use python for Python 3 and python2 for Python 2.
This is a quick fix that uses the platform module to derive information about the system, specifically if it's an Ubuntu system in which case we know how to handle specifying version.

Hi, I can confirm that even on Arch there is a python3 symlink. I believe that
python3 is nowadays available on most distributions, so there is no need
for the command map as we can just replace python with python3.

@mirekys great, I like that way better than the admittedly hacked/ugly bit of code I tossed in to handle the translation. I'll update the pull request.

Thank you, looks good to me now, merging this. Happy new year :)


Charge indicators on atom names seem to have no effect.

The documentation at http://openmopac.net/manual/index.html says the following:
Charges on individual atoms for use in MOZYME can be specified by the symbols "+", "0", and "-" in the atom labels. When the symbol "+" is present, the atom is given a unit positive charge, when "0" is present, the atom will be given a zero charge, and when "-" is present, the atom is given a unit negative charge.  The zero charge option is useful when metal ions in enzymes are being modeled, thus the MoO2 moiety should have MoIV, but by default, Mo is MoVI, i.e., [MoO2 ]2+. To specify that MoIV should be used, after Mo add a "0" to give Mo0. Avoid using these keywords unless necessary.  Thus if the ethyl group, C2H5, is run with [CHARGE=1](http://openmopac.net/manual/charge.html), then the CH2 carbon will automatically be given a positive charge, if CHARGE=-1 is present, the it will be given a negative charge.
However, adding a zero to the Zn atoms in a MOF seems to have no effect. However, the METAL(Zn) keyword does work. Nonetheless using atom names could be useful for e.g. 4-coordinate oxygen atoms where most oxygen atoms are fine, but one or two need tweaking.
The attached tar file has 3 inputs and outputs illustrating the problem. The simple output has no special handling and fails due to charges on the Zn atoms. The files labeled _metal use the METAL keyword, and succeed. That using _atomnames adds the zero charge to the Zn atom names. And fails.
mof.tar.gz

After some examination, it does not appear that the atomic charge labels function as advertised at http://openmopac.net/manual/Labels.html, but they do work for select elements as discussed at http://openmopac.net/manual/mozyme_example_charges.html.
There were some obvious input parsing errors associated with this feature, and I've repaired those in a test branch, but the underlying logic still isn't functioning as intended. It doesn't seem like this feature was fully implemented. Using the METAL keyword with atomic indices does seem to be working, and so ideally I could pair its logic up with the atomic charge labels. However, I presently do not fully understand how METAL is implemented - I can identify where it is ionizing specific elements, but I cannot find where it is ionizing specific atoms. It feels a bit like an obfuscated code challenge, and I can't fix this until my understanding of how MOPAC assigns Lewis structures improves. The actual fix might be quite simple, but it will require this higher level of understanding to proceed.

I've now discussed this matter in more detail with Jimmy Stewart. He has defended the status quo that the METAL and CVB keywords are sufficient for fine-tuning MOZYME's Lewis structure assignment, and my present inclination is to remove the discussion of this incompletely implemented feature from the online manual. There is no fine-grained logic for assigning atomic charge states in the code to connect up with a general-purpose atomic charge label - it either allows an atom to participate in the sigma-bond network or not (as controlled by the METAL keyword), and then it makes pi-bond assignments in a second pass (as controlled by the CVB and SETPI keywords and the extremely limited atomic charge labeling that is presently implemented) before assigning local atomic charges to complete its accounting of electrons. Thus, while there is an assignment of local atomic charges, it is only performed at the end and is not capable of constraining the sigma and pi bonding assignments. The present input syntax for adjusting the Lewis structure is a bit clunky, but it is well-matched to what the code is actually capable of doing.

As long as the METAL keyword works with atom numbers so one can pick out specific atoms, that would be OK. A bit painful, but OK. It seems possible that one might one specific atoms, not all of an element, e.g. “O”.
Paul
3793 Fairway View Drive
Riner VA 24149-2247
USA
@.***
+1.505.603.8182

On Jun 21, 2022, at 12:20 PM, Jonathan E. Moussa @.***> wrote:
I've now discussed this matter in more detail with Jimmy Stewart. He has defended the status quo that the METAL and CVB keywords are sufficient for fine-tuning MOZYME's Lewis structure assignment, and my present inclination is to remove the discussion of this incompletely implemented feature from the online manual. There is no fine-grained logic for assigning atomic charge states in the code to connect up with a general-purpose atomic charge label - it either allows an atom to participate in the sigma-bond network or not (as controlled by the METAL keyword), and then it makes pi-bond assignments in a second pass (as controlled by the CVB and SETPI keywords and the extremely limited atomic charge labeling that is presently implemented) before assigning local atomic charges to complete its accounting of electrons. Thus, while there is an assignment of local atomic charges, it is only performed at the end and is not capable of constraining the sigma and pi bonding assignments. The present input syntax for adjusting the Lewis structure is a bit clunky, but it is well-matched to what the code is actually capable of doing.
—
Reply to this email directly, view it on GitHub https://github.com/openmopac/mopac/issues/40#issuecomment-1161974638, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRFRRA6HGFYRKIZMVOGOC3VQHTT3ANCNFSM5V4K4JGA.
You are receiving this because you authored the thread.

Yes, the METAL keyword can either be used globally (when specified without any arguments), or it can be specified with a list of elements and specific atoms.
If I were to further hijack the atom labels for the purpose of Lewis structure adjustments (and I'm willing to entertain that possibility), the natural set of information, in order of priority, would be:

Sigma bonding radius of the labeled atom (this is internally set to a negative value for fully ionized atoms)
List of atoms with pi bonds to the labeled atom
Target charge state of the labeled atom

While this information would include a charge assignment, it has the lowest priority and is only considered when the sigma and pi bond assignments are compatible with a desired charge state.
As a counterpoint, MOPAC's system for parsing keywords is already extremely clunky, and this type of feature would effectively turn every atom label into its own atom-specific keyword statement that needs special parsing. I suspect that's what turned Jimmy away from going any further down this development path.
What limits my appetite for changing this stuff is that the entire point of QM simulations is that this information is supposed to be determined by the computational search for the electronic ground state. Of course, this process can get trapped in local minima, and I think that it is a very good idea to allow for some kind of "guiding" assignment of atomic spins and charges (e.g. my own semiempirical modeling plans include this concept). Unfortunately, the more natural/intuitive assignments of atomic spin/charge have the lowest priority in MOZYME's current process of preparing a trial ground state and cannot be used to drive/constrain the process without a major overhaul of the Lewis assignment code.

I've revised the manual page to remove any incorrect statements about this feature. I'm now relabeling this Issue to a feature request rather than a bug report. I'll re-evaluate whether or not I'll act on this and with what priority as I eventually attempt to apply MOZYME to an increasingly diverse set of systems. Specifically, I will be mindful of problems or inconveniences that I encounter to understand if new ways of adjusting the Lewis structure would improve user convenience in practice, and if the Lewis assignment code itself needs some revisions or enhancements to resolve problematic cases.


Adam not found in checkpoint

Hi,
some nodes are not in the checkpoints e.g. Not found: Key decoder/attention_decoder/Attn OutputProjection/biases/Adam_1 not found in checkpoint   and I cannot run a MRT model on a pretrained cross entropy model. How can I include them to the saver?
Best,
Stoyan

... and not only Adam. When switched to SGD some others are also missing.
2017-11-23 20:51:10.335741: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key sentence_encoder/input_projection/word_embeddings not found in checkpoint 2017-11-23 20:51:10.339673: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key decoder/attention_decoder/AttnOutputProjection/biases not found in checkpoint 2017-11-23 20:51:10.343059: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key decoder/attention_decoder/AttnOutputProjection/weights not found in checkpoint

Is the neuralmonkey version still the same? It's possible that the names of some variable scopes have changed

We did not test the checkpoint functonality much (for saving and loading variables from a single model part) so it is likely a bug. But it should work when the version that saved the checkpoint is compatible with the version that loads it.

Yes, I forked the repository and didn’t update it since. Used the master to train a cross entropy model and my branch for the MRT. No changes should have happened.

Could you send the version hash so we can test it?

It’s the 0.2.1 version. (I hope this is what you are asking for)

Ok, that's it.. Could you also paste the configuration files so we can see what's going on.. Are  you using the save_checkpoint and load_checkpoint functionality, or are you using the initial_variables config for loading the variables?
As for Adam, it's Ok that Adam nodes are not in the checkpoint since we are saving only trainable variables.

The .ini for training the MRT is 1  and for the cross entropy model 2
I use the initial_variables. I pass it in the .ini

can close this, my bad. Somehow I mixed the encoders, as you can see in the .ini. For the X entropy used the recurrent.py and for MRT sentence_encoder.py.
Sorry for the confusion, I'll stick to the recurrent.py, I guess it doesn´t impact the results after all.

ah, ok..


Live trainning experiment table broken.

Training in the workspace the live experiment table is broken

While at the same time, dvc exp show cli works well.

How to reproduce.
2. In demo repo https://github.com/iterative/vscode-dvc/tree/main/demo.
3. set epochs to a relatively big number (1000).
4. wait for it to run about 100 iterations.
5. you will see the problem above.

Possibly related to https://github.com/iterative/vscode-dvc/issues/1690 / https://github.com/iterative/dvc/issues/7693

Possibly related to #1690 / iterative/dvc#7693

Yes related to https://github.com/iterative/dvc/issues/7693.

Yes related to https://github.com/iterative/dvc/issues/7693.

It does appear to be caused by the dvc exp show output described in https://github.com/iterative/dvc/issues/7693. Currently, I'm not sure what could be done on the VSCode side of things. @karajan1001, what's the status on fixing https://github.com/iterative/dvc/issues/7693?

I'm not sure what could be done on the VSCode side of things. @karajan1001, what's the status on fixing iterative/dvc#7693?

The problem with this is that the dvc exp show can show an experiment table while the vscode-extension can't I need to know what's wrong with the JSON result.

The problem with this is that the dvc exp show can show an experiment table while the vscode-extension can't I need to know what's wrong with the JSON result. Are there any restrictions?

From what I'm understanding, there appears to be an issue with the JSON result of dvc exp show. The latest running checkpoint object usually has a name but it stops showing a name at some point, causing the rendered table to break.
 Hopefully that's enough for @karajan1001 to address it on the DVC side. AFACIT it seems consistent with his findings in https://github.com/iterative/dvc/issues/7693 that the exp ref points to a revision that hasn't been fetched yet.
Is it possible to also make the extension robust to situations where name doesn't exist so that if this ever happens again it won't break the extension?

Yes, on my screenshot we can also see that there is no exp-xxxx in it.

Is it possible to also make the extension robust to situations where name doesn't exist so that if this ever happens again it won't break the extension?

It looks to me like the issue on the VS Code side is here:
https://github.com/iterative/vscode-dvc/blob/6603de9c86d6ff7c96820517440c5f70047aa5c3/extension/src/experiments/model/collect.ts#L121
If the extension falls back to using checkpoint_tip when name doesn't exist, I think the extension would match the CLI that @karajan1001 showed above.
I tried to play around with this and it seems to work except for wanting to collapse constantly:
https://user-images.githubusercontent.com/2308172/202570190-ad19db87-535a-4a5c-8a9f-4896c0db3863.mov
I would submit a PR, but I think someone with better TypeScript skills and familiarity with the codebase could do a much better job much faster.


add draft ADR for static IP support

Changes proposed in this pull request:

add draft ADR for static IP support

security considerations
None

Thanks @bengerman13 Just for clarification, I agree with the guidance that the canonical domain should be www and the apex should redirect to it, that said, does the implementation suggested here support serving the site at the apex with Cloudfront if that is what the user wants?

not any way that is not already supported (e.g. aliases and bypassing our recommendation to use the intermediate external-domains-production.cloud.gov address)
Static addressing and cloudfront are totally incompatible. The only way to accomplish it would be by proxying to cloudfront, which defeats the point of cloudfront.

Is figuring out a way to emit the static IP info when the domain is set up the3 first order of business here? Is this feasible if we determine at some point that there is not an easy way to do that?

There are definitely ways to do it, so I think we can barrel forward without a firm plan on that.
The ideal is using the metadata field from the Open Service Broker API. I have a vague memory of that not being well-supported by CAPI, but I could be wrong, or it could have changed.
As another option, we could emit it as part of the last operation data, which is what the old broker did for ALB names (there are downsides there - namely, we need to make sure it's emitted for any Last Operation step)
Finally, if neither of those work, we could expose this via service keys - this seems like the clumsiest user experience IMO, but maybe the closest to what CAPI/CF want us to do


⚠ Bofrost.at has degraded performance

In 5b77fe5, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1920 ms

Resolved: Bofrost.at performance has improved in 62eca84.


Remove parenthesis from integer test names

As reported here https://github.com/elastic/elasticsearch/issues/35692 there can be issues when running individual integration tests that contain parenthesis in its name.
This means we need to modify all affected *.yml files in rest-api-spec/test folder.

@nik9000 Do you know if the fix will be available for both the ES 5.x and 6.x?

Not in 5.x, no.

@nik9000 Ok, well, then I will need to remove parenthesis anyway (I will be back-porting all esplugin tests back to ES 5.x as much as possible). But hey, thanks a lot Nik!

Verified, parenthesis are not breaking individual integration tests for ES 7.x any longer (master branch as of writing).
Running tests like the following is ok:
gradle integTest \
  -Dtests.class=org.elasticsearch.rest.PrometheusRestHandlerClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=resthandler/30_00_index_level_info/Index level statistics (general)}"

Based on @nik9000 comment above this should be also fixed for ES 6.x.
Closing as WONTFIX for now.


drivers: usb_dc_nrfx: clear write/read flags after ep_disable

If a loaded endpoint was disabled and then reconfigured,
it is not possible to start an IN transfer and
usb_dc_ep_write() returns -EAGAIN.
Clear write/read operations flags after endpoint is disabled.

@emob-nordic @anangl please review

Moved to 2.5 as it needs more discussion.

closing in favor of #34386


Renames COLLECTOR_SD_LOG -> COLLECTOR_EXTRA_LOGGING

Description
Adds configuration to the integration tests to enable it on demand, and removes other references around that point.
Checklist

[ ] Investigated and inspected CI test results
[ ] Updated documentation accordingly

Automated testing
~- [ ] Added unit tests~
~- [ ] Added integration tests~
~- [ ] Added regression tests~
If any of these don't apply, please comment below.
Testing Performed
Tested locally, with and without the variable.

Maybe instead of extra we could use something like inpector?

The main reason I chose extra was to make it agnostic of component, the thinking being that if we wanted to add some more conditional logging elsewhere or in a different component, we wouldn't need to manage multiple different config options to do so.

Also, we don't need to do it in this PR, but a label that could be added on PRs to get this flag set would be a good to have IMO.

Good idea. Should be simple enough to add. I'll do it in this PR.

The main reason I chose extra was to make it agnostic of component, the thinking being that if we wanted to add some more conditional logging elsewhere or in a different component, we wouldn't need to manage multiple different config options to do so.

Yeah, I get this point, I just think extra sounds a little too agnostic or even a little vague. I'd prefer inpector since it doesn't mention a specific provider, yet it implies it is enabling the logs for the inspector.
We could also consider adding another level of log, something like DebugVerbose could just set the inspector to also log out messages.

Perhaps then we can attach this to the TRACE log level and drop the environment variable altogether?

Yeah, I think tying it to the TRACE log level could work, I don't see it being used in our codebase at all, so it should be ok.


Deprecate meta.outputsToInstall

Describe the bug
outputsToInstall apparently is a hack around things going on in nix-env that may have spilled into Nixpkgs somewhen in the past. Currently it's only used in a very few place in Nixpkgs, some of which decidedly remove the attribute from the derivation.
References
https://nixos.org/manual/nixpkgs/unstable/#sec-multiple-outputs-installing-nix-env
https://nixos.org/manual/nix/unstable/command-ref/nix-env/install
Expected behavior
That thing should not exist, as it would break reasonable assumptions about how nix-env may work but doesn't.
@infinisil this may be something for the Nixpkgs Architecture Team to decide.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-09-21-documentation-team-meeting-notes-81/33490/1


Add a static constructor to instantiate Auth Response from string

Feature Request
Describe the solution you'd like
We should be able to use the parse function and use the methodArn to instantiate an object instead of parsing methodArn first.
For example, currently I have to do this:-
        const arnParts = AuthorizerResponse.parseApiGatewayArn(event.methodArn)
        const response = new AuthorizerResponse(
          'apigateway.amazonaws.com',
          arnParts.region,
          arnParts.awsAccountId,
          arnParts.apiId,
          arnParts.stage
        )

And I would like to have a single function to create from methodArn like so
        const response = AuthorizerResponse.fromMethodArn(event.methodArn)

@jappurohit041 this might prove to be even more useful for you.

Solved by #6


Acceptance Testing Schedule Command Bug

Validation for schedule adding may be buggy.

Sorry, but I cannot add the schedule too.

Help:
As a new user, when the help Window is open, when I perform commands, it doesn't update in the UI until I close the help window, this is very inconvenient and I have to open the help window every time after each command.
UI:
Your graph is far too small on my computer, I cannot resize it or see anything about it. In fact, I don't think a graph is the way to go to display the information you want, as the students are not in a sequential order.
The column in between the person's details and the graph has no data displayed, and shows stuff like $mode, $standarddeviation etc on launch. Is this a bug? Contrast is pretty poor and it is hard to see too. It ended up updating afterwards somehow. Is it supposed to display the statistics for the entire list of persons? The class? Not very clear. Maybe don't use Group, or elaborate in the User Guide.
The bottom half of the screen on the 2nd, 3rd and 4th columns are useless, maybe have something there so you don't waste screen real estate.
find, findtags and clear
Using the command find debate or clear gives a white box in the first column.
schedule:
Scheduling a person doesn't seem to do anything at all. Where can I see the schedule list?
edit:
I can't edit a person's phone or a parent's phone alone and I am forced to update them both

Thanks for the feedback guys


cleos get account test1100  unknown key

root@fengqun:/opt/eos_src# cleos get account test1100
924358ms thread-0   main.cpp:2659                 main                 ] Failed with error: unspecified (0)
unknown key

The specified account does not exist on the chain queried.  Query the correct chain, or query for an account that exists.

@jgiszczak  Please help me，Wallet account has been created
root@fengqun:/opt/eos_src# cleos wallet list
Wallets:
[
"beidukeji",
"fengqun",
"test",
"test110",
"test111",
"test66"
]
root@fengqun:/opt/eos_src#
root@fengqun:/opt/eos_src# cleos get account fengqun
2248820ms thread-0   main.cpp:2659                 main                 ] Failed with error: unspecified (0)
unknown key
root@fengqun:/opt/eos_src# cleos get account beidukeji
2258518ms thread-0   main.cpp:2659                 main                 ] Failed with error: unspecified (0)
unknown key

I am also getting the same issue.
My contract account name is battlegame.
I have set my contract.
"user" is the account i have created and also imported the private key with respective public key.
After i do the below command:(i am getting issue)
cleos push action cardgame login '["user"]' -p user@active
error 2019-07-20T19:28:16.254 thread-0  main.cpp:3896                 main                 ] Failed with error: unspecified (0)
unknown key (eosio::chain::name): cardgame

@sugalivijaychari You said that contract name is battlegame, but why did you send transaction to cardgame?


Deprecate QueryBuilder#getResults()

This method promotes inefficiency. It collects query results to a List of Map instances, but in most cases the list must be traversed again to coerce the maps to a requested type, and then again to process the typed results (e.g. serialize as JSON). A more efficient approach would be to stream(), map(), and collect the results, which would require only two traversals. This is not as efficient as using a pipe (which requires only a single traversal), but is often acceptable for small result sets.
The following QueryBuilder methods should be deprecated and ultimately removed:

execute(Connection, Map)
getResult()
getResults()
getUpdateCount()

The executeUpdate(PreparedStatement, Map) method will be updated to store the generated keys for later retrieval via getGeneratedKeys(). It probably should have been doing this all along.

This is done.


Unknown Error: 'CommentedSeq' object has no attribute 'strip'

Version Number
2.0.1-nightly24
What branch are you on?
nightly
Describe the Bug
Fails to build collection. According Mr. Carrot on Discord, issue is also present on 2.0.2 and it happens with a list, but not with a single URL.
Relevant Collection/Overlay/Playlist Definition
Hallmark - Movies:
    summary: A collection of heartwarming movies brimming with love, laughter, and the promise of second chances. Each story is a testament to the enduring magic of heartfelt romance, the joy of true love, and the enchantment of the holiday season.
    mdblist_list:
      url:
        - https://mdblist.com/lists/SynAcksHM/hallmark
        - https://mdblist.com/lists/marko8426/hallmarkallmovies
        - https://mdblist.com/lists/larryconlin/gentle-hallmark-romantic-comedies
        - https://mdblist.com/lists/chrisdmartinez/hallmark-ovlcerrd46
        - https://mdblist.com/lists/womblechutney/hallmark-3k21cd0jzu
        - https://mdblist.com/lists/womblechutney/hallmark-movies
    sync_mode: sync
    smart_label: release.desc
    collection_mode: hide
    visible_library: false
    visible_home: false
    visible_shared: false
    schedule: monthly(1)

Logs
https://gist.github.com/qazero/a7290d11f98c4c6b8e7350f938ff3d76

Solution provided by Sojiro on Discord. Pasting screenshot below for reference:


Feature Request: Spatial Audio Control

I love this plugin, but spatial audio is important to me, and I am not an Android developer. So adding it back in would be a significant challenge. Would you be willing to add it in as a toggle-able option?
Something like the Shutter Volume Control plugin where it can be turned on or off with the wireless button would be great! Extra great if it could also remember the last setting so that I wouldn't have to turn it on every time I enable the plugin.
Thanks again for your work on this!

@antgiant - Is getting ahold of the .wav file separately and then adding it to the video file in something like Premier Pro an option? There are instructions here for grabbing the .wav file: https://community.theta360.guide/t/theta-v-long-video-plug-in-recording-beyond-25-minutes/3483  (scroll down to "Transfer Spatial Audio File to Your Computer" section)

@jcasman Yes I’m willing to combine the files, I expected to do that actually. (By the way Facebook now provides a free tool that can do it as well for those of us without Premier Pro. Spatial Workstation)
However, if you read to the end of your referenced chain or the note on the official version’s description you will note that the wave file is now auto deleted. Hence my request for an option to get the file that doesn’t also require me to become an Android developer.


Header is not at the top

Hallo, sorry if this issues has been post in this Awesome Open Source,
im using
"dependencies": {
"native-base": "^2.4.2",
"react": "16.3.1",
"react-native": "0.55.3",
"react-navigation": "^1.5.11"
my problem is my header didnt show in the top of the phone like this

what's wrong ?
my code is
import React, { Component } from 'react';
import { Container, Header, Left, Body, Right, Button, Icon, Title, Subtitle } from 'native-base';
import {View} from 'react-native';

class HomeScreen extends Component{
    constructor(props){
        super(props);
        this.state ={
            names:[
                {
                    id:1,
                    name:"michael"
                },
                {
                    id:2,
                    name:"lazuardy"
                }
            ]
        };
    }
    render(){
        return(
           <View>
                <Container>
                    <Header>
                        <Left />
                        <Body>
                            <Title>Title</Title>
                            <Subtitle>Subtitle</Subtitle>
                        </Body>
                        <Right />
                    </Header>
                </Container>
           </View>
        )
    }
}
export default HomeScreen

i guess something wrong with react-navigation

@mlazuardy if you are using react navigation, set the header to null.

@akhil-geekyants how sir? Sorry im new in react native,

@akhil-geekyants  you mean <Header/> just like this?

thank you sir, ill try

@mlazuardy
static navigationOptions = {
  header: null,
};

See this link

get it, but should i put that in all component ? or do you have eficient way?

@mlazuardy you can do something like this
const RootStack = StackNavigator(
  {
    Home: {
      screen: HomeScreen,
    },
    Details: {
      screen: DetailsScreen,
    },
  },
  {
    initialRouteName: 'Home',
    navigationOptions: {
     header: null
    },
  }
);


replace import path

fixes #2

I was able to build the replace branch by importing it from driver/sdcard.

Thanks @sago35 now merging.


Display LNURL Error Reason instead of "service offline" message

1⃣ Scan or paste an LNURL that returns an error response according to the LNURL Spec (also applies to payRequest etc.)
Example: LNURL1DP68GURN8GHJ7MR9VAJKUEPWD3HXY6T5WVHXXMMD9AMKJARGV3EXZAE0V9CXJTMKXYHKCMN4WFKZ76TJXDVYVC6RFQM8W52Y8PU9QE2PXDZKXKR34H0W77

Returns:
{"status":"ERROR","reason":"Withdraw is spent."}
2⃣ The Phoenix wallet (at least on iOS) displays "The service (...) appears to be offline, or they have a down server", which is not the case.
It would be helpful to the user to actually know the "reason" from the LNURL response.

Thanks for reporting this issue.
It seems specific to the iOS app. @robbiehanson I think we'll fallback to an "service offline" message if the error is not unreadable/cannot connect : https://github.com/ACINQ/phoenix/blob/3d122f583d00fea107d8e56bc26fd2f1a139216c/phoenix-ios/phoenix-ios/views/send/SendView.swift#L241-L248
On Android we should be ok:
https://github.com/ACINQ/phoenix/blob/3d122f583d00fea107d8e56bc26fd2f1a139216c/phoenix-android/src/main/kotlin/fr/acinq/phoenix/android/payments/ScanDataView.kt#L298-L303
Though on Android we don't have a special Lightning address case (user unknown by this service), which is a nice feature.


Policies do not combine down the OU hierarchy

Some policies store their setting as REG_SZ with 1,2,3,... This scheme for the registry keys prevents values set by GPOs down the OU hierarchy from being combined.
For example, formerly in IE another scheme was used to store such values. Back then it was possible to set a GPO on top level with a set of certain URLs and set an additional GPO on a sub-OU with a set of additional URLs. Both GPOs have been applied and the settings have been combined to a large list. With the new scheme in MS Edge (Chrome) instead of being combined the GPO from the sub-OU replaces all values from the GPO of parent OUs.

@joegoerlich Thanks for the feedback.  We've had other similar requests but no ETA for a change right now.  (For example, see previous discussion on Tech Community: https://techcommunity.microsoft.com/t5/enterprise/microsoft-edge-group-policies-for-lists-like-pop-ups-do-not/m-p/2279658/highlight/true#M3357)
I've passed your feedback on to the team.
-Kelly


'Unfinalized' operations being reverted on Base results in deleting all historical data

When running ponder start I can initially see data being indexed as expected but soon after completing the historical index I see this logged which results in empty tables
8:18:37 AM WARN  realtime   Detected forked 'base' block at height 23385083
8:18:37 AM WARN  realtime   Reconciled 2-block reorg on 'base' with common ancestor block 23385082
8:18:37 AM INFO  database   Reverted 11 unfinalized operations from 'Distributions' table
8:18:37 AM INFO  database   Reverted 230 unfinalized operations from 'Crowdfunds' table
8:18:38 AM INFO  database   Reverted 1253 unfinalized operations from 'Contributions' table

Any ideas how to prevent this from happening?

Yes. I have the same problem as you

@kelvin0x for now i fixed it by patching the finalityBlockCount to be 1, but probably not the best solution
diff --git<EMAIL_ADDRESS><EMAIL_ADDRESS>index 47e377c..a7eba79 100755
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -192,7 +192,7 @@ function getFinalityBlockCount({ chainId }) {
       finalityBlockCount = 240;
       break;
     default:
-      finalityBlockCount = 30;
+      finalityBlockCount = 1;
   }
   return finalityBlockCount;
 }

Thanks for opening, would love to get to the bottom of this. Couple questions:

Could you please share your config file? Specifically curious about what the contract start blocks (eg close to tip? possible that they are greater than latest - 30 ?) and networks (are you using multiple)?
Do those reverted operation counts look like the entire database, or only a few blocks worth?

A few folks have this issue, so very likely an unhandled edge case related to start blocks, finalization windows, etc.

@typedarray thanks for responding!

Config file

import { createConfig } from "@ponder/core";
import { http } from "viem";

import { CrowdfundFactoryABI } from "./abis/CrowdfundFactory";
import { ERC20LaunchCrowdfundABI } from "./abis/ERC20LaunchCrowdfund";
import { TokenDistributorABI } from "./abis/TokenDistributor";

export default createConfig({
  networks: {
    base: {
      chainId: 8453,
      transport: http(process.env.PONDER_RPC_URL_8453),
    },
  },
  contracts: {
    TokenDistributor: {
      network: "base",
      abi: TokenDistributorABI,
      address: "0xc3EF5e6c8cb42C4151859352ACEbE68cf05D8d7d",
      startBlock: 23291762,
      filter: {
        event: "DistributionCreated",
      },
    },
    ERC20LaunchCrowdfund: {
      network: "base",
      abi: CrowdfundFactoryABI,
      address: "0x52e506B58ef5f60Ae74bD04C9bc37A63863Dfe5d",
      startBlock: 23291762,
      filter: {
        event: "ERC20LaunchCrowdfundCreated",
      },
    },
    Contributed: {
      network: "base",
      abi: ERC20LaunchCrowdfundABI,
      startBlock: 23291762,
      filter: {
        event: "Contributed",
      },
      includeTransactionReceipts: true,
    },
  },
});

Yep it's the entire database

@stephancill Are you doing any db.sql operations in your indexing functions? If so could you share some of the indexing function code

@kyscott18 im doing a db lookup in one of the indexing functions
import { ponder } from "@/generated";
import { eq, replaceBigInts } from "@ponder/core";
import { decodeEventLog } from "viem";
import { PartyABI } from "../abis/Party";
import * as schema from "../ponder.schema";

ponder.on("Contributed:Contributed", async ({ event, context }) => {
  const { contributor, amount } = event.args;

  // Find crowdfund to ensure it's from a valid crowdfund
  const [crowdfund] = await context.db.sql
    .select({ address: schema.crowdfunds.address })
    .from(schema.crowdfunds)
    .where(eq(schema.crowdfunds.address, event.log.address))
    .limit(1);

  if (!crowdfund || amount === 0n) {
    return;
  }

  // Find the tokenId for the contribution
  const tokenId = event.transactionReceipt.logs
    .map((log) => {
      try {
        const votingPowerSetLog = decodeEventLog({
          abi: PartyABI,
          topics: log.topics,
          data: log.data,
          eventName: "PartyCardIntrinsicVotingPowerSet",
        });
        return votingPowerSetLog.args.tokenId;
      } catch (error) {
        return null;
      }
    })
    .find((v) => v !== null);

  if (!tokenId) {
    throw new Error(
      `Token ID for contribution not found for ${event.transaction.hash}`
    );
  }

  await context.db.insert(schema.contributions).values({
    id: event.log.id,
    crowdfund: event.log.address,
    contributor,
    amount,
    tokenId,
    logRaw: replaceBigInts(event.log, (v) => v.toString()),
    timestamp: event.block.timestamp,
  });
});


Asm.Js: x64 outparam opt fix

Make sure we allocate enough space in the interpreter for outparam to do the param allocation optimization.
OS#14711984
Found by OSS-Fuzz

This change is

ping


ESLint Integration and Error Resolution

Description:
This PR introduces ESLint into the project to enhance code quality and maintainability. It includes the following changes:

ESLint Configuration: Added a comprehensive ESLint configuration file to enforce coding standards across the codebase.
Error Resolution: Fixed all identified linting errors, including issues related to code style, prop validation, and other best practices.
Code Quality Improvement: Ensured consistent code formatting and reduced potential bugs by adhering to established coding standards.

Impact:

Improved code readability and maintainability.
Enhanced collaboration by standardizing code style across the project.

Next Steps:

Review the ESLint rules and configurations to ensure they align with the team's preferences.
Monitor for any new linting errors as development continues.

Related Issues:

[Issue #74 ] @MinavKaria

Thank you for reviewing this PR!

There were total 34 files which had multiple erros in each of those.
Those errors were resolved manually and
The site was got proper inspection for its working.
Further Improvements:
From now on, on one can push unused data in the repo

Work done! on top priority

@samar12-rad It may take some time to merge your PR

What is the issue @MinavKaria ?

I will resolve branch conflicts.

Please try not break any feature
@samar12-rad

Ok

@samar12-rad Hey resolve the conflict in the weekends, bcoz I'll need to review the code as changes are too much. I'll give level3 to this as there too many eslint issues resolved

You are right. There are too many conflicts, I'll do it on weekend.

@samar12-rad You are good to resolve the conflicts now.

Currently this PR is not needed, When needed, I'll message you. Sorry for inconvenience. Feel free to open new issues,. @samar12-rad


[research] analysis on swap fee rate and pool investor profit

Comparing risk-return profile of pool investors between Uniswap and Liquidity Module

Risk

Impermenant Loss

impermenant loss is caused by reserve token balance changes until withdraw
this risk is exactly same both in Uniswap and Liquidity Module

Slippage Return

in Uniswap, there exists no mechanism to protect liquidity pool from impermenant loss by huge instant price changes
but, in Liquidity Module, there exists a mechanism to partially protect liquidity pool from significant impermenant loss

huge instant price changes lead to longer batch period
which attracts more arbitrageurs, resulting in swap price near global price
then, assuming perfect competition among arbitrageurs, impermenant loss can be partially covered
by the swap price which is near global price

path dependency : this protection mechanism effect is minimized if price change for each batch is minimized

Return

Swap Fee

each executed swap pays swap fee proportional to swap amount
for swap fees from pool-match, both Uniswap and Liquidity Module has same potential fee returns
but, in Liquidity Module, it additionally has swaps which do not spend liquidity from liquidity pool (self-match)
therefore, Liquidity Module has additional fee source compared to Uniswap for given liquidity amount
this can be described as higher capital efficiency of liquidity pool

in short, Liquidity Module pool investors have lower impermenant loss risk, and higher swap fee return than Uniswap

this topic has been researched and presented from economic simulation report


User not disconnected after breakFriendship

I disconnected all users of<EMAIL_ADDRESS>via the service panel > identities > show users > disconnect. Somehow the service wasn't deleted from my FriendMap.
See these server logs. You can see the updateFriendSet call which still contained that service.

<EMAIL_ADDRESS>too. See https://github.com/our-city-app/oca-backend/issues/687
Let's create a migration script that loops all disabled services and checks if there are FriendServiceIdentityConnections that aren't cleaned up. First, check if these FSICs are cleaned up or not for<EMAIL_ADDRESS>If they are cleaned up, then maybe the FriendMaps aren't cleaned up. Test with user b2d2c7aa7c1cea8ae2a88c3b39a0d95e:be-diksmuide.


Difficulty in finding application crash issue after using connectycube in flutter application.

HI, Im using connectycube in my application for video calling services.When i build my app in debug mode,it is working properly without any issue .But when i open my app after final apk building it is crashing and all this occuring after using connectycube sdk in my app.

@TatankaConCube  can you help me out?
This is very major issue for me as my whole app depends on this.

@TatankaConCube

We are investigating your issue, and we will answer you after our internal checks.

Looks like you have a problem with obfuscation of some classes from WebRTC lib. Try add in your build.gradle file of module level next lines in release bid config:
buildTypes {
        release {
            //your code here

            minifyEnabled true
            useProguard true

            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
}

Then create file path/to/project/connectycube-flutter-samples/p2p_call_sample/android/app/proguard-rules.pro with next content
## Flutter wrapper
-keep class io.flutter.app.** { *; }
-keep class io.flutter.plugin.**  { *; }
-keep class io.flutter.util.**  { *; }
-keep class io.flutter.view.**  { *; }
-keep class io.flutter.**  { *; }
-keep class io.flutter.plugins.**  { *; }
-dontwarn io.flutter.embedding.**

## Flutter WebRTC
-keep class com.cloudwebrtc.webrtc.** { *; }
-keep class org.webrtc.** { *; }

And build your release again.

thanks!Now my app is working


Request doesn't have a body prop

Hey guys, I'm having a few issues trying to mock a request to my api, the response object doesn't have a body property.
Here is how I'm doing it
var mock = nock(apiConfig.apiUrl)
            .intercept('/notifications/', 'GET')
            .reply(200, {
                notifications:[
                {id:randomId_1},
                {id:randomId_2}
                ]});
            api.getNotifications(false, false, function(err, data) {
                expect(data).to.deep.equal({notifications:[{id:randomId_1}, {id:randomId_2}]});
                mock.done();
                done();
            });

I even tried to track this on your library and it seems that something is happening in request_overrider.js#L436 with the body.
This test is working for my friends with MAC but it's failling on every Linux machine and shippable.
Let me know if you need anything else from me, thanks

Hi @dkenzox can you tell me which version of Node and Nock you're using?

Sure,
node v0.12.3
nock v2.4.0

Note
Just updated to v2.4.1 and it looks like it is fixed now, maybe this e42028ed82eac582fc73a7cc7c6401a9fdbf51b0 fixed it?

Very good call. I was dealing with similar issues and all is working fine since the upgrade to 2.4.1.


Difference between PrefetchDataZMQ and PrefetchOnGPUs

If I train data by multiGpuTrainer, what's difference between PrefetchDataZMQ and PrefetchOnGPUs?

Sometimes the pre-processing can benefit from running on a GPU, e.g. you can use GPU 1&2 for training and 3&4 for preprocessingdata on-the-fly (OpenCV supports GPUs or custom code)
see: https://github.com/ppwwyyxx/tensorpack/blob/master/tensorpack/dataflow/prefetch.py#L207

Thank you for your comment


Can we choose alert types based on a threshold with in a rule?

Hey,
I have a requirement to raise Service Now ticket if percentage of errors is > 10% and slack alert if it < 2% , is there a way we can achieve this in one rule ?
Or do we have to create multiple rules with different thresholds and alerts?
Any help is appreciated.

https://github.com/Yelp/elastalert/issues/1299

@nsano-rururu , thank you for pointing me to the issue, I looked at and it is not what I am looking for.
Sorry, I am not very clean on the ask.
I am planning to use percentage match rule, but with in that rule, I was to send different type of alert based on match.
Ex:
%match > 50 then I need a Service now Ticket
%match >20 and <50 then I need an Email
%match >10 and <20 then I need a slack notification.
Is this something that is doable??
@Qmando any help is appreciated.

We also recommend asking questions on the Gitter channel
https://gitter.im/Yelp/elastalert


Create README.md

I find a link directly to the Marketplace useful.

The vss-extension.json file has a content/details/path property that points to the the overview.md file - If you could please update that value from overview.md to readme.md so that the next publish will work correctly, I will approve the pr.
Thank you for very much!


mpv: use wrapper for scripts and add mpris script

Motivation for this change
#14040 added the ability to use Lua scripts with mpv. mpv also supports C scripts/plugins which need to be compiled with libmpv. Thus loading C plugins is not possible by simply modifiying mpv's derivation. I've add a wrapper called mpv-with-scripts to solve this problem.
@Profpatsch, could you have a look as the author of #14040?
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

Will take a look later today. Ping me if I forget. :)

Nice touch with the wrapper script.
I wonder if we should make mpv-with-scripts into a function instead? mpv.WithScripts [ mpvScripts.mpris ]

Have you tried it with both scripts? The manpage is a lie, it throws an error if you separate the scripts with a comma. It seems to work when you give it multiple --script options. Maybe open an upstream bug request?

Multiple scripts in a single --script don't work for me either. I will file an upstream bug. Also, was there a --scripts with an s option before? I saw this one in your original PR but it doesn't work at all now.
About mpv.WithScripts I don't know what the preferred/better way is. I've done some grepping on the repo and using a seperate attribute seemed more common to me.

About mpv.WithScripts I don't know what the preferred/better way is. I've done some grepping on the repo and using a seperate attribute seemed more common to me.

Yeah, let’s keep it as it is for now.
Works for me, LGTM!


Make issue, "cannot import name 'Database' from 'db' (unknown location)"

I'm following the set-up instructions. I try make clean reset and get this error.
Traceback (most recent call last):
  File "reset-database.py", line 2, in <module>
    from libs.db_sqlite import SqliteDatabase
  File "/eric/Desktop/Audio fingerprint/audio-fingerprint-identifying-python-master/libs/db_sqlite.py", line 1, in <module>
    from db import Database
ImportError: cannot import name 'Database' from 'db' (unknown location)
make: *** [Makefile:10: reset] Error 1

I assume I'm doing something in the wrong order, but db is an empty folder. Just unsure what I'm missing.

@Scherflinge this is because db directory and db.py under libs share same name,when  execute from db import Database,it find Database in the db directory,a error raised.tryfrom libs.db import Database from libs.config import get_config.

Does it only work with python2 ?
Even I am also getting same error as above

i am running this on Linux with pYthon 2.7 still not working make clean reset command with the suggestion also incorporated by @dianaxun
please help asap


test(@libp2p/tcp): close listener during upgrade

Title
test(@libp2p/tcp): close listener during upgrade

Description
Adds a failing test which shows that the listener is not able to close during a
tcp connection upgrade.

Notes & open questions
Fixed by: #2763

Change checklist

[x] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation if necessary (this includes comments as well)
[ ] I have added tests that prove my fix is effective or that my feature works

Thanks for opening this - this test was integrated into #2744 2763


Making VACUUM no-op if there isn't anything new to delete

Broad Picture
Given that users need to run vacuum manually on the tables, one common way to achieve that is to have a periodic vacuum job that runs on all your tables.
Running on all tables, could mean that we are running it even on tables where we haven't removed any file since the last time vacuum ran.
This could result in increased processing time as well as increased costs from the storage provider such as s3 for LIST calls.
Possible implementation
An ideal solution would check if the table ever has had a vacuum run, and if so, has any new file been removed since then.
A simpler solution would be to see if the last command was vacuum
val lastCommand : Action  = deltaLog.commandLog.last
lastCommand match {
  case Vacuum =>
  case _ => gc()
}

Blocker
This solution requires that we log somewhere if VACUUM ran or not, and from the code it seems like the logging for Vacuum atleast so far is a no-op, here https://github.com/delta-io/delta/blob/6d9420539e9b00f8df8d5bc80ce6aa7a5a9f3b7d/src/main/scala/com/databricks/spark/util/DatabricksLogging.scala#L48-L57
One overkilling way to achieve that would be to add it to transaction log in ? Along with all add/remove/convert operations we log in https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/DeltaOperations.scala.
I might be missing how much more work this could be or if it breaks certain ACID guarantees.
Alternative Ideas

Run VACUUM after writing.

The issue is that every writer needs to call it, and having multiple different writers/jobs that's basically code replication. Maybe having a post write trigger in delta tables could be helpful?

Bear the cost of compute/list

Create a log of removedFiles for each table. Therefore deltaTable.removedFiles == lastTimeWhenJobRan.removedFiles means nothing to do.

One thing to consider is that VACUUM cleans up any files, even files that were never officially added to the transaction log.  So even if there were was no new Add since the last VACUUM, there could be some other random files that need deleting.

Aah I see, thanks for explaining. What are the reasons of these random files that might be added?

many reasons.

failed tasks that wrote out some files, but those files were never committed.
failed operations that wrote out some files. could happen due to transaction conflicts, or any kind of failure.


[ENH] user selected return for classification problems data loading functions

adjusted now to only offer the option of return type, not determine return type from the data. Part of #1778.

Includes a new function load_from_tsfile that will ultimately replace load_from_tsfile_to_dataframe in all loading helper functions. load_from_tsfile returns 2D/3D numpy for equal length problems, DataFrame for unequal problems
EDIT: the return type is now selected by the user, and the data driven version has gone into contrib.
load_from_tsfile now actually checks for the equalLength flag in the header.
A private method _load_provided_dataset that will replace _load_dataset in the baked in loaders. The new method simply looks in the datasets/data directory for the problem requested and fails if it is not there, rather than attempt to download from tsc.com
Adjusts the current _load_dataset to work in local_data directory when it downloads from tsc.com, and change the gitignore to local_data

there are a lot of changes to tests - these seem unrelated to the topic of this PR. Would you like to move these fixes in a different PR, for quick approval, and making the review of the dataIO stuff easier?
if you want the return polymorphic, why not put a convert_to at the end, and give it return_mtype arg? The current behaviour seems very rough and confusing to the user, since the return mtype is now inconsistent and will depend on properties of the dataset (univariate, multivariate, equal length)
in particular, I'd strongly advise that there is one default return_mtype. We've soft-adopted the design principle earlier (core dev vote) that there is one single output mtype.
a clear strategy would be, always convert it internally first to pd_multiindex or df-list, as the most general Panel mtype-s, and then convert it to return_mtype.

no I cant separate out the changes to tests, because they are made necessary
the basic logic of classifiers is that univariate equal length goes into 2D, multivariate equal length into 3D and unequal length into DataFrame. This is completely clear to me, not sure why it would confuse a user. It is entirely consistent with the recent base class restructure.
3.The vote was on the y variable, which is always numpy with us now.
It seems pointless to put data into a more complex data structure then converting it back into a simpler one

ote that I will have do redo the tests as they wil lconflict with https://github.com/alan-turing-institute/sktime/pull/1799, but thats ok, only took 10 mins

and I would add, the whole point is to make the return type dependent on the data type. Thats how classification is now structured. Rather than but everything in nested data frames, we now use the best data structure for the problem. This is how i have always wanted it, but always deferred. Now the base class structure has changed to allow it, I want it this way for the UCR/UEA data sets in .ts format. Users can put their own data in numpy or dataframes, which ever they think is best.

the basic logic of classifiers is that univariate equal length goes into 2D, multivariate equal length into 3D and unequal length into DataFrame. This is completely clear to me, not sure why it would confuse a user.

Well, try to explain it to a user and see what they say.
"The function return is 2D numpy, except when it is 3D numpy, or pandas DataFrame. It's 3D numpy when we can't put it into 2D numpy, and pandas when we can't put it into 3D numpy. Of course we could always return pandas and convert it to numpy 3D on request, but we don't do that"
I honestly challenge you to explain this to someone who's not worked on this.
I strongly recommend to have a return_type argument that controls what comes out of this? And not do conditional type, but raise error messages if the return can't be represented in a given type.

It's easy to explain:
If it is univariate, equal length, it returns 2d
If it's multivariate, equal length, it returns a 3d
If it's unequal length, if returns data frame
If this is not understood, then they have not understood the nature of the data and task, so maybe they will then try do so. It is not a user driven option for this specific function.

If this is not understood, then they have not understood the nature of the data and task, so maybe they will then try do so. It is not a user driven option for this specific function.

The problem with this is that you are loading a file.
You don't know in advance which properties the data will have, e.g., univariate, multivariate, etc. You are introducing what is effectively a non-inspectable condition in the output type, so from a user inspective the output type will be more or less random.
I'd propose an indicative/non-binding core dev vote after the holidays?

You do not in advance the properties of the data you are loading, thats the whole point of the ts format, the meta data is in the header. The output is not in any way random, it is a property of the dataset, and to think the user will know nothing about the property of the data they are loading implies they know nothing about the data they are loading, even to the point they do not understand it is a classification problem. Any output from this function can be passed to any sktime classifier. I generallly think a user will find it easier to understand the difference between a 2D or 3D numpy than it is to understand the whole scitype mtype definition you have invented.
So, sure we can have a vote if you want, but what would we be voting on? The very existence of the loading function, its use in the single problem loaders or planned use in general loading (or all of the above)? I can just put it in contrib and limit use to my group if you object this strongly.

So, sure we can have a vote if you want, but what would we be voting on?

I'd propose, three options, docstring 1, docstring 2, "neither is good"?
Docstring 1 has the return as you describe. Docstring 2 has a default return of pd-multiindex, and an return_mtype arg that allows to pick others such as numpy3D (with error raised if not possible.
Here's my docstring:
    """Load Panel data from a .ts file into an sktime compatible format.

    Parameters
    ----------
    file_path: str
        File lication of the .ts file to read, full path and file name.
    return_mtype: str (optional), default="pd-multiindex"
        in-memory (mtype) format in which the data is loaded
        mtype strings with explanation are in datatypes.MTYPE_REGISTER

    Returns
    -------
    X : object of type `return_mtype`
        the loaded data

    Raises
    ------
    RuntimeError if no file at location exists
    RuntimeError if file exists but is in incorrect format
    RuntimeError if file exists and is in correct format, but `return_mtype` is too specific.
        example: data has unequal length series, but `return_mtype=numpy3D`
            which can deal only with equal length series
    """

Pd-multiindex is not supported by the estimators, so can't be that unless we redesign everything which I don't want to do. Default should be nested data frame as it is now. I don't mind changing it to default to that or switch on user parameter I suppose although it will get messy with exceptions when data doesn't allow the choice.

it needs tidying up, not ready for review, but in principle is structure for load_from_tsfile acceptable @fkiraly? If the user passes an invalid option (say return 2d for multivariate data) would you rather raise an error or just ignore and return the default type? I assume the former

Pd-multiindex is not supported by the estimators, so can't be that unless we redesign everything which I don't want to do.

?? I thought it was, since you've refactored the base class?
I looked at it and there are some minor issues, I'm making a PR with suggested fixes.

Default should be nested data frame as it is now. I don't mind changing it to default to that or switch on user parameter I suppose although it will get messy with exceptions when data doesn't allow the choice.

I don't mind, but all inputs should be supported.

@TonyBagnall, I just checked BaseClassifier, and I think indeed all inputs are supported after your refactor!
While looking, I found a few bugs and inaccurate docstrings, I've fixed them here:
https://github.com/alan-turing-institute/sktime/pull/1804

I think this is still a valuable contribution, @TonyBagnall - no need to close.
Consider handing it over if you don't have the time to work on it.
Or, if my review is a deal breaker, kindly let me know. I thought it was a small but important thing to ask that we are consistent across the package in our naming of data formats.

closed because conflicted, I will revisit at a later date

reopened, as next in line. @fkiraly "Can you kindly use the same strings?" I thought the strings were numpy3D and numpy2D, and I use these (albeit also allowing lower case and np3d as an alternative)
return_data_type = return_data_type.strip().lower()
if return_data_type == "numpy2d" or return_data_type == "np2d":

added numpyflat as in input option for 2d numpy


update kb Dockerfile

What type of PR is this?
/kind bug

What this PR does / why we need it:
kb v0.5.0 requires opencv to run
Which issue(s) this PR fixes:

Fixes #320

As a public dependency, it is recommended to extract OpenCV2  into the reqirements.txt.

This dependency should be fixed to be imported when it is needed(Such as the example of Multi-Edge Inference).

OK, I close this PR.