Structured result support

This PR adds support for handling structured results from function execution.
An example run:
kubeval-failure $ ~/go/bin/kpt fn render --results-dir /tmp/results
Package ".":

[RUNNING] "gcr.io/kpt-fn/kubeval:v0.1"
[FAIL] "gcr.io/kpt-fn/kubeval:v0.1"
  Stderr:
    "[ERROR] selector is required in object 'apps/v1/Deployment//nginx-deployment' in file resources.yaml in field selector"
    "[ERROR] template is required in object 'apps/v1/Deployment//nginx-deployment' in file resources.yaml in field template"
    "[ERROR] Invalid type. Expected: [integer,null], given: string in object 'apps/v1/Deployment//nginx-deployment' in file resources.yaml in field spec.replicas"
    "[ERROR] Failed to parse raw kubeval output:"
    ...(5 line(s) truncated)
  Exit Code: 1

Results saved successfully at "/tmp/results/results.yaml".

Results output:
$ cat /tmp/results/results.yaml
items:
    - image: gcr.io/kpt-fn/kubeval:v0.1
      results:
        - message: selector is required
          severity: error
          resourceRef:
            apiVersion: apps/v1
            kind: Deployment
          field:
            path: selector
          file:
            path: resources.yaml
        - message: template is required
          severity: error
          resourceRef:
            apiVersion: apps/v1
            kind: Deployment
          field:
            path: template
          file:
            path: resources.yaml
        - message: 'Invalid type. Expected: [integer,null], given: string'
          severity: error
          resourceRef:
            apiVersion: apps/v1
            kind: Deployment
          field:
            path: spec.replicas
          file:
            path: resources.yaml
        - message: |
            Failed to parse raw kubeval output:
            Unexpected token E in JSON at position 0

            ERR  - stdin: Failed initializing schema file:///tmp/master-standalone-strict/custom-custom-v1.json: open /tmp/master-standalone-strict/custom-custom-v1.json: no such file or directory
          severity: error
          resourceRef:
            apiVersion: custom.io/v1
            kind: Custom
          file:
            path: resources.yaml
            index: 1

This PR adds support for handling structured results from function execution.
An example run:
kubeval-failure $ ~/go/bin/kpt fn render --results-dir /tmp/results
Package ".":

[RUNNING] "gcr.io/kpt-fn/kubeval:v0.1"
[FAIL] "gcr.io/kpt-fn/kubeval:v0.1"
  Stderr:

This should populate the results section in the output

"[ERROR] selector is required in object 'apps/v1/Deployment//nginx-deployment' in file resources.yaml in field selector"
"[ERROR] template is required in object 'apps/v1/Deployment//nginx-deployment' in file resources.yaml in field template"
"[ERROR] Invalid type. Expected: [integer,null], given: string in object 'apps/v1/Deployment//nginx-deployment' in file resources.yaml in field spec.replicas"
"[ERROR] Failed to parse raw kubeval output:"
...(5 line(s) truncated)

Exit Code: 1
Results saved successfully at "/tmp/results/results.yaml".

Results output:

```shell
$ cat /tmp/results/results.yaml
items:

This needs to be FunctionResultsList

- image: gcr.io/kpt-fn/kubeval:v0.1
  results:
    - message: selector is required
      severity: error
      resourceRef:
        apiVersion: apps/v1
        kind: Deployment
      field:
        path: selector
      file:
        path: resources.yaml
    - message: template is required
      severity: error
      resourceRef:
        apiVersion: apps/v1
        kind: Deployment
      field:
        path: template
      file:
        path: resources.yaml
    - message: 'Invalid type. Expected: [integer,null], given: string'
      severity: error
      resourceRef:
        apiVersion: apps/v1
        kind: Deployment
      field:
        path: spec.replicas
      file:
        path: resources.yaml
    - message: |
        Failed to parse raw kubeval output:
        Unexpected token E in JSON at position 0

        ERR  - stdin: Failed initializing schema file:///tmp/master-standalone-strict/custom-custom-v1.json: open /tmp/master-standalone-strict/custom-custom-v1.json: no such file or directory
      severity: error
      resourceRef:
        apiVersion: custom.io/v1
        kind: Custom
      file:
        path: resources.yaml
        index: 1


Develop a list of best practices

There is a need to develop a list of best practices for use of JSON 5. This issue is a place to organize potential best practices.

From JSON 4 data:
Versions Affected: <= 2.10

Use of <= makes it impossible to identify which version the vulnerability was fixed in. It's not clear if the vulnerability is fixed in 2.11, 2.10.1, or any other version.
The JSON 5 format can be used to identify both the vulnerable and fixed version spaces. Examples need to be produced to illustrate the best practice here.

We have developed a Vulnogram user guide which helps CNAs understand best practices for creating CVE Records. Also, the schema includes descriptions for most fields. We should be covered for now. Can create issues for missing documentation as we find it.


Flatpickr plugin makes a mesh of dates when application default locale is set to nl_NL (Netherlands)

Describe the bug
After I have picked a date in the flatpickr popup it is directly shown in the edit-view and reloaded list-view as  a completely different date in the past. Does it have to do with my application default locale 'nl'  (the Netherlands)?
Because if I remove this application default locale and instead use the data formating options in my edit view (d-m-Y) every date  entered shows up fine.
To Reproduce
Expected behavior
Displaying dates correctly in the choosen locale setting
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

Might be a duplicate of #2856

To clarify, could you show us a screenshot of the configuration of the flatpickr widget?

the day seems to be transformed to the last 2 digits of the year  2024-11-04 becomes 20-11-2004.
It does not matter if I use format d-m-Y or use an emtpy format or use the locale 'de' or 'nl'

When I create a new record in my table and select a date with datapicker it goes allright. But I reslect that record and update the date then the date is screwed up!


Error: Required nested sequences need to be implemented!

Hello,
I've got the following error while trying to parse xsd converted from dtd.
Required nested sequences need to be implemented!
I'm not sure what it means. What am I thinking is that your code doesn't work yet with required nested sequences or maybe there is something wrong with the xsd schema?
Thank you.
Have a great day.

Hi @svrakata,
You are correct.  Required nested sequences are not yet implemented.  Would you be able to share your schema for me to use in testing?
Thanks,
Andy

Hello again,
Here are the schemas.
They are parsed from dtd to xsd via Visual Studio 2017
Sorry for delaying my answer.
Cheers.

Thanks @svrakata.  I’ll try to include this in the next release.


FSMV1.1 release - Updated localization keys for English

FSMV1.1 release - Updated localization keys for English

FSMV1.1 release - Updated localization keys for Hindi

FSMV1.1 release - Updated localization keys for English


maxPatternLength does not error in 3.3.0

I'm on Fusejs 3.3.0, and I've noticed that the maxPatternLength isn't doing anything.  My length is the set to 32.  But when I put in term that are longer, it will still run the search, whereas in 3.2.1 it would error stating that the "Pattern length is too long".
If I navigate to: http://fusejs.io/ it appears to be running version 3.2.1.  If I leave the maxPatternLength to be 32 and search for:
"oldoldoldoldoldoldoldoldoldoldoldold" (36 chars) I get the above exception text in the result box.
If I use my code which is on version 3.3.0 has the maxPatternLength set to 32, and search for the same term ("oldoldoldoldoldoldoldoldoldoldoldold") it returns a match instead of throwing the exception.

Will take a look. Thanks for flagging!


“Since” column on unicode_texas page is useless

See the table.
For ⁇ !!, ≤, ≥, ≠ it says “v6.c”, but that's less than awesome because rakudo version 2015.12 (which implements v6.c) does not have them. There must be a way to indicate this somehow.

Since it wasn't actually tested in v6.c it should say either v6.c.1 or v6.d

Neither of those are appropriate either, since those spec's haven't been decided on.
I would list, for now, the rakudo compiler version they first became available in; we can replace that with the spec once it's decided. I recommend, e.g.
[Starting in Rakudo-2017.02]

Since we don't have this information anywhere else, and we don't have it here, I'm going to remove it. We can address it when we have a unified plan for how to address multiple versions of the spec/compiler adherence to same.


Subscriber struct contains outdated fields

It appears as though the Paypal docs and the subscriber struct don't line up anymore. Am I allowed to amend this and PR?
https://developer.paypal.com/docs/api/subscriptions/v1/#subscriptions_create!path=subscriber&t=request
https://github.com/plutov/paypal/blob/master/types.go#L1255

@Roolps for sure! Send us a PR and happy to help get it reviewed and updated


gitlab-ce: use sops to store GitLab rails secrets in git

GitLab encrypts several types of secrets, including runner tokens, in its postgres database.  For k8s installations the keys to decrypt those values live in in a k8s secret at app-gitlab-ce-rails-secret.
During cluster rebuilds, app-gitlab-ce-rails-secret is reinitialized and the key to unlock tokens in the restored database are lost.  Store app-gitlab-ce-rails-secret in git to prevent that.

bors r+


Graphql: Return 400 instead of 500 when client sends incorrect request in json

Currently Armeria returns a 500 error when a client sends in either invalid json or invalid graphql request.
I agree with what @jrhee17 wrote in https://github.com/line/armeria/pull/4069#discussion_r801222068 , the server should return 400.

Thanks for reporting the bug. As a workaround, we can customize the response status using ServerErrorHandler.


feat: rewrite the scanner in C

Most upstream grammars are now in C as well, it helps with portability & ease of compilation - thanks!

Except from that, it's been working well for me for two weeks now
@zealot128 Do you have any thoughts?

C/C++ is not my strong suite - I just originally forked the repo to get some grammar changes into it.
But since @mclaughlinconnor has no objections and it worked great for them, I see no reason not to merge.
Thanks for the work!

thanks!


input output details on a worker

on the jobs page open a pipeline and press on one of the workers to see its logs,
in the log page there is a "input output details" tab. when pressing on it, the page crashes

On opening an issue, please use the bug template on

Which version you using, I can't reproduce on 1.1.149v


Improvement of extension

Implemented a correct structure of Behat extension.
Started to follow the "Single responsibility" principle.
Improved the logic.
Added configurations to behat.yml.
Added autoloader for all contexts.
Other enhancements.

The setOptions and setNamespaces should not probably merge namespaces and options, but rather replace, I'll create a followup ticket for that, otherwise looks great, thanks for your valuable contributions!


Foundation for Developer Onboarding document

Just starting out with a document to guide in onboarding new developers.
New team members need to be given access to various platforms and projects, and need to be guided through tasks like creating .env files and using the Linkta docs.

Fixes #N/A
Type of change

[x] 📝 Documentation update

How Can this be tested? Testing Plan to review this PR

N/A
Checklist:

[x] I have ensured that my pull request title is descriptive.

Thank you Reland, This is helpful!

@tedgusek @chrisr0892 @lomeintheory @LucieSeidler And everyone else,
This doc now includes a section about setting up local environment variables.
One part is not quite accurate, but it will become accurate after one more PR! It's a long story...


fix: subscriptions to move to dedicated API

Problem description

subscriptions endpoints don't follow the guidelines in https://github.com/camaraproject/Commonalities/blob/r0.4.0-rc.2/documentation/API-design-guidelines.md#resource-based-explicit-subscription
it must be a separate API with -subscriptions in the name
Expected behavior

Create new API,  connectivity-insights-subscriptions. Also note further comments from the PR review
Alternative solution

Additional context

fixed in a23e245 and 66d63bd

I have made a number of suggestions in PR #67  to align event types and scope names with Design Guidelines ch. 11 & 12.
I hope there is no mistake.


Inject, add support for scala-2.12.

Problem
We need to add crossScalaVersions statements to individual
modules while dependencies add support for
scala-2.12.
Additionally some modules in finatra depend on inject-* modules
making this a good base for other modules to add support.
Solution
Add crossScalaVersions statements to inject-* modules
Result
Consumers are able to pull in support for scala-2.12 as well as
finatra modules can begin adding support.
Related: #323, #376

THis was done hastily, will resubmit


Issue with compute_metric plugin used for metric node_volume_total_data

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please let us know in a comment

Problem
We can see that dashboard panels using node_volume_total_data show "No data".
The metric node_volume_total_data is not reported by harvest only node_volume_total_ops is reported:
curl -s -X GET http://localhost:12993/metrics | grep -i node_volume_total
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n02"}<PHONE_NUMBER>7150991
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n05"}<PHONE_NUMBER>0253968
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n07"}<PHONE_NUMBER>7719323
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n01"}<PHONE_NUMBER>7249043
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n04"}<PHONE_NUMBER>1885253
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n06"}<PHONE_NUMBER>2532461
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n03"} 3382.4026900082085
node_volume_total_ops{cluster="xxx33",datacenter="Prod",node="xx33n08"}<PHONE_NUMBER>8664705

The ZAPI config file contains the needed entries:
cat conf/zapiperf/cdot/9.8.0/volume.yaml

name:                     Volume
query:                    volume
object:                   volume

instance_key:             uuid

counters:
 - avg_latency
 - instance_name
 - ...
 - parent_aggr           => aggr
 - read_data
 - read_latency
 - read_ops
 - total_ops
 - vserver_name          => svm
 - write_data
 - write_latency
 - write_ops

plugins:
 - Aggregator:
   # plugin will create summary/average for each object
   # any names after the object names will be treated as label names that will be added to instances
   - node
 - Volume:
     include_constituents: false
 - MetricAgent:
     compute_metric:
       - total_data ADD read_data write_data

Looks like the compute_metric plugin is not working for some reason.
All the configuration YAML files are default from the RPM of ver 24.02.0-1.
There are no messages related to this issues in the poller log file.
Any advise how to troubleshoot this issue?
Configuration
./bin/harvest doctor
Admin:
    httpsd:
        listen: localhost:8887
Tools:
Exporters:
    prometheus:
        exporter: Prometheus
        local_http_addr: <IP_ADDRESS>
        port_range: 12990-13050
        sort_labels: true
Defaults:
    collectors:
        - Zapi
        - ZapiPerf
    use_insecure_tls: true
    exporters:
        - prometheus
Pollers:
    unix:
        datacenter: local
        addr: -REDACTED-
        collectors:
            - Unix
        exporters:
            - prometheus
    xxx33:
        datacenter: Prod
        addr: -REDACTED-
        auth_style: basic_auth
        username: -REDACTED-
        password: -REDACTED-
        exporters:
            - prometheus

Poller
xxx33
Version
24.02.0-1
Poller logs
No response
OS and platform
Red Hat Enterprise Linux release 8.6 (Ootpa)
ONTAP or StorageGRID version
NetApp Release 9.12.1P11
Additional Context
No response
References
No response

Thanks for the bug report @munokar. We discovered the same a couple of weeks ago and it is fixed in nightly.
The changes are all in the template and dashboards which means you can apply them without taking a nightly, but taking the nightly is probably easier.

Thanks for the quick feedback.
Unfortunately I only searched for the full metric name ( node_volume_total_data ) so I didn't find the existing issue.
After updating the template file the to newest one it is working.
Interessting to see that the order of the elements in the list seems to make a difference since I could not identfy any other change in the template file.

No worries! Appreciate you opening the issue and glad it's working after updating. \o/
That's right, the plugin order matters.
The order that plugins are listed in the template file is the order the plugins run. The earlier bug was caused by the Aggregator plugin running before the compute_metric one. That's a problem because the compute_metric plugin is only working on the volume metrics data, not the aggregated-by-node metrics data.
Changing the order means that the compute_metric plugin creates the volume_total_data metric before the Aggregator runs. That way the Aggregator will also aggregate volume_total_data and make it available at the by-node level as node_volume_total_data.

Thanks for the perfect explanation and the very quick response!


No stable tag for PHP 8

Whilst I understand the next major version of Deployer will support PHP 8, it would greatly ease upgrading if the 6.x branch also supported PHP 8. If it technically does already, adding the tag should be trivial. Having the tag means users can upgrade to PHP 8 and later upgrade Deployer, instead of having to do both at the same time. Therefore having a stable PHP 8 tag now is important, rather than waiting for Deployer 7 release.

This is make sense. Will release patch for php8.

Any update on this one yet?

Done. Released v6.8.1.

@antonmedv there is no such version in releases.

Seems he forgot to push the tag...

Yes, I did it, but have to revoke the release and deleted the tag as it not that simple. Due to:
https://github.com/deployphp/deployer/issues/2348
https://github.com/deployphp/deployer/issues/2352
Please follow instructions here: https://github.com/deployphp/deployer/discussions/2351

We were 'lucky' enough to update deployer in the short timeframe that 6.8.1 existed. Because of that, I just discovered that 6.8.1 (i.e. commit 042d1006906002ef580e5453894c4fd47180701e) is based on the commit of release 6.4.5 (its parent is commit be14a4f757cff060ebd3df661df12687e5fbc25e) rather than on 6.8.0. That caused all the features and bugfixes added between 6.4.5 and 6.8.0 to be 'disappeared' in 6.8.1.
With 6.8.1 being retracted that is not a big issue anymore, but branch 6.x still points to the commit that was previously release 6.8.1. I think it would be a good idea to reset that branch to the commit that is version 6.8.0, to not have these 'missing features/bugfixes' problems again when ever trying to release a version 6.8.2.

We were 'lucky' enough to update deployer in the short timeframe that 6.8.1 existed. Because of that, I just discovered that 6.8.1 (i.e. commit 042d1006906002ef580e5453894c4fd47180701e) is based on the commit of release 6.4.5 (its parent is commit be14a4f757cff060ebd3df661df12687e5fbc25e) rather than on 6.8.0. That caused all the features and bugfixes added between 6.4.5 and 6.8.0 to be 'disappeared' in 6.8.1.
With 6.8.1 being retracted that is not a big issue anymore, but branch 6.x still points to the commit that was previously release 6.8.1. I think it would be a good idea to reset that branch to the commit that is version 6.8.0, to not have these 'missing features/bugfixes' problems again when ever trying to release a version 6.8.2.

Will do please open new issues to not forget.

Will do please open new issues to not forget.

Updated to 4e243a6 (HEAD -> 6.x, tag: v6.8.0, origin/6.x) Release v6.8.0


SMS not sent

Ayo,
So basicly, i tryed to phone verify, with 5sim & capmonster, and it just not sent

Idk if you can fix, or idk, i tried with proxy, without, changing country/operator
if u have any question, i will ansanwer faster in discord : Not Sysy's#6700
Thx :)

Do you use the dev branch version or the one in the main branch? I recommend you to try out the dev branch version if you are not doing so already, otherwise the tokens could be the issue too because in general fivesim, vaksms and smshub work fine.

Solved by using vak-sms


ktopt_planning_context.hpp is empty

Right now, all the implementation is in the src/ktopt_planning_context.cpp file, whereas its corresponding header file is empty. Consider moving all the declarations to the header file.

Wait, I was looking at the ktopt_interface.hpp file. My bad.


curl: (77) Problem with the SSL CA cert (path? access rights?)

I've got error when I run test-curl containers.
curl: (77) Problem with the SSL CA cert (path? access rights?)
I've solved this problem adding the following to Dockerfile
RUN apt-get install --no-install-recommends --no-install-suggests -y ca-certificates
Hope to be helpful.

Thanks, it was helpful!
Personally, I would add the following. To get the facebook example working, you need to follow these steps:

Change the following line of Dockerfile:
RUN apt-get install --no-install-recommends --no-install-suggests -y curl to
RUN apt-get install --no-install-recommends --no-install-suggests -y curl ca-certificates
Delete the current docker image as follows:
docker rmi $(docker images test-curl -q)
Build the image from Dockerfile again:
docker build . -t test-curl

It works. Thanks.


On DEV search-api results for c87fa4ec3aa87af19a9ad29d2e2b4c5c contains a null Dataset

Dataset b3d907a4b8a19e79fe095410bdee30ea is the one shows as null in this collection's datasets field while it exists in Neo4j and can be retrieved successfully via entity-api. The problem is during reindex, one of the descendants (00318be0b7cfa3c6ed7fbeab08fe700b) of this dataset doesn't have a record in the UUID MySQL database. As a result, entity-api returns 404. And this further caused an exception during the generate_doc() call on this b3d907a4b8a19e79fe095410bdee30ea dataset, which didn't generate a document of this dataset. Thus null is added to the Collection.datasets field, causing the portal-ui rendering error.

The fix is to handle the exception properly and skip the dataset with missing descendant record in MySQL. Essentially it's a data issue that should be addressed in the database level, but should not cause UI rendering to fail.


Invalid CSS is generated for {screen}:display rules

First of all, thanks a lot for making this library! It's very convenient and fun to use
I noticed an issue when using classes for responsive display properties, such as sm:block, md:table-cell, etc.
For example, sm:flex generates the following CSS:
@media (min-width: 640px) {

  .sm\:flex {
    display: sm:flex;
  }

}

Which should be:
@media (min-width: 640px) {

  .sm\:flex {
    display: flex;
  }

}

Thank you for reporting the issue.

@jeisses This bug seems to be a duplicate of #55 which was fixed in v0.0.3 and which has a test in the codebase already:
https://github.com/green-coder/girouette/blob/46ca014ebdc95a9edbcc46ae4d1f00d261bf683a/lib/girouette/test/girouette/tw/layout_test.cljc#L28-L32
Which version do you use?

I also tested sm:block and md:table-cell, they all work as intended.
Closing as invalid.


KEGG annotations

Hello,
I'm interested in KEGG annotations for HiFi reads. I'm aware that the MEGAN6 community edition used in Taxonomic-Functional-Profiling-Protein workflow doesn't provide these, whereas paid MEGAN6 ultimate edition does. Is there an alternative for this?
Many thanks
Amit

Hi Amit,
KEGG requires licensing, and this is built in with the purchase of MEGAN6 UE. I am unfortunately not aware of any simple alternatives for getting KEGG annotations. If you happen to discover an alternative, please post it here.

Thank you @dportik for your reply! For sure, I'll post here if I come across any such alternative.

I wanted to provide a quick update.
I was able to get KEGG annotations using the command line tools from MEGAN-UE, in conjunction with the mapping file for MEGAN-UE. It did not require a license (likely because we aren't querying the KEGG database).
I've created a separate workflow for this in the Taxonomic-Profiling-Diamond-Megan repo. To use it, you will need to download MEGAN-UE to obtain the binaries sam2rma and rma2info, along with the current MEGAN-UE mapping file.

Thank you for the update. I'll probably require it in the near future, at that time I'll try your workflow.


Adding caisy as cms to the docs

Hi there, thank you for maintaining Astro and its documentation! I love the framework! As a founding member of caisy, I would like to submit some additions to your docs regarding our integration with Astro.
What kind of changes does this PR include?
Changes to the docs content.
Description
Adding caisy as cms/service to the docs.
Our homepage: https://caisy.io

What does this PR change?
Adding caisy to the overview of cms's and adding a new subpage for caisy
Did you change something visual?
Before:

After:

Thanks for reviewing and best regards,
Moritz

Thanks for this CMS guide @MoritzGruber!
I've restructured the guide to better match our other CMS guides, and the layout we've decided on, as well as mark it as a stub, to let people know we're happy for someone to expand it into an in-depth guide!

@MoritzGruber Thank you so much for this contribution! We love having external contributions, and want as many people to enjoy working with Astro as possible!  :raised_hands:
Great job adjusting this to our format, @TheOtterlord ! I am happy to approve the English version immediately!
@MoritzGruber - The German page needs to be reworked to match the English before we can merge this. OR, you are welcome to remove the German for now, and we'll merge the English immediately. Then, you can PR a German translation. The choice is yours! Just let us know!

@sarah11918, I've removed the German version for now. I think it's simpler to move this forward without it first. I'll create another PR for the translation. Thanks for the hint.

Great, @MoritzGruber, then we're happy to add you (and Caisy) to Team Docs!  :partying_face:


refactor: remove unused code and fix typos

Purpose or design rationale of this PR
PR title
Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:

[x] refactor: A code change that doesn't fix a bug, or add a feature, or improves performance

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 52.65%. Comparing base (a22aa8d) to head (45d2dd1).

Additional details and impacted files
@@                Coverage Diff                @@
##           feat/upgrade4    #1470      +/-   ##
=================================================
+ Coverage          52.57%   52.65%   +0.08%
=================================================
  Files                156      156
  Lines              12444    12416      -28
=================================================
- Hits                6542     6538       -4
+ Misses              5341     5322      -19
+ Partials             561      556       -5

Flag
Coverage Δ

database
42.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


feat: add [FIRST | AFTER column_name] support in MySQL alter table

As per MySQL doc, the ALTER TABLE Statement has the definition:
ALTER TABLE tbl_name
    [alter_option [, alter_option] ...]
    [partition_options]

alter_option: {
    table_options
  | ADD [COLUMN] col_name column_definition
        [FIRST | AFTER col_name]
  | ADD [COLUMN] (col_name column_definition,...)
  | ADD {INDEX | KEY} [index_name]
        [index_type] (key_part,...) [index_option] ...
  | ADD {FULLTEXT | SPATIAL} [INDEX | KEY] [index_name]
        (key_part,...) [index_option] ...
  | ADD [CONSTRAINT [symbol]] PRIMARY KEY
        [index_type] (key_part,...)
        [index_option] ...
  | ADD [CONSTRAINT [symbol]] UNIQUE [INDEX | KEY]
        [index_name] [index_type] (key_part,...)
        [index_option] ...
  | ADD [CONSTRAINT [symbol]] FOREIGN KEY
        [index_name] (col_name,...)
        reference_definition
  | ADD [CONSTRAINT [symbol]] CHECK (expr) [[NOT] ENFORCED]
  | DROP {CHECK | CONSTRAINT} symbol
  | ALTER {CHECK | CONSTRAINT} symbol [NOT] ENFORCED
  | ALGORITHM [=] {DEFAULT | INSTANT | INPLACE | COPY}
  | ALTER [COLUMN] col_name {
        SET DEFAULT {literal | (expr)}
      | SET {VISIBLE | INVISIBLE}
      | DROP DEFAULT
    }
  | ALTER INDEX index_name {VISIBLE | INVISIBLE}
  | CHANGE [COLUMN] old_col_name new_col_name column_definition
        [FIRST | AFTER col_name]
  | [DEFAULT] CHARACTER SET [=] charset_name [COLLATE [=] collation_name]
  | CONVERT TO CHARACTER SET charset_name [COLLATE collation_name]
  | {DISABLE | ENABLE} KEYS
  | {DISCARD | IMPORT} TABLESPACE
  | DROP [COLUMN] col_name
  | DROP {INDEX | KEY} index_name
  | DROP PRIMARY KEY
  | DROP FOREIGN KEY fk_symbol
  | FORCE
  | LOCK [=] {DEFAULT | NONE | SHARED | EXCLUSIVE}
  | MODIFY [COLUMN] col_name column_definition
        [FIRST | AFTER col_name]
  | ORDER BY col_name [, col_name] ...
  | RENAME COLUMN old_col_name TO new_col_name
  | RENAME {INDEX | KEY} old_index_name TO new_index_name
  | RENAME [TO | AS] new_tbl_name
  | {WITHOUT | WITH} VALIDATION
}

partition_options:
    partition_option [partition_option] ...

partition_option: {
    ADD PARTITION (partition_definition)
  | DROP PARTITION partition_names
  | DISCARD PARTITION {partition_names | ALL} TABLESPACE
  | IMPORT PARTITION {partition_names | ALL} TABLESPACE
  | TRUNCATE PARTITION {partition_names | ALL}
  | COALESCE PARTITION number
  | REORGANIZE PARTITION partition_names INTO (partition_definitions)
  | EXCHANGE PARTITION partition_name WITH TABLE tbl_name [{WITH | WITHOUT} VALIDATION]
  | ANALYZE PARTITION {partition_names | ALL}
  | CHECK PARTITION {partition_names | ALL}
  | OPTIMIZE PARTITION {partition_names | ALL}
  | REBUILD PARTITION {partition_names | ALL}
  | REPAIR PARTITION {partition_names | ALL}
  | REMOVE PARTITIONING
}

key_part: {col_name [(length)] | (expr)} [ASC | DESC]

index_type:
    USING {BTREE | HASH}

index_option: {
    KEY_BLOCK_SIZE [=] value
  | index_type
  | WITH PARSER parser_name
  | COMMENT 'string'
  | {VISIBLE | INVISIBLE}
}

table_options:
    table_option [[,] table_option] ...

table_option: {
    AUTOEXTEND_SIZE [=] value
  | AUTO_INCREMENT [=] value
  | AVG_ROW_LENGTH [=] value
  | [DEFAULT] CHARACTER SET [=] charset_name
  | CHECKSUM [=] {0 | 1}
  | [DEFAULT] COLLATE [=] collation_name
  | COMMENT [=] 'string'
  | COMPRESSION [=] {'ZLIB' | 'LZ4' | 'NONE'}
  | CONNECTION [=] 'connect_string'
  | {DATA | INDEX} DIRECTORY [=] 'absolute path to directory'
  | DELAY_KEY_WRITE [=] {0 | 1}
  | ENCRYPTION [=] {'Y' | 'N'}
  | ENGINE [=] engine_name
  | ENGINE_ATTRIBUTE [=] 'string'
  | INSERT_METHOD [=] { NO | FIRST | LAST }
  | KEY_BLOCK_SIZE [=] value
  | MAX_ROWS [=] value
  | MIN_ROWS [=] value
  | PACK_KEYS [=] {0 | 1 | DEFAULT}
  | PASSWORD [=] 'string'
  | ROW_FORMAT [=] {DEFAULT | DYNAMIC | FIXED | COMPRESSED | REDUNDANT | COMPACT}
  | SECONDARY_ENGINE_ATTRIBUTE [=] 'string'
  | STATS_AUTO_RECALC [=] {DEFAULT | 0 | 1}
  | STATS_PERSISTENT [=] {DEFAULT | 0 | 1}
  | STATS_SAMPLE_PAGES [=] value
  | TABLESPACE tablespace_name [STORAGE {DISK | MEMORY}]
  | UNION [=] (tbl_name[,tbl_name]...)
}

partition_options:
    (see CREATE TABLE options)

issue reported in #1172 , this PR  add [FIRST | AFTER column_name] support in MySQL alter table

@andygrove Hi, could you please help approve the waiting workflow?

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

130 of 130 (100.0%)  changed or added relevant lines in 6 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.07%) to 87.94%

Files with Coverage Reduction
New Missed Lines
%

src/ast/ddl.rs
2
84.89%

Totals

Change from base Build<PHONE_NUMBER>:
0.07%

Covered Lines:
20614

Relevant Lines:
23441

💛  - Coveralls

Had some fix and running workflow locally with result:
codestyle
cargo +nightly fmt -- --check --config-path <(echo 'license_template_path = "HEADER"')
Warning: Unknown configuration option `license_template_path`

lint
cargo clippy --all-targets --all-features -- -D warnings
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s

compile
cargo check --all-targets --all-features
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s

test
all passed

@andygrove sorry but could you please help approve the workflow again?

Sorry for the delay in reviewing


[RFC] Passing hyperparameters to solvers (but also for i.e. dataset)

When running with the CLI a solver with a specific set of hyperparameters (say a learning rate or the l1_ratio.) Currently BenchOpt syntax for the CLI is a bit heavy handed:
benchopt run -s MySolver[learning_rate=0.1, l1_ratio=3.]

Brackets [, ] syntax is not super convenient in addition in zsh for instance it might throw:
zsh: no matches found: MySolver[learning_rate=0.1, l1_ratio=3.]

Because it uses brackets (solution is to add quotes).
Config files can help to a certain extent but it would be cool if the CLI allowed argparse-like syntax for:
benchopt run -s MySolver --learning_rate 0.1 --l1_ratio=3.

Because everyone is kinda used to the above (see e.g. click or argparse or fire)
Solver hyperparameters CLI is dynamically generated from the parameters attribute of the given solver.
Ofc if one gives several solvers it complicates stuff.
I am nitpicking a bit maybe current system is much safer and both syntax are close. What do you think ?

Also the syntax for the yaml config follows the CLI, aka MySolver[learning_rate=0.1, l1_ratio=3.], which does not follow mlcollections standard of using nested dicts (also it would allow nested params aka passing a function with its args to a solver)


Add any key

Fixes #503, #565, #498

There's a much faster way to implement this.

Also this doesn't trigger hats in the right order.

There's a much faster way to implement this.

I thought about counting the number of keys which were down, but a) that would fail if the window was blurred and b) it's not how you did it in scratch-flash. Clearly I guessed wrong! :P

a) that would fail if the window was blurred

If by "fail" you mean "continue to report that a key is pressed", your solution does that too. If you mean "forever become offset by 1", that's why my code checks the state for that individual key before in/decrementing.
Generally for scratch-flash I did the obviously correct thing and for phosphorus I do the non-obvious error-prone faster thing and try not to make any errors :P

If you mean "forever become offset by 1", that's why my code checks the state for that individual key before in/decrementing.

Yes, that's what I meant. You're very clever.


Changed the crd-install job to use the  flag in its  command.

Checklist

[ ] Update changelog in CHANGELOG.md.
[ ] Make sure values.yaml and values.schema.json are valid.

Update your branch please, you are one version behind in your changelog.


Cannot connect to trino version 468 and Openmetadata 1.6.1

Affected module
Error when create a connection to Trino.
Describe the bug
I follow the test deployment from Quick start intruction. I using Openmetadata version 1.6.1.
I already success connect to postgresql and sql server. But
when I create a trino connection to , I got an error :
test-connection-Trino-1MEORm0f Failed to trigger workflow due to airflow API returned Internal Server Error and response {"error": "Error running automation workflow due to ['NoneType' object has no attribute 'root'] "}
To Reproduce

create a connection to trino with all information, verify ssl = false

Version:

Trino version: 468
Python version: default fof docker images
OpenMetadata version: 1.6.1
OpenMetadata Ingestion package version: docker.getcollate.io/openmetadata/ingestion:1.6.1
Additional context
Trino and Openmetadata in the same docker network.

I had view log and saw this:
File "/home/airflow/.local/lib/python3.10/site-packages/openmetadata_managed_apis/api/routes/run_automation.py", line 65, in run_automation
execute(automation_workflow)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/automations/runner.py", line 43, in execute
return run_workflow(automation_workflow.request, automation_workflow, metadata)
File "/usr/local/lib/python3.10/functools.py", line 889, in wrapper
return dispatch(args[0].class)(*args, **kw)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/automations/runner.py", line 69, in _
connection = get_connection(request.connection.config)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/connections.py", line 54, in get_connection
return get_connection_fn(connection)(connection)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/database/trino/connection.py", line 143, in get_connection
connection.connectionArguments.root["verify"] = {"verify": connection.verify}
AttributeError: 'NoneType' object has no attribute 'root'

And I fixed this by this setup

@ulixius9 shouldn't this be a default config


Apr 699 fat indexer stub additions

The principle of this is: if you set the variable RoatpApiAuthenticationApiBaseAddress to 'stub' in your local, it will get these stubbed records instead of using the roatp-service-api
Once the story is in, I'll update the readme to explain this
Ewan, I've included you more to keep you in the loop that 'review' as such

This needs to be based on #103 instead of master


pending open pr

the issue was assigned to me
please merge this open pr :
https://github.com/Rakesh9100/Beautiify/pull/1331

@sanjay-kv please merge this pr


OTEL metric processing (UX)

Specifically for OTEL metric collection based on our custom prometheus CRD service discovery, within reason we want to :

[ ] Make job names consistent with prometheus-agent
[ ] Make certain labels consistent with prometheus-agent
[ ] Existing opni grafana dashboards should work with OTEL capability

We also want to make sure which metrics are available for scraping to be compatible with the CRD definitions:

[ ] Mount secrets on demand for TLS communication based on promCRD definitions

Note to self:

relabeling configs are important and needed to be applied, since alot of exporters manage private labels that are important
looks like we are not discovering the full set of service monitor targets, need to look more into that
there are issues specifically with node-exporter on RKE2 that appear to prevent static_configs from working correctly.

More notes to self :

instance will always be wrong in the current iteration, but net_host_name  will be accurate

the mismatch between job labelling is the main cause behind some dashboards not working

The root of almost all the current issues surrounding labels on the UX side seems to be the specific way prometheus operator does "implicit" labeling, so I'll have to take an in depth logic of that tomorrow

Fixed most of the dashboards and got auth fully working. Still some labels to clean up.

Dashboard is working now. Minus some issues surrounding data source scrape interval and rate sampling in the infrastructure dashboard. We have essentially close to full parity with prometheus operator.
Current memory savings on 3 node clusters is the order of ~1.2-1.6GB memory depending on the distribution.
Overhead of the OTEL collector is between 300-700MB memory again depending on the distribution. K3s and RKE2 are the worst performers for both OTEL & prometheus agent. I believe there are some issues with underlying metric exporters, and I am doing some additional investigation to be able to report my findings.


feat(spring): complete job executor config options in app.properties

Hi,
I did some follow-up work on pull request #299 and completed the job executor configuration options.
All job acquisition properties listed here are now available as Spring Boot application properties.
Thank you for considering my pull request.
Jan

Hi Jan,
I will have a look and return back to you until end of the week.
Best regards,
Yana

Related to CAM-9212

Hi Jan,
I merged the additional properties.
Do you want to add them in the documentation as well?
https://docs.camunda.org/manual/develop/user-guide/spring-boot-integration/configuration/#camunda-engine-properties
Best regards,
Yana

Hi Yana,
thank you very much.
Yes, I added the additional properties to the documentation and created a PR: https://github.com/camunda/camunda-docs-manual/pull/187
Best regards,
Jan

Hi Jan,
Great, hereby I am closing this PR and I will have a look at the docs.
Best regards,
Yana


no active keys if you don't have an eldest

should fix CORE-2152

r? @oconnor663

I think the first commit there is accidentally included.

I think the rightiest thing to do will be to make GetComputedKeyFamily check GetEldestKID. There will be other errors caused by this caching issue.


SELECT * FROM t_report WHERE date_part('year', report_time) = 2024

"SELECT * FROM t_report WHERE date_part('year', report_time) = 2024
How is this SQL implemented?"

It's easy to add new functions to the library. Here's an example that should work:
public class DatePart extends AbstractTypeConvertingFunction<Date, Integer, DatePart> {

    private final String part;
    protected DatePart(String part, BindableColumn<Date> column) {
        super(column);
        this.part = part;
    }

    @Override
    public Optional<JDBCType> jdbcType() {
        return Optional.of(JDBCType.INTEGER);
    }

    @Override
    public FragmentAndParameters render(RenderingContext renderingContext) {
        return column.render(renderingContext)
                .mapFragment(f -> "date_part('" + part + "', " + f + ")");
    }

    @Override
    protected DatePart copy() {
        return new DatePart(part, column);
    }

    public static DatePart date_part(String part, BindableColumn<Date> column) {
        return new DatePart(part, column);
    }
}

Thank you very much! This way of handling the code is very elegant!
SelectStatementProvider statement = select(DatePart.date_part("'year'", reportTime))
                .from(report)
                .where(memberId, isEqualTo(_memberId))
                .and(deleted, isEqualTo(false))
                .groupBy(reportTime)
                .orderBy(reportTime.descending())
                .build()
                .render(RenderingStrategies.MYBATIS3);

Optional<WhereClauseProvider> whereClause = where(memberId, isEqualToWhenPresent(query.getMemberId()))
                .and(parseStatus, isEqualToWhenPresent(query.getParseStatus()))
                .and(reportType, isEqualToWhenPresent(query.getReportType()))
                .and(DatePart.date_part("'year'", reportTime), isEqualToWhenPresent(query.getYear()))
                .and(deleted, isEqualTo(false))
                .build()
                .render(RenderingStrategies.MYBATIS3);

But there is another problem, which is a bit more complex.
@Select({
            "select * from (",
            "  select row_number() over (partition by name order by report_time desc, created desc) as rn,",
            "    t.*",
            "  from t_report t",
            "    ${whereClause}",
            ") tmp",
            "  where tmp.rn = 1",
            "  order by tmp.report_time desc, tmp.created desc",
    })
    @ResultMap("ReportResult")
    List<Report> findLastReportGroupByName(WhereClauseProvider whereClause);

This is more complex, but you should be able to make it happen:

Build a "model" class that implements BasicColumn and will render as row_number() over (...)
Build some DSL code that will construct the model class with a syntax you are comfortable with


New version: NLPModelsKnitro v0.4.1

Registering package: NLPModelsKnitro
Repository: https://github.com/JuliaSmoothOptimizers/NLPModelsKnitro.jl
Created by: @dpo
Version: v0.4.1
Commit: b88f008fafb50204f1d3cde6852a7333adb620cb
Reviewed by: @dpo
Reference: https://github.com/JuliaSmoothOptimizers/NLPModelsKnitro.jl/commit/b88f008fafb50204f1d3cde6852a7333adb620cb#commitcomment-41059241

[noblock] This module depends on a commercial library.


Remove the SliceIndex implementation from hash types

Draft because on top of #3310
If folk really want to index into a hash they can us as_byte_array then index that.
Includes a bump to the version number of hashes to v0.15.0 - this is because otherwise secp won't build since we are breaking an API that is used in the current release of secp.
Fix: #3115

Instead of the version bump we could merge and backport https://github.com/rust-bitcoin/rust-secp256k1/pull/731 then this PR would build just fine (I think).

I don't think we need to backport it -- the released version of rust-secp is unaffected by anything that happens on git over here.

@apoelstra we don't have to but we might have an easier time if we do. It could even allow range dependency again.

And, speaking of range dependency, the range dependency on hashes in secp is broken in the same way that @Kixunil had to painfully explain to me the other day. It uses <=0.14

It uses <=0.14

Oh, crap, we should fix it and deprecate the reexport while we're at it.

Why is this draft?

Waiting on #3310

You want me to do this and hold the introduce macro PR up some more, don't you, I can just feel it coming :)

If we have two PRs, one of them removing an item and the other moving that same item, I naturally want the removing one go first so that the second is not made larger by having a useless item there.

Needs rebase.

If this gets through CI the dup deps check is not functioning correctly because we should have had to whitelist hashes to get this through.

Shall we put this one in and cut the release? Or just release without it?

I'll go ahead and one-ACK merge this (it's been like a month since Kix's last comment) so we can put it into the release.
Easy to add back in if we get lots of complaints, but we can't remove it without a release.


fix: delete old reports after policy is deleted.

Description
When the report has only result from policies that are not deployed in the cluster anymore, the audit scanner leaves the reports there until the resource is deleted. This is not right. This commit changes this behaviour deleting all the reports that contains only audit result from policies removed from the cluster.
Furthermore, this commit updates the scheme package changing how the known types are registered by using functions available from the policies and wgpolicies packages that does this.
Fix #263
Test
make unit-tests

LMK what you think.

I liked the proposed solution. I'll do that.


reuse free'd memory?

I have a program that in a tight loop needs to allocate and deallocate blocks of memory (let's say of sizes (a,b,c) ),  but they are the same in every iteration. I would have expected mimalloc to re-use previously allocated memory, but instead I see:
allocating Ptr{Nothing} @0x0000048b74020000
allocating Ptr{Nothing} @0x0000048b84020000
freeing Ptr{Float64} @0x0000048b84020000
allocating Ptr{Nothing} @0x0000048b94020000
allocating Ptr{Nothing} @0x0000048598020000
allocating Ptr{Nothing} @0x0000048bbc020000
freeing Ptr{Float64} @0x0000048bbc020000
freeing Ptr{Float64} @0x0000048598020000
freeing Ptr{Float64} @0x0000048b74020000
allocating Ptr{Nothing} @0x0000048be4020000
allocating Ptr{Nothing} @0x0000048598030500
allocating Ptr{Nothing} @0x0000048c0c020000
freeing Ptr{Float64} @0x0000048c0c020000
freeing Ptr{Float64} @0x0000048598030500
freeing Ptr{Float64} @0x0000048b94020000
allocating Ptr{Nothing} @0x0000048c34020000
allocating Ptr{Nothing} @0x0000048598020000
allocating Ptr{Nothing} @0x0000048c44020000
freeing Ptr{Float64} @0x0000048c44020000
freeing Ptr{Float64} @0x0000048598020000
freeing Ptr{Float64} @0x0000048be4020000
allocating Ptr{Nothing} @0x0000048598040000
freeing Ptr{Float64} @0x0000048c34020000
allocating Ptr{Nothing} @0x00000485984402c0
allocating Ptr{Nothing} @0x0000048598450000
freeing Ptr{Float64} @0x0000048598450000
freeing Ptr{Float64} @0x0000048598040000
freeing Ptr{Float64} @0x00000485984402c0
---
allocating Ptr{Nothing} @0x0000048c54020000
allocating Ptr{Nothing} @0x0000048c64020000
freeing Ptr{Float64} @0x0000048c64020000
allocating Ptr{Nothing} @0x0000048c74020000
allocating Ptr{Nothing} @0x0000048598020000
allocating Ptr{Nothing} @0x0000048c9c020000
freeing Ptr{Float64} @0x0000048c9c020000
freeing Ptr{Float64} @0x0000048598020000
freeing Ptr{Float64} @0x0000048c54020000
allocating Ptr{Nothing} @0x0000048cc4020000
allocating Ptr{Nothing} @0x0000048598030a00
allocating Ptr{Nothing} @0x0000048cec020000
freeing Ptr{Float64} @0x0000048cec020000
freeing Ptr{Float64} @0x0000048598030a00
freeing Ptr{Float64} @0x0000048c74020000
allocating Ptr{Nothing} @0x0000048d14020000
allocating Ptr{Nothing} @0x0000048598020000
allocating Ptr{Nothing} @0x0000048d24020000
freeing Ptr{Float64} @0x0000048d24020000
freeing Ptr{Float64} @0x0000048598020000
freeing Ptr{Float64} @0x0000048cc4020000
allocating Ptr{Nothing} @0x0000048598040000
freeing Ptr{Float64} @0x0000048d14020000
allocating Ptr{Nothing} @0x0000048598440300
allocating Ptr{Nothing} @0x0000048598450000
freeing Ptr{Float64} @0x0000048598450000
freeing Ptr{Float64} @0x0000048598040000
freeing Ptr{Float64} @0x0000048598440300
---
allocating Ptr{Nothing} @0x0000048d34020000
allocating Ptr{Nothing} @0x0000048d44020000
freeing Ptr{Float64} @0x0000048d44020000
allocating Ptr{Nothing} @0x0000048d54020000
allocating Ptr{Nothing} @0x0000048598020000
allocating Ptr{Nothing} @0x0000048d7c020000
freeing Ptr{Float64} @0x0000048d7c020000
freeing Ptr{Float64} @0x0000048598020000
freeing Ptr{Float64} @0x0000048d34020000
allocating Ptr{Nothing} @0x0000048da4020000
allocating Ptr{Nothing} @0x0000048598030a00
allocating Ptr{Nothing} @0x0000048dcc020000
freeing Ptr{Float64} @0x0000048dcc020000
freeing Ptr{Float64} @0x0000048598030a00
freeing Ptr{Float64} @0x0000048d54020000
allocating Ptr{Nothing} @0x0000048df4020000
allocating Ptr{Nothing} @0x0000048598020000
allocating Ptr{Nothing} @0x0000048e04020000
freeing Ptr{Float64} @0x0000048e04020000
freeing Ptr{Float64} @0x0000048598020000
freeing Ptr{Float64} @0x0000048da4020000
allocating Ptr{Nothing} @0x0000048598040000
freeing Ptr{Float64} @0x0000048df4020000
allocating Ptr{Nothing} @0x0000048598440380
allocating Ptr{Nothing} @0x0000048598450000
freeing Ptr{Float64} @0x0000048598450000
freeing Ptr{Float64} @0x0000048598040000
freeing Ptr{Float64} @0x0000048598440380

So despite allocating blocks that are of the same size as very recently free'd memory, mimalloc does not reuse it. Is this expected behaviour?

For performance reasons, in my use case, this is a fairly bad thing. I need to multiply a small matrix by a really large matrix and temporarily store the result somewhere. I therefore allocate some space where I'd like to store the result, and subsequently run into a whole bunch of page faults as the OS is lazy and does not actually back that memory with physical memory.
If I do this repeatedly (say in an iterative eigensolver), I keep having to pay the price of all those page faults, unless the underlying memory is re-used (a potential 2x speedup). The memory is actually re-used when I call (mi_reserve_os_memory), but not by default.
I don't understand this - in both cases there should be a thread local free list, which sometimes gets moved to a global free list, after which that data can again be used to serve alloc calls (deferred_free) . Yet only when using a pool of memory in mi_reserve_os_memory do I actually observe this speedup, and do I actually find that memory is indeed re-used.

This is a different issue than what I thought previously. mimalloc will (should) reuse memory of course and always tries to avoid page faults.  I cannot see what goes on in your case from your description. Is this a multithreaded program or single threaded?
When you use mi_reserve_os_memory do you specify that the memory is (eagerly) committed?  As you say, there should be no difference -- can you try with the latest dev-slice as I just fixed a bug where memory would sometimes too eagerly be decommitted. ?

nothing different on dev-slice. It's a single threaded program for now, and I have attached a log with the exact calls to mimalloc. Every allocate call is an alligned_alloc(64,size). As you can see, the pointer for the larger allocations keeps


leader.go crash (on ARM?)

Is this a bug report or a feature request?:
bug
What happened:
{"log":"/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:175\n","stream":"stderr","time":"2017-12-14T21:51:19.205839696Z"}
{"log":"/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:143\n","stream":"stderr","time":"2017-12-14T21:51:19.205899539Z"}
{"log":"/home/miek/g/src/go.universe.tf/metallb/arp-speaker/main.go:204\n","stream":"stderr","time":"2017-12-14T21:51:19.205958289Z"}
{"log":"/home/miek/upstream/go/src/runtime/asm_arm.s:971\n","stream":"stderr","time":"2017-12-14T21:51:19.206014955Z"}
{"log":"I1214 21:51:19.204827       1 leader.go:31] Lost leadership\n","stream":"stderr","time":"2017-12-14T21:51:19.206071725Z"}
{"log":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]\n","stream":"stderr","time":"2017-12-14T21:51:19.207697751Z"}
{"log":"\u0009panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2017-12-14T21:51:19.207999623Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8d48b0]\n","stream":"stderr","time":"2017-12-14T21:51:19.208521597Z"}
{"log":"\n","stream":"stderr","time":"2017-12-14T21:51:19.208590295Z"}
{"log":"goroutine 50 [running]:\n","stream":"stderr","time":"2017-12-14T21:51:19.20911581Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\n","stream":"stderr","time":"2017-12-14T21:51:19.209617368Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0xfc\n","stream":"stderr","time":"2017-12-14T21:51:19.210387882Z"}
{"log":"panic(0x9a51c0, 0x10ec8e8)\n","stream":"stderr","time":"2017-12-14T21:51:19.210857877Z"}
{"log":"\u0009/home/miek/upstream/go/src/runtime/panic.go:491 +0x204\n","stream":"stderr","time":"2017-12-14T21:51:19.211375945Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/resourcelock.(*EndpointsLock).RecordEvent(0x11f78640, 0xa75c1d, 0xd)\n","stream":"stderr","time":"2017-12-14T21:51:19.211865211Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/endpointslock.go:92 +0xfc\n","stream":"stderr","time":"2017-12-14T21:51:19.212389113Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()\n","stream":"stderr","time":"2017-12-14T21:51:19.21270937Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:183 +0x98\n","stream":"stderr","time":"2017-12-14T21:51:19.213238271Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0x123b5fb8)\n","stream":"stderr","time":"2017-12-14T21:51:19.213644205Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x4c\n","stream":"stderr","time":"2017-12-14T21:51:19.214318938Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x123b5fb8, 0x3b9aca00, 0x0, 0x33333333, 0x3ff33333, 0x12268001, 0x11f48000)\n","stream":"stderr","time":"2017-12-14T21:51:19.215461115Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0x90\n","stream":"stderr","time":"2017-12-14T21:51:19.215834601Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0x12391000)\n","stream":"stderr","time":"2017-12-14T21:51:19.216152254Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:175 +0xac\n","stream":"stderr","time":"2017-12-14T21:51:19.21667777Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0x12391000)\n","stream":"stderr","time":"2017-12-14T21:51:19.216996829Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:143 +0x40\n","stream":"stderr","time":"2017-12-14T21:51:19.217517918Z"}
{"log":"main.main.func1(0x12391000)\n","stream":"stderr","time":"2017-12-14T21:51:19.217822498Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/arp-speaker/main.go:204 +0x1c\n","stream":"stderr","time":"2017-12-14T21:51:19.21833817Z"}
{"log":"created by main.main\n","stream":"stderr","time":"2017-12-14T21:51:19.218640251Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/arp-speaker/main.go:204 +0x388\n","stream":"stderr","time":"2017-12-14T21:51:19.219161913Z"}

I'm not sure if I am holding it wrong or that is just doesn't work on ARM.
The
{"log":"I1214 21:51:19.204827       1 leader.go:31] Lost leadership\n","stream":"stderr","time":"2017-12-14T21:51:19.206071725Z"}

Indicates the callback is called just fine.

I going to continue without this code path, and setup a replicaset with replicas=1 for the time being

hmmm, I do pass in an nil EventRecorder


[Suggestion] Enchantment glow effect

This would basically add an extra arguments to allow adding a glow effect to an item you make, or make the item variant of a block glow as well.
Item version:
"var name".itemGlowing = true/false;
Example:
import mods.contenttweaker.VanillaFactory;
import mods.contenttweaker.Item;

var zsItem = VanillaFactory.createItem("zs_item");
zsItem.maxStackSize = 1;
zsItem.rarity = "rare";
zsItem.beaconPayment = true;
zsItem.itemGlowing = true;
zsItem.register();

Read the docs. Glowing is already a property

How is this done then...

What is the exact method for it then.

glowing


Allow permissions tied to tags / sections

Related to #275. Related to #232
A way to attach permissions to specific sections/tags of cards. E.g. "allow anyone to edit cards if they're in section foo"

#331 has some of the design for this.
The list of uids for a given permission can also take special sentinel UIDs: UID_SENTINEL_ANON_USER, UID_SENTINEL_SIGNED_IN_USER, etc that are related to the types of users, allowing any of them to match

Allow editCard permission on sections


Error while running benchmarks: Export ext_benchmarking_commit_db_version_1 not found

I am on rc-5 and trying to run benchmarks. Keep getting the following error no matter what pallet I try to benchmark unless I follow a pattern that I describe later.
Cannot create a runtime: Instantiation("Instantiation: Export ext_benchmarking_commit_db_version_1 not found")
Error: Input("Error executing runtime benchmark: Other(\"Instantiation: Export ext_benchmarking_commit_db_version_1 not found\")")

I am running the following command for benchmarking from node directory and have tried clearing the database before running
cargo run --features runtime-benchmarks -- benchmark --extrinsic=transfer --pallet=balances

Even after commenting out all benchmarking code from my modules and the add_benchmark! in dispatch_benchmark for my modules, i get the same error as above. This is how dispatch_benchmark looks (taken from substrate repo's bin/node/runtime)
fn dispatch_benchmark(
			pallet: Vec<u8>,
			benchmark: Vec<u8>,
			lowest_range_values: Vec<u32>,
			highest_range_values: Vec<u32>,
			steps: Vec<u32>,
			repeat: u32,
		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, sp_runtime::RuntimeString> {
                       use frame_benchmarking::{Benchmarking, BenchmarkBatch, add_benchmark};
                       use frame_system_benchmarking::Module as SystemBench;
                       impl frame_system_benchmarking::Trait for Runtime {}

                       let whitelist = vec![];

                       let mut batches = Vec::<BenchmarkBatch>::new();
                       let params = (&pallet, &benchmark, &lowest_range_values, &highest_range_values, &steps, repeat, &whitelist);

                       add_benchmark!(params, batches, balances, Balances);
                       add_benchmark!(params, batches, frame_system, SystemBench::<Runtime>);

                       if batches.is_empty() { return Err("Benchmark not found for this pallet.".into()) }
                       Ok(batches)

I followed the discussion in this issue https://github.com/paritytech/substrate/issues/6644 and checked with the linked PR but don't see the difference that can cause this.
However, I can get this to work if I follow the steps in the following sequence:

clear db
run benchmark with above-mentioned command - this will crash with the above-mentioned error
run node in dev mode - this will crash as well with the same error
build without benchmark (cargo build) - this will work
run benchmark with above-mentioned command - this produces the benchmark

The code I am running is here.

You are not registering the benchmarking host functions, as shown here: https://github.com/paritytech/substrate/blob/master/bin/node/executor/src/lib.rs#L30

@bkchr Thanks. Your suggestion works. I am closing this issue but can you please explain why build sequence mentioned in the issue worked?

Not sure how that worked :shrug:


Kotlin mutil module Unresolved reference in Gradle 5.2

Before submitting the question, I've implemented a look at it (https://github.com/gradle/gradle/issues/8980 ),But it didn't solve our problem.
A multi module project with Kotlin source code, which used to work, stops working after upgrading to Gradle 5.2, because the Kotlin classes from the compile project('other-module') dependency are not found.
Expected Behavior
The Kotlin classes in the compile project('depend-test') dependency should be found.
Current Behavior
The Kotlin classes in the compile project('depend-test') dependency are not found:

> Task :test-test:compileKotlin FAILED
e: /Users/houshuai/Documents/dev/demo/test/test-test/src/main/kotlin/com/example/test/controller/TestController.kt: (3, 20): Unresolved reference: dependtest
e: /Users/houshuai/Documents/dev/demo/test/test-test/src/main/kotlin/com/example/test/controller/TestController.kt: (22, 18): Unresolved reference: Test

Context
Doesn't work in Gradle 5.2
Steps to Reproduce
See https://github.com/houshuai0816/gradle-multimodule-kotlin-bug

From a quick test, your reproducer does not work in Gradle 5.1 either, I have a compilation error as well.
Can you please add a README to the reproducer that shows what works?

@ljacomet Increased README in project to describe error Messa

It is not the error message I am after, it is the regression. Could you document a working setup?

OK
I created two modules, test-test and depend-test.
The depend-test project is test-test 's dependency.
I tried to call the parameters of depend-test, but it failed to compile and package.
Env

gradle-5.2.1
Kotlin 1.3.31
Springboot 2.1.4
java 1.8

step one

Edit settings.gradle

rootProject.name = 'demo'
include ":depend-test"
include ":test-test"
project(":depend-test").projectDir = file("depend/depend-test")
project(":test-test").projectDir = file("test/test-test")

I used the 1.3.31 version of the kotlin plug-in. The build.gradle file reads as follows

buildscript {
    ext {
        kotlinVersion = '1.3.31'
    }
    repositories {
        mavenCentral()
        maven { url 'http://maven.aliyun.com/nexus/content/groups/public/'}
        maven { url "https://plugins.gradle.org/m2/" }
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlinVersion"
        classpath "org.jetbrains.kotlin:kotlin-allopen:$kotlinVersion"
    }
}

plugins {
    id 'org.springframework.boot' version '2.1.4.RELEASE'
    id 'org.jetbrains.kotlin.jvm' version '1.2.71'
    id 'org.jetbrains.kotlin.plugin.spring' version '1.2.71'
}

allprojects {
    apply plugin: 'idea'
    apply plugin: 'kotlin'

    repositories {
        mavenCentral()
    }
}

subprojects {
    apply plugin: 'kotlin'
    apply plugin: 'java'
    apply plugin: 'eclipse'
    apply plugin: "application"
    apply plugin: 'org.springframework.boot'
    apply plugin: 'io.spring.dependency-management'

    repositories {
        mavenLocal()
        maven { url 'http://maven.aliyun.com/nexus/content/groups/public/'}
        maven { url "https://plugins.gradle.org/m2/" }
        mavenCentral()
        jcenter()
        maven { url "http://repo.spring.io/snapshot" }
        maven { url "http://repo.spring.io/milestone" }
        maven { url 'http://maven.springframework.org/release' }
        maven { url 'http://maven.springframework.org/milestone' }
    }

    version = '1.0'
    apply plugin: 'io.spring.dependency-management'

    group = 'com.mutil.test'
    sourceCompatibility = '1.8'

    compileKotlin {
        kotlinOptions {
            freeCompilerArgs = ['-Xjsr305=strict']
            jvmTarget = '1.8'
        }
    }

    compileTestKotlin {
        kotlinOptions {
            freeCompilerArgs = ['-Xjsr305=strict']
            jvmTarget = '1.8'
        }
    }

    dependencies {
        implementation 'org.springframework.boot:spring-boot-starter'
        implementation 'com.fasterxml.jackson.module:jackson-module-kotlin'
        implementation 'org.jetbrains.kotlin:kotlin-reflect'
        implementation 'org.jetbrains.kotlin:kotlin-stdlib-jdk8'
        testImplementation 'org.springframework.boot:spring-boot-starter-test'
    }
}

dependencies {
    subprojects.forEach {
        archives(it)
    }
}

repositories {
    mavenCentral()
}

step two
build jar for test-test project ,I used two ways, but the results were the same.

terminal use cmd is ./gradlew :test-test:bootJar
user IDEA gradle tool

result
The class file written by kotlin in the submodule cannot be found.

I do not know if the lack of necessary plug-ins caused the failure to package properly.

I spent some more time looking at this, If I remove all Spring references, then compilation succeeds.
I believe the issue is in the integration of the Spring Boot plugin with a Kotlin project, possibly only a local setup issue in your case.
For example, your depends-test project tries to create a boot jar, but that's most likely not what you need. Similarly the application plugin is applied there as well.

@ljacomet Thanks , I went under the Springboot project to ask if there was a solution.

It's really the problem you're talking about.，Thank you very much for having solved my problem. @ljacomet  https://docs.spring.io/spring-boot/docs/2.1.4.RELEASE/gradle-plugin/reference/html/#managing-dependencies-using-in-isolation