mutating session outside of a rule?

Hi, so with the new dynamic rule stuff I've been building a framework that combines odoyle and asami for a reactive, forward and backward-chaining datalog store designed to serve both web apps and games. In combination with Electric clojure this augments odoyle with ergonomic multi-cardinality joins, cleanup handlers (== truth maintenance), maybe even distributed storage later -- so far it's been looking great.
One of the challenges I've run into is mixing odoyle managed mutations like o/insert!, which operate on a *mutable-session*, and mutations from outside a rule, which would swap! the session atom. I think the way odoyle does it creates a race condition: fire-rules basically takes ownership of the session with interior mutability, and any changes to the session that happen while fire-rules is going off they will be lost when the session atom is reset to the result of fire-rules.
This can happen with relatively plain odoyle use. Take for example my recent game jam submission: when you start dragging the launcher it registers an pointerup event handler which inserts the rules to launch the ball. It's hard to trigger in the published demo but as I've been pushing it it happens relatively often that the pointerup event gets essentially skipped because it's processed in the middle of fire-rules.
Thinking about it a bit more I think I should just not use fire-rules at all, use swap! instead of o/reset! everywhere and write my own impure fire-rules!.

If you are doing a swap! on the session atom from inside a rule, that sounds like a problem. Mutating an atom from inside another swap! fn in general is a bad idea, whether you're using odoyle or not. I'm not sure why this would happen with browser events like pointerup; you can mutate an atom from various browser events and trigger your rules just fine. It sounds like one or more of your rules are ultimately trying to mutate an atom, though. Is there a reason they need to do this?

Is there a reason they need to do this?

To bridge odoyle to electric, it goes through missionary (m/observe), which doesn't do dynamic binding conveyance.
My solution has been to make this change to o/fire-rules
+   (let [session (cond-> session (:transform opts) ((:transform opts)))
          then-queue (:then-queue session)
          then-finally-queue (:then-finally-queue session)]

Then in my code I can queue up work instead of touching *mutable-session*:
(def fact-queue (js/Array.))
(defn add-work! [f]
  (.push fact-queue f))
(defn fire-rules [s]
  (o/fire-rules s {:transform (fn [session]
                                (let [s (reduce (fn[s f](f s)) session fact-queue)]
                                  (set! fact-queue (js/Array.))
                                  s))}))

(add-work! (fn [s](o/insert s e a v)))

I think something like this is also necessary for a multi threaded fire-rules, though


Allow granting / requesting access at invite time

Currently for users to proxy with each other they first need to exchange invites, then need a separate round trip to request/grant or offer/accept permission.  We should add the ability to grant or request access when inviting a user.  There are some UX mocks for this at https://folio.googleplex.com/izzie/uProxy/091015_InviteFlow#%2FInvite Flow_v3.png%3Fz=half (screens "02_Invite a Gmail contact" and "03_Generate an invite code").
TBD:

Invite tokens are currently multi-use.  Will this be a problem for tokens that offer access?
Currently Quiver immediately generates an invite token after sign in or clicking on the invite-user screen.  Should we delay generating the invite URL until after the user has chosen to check offer/request?

This is done


Shift the build process to PEP-517

Evaluate and change the build process to PEP-517

I would like to work on this, can i take this up?

Sounds good. @rhdolin  Is it possible to assign this issue ?  We would like to avoid any wasted effort if someone else is working on this too.

Yes, let's assign to @Arraxx . Aman, thanks for taking this one on.

Thanks for assigning me this issue. i have started reading the PEP 517 documentation and will try to fix this ASAP.

@Arraxx Are you still working on this issue? If not can I take it from here?

Yes @theanmolsharma, I am still working on the issue. I'm relatively new to python and working on this issue will be a great if you find another issue.
If I found myself unable to solve this issue I will mention you for the same.

@Arraxx Sure man, take your time.


Postgres Handler/Example not working

When attempting to use the Postgres Handler, I get the following error:

I did some research and found the item below. Does it indicate you need to some adjustments to make Mesh and Postgraphile more compatible?

Are you using the latest version "@graphql-mesh/postgraphile": "0.8.14"? Because this has been already fixed 2 days ago.

Ah! No. Didn’t realize you guys were already on top of it! Will try it out
and report back. Thanks!
On Tue, Feb 16, 2021 at 9:32 PM Arda TANRIKULU<EMAIL_ADDRESS>wrote:

Are you using the latest version "@graphql-mesh/postgraphile": "0.8.14"?
Because this has been already fixed 2 days ago.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Urigo/graphql-mesh/issues/1621#issuecomment-780254450,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AO3YXTGOBCU6HRRATLWJCCDS7MTC7ANCNFSM4XXJKINQ
.

Closing for now :)


Synthesizing let accessors for storyboard identifiers

Thanks for the PR! I'll take a look as soon as I have a bit of time to review. :)

:+1:

Looked at the PR. Thanks again for creating it!
I have a few questions;

What is your use case that you need to retrieve the identifier of a VC instead of the VC itself?
You're using the type name instead of the identifier name for the generated property, this could conflict if I have 2 VCs of the same type in 1 storyboard, both with an identifier. Shouldn't we use the identifier so it becomes:

let avpVC = "avpVC"
func avpVC(_: Void) -> AVPlayerViewController? {
        return UIStoryboard(resource: self).instantiateViewControllerWithIdentifier(self. avpVC) as? AVPlayerViewController
}

Sorry for the late reply!

What is your use case that you need to retrieve the identifier of a VC instead of the VC itself?

I have a custom set up so that I don't instantiate the VCs from UIStoryboard, but instead each VC has its own initializer to do proper compile-time-safe dependency injection. For this I need the identifier (to pass it to a common parent class that instantiates the controller, which is written in Obj-C cause Swift doesn't allow to return anything other than self from an init)

You're using the type name instead of the identifier name for the generated property, this could conflict if I have 2 VCs of the same type in 1 storyboard, both with an identifier

I remove duplicates if that's the case. Is it normal to have 2 of the same type in one storyboard? I can change it to the identifier though. I remember I did that initially and changed it, but I can't remember why.

I remove duplicates if that's the case. Is it normal to have 2 of the same type in one storyboard? I can change it to the identifier though. I remember I did that initially and changed it, but I can't remember why.

I made that change :) let me know how it looks!

Thanks for the update! Little busy over here, but I'll take a better look asap and see if I can merge it in! :)


Process multiple games concurrently

Due to google sheets limitations we need to process all the games in one function to stop timeouts

fixed by #23


Execution failed for task ':app:lintVitalRelease'

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':app:lintVitalRelease'.

Could not resolve all artifacts for configuration ':image_picker_android:debugUnitTestRuntimeClasspath'.
Failed to transform bcprov-jdk15on-1.68.jar (org.bouncycastle:bcprov-jdk15on:1.68) to match attributes {artifactType=processed-jar, org.gradle.category=library, org.gradle.libraryelements=jar, org.gradle.status=release, org.gradle.usage=java-runtime}.
Execution failed for JetifyTransform: C:\Users\Taufiqsejati.gradle\caches\modules-2\files-2.1\org.bouncycastle\bcprov-jdk15on\1.68\46a080368d38b428d237a59458f9bc915222894d\bcprov-jdk15on-1.68.jar.
Failed to transform 'C:\Users\Taufiqsejati.gradle\caches\modules-2\files-2.1\org.bouncycastle\bcprov-jdk15on\1.68\46a080368d38b428d237a59458f9bc915222894d\bcprov-jdk15on-1.68.jar' using Jetifier. Reason: IllegalArgumentException, message: Unsupported class file major version 59. (Run with --stacktrace for more details.)
- Check out existing issues at https://issuetracker.google.com/issues?q=componentid:460323&s=modified_time:desc, it's possible that this issue has already been filed there.
- If this issue has not been filed, please report it at https://issuetracker.google.com/issues/new?component=460323 (run with --stacktrace and provide a stack trace if possible).

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 3m 6s
Running Gradle task 'assembleRelease'...                          188.3s
Gradle task assembleRelease failed with exit code 1

I'm resolve addiing this to grade in app scope:
´´´
lintOptions {
disable 'InvalidPackage'
disable "Instantiatable"
checkReleaseBuilds false
abortOnError false
}
´´´

This still an issue?
Seems like a Android config problem. Take a look to image_picker config, maybe help

Due to lack of response, I will close this issue.
If the bug still exists, feel free to open a new issue


Added clock tooltip with optional alternate format

I wanted the ability to see the current date when I hover my mouse over the clock module.
Adds the following settings:

clock/tooltip
clock/tooltip-format (default is same as normal format)

Thanks!


Documents steps to meet pipenv prereq

Background
pipenv was not installed on two different systems that I use. This PR adds the steps to install it, and add it to the path in order to satisfy that prerequisite.
Changes

README.md file

Testing

Confirmed it works across two systems

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Publish latest version

Please publish 2.2.1 when you get a chance.

Oops, sorry about that.  It should be published now.


[Bug] PDF Export Button Uncaught Malformed table row, a cell is undefined

Hi,
There is a bug on HTML 5 PDF export button is pressed when there is no data available on tbody. Console.log will print
Uncaught Malformed table row, a cell is undefined.
I have created a jsfiddle to reproduce the bug, the original source code is clone from https://datatables.net/extensions/buttons/examples/html5/simple.html
https://jsfiddle.net/1o9vcv3r/2/
Expected behavior - The PDF export button should work properly likes other export buttons do even though no data is available.

It was fixed in the Buttons repo.


[WIP] debug CI failures in #684

/kind failing-test
What this PR does / why we need it:
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?
NONE

Apparently, sudo changes go env in the following way:
 GOPATH="/home/runner/work/cri-tools/cri-tools"
 GOPRIVATE=""
 GOPROXY="https://proxy.golang.org,direct"
-GOROOT="/opt/hostedtoolcache/go/1.15.4/x64"
+GOROOT="/opt/hostedtoolcache/go/1.14.11/x64"
 GOSUMDB="sum.golang.org"
 GOTMPDIR=""
-GOTOOLDIR="/opt/hostedtoolcache/go/1.15.4/x64/pkg/tool/linux_amd64"
+GOTOOLDIR="/opt/hostedtoolcache/go/1.14.11/x64/pkg/tool/linux_amd64"
 GCCGO="gccgo"
 AR="ar"
 CC="gcc"

which is caused by env var GOROOT which is different for normal user and root (surely there are other changes but they are not relevant I guess).
and this is the cause of the issue I guess.

OK, apparently there are myriad ways to do that, and some of them even work!
+ go env GOROOT
/opt/hostedtoolcache/go/1.15.4/x64
+ sudo go env GOROOT
/opt/hostedtoolcache/go/1.14.11/x64
+ sudo -E go env GOROOT
/opt/hostedtoolcache/go/1.15.4/x64
+ sudo --preserve-env=GOROOT go env GOROOT
/opt/hostedtoolcache/go/1.14.11/x64
+ sudo GOROOT=/opt/hostedtoolcache/go/1.15.4/x64 go env GOROOT
/opt/hostedtoolcache/go/1.14.11/x64
+ sudo env GOROOT=/opt/hostedtoolcache/go/1.15.4/x64 go env GOROOT
/opt/hostedtoolcache/go/1.15.4/x64

To simplify things, let's use sudo -E

We're done here; opened #689 as a fix


using --exit flag returns exitCode 0 even if tests failed

Hi,
I recently installed the latest serverless-mocha-plugin (v1.9.0) that has a support for --exit flag (yay) to exit after the test run is completed.
However, I noticed that, even though the mocha tests failed, the exitCode was still 0.
This has direct implications when running in CI environment, since the exit code is 0 and CI thinks, the run was successful.

Closing this issue, since I was using async/await and one of the promise object was expecting something which caused mocha to not exit.
For context, I should be ending the pg-pool connection instead of just leaving at releasing the db_client.


Wrong Error response with Enum as query Param

When we add an Enum as a QueryParam in API structure and provide an invalid value in the API request it gives 404 error response. The correct response for the user would be 400 Bad request error.
Environment Details

Helidon Version: 3.2.1
Helidon MP
JDK version: 17
OS: MacOs
Docker version (if applicable):

Problem Description
Current Behaviour
We get 404 error response from Helidon.
Expected Behaviour
We should get 400 Bad request response from Helidon
Problem is easily reproducible"
Steps to reproduce

Create an API with an Enum as a query Param
Call the API with an invalid value(Any value which is not in the enum)

Added Screenshots of sample code and response.

Hello
using 400 is reserved for bad request on the protocol level - this request is not wrong on the protocol level (the query is valid), so Jersey probably uses 404 instead, as it cannot invoke the endpoint.
If the enum is yours, you can work around this by specifying a method fromString on the enum that throws appropriate JAX-RS exception (or an exception that you map to it).
The specification says the following - as you can see, 400 is the correct default response as defined by the specification for @QUeryParam:
A WebApplicationException thrown during construction of field or property values using any of the 5 steps listed above is processed directly as described in [Exceptions](https://jakarta.ee/specifications/restful-ws/3.0/jakarta-restful-ws-spec-3.0#method_exc). Other exceptions thrown during construction of field or property values using any of the 5 steps listed above are treated as client errors: if the field or property is annotated with @MatrixParam, @QueryParam or @PathParam then an implementation MUST generate an instance of NotFoundException (404 status) that wraps the thrown exception and no entity; if the field or property is annotated with @HeaderParam or @CookieParam then an implementation MUST generate an instance of BadRequestException (400 status) that wraps the thrown exception and no entity. [Exceptions](https://jakarta.ee/specifications/restful-ws/3.0/jakarta-restful-ws-spec-3.0#method_exc) MUST be processed as described in Exceptions.

Resolving this issue - I have provided an explanation, and a workaround.


Html structure

Following on from #406, this has been updated to use a static regex.
I ran the bench tests for this PR and v0.3.2 (50000 iterations) and there doesn't seem to be any significant difference:

v0.3.2
This PR

marked
171160ms
169728ms.

marked (gfm)
186709ms
188834ms.

marked (pedantic)
161621ms
161455ms.

Closing as stale, merge conflicts, and out of focus. See #956


Responsive work and spacing with "Our Work" section 🧑🏭

The "Our Work" section needs to be more responsive for mobile users. Some of the text is hard to see 👁, and this will need to be changed (styles for the card can be found within the /components/styles folder under it's respective name). Also, the top card is too close to the title of the page, and this will need to be moved down.

Wish you're having a great day! Can I try and solve this issue?  😊


Digest Calculation for Filesystem Content

Filesystem content may be provided in various technical formats: zip, tgz, tar,...
So far, the chosen archive format is taken in its binary form to calculate a digest.
It would be a good idea to have a format-agnostic digest for such content used for a dedicated FileSystemContent
Artifact type.
What would you like to be added:
Git already has solved this problem by introducing its Tree Hashes. So it seems to be useful to implement
a compatible hashing for filesystem content based on the various archive formats.
It should be added a s a dedicated digest handler for the filesystem content artifact type.
Why is this needed:
This mechanism provides a format agnostic digest for file system content. It offers the possibility to repack content without loosing the validity of its digest used as part of the signing process.

A package providing a Git-like treehash for filesystem or archive content and an incremental filetree compose is provided.
But it is not yet used anywhere.
We have to establish DirectoryTree-like resources which should use such a digester.

This is related to #42
It has been decided to keep this issue on hold until it is necessary to implement

implemented, but no real use-case so far.


Reviewing APIView latest designs

Hellooooo
Would love to book sometime with the architects (minimum of an hour) to review the design improvements of the APIView tool.
@weshaggard  @praveenkuttappan @chidozieononiwu @kurtzeborn @mikekistler  as additional attendants

scheduled for 2/7 2-3pm pst

@lmazuel fyi since you have asked to be part of this APIView demo

Recording[MS INTERNAL ONLY]


x/tools/go/loader: should respect PKG_CONFIG environmental variable

When loading cgo, the go tool allows the path to the pkg-config tool to be overridden with the PKG_CONFIG environmental variable. The x/tools loader, however, hardcodes the name pkg-config for a $PATH search. It should instead only fallback to such a search after looking at the PKG_CONFIG environmental variable.

go/loader is deprecated in favor of go/packages. Please give it a try. (The same bug may well exist there. If so, please let us know.)
cc @matloob

The go/packages < 1.12 fallback code uses the same cgo code as go/loader. But it looks like that's all slated to be deleted imminently?

The go/packages fallback code was just deleted as of golang.org/cl/166537.
As @josharian noted above, go/loader is deprecated. We don't plan on making changes to go/loader except to fix serious bugs.

Closing as the package is deprecated.


Make The Build Green Again

This PR:

Makes the build green again.

Thanks!
Strange, you would have thought Environment.Line would know better...


pkg/transport: reload TLS certificates for every use

This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.
Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.
Fixes #7576

Caveats: This requires Go 1.8.1

Commit title should be pkg/transport: reload ...?

So the existing TLS connections will not be terminated even if the cert is changed? Is this the expected behavior?

That is correct. Because the TLS negotiation occurs only when the connection is opened, ongoing connection can continue.

@tgrosinger
yea. i know how tls works. my concerns is that users might expect the existing tls connections to be terminated when they switch key/cert. but i am not a security expert and do not know the common practice. it would be great if you can feed me any related info: like what other projects do online cert/key change. thanks.

Not terminating existing connections is the usual case. Main reason to replace certs is local renewal.  In the more rare case of moving from one trust root to another, it's a conscious act, and restarting the service would be a good way to terminate all connections.
Not killing existing connections matches expectation (either automatic reload, or reloading on a SIGHUP.)

@Spindel thanks for the explanation. make sense.

For our use case we are running etcd in a Kubernetes pod, so we are using an in-memory EmptyDir volume to help with performance.
I agree that caching would be desirable, however this patch was meant to be as minimal as possible. It's something I wanted to share with the community for others to build off of, not necessarily to be merged into etcd proper as it currently stands.

Ok, I can confirm that this works with

good cert etcd cluster
now has bad etcd cluster
client connection to this bad cert etcd cluster fails with "transport: x509: certificate has expired or is not yet valid"; please retry

Test fails because we still need Certificates: []tls.Certificate{*tlsCert} for initial listener start.
We need integration test around this, anyway. So I will cherry-pick this patch and create a separate PR with tests (today or tomorrow).
Thanks!

Closing in favor of https://github.com/coreos/etcd/pull/7829.


docs: [BD-46] added "Elevation" page to "Foundations" section

Description
Added "Elevation" page to "Foundations" section.
Figma
Deploy preview
Elevation page
Merge Checklist

[ ] If your update includes visual changes, have they been reviewed by a designer? Send them a link to the Netlify deploy preview, if applicable.
[x] Does your change adhere to the documented style conventions?
[x] Do any prop types have missing descriptions in the Props API tables in the documentation site (check deploy preview)?
[x] Were your changes tested using all available themes (see theme switcher in the header of the deploy preview, under the "Settings" icon)?
[x] Were your changes tested in the example app?
[ ] Is there adequate test coverage for your changes?
[x] Consider whether this change needs to reviewed/QA'ed for accessibility (a11y). If so, please add wittjeff and adamstankiewicz as reviewers on this PR.

Post-merge Checklist

[x] Verify your changes were released to NPM at the expected version.
[x] If you'd like, share your contribution in #show-and-tell.
[x] 🎉 🙌 Celebrate! Thanks for your contribution.

Thanks for the pull request, @PKulkoRaccoonGang! I've created BLENDED-1254 to keep track of it in Jira.   More details are on the BD-46 project page.
When this pull request is ready, tag your edX technical lead.

@PKulkoRaccoonGang 🎉 Your pull request was merged!  Please take a moment  to answer a two question survey so we can improve your experience in the future.

:tada: This PR is included in version 19.25.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Simulation Verilog

Hi there,
Thanks for taking time reading my issue.
I am trying to use CIRCT and Calyx to lower my MLIR which is a simple four-element vector addition.
I used the following lowering pipeline to get Verilog:
./mlir-opt --lower-affine
./circt-opt --lower-scf-to-calyx --canonicalize --verify-diagnostics
./circt-translate --export-calyx
futil -l ./ -b verilog --synthesis --disable-init --nested
After getting Verilog, I exported the Verilog code into Vivado to do the simulation. I noticed a couple of issues impeding my simulation:

There are three input and three output ports, which is supposed to be two input and one output according to my MLIR. But then I figured out only two input ports read data from outside and only one output has the driver, equivalent to two input and one output, making sense to me at this stage.
The address signal of the output port is a binary signal without bit-width, meaning the output port cannot write data to external memory. So I couldn't get the correct results from my test bench.
There are redundant finite state machine signals and assign statements, is it possible to make the Verilog more compact and readable?

I attached my MLIR, generated Verilog, and my test bench here. Hope they are easy to understand. I am looking forwards to your reply!
va_for_loop.mlir.txt
va_for_loop_calyx_naive.sv.txt
va_1x4_tb.sv.txt
Many thanks.

Digging into the example a bit, here are some answers:

When you define a memory as an argument (input or output), the CIRCT Calyx compiler will implement it by taking all the ports of a std_mem_d1 primitive and putting them into port arguments of the containing component. This is what you're seeing in the example: your three memories get turned into lots of ports which correspond to the memories. Calyx supports the ref keyword to make this easier to implement and optimize but the CIRCT compiler currently does not use it
The output signal is a binary because your memory is a 1-element memory which means a binary signal is sufficient to address it. If you change your example to have larger memories, Calyx with generate the appropriate signals.
Yes, remove the --nested flag from the final command. This will generate wires that track the FSM signals better.

I haven't looked into your testbench yet but let me know if these answers help? If your TB simulates, can you send me the VCD file so I can take a closer look?

Thanks a lot for the reply.
va_for_loop.mlir.txt
Probably my MLIR file is incorrect. As I would like to realize 4-element vector-add, but IO memories only contains one element, e.g., %arg0: memref<1xi32, 1>, %arg1: memref<1xi32, 1>, %arg2: memref<1xi32, 1>.
I am trying to do behavior simulation of the generated Verilog in Vivadova_for_loop_calyx_naive.sv.txt. It is little vague which port is input and output. But nevertheless I found 'ext_mem1_write_data' and 'ext_mem2_write_data' don't have drivers and deliver 'xxxx_xxxx'. So I think these two ports should be input, and 'ext_mem0_write_data' is the output port.
I input one value each clock cycle, meaning that I input four values '0,1,2,3' to 'ext_mem1_read_data' and '0,2,4,6' to 'ext_mem2_read_data' in four clock cycles. But I couldn't get correct results from 'ext_mem0_write_data'. So I just wonder if I didn't stimulate the input improperly. As I am using Vivado, so this is the waveform file I got. In case of you cannot open it, I also attached the screenshot here.
va_1x4_tb_behav.zip

Looking forwards to your reply.

Ah, thanks for the simulation trace. Couple more pointers for you:

Yup! That's exactly why you have only one element in your memory. If you change it to memref<4xi32>, it should create an array with 4 elements.
The primitives are implement using the ports in std_mem_d1. You can look at the Verilog for that module to get a sense of how the ports should be used.
read_data is the "output" port of a memory and combinationally updates when you provide a value to addr0. The write_data port writes a value into the address specified by addr0 only when write_en is also asserted.

The reason you might be having a hard time simulating this design appropriately is because there are no memories in the program–your test bench needs to be one that models the behavior of the memories and maintain the interface with specified by std_mem_d1.
If I may, I really recommend going through the language tutorial to get a sense of how we simulate programs with Calyx's default testbench. In the meantime, I'm going to go and attempt to write a testbench for you to and see if I can make your design work.

Many thanks for the help.
The Verilog code seems not the same with my uploaded MLIR. Can I ask what is your lowering pipelines get the Verilog you sent just now?

Hi there,
Thanks for the previous explanation. Vector-add examples work correctly now and move to matrix-multiplication.
The generated Verilog seems cannot recognize each row of the output matrix. For example we are trying 4x4,
x

the correct result should be:

`
But the code give us the result:
0,0,0,0
0,4,12,24
24,32,48,72
72,84,108,144
`
Elements of the output matrix keep accumulating until the last one. I think the reason is that fsm are not mapped correctly from MLIR to Verilog. I added a Verilog snipped in the module, and revised the RAM in core.sv and solved this issue for any size of matrix multiplications.
The added Verilog in the module (4x4 gemm for example):
always_ff @(posedge clk or posedge reset)begin
if (reset)
we_cntr<=32'b0;
else if (we_cntr<'d4 && ext_mem0_write_en) //length of each MAC
we_cntr<=we_cntr+1'b1;
else if (we_cntr=='d4) //length of each MAC
we_cntr<='b0;
else
we_cntr<=we_cntr;
end
Revised BRAM snippet:
  assign read_data = we_cntr=='b0 ? 'b0 : mem[addr0];
So that we can make the first element to zero before the multiplication.
Although the manual modification can make it work, it would hinder the entire automation pipelines. I just wonder is there any better method to address this issue if it is because of Calyx? I attached verilog files and MLIR there.
The second question is that how can I wrap ext_mem0_read_* and ext_mem0_write_* with AXI interface using the Calyx tool? Do I need to change the IO to @external in calyx dialect?
The questions might seems little annoying, appreciate you taking time to read it.
Many thanks!
gemm4x4.zip

Very interesting! It would be good to narrow down whether this bug is on the CIRCT side (translating to Calyx) or on the Calyx side (compiling Calyx IL to Verilog). I see the zip archive there has the original (affine dialect) MLIR cod and the Verilog—any chance you could generate the Calyx IL code that results from the CIRCT-side compilation so we can look at that?
If you're interested, it's possible to execute Calyx code directly with our debugger, which would help distinguish whether the Calyx code is wrong or whether it's being compiled wrong.

The second question is that how can I wrap ext_mem0_read_* and ext_mem0_write_* with AXI interface using the Calyx tool? Do I need to change the IO to @external in calyx dialect?

Wrapping memories in an AXI interface in Calyx is currently under the umbrella of our support for the Xilinx toolchain. You can enable that backend by passing -b xilinx to the compiler, like this as an example:
cargo run -- -b xilinx examples/futil/dot-product.futil

That produces Verilog with AXI wrappers for the @external memories.

Our Calyx IL does not have @external memories that is the point I am curious about. Without '@external', the error was launched:

Ah, I see now! I didn't previously quite grok that the Calyx program didn't have any memories at all; they have already been "externalized." This is a little hard to explain, but bear with me: if you look at the (native) Calyx code, you'll see that the top-level component (mlir_funcSYCL_class_mxm_kernel) has a bunch of input ports like ext_mem0_read_data, ext_mem0_done: several ports for each memory. The idea is that the module is assuming someone else has instantiated these memories and is handing over all their ports to access them—for example, that's exactly what @rachitnigam's testbench above provided.
In the native Calyx ecosystem, we achieve the same effect by declaring @external memories. These "look like" real memories but actually get turned into the same thing as above during compilation: there is no real memory, and instead someone "external" provides the interface ports to the memories. It looks like this:
https://github.com/cucapra/calyx/blob/d03d22ffb3f5b13d3091e2d3e82d8fb3d0963f54/examples/tutorial/language-tutorial-iterate.futil#L5
It looks like the SCFToCalyx pass is responsible for creating the former "add a bunch of ports" style of code, as described by the MemoryPortsImpl variant of MemoryInterface:
https://github.com/llvm/circt/blob/7bbf7adc882686ea337d4ebb371cf204f6b04d73/include/circt/Dialect/Calyx/CalyxLoweringUtils.h#L87-L93
It seems to me that we might want to change this pass to use the @external memory style instead. It would ultimately compile to the same thing, and it would let us do stuff in Calyx that is currently impossible: most relevantly here, running the debugger and using the Xilinx backend to generate AXI wrappers for the memories.

Hey @zzy666666zzy, I'm working on a pass to automate the transformation of ports into @external cells which will allow you to use the Cider and fud tools in this repo: https://github.com/cucapra/calyx/pull/1488
I'll ping the thread once its done with instructions for you to try it out. Hopefully, this will also make it easier to debug the bug in the program above which likely caused by CIRCT lowering.

Okay, the pass I linked above is working! Once it is merged, you can run the following command to extract cells marked with @external from your program:
cargo run -- gemm4x4_calyx.txt  -p validate -p discover-external -p validate -m file -x discover-external:default=16

The cargo run command rebuilds the compiler from source in the repository. If you're using the crate, you'll have to update it first and then run the compiler:
cargo install calyx
calyx gemm4x4_calyx.txt  -p validate -p discover-external -p validate -m file -x discover-external:default=16

You'll notice that your get the following warnings:
[WARN  calyx_opt::passes::discover_external] Unable to infer parameter value for ext_mem0_ in std_mem_d1, defaulting to 4
[WARN  calyx_opt::passes::discover_external] Unable to infer parameter value for ext_mem2_ in std_mem_d1, defaulting to 4
[WARN  calyx_opt::passes::discover_external] Unable to infer parameter value for ext_mem1_ in std_mem_d1, defaulting to 4

This is because the pass has no way of inferring the size of the memory for you. We provide the -x discover-external:default=16 to say that when a parameter to a cell (in this case, the size of the memory) cannot be inferred, please use the value 16 which works for your example since your memory has 4x4=16 elements.
If you look at the output of the pass, you can see the we've removed a lot of the ports and added a bunch of new cells marked with @external:
cells {
    ...
    @external @generated ext_mem1_ = std_mem_d1(32, 4, 4);
    @external @generated ext_mem0_ = std_mem_d1(32, 4, 4);
    @external @generated ext_mem2_ = std_mem_d1(32, 4, 4);
}

These are exactly the cells we need to make the program work with the AXI generator and the debugger! Next, we need to specify some data using Calyx testbench's data format which is just a JSON file:
{
"ext_mem0_":{
  "data":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "format":{"numeric_type":"bitnum","is_signed":false,"width":32}},
"ext_mem1_":{
  "data":[0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3],
  "format":{"numeric_type":"bitnum","is_signed":false,"width":32}},
"ext_mem2_":{"data":[0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3],
  "format":{"numeric_type":"bitnum","is_signed":false,"width":32}}
}

Then, I ran the following fud command to simulate the program through icarus-verilog:
fud e -s verilog.data data.json --to dat --through icarus-verilog out.futil

Sadly, fud will complain that the main module is missing. This is because the Calyx testbench hardcodes main as the name of the DUT. To fix this, I renamed the top module from mlir_funcSYCL_class_mxm_kernel to main and this allows fud to generate the output:
{"cycles":1940,"memories":{"ext_mem0_":[0,0,0,0,0,4,12,24,24,32,48,72,72,84,108,144],"ext_mem1_":[0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3],"ext_mem2_":[0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3]}}

This reproduces the erroneous output you are getting. I'm going wrap this comment up at this point but will report back once I find the issue! Hopefully this is helpful.
Please ask questions if you run into any trouble understanding this!

Okay, I've figured out the bug! The problem is this group:
    group assign_while_0_init_1 {
      while_0_arg1_reg.in = load_0_reg.out; // incorrect init
      while_0_arg1_reg.write_en = 1'd1;
      assign_while_0_init_1[done] = while_0_arg1_reg.done;
    }

Note that it resets the values of the while_0_arg1_reg to the value of load_0_reg. This is incorrect and if we instead make it set the value to 0, the program generates the expected out:
    group assign_while_0_init_1 {
      while_0_arg1_reg.in = 32'd0;
      while_0_arg1_reg.write_en = 1'd1;
      assign_while_0_init_1[done] = while_0_arg1_reg.done;
    }

Running fud, we get the correct answer:
{"cycles":2068,"memories":{"ext_mem0_":[0,0,0,0,0,4,8,12,0,8,16,24,0,12,24,36],"ext_mem1_":[0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3],"ext_mem2_":[0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3]}}

We'll still have to figure out where the CIRCT compilation introduces this bug...

Can we double check the input MLIR?
This is what I have from the thread:
// Is this IR correct?
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<16xi32, 1>, %arg1: memref<16xi32, 1>, %arg2: memref<16xi32, 1>) {
  affine.for %arg3 = 0 to 4 {
    affine.for %arg4 = 0 to 4 {
      %0 = affine.load %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
      %1 = affine.for %arg5 = 0 to 4 iter_args(%arg6 = %0) -> (i32) {
        %2 = affine.load %arg2[%arg5 * 4 + %arg4] : memref<16xi32, 1>
        %3 = affine.load %arg1[%arg5 + %arg3 * 4] : memref<16xi32, 1>
        %4 = arith.muli %2, %3 : i32
        %5 = arith.addi %4, %arg6 : i32
        affine.store %5, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
        affine.yield %5 : i32
      }
    }
  }
  return
}

This is two attempts at rewriting:
// The above didn't make sense to me. This is how I rewrote the gemv kernel
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<16xi32, 1>, %arg1: memref<16xi32, 1>, %arg2: memref<16xi32, 1>) {
  affine.for %arg3 = 0 to 4 {
    affine.for %arg4 = 0 to 4 {
      // Start the inner product at constant 0
      %c0_i32 = arith.constant 0 : i32
      %1 = affine.for %arg5 = 0 to 4 iter_args(%arg6 = %c0_i32) -> (i32) {
        %2 = affine.load %arg2[%arg5 * 4 + %arg4] : memref<16xi32, 1>
        %3 = affine.load %arg1[%arg5 + %arg3 * 4] : memref<16xi32, 1>
        %4 = arith.muli %2, %3 : i32
        %5 = arith.addi %4, %arg6 : i32
        affine.yield %5 : i32
      }
      // Assign out[%arg3][%arg4] with inner product result
      affine.store %1, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
    }
  }
  return
}

// Can use repeated loads and stores instead of carrying an iterArg
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<16xi32, 1>, %arg1: memref<16xi32, 1>, %arg2: memref<16xi32, 1>) {
  affine.for %arg3 = 0 to 4 {
    affine.for %arg4 = 0 to 4 {
      // Start the inner product at constant 0
      %c0_i32 = arith.constant 0 : i32
      affine.store %c0_i32, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
      affine.for %arg5 = 0 to 4 {
        %2 = affine.load %arg2[%arg5 * 4 + %arg4] : memref<16xi32, 1>
        %3 = affine.load %arg1[%arg5 + %arg3 * 4] : memref<16xi32, 1>
        %4 = affine.load %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
        %5 = arith.muli %2, %3 : i32
        %6 = arith.addi %5, %4 : i32
        affine.store %6, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
      }
    }
  }
  return
}

We can use the linalg dialect and -convert-linalg-to-affine-loops to make it less confusing to check too:
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<4x4xi32, 1>, %arg1: memref<4x4xi32, 1>, %arg2: memref<4x4xi32, 1>) {
    affine.for %arg3 = 0 to 4 {
      affine.for %arg4 = 0 to 4 {
        affine.for %arg5 = 0 to 4 {
          %0 = affine.load %arg0[%arg3, %arg5] : memref<4x4xi32, 1>
          %1 = affine.load %arg1[%arg5, %arg4] : memref<4x4xi32, 1>
          %2 = affine.load %arg2[%arg3, %arg4] : memref<4x4xi32, 1>
          %3 = arith.muli %0, %1 : i32
          %4 = arith.addi %2, %3 : i32
          affine.store %4, %arg2[%arg3, %arg4] : memref<4x4xi32, 1>
        }
      }
    }
    return
  }

Oh hey, thats a good catch @matth2k! @zzy666666zzy, it seems the input MLIR code is wrong because it does not correctly load the accumulation variable causing the downstream problem.

I've also written up some documentation for two new passes that should make it easier to interact with the Calyx compiler when using CIRCT generated code! https://docs.calyxir.org/fud/circt.html#using-native-tools-with-mlir-generated-calyx

Hi @matth2k,
Thanks for the corrected MLIR. I have tried the provided three MLIR, only
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<16xi32, 1>, %arg1: memref<16xi32, 1>, %arg2: memref<16xi32, 1>) {
  affine.for %arg3 = 0 to 4 {
    affine.for %arg4 = 0 to 4 {
      // Start the inner product at constant 0
      %c0_i32 = arith.constant 0 : i32
      %1 = affine.for %arg5 = 0 to 4 iter_args(%arg6 = %c0_i32) -> (i32) {
        %2 = affine.load %arg2[%arg5 * 4 + %arg4] : memref<16xi32, 1>
        %3 = affine.load %arg1[%arg5 + %arg3 * 4] : memref<16xi32, 1>
        %4 = arith.muli %2, %3 : i32
        %5 = arith.addi %4, %arg6 : i32
        affine.yield %5 : i32
      }
      // Assign out[%arg3][%arg4] with inner product result
      affine.store %1, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
    }
  }
  return
}

works for my current testbench.
This version
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<16xi32, 1>, %arg1: memref<16xi32, 1>, %arg2: memref<16xi32, 1>) {
  affine.for %arg3 = 0 to 4 {
    affine.for %arg4 = 0 to 4 {
      // Start the inner product at constant 0
      %c0_i32 = arith.constant 0 : i32
      affine.store %c0_i32, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
      affine.for %arg5 = 0 to 4 {
        %2 = affine.load %arg2[%arg5 * 4 + %arg4] : memref<16xi32, 1>
        %3 = affine.load %arg1[%arg5 + %arg3 * 4] : memref<16xi32, 1>
        %4 = affine.load %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
        %5 = arith.muli %2, %3 : i32
        %6 = arith.addi %5, %4 : i32
        affine.store %6, %arg0[%arg4 + %arg3 * 4] : memref<16xi32, 1>
      }
    }
  }
  return
}

writes back resultant BRAM with all zeros.
And this version:
func.func @mlir_funcSYCL_class_mxm_kernel(%arg0: memref<4x4xi32, 1>, %arg1: memref<4x4xi32, 1>, %arg2: memref<4x4xi32, 1>) {
    affine.for %arg3 = 0 to 4 {
      affine.for %arg4 = 0 to 4 {
        affine.for %arg5 = 0 to 4 {
          %0 = affine.load %arg0[%arg3, %arg5] : memref<4x4xi32, 1>
          %1 = affine.load %arg1[%arg5, %arg4] : memref<4x4xi32, 1>
          %2 = affine.load %arg2[%arg3, %arg4] : memref<4x4xi32, 1>
          %3 = arith.muli %0, %1 : i32
          %4 = arith.addi %2, %3 : i32
          affine.store %4, %arg2[%arg3, %arg4] : memref<4x4xi32, 1>
        }
      }
    }
    return
  }

seems change the destination pointer from %arg0 to %arg2, so input and output ports in testbench should be changed accordingly. I have managed to use the working version to generate AXI wrapped Top-module. Thanks a lot.
Thanks @rachitnigam for the complemented document, can I ask if it is possible to avoid using XRT, but to use standalone (baremetal) and a .tcl script to generate a block design and integrate it with ARM CPU?

(Just wanted to say AMAZING WORK to @rachitnigam for improvising those extra passes to help debug this! Really really cool that it was possible to "recover" @external memories from what CIRCT is currently producing.)

can I ask if it is possible to avoid using XRT, but to use standalone (baremetal) and a .tcl script to generate a block design and integrate it with ARM CPU?

Good question. Looking at the generated Verilog, is there anything in there that prevents this use case? That is, our current AXI-wrapping pass is indeed meant to be used from XRT, but it "mostly" just consists of AXI interfaces to the relevant memories (and control signals). It seems like this could be a good starting point for integrating with something else instead, such as a soft ARM core, if it's willing to speak AXI. But are there particular things we're generating there that prevent that for your situation?

Thanks @rachitnigam for the complemented document, can I ask if it is possible to avoid using XRT, but to use standalone (baremetal) and a .tcl script to generate a block design and integrate it with ARM CPU?

I think it should be possible. As @sampsyo said, the code is designed to generate generic AXI controller but uses XRT to generate and interface with the host code. Can you take a shot at using it and seeing what kinds of errors show up? We can help you once we have a sense of what kinds of problems you're running into.

Hi there! Debugging Xilinx toolchain problems can be a huge pain; sorry you've run into trouble here. Here are a couple of thoughts about this:

Just to check: is your gen_xo.tcl script the same as ours? I'm guessing yes, but just wanted to confirm.
It looks like you're running vivado manually—I imagine you got this command from a fud execution, right? Just wanted to make sure the same thing happens when you do, like, fud --to xclbin as when running the command manually.
My best guess is that this has to do with the version of Vivado/Vitis you have. We have been testing against 2019.1. It seems plausible to me that more recent versions of Vivado are expecting a kernel.xml in a somewhat different format. We should conform!
A good way to get a more specific error out of Vivado might be to split up the kernel.xml file onto different lines. You'll notice that we generate it with no line breaks; I've pasted a version of your file below that I prettified with xmllint. Vivado will give a more specific line number on this file!

prettified kernel.xml
<?xml version="1.0" encoding="UTF-8"?>
<root>
  <versionMajor>1</versionMajor>
  <versionMinor>6</versionMinor>
  <kernel>
    <name>Toplevel</name>
    <language>ip_c</language>
    <vlnv>capra.cs.cornell.edu:kernel:Toplevel:1.0</vlnv>
    <attributes/>
    <preferredWorkGroupSizeMultiple>0</preferredWorkGroupSizeMultiple>
    <workGroupSize>1</workGroupSize>
    <interrupt>false</interrupt>
    <hwControlProtocol>ap_ctrl_hs</hwControlProtocol>
    <ports>
      <port>
        <name>s_axi_control</name>
        <mode>slave</mode>
        <range>0x1000</range>
        <dataWidth>32</dataWidth>
        <portType>addressable</portType>
        <base>0x0</base>
      </port>
      <port>
        <name>m0_axi</name>
        <mode>master</mode>
        <range>0xFFFFFFFFFFFFFFFF</range>
        <dataWidth>512</dataWidth>
        <portType>addressable</portType>
        <base>0x0</base>
      </port>
      <port>
        <name>m1_axi</name>
        <mode>master</mode>
        <range>0xFFFFFFFFFFFFFFFF</range>
        <dataWidth>512</dataWidth>
        <portType>addressable</portType>
        <base>0x0</base>
      </port>
      <port>
        <name>m2_axi</name>
        <mode>master</mode>
        <range>0xFFFFFFFFFFFFFFFF</range>
        <dataWidth>512</dataWidth>
        <portType>addressable</portType>
        <base>0x0</base>
      </port>
    </ports>
    <args>
      <arg>
        <name>timeout</name>
        <addressQualifier>0</addressQualifier>
        <id>0</id>
        <port>s_axi_control</port>
        <size>0x4</size>
        <offset>0x010</offset>
        <type>uint</type>
        <hostOffset>0x0</hostOffset>
        <hostSize>0x4</hostSize>
      </arg>
      <arg>
        <name>ext_mem0_</name>
        <addressQualifier>1</addressQualifier>
        <id>1</id>
        <port>m0_axi</port>
        <size>0x8</size>
        <offset>0x18</offset>
        <type>int*</type>
        <hostOffset>0x0</hostOffset>
        <hostSize>0x8</hostSize>
      </arg>
      <arg>
        <name>ext_mem2_</name>
        <addressQualifier>1</addressQualifier>
        <id>2</id>
        <port>m1_axi</port>
        <size>0x8</size>
        <offset>0x20</offset>
        <type>int*</type>
        <hostOffset>0x0</hostOffset>
        <hostSize>0x8</hostSize>
      </arg>
      <arg>
        <name>ext_mem1_</name>
        <addressQualifier>1</addressQualifier>
        <id>3</id>
        <port>m2_axi</port>
        <size>0x8</size>
        <offset>0x28</offset>
        <type>int*</type>
        <hostOffset>0x0</hostOffset>
        <hostSize>0x8</hostSize>
      </arg>
    </args>
  </kernel>
</root>

Looking at the XML file directly, it seems like we're using a different XML format from what the Xilinx user guide recommends. We generate, for example:
      <port>
        <name>s_axi_control</name>
        <mode>slave</mode>
        <range>0x1000</range>
        <dataWidth>32</dataWidth>
        <portType>addressable</portType>
        <base>0x0</base>
      </port>

Whereas the Xilinx manual recommends:
 <port name="s_axi_control" mode="slave" range="0x1000" dataWidth="32" portType="addressable" base="0x0"/>

That is, it uses XML attributes instead of nested tags. It is possible that Vivado got more strict about this format between 2019 and 2023. We should definitely switch to the recommended format…

I dug into this a little more and it turns out that our kernel.xml generation had actually just broken in the last couple of months! #1529 should fix this.

Hi there! Debugging Xilinx toolchain problems can be a huge pain; sorry you've run into trouble here. Here are a couple of thoughts about this:
* Just to check: is your `gen_xo.tcl` script the same as [ours](https://github.com/cucapra/calyx/blob/master/fud/bitstream/gen_xo.tcl)? I'm guessing yes, but just wanted to confirm.

* It looks like you're running `vivado` manually—I imagine you got this command from a `fud` execution, right? Just wanted to make sure the same thing happens when you do, like, `fud --to xclbin` as when running the command manually.

* My best guess is that this has to do with the version of Vivado/Vitis you have. We have been testing against 2019.1. It seems plausible to me that more recent versions of Vivado are expecting a `kernel.xml` in a somewhat different format. We should conform!

* A good way to get a more specific error out of Vivado might be to split up the `kernel.xml` file onto different lines. You'll notice that we generate it with no line breaks; I've pasted a version of your file below that I prettified with `xmllint`. Vivado will give a more specific line number on this file!

prettified kernel.xml

Hi @sampsyo, thanks a lot for your reply.

Yes, I used the provided .tcl file just changed the path including our Verilog file.
Yes, I used step-by-step calyx command lines. And also used fud --to xclbin to generate .xo, but got following error:

$ fud e examples/futil/dot-product.futil -o foo.xclbin --to xclbin
Traceback (most recent call last):
  File "/home/zzy666/.local/bin/fud", line 8, in <module>
    sys.exit(main())
  File "/home/zzy666/.local/lib/python3.9/site-packages/fud/main.py", line 342, in main
    exec.run_fud(args, cfg)
  File "/home/zzy666/.local/lib/python3.9/site-packages/fud/exec.py", line 105, in run_fud
    step()
  File "/home/zzy666/.local/lib/python3.9/site-packages/fud/stages/__init__.py", line 51, in __call__
    self.output.data = self.func(*self.args)
  File "/home/zzy666/.local/lib/python3.9/site-packages/fud/stages/xilinx/xclbin.py", line 82, in package_xo
    port_names = list(get_ports(Path(tmpdir) / "kernel.xml"))
  File "/home/zzy666/.local/lib/python3.9/site-packages/fud/stages/xilinx/xclbin.py", line 24, in get_ports
    if port.attrib["mode"] == "master":
KeyError: 'mode'

Seems also because of kernel.xml

I agree, I checked my another xml file example

<args>
<arg name="m_ptr" addressQualifier="1" id="0" port="M_AXI_GMEM" size="0x8" offset="0x10" hostOffset="0x0" hostSize="0x8" type="void*"/>
<arg name="m_ptr1" addressQualifier="1" id="1" port="M_AXI_GMEM" size="0x8" offset="0x1C" hostOffset="0x0" hostSize="0x8" type="void*"/>
<arg name="m_ptr2" addressQualifier="1" id="2" port="M_AXI_GMEM" size="0x8" offset="0x28" hostOffset="0x0" hostSize="0x8" type="void*"/>
</args>

It looks different to calyx generated xml:
      <port>
        <name>s_axi_control</name>
        <mode>slave</mode>
        <range>0x1000</range>
        <dataWidth>32</dataWidth>
        <portType>addressable</portType>
        <base>0x0</base>
      </port>

Thanks, xmllint is very useful. I just open the .xml file in my browser and it looks prettier.

I am investigating the source code as well, but it takes time to figure our the structure.
Thanks.

Yep, I think that confirms it! The problem I investigated above in https://github.com/cucapra/calyx/pull/1529 appears to be at fault. I'll merge that now; Calyx should then generate the right kernel.xml for you.

Yep, I think that confirms it! The problem I investigated above in #1529 appears to be at fault. I'll merge that now; Calyx should then generate the right kernel.xml for you.

Thanks for fixing that issue. Now I have correct xml fed into vivado and got .xo file.
Before I input .xo into v++, I made my .xpfm file because I am using a different board and set the frequency to 150 or 100 MHz for the IP core. When I run
$Vpp_PATH/v++ -O3 -t hw --platform=$xpfm_PATH --save-temps --profile.data all:all:all --profile.exec all:all:all -lo $SPTH/kernel.xclbin $SPTH/kernel.xo

I got this errors in v++ link about clock frequency mismatch.
Because the clock is from PS sides, so it cannot output exact 100MHz, so output 99.99MHz.
ERROR: [VPL 41-237] Bus Interface property FREQ_HZ does not match between /axi_ic_zynq_ultra_ps_e_0_S_AXI_HP0_FPD/s00_couplers/auto_ds/S_AXI(99990000) and /Toplevel_1/m0_axi(100000000)
ERROR: [VPL 41-237] Bus Interface property FREQ_HZ does not match between /System_DPA/dpa_mon1/MON_M_AXI(99990000) and /Toplevel_1/m0_axi(100000000)
ERROR: [VPL 41-237] Bus Interface property FREQ_HZ does not match between /axi_ic_zynq_ultra_ps_e_0_S_AXI_HP1_FPD/s00_couplers/auto_ds/S_AXI(99990000) and /Toplevel_1/m1_axi(100000000)
ERROR: [VPL 41-237] Bus Interface property FREQ_HZ does not match between /System_DPA/dpa_mon2/MON_M_AXI(99990000) and /Toplevel_1/m1_axi(100000000)
ERROR: [VPL 41-237] Bus Interface property FREQ_HZ does not match between /axi_ic_zynq_ultra_ps_e_0_S_AXI_HP2_FPD/s00_couplers/auto_ds/S_AXI(99990000) and /Toplevel_1/m2_axi(100000000)

When I open the generated Vivado project from v++, .sv files from primitives were not imported either.
I tried to use fun e --to to try the example dot_multiply.futil, and got this error:
[fud] ERROR: `.futil' corresponds to multiple stages: ['futil', 'calyx']. Please provide an explicit stage using --to or --from.

I am bit confused as I used --to, but still no luck.

Regarding the primitives missing files, I just changed the path in the .tcl and got them imported.

Other bug is that in the Toplevel module, the port of instantiated hardware is different:
The ports' name of the my hardware module is:
  input logic [31:0] ext_mem0_read_data,
  input logic ext_mem0_done,
  input logic [31:0] ext_mem1_read_data,
  input logic ext_mem1_done,
  input logic [31:0] ext_mem2_read_data,
  input logic ext_mem2_done,
  input logic clk,
  input logic reset,
  input logic go,
  output logic [31:0] ext_mem0_write_data,
  output logic [3:0] ext_mem0_addr0,
  output logic ext_mem0_write_en,
  output logic [31:0] ext_mem1_write_data,
  output logic [3:0] ext_mem1_addr0,
  output logic ext_mem1_write_en,
  output logic [31:0] ext_mem2_write_data,
  output logic [3:0] ext_mem2_addr0,
  output logic ext_mem2_write_en,
  output logic done

In the top level, the instantiated ports' name is:
        .clk(ap_clk),
        .done(kernel_done),
        .ext_mem0__addr0(ext_mem0__addr0),
        .ext_mem0__clk(),
        .ext_mem0__done(ext_mem0__done),
        .ext_mem0__read_data(ext_mem0__read_data),
        .ext_mem0__write_data(ext_mem0__write_data),
        .ext_mem0__write_en(ext_mem0__write_en),
        .ext_mem1__addr0(ext_mem1__addr0),
        .ext_mem1__clk(),
        .ext_mem1__done(ext_mem1__done),
        .ext_mem1__read_data(ext_mem1__read_data),
        .ext_mem1__write_data(ext_mem1__write_data),
        .ext_mem1__write_en(ext_mem1__write_en),
        .ext_mem2__addr0(ext_mem2__addr0),
        .ext_mem2__clk(),
        .ext_mem2__done(ext_mem2__done),
        .ext_mem2__read_data(ext_mem2__read_data),
        .ext_mem2__write_data(ext_mem2__write_data),
        .ext_mem2__write_en(ext_mem2__write_en),
        .go(kernel_start),
        .reset(reset || memories_sent)

There are two underscores __. So the Toplevel  cannot be synthesized as well.

@zzy666666zzy I think the double underscore are coming from the discover-external pass. If you look at the output of:
calyx -p discover-external <file>

You'll notice that the inferred memories have an _ at the end which is causing this problem. This is because when discovering the memory names, we look at port prefixes and extract the one that ends up producing all the ports on the memory primitive.
One quick solution to this is adding a strip-suffix flag to the discover externals pass and have it remove the extra _.

@zzy666666zzy I've added this feature in a pull request: https://github.com/cucapra/calyx/pull/1534. Once it's merged, you can rebuild the compiler and add the flag to remove the extra _ from the suffix of the inferred memories.

Hi @rachitnigam ,
Many thanks for the answers so far. We are very close to our objective.
We are trying to avoid using XRT because we would like to target a standalone embedded platform without an operating system. So we need to configure the AXI-wrapped IP core on the ARM core side (Vitis or Vivado SDK) manually to send data to the IP core and get the result. Our case still sticks to GEMM, where we instantiate two input matrices in ARM and get the multiplication result from the IP core. Now we have the block design:

with mapped address:

So I am trying to send some data to m1_axi, m2_axi, I found the address offset in the Toplevel AXI verilog code. For example, offset of m1_axi is 32 in decimal and 20 in hex:

Accordingly, in SDK main function I use two pointers:
volatile int *slave_addr_m1 = (int *) 0x43C00020;
volatile int *slave_addr_m2 = (int *) 0x43C00028;

pointing to m1_axi, m2_axi.
    Control_axi inst_control_axi (
        ....
        .ext_mem0_(ext_mem0_),//BASE_ADDRESS of Master0
        .ext_mem1_(ext_mem1_),//BASE_ADDRESS of Master1
        .ext_mem2_(ext_mem2_),//BASE_ADDRESS of Master2
         ...
    );

Now, we use ILA and waveforms show that we can configure s_control_axi, now. But we don't know how to send our matrices to m1_axi and m2_axi from the ARM core?
We don't have idea what the offsets of m0_axi_RDATA and m0_axi_ARADDR are, and couldn't configure the registers accordingly.
I appreciate it if there is any solution for my issue.
Many thanks!

Hi @rachitnigam @sampsyo ,
Thanks for the suggestions, I have probably managed to configure the address and offsets of the IP core.
I am trying to verify a simple 4x4 GEMM. When I use ILA to observe the data transferred from ARM to the IP, I found two issues:
1. I initialize integer elements in matrix A as
0,0,0,0
1,1,1,1
2,2,2,2
3,3,3,3

and matrix B as
0,1,2,3
0,1,2,3
0,1,2,3
0,1,2,3

and transfer it to the IP, Let's see the transferred data of the 2nd row in matrix A. All 32-bit data was duplicated up to 512-bit in each handshake:

Also, in the 1st row in matrix B, duplicated data contains two elements, but I think each handshake should just transfer one element:

.
This data transfer seems not ideal. Maybe I should modify my host code on ARM?
2. I try to catch the ap_done signal, indicating the computing is finished, but the host program hangs in if((*slaveaddr_ap_done)==1) in my code. I think it is because of SEND_TO_HOST_DONE and related signals in AXI Toplevel.sv.
I attached my host code, and address info from Vivado address editor here for a better understanding of my confusion.
helloworld.txt

I really appreciate it if there would be some advice.
Thanks a lot!

That's incredible! Thank you for pushing the Calyx design through. Please feel free to open more issues if you need any help from us

Indeed; awesome! We would love to hear how this project proceeds if you ever have any updates to share. :smiley:


Make compatibility with reason-apollo optional

Currently we require reason-apollo to be installed along with reason-apollo-hooks to ease the transition, now we can drop it and make this optional

That would be very nice!
There are however some useful bindings that come with reason-apollo, particularly when it comes to working with cache. ApolloClient had WriteQuery and ReadQuery for this (which could actually benefit from some improvements..) Do you think we should create our own bindings for those?

I think it makes sense! Ideally we should split the bindings like react-apollo repo. common, components, hoc, hooks. Then there's react-apollo which has all of them.
https://github.com/apollographql/react-apollo/tree/master/packages

That would be much better indeed @cem2ran, I think I can talk with @Gregoirevda about this. It'll scale much better

Sure, let's do that

With Apollo Client 3.0 deprecating the react-apollo package, and the reason-apollo bindings not really being updated, isn't it better to make this package self-containing? Include the read/write query code and some of the types in this package. This also means we can easily update to the Apollo Client 3.0 beta.

We could probably make this repo a monorepo and have a separated package for apollo-client


latest version breaking on android with react-native 0.41.2

I followed example and used following code, but it is just crashing  app as soon as I am clicking on use camera options on the other hand it is working perfectly when I am using select image from library.

ImagePicker.showImagePicker(options, (response) => {
console.log('Response = ', response);
if (response.didCancel) {
console.log('User cancelled image picker');
}
else if (response.error) {
console.log('ImagePicker Error: ', response.error);
}
else if (response.customButton) {
console.log('User tapped custom button: ', response.customButton);
}
else {
let source = { uri: response.uri };
this.setState({
avatarSource: source
});
}
});

"dependencies": {
"react": "15.4.2",
"react-native": "0.41.2",
"react-native-image-picker": "^0.25.5"
},

@manjeets12 Could you give more information about crash? I mean logs from Android.
Did you add the next lines to the android/app/src/main/AndroidManifest.xml
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

Yes I have added those lines,
here is my code and errors image

const imageOptions = {
quality: 1,
maxWidth: 500,
maxHeight: 500,
storageOptions: {
skipBackup: true
}
};
ImagePicker.launchCamera(imageOptions, (response)  => {
// Same code as in above section!
let source = { uri: response.uri };
this.setState({
avatarSource: source
});
});

@manjeets12 What version of Android did you use for testing? Was it device or emulator? If it was emulator, will you write emulator params for repeating bug?

I am testing on my micromax mobile (android 5.1)

Thanks.

@manjeets12 Do you use proguard in your Android project?

no I didn't , even I don't have a idea about that,

yes I just seen there is an autogenerated progaurd file

Add project specific ProGuard rules here.
By default, the flags in this file are appended to flags specified
in /usr/local/Cellar/android-sdk/24.3.3/tools/proguard/proguard-android.txt
You can edit the include path and order by changing the proguardFiles
directive in build.gradle.

For more details, see
http://developer.android.com/guide/developing/tools/proguard.html
Add any project specific keep options here:
If your project uses WebView with JS, uncomment the following
and specify the fully qualified class name to the JavaScript interface
class:
#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
public *;
#}
Disabling obfuscation is useful if you collect stack traces from production crashes
(unless you are using a system that supports de-obfuscate the stack traces).
-dontobfuscate
React Native
Keep our interfaces so they can be used by other ProGuard rules.
See http://sourceforge.net/p/proguard/bugs/466/
-keep,allowobfuscation @interface com.facebook.proguard.annotations.DoNotStrip
-keep,allowobfuscation @interface com.facebook.proguard.annotations.KeepGettersAndSetters
-keep,allowobfuscation @interface com.facebook.common.internal.DoNotStrip
Do not strip any method/class that is annotated with @DoNotStrip
-keep @com.facebook.proguard.annotations.DoNotStrip class *
-keep @com.facebook.common.internal.DoNotStrip class *
-keepclassmembers class * {
@com.facebook.proguard.annotations.DoNotStrip *;
@com.facebook.common.internal.DoNotStrip *;
}
-keepclassmembers @com.facebook.proguard.annotations.KeepGettersAndSetters class * {
void set*(**);
*** get();
}
-keep class * extends com.facebook.react.bridge.JavaScriptModule { *; }
-keep class * extends com.facebook.react.bridge.NativeModule { *; }
-keepclassmembers,includedescriptorclasses class * { native ; }
-keepclassmembers class *  { @com.facebook.react.uimanager.UIProp ; }
-keepclassmembers class *  { @com.facebook.react.uimanager.annotations.ReactProp ; }
-keepclassmembers class *  { @com.facebook.react.uimanager.annotations.ReactPropGroup ; }
-dontwarn com.facebook.react.**
okhttp
-keepattributes Signature
-keepattributes Annotation
-keep class okhttp3.** { ; }
-keep interface okhttp3.* { ; }
-dontwarn okhttp3.*
okio
-keep class sun.misc.Unsafe { ; }
-dontwarn java.nio.file.
-dontwarn org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement
-dontwarn okio.**

@manjeets12 please look at android/app/build.gradle. What does enableProguardInReleaseBuilds key value have? True or False?

@rusfearuth  actually it is false right as I am still working on development version, will set set to true for realease version

Same problem here, but with react-native version 0.40.0
I am testing with android 5.1 too.
I will came back as soon as test with another device.
OBS:
launchImageLibrary works without problems, the error appear when I call launchCamera

I notice that the provider authorities generated by gradle disagree with authority requested by code
The gradle replaced android:authorities="${applicationId}.provider" with android:authorities="com.imagepicker.provider" instead of the final applicationId.
Apparently this is a gradle bug, but for fast fix I replaced ${applicationId} with my aplication id in node_modules/react-native-image-picker/android/src/main/AndroidManifest.xml

To avoid edit react-native-image-picker source I edit my manifest and added:
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    [...]
      <provider tools:replace="android:authorities"
          android:name="android.support.v4.content.FileProvider"
          android:authorities="${applicationId}.provider"
          android:exported="false"
          android:grantUriPermissions="true">
          <meta-data
              android:name="android.support.FILE_PROVIDER_PATHS"
              android:resource="@xml/provider_paths" />
      </provider>
    </application>

</manifest>

play attention to the addition of namespace tools on line 2
this force gradle use your definition of provider instead of the definition of react-native-image-picker.

@dsvictor94 👍 Going to make fix

Sorry, I installed the master branch to teste if the commit solve the problem I it just added this warning
Warning: /home/victor/victor/kissed/android/app/src/main/AndroidManifest.xml:13:9 Warning:
	provider#android.support.v4.content.FileProvider@android:authorities was tagged at AndroidManifest.xml:13 to replace other declarations but no other declaration present
/home/victor/victor/kissed/android/app/src/main/AndroidManifest.xml:13:9 Warning:
	provider#android.support.v4.content.FileProvider@android:authorities was tagged at AndroidManifest.xml:13 to replace other declarations but no other declaration present

and nothing solved, the merged manifest remains incorrect
        [...]
        <provider
            android:name="android.support.v4.content.FileProvider"
            android:authorities="com.imagepicker.provider"
            android:exported="false"
            android:grantUriPermissions="true" >
            <meta-data
                android:name="android.support.FILE_PROVIDER_PATHS"
                android:resource="@xml/provider_paths" />
        </provider>
    </application>

I think I was misunderstood in my last comment (probably because of my poor English), the workaround  must be applied to the main manifest (the same you add permissions), not in the library manifest.
I thing there are nothing you can do based on my workaround except add a note in installation instruction

@rusfearuth ^ seems like we should roll back this commit

@marcshilling Yeah... I think too and look better solution.

@dsvictor94 Have you changed android section in your app gradle build file?

bug still on lastest version 0.25.7 (my device : android 4.4.2).
Crash on Camera option.
Debug log :
"02-27 18:18:19.858 29125-29172/com.ezeesell W/ContextImpl: Failed to ensure directory: /storage/extSdCard/Android/data/com.ezeesell/files/Pictures"

@hoavo It seems that catalog wasn't created

I think nobody have change the build.gradle file, but if it helps
build.gradle.txt

@dsvictor94 I've changed )

@dsvictor94 Okey, I'm going to create test project with using settings like yours.

@manjeets12 and @dsvictor94
Could you check android/build.gradle?
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.2.+' // <============= is it lower 2.2?
    }
}

if it's lower, you will need to update it to 2.2.+
Then you should update android/gradle/wrapper/gradle-wrapper.properties to
#Thu Jun 09 11:55:07 EDT 2016
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-2.14.1-all.zip ### <====== TO THIS

And try again.
Waiting for your report )

@hoavo Did crash happen after you confirm of taken photo?

I am without a device now, but I build and inspect de manifest and apparently solves the problem.
Sorry when I said the my gradle version is 2.14.1, I saw the gradle version to the wrong project. It was 2.4.

@dsvictor94 If I understand right, now is it fine?

I can't make sure because because I am without a device, but yes, it's fine now.

I tested today and upgrading the gradle (and de gradle plugin) solves the provider authorities problem. @rusfearuth, thanks a lot by your fast support

@rusfearuth Crash after capture a photo.
Thanks for your support

@hoavo I found why crash happens. Your case described in #478.

@dsvictor94 @manjeets12 I think we can close this issue. Do you?

@rusfearuth I will confirm it soon,

My device(SO-01F - 4.4.2) crashed by FileProvider issue.
Both build.gradle and gradle-wrapper.properties are following above codes.
But I found react-native-image-crop-picker working on using camera.
So I changed the line in ImagePickerModule like react-native-image-crop-picker.
Now it seems to me working, but I don't know why it works.
// In compatUriFromFile
// ...
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP)
    {
      result = Uri.fromFile(file);
    }
// ...

@tanaka0x Could you check a new version?

@rusfearuth I tried 0.26.1 with RN 0.42, it can take a photo using camera.
I think this issue was fixed in my case.
However sometimes another problem occured I think it is related to #471


Binding getter to attribute without computedFrom causes infinte loop

I have a custom element for media uploads that displays either the media that has already been saved or the blob url of the media about to be uploaded.
In my VM I have a getter to workout the correct url for the media:
export class UploadField {
  @bindable field;

  fileToUrl(fileList) {
    return fileList ? URL.createObjectURL(fileList[0]) : false;
  }

  get src() {
    return this.fileToUrl(this.field.files) || this.field.src || '';
  }
}

and in my view I have:
<video controls if.bind="field.element == 'video' && src" src.bind="src"></video>
<img if.bind="field.element == 'img' && src" src.bind="src"></img>

This appears to work, however on closer inspection if I add a file to the input it seems to infinite loop and repeatedly creates new blob urls and updates the src. Adding @computedFrom resolves the issue and everything works as expected:
@computedFrom('field.files', 'field.src')
get src() {
  return this.fileToUrl(this.field.files) || this.field.src || '';
}

@EisenbergEffect is this expected behaviour?

yes, getters are observed with dirty-checking.


Add support for elaborated types.

Things like JS::Latin1Char now show up as elaborated types and we need to use clang_Type_getNamedType to resolve them.
This should fix #1.

cc @emilio @nox

@edunham This repo is not reported to IRC or homu yet. Can you turn that on?

So that's something that changed in libclang itself or something like that?

@nox It seems so.
@metajack Thanks for digging into this, my laptop OOMd the first time I tried to build LLVM 3.9 so I left it building again with -j1 and went to bed. Adding a test for this would be ideal (tests/headers/x.hpp + make test, yeah, I know), but I can take care of it.

@emilio Tests added and @nox's comment addressed.

@emilio re-r?

r+, I'm looking into CI issues right now (#5) that should allow us to autoland this instead of merging manually, but feel free to do so.

@bors-servo: r+

:pushpin: Commit 1c4ca74 has been approved by emilio

:hourglass: Testing commit 1c4ca74 with merge 00ddb0e...

:broken_heart: Test failed - travis

:umbrella: The latest upstream changes (presumably #6) made this pull request unmergeable. Please resolve the merge conflicts.


Publish ranges of amount to operate in buy orders

Add new fields in order schema
Create scene to manage input
Add input validation
Add handlers to build description
Add logic to relaunch orders
Add switch to change logic between simple and range orders

Resolves #151 (partially)

Seller take an order
Seller add fiat amount correctly
Then didn't pay the invoice
The order is canceled but order.fiat_amount is not set back to 0


[SUPPORT] MOR table found duplicate and process so slowly

I want to have a pipeline consume incremental data from Kafka. first i have a full data import to a hudi mor table file size around 23G, When this is done, everything is all right. But when i start consuming incremental data from kafka, it's soooo slow and then when i check the data, i found much duplicate records, and after runing incremental data ingesting the file size now is 93.9G, it made me curious, cause i used GLOBAL_BOOM, from the doc i saw the desc below, so i think if i got the same record key, then no need to worry about the duplicate, the same record key records will be updated.

/**

Only applies if index type is GLOBAL_BLOOM.

When set to true, an update to a record with a different partition from its existing one
will insert the record to the new partition and delete it from the old partition.

When set to false, a record will be updated to the old partition.
*/

And u see just a little bit data, it process so slowly, need your help  @bvaradar , thanks sooo much.

Here is my option:

hoodie.datasource.write.recordkey.field -> "id"
hoodie.datasource.write.precombine.field -> "_ts"
hoodie.datasource.write.partitionpath.field -> "dt"
hoodie.datasource.write.table.type -> "MERGE_ON_READ"
hoodie.datasource.hive_sync.database -> "schema"
hoodie.datasource.hive_sync.table -> "table"
hoodie.datasource.hive_sync.enable -> "true"
hoodie.datasource.hive_sync.jdbcurl -> "jdbc:hive2://xxxxxxx"
hoodie.table.name -> "table"
hoodie.datasource.hive_sync.partition_extractor_class -> "org.apache.hudi.hive.MultiPartKeysValueExtractor"
hoodie.index.type -> "GLOBAL_BLOOM"
hoodie.datasource.hive_sync.partition_fields -> "dt"
hoodie.compact.inline -> "true"
hoodie.parquet.small.file.limit -> "104857600"

Hudi version :
0.6
Spark version :
2.4.6
Hive version :
3.1.2
Hadoop version :
3.1.2
Storage (HDFS/S3/GCS..) :
HDFS
Running on Docker? (yes/no) :
no

@nsivabalan : Can you take a look at this.
Thanks,
Balaji.V

@nsivabalan : Can you take a look at this.
Thanks,
Balaji.V

Thanks for your quick reply

@so-lazy :would u mind elaborating more on your use-case. did you choose Global_bloom intentionally?
And by this statement of yours "i found much duplicate records,..", did you mean to insinuate that compaction hasn't happened and hence you found duplicates or did you refer in general your dataset has duplicates?
Do you want to do dedup for your use-case in general?

If you wish to understand diff indexing schemes, please refer to this blog.

@so-lazy :would u mind elaborating more on your use-case. did you choose Global_bloom intentionally?
And by this statement of yours "i found much duplicate records,..", did you mean to insinuate that compaction hasn't happened and hence you found duplicates or did you refer in general your dataset has duplicates?
Do you want to do dedup for your use-case in general?

This is my mysql table
id | name | add_time
1  | "so-lazy" | 2020-12-26
yeah, for my use case, i consume binlog delta data from kafka and those data they have primary key "id", i set dt based on the column add_time,  and format is yyyy-MM-dd. What i want is if one row id is 1 and at the first time hudi  upserts this row into table partitiion 2020-12-26, then next time this id 1 come again,it will be updated.
But now what i found is , when i use spark-sql search "select * from table where id = 1" on hive table synced by hudi, i saw many the same records they are totally the same, all the column values  they are  the same, also in the same parquet file, but when i read this parquet file by spark, i can only find one row which id = 1 in that parquet file.
Sir do u understand what i mean, thanks sooo much.

I run select id,count(1) from table_ro group by id having count(1) > 1 got this

then i query data by id got this

then i go to check the parquet file
val df = spark.read.parquet("/hudi/2020-12/c147c22c-7e86-4ae9-bc77-e2a2a52cc897-0_0-230-113454_20201215205055.parquet") df.createOrReplaceTempView("table") spark.sql("select _hoodie_commit_time,_hoodie_commit_seqno,_hoodie_record_key,_hoodie_partition_path,_hoodie_file_name,id from table WHERE id = 338409469").show

I run select id,count(1) from table_ro group by id having count(1) > 1 got this

then i query data by id got this

then i go to check the parquet file
val df = spark.read.parquet("/hudi/2020-12/c147c22c-7e86-4ae9-bc77-e2a2a52cc897-0_0-230-113454_20201215205055.parquet") df.createOrReplaceTempView("table") spark.sql("select _hoodie_commit_time,_hoodie_commit_seqno,_hoodie_record_key,_hoodie_partition_path,_hoodie_file_name,id from table WHERE id = 338409469").show

when you use spark-sql,you can try set spark.sql.hive.convertMetastoreParquet = false;
I ran into the same problem before yesterday

@nsivabalan Sir,can u help me out of this? I am so curious, thanks too much

@nsivabalan Sir,can u help me out of this? I am so curious, thanks too much

@so-lazy : I am looping in @bvaradar to help you out here. But in the mean time, some context around Global_Bloom. Hudi has two kinds of indexes, regular and global. in regular bloom, all record keys within a partition are unique, but there could be same record key across diff partitions. Within same partition, hudi will take care of updating the records based on record keys and will serve you only the latest snapshot for every record key of interest.
Where as in Global versions, record keys across the entire dataset is unique. in other words, there can't be same record key in different partitions. So, incase you insert a record, rec_1 in partition1 and later try to insert the same record(rec_1) to a diff partition, say partition2, Hudi by default will update the record in partition1. But there is a config which you can set, on which case, hudi will delete this record, rec1 of interest from partition1 and will insert to partition2.
This is the major difference between regular and global versions of index. Since in Global version, all partitions need to be looked up for all records, it is known to be less performant compared to regular index. So, unless you have this requirement, would suggest you to use regular indexes (BLOOM for ex).

@so-lazy : I am looping in @bvaradar to help you out here. But in the mean time, some context around Global_Bloom. Hudi has two kinds of indexes, regular and global. in regular bloom, all record keys within a partition are unique, but there could be same record key across diff partitions. Within same partition, hudi will take care of updating the records based on record keys and will serve you only the latest snapshot for every record key of interest.
Where as in Global versions, record keys across the entire dataset is unique. in other words, there can't be same record key in different partitions. So, incase you insert a record, rec_1 in partition1 and later try to insert the same record(rec_1) to a diff partition, say partition2, Hudi by default will update the record in partition1. But there is a config which you can set, on which case, hudi will delete this record, rec1 of interest from partition1 and will insert to partition2.
This is the major difference between regular and global versions of index. Since in Global version, all partitions need to be looked up for all records, it is known to be less performant compared to regular index. So, unless you have this requirement, would suggest you to use regular indexes (BLOOM for ex).

@so-lazy : I think the hive table may not be a hudi table. can you show the output of the following hive command ?
desc formatted table
Also, can you please attach the listing of .hoodie folder ?
Thanks,
Balaji.V

@so-lazy : I think the hive table may not be a hudi table. can you show the output of the following hive command ?
desc formatted table
Also, can you please attach the listing of .hoodie folder ?
Thanks,
Balaji.V

@so-lazy : I think the hive table may not be a hudi table. can you show the output of the following hive command ?
desc formatted table
Also, can you please attach the listing of .hoodie folder ?
Thanks,
Balaji.V

@bvaradar Here are my table desc and .hoodie folder, thanks so much.
desc table.txt
hoodie.tar.gz

@so-lazy : I think the hive table may not be a hudi table. can you show the output of the following hive command ?
desc formatted table
Also, can you please attach the listing of .hoodie folder ?
Thanks,
Balaji.V

@bvaradar Here are my table desc and .hoodie folder, thanks so much.
desc table.txt
hoodie.tar.gz

@so-lazy :
when you query through spark datasource (not just single file), are you able to see unique record ?
val df = spark.read.format("hudi").load("hdfs://hadoop01:9000/hudi/cars/carsdata/inf_car_bin/*")
....
Also, Are you passing the config (spark.sql.hive.convertMetastoreParquet=false) when you are launching spark ? https://hudi.apache.org/docs/querying_data.html#spark-sql.
Also, I see you have space around "=" sign (set spark.sql.hive.convertMetastoreParquet = false;)   Try removing it. Please also enable INFO logging and run the select group by query and attach them.

@so-lazy :
when you query through spark datasource (not just single file), are you able to see unique record ?
val df = spark.read.format("hudi").load("hdfs://hadoop01:9000/hudi/cars/carsdata/inf_car_bin/*")
....
Also, Are you passing the config (spark.sql.hive.convertMetastoreParquet=false) when you are launching spark ? https://hudi.apache.org/docs/querying_data.html#spark-sql.
Also, I see you have space around "=" sign (set spark.sql.hive.convertMetastoreParquet = false;)   Try removing it. Please also enable INFO logging and run the select group by query and attach them.

@so-lazy :

Also, I see you have space around "=" sign (set spark.sql.hive.convertMetastoreParquet = false;) Try removing it. Please also enable INFO logging and run the select group by query and attach them if the problem persists.

@bvaradar Sorry,bvaradar, these days i was so busy didn't reply on time. Today i followed your suggest and attach  screen shot.

val df = spark.read.format("hudi").load("hdfs://hadoop01:9000/hudi/cars/carsdata/inf_car_bin/*")
run this i got unique record

run sql on hive ro table

Also, Are you passing the config (spark.sql.hive.convertMetastoreParquet=false) when you are launching spark ? https://hudi.apache.org/docs/querying_data.html#spark-sql.

Yeah after passing this config, i can see unique record. thanks so much , and now i am gonna try other index type, for global_bloom , it's so slowly may be there are some problems in my program. Hudi is a good software , thanks for your time and effort.

@so-lazy :

Also, I see you have space around "=" sign (set spark.sql.hive.convertMetastoreParquet = false;) Try removing it. Please also enable INFO logging and run the select group by query and attach them if the problem persists.

@bvaradar Sorry,bvaradar, these days i was so busy didn't reply on time. Today i followed your suggest and attach  screen shot.

val df = spark.read.format("hudi").load("hdfs://hadoop01:9000/hudi/cars/carsdata/inf_car_bin/*")
run this i got unique record

run sql on hive ro table

Also, Are you passing the config (spark.sql.hive.convertMetastoreParquet=false) when you are launching spark ? https://hudi.apache.org/docs/querying_data.html#spark-sql.

Yeah after passing this config, i can see unique record. thanks so much , and now i am gonna try other index type, for global_bloom , it's so slowly may be there are some problems in my program. Hudi is a good software , thanks for your time and effort.

@bvaradar sir, now i used global simple index,  but for some satages
Getting small files from partitions
Compacting file slices
they cost so long mintues, and i attach my option, am i wrong with some config?
By the way i searched a lot about global_bloom and global_simple, i am still not so clear, what's the  difference between them?

@bvaradar sir, now i used global simple index,  but for some satages
Getting small files from partitions
Compacting file slices
they cost so long mintues, and i attach my option, am i wrong with some config?
By the way i searched a lot about global_bloom and global_simple, i am still not so clear, what's the  difference between them?

cc @nsivabalan
Not sure if you saw this blog about index usages :https://hudi.apache.org/blog/hudi-indexing-mechanisms/
The stage names could be misleading. It is likely the index lookup is running and not finding the small files.  Regarding compaction, I see that there are 413 tasks. Do you have sufficient executors to have them run in parallel. Each task would be compacting the delta files and creating a new parquet file.

cc @nsivabalan
Not sure if you saw this blog about index usages :https://hudi.apache.org/blog/hudi-indexing-mechanisms/
The stage names could be misleading. It is likely the index lookup is running and not finding the small files.  Regarding compaction, I see that there are 413 tasks. Do you have sufficient executors to have them run in parallel. Each task would be compacting the delta files and creating a new parquet file.

@so-lazy : Can you please respond to Balaji's comment when you get a chance.
few more questions as we triage the issue.
When you loaded the data to hudi for the first time, did you use bulk-insert of insert operation. Were the configs you used to load the data into hudi first time is same as the ones you have given in description of this issue? If not, would you mind posting those configs as well.

Closing due to inactivity. but feel free to reopen to create a new ticket. would be happy to assist you.


feat(slider): create a CDK slider component

It would be nice to have a CDK slider that allows for more user customization than mat-slider.
Things to support:

Multiple thumbs (should follow a11y pattern here: http://oaa-accessibility.org/example/32/)
Provides elements to style for things like thumb, filled track, non-filled track, etc, but leaves that styling up to the user
Should be able to base mat-slider off of it

multiple thumbs would be a nice feature.
If you click the track, I think you'd want the closest thumb to move to the click position.
And if you drag one thumb into another there are two options:

STOP - as done in your link: http://oaa-accessibility.org/example/32/
PUSH - push the other thumb as you drag the other into it.

If two thumbs are on top of each other, it creates an awkward situation.
When you click and drag, which one does it pickup?
If you drag one direction, it could be pushing against the other (as in the oaa-accessibility demo)
So maybe the best solution would be to not "pickup" a thumb initially and wait to see which direction the user drags and then pickup the thumb open to that direction. But that may be difficult to code.

I would love to have multiple thumbs feature
But most importantly, I would like the feature for thumb to be always visible (and not just on focus)

One extra feature I would like:
if there is a max number set (let's say 100) I would like that if I slide to that number and hold on a bit.... the scale widens... to my desired number .... this is a nice to have.... but the number 2 on my list is a must imho

When we can expect range slider in angular material guys..?

Came across #4803 and since that was closed, going to post this here:
Figured out how to show the slider thumb label underneath the slider / didn't want to open a new issue / for posterity / didn't find this talked about elsewhere:
jJst noticed @vsarathy mentioned it (almost a year ago, but).
::ng-deep {
  .mat-slider-horizontal .mat-slider-thumb-label{
    top: 10px !important;
    transform: translateY(-26px) scale(0.1) rotate(225deg) !important;
  }
  .mat-slider-horizontal.cdk-focused .mat-slider-thumb-label {
    transform: rotate(225deg) !important;
  }
  .mat-slider-thumb-label-text {
    transform: rotate(135deg) !important;
  }
}

STOP - as done in your link: http://oaa-accessibility.org/example/32/
PUSH - push the other thumb as you drag the other into it.

I wonder if any thumb should be freely dragged from 0 to 100% without bumping into each other (I mean when thumb is blocked for further drag to specified side) and without linking that this one thumb is for left part and another is for right part. Min and max values would be reactively calculated depending on current thumbs positions. It might provide better UX for specified cases depending on user actual needs (just speculating though, I didn't do any research on this).

Is this feature already available?

Any news on this? Have a project that would greatly benefit from more customization options

Hey, unfortunately this is not started and not planned at the moment. Realistically, with the amount of other things taking priority over this, I wouldn't bet on this getting done in 2023 :(


Blog Post Image for Release Notes 1.8

Related Issue
https://github.com/flowforge/website/pull/820
Description
Considerations:
Where the image will be displayed/shared: blog post
For next week's Release Notes, it would be great to have an image showcasing the new highlights of the 1.8 release.
The highlights for the image are:

High Availability (https://github.com/flowforge/flowforge/issues/2156)
DevOps Pipelines (https://github.com/flowforge/flowforge/issues/1492)

Inspirations:
/blog/2023/04/images/release-1.6.0.png

@Yndira-FlowForge "visualisation of" under-sells it. "Enhanced DevOps Pipelines" maybe a little better? Or "DevOps Deployment Pipelines" - as even though this functionality was in before, it was very well hidden

as even though this functionality was in before, it was very well hidden

Yeah, I noticed we mentioned it on release 1.4
Here are the two options


Fix for 592

Description
Go back to tr/catch for requiring async code.
Issues Resolved
Closes #592.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@harshavamsi You are absolutely right. Switch to using nox in CI.


Godot editor UI elements don't focus correctly when launched from Terminal

MacBook Pro running OSX Sierra 10.12.6
godot current master.
Issue description:
Godot editor window focus is buggy, doesn't always assume a "foreground" position when launched from the Terminal
Steps to reproduce:

git clone godot
scons build following instructions
Focus your web browser i.e. firefox
Go back to terminal and run "./bin/godo..."
Top title bar will say "Firefox" and godot will seem to only register double click events on the UI and not single clicks (window border will be grayed out even though it's focused...). You can fix by minimising godot editor, selecting Terminal in the dock, manually left pressing the terminal area and then maximising godot. Then it will say "godot.osx.tools.64" in the top left system menu bar and things work as expected

You have to repeat these steps every time the active window changes i.e. when moving from the project selector to the main editor
This one had me scratching my head. I almost posted a bug that touchpad isn't working ..

#8907 duplicate, use .app bundle, instead of stand-alone executable.

Closing as duplicate of #8907, will add another comment there.


LongRunning Client.GetOperation() fails with async speech operation name.

Sample code here:
https://github.com/SurferJeffAtGoogle/csharp-docs-samples/blob/f219a7cafd26b64212574ab0e44ab2d6134477bc/speech/api/TranscribeAsync/Program.cs
Yields this error:
Test Name:	GoogleCloudSamples.TranscribeAsyncTest.TranscribeAudio
Test FullName:	GoogleCloudSamples.TranscribeAsyncTest.TranscribeAudio
Test Source:	C:\Users\Jeffrey Rennie\gitrepos\dotnet-docs-samples\speech\api\TranscribeAsyncTest\TranscribeAsyncTest.cs : line 51
Test Outcome:	Failed
Test Duration:	0:00:32.902

Result StackTrace:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Grpc.Core.Internal.AsyncCall`2.UnaryCall(TRequest msg)
   at Grpc.Core.Calls.BlockingUnaryCall[TRequest,TResponse](CallInvocationDetails`2 call, TRequest req)
   at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
   at Grpc.Core.Internal.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
   at Google.Longrunning.Operations.OperationsClient.GetOperation(GetOperationRequest request, CallOptions options)
   at Google.Api.Gax.Grpc.ApiCall.<>c__DisplayClass0_0`2.<Create>b__1(TRequest req, CallSettings cs)
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass1_0`2.<WithRetry>b__0(TRequest request, CallSettings callSettings)
   at Google.Api.Gax.Grpc.ApiCall`2.<>c__DisplayClass6_0.<Sync>b__0(CallSettings callSettings)
   at Google.Api.Gax.Grpc.ApiCall`2.Call[T](TRequest request, CallSettings perCallCallSettings, Func`2 fn)
   at Google.Api.Gax.Grpc.ApiCall`2.Sync(TRequest request, CallSettings perCallCallSettings)
   at Google.Longrunning.OperationsClientImpl.GetOperation(String name, CallSettings callSettings)
   at GoogleCloudSamples.TranscribeAsync.Main(String[] args) in C:\Users\Jeffrey Rennie\gitrepos\dotnet-docs-samples\speech\api\TranscribeAsync\Program.cs:line 48
   at GoogleCloudSamples.CommandLineRunner.Run(String[] arguments) in C:\Users\Jeffrey Rennie\gitrepos\dotnet-docs-samples\testutil\TestUtil.cs:line 107
   at GoogleCloudSamples.TranscribeAsyncTest.Run(String[] arguments) in C:\Users\Jeffrey Rennie\gitrepos\dotnet-docs-samples\speech\api\TranscribeAsyncTest\TranscribeAsyncTest.cs:line 38
   at GoogleCloudSamples.TranscribeAsyncTest.TranscribeAudio() in C:\Users\Jeffrey Rennie\gitrepos\dotnet-docs-samples\speech\api\TranscribeAsyncTest\TranscribeAsyncTest.cs:line 52
Result Message:	Grpc.Core.RpcException : Status(StatusCode=Unknown, Detail="")

Looking at the asyncResponse:
+		asyncResponse	{{ "name": "2592828109219406248" }}	Google.Longrunning.Operation

Right, that code won't work because OperationsClient needs to be tied to the gRPC channel that the Speech API uses. Basically we still need to overhaul the Speech client to expose the LRO part.
If you really want to work on this right now, you could manually create a channel, and pass that to both the SpeechClient.Create and OperationsClient.Create methods - but I'd wait if you possibly can.

I'll wait.  When the speech client is ready, please close this bug to signal me to write the sample.

v1.0.0-alpha03 is now on nuget :)


Fix isuue #1271 | Medium Issue 1 (CSS)

Pull Request Type
What kind of change does this PR introduce?

Solved an issue/task

What is the current behavior?

Issue Number: https://github.com/fork-commit-merge/fork-commit-merge/issues/1271
What is the new behavior?

Implement this class in the styles.css

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out our forkcommitmerge.io website repository. The repository contain real issues that you can help to resolve or create some of your own if you see any bugs or want to add new features to the site, etc. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!