Dynamic slots allocation with telemetry base structure

Memory window 2 is divided into 8 pages: 7 slots and a page with descriptors for those slots. Now they are allocated dynamically. Log backend mtrace has been adjusted to this mechanism as it has been hardcoded before.
Base telemetry data structure has been added, slot for telemetry is allocated and some data is written.
To enable this mechanism please add:
CONFIG_DYNAMIC_SLOTS=y
CONFIG_DEBUG_MEMORY_WINDOW=y
CONFIG_TELEMETRY_INTEL=y
CONFIG_LOG_BACKEND_ADSP_MTRACE=y
to the board config.

@kv2019i We lost our conversation due to my mistake, but as to cavs not working, I remember that I had no output on ACE when HP_SRAM_RESERVE and MEMORY_WIN_2_SIZE were not set properly. Before I set them right only for ACE platforms. Now I've added the same for CAVS. Please check again. It's getting like really late and as you know I'm leaving the project today, so most probably I will not be able to check if it works. I only managed to test on ACE platform.
I've also tried to add sparse cache fixes @lyakh suggested, but did not manage to test if I've done it all right as well.
The person that will take over this PR should modify west.yml https://github.com/thesofproject/sof/pull/6937 like @marc-hb  suggested and check if that helped.

@piotrkmx  wrote:

@kv2019i We lost our conversation due to my mistake, but as to cavs not working, I remember that I had no
output on ACE when HP_SRAM_RESERVE and MEMORY_WIN_2_SIZE were not set properly. Before I set them right only for
ACE platforms.

Hmm, this is probably the reason. I tried again, but stll not getting logs via the Linux driver. I now debugged a bit and I can see mtrace_out() called, and can see the DSP write_ptr updated, but the kernel side is not seeing the updates, so something is still missing. But initialization seem to be ok, and mtrace is getting called for logs.

It was replaced by another PR. please close it.

This PR was discarded and the new design was merged :https://github.com/zephyrproject-rtos/zephyr/commit/0bff695d2c997363eb0c6cbf9421b056d645ea4a.  So please close it.


Added B rank enabled/disabled via UI config

along with all ranks enabled/disabled via user config

Thank you!! Hopefully that wasn't too painful.  :)


[TASK] Run brew without sudo

Homebrew prefers not to be executed as sudo

@dhuf: Thanks for your PR ;-)


Propose Demos for Weekly All-Hands Here!

We've allocated time for Demos on the Weekly All-Hands call. What should be showcased there? Who should we invite to give demos?
Add comments here if you have ideas, or if you would like to give a demo.

Should we invite people from awesome-ipfs?

Feels like this should be a little bit tied to awesome-ipfs. At least we can invite the people who create stuff that goes into the list to talk about their own thing.
Also, people from the things mentioned here: https://github.com/ipfs/awesome-ipfs/issues/65

I would love to see the IPFS Pics people come and talk about their work. Also, I would like to know more about Mesh networks and if they use IPFS at all (@lgierth?).

Yeah, maybe @cloutier is interested in participating next Monday calls to talk about IPFS Pics?

Last week, July 3, 2017 we had:

IIIF Annotations over IPFS - https://www.youtube.com/watch?v=hmAniA6g9D0&feature=youtu.be&t=10m40s
Ethereum over IPFS - https://www.youtube.com/watch?v=hmAniA6g9D0&feature=youtu.be&t=35m42s

🎉


[MapBox lookup] Ommited result

Expected behavior
Using Mapbox as the lookup API I'm getting some ommited results
e.g -> "geometry"=>{"type"=>"Point", "coordinates"=>[X, Y], "interpolated"=>true, "omitted"=>true}
I want to accept these results as valid instead of ignoring and using the next one in the array.
Actual behavior
Geocoder actually ignores ommited: true results instead of using them
Steps to reproduce
Geocoder.search("Rua Egas Moniz 8, 1900-218, Lisboa")
Environment info

Geocoder version: 1.8.3
Rails version: 8.0.1
Database (if applicable):
Lookup (if applicable): mapbox

I'm not sure I understand this issue. Some of the results returned by Geocoder have omitted: true and some don't. Are you saying Geocoder should be handling these results differently?

@alexreisner if the first result is ommited it's ignored and the second one is used.
I'd like a way to override this behaviour. To use the first value even if it's ommited

Hmm, I think I need more context. Can you share some code that shows the result being ignored?


Downloading is very slow

Relevant issue mps-youtube/pafy#199.
For the time being, we could just add this header "Range": "bytes=0-" when downloading streams as mentioned in https://github.com/mps-youtube/pafy/issues/199#issuecomment-368257981 to fix on download speeds on relatively smaller downloads.

87984addcc1e1d9c725872a1197fa32cec426cc8 partially fixes this.


Color active state on link does not return to original color

Version
3.0.2
Reproduction link
https://codesandbox.io/embed/vue-template-lgn63
Steps to reproduce

Open the project in the browser.
The link with the name Say Hello appears in the browser.
The default color is green of the Say Hello link is green, when hovering or clicking on the link the color changes from green to gray.
The gray color remains in the Say Hello link, the expected behavior is that once you click and refresh the page the color returns to the original which is green.
In tests carried out in development environments, low resolutions are being presented with smart phone views.

What is expected?
Fix the original color state bug, if the mouse cursor is positioned or the link is touched change color and return to its original state.
What is actually happening?
The color of the link in desktop resolutions works correctly when you hover over the link or click. In low resolutions and smart phones when touching the link changes the active state color and does not return to the original color of the normal state.

If you modify the history manually, Vue router cannot be aware of the changes. When redirecting, use next, make sure to not enter an infinite redirection though:
router.beforeEach((to, from, next) => {
  if (to.path !== "/goodbye" || to.query.name === 'bla') return next();
  const fromForward = document.location.pathname !== "/hello";
  next({
    path: '/goodbye?name=bla',
    replace: fromForward
  });
});

I used the query to avoid an infinite redirection but it's just to show the proper behaviour

Hi @posva,
The problem is not an infinite loop of redirections. The problem is that the color of the button remains gray after clicking, the original color of the button is green, when clicking in say hello the color of the button changes to gray and the only way to return the original color is making click on another part of the page. The expected behavior is that the button returns to the green color once it is clicked.
I copy a link where the problem is explained in a video.
https://www.loom.com/share/fb2d7b8bdfe44c88b619b5e9e6017453
Thank you.

I know it's not the problem you are seeing, but your problem comes from
using history.pushState. The infinite redirection thing is a problem you
will face if you redirect using next instead of calling
history.pushState which is not supported because it makes it impossible
for vue-router to know the new url. So to be clear, don't use
history.pushState (or replaceState) in the guard, use next to redirect
to a different location
On Tue, Jun 25, 2019 at 5:33 PM juanescarraga<EMAIL_ADDRESS>wrote:

Hi @posva https://github.com/posva,
The problem is not an infinite loop of redirections. The problem is that
the color of the button remains gray after clicking, the original color of
the button is green, when clicking in say hello the color of the button
changes to gray and the only way to return the original color is making
click on another part of the page. The expected behavior is that the button
returns to the green color once it is clicked.
I copy a link where the problem is explained in a video.
https://www.loom.com/share/fb2d7b8bdfe44c88b619b5e9e6017453
Thank you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vuejs/vue-router/issues/2819?email_source=notifications&email_token=AAFCE4MOMWKFO6NWRK6T25DP4I3E3A5CNFSM4H2ACFU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYQUZZA#issuecomment-505498852,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFCE4NRGYNYJPVPXPNNSETP4I3E3ANCNFSM4H2ACFUQ
.

--
Eduardo San Martin Morote

Hi @posva ,
Update the project code with the suggested suggestions, in the project view the buttons are purple by default, when hovering in the Say Hello button, the color changes to gray and when you click on the Say Hello button, the message appears. Hello, Foo! and the color of the button changes to gray and stays in that color, the expected operation is that the Say Hello button once the click is done returns to its original color in this case the purple one.
https://codesandbox.io/embed/vue-template-lgn63
Thanks,


fix(mission_planner): fix function name renaming

Signed-off-by: Takagi, Isamu<EMAIL_ADDRESS>Description
Fix https://github.com/autowarefoundation/autoware.universe/pull/1737
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[ ] I've confirmed the contribution guidelines.
[ ] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

before
[component_container-43] /opt/ros/galactic/lib/rclcpp_components/component_container: symbol lookup error: /home/kosuke55/pilot-auto/install/mission_planner/lib/libmission_planner_component.so: undefined symbol: _Z9set_colorPN8std_msgs3msg10ColorRGBA_ISaIvEEEdddd
[ERROR] [component_container-43]: process has died [pid 3330998, exit code 127, cmd '/opt/ros/galactic/lib/rclcpp_components/component_container --ros-args -r __node:=mission_planning_container -r __ns:=/planning/mission_planning --params-file /tmp/launch_params_2ef6pz65'].

after


test: handling multi-request tests better

This may help https://github.com/envoyproxy/envoy/issues/26258

Cool -- did you try (say) running the test ref'd in the bug 10k times with & without this change?


About evaluation metric

(3) Image-to-image alignment (I2I): evaluates the image-image CLIP similarity in exemplar-driven inpainting.

Hello, how do I compute the I2I metric?

hi, you can use CLIP image model to extract two image embeddings from two images (output and exemplar), then compute their cosine similarity.


Set #default_options in subscriptions requests.

Coverage increased (+0.004%) to 99.501% when pulling d5bda5502269bea37649f71b35f4b17bab914e76 on frodsan:refactor/use-default-options into b10e9190257b283010f9819d4b9977adccf4f8ec on hyperoslo:master.

I'm running out of :cake: here!

Coverage increased (+0.005%) to 99.502% when pulling 9b675c7e540bb183bad02c9bff7143ab9b6bc731 on frodsan:refactor/use-default-options into b10e9190257b283010f9819d4b9977adccf4f8ec on hyperoslo:master.


authenticate() not working on macOS

Steps to reproduce
Tell us about your environment:

Puppeteer version: 2.0.0
Platform / OS version: macOS Catalina 10.15.2
Node.js version: 12.14.0

    const browser = await puppeteer.launch({
        headless: false,
        defaultViewport: null,
        args: ['--proxy-server=' + taskProxy]
    });
    const page = await browser.newPage();
    await page.authenticate({
        username: taskUser,
        password: taskPass
    });

    await page.goto(url);

What is the expected result?
Page will authenticate the proxy and the given URL will be loaded.
What happens instead?
Page won't authenticate and asks user to manually enter user/pass.

You can use https://github.com/gajus/puppeteer-proxy as an alternative to --proxy-server/ page.authenticate. With puppeteer-proxy, you just pass your proxyUrl in a format http://username:password@proxy.

Wish I knew about this earlier, thanks!


Is it a bad practise to wrap an express app by a user defined object?

Hi guys.
I am implementing a middleware which will check whether the given request contains all the required params/body.
app.use(checkRequestData);

app.post("/user", controller.addUser);
app.get("/user/:id", controller.getUser);

Now in checkRequestData, I can get req.body but req.params.id is returning undefined. To get the req.params.id, I have written the following function

exports.checkRequestData = (req, res, next) => {
    /**
     * check whether request carries required data
     */
    next();
};

exports.applyMiddleware = (app) => (...middleware) => {
    return new class {

        middeware(req, res, next){
            middlewares.forEach((mw) => {
                mw(req, res, next);
            });
        }

        get(route, controller){
            return app.get(route, this.middeware, controller);
        }

        post(route, controller){
            return app.post(route, this.middeware, controller);
        }
    }
};

And the server file is modified as follows:
const _app = applyMiddleware(app)(checkRequestData);

_app.post("/user", controller.addUser);
_app.get("/user/:id", controller.getUser);

Using this method I am able to get the request data.
Is this the right approach of using express middlewares? Will this cause any issues to the in-built express methods?

Hi @gmk292.  This is probably not the way I would do it, but that is not to say it is wrong.  One other approach you might take is to create an array of reusable middleware and use it on each route.  The reason params is missing is because no route has been matched for that middleware.  You could get around this like so:
app.post("/user", checkRequestData, controller.addUser);
app.get("/user/:id", checkRequestData, controller.getUser);

If you have multiple of these middleware you can group them in an array like this:
const middleware = [checkRequestData, handleRequestDataErrors];

app.post("/user", middleware, controller.addUser);
app.get("/user/:id", middleware, controller.getUser);

This composition approach gets rid of the custom wrapper, and sticks with just usage of the express api.  Usually this makes for easier debugging in the future for you and anyone else who works on your project.  Good luck.

Thanks for your answer. I will follow your advise.


Make output JSON serializable

92568700ad1bd560d3f92213b52a6b11a1647d13 breaks assumed roles in AWS CLI because it adds HTTP headers in a botocore.vendored.requests.structures.CaseInsensitiveDict.  AWS CLI caches details to disk by serializing to JSON.
Resolve this by coercing into a normal dict.

[Current coverage][cc-pull] is 97.31%

Merging [#961][cc-pull] into [develop][cc-base-branch] will not change coverage

@@            develop       #961   diff @@
==========================================
  Files            43         43
  Lines          7014       7014
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           6826       6826
  Misses          188        188
  Partials          0          0

Powered by Codecov. Last updated by [fac5529...4660ab1][cc-compare]
[cc-base-branch]: https://codecov.io/gh/boto/botocore/branch/develop?src=pr
[cc-compare]: https://codecov.io/gh/boto/botocore/compare/fac55296d06b840155bc90b3499a2cda1d5bad09...4660ab10364724bd0503cee8f186a3f0d7115051
[cc-pull]: https://codecov.io/gh/boto/botocore/pull/961?src=pr

Could you please add a test for this?

I'll go ahead and take this and add a test.  I'd like to get this fixed asap so I'll add a test.

Pulled in this PR and added a test.  Let's track the remaining work in https://github.com/boto/botocore/pull/962


Added a python problem for longest string

Created Longest Substring with At Most K Distinct Characters in python as part of Hactoberfest 2024 Contribution.

@HarshwardhanPatil07 : Please consider this PR as Hacktoberfest 2024 Contribution.


ft2-clone: 1.74 -> 1.75

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/8bitbubsy/ft2-clone/releases.
meta.description for ft2-clone is: A highly accurate clone of the classic Fasttracker II software for MS-DOS
meta.homepage for ft2-clone is: https://16-bits.org/ft2.php
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.75 with grep in /nix/store/cc3i72z1ff6bpz181h7ckx0499q357ac-ft2-clone-1.75
found 1.75 in filename of file in /nix/store/cc3i72z1ff6bpz181h7ckx0499q357ac-ft2-clone-1.75

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
ft2-clone

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/cc3i72z1ff6bpz181h7ckx0499q357ac-ft2-clone-1.75 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A ft2-clone https://github.com/r-ryantm/nixpkgs/archive/4c387d8345e43c6a006f5a74faea542c1a80cbab.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/4c387d8345e43c6a006f5a74faea542c1a80cbab#ft2-clone

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/cc3i72z1ff6bpz181h7ckx0499q357ac-ft2-clone-1.75
ls -la /nix/store/cc3i72z1ff6bpz181h7ckx0499q357ac-ft2-clone-1.75/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

ft2-clone

Maintainer pings
cc @fgaz for testing.

@ofborg eval


Simplify Collective getTransactions method's query

Small improvement to make query simpler.
references https://github.com/opencollective/opencollective/issues/1519

I don't think this works.
If includeUsedVirtualCardsEmittedByOthers is false

It will wrongly include the transactions made using gift cards emitted by others

Admitting we're interested in getting user transactions, the new query will return entries that match:
{CollectiveId: UserCollectiveId, UsingVirtualCardFromCollectiveId: RandomOrganizationCollectiveId}

We don't want them as we've set includeUsedVirtualCardsEmittedByOthers to false.
This was the reason for the:
[
  { CollectiveId: this.id, UsingVirtualCardFromCollectiveId: null },
  { UsingVirtualCardFromCollectiveId: this.id },
];

Aka get the transactions I've made without using a virtual card (UsingVirtualCardFromCollectiveId=null), or the ones made using one of my virtual cards.

It will not include the transactions made using own virtual cards

When making a transaction using a virtualcard, CollectiveId is referencing the user that made the transaction while the organization that emitted the virtual card is referenced by UsingVirtualCardFromCollectiveId. We must use this field to return organization the transactions that was made using one of their virtualcard.

I get that this can be confusing and I've started writing some documentation to make it clearer. I'll need to reference this in function's comments. Do you know what would be the best place for me to start uploading these (markdown) documentation? There is the docs/ folder, but i'd love to use the github wiki so we don't have to commit to update it.

Hum I think now it's clearer for me @Betree . Now I understand the importance of UsingVirtualCardFromCollectiveId: null. thanks for clarifying. Closing here..


Kibana and logging across namespaces

Hey,
Is there any way to use the current Kibana & Elasticsearch setup for logging across multiple namespaces?
The way I have it working right now is with an extra fluent daemon set in the default namespace, but maybe there is a better solution.
Thanks :)

hi @vasil-moneybird - what are you looking to log ? docker logs ?

Pod logs


Prototype CRD API generation (not working)

Description
To configure auth providers in StackRox you need to send a REST call to Central (requiring API permissions etc).
A declarative approach would circumvent the auth and state management issues.
In that PR I tried to add a CRD to configure an AuthProvider , in the CRDs reconcile function it must then take the data from the CRD and send it to the AuthProvider data-layer/service layer in StackRox.
The problem in that prototype was that stackrox is defined as stackrox/rox in StackRox's go.mod but is located in github.com/stackrox/stackrox which leads to conflicts for imports and relative pathfinding for kubebuilder and code-generator.
To integrate the CRD it is recommended to put it into a separate repository and depended on it as a library.
Central then registers the reconciler, creates a new controller manager and starts it in a separate go-routine.
Why is declarative configuration an advantage?

fleetshard must not only deploy Central, but also after a successful deployment make changes at runtime
fleetshard needs API access and correct permissions to create auth provider in Central
Leverage Kubernetes API's RBAC system

Notes

Custom repository for CRD APIs
Helm chart must install CRD from crds/ directory

Helm chart must be up-to-date with CRDs
Helm chart build process could download CRD definition from separate repo or loading it from a vendored directory

Controller Manager is started as a separate go-routine

Reconcile function should be owned by Central to avoid cyclic imports between Central <> CRD repo
Rudimentary entry point which is called by Central here
AuthProvider database access must be implemented here (or passing a function to a reconciler)

Central needs RBAC to read and write to CRDs

Why is declarative configuration an advantage?

What are the disadvantages? I can think of "backward reconciliation": imagine someone makes changes to the auth provider configured via the CRD, e.g. adds more rules. Will that change be reflected in the CRD? If not, if someone adds another, potentially conflicting, change to the CRD, how the reconciler shall reconcile?

imagine someone makes changes to the auth provider configured via the CRD, e.g. adds more rules. Will that change be reflected in the CRD?

This is depended on the implementer. We can use the CRD as a storage backend (in fact it is), thus removing postgres/rocksdb for AuthProvider config altogether.
If a hybrid approach is chosen either disabling the configuration from the UI or Central modifies the CRD directly on UI updates. This must be implemented in the Central's data layer.
The reconcile function is only responsible for setting up the desired state from the CRD to match the server/application state.

If not, if someone adds another, potentially conflicting, change to the CRD, how the reconciler shall reconcile?

Overwriting UI changes, treating etcd like any other database. If a change is made directly to rocksdb/postgres these are overwritten directly too.


Melody search tool sometimes does not allow notes to be entered

Seemingly dependent on window sizes, sometimes you can't enter certain notes.
One issue I've found is that the bounding box of the staff area is only calculated at page load, so if you do anything to resize the page mid-use, things get wonky.
Another issue is that we should not allow wrapping of the staff.

same issue as #1128


wasm RFC

Summary
Motivation
Reference-level explanation

Introduce cargo workspaces
Provide methods to index on server

Library
tantivy-cli/indexer

Make the wasm library easy to compile and integrate

Add functions for
Enable integrations

Drawbacks

Incompatibility/lack of features
Programming style
CI infra/build times

Alternatives
Unresolved questions
Future possibilities

Summary
Commit to wasm as one of the targets for tantivy.
Motivation
Makes tantivy available to server-side and web developers natively. Enables developers to use the same index format between server and gives client native bindings to read and query the index. eg. Client-side index queries on small-enough index files.
I expect this to give us a competitive advantage over Lucene and help library adoption rates.
Reference-level explanation
Introduce cargo workspaces
Using Cloudflare's wirefilter as an example, we would move current src/ directory to server/ and create a new directory wasm/.
Provide methods to index on server
The server indexer has 2 entry-points: tantivy-cli and library.
Library
Add a method to IndexWriter that serializes the index to a file.
impl IndexWriter {
...
/// Serialize the index as of the last commit to a file under a given Path
/// Return a result with the Opstamp of serialized index.
pub fn serialize_for_wasm(&self, filename: Path) -> Result<Opstamp> {
}

Helps user build a tantivy index that is later serialized into a binary format that tantivy wasm understands.
tantivy-cli/indexer
Add a flag/question at the end to give users an option to serialize the index to wasm format.
Make the wasm library easy to compile and integrate
Add functions for

[ ] reading the index file
[ ] parsing queries
[ ] performing searches
[ ] collecting and presenting results as JsValues

Enable integrations
Use wasm-pack to build tantivy-wasm. Ship the repo with a JS/HTML component that makes it easy to integrate tantivy wasm to web applications (backend and frontend).
Drawbacks
tantivy was originally conceived as a library for developing server-side indexers.
Making a serious commitment to wasm will affect feature development, programming style and devops infrastructure.
Incompatibility/lack of features
Although wasm support is being increasingly adopted by browser engines (Chromium and Firefox), the API surface is still limited and continues to change fast. For example, threading support is currently a work in progress in the wasm runtime.
Having a clear focus on 1 type of platform (servers) allows us to optimize our solution using SIMD, Rust intrinsics for different platforms and system-specific structure for lock-less programming provided by crossbeam.
Programming style
Changes in any relevant traits and structures cannot break the wasm build. Will require implementing some functionality twice with conditional compilation flags. This might introduce compromises in code, when Linux-specific features are sacrificed for the sake of wasm-compatibility.
Lose system-specific performance gains.
 The probability of wasm being abandoned is much greater than Linux.
CI infra/build times
Including the wasm target and dependencies (wasm-bindgen, wasm-pack) in every CI build will increase CI time. Since these dependencies are yet to stabilize, we run the risk of being guinea pigs for the bugs in wasm-bindgen, which might break our builds and introduce potentially indefinite delays.
Alternatives
Start a separate project under the tantivy organization. Only guarantee about wasm-compatibility every git tag/release checkpoint. This will keep the same rate of development of features.
Unresolved questions
How to test wasm? Headless browser or in pure Rust?
Future possibilities
Extend tantivy wasm to support the IndexWriter trait. Enable building a wasm application that indexes uploaded files in the tantivy format in the browser. Users will be able to build an index in browser memory and download it to run on their server.
This will further extend tantivy wasm demo and allow users to build an index and run queries against it on client-side.

Keen to hear your thoughts @jonfk

A couple of remarks.
If I recall correctly, the smallest wasm tantivy build I managed to get so far was 900KB and that involved a lot of cheating & trimming. Some of this work will affect external crates, like the stemming crate.
I haven't followed what was the state on the standardization of the "WASI", but if we get an Mmap there, it might a very nice use case outside of browser development.
This could be a very nice way to distribute tantivy-cli on any platform, in a very safe way.
Possibly it could also be a way to make tantivy usable from different language without too much sweat.

According to this doc Mmap is not likely to be part of the standardized WASI any time soon.
That's a bummer.
On tantivy side, that will mean some heavy changes on the way we do io if we want to get compatible eventually.

thanks for your notes. Below are my thoughts on the 3 specific topics.
size
I have grepped for stemmer and found occurrences in the tokenizer and query parser modules. I suggest we stick to the MVP, which allows reading the index without building one. This should only affect tokenising user queries, if I understand this correctly?
mmap
I am going off the wasm spec, rather than the WASI.
According to the wasm design doc mmap support is in the future ideas section. I am guessing the rust-wasm team will wait for it to stabilise and hopefully provide a wrapper in the mmap crate.
Either way you make an important point. For the foreseeable future, mmap calls won't be supported by wasm. This means we will need to change our internals and keep them compatible for the same index format across different platforms. If and when mmap support is added, we will need to add conditional compilation flags inside the core library to allow mmap on wasm and server-side.
distribution as a wasi binary for other languages to use
I didn't think of it. My original suggestion was to build tantivy wasm for browsers. However, you are right to draw attention to the potential of wasmer and other wasm runtimes that can help us. They will provide the wrappers and integrations of wasm binaries with CPython, Java. This will save us time, while enabling more developers to embed tantivy across different applications.
Overall
I prefer to under promise and over deliver than vice versa. if wasm is hard to manage and will make it hard to work on server-side features, I would not accept this. In my opinion, promising wasm support now and ignoring or removing it later seems more damaging to tantivy than not promising it at all.
I am still curious to get more thoughts on pain points/costs/disadvantages there might be of adding wasm as a first-class target. It feels like a big commitment for development, toolchain inclusion and CI/ops provisioning.

This would also make it easy to use tantivy in WebExtensions for Firefox/Chrome or MailExtensions in Thunderbird.

Is this RFC about having a .wasm binary for this crate and use it in JavaScript or other bindings, or having a WASM target-compatible library that serves as other library's dependency?
If the latter is the choice, there is an very simple guide on the WebAssembly Book that might help.

Update: I found that this crate can be built to WASM target with wasm-bindgen feature enabled and default features disabled.
So I think it could be used like this:
[dependencies]
tantivy = { version = "0.11.3", default-features = false, features = ["wasm-bidgen"] }

But I cannot make sure since the current test suite relies on mmap.

did not look deeper, but could this help:
https://github.com/WebAssembly/wasi-libc/pull/197

@urbien This part is probably more relevant.
https://github.com/WebAssembly/wasi-libc/blob/3e9892fc41dd83fa66f192f963f54c72fb8321c1/libc-bottom-half/mman/mman.c
So the WASI-libc does add support for mmap, but it will read and load the entire file in anonymous memory.
That's kind of lame but that should be ok for ppl who want to try it out.

I came here looking for a static web site search engine. Think Wikipedia size data with all the files pre-processed and search indexes laid down on disk for static serving. A WASM module in the browser would read the inverted index files as needed to execute the search at hand.
I'm not familiar with the layout of your index files, but I wouldn't want each individual file too big or too small, and a single search shouldn't need too many different files to complete, since each "seek" could be another network fetch.
A website archived to a content addressable storage and wanted to include search would need everything pre-computed at build and upload time. Keeping the content un-changeable and without any backend infrastructure enable reliable sneaker-net movement of these archives and usage across air-gapped networks.

@ngbrown Do you have a specific use case?

One very specific example, but it's not the first use of a static search that I've thought of, would be the need for an air-gapped Wikipedia because some countries block it now. The IPFS has a project that provides this (https://blog.ipfs.io/24-uncensorable-wikipedia/) along with several methods to transport the snapshot across the block, one option is pinning on nodes that have connectivity to both partitions of the network, and the other is to copy an entire snapshot package to a USB drive (https://dweb-primer.ipfs.io/avenues-for-access/sneakernets).
If you check out their snapshot, you'll see it has no search. dealing with Wikipedia without search is less than ideal. As far as I know, there's no working solution for a static search on sites this big because the indexes would be very large to download just for that specific user's needs. Javascript and WASM is allowable as part of this snapshot so that's why I thought a full search engine like tantivy could be leveraged.

@ngbrown searching static website is common, but they are usually small enough that javascript libraries do ok.
I have compiled tantivy to WASM before, but the resulting file is around 1MB at least. (more with stemming), so it is not really worth doing it for a simple use case.
Wikipedia on a USB key is interesting. For different reasons, current version of tantivy is not great for this use case however...
but we (at Quickwit) are currently working on a version that should up for the task.

I've used the in-memory JavaScript search solutions for the smaller use cases (think blog or help documentation). So they do work, but I don't know of one that partitions the indexes and only loads segments on demand for the really big cases.
I really don't think a 1MB WASM file would be a big deal compared to the size of the indexes for a big site like Wikipedia.  Microsoft is downloading multiples of that to get .NET running in WASM.
Do you have any more information of this alternative?
The good news is that this use case doesn't need backwards compatible files. They would just get re-built each time, for each version.

I think this blocks https://github.com/matrix-org/seshat/issues/84 which is useful in element matrix web client. Is it possible to remove non wasm compatible parts (I see mmap in this issue) via some cfg?

I've managed to make tantivy work fully in the browser with databases of arbitrary size (tested with a 14GB database and it works great).
The code and demo is here: https://github.com/tantivy-search/tantivy/pull/1067
It has a demo of Wikipedia full text search that could be put on IPFS as mentioned by @gbrown

@phiresky That's awesome. I'll check out your proof of concept, thanks for sharing it. With Tantivy's use of mmap for storing the index, what does memory usage look like to the end user on your proof of concept? (i.e. in Activity Monitor).
(warning: I'm not well-versed in mmap'd files so there maybe be incorrect assumptions in the following question...)
Is there a way to tell the OS to limit resident memory usage (rss) with an mmap'd file? I want to use Tantivy via WASM for a large client-side index, but I'm concerned the end user will perceive a bunch of memory usage (i.e. in Activity Monitor) when, in practice, the OS should manage the mmap'd file's resident memory usage as it pleases.

With Tantivy's use of mmap for storing the index, what does memory usage look like to the end user on your proof of concept?

In my POC I replace memory mapping with a manual "page cache". Basically a replacement for what the OS does when memory mapping. With memory mapping the OS chooses how much of it to keep it memory based on how much RAM you have, and automatically evicts it when other stuff needs the space. We can't really do this automatic choosing in the browser since we can't know what other programs on the computer need.
So you can basically choose whatever memory usage you want. You actually have to in my POC, since otherwise it will cache everything forever. So it would probably need an LRU system with a fixed limit on memory usage. Depending on your needs you might want to cache e.g. the whole .fieldnorm files and fetch everything else on demand.
In my tests, normal queries fetch around 1-10MB of data, which is then the same as the memory usage (except for internal data structures, but those shouldn't be very large). So it really shouldn't be much more than a normal website.

Thanks for providing Tantivy and keeping this ticket open.
Even though I know it's not supported, I tried and seemed to get close, but can't get Tantivy working in WASM even though it compiles. I get a panic when actually calling Tantivy code in a WASM context. I know this isn't supported yet, so I'm not surprised it doesn't work, but if anybody has any quick pointers on hacks/workarounds the code is below.
My strategy was:

Tell Tantivy I only care about an in-memory index, i.e. Index::create_in_ram(schema.clone());
Set default-features = false to get Tantivy compiling in WASM (this might break things?)
Make sure WASM/Rust communication is working using more basic functions (it is)
Make sure my search index works from cargo run (non-WASM target) (it does)

I think my WASM/Rust/JS is right. I think my Tantivy code is right. I told Tantivy I only care about in-memory index. But still can't get the combination of technologies working.
The update from 0.14 here make it seems like the underlying data storage Tantivy uses is getting closer to some form of in-memory + WASM capability, but I'm not sure if it can work in any WASM environment yet (even if limited?).
Is the underlying issue with WASM the data storage aspect? For me, I want the Lucene-like capabilities, stemming, etc. but am okay with an in-memory index.

The panic, as close as I can get!

Cargo.toml
[package]
# search client!
name = "sc"
version = "0.1.0"
edition = "2021"
default-run = "sc"

[lib]
crate-type = ["cdylib"]

[dependencies]
anyhow = "1.0"
fake = { version = "2.6.1", features=['derive'] }
num-format = "0.4.4"
rand = "0.8.5" # required by fake
tantivy = { version = "0.19.2", default-features = false }
uuid = "1.3.3"
wasm-bindgen = "0.2.86"
# needed for WASM
getrandom = { version = "0.2.2", features = ["js"] }
web-sys = { version = "0.3.63", features = ['console'] }

[target.'cfg(target_arch = "wasm32")'.dependencies]
console_error_panic_hook = "0.1.6"

src/lib.rs
use wasm_bindgen::prelude::*;

mod logging;
mod search_index;
mod test_data;

#[wasm_bindgen]
extern "C" {
    fn get_js_name() -> JsValue;
}

#[wasm_bindgen]
pub fn get_rust_name() -> String {
    "BenRs".to_string()
}

#[wasm_bindgen(start)]
pub fn start() {
    let js_name = get_js_name();
    let js_name_string = js_name.as_string().unwrap();
    println!("Hello {}, you wild alien from JavaScript!", js_name_string);

    println!("test search");
    let goofy_string = search_index::test_two();
    search_index::test();
    println!("test search: done, string is {}", goofy_string);
}

src/test_data/mod.rs
use fake::Fake;

// using `faker` module with locales
use fake::faker::name::raw::*;
use fake::locales::*;

pub fn fake_names(count: usize) -> Vec<String> {
    (0..count).map(|_| Name(EN).fake()).collect()
}

src/search_index/mod.rs
use std::time::Instant;
use num_format::{Locale, ToFormattedString};
use tantivy::collector::TopDocs;
use tantivy::query::QueryParser;
use tantivy::schema::*;
use tantivy::{doc, Index, ReloadPolicy};
use uuid::Uuid;
use crate::test_data::fake_names;

const NUM_TEST_RECORDS: usize = 100_000;
// const NUM_TEST_RECORDS: usize = 1_000_000;
// const NUM_TEST_RECORDS: usize = 1;
const TEST_QUERY: &'static str = "standefer";
const DEBUG: bool = false;

#[derive(Debug)]
struct TestDoc {
    uuid: Uuid,
    name: String,
}

pub fn test_two() -> String {
    println!("Geting a string from within search_index module");
    "hi from search_index!".to_string()
}

pub fn test() -> tantivy::Result<()> {
    let mut now: Instant;

    // Define schema
    let mut schema_builder = Schema::builder();
    schema_builder.add_u64_field("uuid_hi", STORED);
    schema_builder.add_u64_field("uuid_lo", STORED);
    schema_builder.add_text_field("name", TEXT | STORED);
    let schema = schema_builder.build();

    // Create the index, this will create meta.json in the directory
    let index = Index::create_in_ram(schema.clone());

    // Get fake data
    now = Instant::now();
    let mut names = fake_names(NUM_TEST_RECORDS);
    names.push("Ben Standefer".to_string());
    let docs: Vec<TestDoc> = names.into_iter().map(|name| {
        TestDoc {
            uuid: Uuid::new_v4(),
            name: name.to_string(),
        }
    }).collect();
    if DEBUG {
        for doc in &docs {
            println!("{:?}", doc);
        }
    }
    println!("Generating data took: {:?} ({} records)", now.elapsed(), NUM_TEST_RECORDS.to_formatted_string(&Locale::en));

    // Write
    now = Instant::now();
    let mut index_writer = index.writer(50_000_000)?;
    let uuid_hi_field = schema.get_field("uuid_hi").unwrap();
    let uuid_lo_field = schema.get_field("uuid_lo").unwrap();
    let name_field = schema.get_field("name").unwrap();
    for doc in &docs {
        let (uuid_hi, uuid_lo) = doc.uuid.as_u64_pair();
        index_writer.add_document(doc!(
            uuid_hi_field => uuid_hi,
            uuid_lo_field => uuid_lo,
            name_field => doc.name.to_string(),
        ))?;

    }
    index_writer.commit()?;
    println!("Indexing data took: {:?}", now.elapsed());

    // Search
    now = Instant::now();
    let reader = index
        .reader_builder()
        .reload_policy(ReloadPolicy::OnCommit)
        .try_into()?;
    let searcher = reader.searcher();
    let query_parser = QueryParser::for_index(&index, vec![name_field]);
    let query = query_parser.parse_query(TEST_QUERY)?;
    let top_docs = searcher.search(&query, &TopDocs::with_limit(10))?;
    for (_score, doc_address) in top_docs {
        let retrieved_doc = searcher.doc(doc_address)?;
        println!("{}", schema.to_json(&retrieved_doc));
        println!("{}", Uuid::from_u64_pair(
            retrieved_doc.get_first(uuid_hi_field).unwrap().as_u64().unwrap(),
            retrieved_doc.get_first(uuid_lo_field).unwrap().as_u64().unwrap(),
        ));
    }
    println!("Search took: {:?}", now.elapsed());

    Ok(())
}

src/logging.rs
#[cfg(target_arch = "wasm32")]
#[macro_export]
macro_rules! println {
    ($($arg:tt)*) => (web_sys::console::log_1(&format!($($arg)*).into()))
}

#[cfg(not(target_arch = "wasm32"))]
#[macro_export]
macro_rules! println {
    ($($arg:tt)*) => (std::println!($($arg)*))
}

src/target_test.rs
pub fn test() {
    #[cfg(target_arch = "wasm32")]
    {
        println!("target test: wasm32");
    }

    #[cfg(not(target_arch = "wasm32"))]
    {
        println!("target test: NOT wasm32");
    }
}

src/main.rs
use anyhow::Result;

mod logging;
mod target_test;
mod search_index;
mod test_data;

fn main() -> Result<()> {
    target_test::test();

    println!("Let's do this!");
    search_index::test()?;

    Ok(())
}

index.html
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Hello from Rust!</title>
    <script type="module">
      import init, { greet } from './sc.js';

      async function run() {
        await init('./sc_bg.wasm');
        console.log(greet("World"));
      }

      run();
    </script>
</head>
<body></body>
</html>

index.js
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Hello from Rust!</title>
    <script type="module">
      import init, { greet } from './sc.js';

      async function run() {
        await init('./sc_bg.wasm');
        console.log(greet("World"));
      }

      run();
    </script>
</head>
<body></body>
</html>

package.json
{
  "scripts": {
    "build": "webpack",
    "serve": "webpack serve"
  },
  "devDependencies": {
    "@babel/core": "^7.22.1",
    "@babel/preset-env": "^7.22.4",
    "@wasm-tool/wasm-pack-plugin": "1.5.0",
    "babel-loader": "^9.1.2",
    "html-webpack-plugin": "^5.3.2",
    "source-map-loader": "^4.0.1",
    "text-encoding": "^0.7.0",
    "webpack": "^5.49.0",
    "webpack-cli": "^4.7.2",
    "webpack-dev-server": "^3.11.2"
  }
}

.babelrc
{
  "presets": ["@babel/preset-env"]
}

webpack.config.js
const path = require('path');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const webpack = require('webpack');
const WasmPackPlugin = require("@wasm-tool/wasm-pack-plugin");

module.exports = {
  entry: './index.js',
  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: 'index.js',
  },
  plugins: [
    new HtmlWebpackPlugin(),
    new WasmPackPlugin({
      crateDirectory: path.resolve(__dirname, ".")
    }),
    // Have this example work in Edge which doesn't ship `TextEncoder` or
    // `TextDecoder` at this time.
    new webpack.ProvidePlugin({
      TextDecoder: ['text-encoding', 'TextDecoder'],
      TextEncoder: ['text-encoding', 'TextEncoder']
    })
  ],
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['@babel/preset-env'],
          },
        },
      },
      {
        test: /\.js$/,
        enforce: "pre",
        use: ["source-map-loader"],
      },
    ],
  },
  mode: 'development',
  experiments: {
    asyncWebAssembly: true
  }
};

The panic, as close as I can get!
hi, looks like the error comes from Instant which does not support wasm, see https://internals.rust-lang.org/t/is-std-instant-on-webassembly-possible/18913

You may want to check out wasix https://wasmer.io/posts/announcing-wasix, they also got a version of tantivy working
https://github.com/wasix-org/tantivy/tree/wasix

Whether or not Instant works seems to depend on your WASM environment. I have managed to get tantivy (and Instant) to work in a Fastly Compute@Edge environment, but with a lot of caveats and sharp edges

@aguynamedben And also you can check Tantivy fork and search server based on this fork https://github.com/izihawa/summa
It is patched for WASM:

usize pointers
more async functions chains in all code for parallel networking in browser environment
changed Boxes to Arcs and added Send in several places
Also, I did small patches for my case: parallelized compression of .store, supported custom attributes for segments and entire index, random collector for subsampling.

Both fork and search server has been in production for a long time, I'm keeping it in line with Tantivy master branch.

Here is WASM package https://github.com/izihawa/summa/tree/master/summa-wasm, it works for me in all browsers including Safari on iOS, but I have never prepared it for being public so it suffers from the lack of any documentation, except for several indirectly related articles in blog: 1 and 2
Anyway, you can look how it is done. It utilizes ThreadPool based on WebWorkers for paralellizing search load. Together with async code patches and accurate using (e.g. using hotcache and not using fieldnorms), it works very fast on multi-segment indices and multi-term queries even if index is living in the network.


Update Kibana package so we can use it for fleet-server bootstrap tests

What does this PR do?

Why is it important?

Checklist

[ ] My code follows the style guidelines of this project
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

Related issues

:green_heart: Build Succeeded

Buildkite Build
Commit: 10e337e09202941790fb2a38954987f12aa0c808

Replacing this with an additional function in the estools testing package of the elastic-agent repo


Locate range distance issue

why /locate rande distance so small? ~134 blocks

where is the problem? Is this wrong? because the distance just represents the air - line distance to the point...the y axis might be wrong, shouldn't make a big difference iirc

It's not about Y, it's about the fact that locate can't find anything beyond ~200 blocks, whereas on JAVA locate finds biomes/structures up to a few thousand blocks away

yeah this doesnt make any sense cuz its using the same 100 chunk radius that is also used in the java client. im gonna look into this tomorrow

it happend only with biomes.


Cannot run the mnist_linear problem

I've obtained the MINST data set, followed the build instructions and managed to get a working system.
symptoms

Running the tests passes.
The load example works
fortytwo works.

Running mnist_linear results in the following:
[jlouis@lady-of-pain tensorflow-ocaml]$ make mnist_linear
ocamlbuild examples/mnist/mnist_linear.native
Finished, 0 targets (0 cached) in 00:00:00.
Finished, 109 targets (109 cached) in 00:00:00.
LD_LIBRARY_PATH=./lib: ./mnist_linear.native
Uncaught exception:

  (Failure "5 FeedInputs: unable to find feed output \192\021\242")

Raised at file "pervasives.ml", line 32, characters 22-33

Expected output:
I would have expected this to run the system to completion and print out some data from the training around 92% accuracy.
Discussion:

I have relatively little idea of what is going wrong, everything is guesswork.
System is a fairly updated ArchLinux installation, libtensorflow.so is either yours or one I built myself, no difference.
OCaml is 4.03.0+flambda
Python is 3.5.1

Are there any other data you could be interested in knowing? I could add some debug-printfs to the code and figure out what call it is that fails more specifically, but if you happen to know exactly what the problem is, I'd rather just get your input first.
I'm at the HEAD of master:
[jlouis@lady-of-pain tensorflow-ocaml]$ git show | head -n 6
commit 4fd7c65a664bcef8816372ba464ff93aa1e60a1b
Author: laurent<EMAIL_ADDRESS>Date:   Sun May 1 17:02:50 2016 +0100

    Move the lstm dependent part away.

More stats:

mnist_svn fails
mnist_nn fails (PruneForTargets)
mnist_conv works

Thanks for the detailed feedback, that's very interesting.
There was a similar problem caused by some memory corruption issue in the ctypes wrapper a couple weeks back but I thought this was now fixed.
I'm able to reproduce your issue after switching to 4.03.0-flambda, I'll look into this but meanwhile maybe you could use 4.03.0 ? Things seem to work properly with this compiler version, at least for me.

This was caused by flambda optimizing an ignore statement. Thanks to the help of @hhugo I just pushed a bugfix that should work for flambda. Let me know if you still have any issue.

Thank you for the quick analysis and fix. I think I'll run this on 4.03.0 for now. I don't need the extra speed provided by flambda short-term anyway.


Support deployment to an existing Azure vNet in future versions of the Elastic ARM template

I attempted deploying to an existing Azure Resource Group which is already running another Cluster of .Net compute nodes (Service Fabric) but the deployment failed. If possible it would be great to support deploying an ES cluster to an existing Azure vNet.

++ this is high on our backlog

@Mpdreamz - Let me know if you need any guinea pigs :+1:

any updates on this?

#37 is a possible solution.

@ajhewett I'm working on updating this in line with new changes to the template

This will be in the next template release, once it's gone through the approval process.


Fix Global Opt for 1x1 Filter Convs

Using this issue to note down internal discussion about conv performance.
Problem
1x1 filter convolutions get converted to linalg.generic ops during the global optimization pipeline. For example:
%8 = linalg.conv_2d_nchw_fchw {dilations = dense<1> : vector<2xi64>, strides = dense<1> : vector<2xi64>} ins(%0, %2 : tensor<1x8x128x128xf16>, tensor<8x8x1x1xf16>) outs(%7 : tensor<1x8x128x128xf32>) -> tensor<1x8x128x128xf32>

Convert1X1FilterConv2DToMatmulPass (+ dropping unit dims) generalizes it to:
 %8 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d3, d1, d2)>, affine_map<(d0, d1, d2, d3) -> (d0, d3)>, affine_map<(d0, d1, d2, d3) -> (d0, d1, d2)>], iterator_types = ["parallel", "parallel", "parallel", "reduction"]} ins(%collapsed, %collapsed_0 : tensor<8x128x128xf16>, tensor<8x8xf16>) outs(%7 : tensor<8x128x128xf32>) {
    ^bb0(%in: f16, %in_2: f16, %out: f32):
      %16 = arith.extf %in : f16 to f32
      %17 = arith.extf %in_2 : f16 to f32
      %18 = arith.mulf %16, %17 : f32
      %19 = arith.addf %out, %18 : f32
      linalg.yield %19 : f32
    } -> tensor<8x128x128xf32>

This is problematic because the reduction dimension d3 is the outermost dimension of %collapsed (first input to the generic). Ideally, we would transpose %collapsed with the permutation [1, 2, 0] to make the reduction the innermost and expect the transpose to fuse with its producer. This doesn't get handled during transpose propagation either.
For some more context, there %collapsed is produced by transpose([1, 2, 0]) -> pad -> collapse_shape(remove unit dim). If the generic conv could fuse with that transpose, it would make the access contiguous along the reduction dimension.
Possible Solutions:
@MaheshRavishankar suggested either:

Teach transpose propagation to propagate across pads
OR
Reorder Global Optimization to be:  transpose propagation -> pad to intrinsic -> generalization

Related Issue: https://github.com/iree-org/iree/issues/19230
cc @qedawkins

I think we might want to reorder Convert1X1FilterConv2DToMatmulPass and ConvertConvToChannelsLastPass so that ConvertConvToChannelsLastPass runs first. They are adjacent so it should have limited side-effects and I think it does what we want: transposes inputs and converts to preferable conv op variants.
Switching them with no other changes results in:
%602 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>, affine_map<(d0, d1, d2, d3) -> (d3, d2)>, affine_map<(d0, d1, d2, d3) -> (d0, d1, d2)>], iterator_types = ["parallel", "parallel", "parallel", "reduction"]} ins(%599, %__hoisted_tensor_8x16xf16 : tensor<128x128x8xf16>, tensor<8x16xf16>) outs(%601 : tensor<128x128x16xf32>) {
^bb0(%in: f16, %in_324: f16, %out: f32):
  %630 = arith.extf %in : f16 to f32
  %631 = arith.extf %in_324 : f16 to f32
  %632 = arith.mulf %630, %631 : f32
  %633 = arith.addf %out, %632 : f32
  linalg.yield %633 : f32
} -> tensor<128x128x16xf32>

If this works, let's do this. Thanks @IanWood1 for finding a simpler solution


Embedded Rabbit MQ is not downloading and starting after deleting .embeddedRabbitMQ Directory

Hi,
I am using Embedded RabbitMQ for running Integration Tests.
Earlier today, everything worked pretty well but after deleting .embeddedRabbitMQ directory and clearing the extracted files in Temp directory, I am not able to run anything smoothly.
I am getting below exception, even newly it is not getting downloaded.
Code
  var configBuilder = new EmbeddedRabbitMqConfig.Builder();
        configBuilder.downloadFrom(OfficialArtifactRepository.GITHUB);
        configBuilder.port(5673);
        configBuilder.useCachedDownload(false);
        var config = configBuilder.build();
        var command = new RabbitMqCommand(config, "rabbitmq-server", "start");
        command.call();

Need your help.
Exception Caused By
java.lang.IllegalArgumentException: The given command could not be found using the path: C:\Users\<username>\AppData\Local\Temp\rabbitmq_server-3.8.0\sbin\rabbitmq-server.bat

@AlejandroRivera  Need your suggestion in resolving this issue.

Hi @VIGNESHBRB...
It's going to be hard for me to help out here since i don't have a Windows machine i can test things, but do you see the logs where the file is being downloaded and extracted?  From the error message, it seems like the file isn't found and i imagine the issue is the extraction process failing for some reason.
The logs that show the extraction It should look like this: https://travis-ci.org/github/AlejandroRivera/embedded-rabbitmq/builds/463047276#L2522
If you don't see those logs, make sure to enabled DEBUG-level logs on your config (SLF4J or whatever logging library you're using).
I'd also recommend you set a breakpoint in your test right before the test finishes and the tear-down takes place and then navigating on your filesystem to see the actual extracted files and ensuring things are there.
Good luck!

@AlejandroRivera Hi. Download and extraction is working fine after performing each operation separately.
Like:
For Downloading:
  var downloaderFactory = new DownloaderFactory(config);
        var downloader = downloaderFactory.getNewInstance();
        downloader.run();

For Extracting:
   var extractorFactory = new ExtractorFactory(config);
        var extractor = extractorFactory.getNewInstance();
        extractor.run();

For Running:
 var command = new RabbitMqCommand(config, "rabbitmqctl", "start_app");
        rabbitMqProcess = command.call().getFuture();

When I tried to use as given in README:
EmbeddedRabbitMq rabbitMq = new EmbeddedRabbitMq(config);
        rabbitMq.start();

I am getting:
io.arivera.oss.embedded.rabbitmq.helpers.StartupException: Could not confirm RabbitMQ Server initialization completed successfully within 3000ms

Hi again,
Without having access to the logs from the process that's trying to start the RabbitMQ server it will be hard to know what's the issue. However, it's clear that the issue here is being able to run RabbitMQ on your host and it's not related to this library.
I'd suggest:
a) Determining if there's not already some process that was left hanging that's using the port number you're trying to use -- this is a common issue when RabbitMQ isn't properly terminated (like if your JVM instance crashes or you don't make it stop programmatically.
b) look for RabbitMQ log file. See http://www.rabbitmq.com/logging.html
c) Look for an Erlang's crash dump (erl_crash.dump ). See https://www.rabbitmq.com/troubleshooting.html#crash-dumps
d) Manually download and run the RabbitMQ server (ie. without using this library) on your system. Once you're able to run it like that, you should also be able to run it using this library.
Good luck again!


Proposal: operator tags

When building a circuit is often useful to have some extra information about certain parts of the circuit.  For example you may want to preserve a certain part of the circuit while performing a transformation (optimization pass).  There isn't a great way to do this right now, but we were able to tag our operations, then we could use the tag to do this.
Tagging could also be used for diagrams.
Tagging might also be important when differentiating different "types" of gates for the same unitary.  For example sometimes exp(iZt) can be compiled out of a circuit, but sometimes you want it to stick around.

Thoughts?

Amusingly got a request for this today from someone else.

It seems like it would be low effort and generally useful for a variety of tasks.
It seems like there are a fair amount of different tasks where we have a generally accepted behavior or optimization pass, except we want to ignore one specific use case, so this would be a good way to accomplish all of those kinds of tasks.

Another question is whether Gates should also have tags.  Any gate with a tag transfers it to its operator when applied using on

Another question is about equality.  I guess gates and ops with different tags need to not be different.  I was going to lazily add the tags, but if we want that to work then there is a bit more work.

How much structure do you want tags to have? A string field? a list of strings? something more like an enum?

list of strings was my thought

This has been done in #2407

This has been done in #2407

Correction: #2407 was dropped in favor of #2670, which has been submitted.


Highlight group 'Normal' has no background highlight warning

Hello, first thanks for this great plugin!
After updating, I keep seeing Highlight group 'Normal' has no background highlight notification the first time a notification is displayed.
I have no idea why this shows up, because my Normal highlight has background color:

I modified my config to include the background_colour, but the warning still seems like a bug to me. By quick glance it seems only the bg property is checked, but not the guibg.
I'm using material.nvim colorscheme (if that somehow relates):
  use({ -- Colorscheme
    "marko-cerovac/material.nvim",
    branch = "main",
    config = function()
      vim.g.material_style = "deep ocean"
      require("material").setup({
        contrast = {
          sidebars = true, -- Enable contrast for sidebar-like windows ( for example Nvim-Tree )
          floating_windows = true, -- Enable contrast for floating windows
          line_numbers = true, -- Enable contrast background for line numbers
          sign_column = true, -- Enable contrast background the sign column
          cursor_line = false, -- Enable darker background for the cursor line
          non_current_windows = false, -- Enable darker background for non-current windows
          popup_menu = true, -- Enable lighter background for the popup menu
        },

        contrast_filetypes = {
          "terminal", -- Darker terminal background
          "term", -- Darker terminal background
          "packer", -- Darker packer background
          "qf", -- Darker qf list background
        },

        disable = {
          term_colors = true,
          borders = false,
          colored_cursor = true,
        },

        custom_highlights = {
          DiffAdd = { bg = "#002500" },
          DiffDelete = { bg = "#250000" },
          DiffChange = { bg = "None" },
          DiffText = { bg = "None", fg = "#999900" },
          NvimTreeNormal = { fg = "#A6ACCD" },
          NeomakeVirtualtextErrorDefault = { fg = "#AF111A" },
          FloatTitle = { fg = "#a6accd", bg = "#090b10" },
        },
      })
      vim.cmd([[colorscheme material]])
    end,
  })

The synIDattr() function doesn't distinguish between gui/cterm values, it just returns what's currently in use (determined by termguicolors option. The logic for the warning is simply
local bg = vim.fn.synIDattr(vim.fn.synIDtrans(vim.fn.hlID(colour_or_group)), "bg#")
if bg == "" or bg == "none" then
  -- Warn
end

so you can see locally why that's being triggered. I've tried with the material config you provided and don't see the warning so there might be something else setting the highlight?

Running :echo synIDattr(synIDtrans(hlID('Normal')), 'bg#') results in #0f111a.
Same result with lua :lua print(vim.inspect(vim.fn.synIDattr(vim.fn.synIDtrans(vim.fn.hlID("Normal")), "bg#"))) results in "#0f111a".

OK since I can't reproduce you'll either need to provide a minimal init.lua that reproduces or you can try debugging yourself by adding a print for the value as the warning is emitted in config module https://github.com/rcarriga/nvim-notify/blob/master/lua/notify/config/init.lua#L71

Seems like in that part of code, the background is "".
I tried running :verbose hi Normal and it's same before and after the first notification.
However, I just noticed this problem only happens when the notification is triggered from nvim-tree, if I do :lua vim.notify('Test') from a regular buffer, the background is set and no warning shows up. If I run the same command inside nvim-tree (or just do any action that creates a notification eg. rename a file) the background reports as empty and warning is shown.

I too am seeing the same issue.
For me it seems to be when I have my TokyoNight theme settings to show a transparent background...
require("tokyonight").setup({
  style = "night",
  transparent = true,
  styles = {
    sidebars = "transparent",
    floats = "transparent",
  },
})

When I comment out the transparent settings I don't see the waring popup...
require("tokyonight").setup({
  style = "night",
  -- transparent = true,
  -- styles = {
  --   sidebars = "transparent",
  --   floats = "transparent",
  -- },
})

I too have the tokyonight theme and see the same problem with transparency.

@bushblade @zakissimo This is the expected behaviour, please follow the notification and set the background_colour setting. This issue is for when a transparent background is not being used.
@tomaskallup I believe I've found the issue, it's due to nvim-tree using a custom winhl setting and how the underlying implementation of it changed in latest neovim. Please try the latest commit

For nvim-notify to provide an opacity effect, it must know the background colour to fade to/from. If you don't have a background colour for the "Normal" highlight group, it will not know what colour to use so you must tell it. This should be whatever your terminal background colour is (even if it is somewhat transparent itself).
The notification tells you how to set it
  notify.setup({
    background_colour = "#121212", --Change to terminal background colour
  })

@rcarriga Thanks, sorry yes I did notice that just after.
This seems to work for me in my Noice setup...
    use({
      "folke/noice.nvim",
      event = "VimEnter",
      config = function()
        require("noice").setup()
        require("notify").setup({
          background_colour = "#1a1b26",
        })
      end,
      requires = {
        -- if you lazy-load any plugin below, make sure to add proper `module="..."` entries
        "MunifTanjim/nui.nvim",
        "rcarriga/nvim-notify",
      },
    })

@rcarriga Thanks, with the latest commit it seems to work fine.


[Questions] misc questions on Figure 8 (order processing flow chart)

Should there be a decision/exit point for orderErrors (as described in chapter <IP_ADDRESS>) in Figure 8?

Phrasing of condition "1st node in deviation range" misleading? I would prefer e.g. "Vehicle within deviation range of first node"
Equivalently for "Reject; Report noRouteError; (1st node as reference)" -> Would a better alternative for the bold text be "nodeId of 1st node as reference"?

Is the definition of "active order" correct? The condition states
AGV has current order:
actionStates not only FINISHED/EMPTY
&& nodeStates not empty
&& edgeStates not empty

Isn't logical OR (||) correct instead of logical AND (&&)?

What exactly does lastNode mean in the context of this condition?

Is it the lastNodeId, or all elements that belong to the regarding node's JSON object?

I have similiar questions about active order. If all the actions finished but agv not reach the final node, I thought it is still an active order.

I think lastNode means lastNodeId and the && (logical AND) makes sense. If you are at this point in the flow chart, you have the following situation:

the vehicle reached the decision point and has no active order (no horizon).
an order update is received. This update is only valid if it starts at the decision point --> nodeId of the first node of the update must be equal to the current lastNodeId AND the sequenceId of the first node must be identical to the current lastNodeSequenceId. If either nodeId or sequenceId doesn't fulfill these requirement the update isn't a correct continuation if the old order.

This check is some different to the other check 'if start of new base is end of old base'. If there is no active order the only information about the old order in the state message is the lastNodeId and lastNodeSeqeunceId (node state is empty). If the decision point isn't reached the full information of the decision node is available ( the decision point is inside the node state) and a check if the start node of an update is identical to the decision point can include more fields of a node. (Yes, 'if start of new base is end of old base' is a little bit sketchy).
An order is base + horizon. The base cannot be changed by an order update (one reason for the checks above; decision point is part of the base). A horizon can be changed by an update.
The nodes and edges of the horizon are included in the nodeStates resp. edgeStates, therefore a non empty horizon means the AGV has a current order and cannot accept a new order. The MC has to send first a 'cancel order'.

Thank you, @collani-bosch for the detailed explanation of the fourth bullet point of the question. I think there we can agree that lastNode rather means lastNodeId, and that the AND-combination of both, lastNodeId and lastNodeSequenceId is correct.
Do you have an opinion on bullet point 3, which in the diagram looks like this:

Do you agree that the three conditions should be OR'ed instead of AND'ed?


[PROPOSAL] Optimize embedding processors for update scenario

What/Why
What are you proposing?
Text embedding processors can be more intelligent and skip unnecessary calls to model in update scenarios.
What users have asked for this feature?
I do have one customer case where they are doing following:

configure remote model
configure text embedding processor and attach it to ingest pipeline that is set as default at index level
ingest document, that creates embeddings
update some document fields not related to embedding field or original field that is base for embedding generation

our text embedding processor does not analyze document state and what exactly change, it just calls the model again. That doesn't add any value as it receives same embeddings and set them for new document. That model call can be avoided by simply copying embeddings from original document.
What problems are you trying to solve?
We can save on number of calls to remote model. For end customer that means:

lower bill as they typically pay by number of calls
increase stability as system can reach rate limit and other calls can be throttled by the model

What is the developer experience going to be?
We need a way to make this behavior configurable. In some cases you need to regenerate embeddings even if they previously were in the document, for example you've deployed updated version of the model.
I suggest following logic:

if embeddings in current document are empty - always make a call to the model
if embeddings are not empty - check the flag. if flag says 'update' then do the call. Otherwise copy embeddings from original document.
default behavior should be - 'always update`. This will ensure backward compatibility with today's code
processor behavior of update/not update should be configurable at the processor level to allow flexibility, so different processors may be configured differently in one cluster.

Are there any security considerations?
No security concerns regarding making calls to model, this functionality exists today. New flag that regulatesd processor behavior should be added in a safely manner.
Are there any breaking changes to the API
No, as per suggested logic today's behavior will remain the same
What is the user experience going to be?
If user wants to fine tune the processor behavior they will need to set a new processor parameter.
Are there breaking changes to the User Experience?
No
Why should it be built? Any reason not to?
Main reason is saving cost on calls to model and lowering chances of requests being throttled. If customer having such problems today they may not even realize the system is making unnecessary calls to model.
What will it take to execute?
This will be a code change in the plugin code, for every processor that we want to onboard for this feature. change is something like:

add parameter to processor factory so new behavior can be enabled
modify processor as per suggested logic - check if in original document the field for embedding exists, if it not - call the model, today's logic. if it does - check the param, if it says not to update - return, otherwise call the model.

Any remaining open questions?
Some edge cases may be:

what if field with embeddings is marked as "excluded" in index mapping. I suggest we do the call to model
Some open questions related to implementation:
if document that is passed to the processor in case of "update_by_query" has the original document. Need to check what's the input in https://github.com/opensearch-project/neural-search/blob/main/src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java#L120
do "update" and "update_by_query" behave the same? they probably do but that needs verification

[Catch All Triage - Attendees 1, 2, 3, 4, 5]

Agree to add a processor-level flag to update. Are there any reasons we shouldn't implement this feature?

Will look into this.


missing tag filter

Describe the solution you'd like
Is there a way to filter memos that do not have any tags, so that one can easily find the ones where one forgot to add a tag?
Type of feature
User Interface (UI)
Additional context
No response

Not sure if it is applicable but one way would be to have a filter called 'uncategorized' that is the default for every memo and it can be changed to other tags before creation. 🤷🏽♂

yi'qian以前的版本是有“捷径”功能来筛选带有附件的内容，以及筛选没有标签的内容。目前为止，这项功能被删除了。

Not sure if it is applicable but one way would be to have a filter called 'uncategorized' that is the default for every memo and it can be changed to other tags before creation. 🤷🏽♂

Might be a solution. But probably easier to add a filter by default which will simply query all memos which do not have any tag.

Issue is not in English. It has been translated automatically.

 Previous versions of yi'qian had a "shortcut" function to filter content with attachments and content without tags. As of now, this feature has been removed.

Was there a particular reason to remove that functionality?

Issue is not in English. It has been translated automatically.

 Previous versions of yi'qian had a "shortcut" function to filter content with attachments and content without tags. As of now, this feature has been removed.

Was there a particular reason to remove that functionality?

https://github.com/usememos/memos/issues/2593
This feature is being re-added and may take some time.

 以前的版本是有“捷径”功能来筛选带有附件的内容，以及筛选没有标签的内容。目前为止，这项功能被删除了。

可以分享一下右上角的热力图和搜索的样式吗😍

 以前的版本是有“捷径”功能来筛选带有附件的内容，以及筛选没有标签的内容。目前为止，这项功能被删除了。

可以分享一下右上角的热力图和搜索的样式吗😍

这是以前Memos自带的热力图，不知道为什么在最近的版本里被删掉了。
可以拉取以前版本的Memos镜像，就能看到这项功能了。


aredis_om and pydantic-v2 @model_validator RecursionError

When trying to add a pydantic v2 model validator to my aredis_om JsonModel, a RecursionError occurs. When testing both aredis_om and the validator separately, both work without error. When testing together a maximum depth search error occurs.
Here is example code that recreates this issue...
from aredis_om import JsonModel
from pydantic import model_validator

class Test(JsonModel):
    name: str

    @model_validator(mode='after')
    async def create_name(self):
        self.name = 'this is different'
        return self

    @classmethod
    async def create(cls, name):
        test = cls(name=name)
        await test.save()

I think this will probably be solved by #603


Added the package info folder path parameter to yml template

The original way of reading *.json under PackageInfo is not working at the case which artifact is under ci.yml but old mgmt which is not used anymore.
E.g. azure-keyvault in python sdk. Link
The PR is to add a new parameter in order not to break the contract between yml template and scripts.
The test pipeline:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1125426&view=logs&s=cc1c0c38-0f79-5da2-c09f-847e3abdf9ce

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

We found this situation happening in other repos as well (.NET specifically). After chatting with @weshaggard we should consider raising a warning when files are specified which do not have corresponding PackageInfo files.
This will prevent build errors and keep the processing logic identical.
We should also open up work items to enumerate all of the packages which don't produce PackageInfo so we can figure out if anything special needs to be done for those packages or their information in ci.yml

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow


fixed rating issue on server side

Description
Fixed 2 things:

Status now updates dynamically in the map tracker

Rating a user now persists

Low priority but user SHOULD not be able to rate the user again but w/e.

I used my local db/server to test seed data

Published to https://exp.host/@heartbleed/whimsical?release-channel=pr65


[add] env AWS_REGION

add env AWS_REGION
add secret hash in register, close #5
updae aws-sdk-go
fix README.md

Hello
This PR is related to #5 And fix AWS_REGION bug which COGNITO_USER_POOL_ID should be the same AWS_REGION.
@br4in3x please let me know if it's ok with you or if I should change anything.
Best wishes


Install Jinja2 templates for click_completion

The new version of click_completion requires its Jinja2 templates to be installed along with the package in order to work. If not, any invocation of pipenv --completion results in a jinja2.exceptions.TemplateNotFound for the template in question.
I’ve just trivially added the pattern to setup.py, and tested that it works.

@techalchemy We’ll need another release very soon 🤦♂

Today is the plan

Is this sufficient to make sure they end up in the sdist too?

@ilovezfs It should, I think.

We didn’t actually release an sdist yet but I would believe that our manifest doesn’t include these. Feel free to PR a fix and thanks for the heads up!


Server Side Rendering

How can scala-dom-types api be used to render html on the server side (non js) ? eg,
val indexhtml = div("hello world")
val indexhtmlstr = indexhtml.toString
// then serve indexhtmlstr, eg, via http server.

though one other way would be to use the Laminar api .L._ , to build html and then render it via scala.js on node/graal on server side.

If you wanted to use Laminar for this, then yes, in principle you can use node with jsdom / graal / headless browser / etc. to render the HTML on the backend. Here's one approach using headless browser.
All of the above is useful if you want to share your HTML-generating code between frontend and backend, but it's overkill if you only want to generate some HTML on the backend. If that's the only thing you need, if you don't also need access to indexHtml variable from your scala.js code, then there's a simple way:
Instantiate the html tags and properties that you need, implementing def apply and def :=, an implicit conversion from String to Modifier, a toHtmlString method, etc. You don't need any JS environment for this, just keep track of element children and prop-value pairs yourself, and recursively compile all that into an HTML string when requested. To get started, you might want to look at some code from https://github.com/raquo/scala-dom-builder, although it does not contain a full solution to this, and it's burdened by a cross-platform setup which you don't need.
Also you might want to add some document-level tags like html, body and head if you need them.
Implementing any kind of rendering is out of scope for Scala DOM Types, but anyone so inclined could potentially make a library that would do this using the types defined here. This would be a nice addition.

Also if you implement toHtmlString yourself, be careful to encode the data correctly – quotes, angle brackets, etc. Otherwise the user data you render might end up breaking your website layout or giving you XSS.

Implementing any kind of rendering is out of scope for Scala DOM Types, but anyone so inclined could potentially make a library that would do this using the types defined here. This would be a nice addition.

Would there be some code to this end in Laminar already?

No, because Laminar does not compile HTML strings – it creates and updates DOM elements using JS DOM APIs which differentiate between structure and data, avoiding the problem.
See https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html for a quick cheatsheet, especially the Output Encoding section. It's not very instructive though, there are probably more comprehensive pages out there.
I should have mentioned it from the start but I forgot – ScalaTags already knows how to output HTML in a similar manner to what you want, so you might want to either use it instead, or look at its implementation – not just of how they do data encoding, but also the compilation of HTML in general. The data encoding method def escape is defined in Escaping.scala, see how it's used in their code.


Rwa 3731 performance degradation

https://tools.hmcts.net/jira/browse/RWA-3731

In production the SKU is overridden here: https://github.com/hmcts/wa-task-management-api/blob/master/infrastructure/prod.tfvars#L3


Add a V2 version of typed connect that supports mapXToProps fns

@keybase/react-hackers this adds a new typed connector that supports all the mapXToProps.
a mergeProps function is required because we can't merge types together.

Closing in favor of using redux's type defs

@MarcoPolo on master right now the typed connect still does the state=> (state) which we should avoid due to performance reasons right? so we still need a pr to address that?


On hybrid graphics, blacklisting NVIDIA modules causes delay in windows opening

I have a laptop with inegrated graphics (Intel UHD 630) and discrete graphics (renderer: NVIDIA GeForce GTX with Max-Q Design/PCIe/SSE2).
NVIDIA's power management for this card is... not particularly good, so if I want to save power, I sometimes blacklist the following modules on boot via kernel parameters: nvidia,nvidia_drm,nvidia_modeset. This usually means it doesn't turn on at all, giving me 5-8W idle power usage instead of 15-18W.
However, when running Hyprland with these modules blacklisted, window creation is delayed by more than 2 seconds.
Without blacklisting these modules,
Steps to reproduce

Boot with the following kernel parameters added: module_blacklist=nvidia,nvidia_drm,nvidia_modeset
Run Hyprland
Attempt to open windows

Different applications seem to act differently. I tested a couple, with the results below.
Window creation testing
Large (~2.3-2.5s) delay? Appears to vary

Kitty (any window)
Firefox (first opened window)
Waybar (first time)

Small (noticeable though) delay:

Thunar
Firefox's second window
running kitty with KITTY_DISABLE_WAYLAND=1 kitty

running kitty from a kitty window within Hyprland has the same delay, and gives:
[glfw error 65544]: process_desktop_settings: failed with error: [org.freedesktop.DBus.Error.UnknownMethod] No such interface “org.freedesktop.portal.Settings” on object at path /org/freedesktop/portal/desktop

Hitting ctrl+enter (my keybind to launch kitty) continuously does nothing until the delay is up, and then they all appear at the same time.
Related packages, config
hyprland-git: r2060.250d61e-1, apparently release 6.0.9-arch1-1,
xdg-desktop-portal: 1.15.0-1
xdg-desktop-portal-gtk: 1.14.0-1
NVIDIA modules are not in my mkinitcpio precisely so I can do this hacky powersaving, though my Hyprland wrapper does have the lines specified at https://wiki.hyprland.org/Nvidia/
Logs
I redirected Hyprland's output to a file (as well as current kernel parameters) so you can see what might be up. In each session, I run a script (runKittiesAndThunar) to launch 3 kitty instances and 1 thunar instance. Knda wish the logs had timestamps, happy to rerun with those enabled if there's a way. These are attached.
intel.log: followed the steps to reproduce
intelNVIDIABlacklist.log: did not blacklist any modules.
Whew. Sorry for that wall of text.

The problem with nvidia is that I do not offer any official support for such issues.  Have you tried hyprland-nvidia-git from the AUR?
I don't even have a machine to test, as I have not touched nvidia in ages due to their drivers.
Apart from that, you can join the discord server and ask around for similiar issues, maybe someone else had this.

Unfortunately, my worry is that the bug is happening with the NVIDIA drivers specifically not launched, so I don't think it's the drivers' work. I'm wondering if it might be the case with any hybrid machine? I can try the hyprland-nvidia-git package and report though.

Now that you tell, I noticed it on my laptop, I have the nvidia card blacklisted too.
Just thinking... it might not be because of the nvidia card blacklisting? It could be just the intel graphics itself... I have an 11th gen intel.

I am testing the open source Nvidia drivers. Now I can black list the nvidia card and have 0 lag using the integrated intel card alone.  Why does the drivers have an impact although you have black listed them?

Could you, using hybrid graphics mode, run inxi -G to check what's rendering for you?

renderer: Mesa Intel Xe Graphics     (TGL GT2)

I seem to have the same issue with nouveau - delays that feel like anywhere between 0.5 and 2 seconds, though it's worth noting inxi -G has my renderer as "N/A" , and glxgears will not run.
Blacklisted nvidia modules on boot, and had uncommented nouveau blacklistings in /usr/lib/modprobe.d.
inxi -G output:
Graphics:
  Device-1: Intel CoffeeLake-H GT2 [UHD Graphics 630] driver: i915 v: kernel
  Device-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] driver: nouveau
    v: kernel
  Device-3: Sunplus Innovation HD WebCam type: USB driver: uvcvideo
  Display: wayland server: X.org v: <IP_ADDRESS> with: Xwayland v: 22.1.5
    compositor: Hyprland driver: X: loaded: intel unloaded: modesetting
    dri: i965 gpu: i915,nouveau resolution: 1920x1080
  API: OpenGL v: N/A renderer: N/A

Kernel parameters: vmlinuz-linux module_blacklist=nvidia,nvidia_drm,nvidia_modeset root=/dev/nvme1n1p1 rw add_efi_memmap initrd=intel-ucode.img initrd=initramfs-linux.img nopti intel_iommu=on iommu=on sysrq_always_enabled=1 quiet splash loglevel=3 rd.udev.log_priority=3 i915.fastboot=1 i915.enable_guc=2 i915.modeset=1

yeah because
a) doesnt seem like a hyprland issue
b) I've not had a single nvidia product for ages.

 Maybe we can find some help in Archlinux forums or reddit.

Something worth nothing:

Plasma Wayland has this bug too.
Plasma X11 has not this bug.
Plasma does not use wlroots.
So the problem is something wrong with Wayland itself.

if wayland plasma has the exact bug, then I will close this, as it's not a hl issue. Most likely a driver one.

Occurs even when modules nvidia, nvidia_modeset, nvidia_uvm and nvidia_drm aren't added to the initramfs on Arch Linux. Adding them fixes the issue.

I seems to find a clue. through lsof we can see the libnvidia is opened for all apps after hyprland is launched, but I don't know how to prevent that

Solved
https://wiki.archlinux.org/title/wayland#Avoid_loading_NVIDIA_modules


MacOS support

I get error[E0425]: cannot find function `prctl` in crate `libc` which suggests Linux-only.

Correct, runwasi is only supported on Linux at the moment.
We could look at taking contributions to make macos work.
There is already folks looking at making it work on Windows which would help because we'll split the code out based on platform.

+1.
Tried building this today on Mac M1. Error seems to be different.
error: failed to run custom build command for `procfs v0.15.1`

Caused by:
  process didn't exit successfully: `/Users/home/project/rust/wasm/runwasi/./target/debug/build/procfs-1372915205a1c053/build-script-build` (exit status: 1)
  --- stderr
  Building procfs on an for a unsupported platform. Currently only linux and android are supported
  (Your current target_os is macos)
warning: build failed, waiting for other jobs to finish...
make: *** [build-wasm] Error 101

We currently take unix builds as meaning linux, although macos is also unix.
If we plan to support macos we would need to clean that up.
runwasi has gained a lot of new features since this issue was created, and we havent been paying attention to macos` since then.
The main challenge to have a macos build would be implementing an Instance for it. The first step would be creating a stub like we currently do for windows here, and disambiguate all the unix checks into linux and macos. The linux implementation is "easy" because we can piggiback on youki. There's work in progress for the windows implementation, but we don't currently have anyone working on a macos implementation.

What are you trying to achieve? - Is there a specific problem you are facing?
If you just want to compile runwasi with your MacOS, I recommend an Ubuntu Linux VM (that's what I use). But a Docker container is also possible.
If you want to compile a runwasi shim for using it on MacOS then the complete implementation is missing and there are open design questions like do we want to wrap the runtime in a container and how could that look like in MacOS since Apple does not officially support Linux like containers.
However, there is no active development on building a MacOS implementation of runwasi.

What are you trying to achieve? - Is there a specific problem you are facing?
If you just want to compile runwasi with your MacOS, I recommend an Ubuntu Linux VM (that's what I use). But a Docker container is also possible.
If you want to compile a runwasi shim for using it on MacOS then the complete implementation is missing and there are open design questions like do we want to wrap the runtime in a container and how could that look like in MacOS since Apple does not officially support Linux like containers.
However, there is no active development on building a MacOS implementation of runwasi.

I just want to look at the code and try to build it and learn its implementation logic. Thank you for your reply.

I'm happy that you are looking into this repository! If you have specific questions you can chat with the team on our CNCF Slack channel: https://cloud-native.slack.com/archives/C04LTPB6Z0V

@0xE282B0 I think people are trying to achieve being able to run wasi workloads without a second level of emulation, i.e. directly under their main operating environment.


Extend cgpm_analyze/grammar.y with lovecat analysis

Easily to extend the little language inside cgpm_analyze to achieve.
ANALYZE m FOR 10 ITERATION(
    lovecat;
)

Although we should use another name i.e. cpp-backend (for lovecat) vs python-backend (for gpmcc).
Still undertermined whether cgpm.crosscat.engine and cgpm.crosscat.state will perform the delegation, or cgpm_metamodel.py here in bayeslite.

https://github.com/probcomp/bayeslite/commit/f11c3fdf6967ae31694e224013b3f959252400da


[Bug]: onregionchange infinite loop / crash

Mapbox Implementation
Mapbox
Mapbox Version
default
Platform
Android
@rnmapbox/maps version
10.0.0-beta.68
Standalone component to reproduce
/**
 * Sample React Native App
 * https://github.com/facebook/react-native
 *
 * @format
 */

import React, {useEffect, useState} from 'react';
import {StyleSheet, View} from 'react-native';
import MapboxGL, {
  Camera,
  FillLayer,
  MapView,
  ShapeSource,
} from '@rnmapbox/maps';

MapboxGL.setAccessToken(<ACCESS_TOKEN);

function App(){
  const [region, setRegion] = useState();
  const [visibleBounds, setVisibleBounds] = useState();

  const onRegionDidChange = r => {
    console.log('region changed');
    setRegion(r);
  };

  useEffect(() => {
    console.log(region?.properties?.zoomLevel);
    if (region?.properties?.visibleBounds) {
      setVisibleBounds(region?.properties?.visibleBounds);
    }
  }, [region]);

  return (
    <View style={styles.container}>
      <MapView
        style={styles.map}
        onRegionDidChange={onRegionDidChange}
        styleURL={MapboxGL.StyleURL.Street}
        rotateEnabled={false}>
        <Camera
          maxZoomLevel={25}
          animationDuration={800}
          animationMode={'flyTo'}
        />
        {(region?.properties?.zoomLevel ?? 1) >= 3.5 && (
          <MapCubes visibleBounds={visibleBounds} />
        )}
      </MapView>
    </View>
  );
}

const COORDS = [
  [
    [-50.2734375, 55.578344672182],
    [-53.4375, 47.989921667414],
    [-42.5390625, 47.989921667414],
    [-41.484375, 55.578344672182],
    [-50.2734375, 55.578344672182],
  ],
  [
    [-50.273439, 55.57834467219],
    [-53.4375, 47.989921667414],
    [-42.5390625, 47.989921667414],
    [-41.484375, 55.578344672182],
    [-50.273439, 55.57834467219],
  ],
  [
    [-50.273439, 55.57834467219],
    [-53.4375, 47.989921667414],
    [-42.5390625, 47.989921667414],
    [-41.484375, 55.578344672182],
    [-50.273439, 55.57834467219],
  ],
];

function createNewFeatureCollection() {
  console.log('createNewFeatureCollection start');
  return new Promise(resolve => {
    setTimeout(() => {
      console.log('create new data');
      const idx = Math.random() * (3 - 1) + 1;
      console.log();
      const res = {
        type: 'FeatureCollection',
        features: [
          {
            type: 'Feature',
            properties: {
              description: null,
            },
            geometry: {
              type: 'Polygon',
              coordinates: COORDS[idx],
            },
          },
        ],
      };
      resolve(res);
    }, 1000);
  });
}

function MapCubes({visibleBounds}) {
  const [squaresFeatureCollection, setSquaresFeatureCollection] = useState();

  useEffect(() => {
    console.log('setSquaresFeatureCollection');
    (async () => {
      setSquaresFeatureCollection(await createNewFeatureCollection());
    })();
  }, [visibleBounds]);

  return squaresFeatureCollection?.features.length ? (
    //FIXME: squaresBought: need tests
    //squaresIcons: features[x].properties.icon attribute
    <ShapeSource
      maxZoomLevel={300}
      id="squareSource"
      shape={squaresFeatureCollection}>
      <FillLayer id="squares" key={'Squares'} style={squareStyle} />
    </ShapeSource>
  ) : null;
}

const squareStyle = {
  fillOpacity: 0.5,
  fillColor: '#E7EEF1',
  fillOutlineColor: '#005AEE',
};

const styles = StyleSheet.create({
  container: {
    height: '100%',
    width: '100%',
  },
  map: {
    flex: 1,
  },
});

export default App;

Observed behavior and steps to reproduce
In the onregion change listener, there is a check on the zoom. I'd like to show a square in the ocean (55.578344672182, -50.2734375). so, when the zoom is correct, I call an async service in order to get the feature collection to show the new layer over the map, but it seems there is a loop (I got this behaviour in my app) or a crash (what happened in the provided repro sample) there. I did some debug, and my opinion is that when the layer appears, in someway the onRegiondidCnahge is called again, as well as you're moving the map. The very strange situation, is that this loop (or crash) happen only when I call an  api too get the feature collection data, or when I use an async operation. you can see that in the visibleBounds useEffect. if you use the constants data to set the feature collection component state, the error is gone, and the app doesn't crash
Expected behavior
No response
Notes / preliminary analysis
No response
Additional links and references
I also tried to create an expo snack sample: but I got this error there: Failed to resolve dependency '<EMAIL_ADDRESS>(Network request failed)

I ran into the same issue with onRegionDidChange or onMapIdle. Did you figure out how to fix it? It only happens to Android.

hi! yea...I used a terrible hack to manage this issue.
I save the bounds in a ref variblae, and when the onRegiondidchange listener if fired, I check if the current bounds are different from the previous value, and only in this case do the necessary things.
something like this:
	const [visibleBounds, setVisibleBounds] = useState<Position[]>()
	const prevBounds: MutableRefObject<Position[]> = useRef<Position[]>([
		[-1, -1],
		[-1, -1]
	])

const onRegionDidchange = (feature: Feature<Point, RegionPayload>) => setRegion(feature.properties)

and then:
	useEffect(() => {
		if (!region) return
		const currentZoomLevel = region.zoomLevel
		const nextProjection = currentZoomLevel < ZOOM_LEVEL_GLOBE ? MAP_PROJECTION_MODE.GLOBE : MAP_PROJECTION_MODE.MERCATOR
		if (nextProjection !== projection) {
			setProjection(currentZoomLevel < ZOOM_LEVEL_GLOBE ? MAP_PROJECTION_MODE.GLOBE : MAP_PROJECTION_MODE.MERCATOR)
		}
		//FIXME: terrible hack to avoid map re-render
		if ((region?.zoomLevel ?? 1) >= MAPBOX_SQUARE_ZOOM && MapService.areBoundsDifferent(prevBounds.current, region?.visibleBounds)) {
			setVisibleBounds(region?.visibleBounds ?? [])
			prevBounds.current = region?.visibleBounds
		}
		// eslint-disable-next-line react-hooks/exhaustive-deps
	}, [region])

and this is how I use the the map component:
	<MapView
		style={styles.map}
		onRegionDidChange={onRegionDidchange}
		styleURL={MAP_STYLES.DEFAULT}
		projection={projection}
		rotateEnabled={false}>
		<Camera ref={mapCameraRef} maxZoomLevel={MAX_ZOOM_LEVEL} animationDuration={800} animationMode={'flyTo'} />
		<MapboxGL.UserLocation visible={hasLocationPermission} />
		{(region?.zoomLevel ?? 1) >= MAPBOX_SQUARE_ZOOM && <MapCubes visibleBounds={visibleBounds} onCubePress={onCubePress} />}
	</MapView>

hope this can help


Pause is not working

Hello and thanks for this library.
I use this library together with VueJS 3 and it works quite well.
However, the pause() function does not work correctly, it is a small mistake in the code.
In the code you wrote:
    pause() {
        this.audio && !this.audio.paused && this.audio.play();
    }

Correct would be:
    pause() {
        this.audio && !this.audio.paused && this.audio.pause();
    }

In addition to that, the pause() function can only be active after Fetch API finished.
Basically, you call the function in vain, you have to wait for the completion of the Fetch stream.

@softy2k , I'm sorry for only seeing this Issue now. I will fix this problem in the next version, thank you for your suggestion.

@softy2k https://github.com/kvsur/openai-speech-stream-player/releases/tag/1.0.8