Results returned are different even if I only use one thread

Hi,
I found that if I use the hnsw method in nmslib, I'll get slightly different results each run, even if I only use one single thread. But this behavior is not appeared if I use hnswlib.
Any comments?
Thanks,
Bo

What is eactly a run?

@searchivarius , thanks for your quick reply. Please see the codes below.
knn_index = nmslib.init(method = 'hnsw', space = 'l2', data_type = nmslib.DataType.DENSE_VECTOR)
knn_index.addDataPointBatch(adata.obsm['X_pca'])
knn_index.createIndex({'M': 15, 'indexThreadQty': 1, 'efConstruction': 100})
knn_index.setQueryTimeParams({'efSearch': 100})
results = knn_index.knnQueryBatch(adata.obsm['X_pca'], k = 100, num_threads = 1)
I have run the same codes twice and get different results.

@bli25broad possibly, this is some older version which didn't consistently initialized the random number generator.

@searchivarius , NMSLIB version is 1.7.2 . Isn't it up-to-date?

@bli25broad could you

double-check that it's 1.7.2, print nmslib.version
what happens when you run knnQueryBatch several times? (but indexing once) do you get different results?
thanks!

@searchivarius
re:1. I think it is 1.7.2. See below.

print(nmslib.version)
1.7.2

re:2. If I index once, I will not get different results.
Thanks,
Bo

@bli25broad thanks, so it's indexing being non-deterministic. Ok, the next question: is there a chance that you call init() twice in the same Python script?

@searchivarius , I guess not?
What I did is paste the codes twice in iPython. This should create two different index objects, won't it?
Best,
Bo

@bli25broad the library initializes the random number generator only once. It is currently not being re-initialized by the initIndex function.


Cluster for printing and distributing code?

Wisconsin, 1856:
SEc. 367. The secretary of state shall, forthwith, after the passage of. this act, cause six thousand copies thereof to be printed in a separate pamphlet, as session laws are
now printed, and he shall, as soon as said pamplet is
printed, distribute the same as the session laws are now
required to be distributed, by law, and the remaining
copies shall be retained in the office of the secretary of
state, to be sold at cost to any person desiring to purchase
the same, (not more than five copies to one person), and
it shall not be necestiry to print the [this] act in the
session laws.
Does something like this appear anywhere else?

It appears in a few different places, but wasn't caught in a cluster, probably because we require a minimum number of documents in a cluster. Here is a list of all sections which have "to be printed," though some of these are on a different topic (which laws can be used in evidence).
AL1852-001560.txt
CA1872-015520.txt
CO1877-003870.txt
CT1879-000470.txt
FL1892-006310.txt
FL1892-006900.txt
GB1875-004210.txt
GB1875-008460.txt
GB1875-008510.txt
GB1875-008530.txt
HI1897-002010.txt
IN1843-006690.txt
IN1852-002690.txt
IN1852-002700.txt
KS1868-003720.txt
MT1895-018490.txt
NC1868-006810.txt
NC1868-006920.txt
NY1850-003910.txt
NY1850-019420.txt
NY1876-000230.txt
NY1879-000250.txt
OK1891-001970.txt
OK1893-004270.txt
OR1862-008900.txt
TX1855-009610.txt
VA1893-000250.txt
WI1856-003670.txt
WV1868-021590.txt


British or American English?

The age-old question about British or American English :)
I'll approve this, but I think we should document in our style guide which one we are going for so that we can be consistent over time. Or do you believe this could vary by country and that Belgium rather use American spelling?

Originally posted by @Ainali in https://github.com/govdirectory/website/pull/323#pullrequestreview-1747673616
I think we should do a combination of these. First, if a country has a preference, then we should use that. And if not, I suggest we default to British English, but if someone has a good argument for American, I am easily convinced. And then we should document this in appropriate places.

Kind of related to #27

Kind of related to #27

A little bit, yes, but I think this is particularly needed for all the text that we don't get from Wikidata. Issue 27 mostly focuses on the values we retrieve.
(There is also a larger question about making the entire site translateable, but one small step at the time. :smiley: )

How do our current spellcheck work in relation to this @Ainali ?

I am not entirely sure, and on the train, but think it is set on en-us.


Linux scripts to rename, publish debug & release

Hello,
I developed these scripts for my own usage and figured I'd offer them to you guys for future people who decide to mod Valheim on Linux.

Thanks for the contribution, looks good but I haven't tried it out yet


Make palette observable to allow changes to palette

LGTM, thank you very much! Will merge and release as new minor version.


feat(vscode): add rem to px preview

close #2340
Preview

Add Configuration

Thanks for refactoring the configs!

We also need it here。thank you


Switch to BufRng instead of finite buffer

Currently we just panic if the input buffer runs out of data. https://github.com/fitzgen/bufrng is a simple but brilliant idea - return zeroes after the input data from the fuzzer is exhausted. It is much better than a ring buffer because it does not mess up coverage information, but at the same time is not prone to panics.

More thorough writeup about BufRng: https://fitzgeraldnick.com/2019/09/04/combining-coverage-guided-and-generation-based-fuzzing.html

I'm told using this Arbitrary implementation would be even better, since it is even more transparent for the fuzzer: https://github.com/rust-fuzz/arbitrary
However, I have not done any measurements on that myself.


Graph MA User Full Import Broken in 2/2020 release

User Full Import is broken in the Graph MA.  Syntax is wrong and appears to be a Delta Import so: Full User Import = Broken Delta Import.  Groups work OK, at least in a Full Import.
HttpStatusCode: BadRequest;
url: https://graph.microsoft.com:443/v1.0/users/delta/?$deltaToken=latest&$select=country,userType,faxNumber,postalCode,employeeId,streetAddress,mailNickname,proxyAddresses,officeLocation,displayName,businessPhones,deviceEnrollmentLimit,city,id,jobTitle,companyName,surname,mobilePhone,mail,department,givenName,accountEnabled,manager;
Response: {
"message": "Invalid Request: $select is not supported for these properties.",

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 94c4df6e-f886-e5b8-ae28-183def499df2
Version Independent ID: 6ee14064-16b8-27e9-95ef-60fdec8f0da2
Content: Connector Version Release History
Content Source: MIMDocs/reference/microsoft-identity-manager-2016-connector-version-history.md
Product: microsoft-identity-manager
GitHub Login: @EugeneSergeev
Microsoft Alias: esergeev

thanks for your feedback, however, next time please use<EMAIL_ADDRESS>for the feedback as this feedback is supposed to be about documentation, not to troubleshoot issues.
as for your problem: not a connector issue. Try this request in Graph Explorer - you'll see that /delta/ query does not support deviceEnrollmentLimit attribute. Just remove it from your list.

Ah, thanks – for some reason the attribute was checked by default after the MA install.
Jason Wagner
From: Eugene Sergeev<EMAIL_ADDRESS>Sent: Wednesday, February 19, 2020 3:38 PM
To: MicrosoftDocs/MIMDocs<EMAIL_ADDRESS>Cc: Jason542<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/MIMDocs] Graph MA User Full Import Broken in 2/2020 release (#198)
thanks for your feedback, however, next time please use MIM2016<EMAIL_ADDRESS>for the feedback as this feedback is supposed to be about documentation, not to troubleshoot issues.
as for your problem: not a connector issue. Try this request in Graph Explorer - you'll see that /delta/ query does not support deviceEnrollmentLimit attribute. Just remove it from your list.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FMIMDocs%2Fissues%2F198%3Femail_source%3Dnotifications%26email_token%3DAM6ZG337546Q4RHIBNIDYGLRDWKDVA5CNFSM4KYBD7Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMJNSJA%23issuecomment-588437796&data=02|01||09bb4f098e0049c98a9e08d7b57ba742|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637177414991874374&sdata=fIQQNUddLsyryozLaS0XUBkiHd7YF%2FmMH9UEORxSeyU%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM6ZG32SNAO36FE4U6JCRSDRDWKDVANCNFSM4KYBD7QQ&data=02|01||09bb4f098e0049c98a9e08d7b57ba742|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637177414991874374&sdata=0Abi9OrBW7O0gwPNjYD%2BewE%2Fa56ImWq4XVk6jy3KjMg%3D&reserved=0.

#please-close


new rule - always enforce from __future__ import annotations

there are existing rules that enforce it in some cases, but i would like one that enforces it all the time

You should be able to achieve this by marking it as a required import: https://beta.ruff.rs/docs/settings/#isort-required-imports
Does that work, or is there a limitation?

perfect, thanks!


Merge embassy branch into main

I think once this is merged I can bring in my changes.

@bentwire Sorry, I've been slowed down by not being able to test, but if it's working for you I can just go ahead and merge?

@fmckeogh I'm not in any rush if you want to test.
It's working for me but I'd feel better if it was working for both of us.


A bug in sdk

I believe this is a bug. (but I just ran across it when reading code, didn't actually isolate it and test it).
https://github.com/onflow/flow-js-sdk/blob/5678e821ebe37ad961af920c659a1cebf2f60d11/packages/interaction/src/interaction.js#L100
I apologize if I'm wrong.

cc: @JeffreyDoyle

Fix should be included in #568


Hacktoberfest

Check and validate with latest framworks

Hii @pfaffenrodt can you describe more what is this issue about?

@divyansh-dxn I had no time to check if everything is fine. and maybe upgrade some things to unify with the latest version from leanback/recyclerview framework

hacktoberfest is over


Add ViewRangefinder3d to reduce boilerplate when enqueuing standard 3D PhaseItems.

Objective
Reduce the boilerplate code needed to make draw order sorting work correctly when queuing items through new common functionality. Also fix several instances in the bevy code-base (mostly examples) where this boilerplate appears to be incorrect.
Solution

Moved the logic for handling back-to-front vs front-to-back draw ordering into the PhaseItems by inverting the sort key ordering of Opaque3d and AlphaMask3d. The means that all the standard 3d rendering phases measure distance in the same way. Clients of these structs no longer need to know to negate the distance.
Added a new utility struct, ViewRangefinder3d, which encapsulates the maths needed to calculate a "distance" from an ExtractedView and a mesh's transform matrix.
Converted all the occurrences of the distance calculations in Bevy and its examples to use ViewRangefinder3d. Several of these occurrences appear to be buggy because they don't invert the view matrix or don't negate the distance where appropriate. This leads me to the view that Bevy should expose a facility to correctly perform this calculation.

Migration Guide
Code which creates Opaque3d, AlphaMask3d, or Transparent3d phase items should use ViewRangefinder3d to calculate the distance value.
Code which manually calculated the distance for Opaque3d or AlphaMask3d phase items and correctly negated the z value will no longer depth sort correctly. However, incorrect depth sorting for these types will not impact the rendered output as sorting is only a performance optimisation when drawing with depth-testing enabled. Code which manually calculated the distance for Transparent3d phase items will continue to work as before.

I've merged in the latest main and updated the PhaseItem::sort() impls added by PR #5049 to be consistent with the new SortKey behaviour.

As suggested, I've i) added the inline annotation and ii) added a utility function to ExtractedView for obtaining a ViewRangefinder3d.
I've run the many_cubes benchmark on both this and the main branch, and there does not appear to be any difference in performance above the noise threshold.

bors r+


newsflash: 2.2.4 -> 2.3.0

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Thanks @Stunkymonkey! There was just one thing I noticed: Since #221716, we should also be updating the Cargo.lock on each update.

you are totally right. Thanks for the update on rust packages.

Result of nixpkgs-review pr 229970 run on x86_64-linux 1

1 package built:

newsflash


Sparse digital waveforms look wrong

This is caused by the new digital waveform renderer being based on the analog waveform. We don't want to interpolate between 0 and 1 when rendering!


[google_sign_in] Add serverClientId option to return serverAuthCode

This should fix issues flutter/flutter#16613 and flutter/flutter#17813
There are a number of formatting changes introduced by the flutter_plugin_tool...I'm happy to back those out if desired.

I signed it!

Also should fix flutter/flutter#14245

I signed it!

@kroike (Not sure if you are the right person to ping) Any chance of getting a review on this?

@kroikie (mis-spelled the first time) Any chance of getting a review on this?

@bparrishMines When will this pullRequest to be confirmed and merged ?

This is a necessary feature. Would love to see this merged! @bparrishMines

Hi @fbcouch,
Thank you for the contribution!
Unfortunately, we currently don't have a harness for testing the platform side of plugins, current and new features are not properly covered by tests. In order to maintain a quality bar for flutter/plugins we are going to avoid adding new features that aren't critical until they can be properly tested.
The test harness work is tracked in flutter/flutter#10824 and flutter/flutter#26080, your feedback on the testing work will be really appreciated (will it properly support the testing needs of this PR?) some design details should be available on flutter/flutter#10824 soon(contributions to that effort are welcomed!).
Would you mind re-opening this PR with e2e tests once a test harness is ready?
Meanwhile, you can use this branch in your project by including the lines below in your pubspec.yaml.
dependencies:
  google_sign_in:
    git:
      url: git://github.com/fbcouch/plugins.git
      path: packages/google_sign_in
      ref: server-auth-code

any idea when this will get pulled?

@bparrishMines Do you know if all of this will be usable with those test harnesses? In the specific case of this PR, we need to get a code from one google client ID for another google client ID and then exchange that code for an access token (as you would on a server) and check if it has a refresh token or not. To be honest, since this meets my needs, it's not likely I'm going to set all of that up for this small change, so someone else may want to take over from here.

apologies for probably just being a n00b here :see_no_evil: , but can we get an example with using this new option?
currently, server-side I can return something  like:
{"token_type":"Bearer","access_token":"ya...","scope":"email profile https://www.googleapis.com/auth/userinfo.profile openid https://www.googleapis.com/auth/userinfo.email","login_hint":"...","expires_in":3599,"id_token":"ey...","session_state":{"extraQueryParams":{"authuser":"0"}}}

from my understanding from peeking at the code, the intended way to login with the flutter library is like:
GoogleSignIn _googleSignIn = GoogleSignIn(
  serverClientId: <insert server client id here>
  scopes: <String>[
    <the scopes you want>,
  ],
);

Future<void> login() async {
  try {
    await _googleSignIn.signIn();
  } catch (error) {
    print(error);
  }
}

am I missing anything? (probably something simple I just overlooked such as additional configuration needed... :/)
tl;dr attempting to authenticate without firebase, is this even the right way to attempt this?

Try calling await _googleSignIn.signOut(); before signIn...I'm not 100% sure if you need that every time or only once in order to get the refresh token

I just tried this code in my project, but im getting the below error
E/flutter (30432): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: PlatformException(exception, two different server client ids provided, null)
E/flutter (30432): #0      StandardMethodCodec.decodeEnvelope
package:flutter/…/services/message_codecs.dart:564
E/flutter (30432): #1      MethodChannel.invokeMethod
package:flutter/…/services/platform_channel.dart:302
E/flutter (30432): <asynchronous suspension>
E/flutter (30432): #2      GoogleSignIn._ensureInitialized
package:google_sign_in/google_sign_in.dart:252
E/flutter (30432): #3      GoogleSignIn._callMethod
package:google_sign_in/google_sign_in.dart:228
E/flutter (30432): <asynchronous suspension>
E/flutter (30432): #4      GoogleSignIn._addMethodCall
package:google_sign_in/google_sign_in.dart:276
E/flutter (30432): #5      GoogleSignIn.signOut
package:google_sign_in/google_sign_in.dart:354

It is not the package problem,  i used firebase and own client id, i just copied the clinet id from googleservies.json to this package. and it is working fine.

I have checked the code working fine, but signInSilently is always getting bull if we provided the own clinet id otherwise working fine.

any updates?

@fbcouch Seems like you have not been actively working on this for a while. Are you still able to work on this?

@cyanglaz I might circle back eventually, but it will probably be awhile. If someone else wants to take it over, that's fine with me.

I started modifying the google_sign_in plugin in the same manner as this PR, but I got the same error as @pavankumarkatakam.
E/flutter (30432): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: PlatformException(exception, two different server client ids provided, null)

I'm using Firebase and I want to get the auth code so I can pass it to my server. I'm wondering if Firebase is setting the serverClientId silently, and the Java class can't handle 2 of them.

Were you able to solve it?

I started modifying the google_sign_in plugin in the same manner as this PR, but I got the same error as @pavankumarkatakam.
E/flutter (30432): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: PlatformException(exception, two different server client ids provided, null)

I'm using Firebase and I want to get the auth code so I can pass it to my server. I'm wondering if Firebase is setting the serverClientId silently, and the Java class can't handle 2 of them.

Were you able to solve it?

No. I haven't tried to move forward with it. I'm using Firebase with Google Sign In, and I assume that there's something in the Firebase Android/iOS library that is setting (or adding) a server client id, because it's required to use Firebase, but that's not clearly documented. So, if you want to use Firebase you can't set your own server client id to enable authorization code flow for your server.

Since the author is not able to working on this any time soon. I am closing this PR.
The same issue has been worked on by @yamarkz recently in https://github.com/flutter/plugins/pull/2116

Esto debería solucionar los problemas de flutter / flutter # 16613 y flutter / flutter # 17813
Hay una serie de cambios de formato introducidos por flutter_plugin_tool ... Estoy feliz de retirarlos si lo desea.

Does anyone know how to implement the serverClientId option to return serverAuthCode?


Copy local files to current working directory

created a DTConfig object with local_resources specified that is passed to instances of DTChecker, DTFinder and DTParser
created a copy_local_files function that ensures local files defined by config's local_resources attribute are copied to the cwd where pytest carries out its tests
implemented the pytest_unconfigure hook to ensure all copied local files are deleted when testing is complete

Note: commented out the plugin tests which are currently failing since pytester creates a temporary directory for testing and cannot find the local resources.

@ev-br @melissawm
I'm unable to figure out why the plugin tests are failing. Some assistance please.
The local_files path is accurate but the files cannot be found.

@ev-br unfortunately, I am still stuck on this.
The plugin test cases in test_pytest_configuration seem not to find the local files from the stated file path. I tried using the os.path.abspath() function to pass the absolute file path which should be something like/home/runner/work/scpdt/scpdt... on the CI. But it's using the pytester temporary directory instead, thus not being found.
I have not given up though.

I think I will need to specify to the user to define local_resources if any, otherwise set the config variable to default DTConfig().

The main thing at this stage IMO is to converge on some way of storing a passing around various bits of configuration. Ideally, tack averything we need to the pytest configuration object, if one exists.

Let me look into this. I know the only way to utilize pytest's Config object is through ini options. But I want to see if we can maintain use of DTConfig instead which I feel is much easier to use.

I tried implementing @melissawm 's solution of using ini options to specify local_resources and it works. It feels more robust and less hacky. It also removes the need for global variables and clean-up after tests.
However, the local files cannot be found when running pytest in the root directory. It only works wherever the files are... locally, this is in the tests directory. I am currently trying to find a solution to this. I would really appreciate your insight. @melissawm @ev-br

FWIW I'd be wary of solutions which work with ini files only or contest.py only

FWIW I'd be wary of solutions which work with ini files only or contest.py only

Well, the previous implementation of copying files to the cwd worked fine but I am unable to perform a proper clean-up without the copied_files global variable since I can't keep track of copied files and files that didn't need to be copied. I still cannot find a workaround, unfortunately.

Ok, we can probably live with globals then. One question before we declare it done though. Can we attach what we need here to the pytest config object and pass it around?

Can we attach what we need here to the pytest config object and pass it around?

I think the only way that is possible is through cli and ini options. Maybe @melissawm can weigh in on this.

To keep the ball rolling, I think we should merge the working version with global lists etc, only please rename different config variables. Then we add other features and then consolidate.

As discussed, let's merge this in its current form to keep the ball rolling. Thanks Sheila!


Use CMake RESCAN for MKL linking

What
Resolve static library linking errors on Linux x86-64 builds.
Requires CMake 3.24 or higher.
Without this change I consistently fail to link when following the build instructions on Ubuntu 20.04 and LLVM 15.0.7:
stefan@stefan-myscaledb-test2:~/MyScaleDB$ cmake --build build
...
/usr/bin/x86_64-linux-gnu-ld: ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_core.a(slarft.o): in function `mkl_lapack_slarft':
slarft_gen.f:(.text+0xd61): undefined reference to `mkl_blas_strmm'
/usr/bin/x86_64-linux-gnu-ld: slarft_gen.f:(.text+0x226b): undefined reference to `mkl_blas_strmm'
/usr/bin/x86_64-linux-gnu-ld: ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_core.a(slarfb.o): in function `mkl_lapack_slarfb':
slarfb_gen.f:(.text+0x20b): undefined reference to `mkl_blas_strmm'
/usr/bin/x86_64-linux-gnu-ld: slarfb_gen.f:(.text+0x344): undefined reference to `mkl_blas_strmm'
/usr/bin/x86_64-linux-gnu-ld: slarfb_gen.f:(.text+0x481): undefined reference to `mkl_blas_strmm'
/usr/bin/x86_64-linux-gnu-ld: ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_core.a(slarfb.o):slarfb_gen.f:(.text+0x689): more undefined references to `mkl_blas_strmm' follow
/usr/bin/x86_64-linux-gnu-ld: ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_core.a(slarf.o): in function `mkl_lapack_slarf':

... and many other similar symbols.
How
After snooping around with nm on the MKL libraries I noticed all the required symbols are there in the 3 static libs linked, but the linking process is failing because it's not doing multiple passes to resolve the symbols, see this snippet from the build above:
: && /usr/bin/clang++-15 --target=x86_64-linux-gnu --sysroot=/home/stefan/MyScaleDB/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc --gcc-toolchain=/home/stefan/MyScaleDB/cmake/linux/../../contrib/sysroot/linux-x86_64 -std=c++20
...
 rust/supercrate/libsupercrate.a  contrib/search-index/libmyscale_search_index_static.a  contrib/search-index/contrib/faiss/faiss/libfaiss.a  ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_intel_lp64.a  ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_intel_thread.a  ../cmake/linux/../../contrib/sysroot/linux-x86_64/mkl/libmkl_core.a  ../cmake/linux/../../contrib/sysroot/linux-x86_64/lib/x86_64-linux-gnu/libomp.so.5
...

Notice the lack of -Wl,--start-group and -Wl,--end-group.
It appears the correct way to link static MKL is with multiple passes, see:

https://github.com/facebookresearch/faiss/issues/96#issuecomment-299108019
Intel® oneAPI Math Kernel Library Link Line Advisor

I resolved this problem by using the RESCAN predefined feature introduced in CMake 3.24:

Some linkers are single-pass only. For such linkers, circular references between libraries typically result in unresolved symbols. This feature instructs the linker to search the specified static libraries repeatedly until no new undefined references are created.

Normally, a static library is searched only once in the order that it is specified on the command line. If a symbol in that library is needed to resolve an undefined symbol referred to by an object in a library that appears later on the command line, the linker would not be able to resolve that reference. By grouping the static libraries with the RESCAN feature, they will all be searched repeatedly until all possible references are resolved. This will use linker options like --start-group and --end-group, or on SunOS, -z rescan-start and -z rescan-end.

Closing this as it was pointed out to me in Discord the correct approach is to use the existing scripts which require separate installation of MKL and referencing it via LD_LIBRARY_PATH. Live and learn.

We are planning to remove the mkl libraries from faiss and change the relevant matrix libraries to blas and lapack. Thanks for your contributions, we will review the PR in MyScaleDB for how to build from source.


Enable logging when creating apps.

Logging will eventually get disabled by the platform and that's okay, just make sure that customers can get to the logs as easily as possible for fresh apps. There are two reasons for this:
1. The first deployment is the most likely deployment to fail
We see most deployment failures on the initial deployment so anything we can do to improve that experience is a benefit to the customer.
2. We immediately direct customers to the logs after a deployment
After a deployment, the notification includes a "stream logs" button, which, when clicked, launches into a modal to verify that logging needs to be set and the app needs restarting. This is unnecessary.

Fixed in https://github.com/microsoft/vscode-azureappservice/pull/1370


⚠ ESPN has degraded performance

In e6b5e4a, ESPN (https://espn.com) experienced degraded performance:

HTTP code: 200
Response time: 3147 ms

Resolved: ESPN performance has improved in 1337d38.


Java heap objects overhaul

We need to ensure, all short-living objects, like INDArray impls, DataBuffer, DataSet, etc - are all lightweight objects, or pointers to off-heap too. That'll give us further reduction of GC pressure.
To start with: check for legacy fields for all those classes, and get rid of them.

https://github.com/deeplearning4j/DataVec/issues/301

ND4j part got overhaul today, DataBuffer is 80 bytes now, looks like nothing to remove now :/


Making it multiplatform

Hi,
I just stumbled upon your project while looking for a pure Kotlin graphql parser I could use in my Kotlin multiplatform lib. And it looks great!
It seems that your lib works platform independently except from the modelreader package, which uses kotlin.reflect.full.declaredMemberProperties
Do you have plans to turn this into a multiplatform implementation?
I'm at the beginning to experiment with Kotlin multiplatform so I don't know too much about it, but it doesn't seem difficult with the current state of graphql-parser-kotlin to make it multiplatform.

Hey @beepsoft, there aren't any plans to maintain this, but I was curious about multiplatform so I gave it a crack.
Looks like kotlin.text's Char.isDigit and Char.isLetterOrDigit need some platform implementations (not available in JS) but other than that it wasn't too painful.

Yes, I just realized too that iDigit and friends are missing from javascript platform, but are available (and work in graphql-parser-kotlin) in jvm and native. Too bad. However I like the idea of a multiplatform graphql-parser-kotlin so I will ask around the forums to see whether we can make this work.

I added a JS target and simple implementation of isDigit/isLetterOrDigit

Nice! I tried to find a way to add isDigit to kotlin.text.Char for JS, so that your code could be used without change but that doesn't seem possible. Using a custom isDigit/isLetterOrDigit as you did seems the only possible way for now.
As this is now a truly multiplatform lib you may add it to https://github.com/AAkira/Kotlin-Multiplatform-Libraries, what do you think?
By the way, I really appreciate how you not maintain this project. 😃


Exclude not working for AllCops in .rubocop.yml?

I have some code that I don't want to be checked by the cops and hence configured the respective files to be excluded in .rubocop.yml:
AllCops:
  Exclude:
    - 'app/controllers/commontator/*.rb'

Unfortunately, the cops are run for files within that directory. If I explicitly exclude the same pattern for one of the violated cops, the violation is not reported anymore:
Metrics/ClassLength:
  Exclude:
    - 'app/controllers/commontator/*.rb'

Any idea why that is the case? I'm using rubocop 0.35.1 and this is the complete config (without the pattern in question):
AllCops:
  Exclude:
    - 'db/schema.rb'
    - 'vendor/**/*'
  DisplayCopNames: true
  RunRailsCops: true

Documentation:
  Enabled: false

Metrics/LineLength:
  Max: 180

Metrics/MethodLength:
  Max: 20

So if you add the exclusion in question, so that your .rubocop.yml becomes
AllCops:
  Exclude:
    - 'db/schema.rb'
    - 'vendor/**/*'
    - 'app/controllers/commontator/*.rb'
  DisplayCopNames: true
  RunRailsCops: true

Documentation:
  Enabled: false

Metrics/LineLength:
  Max: 180

Metrics/MethodLength:
  Max: 20

and run rubocop --list, then files under app/controllers/commontator/ show up in the list?

No, they don't show up. I think I know why this happens. I have a rubocop plugin for vim which runs rubocop when a file is saved but this does not care about the config rubocop uses when run from the command line. I'll try to figure it out or ask in the plugin repository. Nevermind.

There's an option called --force-exclusion, which editor plugins should use. The problem is that when you give a single file name to rubocop, then Exclude parameters don't come into play. Unless you run with --force-exclusion. I see that information about this is missing on the README page.

I was seeing this in 0.46 after not seeing it in 0.43. Turned out I had two AllCops stanzas in my .rubocop.yml. Once I merged them, the problem went away.


Update connection names from sMRIPrep's anat_preproc_wf

A recent PR in smriprep breaks connections between the anat and func preproc wfs:
https://github.com/poldracklab/smriprep/pull/108
Here:
https://github.com/poldracklab/fmriprep/blob/2c0a8b39af4fac7416ce5d242c45f86e4a195b91/fmriprep/workflows/base.py#L624
Should be an easy fix, I can submit a PR.

As long as we do not release another patch release of 0.3.x, this should not be a problem. I have updated the title. Thanks for reporting!

PR has been merged. Not an issue anymore! Thanks Alejandro.


[QUESTION] Undefined constant Junges\Kafka\RD_KAFKA_PARTITION_UA

@irfanpathanudr this constant comes from librdkafka, not this package. You need to install this library in order to use this laravel-kafka. This is the installation guide

My PHP Version 8.0.12

run commond
composer require mateusjunges/laravel-kafka
but composer show error

mateusjunges/laravel-kafka[v1.4.0, ..., v1.4.4] require ext-rdkafka ^5.0|^4.0 -> it is missing from your system. Install or enable PHP's rdkafka extension.

Root composer.json requires mateusjunges/laravel-kafka ^1.4 -> satisfiable by mateusjunges/laravel-kafka[v1.4.0, ..., v1.4.4].

run commond with
composer require mateusjunges/laravel-kafka --ignore-platform-reqs
no error come but kafka client throw error
Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA"
@mateusjunges Can you pl. help me to come out of this issue? It'll huge thanks for you :)

@irfanpathanudr This phpinfo() shows the web configuration for PHP. Some installations have a different configuration on CLI. Does rdkafka show up when you run php -i on the terminal, and which version?
You can also checked if the rdkafka extension is enabled in terminal at all by running php -m

thank you @rtuin
Call to undefined method RdKafka\ProducerTopic::producev()

can you pl. help me @rtuin

Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA" error occurs

@irfanpathanudr The Call to undefined method RdKafka\ProducerTopic::producev() issue occured because the librdkafka version is under 0.11.4, if you upgrade it should work.

@mateusjunges
my requirement is to setup Kafka client and my system configuration is
Ubuntu 18.04
php 7.2,
laravel/framework: "^6.2"  for Admin,
laravel/lumen-framework": "^7.0" for API,
Rdkafka 5.0 , librdkafka 0.11.3.
ubuntu 18.4 have only supporte librdkafka 0.11.3 version mention in  bellow image or link.

https://repology.org/project/librdkafka/versions
http://<IP_ADDRESS>/upstreamproject/details/librdkafka
so please help me out to communicate with kafka

@irfanpathanudr  It will not work on php 7.2, the minimun php version required is 8.0. About the librdkafka version, i can add a workaround to use the produce method instead, if the version is unde 0.11.4, but it will not be able to add message headers.

I suggest you do not run composer require with the --ignore-platform-reqs flag. The platform requirements defined in laravel-kafka's  composer.json is meant to prevent the issues you are running into. This applies to most if not all packages.

@irfanpathanudr i've added support for librdkafka v0.11.3, but as @rtuin mentioned, you should not run composer with --ignore-platform-reqs flag. You are only running into issues because of that, and i can't support you if your system does not follow the minimun requirements needed to use this package. I closed this with #43, but feel free to reopen or create another issue after you fix your system with the minimun requirements of this package.

@mateusjunges can you plz help me on this error is coming when am trying to send message to kafka.
Here is my code,

Kafka::publishOn('broker', 'topic')
->withBodyKey('foo', 'bar')
->send();
    return response('ok');

local.ERROR: Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA" {"exception":"[object] (Error(code: 0): Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA" at /var/www/html/smartplement-asset/vendor/mateusjunges/laravel-kafka/src/AbstractMessage.php:9)
Hi, im getting the same error while on Ubuntu 18.4 using build 1.6.6

This issue still persist when I try to use phpfranken and octane.
```bash
Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA" {"exception":"[object] (Error(code: 0): Undefined constant \"Junges\\Kafka\\RD_KAFKA_PARTITION_UA\" at /var/www/gateway/vendor/mateusjunges/laravel-kafka/src/AbstractMessage.php:11)
[stacktrace]
#0 /var/www/gateway/app/Actions/CreateWalletAction.php(21): Junges\\Kafka\\AbstractMessage->__construct(NULL, NULL, NULL, '{\"id\":\"264e48de...')
#1 /var/www/gateway/app/Http/Controllers/Api/V1/Auth/AuthController.php(51): App\\Actions\\CreateWalletAction->handle(Object(App\\Models\\User))
#2 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): App\\Http\\Controllers\\Api\\V1\\Auth\\AuthController->register(Object(App\\Http\\Requests\\V1\\Auth\\RegisterRequest))
#3 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(43): Illuminate\\Routing\\Controller->callAction('register', Array)
#4 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Route.php(259): Illuminate\\Routing\\ControllerDispatcher->dispatch(Object(Illuminate\\Routing\\Route), Object(App\\Http\\Controllers\\Api\\V1\\Auth\\AuthController), 'register')
#5 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Route.php(205): Illuminate\\Routing\\Route->runController()
#6 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Router.php(806): Illuminate\\Routing\\Route->run()
#7 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Routing\\Router->Illuminate\\Routing\\{closure}(Object(Illuminate\\Http\\Request))
#8 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#9 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#10 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Middleware/ThrottleRequests.php(159): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#11 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Middleware/ThrottleRequests.php(125): Illuminate\\Routing\\Middleware\\ThrottleRequests->handleRequest(Object(Illuminate\\Http\\Request), Object(Closure), Array)
#12 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Middleware/ThrottleRequests.php(87): Illuminate\\Routing\\Middleware\\ThrottleRequests->handleRequestUsingNamedLimiter(Object(Illuminate\\Http\\Request), Object(Closure), 'api', Object(Closure))
#13 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Routing\\Middleware\\ThrottleRequests->handle(Object(Illuminate\\Http\\Request), Object(Closure), 'api')
#14 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#15 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Router.php(805): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#16 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Router.php(784): Illuminate\\Routing\\Router->runRouteWithinStack(Object(Illuminate\\Routing\\Route), Object(Illuminate\\Http\\Request))
#17 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Router.php(748): Illuminate\\Routing\\Router->runRoute(Object(Illuminate\\Http\\Request), Object(Illuminate\\Routing\\Route))
#18 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Routing/Router.php(737): Illuminate\\Routing\\Router->dispatchToRoute(Object(Illuminate\\Http\\Request))
#19 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(200): Illuminate\\Routing\\Router->dispatch(Object(Illuminate\\Http\\Request))
#20 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}(Object(Illuminate\\Http\\Request))
#21 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#22 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#23 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#24 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#25 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#26 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#27 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#28 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#29 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(99): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#30 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#31 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(62): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#32 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Http\\Middleware\\HandleCors->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#33 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(39): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#34 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Http\\Middleware\\TrustProxies->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#35 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#36 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(175): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#37 /var/www/gateway/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(144): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter(Object(Illuminate\\Http\\Request))
#38 /var/www/gateway/vendor/laravel/octane/src/ApplicationGateway.php(36): Illuminate\\Foundation\\Http\\Kernel->handle(Object(Illuminate\\Http\\Request))
#39 /var/www/gateway/vendor/laravel/octane/src/Worker.php(84): Laravel\\Octane\\ApplicationGateway->handle(Object(Illuminate\\Http\\Request))
#40 /var/www/gateway/vendor/laravel/octane/bin/frankenphp-worker.php(53): Laravel\\Octane\\Worker->handle(Object(Illuminate\\Http\\Request), Object(Laravel\\Octane\\RequestContext))
#41 [internal function]: {closure}()
#42 /var/www/gateway/vendor/laravel/octane/bin/frankenphp-worker.php(74): frankenphp_handle_request(Object(Closure))
#43 /var/www/gateway/public/frankenphp-worker.php(3): require('/var/www/gatewa...')
#44 {main}
"}

@mateusjunges   Facing below issue while producing message
Undefined constant "Junges\Kafka\RD_KAFKA_PARTITION_UA"
configuration.
rdkafka
rdkafka support => enabled
librdkafka version (runtime) => 1.8.0
librdkafka version (build) => <IP_ADDRESS>

I am v2 documentation from this website https://junges.dev/documentation/laravel-kafka/v2.0/1-introduction.
I am using php -8.3
laravel  - 11,
mateusjunges/laravel-kafka : "^2.1"

You need to restart apache (If you are using Octane reload and update)


[SPARK-6166] [Spark Core] Limit number of concurrent outbound connections.

The overall number of requests in the cluster comes down with this (though in theory, you can still have one node getting swamped). In our experiments this has allowed jobs to progress where they previously die due to workers repeatedly going down when number of requests go up.
Typically happens when you have large number of nodes, and data per node is not too high (in one of the many jobs in the application) - drastic lowering of spark.reducer.maxSizeInFlightMb results is very poor performance overall.
Note: I have not included any testcases - suggestions on how to do the same in current spark test framework would be great - I have not looked at spark test suites in a while. Simulating this is slightly 'interesting'

Merged build triggered.

Merged build started.

Test build #31623 has started for   PR 5852 at commit 32168fc.

Merged build triggered.

Merged build started.

Test build #31625 has started for   PR 5852 at commit 0d8088a.

Test build #31623 has finished for   PR 5852 at commit 32168fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31623/
Test PASSed.

Merged build finished. Test PASSed.

Test build #31625 has finished for   PR 5852 at commit 0d8088a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31625/
Test PASSed.

Any updates on pushing this in ?
Our jobs do not run without using this and setting it to some modest value like 25.
The performance impact is minimal from our observations (since you are still having 25 outstanding requests to hosts) - while allowing jobs which fail earlier to run to completion : particularly in yarn mode where jobs get killed when memory usage shoots up beyond the container limit and the netty code is extremely 'generous' with using off heap buffers.

Test build #46796 has started for   PR 5852 at commit 0d8088a.

Test build #46796 has finished for   PR 5852 at commit 0d8088a.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46796/
Test FAILed.


PP-4949 Use "@pact-foundation/pact" dependency instead of deprecated "pact" dependency

WHAT

Remove pact dependency and use @pact-foundation/pact instead, because pact v4.x.x is no longer being supported

Will require not need to be updated for the test files?

Of course, just missed that one! :)


How to compile sol files to llvm ir files?

As https://era.zksync.io/docs/dev/compiler-toolchain/overview.html#high-level-source-code-compilers mentioned, I first compiled the sol file to a yul file. I know this is the first step, but I could not compile the yul file to an LLVM ir file using Zksolc. What should I do?

@hrshy0629 you can run zksolc input.yul --yul --debug-output-dir <dir>. The LLVM IR will be in <dir>.
I hope it helps!

@hedgar2017 It reported a lot of errors and I wasn't sure what I was doing wrong. I'll start by running solc -o solcC --ir test.sol, which will generate a yul file, Then run ./zksolc-linux-amd64-musl-v1.3.8 solcC/Ballot.yul --yul --debug-output-dir llvm, but it reports a lot of errors and there is no llvm ir file. The case I used is from the web https://docs.soliditylang.org/en/v0.8.17/solidity-by-example.html.


Btw, Has anyone figured out how to get valid option codes from the API since they expired the old field a few years ago?


Allow MultiLeafKnnCollector.greediness to be configurable

Description
In Lucene 9.10, we introduced a new algorithm for sharing state between segments to speed up the vector search process. This article explains the features quite well, and it looks like the idea of the greediness parameter is that we can tune it to control the latency-recall tradeoff: low value means more relaxed search (higher recall but higher latency, global queue less important) and high value means more restrictive search (lower latency but lower recall, global queue more important).
However the current code doesn't allow the parameter to be tunable: It will always use the default value of 0.9f. I think allow it to be configurable would allow each application to use a value suited to its specific use case and document/query patterns.
As per where user can configure this parameters, I'm thinking of 2 places:

Allow it to be configured in AbstractKnnVectorQuery, and propagate the value down to MultiLeafKnnCollector
Allow it to be configured only in TopKnnCollectorManager, and users would need to override AbstractKnnVectorQuery.getKnnCollectorManager()

We can make a PR, but would like to hear from the community first.

Lucene has tens of parameters like this one, exposing them all would make our APIs look rather bad. I wonder how you envision users to tune this parameter, would it be good enough if we made it configurable through something like a system property rather than through the API?

Do we have other internal API parameters via system property? I'm wondering why you think that's preferable to adding Java functions?

I think system property might be prohibitive for configurability. This parameter is a query-time control, but system property inherently enforce a single value for the whole application (and has to be determined at the start). Some use cases that I can see during tuning are:

Users start a Lucene service, maybe a HTTP wrapper. Then they would want to run multiple queries, and see how the results look like with different value of greediness. If the cost of starting up the service is high, then reusing a single JVM instance would be ideal.
Users might want to tune the value with the actual production settings, or even production instances. In this case, it would only be possible if the value is tunable through a query parameter (that is used to create the AbstractKnnVectorQuery).
Different set of queries may need different value of greediness. For example, for tail queries, which have small lexical match-set, we may want to get more recall here; for head queries, which already had large lexical match-set, we may want to tune for better latency.

With that being said, I agree that having too many parameters would also be bad for API brevity. What if we allow configuring this parameter with a separate method after the AbstractKnnVectorQuery is created? And if there are multiple parameters such as in case of AbstractVectorSimilarityQuery, we group them in a single struct-like object (such as TunableParameters; but of course we should try to reduce them as many as possible).

Do we have other internal API parameters via system property? I'm wondering why you think that's preferable to adding Java functions?

I don't think we do indeed, I'm trying to be creative to avoid the API tax of exposing all these low-level tuning knobs. To my knowledge, most of these knobs are not exposed, e.g. we don't allow users to tell us whether disjunctions should run using MAXSCORE or WAND, we don't allow users to configure the threshold in IndexOrDocValuesQuery, we don't allow users to configure whether filtered vector search queries should use the HNSW graph or perform a brute force search, etc.

Different set of queries may need different value of greediness. For example, for tail queries, which have small lexical match-set, we may want to get more recall here

FWIW I'd be less concerned about exposing a higher-level configuration option on the query, such as a "desired recall" or something along these lines and then make decisions for greediness (and other things) based on this option.

FWIW I'd be less concerned about exposing a higher-level configuration option on the query, such as a "desired recall" or something along these lines and then make decisions for greediness (and other things) based on this option.

This is an interesting idea, but I'm quite new to this. Can you elaborate it? How does this desired recall look like and what type of recall it tries to achieve?

[Disclaimer: I work with @dungba88 at Amazon Product Search team, and I also suggested he open this issue ;)]

Lucene has tens of parameters like this one, exposing them all would make our APIs look rather bad. I wonder how you envision users to tune this parameter, would it be good enough if we made it configurable through something like a system property rather than through the API?

This is indeed a problem/challenge for Lucene.  In cases like this I like the "simple things should be simple, complex things should be possible" mantra.  I.e. Lucene should have good defaults for interesting parameters like g, but really should expose the option for complex apps to tune it?
I guess we can tackle case by case... complex apps could always poach/fork the Lucene implementation and then change anything, worst case.

Here, g controls the greediness of non-competitive search and is some number less than 1. In effect, g is a free parameter we can use to control recall vs the speed up.

Hmm the thing is, if I'm reading the blog post correctly, this parameter g (greediness) is quite critical/fundamental a knob in this cool cross-concurrent-segment search algo, to trade off how the algorithm prefers exploiting vs exploration?  Why fuzz it up by adding a layer of indirection ("desired recall")?  Also, it seems likely the recall vs performance tradeoff might be model dependent?  Are all vectors really "created equal" (I honestly don't know how much variance in behavior there is among sets of vectors....).
How about if we expose tuning g through API, somehow, but mark it @lucene.experimental so we have the freedom to change/break it if we switch to "desired recall" in the future?  And mark it "expert"?

I have been thinking more and more of how we can pass "custom" or "complex" parameters via the API.
FWIW, this greediness is honestly focused on graph based indices only. If we ever did another type of index, its behavior would change.
This is a similar conversation around folks adding efSearch as a parameter. Or possibly custom kNN collector behavior (your own early stopping logic...).
A more opaque "hnsw knn search parameters" might be a better way, with a simple interface that is accepted in the queries. But adding an individual parameter for all the things you can do during query time with graph exploration would be complicated.

This is a similar conversation around folks adding efSearch as a parameter. Or possibly custom kNN collector behavior (your own early stopping logic...).

Is efSearch (a Faiss parameter) like the index-time beamWidth, but it applies to search-time?  Like it's the size N of the priority queue holding most promising vectors so far, such that N can be bigger than the eventual K that are finally returned?  Lucene does not expose this today?  Maybe it should be called searchBeamWidth if this logic is right :)
So the workaround for users now is to just ask for a larger top N and discard all but the top K in the end?  knnPerfTest (for benchmarking Lucene HNSW/KNN in luceneutil) seems to take this approach (it calls it fanout).
Another example might be interval from the MultiLeafKnnCollector -- crazy expert level parameter.  Though that one is more a tradeoff of concurrency vs total CPU cost.  Greediness seems more application / vector / model dependent and quite fundamental to this algorithm.

A more opaque "hnsw knn search parameters" might be a better way, with a simple interface that is accepted in the queries. But adding an individual parameter for all the things you can do during query time with graph exploration would be complicated.

+1 it'd be nice to find some simple way to allow "expert" tunables to be set rather than one at a time poking them through the APIs ad-hoc.  Though, lacking such a general mechanism also shouldn't block progress...

FWIW, this greediness is honestly focused on graph based indices only. If we ever did another type of index, its behavior would change.

But let's design for today?  We shouldn't let a possible future path ("maybe we some day expose non-graph-based approximate KNN") limit (too much) what we expose today?

So the workaround for users now is to just ask for a larger top N and discard all but the top K in the end?

Yes, that is the current solution.

+1 it'd be nice to find some simple way to allow "expert" tunables to be set rather than one at a time poking them through the APIs ad-hoc. Though, lacking such a general mechanism also shouldn't block progress......But let's design for today?

I agree, just wanting to clarify that the expert settings would be tied to a particular index format. Users would need to know the format they are using when utilizing the parameters.
Additionally, there are some other more advanced things you can do with HNSW (early exit based on other criteria) and quantization (dynamically gather more values based on some error bias).
As for how to set these tunables, its possible now to create a sub-class of the knn queries and override getKnnCollectorManager. Maybe MultiLeafKnnCollector just needs to be updated to allow more inputs (greediness is static and cannot be set via the ctor). That way folks can pretty easily sub-class the knn queries and return a customized multi-leaf collector.

As for how to set these tunables, its possible now to create a sub-class of the knn queries and override getKnnCollectorManager. Maybe MultiLeafKnnCollector just needs to be updated to allow more inputs (greediness is static and cannot be set via the ctor). That way folks can pretty easily sub-class the knn queries and return a customized multi-leaf collector.

+1, that sounds like a reasonable approach for expert usage?

I opened a PR with that approach.
As for

Allow it to be configured only in TopKnnCollectorManager, and users would need to override AbstractKnnVectorQuery.getKnnCollectorManager()

TopKnnCollectorManager is so simple, it just doesn't seem worth it. I would just assume somebody provide their own manager which returns the appropriate collector with the options they care about.

TopKnnCollectorManager is so simple, it just doesn't seem worth it. I would just assume somebody provide their own manager which returns the appropriate collector with the options they care about.

You are right, allowing MultiLeafKnnCollector to be extensible is the better way.


Ensure 1.0.0 tools will work with 1.0.0 runtime

When the tools packages (listed below) RTM, We'll need to ensure they work with the 1.0.0 runtime in addition to the 1.1.0 runtime.

Microsoft.EntityFrameworkCore.Design
Microsoft.EntityFrameworkCore.Tools.*

We'll especially need to investigate what to do with the Microsoft.EntityFrameworkCore.Design package. The original assumption was that this assembly would version alongside the runtime. We'll either need to make sure the 1.0.0 version plays nicely with the 1.1.0 runtime, or release both a 1.0.0 and 1.1.0 version when the tools RTM.

Having a single version is not going to work. We added new migration operations which require referencing MS.EFCore.Relational 1.1.0.
So I guess we need to decide when to rev the number:

With each version of the runtime
When we need new features from the runtime

Long term I think we want to get the runtime/tooling package versions back in lock step (so option 1). So I think that approach would be fine for now too.

Ok, then we need to retroactively create a version of Microsoft.EntityFrameworkCore.Design that works with 1.0.1 and RTM it with the tools (or with the next preview).
Microsoft.EntityFrameworkCore.Design should not move to the tools repo. This solves the Reverse Engineering tests issue.

Also, we should start versioning Microsoft.EntityFrameworkCore.Design with the runtime from here on out.

This plan fits the original architecture.

Done

The plan is to release a 1.0.0-compatible version of Microsoft.EntityFramework.Design from dev branch of the aspnet/EntityFramework.Tools repo, and a 1.1.0-compatible version from the dev branch this repo.
After the tools RTM, we can transition the code back to the next rel/1.0.x branch on this repo.

Re-opening since our build infrastructure didn't support this.

The current plan is to:

Make Tools work with Design 1.0.0-preview2 (#6743)
Release Design with 1.0.2 (#6742)
Remove the work for step 1
Release Tools 1.0.0

That plan won't work. 1.0.0-preview2 doesn't contain the types we need. We have to go back to the original plan of releasing a 1.0.1 version with the next release of the tools.


Is imroy pubsubclient more reliable?

Sorry to raise this question here, but I'm wondering whether somebody here knows if https://github.com/imroy/pubsubclient is maybe more reliable/stable than the orignal PubSubClient for ESP8266. I've seen some hiccups on ESP8266 and Homie (reported here earlier), and I'm wondering whether this library maybe fixes those?

I also asked for feedback on the openHAB forum, maybe you saw it. This is something I definitely consider, and I started to work on it by abstracting MQTT functions into the MqttClient class, so it will be a piece of cake to make the switch.

I will try to benchmark the two libraries, to monitor the times it takes to connect / subscribe / publish, how many times it reconnects, and how much memory it uses.

FWIW, I've done a few tests with a simple sketch that subscribes to a single topic, prints the payload string to Serial: there's not much difference between the two libraries in terms of stability. I can fire off about 100 messages (qos=0, qos=1) and they tend to arrive without resetting the device. Anything above that number quickly crashes the module.
On the other hand, if I attempt to publish what the device received (i.e. play "echo" server), things quickly fall apart: while PubSubClient manages to echo two or three messages, the imroy version just quietly dies.
This is meant as a heads-up, that it's probably not worth your while to pursue the modularization, so please close this asap. :-)

Alright, thanks for testing! I will try to learn more about the MQTT network protocol, and attempt to implement something with the ESPAsyncTCP library. As it is non-blocking, it might handle a lot more workload and therefore be more stable. However, there is not support for SSL for now (https://github.com/me-no-dev/ESPAsyncTCP/issues/9).

Well, to be honest, within a closed environment, TLS is probably overrated anyway. 😆
/me ducks

@marvinroger You should speak to @ckrey about this: he's our MQTT protocol guru. Honestly: he built OwnTracks for iOS and has done tons of MQTT protocol tests with different brokers. I'll ask him to create a repo to start working on this with you.

https://github.com/ckrey/mqtt-esp

I can't work on this right now, I have my exams soons so I'll have to slow down. :confused:

I had some issues in the past with pubsubclient and 10+ retained messages.
After connection all the messages are bombed into my ESP8266 and forces sometimes a crash.
To avoid this, I added a ring buffer and push all arrived messages (ok maybe not all ;) )
In my loop I process/pop all stored messages.
This is working really stable for me by the way

Here is the asynchronous library I've been working on this week-end: marvinroger/async-mqtt-client. It seems pretty stable, but some exceptions seem to appear on disconnect... It would help a lot if I can have feedback on it. :smiley:

AsyncMqttClient will be used, see https://github.com/marvinroger/homie-esp8266/issues/87


feat: add ui5-linter to template | bump devDeps

Since v1 of the ui5-linter is released, it should be part of the TS template.
[CoPilot]
This pull request includes updates to the package.json files in the generators/app/templates and root directories to add the new UI5 linting capabilities and update dependencies.
Linting Enhancements:

generators/app/templates/package.json: Added a new script "ui5lint": "ui5lint" for UI5 linting.
generators/app/templates/package.json: Added @ui5/linter version ^1.0.2 to devDependencies to support the new UI5 linting script.

Dependency Updates:

generators/app/templates/package.json: Updated versions of typescript-eslint, globals, and typescript to ^8.8.1, ^15.11.0, and ^5.6.3 respectively.
package.json: Updated versions of eslint, eslint-plugin-jsdoc, globals, and husky to ^9.12.1, ^50.3.2, ^15.11.0, and ^9.1.6 respectively.

Thanks!


Merge from master to pre-release

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: zunhu sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Database dump (.dmp) import fails

After adding Oracle DB dump (.dmp) file to /docker-entrypoint-initdb.d/, I get the following error during container startup.
Import: Release <IP_ADDRESS>.0 - Production on Thu Dec 7 10:28:08 2017

Copyright (c) 1982, 2015, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 12c Standard Edition Release <IP_ADDRESS>.0 - 64bit Production
ORA-39002: invalid operation
ORA-39070: Unable to open the log file.
ORA-29283: invalid file operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file operation

This happens due to lack of permissions on /docker-entrypoint-initdb.d/ directory:
root@77c0dae5dcba:/# ls -l
total 80
drwxr-xr-x   2 root root 4096 Nov 27  2015 bin
drwxr-xr-x   2 root root 4096 Apr 10  2014 boot
drwxr-xr-x   5 root root  360 Dec  7 10:31 dev
drwxr-xr-x   2 root root 4096 Dec  7 10:31 docker-entrypoint-initdb.d
...

RUN chown -R oracle:dba docker-entrypoint-initdb.d in Dockerfile does not work. However, if the same is done in entrypoint.sh, it works. So, the solution is, probably, to add appropriate chown to entrypoint.sh.

Thans for PR, Have you tested with a new build?

Tested with new build. The problem is solved!


Notebook that extracts SiteIDs from an excel sheet

Oh yes, good point about the sensitive data, I was thinking about that, but figured it's ok as it's private, but better safe than sorry.
Sometimes, having expected/example output is useful, but I don't think this notebook is the case. When that will be useful I might make a mini fake data example for those cases.
I will recreate a fresh the branch to make the commits harder to find (if you think necessary I can also make sure that they are deleted properly.)
More info about what the notebook is doing sounds good. 👌


Chapter not available (Source: 1stKissNovel)

Steps to reproduce

Novel: The Rise of the White Lotus w/ source as 1stKissNovel
From Chapter 104 - 115, it just shows that chapter is not available.
Would like for a fix.

Expected behavior
The said chapters should show the content.
Actual behavior
The said chapters just shows that the chapter(s) is/are not availabl

e
LNReader version
<IP_ADDRESS>
Android version
Android 12
Device
Realme 7 Pro
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with an source, I should be opening an issue in the sources repository.
[X] I have updated the app to version 1.1.12.
[X] I will fill out all of the requested information in this form.

If this is an issue with an source, I should be opening an issue in the sources repository.


Installation fails on Node 12

Running npm install shards-ui fails on Node.js 12 (on macOS). There's a compilation error on iltorb, which seems to be caused by an outdated dependency (see: https://github.com/MayhemYDG/iltorb/issues/85 )
Here's the full error:
$ npm install --save shards-ui

><EMAIL_ADDRESS>install /Users/alessandro/Desktop/Code/platform/controller/node_modules/iltorb
> detect-libc prebuild-install || node-gyp rebuild

prebuild-install WARN install No prebuilt binaries found (target=12.1.0 runtime=node arch=x64 platform=darwin)
  CC(target) Release/obj.target/iltorb/brotli/common/dictionary.o
  CC(target) Release/obj.target/iltorb/brotli/dec/bit_reader.o
  CC(target) Release/obj.target/iltorb/brotli/dec/decode.o
  CC(target) Release/obj.target/iltorb/brotli/dec/huffman.o
  CC(target) Release/obj.target/iltorb/brotli/dec/state.o
  CC(target) Release/obj.target/iltorb/brotli/enc/backward_references.o
  CC(target) Release/obj.target/iltorb/brotli/enc/backward_references_hq.o
  CC(target) Release/obj.target/iltorb/brotli/enc/bit_cost.o
  CC(target) Release/obj.target/iltorb/brotli/enc/block_splitter.o
  CC(target) Release/obj.target/iltorb/brotli/enc/brotli_bit_stream.o
  CC(target) Release/obj.target/iltorb/brotli/enc/cluster.o
  CC(target) Release/obj.target/iltorb/brotli/enc/compress_fragment.o
  CC(target) Release/obj.target/iltorb/brotli/enc/compress_fragment_two_pass.o
  CC(target) Release/obj.target/iltorb/brotli/enc/dictionary_hash.o
  CC(target) Release/obj.target/iltorb/brotli/enc/encode.o
  CC(target) Release/obj.target/iltorb/brotli/enc/entropy_encode.o
  CC(target) Release/obj.target/iltorb/brotli/enc/histogram.o
  CC(target) Release/obj.target/iltorb/brotli/enc/literal_cost.o
  CC(target) Release/obj.target/iltorb/brotli/enc/memory.o
  CC(target) Release/obj.target/iltorb/brotli/enc/metablock.o
  CC(target) Release/obj.target/iltorb/brotli/enc/static_dict.o
  CC(target) Release/obj.target/iltorb/brotli/enc/utf8_util.o
  CXX(target) Release/obj.target/iltorb/src/common/allocator.o
  CXX(target) Release/obj.target/iltorb/src/common/stream_coder.o
  CXX(target) Release/obj.target/iltorb/src/dec/stream_decode.o
../src/dec/stream_decode.cc:14:72: error: no matching member function for call to 'ToObject'
    Local<Object> dictionary = Nan::Get(params, key).ToLocalChecked()->ToObject();
                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2532:44: note: candidate function not viable:
      requires single argument 'context', but no arguments were provided
  V8_WARN_UNUSED_RESULT MaybeLocal<Object> ToObject(
                                           ^
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2546:35: note: candidate function not viable:
      requires single argument 'isolate', but no arguments were provided
                    Local<Object> ToObject(Isolate* isolate) const);
                                  ^
../src/dec/stream_decode.cc:40:49: error: no matching member function for call to 'ToObject'
  StreamDecode* obj = new StreamDecode(info[0]->ToObject());
                                       ~~~~~~~~~^~~~~~~~
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2532:44: note: candidate function not viable:
      requires single argument 'context', but no arguments were provided
  V8_WARN_UNUSED_RESULT MaybeLocal<Object> ToObject(
                                           ^
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2546:35: note: candidate function not viable:
      requires single argument 'isolate', but no arguments were provided
                    Local<Object> ToObject(Isolate* isolate) const);
                                  ^
../src/dec/stream_decode.cc:48:35: error: no matching member function for call to 'ToObject'
  Local<Object> buffer = info[0]->ToObject();
                         ~~~~~~~~~^~~~~~~~
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2532:44: note: candidate function not viable:
      requires single argument 'context', but no arguments were provided
  V8_WARN_UNUSED_RESULT MaybeLocal<Object> ToObject(
                                           ^
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2546:35: note: candidate function not viable:
      requires single argument 'isolate', but no arguments were provided
                    Local<Object> ToObject(Isolate* isolate) const);
                                  ^
../src/dec/stream_decode.cc:54:16: error: no matching member function for call to 'BooleanValue'
  if (info[2]->BooleanValue()) {
      ~~~~~~~~~^~~~~~~~~~~~
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2559:8: note: candidate function not viable:
      requires single argument 'isolate', but no arguments were provided
  bool BooleanValue(Isolate* isolate) const;
       ^
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2562:51: note: candidate function not viable:
      requires single argument 'context', but no arguments were provided
                V8_WARN_UNUSED_RESULT Maybe<bool> BooleanValue(
                                                  ^
../src/dec/stream_decode.cc:70:16: error: no matching member function for call to 'BooleanValue'
  if (info[1]->BooleanValue()) {
      ~~~~~~~~~^~~~~~~~~~~~
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2559:8: note: candidate function not viable:
      requires single argument 'isolate', but no arguments were provided
  bool BooleanValue(Isolate* isolate) const;
       ^
/Users/alessandro/.node-gyp/12.1.0/include/node/v8.h:2562:51: note: candidate function not viable:
      requires single argument 'context', but no arguments were provided
                V8_WARN_UNUSED_RESULT Maybe<bool> BooleanValue(
                                                  ^
5 errors generated.
make: *** [Release/obj.target/iltorb/src/dec/stream_decode.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/alessandro/.nvm/versions/node/v12.1.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:196:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:256:12)
gyp ERR! System Darwin 18.5.0
gyp ERR! command "/Users/alessandro/.nvm/versions/node/v12.1.0/bin/node" "/Users/alessandro/.nvm/versions/node/v12.1.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/alessandro/Desktop/Code/platform/controller/node_modules/iltorb
gyp ERR! node -v v12.1.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>- 3 but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>- 3 but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>install: `detect-libc prebuild-install || node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Hey guy, how about this issue?

@ItalyPaleAle Can you please provide the exact version of Node you're using while having this issue?

@cookieshell it looks like it's working now, a dependency must have been updated. It works on 12.4.0 on both macOS and Linux. Closing :)


Bad access error when deserializing a fully connected TF Lite model

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
('v1.3.0-rc1-5910-ge2174cc943', '1.4.0')
Describe the problem
I trained a very simple feed forward network (model structure is y = W*x + b so basically linear regression) using raw tensorflow code (matrix multiply, bias add and relu) so not using any Estimators or higher order APIs. My main purpose was to see if I could serialize the model, convert it to .lite format and then deserialize it correctly in C++.
I did freeze_graph on this graph.pbtxt and model checkpoints using tensorflow's provided freeze_graph method, and converted to .lite format using the bazel command line instruction (in the lite documentation) Lite conversion command used is
bazel run --config=opt \
  //tensorflow/contrib/lite/toco:toco -- \
  --input_file=frozen_model.pb  \
 --output_file=frozen_model.lite \
 --input_format=TENSORFLOW_GRAPHDEF \
  --output_format=TFLITE \
  --inference_type=FLOAT \
  --input_shape=**1,1** \
 --input_array=**x** \
 --output_array=**output**

As x is the name of the input 1-Dim placeholder tensor, output is the name of the result of W*x+b. I did not include y as part of the input_array as running with --input_array=x,y, --input_shape=1,1:1,1 as this produced a .lite model with 0 bytes (no output)
Using code very similar to what is in tflite_driver.cc and tflite_driver_test.cc, I was able to get the mobile_net example working fine but this simple model I described above fails with a EXC_BAD_ACCESS error. Upon debugging I saw that the error happens at fully_conncted.cc
template <KernelType kernel_type>
TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
....
  switch (input->type) {  // Already know in/out types are same.
    case kTfLiteFloat32:
      return EvalFloat<**kernel_type**>(context, node, params, data, input, filter,
                                    bias, output);
....
  }
  return kTfLiteOk;
}

kernel_type is kPie at this point. Looking further into fully_connected.cc, I notice two things.
enum KernelType {
  kReference,
  kGenericOptimized,  // Neon-free
  kNeonOptimized,
  kPie,  **// Used by the PIE team**
};

And also
TfLiteRegistration* Register_FULLY_CONNECTED() {
  **// TODO(ahentz): We don't have a dedicated quantized version of the PIE
  // kernel. For now, the quantized version just defer to the corresponding
  // optimized MINI kernel. At some point we will allow different libraries to
  // be built with different kernels, but for now we have to pick one here.
  return Register_FULLY_CONNECTED_PIE();**
#ifdef USE_NEON
  return Register_FULLY_CONNECTED_NEON_OPT();
#else
  return Register_FULLY_CONNECTED_GENERIC_OPT();
#endif
}

Specifically, the EXC_BAD_ACCESS error happens at
TfLiteStatus EvalPie(TfLiteContext* context, TfLiteNode* node,
                     ...
  // Output = bias if bias tensor exists.
  if (bias) {
    tensor_utils::VectorBatchVectorAssign(bias->data.f, num_units, batch_size,
                                          output->data.f);
  }

As we can see the filter and bias terms have bad memory addresses, the input and output tensors have valid memory addresses though
My questions are

Why is kPie kernel_type being picked for my simple model?
Based on the comments it looks like kPie is not supported for external use, so could this explain the bad access error?
Is this an issue with how I converted to lite format? Here is my command again

bazel run --config=opt \
  //tensorflow/contrib/lite/toco:toco -- \
  --input_file=frozen_model.pb  \
 --output_file=frozen_model.lite \
 --input_format=TENSORFLOW_GRAPHDEF \
  --output_format=TFLITE \
  --inference_type=FLOAT \
  --input_shape=**1,1** \
 --input_array=**x** \
 --output_array=**output**

Is there any example of a very simple model composed out of matmul, add and relu which can be converted to .lite in such a way that it can be correctly deserialized in C++, just like the mobile_net example? I can build up from such an example

Nagging Assignee @aselle: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@ushnish, were you able to resolve your issue?

Unfortunately I was not able to resolve this issue. Has anyone tried to run the end-to-end example I provided? I’ll be happy to revisit this issue if the example I provided works now

Nagging Assignee @aselle: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aselle: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aselle: It has been 16 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aselle: It has been 77 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aselle: It has been 92 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I just went over your example, and it should work without issues.
My guess is that your inference code is not holding on to the flatbuffer bytes (or of the FlatBufferModel) for the whole duration of inference.


[BUG] Can delete the last host in the hosts list

Describe the bug

You can delete the last host in the hosts list, but it never finishes. It just schedules it for removal
To Reproduce
Steps to reproduce the behavior:

Create 1 node harvester cluster
Log in
Navigate to hosts page
Delete host
Expected behavior

I'm not sure what the desired functionality is here, but here's a few options

We could disable the host delete button when there is only 1 host in the list (i'd probably recommend this)
We could make the host removal work
We could throw an error when trying to remove the last host
Support bundle

supportbundle_2bd06f00-af9f-4a16-aded-42e4f232944f_2022-03-01T19-43-13Z.zip
Environment:

Harvester ISO version: master-bfd6a8a9-head
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): KVM virtualized environment 1 node

Additional context
Add any other context about the problem here.

Verified fixed on master-03ee6213-head(04/15). Close this issue.
Result
When there is only one node on the Hosts page, Harvester won't provide the Delete host button and relative menu options
to prevent user deleting the last node

Test Information

Test Environment: 1 node harvester on local kvm
Harvester version: master-03ee6213-head

Verify Steps

Prepare a one node harvester cluster
Access Hosts page
Check if there is any delete button or options in this page


(WIP) Refactor HTTP::Client to use a transport

This PR is a first step to refactor HTTP::Client to make it more customizable by extracting the actual connection management to a Transport class.
Every HTTP::Client uses a configurable transport to establish the actual IO connection. By default, this is HTTP::Client.default_transport which is an instance of DefaultTransport and basically enables connections to any TCP endpoint (the implementation is currently very rudimentary).
If HTTP::Client is initialized with a host and port argument, it will use a TCPTransport connecting to a fixed endpoint. This is basically the behaviour as it was until now: the Client instance is bound to a specific connection.
The static methods like HTTP::Client.get will use HTTP::Client.default_client by default. But they can also receive a transport argument and create a custom client using that transport (thus it won't be reused).
This proposal allows for more advanced features to be added to HTTP::Client, which should rather be a high-level interface while Transport does the heavy lifting.
One such feature is to make requests to Unix sockets (#2735) which is already implemented in this PR.
Further enhancements would be connecting through a proxy transport as well as an advanced connection management for DefaultTransport which can pool and re-use connections.
While still work in progress I'd like to get some feedback on the general design. Please don't comment on implementation details, I know there are many rough edges. The docs are not updated yet either.
The interface between HTTP::Client and Transport probably needs to be more refined.
Currently, I've basically extracted the part where a  TCPSocket was created and replaced it with transport.connect(uri, request). The transport establishes an appropriate connection and returns the IO. The decision where to connect can be based on the URI and request.
I'm contemplating about moving the request and response serialization entirely to the transport. This would move more low level stuff out of the client. It would also allow the transport to easily return responses without reading them from an IO. Practical applications for this are for example a response cache or implementing a transport for the file scheme to serve local files as HTTP responses directly without serializing through an IO.

It seems Transport is also a thing in Go. Since we are copying Go so much, I can't understand why so many of you are against the idea of making Crystal more typed and more similar to other typed languages.

@asterite I'm not sure if your comment is sarcastic or just off-topic.
But yes, Go has such a transport as well and so have other HTTP clients in other languages. At least this has nothing to do with the type system.
I was just compiling a list of similar implementations for reference:

Ruby: Hurley has a connection property which is a lambda request -> response
Go: RoundTripper (interface request -> response) and Transport (implementation, including proxy, reusing connections, HTTP/2 and a lot of basic HTTP workflow like redirects).
Java: com.google.api.client.http.HttpTransport: builds a request (method, url -> request ) and request.execute() gets the response.
Ruby: Faraday implements a middleware interface similar to HTTP::Handler.

Cool, I thought Transport was Go-exclusive.

port is now required?

Internally, port was always required. You can't connect to a TCP socket without a port. But formerly, if omitted in the constructor, it was automatically deduced from tls which meant "use tls" or "don't use tls". The meaning of tls has no changed, it no longer enforces to use tls for every request, but only those that are actually using https scheme.
This means, you can use the same client to do https and http requests.
It would be possible to provide a default argument for port, but I'm not sure that makes much sense. If tls = false it's easy, but if tls = true either 80 or 443 make sense. So I think it's better to make it required in any case.

This means, you can use the same client to do https and http requests

Huh?

client = HTTP::Client.new(tls: true)
client.get("https://crystal-lang.org/")
client.get("http://crystal-lang.org/") # -> this wouldn't work with the previous meaning of `tls`

Strange, I thought we only allowed URIs, not full URLs, when initializing a client. But maybe it makes sense, I don't know. In the old ways it used to connect to a server with that base URL, I don't know what's going on now, but it's probably okay...

I love the first example more
client = HTTP::Client.new
client.get("https://crystal-lang.org/")
client.get("https://github.com/crystal-lang")

It would be possible to provide a default argument for port, but I'm not sure that makes much sense. If tls = false it's easy, but if tls = true either 80 or 443 make sense. So I think it's better to make it required in any case.

IMO that's too common use-case, why not use 80/443 as default, depending on http/https scheme?

@bararchy It depends on the use case. If you just need to query one API endpoint, it's much easier to use a client that's specifically connected to that endpoint.
@Sija Because the constructor HTTP::Client.new(host) wouldn't know the scheme you want to send requests with in advance. It could be viable to add an option to specify the port as "http" or "https" and derive the default value from that. But you'll need to specify either a port or a scheme, and specifying a port makes more sense to me. From a port you can always deduce the default scheme (80 -> "http", 443 -> "https", other -> nil), from just a scheme, you can't be certain about which port to use.
There is also the possibility to use a URI as argument, which can include scheme, host and port:
HTTP::Client.new(URI.parse("https://example.com")) # connects to default port 443
HTTP::Client.new(URI.parse("http://example.com:1234)) # connects to port 1234

So how does keep-alive work?

@asterite I think that should be implemented in the transport. That's why it probably makes sense to read the response there (last paragraph in the OP). For now, the TCPTransport keeps the connection and reuses it by default (until close is called, currently from keep-alive handling in the client).
DefaultTransport opens a new connection for every request, but as already mentioned should be enhanced to use a connection pool for re-using TCP connections across clients.

@ysbaddaden This PR is a proof of concept and not intended to be reviewed. It's far to incomplete for that anyway. I've just been exploring this approach to improve and simplify the http client implementation. And I wanted to share that and use it to support a discussion. Maybe it would have been better to open an issue and posted the code parallel to that. In fact, I can still do that.
TCPTransport is only there to keep the current behaviour of connecting the client to one specific server (and keeping that connection open) because DefaultTransport is not fully implemented. It doesn't have to stay that way and it can probably be removed later. TCPTransfer is not meant to be explicitly used anyway. It's mostly an interim internal implementation.
UnixTransport is already included in this PR and I don't think there is a viable alternative to that if we want HTTP::Client to be able to connect to Unix sockets. We have IO objects, but they need to be created somewhere. Maybe it can be simplified to pass in a UnixSocket (or any IO) directly as transport.

It's a bit strange because in Go a Transport (well, RoundTripper) takes a Request and returns a Response, not an IO.
I think Transport will make it easier to test an HTTP::Client, too. Right now there are no (any?) tests for HTTP::Client.

I really don't want to make things more complex here either. Why isn't IO enough? HTTP is a wire protocol, you write a request and receive a response. IO covers this exact API perfectly.
If we want middleware - which is looks to me is what you really want - let's implement middleware not this.
And lastly, @asterite, I work on crystal because it's not just another typed language.

Passing the client simply an IO can't be enough. How would you make simultaneous requests over the same client if it can't create a new IO itself? Or re-use connections to different servers?
Middleware is related but not dependend on this nor could this be implemented using middleware unless you would use the middleware to completely hijack the requests-response roundtrip.

@straight-shoota It seems there's a misunderstanding about how HTTP::Client was originally designed to work. The idea is that you create it with one server in mind, and execute queries against one server.

How would you make simultaneous requests over the same client if it can't create a new IO itself?

This wasn't intended in the original design. You can have one request per time. To execute a next request you must consume the last response. That's also why "keep-alive" is important here: with one server, "keep-alive" is true by default (depending on the HTTP protocol of the server), and so you can keep that connection alive.
It seems this PR is completely changing how HTTP::Client works. That's why for me it makes no sense to do HTTP::Client.new (to which server??)
That's also why doing this:
client.get("http://...")

makes no sense to me. You are targeting a single server, you don't need to repeat the scheme, host, etc. Just send the path.
If you want to redesign HTTP::Client, that's fine. But it shouldn't be done in a PR that completely changes how it works. I know you opened another issue for this, but it seems we first have to clarify how we want HTTP::Client to work.

Or re-use connections to different servers?

That's why we need a pool of HTTP::Clients. But you'd have one for each different server, so doing HTTP::Client.get(...) would, in my mind, use that pool, and try to see if there's already an existing client connected to that server that can be used. Of course this is difficult if we have to track that each client can only perform one request at a time...

Indeed, it's far easier to think of a pool of http clients, instead of a http client which contains a pool.

You are probably right, though I wouldn't know how to implement keep alive that way. And then an instance of an HTTP Client probably makes no sense if we go that way, because it's more useful to use the class methods.
If someone wants to implement this, go ahead.

Yes, the implementation of the connection pool is going to be very complex. I've looked through Go's Transport implementation, it's really huge.
I think instances of HTTP::Client are still useful to hold custom configuration for SSL context, high-level HTTP protocol features such as cookies and following redirects, and additional middleware.

@ysbaddaden I'm starting to agree with @straight-shoota . At least Go's http client can be used by multiple fibers and it works fine, so you can have multiple concurrent requests, something that right now doesn't work well... and if we don't do that, then any class that wraps an API won't be concurrent by default, and all of that logic must be implemented in each case, which is a burden.
I think @waj was also interested in having HTTP::Client have a pool of connections, though I don't think he'll have time to do this, but maybe he has some time to comment on this.

I'm starting to want to clean-sheet the whole of HTTP to be honest...

I like this idea.  I have been swapping out or monkey patching the socket to do some fun things.  but I would be interested in where this goes.  I am still a little fuzzy on all the features that you intend to get out of this.  Is it fair to say you want:

concurrency support
connection pooling
agnostic IO

#codetriage This PR appears to have been a good discussion point months ago, and stalled.
It's unclear what the future ought to be for HTTP::Client behaviors and how extensively the default behaviors should be - this probably deserves a scoping session to determine what behaviors we've seen from other languages/libraries/frameworks.


Add chapter around how memory is laid out for descriptor binding

Similar to to the Vertex Input Data chapter, but around the use of buffer memory blocks in relation to each of the parameters in DescriptorBufferInfo, DescriptorSetLayoutBinding, WriteDescriptorSet and vkCmdBindDescriptorSets

+1 - this would be really useful

closed from https://github.com/KhronosGroup/Vulkan-Guide/pull/161