Redis: can not check if index exists and can not create index if it does not

I really really love langchain. But you are moving too fast, releasing integration after integration without documenting the existing stuff enough or explaining how to implement real life use cases.
Here is what I am failing to do, probably one of the most basic tasks:
If my Redis server does not have a specific index, create one. Otherwise load from the index. There is a _check_index_exists method in the lib. There is also a call to create_index but it is burried inside from_texts.
Not sure how to proceed from here

Thank you for bringing this issue to our attention. We apologize for the lack of documentation and explanation on how to implement real life use cases.
To check if an index exists and create one if it does not, you can use the following code example:
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores.redis import Redis

# Load documents
loader = TextLoader('../../../state_of_the_union.txt')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

# Create embeddings
embeddings = OpenAIEmbeddings()

# Create Redis instance
rds = Redis.from_documents(docs, embeddings, redis_url="redis://localhost:6379",  index_name='link')

# Check if index exists
if rds.index_name is not None:
    # Load from index
    query = "What did the president say about Ketanji Brown Jackson"
    results = rds.similarity_search(query)
    print(results[0].page_content)
else:
    # Create index
    print(rds.add_texts(["Ankush went to Princeton"]))

# Query
query = "Princeton"
results = rds.similarity_search(query)
print(results[0].page_content)

We have also created a tutorial notebook that goes over different options for using the vector store as a retriever, as well as how to use Redis and SQLAlchemy caches. You can find the tutorial notebook here: [link to tutorial notebook].
We hope this helps and we apologize again for the lack of documentation. Please let us know if you have any further questions or concerns.


@JuliaRegistrator register

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/74208
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.2.0 -m "<description of version>" 850050a9d9346fc354e7aa83109e2801f2ab835a
git push origin v0.2.0

Also, note the warning: This looks like a new registration that registers version 0.2.0.
Ideally, you should register an initial release with 0.0.1, 0.1.0 or 1.0.0 version numbers
This can be safely ignored. However, if you want to fix this you can do so. Call register() again after making the fix. This will update the Pull request.

closed


Add named fragments for the headings in our docs

Expected Behavior
Changing the text of a Heading in a doc shouldn't break all links pointing to that heading.
Actual Behavior
Changing the words of a Heading in one of our docs completely breaks any other docs who were linking to it.  This happens because github automatically generates the "fragment identifier" (#foo-bar-baz) from the text of the heading. ## Getting Started With Workspaces becomes #getting-started-with-workspaces.  Changing this text in any way breaks incoming links.
We can fix this by introducing explicit named anchors to our headings like this:
#  Getting Started With Workspaces

This will allow links from other docs to point at workspaces.md#workspaces-intro regardless of the text in the heading.  It would also allow us to preserve those links even when we completely reorganize the structure of a document.

 Getting Started With Workspaces

That seems like a pretty big ask for folks to create on their own, and then asking reviewers to catch. @sbwsg what do you think if we had some automation that added these kinds of links automatically?
e.g. a periodic job / post merge job that creates (and merges) a PR that adds these links when they are missing
If that doesn't work, I'd rather explore a different docs format (e.g. ascii doc) than ask every doc writer to add these links manually, but I hope we don't have to do that.
Another avenue to explore would be automation that can look across all our docs and find dead links.

That seems like a pretty big ask for folks to create on their own, and then asking reviewers to catch.

I'm a tad confused by this.  Is it more work than also picking up on grammar, spelling, punctuation etc in a doc review?  The named anchors only have to be added once and then preserved when headings are changed or docs are reorganized.

Just as an example (not planning to make a PR until/unless we reach agreement on best way forward), I've updated workspaces.md to include the named fragments: https://github.com/tektoncd/pipeline/compare/master...sbwsg:name-fragments
It only took a few minutes to add these and the benefit will be that we can keep the same named anchors even if we reorganize the info.  So imagine we rewrote the portion of the doc where we list the supported volumesources: As long as we maintain a  in the doc at the part where we describe the emptydir source then users will continue to land at the correct updated information regardless of any changes to wording / structure of the document.

I'm just having a lot of trouble imagining maintaining this over time - basically the more manual gotchas we rely on, the more we'll miss. Happy to discuss more tho!
In the meantime I'm not sure about having "good first issue" on this until we discuss more - maybe we can discuss in the productivity and/or docs working groups.

I'm just having a lot of trouble imagining maintaining this over time - basically the more manual gotchas we rely on, the more we'll miss. Happy to discuss more tho!
In the meantime I'm not sure about having "good first issue" on this until we discuss more - maybe we can discuss in the productivity and/or docs working groups.

I do agree with @bobcatfish, maintaining this over time might be a tiny headache. I think this might have been a reason why we had the markdown linter (although I am not sure it was checking for that). I would rather rely on a linter that is capable of erroring out if an anchor link is pointing to the void than having to add those on all the title.

Automation and linting don't account for external links into our docs.  Using named anchors makes the backlinks constant so that outside articles can point into sections of our documentation.
On the subject of maintenance burden I think we need to quantify it rather than speak vaguely.
Current situation: A word is changed in a heading somewhere in a doc. This gets reviewed, approved, and merged. A user then has to notice that links inside our other docs are now broken. We then waste at least 2 people's time - the person who has to make the PR to fix it and the one who has to lgtm + approve. Multiply that by each broken link, since it's unlikely we catch all of them and fix them in a single PR. There's a lag here as well: humans have to notice the broken links. So we add automation to check our internal docs. But automation will never be able to catch all the broken backlinks from external sites into our pages.
With named anchors the work is actually reduced: reviewers notice during a docs PR review that the link name is changed or removed and ask for it to be put back / reorganized with the content. Backlinks from external sites remain constant and we can have automation check our internal doc links if we want to.
So I dunno. I think this is less work, more explicit, and more useful than what we do currently.

This is true, but it's also true any time we will want to refactor our docs (like https://github.com/tektoncd/pipeline/pull/2364). Some of those header might change significantly and thus the name would have to be changed, and you have broken link again. This is a problem that all docs are facing ; how many times do I ran into dead "anchor" links on the k8s doc, openshift doc, gcloud doc, git doc, …. In general I do agree with you but…
Questions are : When do we account for broken links ? And which ones can we "limit" ?
We can't make sure external links won't be broken (if the doc gets refactored, or moved), especially if it's a link targeting the tektoncd/pipeline GitHub repository (but any really). There is no guaranteed made that github will always serve the content in a way or another, same for the folder we put our docs in, at some point it might change, etc.. The story is a bit different on tekton.dev.
We can make sure "local" links are not broken, aka links in the same repository, or maybe in all content served my tekton.dev. At least our docs would always be consistent and without dead link.
This put some html tags in a markdown file, this assumes writers know html, this also assumes it will be published as html. Those assumption might not always be true, an example, on tektoncd/cli, part of our docs in markdown becomes a man page. Having those means complexifying the code to generate the man page from markdown, and remove one benefit of what markdown is supposed to provide (aka having a text-only syntax, not relying on html — or another other "output" implementation detail). pandoc version of Markdown has a "cleaner" way to do that : heading identifiers
For me forcing those would add burden on the contributors on something that is not fully solvable, that can be limited a little bit (at least on what we control) and that is a tooling limitation (markdown doesn't support this, nor we have a tool that would make us translate old link to new ones, …). I am not entirely against but I don't know if the tradeoff balance is positive or not.

OK sounds good to me.  Your points about pandoc offering this as a feature, github not guaranteeing our links, and the CLI having to filter out html make sense to me.  I'm going to close this issue now.  I'm not going to argue in favor of named fragments anymore because they're clearly not the right solution.

Since I was out yesterday and Vincent asked for my opinion, my TL;DR is
that while named anchors are a great idea, they only solve the problem if a
heading is slightly reworded. If the section is deleted, moved, or
rewritten and the heading must change completely, the anchor goes out of
sync with the heading meaning-wise and we'll have links like
Installing Smurfcaptor 9000 On
Debian. Whenever I make edits in a
doc, I check and fix all links it contains. If we do this for all docs
every time we edit them, we'll maintain at least a level of link
validation. In Bazel docs, we had a linter that checked links and blocked
the update CL/PR if broken links were detected.
Disclaimer: The above thoughts are mid-coffee.
On Mon, Apr 27, 2020 at 9:53 AM Scott<EMAIL_ADDRESS>wrote:

Closed #2488 https://github.com/tektoncd/pipeline/issues/2488.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tektoncd/pipeline/issues/2488#event-3275888807, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AG3UR4US6KOWE5O7442N6TLROWE4TANCNFSM4MQC7JEQ
.


add initial support environment variables

This PR

This PR adds a basic implementation of the Environment Variables feature provider
Supported Types
The provider supports the following types:

Int
float
String
bool

Please note that Struct type is not currently supported yet.

doc: https://github.com/jbovet/rust-sdk-contrib/blob/main/crates/env-var/README.md
changelog: https://github.com/jbovet/rust-sdk-contrib/blob/main/crates/env-var/CHANGELOG.md
Related Issues

related #339
Notes

How to test

cucumber features supported

My apologies for the delayed response! :-)

@beeme1mr @sheepduke could we merge this PR?

Hey @jbovet, sorry for the delay. I've merged this PR. Thanks!

Hey @jbovet, sorry for the delay. I've merged this PR. Thanks!

Happy to help @beeme1mr


Add missing class doc to NavigationMeshGenerator

Adds missing class documentation for NavigationMeshGenerator bake() and clear() functions.

Could you also add a brief description and description? In 3.x, the brief description and description both are:

This class is responsible for creating and clearing navigation meshes.

I'd suggest expanding upon the full description, and changing the brief descriptiuon to:

Helper class for creating and clearing navigation meshes.

Also, it should be noted explicitly that NavigationMeshGenerator is only for 3D (if I'm not mistaken).

@Calinou
Technically the NavigationMeshGenerator is 3D only but by converting Vector2 Polygons to Vector3 and back e.g. as the NavigationServer does behind the scene, it can be used as well for 2D but is not a straightforward use.

Thanks!

Cherry-picked for 3.5.
Since there's no NavigationRegion in 3.x I replaced the references with NavigationServer, might need double check that the docs added here (and in other navigation PRs cherry-picked today) are accurate for 3.x.

Will try to check later.
For general navigation 4.0 -> 3.5 renames:
NavigationRegion3D -> NavigationMeshInstance
NavigationRegion2D -> NavigationPolygonInstance
NavigationServer3D -> NavigationServer


Provide a config option to auto-detect a running jupyter server on localhost

survey feedback:
Provide a config option to auto-detect a running jupyter server on localhost...settings could be 'auto' 'prompt' 'off'

Based on our current implementation, it's not feasible to connect to a running server from a security point of view.


Problem running on OS: missing OpenMP & No module named pyopengv

I already brew install open-mpi.
But when I run python3 setup.py build, everything seems ok except for:

-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)

Then when I try bin/opensfm_run_all data/berlin, I get:

Running using Python command: python
Traceback (most recent call last):
File ".../OpenSfM/bin/opensfm", line 10, in
from opensfm import commands
File ".../OpenSfM/opensfm/commands/init.py", line 4, in
from . import match_features
File ".../OpenSfM/opensfm/commands/match_features.py", line 11, in
from opensfm import matching
File ".../OpenSfM/opensfm/matching.py", line 3, in
import pyopengv
ImportError: No module named pyopengv

I run make install again, all looks fine:

[ 92%] Built target opengv
[ 97%] Built target random_generators
[100%] Built target pyopengv
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libopengv.a
-- Up-to-date: /usr/local/lib/cmake/opengv-1.0/opengvConfig.cmake
-- Up-to-date: /usr/local/lib/cmake/opengv-1.0/opengvConfigVersion.cmake
-- Up-to-date: /usr/local/lib/cmake/opengv-1.0/opengvTargets.cmake
-- Up-to-date: /usr/local/lib/cmake/opengv-1.0/opengvTargets-release.cmake
-- Up-to-date: /usr/local/include
-- Up-to-date: /usr/local/include/opengv
-- Up-to-date: /usr/local/include/opengv/sac_problems
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/RotationOnlySacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/NoncentralRelativePoseSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/CentralRelativePoseSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/TranslationOnlySacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/EigensolverSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/MultiCentralRelativePoseSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/relative_pose/MultiNoncentralRelativePoseSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/point_cloud
-- Up-to-date: /usr/local/include/opengv/sac_problems/point_cloud/PointCloudSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/absolute_pose
-- Up-to-date: /usr/local/include/opengv/sac_problems/absolute_pose/AbsolutePoseSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac_problems/absolute_pose/MultiNoncentralAbsolutePoseSacProblem.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose
-- Up-to-date: /usr/local/include/opengv/relative_pose/NoncentralRelativeAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/CentralRelativeWeightingAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/RelativeMultiAdapterBase.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/MACentralRelative.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/CentralRelativeMultiAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/NoncentralRelativeMultiAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/MANoncentralRelative.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/CentralRelativeAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/MANoncentralRelativeMulti.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/methods.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/main.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/sixpt
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/sixpt/modules.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/ge
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/ge/modules.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/fivept_stewenius
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/fivept_stewenius/modules.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/fivept_nister
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/fivept_nister/modules.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/fivept_kneip
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/fivept_kneip/modules.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/eigensolver
-- Up-to-date: /usr/local/include/opengv/relative_pose/modules/eigensolver/modules.hpp
-- Up-to-date: /usr/local/include/opengv/relative_pose/RelativeAdapterBase.hpp
-- Up-to-date: /usr/local/include/opengv/point_cloud
-- Up-to-date: /usr/local/include/opengv/point_cloud/PointCloudAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/point_cloud/MAPointCloud.hpp
-- Up-to-date: /usr/local/include/opengv/point_cloud/methods.hpp
-- Up-to-date: /usr/local/include/opengv/point_cloud/PointCloudAdapterBase.hpp
-- Up-to-date: /usr/local/include/opengv/sac
-- Up-to-date: /usr/local/include/opengv/sac/SampleConsensus.hpp
-- Up-to-date: /usr/local/include/opengv/sac/MultiSampleConsensusProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac/Ransac.hpp
-- Up-to-date: /usr/local/include/opengv/sac/MultiRansac.hpp
-- Up-to-date: /usr/local/include/opengv/sac/SampleConsensusProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation
-- Up-to-date: /usr/local/include/opengv/sac/implementation/SampleConsensus.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation/MultiSampleConsensusProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation/Ransac.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation/MultiRansac.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation/SampleConsensusProblem.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation/Lmeds.hpp
-- Up-to-date: /usr/local/include/opengv/sac/implementation/MultiSampleConsensus.hpp
-- Up-to-date: /usr/local/include/opengv/sac/Lmeds.hpp
-- Up-to-date: /usr/local/include/opengv/sac/MultiSampleConsensus.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose
-- Up-to-date: /usr/local/include/opengv/absolute_pose/CentralAbsoluteAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/AbsoluteMultiAdapterBase.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/MACentralAbsolute.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/AbsoluteAdapterBase.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/NoncentralAbsoluteAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/NoncentralAbsoluteMultiAdapter.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/methods.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/main.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/Epnp.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/upnp4.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp1
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp1/modules.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/upnp2.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gp3p
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gp3p/modules.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp4
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp4/modules.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp3
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp3/modules.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp2
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp2/modules.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp5
-- Up-to-date: /usr/local/include/opengv/absolute_pose/modules/gpnp5/modules.hpp
-- Up-to-date: /usr/local/include/opengv/absolute_pose/MANoncentralAbsolute.hpp
-- Up-to-date: /usr/local/include/opengv/math
-- Up-to-date: /usr/local/include/opengv/math/cayley.hpp
-- Up-to-date: /usr/local/include/opengv/math/arun.hpp
-- Up-to-date: /usr/local/include/opengv/math/quaternion.hpp
-- Up-to-date: /usr/local/include/opengv/math/gauss_jordan.hpp
-- Up-to-date: /usr/local/include/opengv/math/Sturm.hpp
-- Up-to-date: /usr/local/include/opengv/math/roots.hpp
-- Up-to-date: /usr/local/include/opengv/OptimizationFunctor.hpp
-- Up-to-date: /usr/local/include/opengv/triangulation
-- Up-to-date: /usr/local/include/opengv/triangulation/methods.hpp
-- Up-to-date: /usr/local/include/opengv/Indices.hpp
-- Up-to-date: /usr/local/include/opengv/types.hpp
-- Up-to-date: /usr/local/lib/python3.6/dist-packages/pyopengv.cpython-37m-darwin.so

I'm on MacOS High Sierra.
Note: I tried issues 366 and 179 solutions, but these don't seem to help me.

Thank you.


check request header to identify user is using native app

Any request header that can be used to check whether user is using manifold JS installed on phone(native app) or using browser ?

This is something we need to put in the Wiki, I think just providing a list of the user agents for our supported platforms would be quite helpful.

What I meant was how to identify whether user is using a browser to view our website or using manifoldjs webview to view our website? Is there a way when manifoldjs webview doing HTTP request, it insert a header into the request so that we can identify it from website server side?


Fix for tangent linear

What changes does this PR introduce?

fix few bugs in tangent-linear (hand-written) src code '
also add new target "ftlobj" to Makefile (similar to "obj" and "adobj" for forward and AD built).

What is the current behaviour?

wrong number of arguments in autodiff_inadmode_set/unset TL versions (left from PR #411)
mask with wrong size in ctrl_bound_2d TL version (left from PR #445).

What is the new behaviour
fixed
Does this PR introduce a breaking change?
Only Tangent-Linear specific changes and nothing else.
Other information:
Suggested addition to tag-index
o tangent-linear:

fix few hand-written tangent-linear src code (in autodiff & ctrl pkgs);
add new target "ftlobj" to Makefile (similar to "obj" & "adobj" for forward
and AD built) and corresponding switch in testreport.

Might have done something wrong, but currently I am getting (e.g., from
file: offline_exf_seaice/input_ad/tr_checklist.adm ):
tlmGrd tlmCst tlmGrd tlmFwd T+ S+ aSI+ hSI+ uSI+ vSI+
but with with modified sed command I get:
tlmGrdtlmCsttlmGrdtlmFwd T+ S+ aSI+ hSI+ uSI+ vSI+
On Mon, Oct 11, 2021 at 07:55:45AM -0700, Oliver Jahn wrote:

One could use extended regular expressions, which are POSIX compliant, and spell out the word boundary match:
listChk=`echo $listChk | sed -E 's/(^|[^a-zA-Z0-9_])adm/tlm/g'`

This should work both on BSD/Mac and GNU, but maybe not on very old systems...
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/MITgcm/MITgcm/pull/546#issuecomment-940104281

@jahn and @mjlosch May be simply this:
``listChk=echo $listChk | sed 's/^adm/tlm/' | sed 's/ adm/ tlm/g'
would work ?


Map View Improvements

[x] Use our pin design (not default Apple pin) for map view
[x] Add arrow in bottom left corner to go to current location (like in Apple maps)

Closed with #83


Autolaunch and UI on preferences

Preferences settings not sticky eg:
When check Autolaunch, it checks, use system rclone checked checks, but when UI opened, Autolaunch and rclone is unchecked.
Expected: check status remains and UI not self close.

Fixed in 2.0.0-beta


Rework Dapp Connection Screen

Below an image is presented about dapp connection
We need to rework dapp connection screen. This screen is about dapp requesting from the wallet to provide address of active account. Should not seem like a transaction.

Identicon should not be created for the URL
Allow Access is misleading, since we do not providing access to the wallet here.
Concrete explanation shall be displayed, we are not connecting to the account, we are getting the address of current account.
Site URL should not be displayed 2 times

Dapp Connection Screen
CASE 1. The dApp name can be fetched, the URL can be fetched, dApp favicon/logo can be fetched and displayed.

CASE 2. The dApp name can not be fetched, dApp favicon/logo can not be fetched and displayed so we are replacing it with the generic label "DAPP" and generic icon.

CASE 3. Unknown URL! Rare case that can still occur. URL cannot be fetched by Superhero Wallet. In this case neither the favicon nor the dApp name will be available. Most probably it's a risky app with basically "unknown source". User should be warned about potential risks.

Figma reference: https://www.figma.com/design/3oGLWzSH0oJljo4RETZtur/Superhero-Wallet-UI-(✔Updated)?node-id=37239-260033&node-type=frame&t=xkVASfrr0tXWPBQs-0

@sotiris @CedrikNikita IMO there's an issue with the way we grant permission to third party dApp to view not only the current account but also all current active accounts in the future. In other words the dApp asks only once for permission to view current address but it will be able to view all "currently active addresses" until Superhero Wallet is connected. @CedrikNikita correct me if I'm wrong here.
I am concerned that the user is not aware what's happening. As a user I'd expect to give permission concerning only my current account address when connecting, logging in, providing permission to a dApp or platform.
I might want to use different accounts for different platforms and purposes. I would not expect to grant access to the dApp to view all my wallet accounts and that's what we are currently doing. I think we should reconsider this behavior and ask for permission each time the currently active account is changed.

@onvisions dapp requests access to one address is not this the case?

@onvisions dapp requests access to one address is not this the case?

As far as I understood no, this is not the case and that's why I raised my concerns from user perspective. Let's sync with @CedrikNikita to be on the same page if it's needed.
As far as the current task is concerned I have created the design according to request:
the dApp is asking for permission to view user's currently active account address (the address that is active at the moment of granting the permission).

@onvisions , @smaroudasunicorn - I confirm that the DAPPs are actually connecting with the wallet, not the addresses. For AE if a DAPP is connected whenever user changes the active account the DAPP knows about that and can make an action based on this action. The best example is our testing DAPP: https://docs.aeternity.com/aepp-sdk-js/v13.3.2/examples/browser/aepp/
Try to connect and open the "Basic functionality" tab. Then open the extension and try to switch accounts. You'll see that the testing DAPP will update the active account on the fly.
This is why I agree with Tsvetan that this connect modal is not fully informative.
MetaMask is asking you to list the accounts you want to use for connection:

Dapp Connection Screen
CASE 1. The dApp name can be fetched, the URL can be fetched, dApp favicon/logo can be fetched and displayed.

I suggest to not put the dapp name as a tag above the arrow. There is a limited space in this area and in my opinion putting the name there gives more risk of breaking the layout than a real value to the user.
Also regarding the icon: should we create our own list of icons we can display? I'm asking because the situation is the same as with the app browser: we are not able to obtain the dapp icon.

CASE 2. The dApp name can not be fetched, dApp favicon/logo can not be fetched and displayed so we are replacing it with the generic label "DAPP" and generic icon.

The tags above the arrow are uppercased, so we are not able to display "dAPP". What do you think about using "DAPP"?
Also the "dApp" string displayed as the name is our own label which should be used when there is no name?

CASE 3. Unknown URL! Rare case that can still occur. URL cannot be fetched by Superhero Wallet. In this case neither the favicon nor the dApp name will be available. Most probably it's a risky app with basically "unknown source". User should be warned about potential risks.

Not sure if this is possible. We always have the access to the URL of the app that tries to connect. @CedrikNikita am I right?

Also I see following issues in provided designs:

I think reusing the transaction details header section is bad because it forces to display the DAPP data (name+address) in a narrow column which is problematic with long names and addresses.
The dapp name and address are displayed twice: both in the header and in the area beneath it.

As we have 100% control on the connection modals we can display those screens in totally different way than the transaction details. What do you think about following structure:

This is a working POC done with the code and it didn't require much time to do. Obviously this is only a concept and can be further updated.

One more issue raised by @peronczyk (not mentioned above) duplicate info:

I agree about this one we might get rid of:

About the fact we are currently not able to fetch the icon/favicon of any dApp - my suggestion will be to create and keep icons for a list of known and trusted dApps (like Superhero DEX) and use the generic dApp icon for all the rest.

About the label my suggestion was to make an exception of the capitalized letters in this case and use dAPP instead of DAPP to emphasize it's a "decentralized app" (it's usually written in this way). However it won't be an issue to use DAPP if making an exception is a problem.

About Case 3 (Unknown URL) - we can have such a case. The wording of the warning text (in yellow) is synced with @smaroudasunicorn

About your suggestion to change the layout: @smaroudasunicorn if you think it's a better idea to use the suggested alternative layout (mockup provided by @peronczyk) I'll update the design accordingly. I was trying to utilize what we already have as visualization for transactions and smart contract calls.

I agree with Bart Comments

I agree to change the layout of dapp connection instead of using the transaction modal one. Maybe to combine the above approaches above (keep the arrow i.e.)
I agree about icons
If we change the layout this will be removed. Even with this layout propose to remove the dAPP box. I agree to remove also DAPP name box.
I think depends on the caller if i am not mistaken about the URL...
As mentioned in the one, we could keep the visualization with the arrow and have something combined from Bart's proposal @onvisions
Regarding account change, we can have it but i do not consider it as MVP and would like to avoid more time needed to deliver.

OK, so now I'm waiting for updated designs. Here's working POC of my changes: https://github.com/superhero-com/superhero-wallet/pull/3385

@peronczyk @smaroudasunicorn
Updated dApp connection template
1. dApp name can be fetched

2. dApp name can not be fetched

3. Trusted dApp

4. Unknown URL = unknown source

Figma reference: https://www.figma.com/design/3oGLWzSH0oJljo4RETZtur/Superhero-Wallet-UI-(✔Updated)?node-id=37239-260033&node-type=frame&t=FMM3evEbkmvY73q9-0

The task is ready to be tested @Liubov-crypto
Regarding this part:

we decided to use the dapp icon instead of the warning, but colored in orange.
Modals changed:

Connect modal
Share address list modal

Affected features:

Account select items:  - please test if nothing has changed.

I found some discrepancies between Figma design and wallet:

different text
different buttons title
':' is present in wallet but missing in Figma design

As I understood we should show the short Dapp name, not to double url:

Different modal for known Dapp:

When I interacted with superhero dex there were some connection issues, I was disconnected and observed that our connection modal also has some changes. Is it expected in this case?

https://github.com/user-attachments/assets/b6f03b07-df92-48c5-a70b-6c36ce6e573e
@peronczyk please check it.

@onvisions  @peronczyk  One more question: do we decided to leave this screen as it is?
https://github.com/user-attachments/assets/aea4d0b2-51b2-441b-82c2-e3a62c50098c

I agree with what is mentioned above by @Liubov-crypto.
We can leave the ":" as implemented.
The Wallet Connect is not affected by this changes and is not within the scope of the task as far as I know.

2 more remarks on my side:

According to design the URL is always one line and horizontally scrollable (so the user can mark the text and see the whole URL or copy it in case they would like to).

From the screenshots I noticed that the account background is not matching the avatar as expected.

All the UI related fixes are there.
@Liubov-crypto

When I interacted with superhero dex there were some connection issues, I was disconnected and observed that our connection modal also has some changes. Is it expected in this case?

The disconnecting happens also on dev. But I don't fully understand this. The PR is about changing the connection screen.

When it opens as web wallet Propose Tx text is missing:

This is how the dapp asks for permission. We are not controlling this. Can you compare with dev?

All the UI related fixes are there.
@Liubov-crypto

When I interacted with superhero dex there were some connection issues, I was disconnected and observed that our connection modal also has some changes. Is it expected in this case?

The disconnecting happens also on dev. But I don't fully understand this. The PR is about changing the connection screen.

When it opens as web wallet Propose Tx text is missing:

This is how the dapp asks for permission. We are not controlling this. Can you compare with dev?

I tested with dev and there was no disconnection issue for me.

Would we like to show superhero icon here? or leave it as it is?

the rest is fixed, LGTM

Superhero.com and aescan added to trusted dapps. Can you retest @Liubov-crypto ?

I found that Dapp name is missing in Permissions Settings:
https://github.com/user-attachments/assets/dbf1e544-420d-4046-b99f-ad5bce4102b1
Dev:

@peronczyk please check it

Fixed @Liubov-crypto
Please retest.

LGTM


Statically link libstdc++ at least for musl builds

A lot of jdk docker images are
based on alpine without libstdc++ preinstalled. It is great to be able to use libasyncprofiler without having to modify the docker image that java is running on.
Please link libstdc++ statically but keep other dependencies like musl libc dynamically linked.
I can make a pull request if this is something that would be ok to have in all musl release builds, I am guessing something like this
g++ -shared -o your_library.so your_object_files.o -static-libstdc++ -Wl,-Bdynamic -lc

We're building AP with -static-libstdc++ on musl in our async-profiler fork used in gProfiler. For reference this is the change. Note that it greatly increases the size (300-400k -> 8mb).
As a generic solution we might want to have 2 separate AP builds, one static and one not static, because not all musl-based JVMs lack libstdc++, so we can use the appropriate one. (to reduce memory overhead & loaded code, etc).

As @Jongy noticed, static linking may significantly impact the library size - this is not something that can be done unconditionally. At the same time, I don't want to multiply the number of builds.
Ideal solution would be to get rid of libstdc++ dependency at all. This is doable but requires a fair amount of routine.

Would it not be possible to add the libstdc++.so to the jar file which already contains libasyncprofiler.so ? I bet there is a way to set LD_LIBRARY_PATH so it would only load this copy if libstdc++ is non-existent on the system, and libstdc++ should be known at build time which version of it is the compatible one, so it could be included in the jar. This way, there is no increase in memory usage that can be seen with static linking, but asyncprofiler is usable on stock docker images where libstdc++ is not preinstalled (most notably a lot of alpine based openjdk distributions). This or the rewrite where libstdc++ is removed completely (but I can understand if that is a larger task)

You can build libasyncProfiler.so statically linked with libstdc++ with no modifications to async-profiler source code by passing custom CXXFLAGS value to make command.
We have no plans to provide multiple builds of libasyncProfiler.so that differ only by used compiler/linker flags. At the same time we welcome changes that will decrease the number of produced binaries and thus improve user experience and reduce maintenance overhead. As mentioned above, one of the possible approaches to that is getting rid of libstdc++ dependency altogether.

Closing this issue in favor of #872.


Update de.json

Harmonized and corrected German translation.

@berndhose: Thank you for contributing to the project!  A couple of things:

If you're going to contribute, you need to follow the guidelines, which means opening a JIRA issue, first, and then tagging both the pull request and your commits with the JIRA issue (see the other open pull requests for examples).
#375 has been opened by someone else dealing with updating the German translation.  I suggest that it might be better to review that pull request and add your input on the updates suggested there to make sure they are accurate.

OK, my fault. I have reviewed the proposed changes of #375 and they seem to be ok, apart from one occurrence of the word "Passwörten", which should read "Passwörter".

Alrighty. I've copied over the comment above to #375.
@berndhose, if you find any other problems with #375, please feel free to comment directly there.


Compile "Official Build"

Flag is_official_build=true enables some optimizations and other Ungoogled Chromium distributions (Debian, Portable Linux) also enable this.
It should also be enabled for the Windows distribution.

Fixed with 103.0.5060.134-1.2


The link to the book is broken

Hi,
Noticed the link in the readme is not right anymore. Looks like all URIs are without "/sutton/" path now.
Many thanks.

Fixed. Thanks for pointing it out!


Basil 19 built in non returning functions

I don't think it really makes sense to put this at the level of the BAP to IR translation and loading the BAP representation from the parser? I'd prefer to keep the BAP AST to be a fairly direct representation of what we get from BAP, and keep transformations like this as separate passes over the IR. For instance, we may need to apply this sort of pass early on, but then also later again after resolving indirect calls?
This is also a shallow approach that doesn't seem like it will consistently deal with nested non-returning calls (where if f always calls g which always calls h which is non-returning, then f and g should also both also non-returning). We really want something more like the approach described in 'Binary Code is Not Easy' to handle those cases.
The result of the translation also seems like it's dependent on the order the procedure are iterated over, as it uses Procedure.nonReturning of other procedures that are called by the current procedure to set the returnTarget of DirectCalls, but sets Procedure.nonReturning for the current procedure. Parallelising that like you have seems like it would potentially be non-deterministic, which we don't want.

Oh, well you should probably mark this pull request as a draft then until it's ready then.
Where you've done it there isn't the IR stage, that's at the translation step from the BAP representation to the IR representation, and loading the BAP representation from the ADT parser. The IR stage is where the program is represented by the classes in the IR package. The pipeline is parser -> BAP representation -> IR representation -> Boogie representation -> final .bpl file. The CFG is created from the IR for the purposes of the analysis, and then the IR is modified based on the results of the analysis. It would be possible to do this by making use of the CFG or not, whichever works best.

I've updated the implementation to work similar the "Binary Code is Hard" article.
I've also added code to remove dead code after while (true) endless loops.

For the test indirect_call/gcc_O2, greet is incorrectly identified as non-returning.
For the test malloc_with_local3/gcc_O2, printCharValue is incorrectly identified as non-returning, and the only block removed is immediately following the call, instead of all inaccessible blocks as you would expect.

These were not working prior to my additions.

Looking at those cases more closely, this seems to be an issue with tail calls.

Bap also flags some functions with the noreturn attribute which isn't currently loaded from the ADT into the IR, it may be worth using that also.

That's a good find and we probably should incorporate that, but I'm not sure how useful it is to rely on, since BAP doesn't seem to add it for __stack_chk_fail which is an obvious case of an external non-returning call.

In Binary Code Is Not Easy, their algorithm for detecting non-returning calls (in Figure 8) says that if a function has a reachable return block or tail calls a returning function, then it is a returning function. The problem with indirect_call/gcc_O2 and malloc_with_local3/gcc_O2 is that tail calls aren't taken into account at all - is this something that will be addressed for #101?
This still isn't separating the deletion of unreachable blocks from the identification of the non-returning calls. You have just marked which blocks to delete at the same time as identifying non-returning calls, and then created a separate call to remove them. I'm sorry if this was unclear, but the motivation for not deleting the now-unreachable blocks until later was that we might require them later, so it wouldn't be safe to assume they are still unreachable at this later call.
This also still isn't removing all unreachable blocks either, as shown in  your basic_nonreturning_function_exit and basic_function_calling_nonreturning_function_exit examples.

In Binary Code Is Not Easy, their algorithm for detecting non-returning calls (in Figure 8) says that if a function has a reachable return block or tail calls a returning function, then it is a returning function. The problem with indirect_call/gcc_O2 and malloc_with_local3/gcc_O2 is that tail calls aren't taken into account at all - is this something that will be addressed for #101?

This is correct, tail calls, are not taken into account just yet, and will actually need to be a seperate process. This is becaue BAP will mark calls as non-returning when it sees a jump to a function rather than a BL instruction. indirect_call/gcc_O2 and malloc_with_local3/gcc_O2 were already marked as non-returning, because of BAP and how it handles tail calls.

This still isn't separating the deletion of unreachable blocks from the identification of the non-returning calls. You have just marked which blocks to delete at the same time as identifying non-returning calls, and then created a separate call to remove them. I'm sorry if this was unclear, but the motivation for not deleting the now-unreachable blocks until later was that we might require them later, so it wouldn't be safe to assume they are still unreachable at this later call.

I'm not sure you actually understand what is going on here, nor why blocks need to be marked as deleted. I'm also at a loss as to what exactly you want. Please explain!!
The purpose of deleting the blocks is to eliminate blocks that happen after a return call due to the complier producing stack frame instructions that are never executed. Hence why I mark them as deleted, then delete them as you requested in later parts of the process.

This also still isn't removing all unreachable blocks either, as shown in your basic_nonreturning_function_exit and basic_function_calling_nonreturning_function_exit examples.

Can you provide examples of this?

Just a note, Bap also flags some calls with the noreturn attribute which isn't currently loaded from the ADT into the IR, it may be worth using that also.

This is added into the IR using None as the returnTarget for those calls.

In Binary Code Is Not Easy, their algorithm for detecting non-returning calls (in Figure 8) says that if a function has a reachable return block or tail calls a returning function, then it is a returning function.

This is correct, however there is a diferentiation between a tail call is returning, and a tailing call that is non-returning. Meaning, that tail calls that are non-returning, are not counted as a returning block. You will also notice line 5 of FIgure 8, says continue finding non-returning functions until you have found no more. This is because, the change in the function's non-returning state changes the state of ofther function calling it.
The other point here is the use "reachable", if there is a return block, that is returning after a non-returning block, then return block is not reachable. Therefore, as part of the analysis, we need to check these for these occurances. Hence why soft deleting is required.
We discussed this at the beginning of the task, and it was decided to delete these blocks, I'm not sure why now we want to have the status of these blocks simultainously in two states, deleted and not there. Maybe you haven't been clear as to what you wanted. So please clarify as I'm at a loss of what to do next here.

I'm not sure what needs to be changed here, there isn't any real understanding of the reviewers, nor any idea of how it should be done. So I'll leave this with liam.


Misc. improvements and cleanups

Thank you for the feedback, pushed fixes.

 That resolves the major concern I found while reviewing the PSBT changes, although I'm still learning the PSBT code, so may have further comments in coming days.
Hopefully it goes without saying that you are welcome to cherry-pick, squash, etc. any of my commits as you see fit into your branch.

NACK on removing eccLib from p2tr.
Perhaps it is because there is a lack of support from libraries and lots of people are "rolling their own" for now... but I have seen examples of people sending to v1 segwit programs where the 32 bytes following is not a valid point (even though it is less than P).
Bitcoin Core does not reject these, and they are essentially a black hole.
(Perhaps a standardness rule could be added to Bitcoin Core to prevent their broadcasting/propagation)
At the very least, we should require it for the address module, and anyone trying to get the address from p2tr.
But the requirement of being a valid point is especially concerning in this case since a v0 p2wsh is also 32 bytes, but it's a hash.

If Bitcoin Core included a standardness rule, I might consider it.

I made an issue for discussion here

I would be comfortable removing eccLib once an official release with that standardness rule is over 60%-ish of the observable network...

hmm... looks like consensus is opposite (standardness rule should come after every client/library under the sun has protected against it)
https://github.com/bitcoin/bitcoin/pull/24106#issuecomment-1017498437

OK, I see your point.
I think more and more that we should drop the verifyEcc stuff though. Per a side chat, we're at least marginally concerned about performance, and those are rather expensive tests to be performing on each instantiation of p2tr. bitcoinjs-lib should either provide the ECC implementation, or accept that it may be injected with one that doesn't meet the expected requirements.
Given all of this, it might make sense to apply the Factory pattern to PSBT, make an ECC library required to use PSBT in general, and avoid the conditional/optional threading. That would then open up later refactors to replace types.isPoint with a proper point check for non-Taproot as well.

That is true. It's pretty expensive.
Perhaps we can cache eccLib...?
When comparing two objects, it returns true for equals when the object is the same reference.
We could use that and have some sort of eccLib cache that is rooted in the main library, and will use the cached library if none is passed in, and even if the same library is passed multiple times, skip the check if the reference to the library object is the same?
ie.
// eccCache.ts (do not export this in the index.ts, it should not be exposed to the public API)
export const _ECCLIB_CACHE: { eccLib?: TinySecp256k1Lib } = {};

// p2tr.ts
import { _ECCLIB_CACHE } from '../eccCache';

if (_ECCLIB_CACHE.eccLib !== eccLib) {
  check(eccLib);
  _ECCLIB_CACHE.eccLib = eccLib;
}

I think more and more that we should drop the verifyEcc stuff though. Per a side chat, we're at least marginally concerned about performance

this indeed can be a performance problem. One option is to add a skipEccValidation boolean flag to PaymentOpts:

export interface PaymentOpts {
  ...
  eccLib?: TinySecp256k1Interface;
  skipEccValidation?: boolean;
}

The caching approach would also solve the the "threading" of eccLib in PSBT.
But we can have the scenario where :

a p2tr(..., { eccLib: ecc }) call is made somewhere in the code
and later, in different parts of the code a p2tr(...) call is made (without eccLib) and it works (the eccLib is cached)
someone removes/adds a condition for the first call (with eccLib). Then the second call (without eccLib) fails

Same goes for new Psbt({ eccLib: ecc }) vs new Psbt()
Or... are we planning to use the cache only for removing multiple verify calls for eccLib?

We could do both.
Maybe even provide a public setter that will verify and set the cache so people don't have to use the weird options to begin with.
"Only insert this option once, any subsequent usage isn't needed" is a weird API.

Maybe even provide a public setter that will verify and set the cache...

Yes. I would prefer something explicit as in the code snippet below for several reasons:

if the eccLib is invalid, then it will fail at the initEccLib(...) invocation place (which can make it easy to investigate/debug), instead of failing inside p2tr or Psbt
if p2tr is used without an eccLib then the error message can clearly state that: "No ECC Library provided. You must call initEccLib(...) with a valid TinySecp256k1Interface instance"

side note: rename TinySecp256k1Interface to Secp256k1Interface?

it makes the code-base a little easier to work with (no need to pass eccLib around)
it makes the public API a little simpler (no eccLib option for p2tr and Psbt)
calling initEccLib(...) is optional (only required if p2tr is used)
calling initEccLib(...) multiple times with the same TinySecp256k1Interface instance is a noop

// client code
import { payments, address, initEccLib } from 'bitcoinjs-lib'

initEccLib(clientEccLib)

I'm in full support of an explicit initEccLib and then having p2tr throw a useful error if it's used prior to that being called.
@motorina0 would you like me to whip that up, or you?

That boat has already sort of sailed with bip32/ecpair tbh.
The answer would be to have a single file that imports bitcoinjs, bip32/ecpair, and tiny-secp256k1, mashes them together, and re-exports the eccLib-included objects. Other files would then import from that one file.

those deps do not have an init call.

They kind of have (its a factory pattern):
import BIP32Factory from 'bip32';
import ECPairFactory from 'ecpair';
import { payments, address, initEccLib } from 'bitcoinjs-lib'
import * as ecc from 'tiny-secp256k1';

const bip32 = BIP32Factory(ecc);
const ECPair = ECPairFactory(ecc);

// suggested change:
initEccLib(ecc)
...

Having a side-effecting call plus extra arguments is a strange mix.

it is either the initEccLib OR the extra argument ({ ..., eccLib?: TinySecp256k1Interface }). It does not make sense to have both
I don't know if having an init() method is a "side-effect" call, but I get your point. It its more like "enable extra functionality". The lib itself is stateless.
the "extra argument" has the disadvantage of having to "thread" eccLib (as @brandonblack  correctly observed for Psbt)
the init() has the disadvantage that the client application must make sure to call it at least once.
other options... ideas are welcomed

@brandonblack I would like to get these changes in the p2tr-v1 branch. Should I wait for the Psbt review, or is it OK for you to open a new PR?

Fine with merging this as is. Still getting my head around PSBT, but hopefully others can continue reviewing in the meantime.

Yeah merge this into the p2tr-v1 branch for now. We can continue discussion there.

PR merged! Thanks again @brandonblack
I have open this PR https://github.com/bitcoincoretech/bitcoinjs-lib/pull/5 against p2tr-v1 with the initEccLib chages.
Please have a look (and comment) before I merge it.


Vendor dependencies with godep.

Because the new infux client is completely incompatible with our code.

gosh, and since the influx client is inside their main repo we're vendoring the whole thing :pensive:


Upgrade Browserify/Babelify

Should be pretty simple:

Upgrade Babelify to 7.2.0
Install babel-preset-es2015
Add the following snippet to the task, in lieu of the current transform call:

b.transform(babel, { presets: ["es2015"] });

Not sure what we benefit from this, but it's worth a shot. Startup time seems to take a hit, but after the first run it's pretty quick. May be worth delaying until the point that Browserify starts to see major improvements for things like module resolution.

Probably going to hold off on this, at least for the Babelify stuff, since this jams our disk usage up from ~50MB to over 100MB, most of which comes from babel-preset-es2015. We'll revisit this when Grunt stable hits 1.0.0.


Add integration testing for uncovered APIs

fixes #1074

Based on the PR https://github.com/eclipse-xpanse/xpanse/pull/1463.

Based on the PR https://github.com/eclipse-xpanse/xpanse/pull/1463, merge it after the PR #1463 is merged.


Use mongodb kotlin bson implementation

I noticed there is a kbson implementation inside the mongodb organisation as well, seems like the latest version 0.2.0 is also released to maven central.
Maybe it's a good idea for KMongo to use this instead of the kbson implementation it uses currently, because the one in the mongodb organisation might be more "official"?
Link to the repository: https://github.com/mongodb/kbson

There will be soon an official mongo driver for mongodb - stay tunned ;)


Fix setup for development link step.

This was a typo as agreed upon at https://github.com/tensorflow/tensorflow/issues/2591

Can one of the admins verify this patch?


Error

There is a small issue with the Layout graphic.  The Power Switch and the Mid Muscle Electrode Snap are backwards.  They just need to change positions with each other.

This has been fixed. Thanks!


aio app test is failing on EBADF bad file descriptor (Windows only)

Expected Behaviour
On Windows i expect to be able to test my firefly application by running the command aio app test
Actual Behaviour
Crash
`D:\code\2021\git\demos\gettingstarted>aio app test
child_process.js:114
  p.open(fd);
    ^

Error: EBADF: bad file descriptor, uv_pipe_open
    at Object._forkChild (child_process.js:114:5)
    at setupChildProcessIpcChannel (internal/bootstrap/pre_execution.js:328:30)
    at prepareMainThreadExecution (internal/bootstrap/pre_execution.js:49:3)
    at internal/main/run_main_module.js:7:1 {
  errno: -4083,
  code: 'EBADF',
  syscall: 'uv_pipe_open'
}
 »   Error: Command failed with ENOENT: jest --passWithNoTests ./test
 »   spawn jest ENOENT`

Reproduce Scenario (including but not limited to)
aio app init getting started
Adobe IO Demo org
default options the rest of the way
Steps to Reproduce
Environment Info
  System:
    OS: Windows 10 10.0.19042
    CPU: (8) x64 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
    Memory: 23.78 GB / 47.91 GB
  Binaries:
    Node: 12.13.0
    Yarn: 1.22.5
    npm: 6.12.0
  Virtualization:
    Docker: 20.10.2
  npmGlobalPackages:
    @adobe/aio-cli: Not Found

Sample Code that illustrates the problem
Logs taken while reproducing problem

also add aio app test to the e2e tests for the aio-cli, and add the e2e tests to the Github Actions.


Transformer not working in v4 strapi

Hello! Thanks for the plugin!
I'm trying to use transformer in a strapi v4 app. When calling the endpoint the data is not transformed.
I followed the documentation and it still didn't work
my ./config/plugins.js
module.exports = ({ env }) => ({
 'transformer': {
    enabled: true,
    config: {
      prefix: '/api/',
      responseTransforms: {
        removeAttributesKey: true,
        removeDataKey: true,
      }
    }
  },
});

my package.json
    "@strapi/plugin-graphql": "^4.2.0",
    "@strapi/plugin-i18n": "4.2.0",
    "@strapi/plugin-users-permissions": "4.2.0",
    "@strapi/strapi": "4.2.0",
    "pg": "8.6.0",
    "strapi-plugin-ckeditor5": "^2.1.1-rc.1",
    "strapi-plugin-transformer": "^2.0.2"

Same here. Updated from v4.1.8 to v4.2.3. Attributes and Data keys are back in the json response

Thanks for the reply. I do not use cache plugin. The only plugin I use is the transformer plugin.
I soured the build cache as well. I still get the json with the data and attribute keys

And the plugin gets recognized correctly, if you check the Plugins Page in the Admin Panel?

My panel

Hmm strange as far as I know no major changes were released that should break the plugin.
Do you have a sample repo I can look at? Did you change any api settings?

The same issue in the Strapi 4.2.3, only default plugins installed.

Sale 4.3.0 here

@ComfortablyCoding
https://github.com/fernando23depaula/strapi-cms

Thanks, will have a look.

The provided example does not look to be public anymore. I just tested it myself on a fresh install of the latest v4 and the plugin is working as expected.


Add features for popular data structures

https://crates.io/keywords/data-structures
https://lib.rs/data-structures
https://crates.io/crates/smallvec
serde_json
bytes
beef

Syn would be interesting, but Hard

Also rustdoc::Clean


Add option to defer by one hour

This gives an option to defer a task by one hour. The code is pretty much a copy of deferDay.

I will add in the description that you did the modification to Due Selected Tasks script. And I like the option to defer an hour proposed here. Thank you.


Jobs.waitUntilDone() does not wait forever when 0 ms is specified

Jobs.waitUntilDone() waits 0 ms when a 0 ms timeout is specified, not infinitely, as the documentation says. Either JobExecutor.waitUntilDone() should be fixed, or the documentation for Jobs.waitUntilDone() updated.

There are two implementations of this waitUntilDone:

JobQueues (used by local and SSH)
SchedulerConnection (used by slurm, sge, etc).

Unfortunately, they are not consistent with each other nor with the documentation.
Documentation says:   timeout >= 0, where 0 is wait forever
JobQueues: timeout = 0 is return immediately, and negative results in exception
SchedulerConnection: timeout < 0 is return immediately, timeout == 0 is wait forever

waitUntilRunning has the same problem

Many ways to fix this. Some ideas:

timeout <= 0 is return immediately. No shortcut to wait forever.
timeout == 0 is wait forever, timeout <= 0 is return immediately
timeout < 0 is wait forever, timeout == 0 is return immediately

Nice thing about the first option is that the semantics are simple and clear. The other two seem to be an arbitrary choice, an I can never remember which one is taken (as the implementations nicely illustrate).
Opinions ?

One more comment on this. The Thread.wait(long timeout) has the following semantics:

timeout == 0 is wait forever
timeout < 0 results in IllegalArgumentException

So this more or less matches our documentation (although we are not clear about the exception).
Will fix this to use these semantics.

I propose we follow Java convention (and the intended behaviour at the moment I think):
From Thread
public final void join(long millis)
                throws InterruptedException

Waits at most millis milliseconds for this thread to die. A timeout of 0 means to wait forever.

This implementation uses a loop of this.wait calls conditioned on this.isAlive. As a thread terminates the this.notifyAll method is invoked. It is recommended that applications not use wait, notify, or notifyAll on Thread instances.

Parameters:
    millis - the time to wait in milliseconds
Throws:
    IllegalArgumentException - if the value of millis is negative
    InterruptedException - if any thread has interrupted the current thread. The interrupted status of the current thread is cleared when this exception is thrown.

Also, we should have a few unit tests for this behaviour :-)

Yup. Fixes committed to hotfix-wait branch. Will add unit tests now.

That's going to be a really long test ;-)

My point exactly :-). No one really wants to wait forever, that option is just a convenient way of avoiding some error handling that shouldn't be avoided. But I agree that if this is how it's done in Java, then Xenon should do the same. Principle of least astonishment and all...

I've cunningly avoided this problem by adapting to Javadoc to state that "the timeout will be ignored when 0, and the method will wait until the job is done / running"

.... cause who wants to wait forever ?
(cue Queen soundtrack ;-)

Now where's my eScience sword? There can be only one! (At least when programming in Python, due to the global interpreter lock.)


Refactor usage of creation pattern

Cascade of #37
I.e., bulk out concrete __init__ with args (func sig) rather than just passing a single config.

When I started doing this, it quickly occurred to me that we might not be using pedantic pydantically.  Instead of writing normal python classes that have pydantic model configs, it would be more "pydanically correct" to just make all classes pydantic models.
@amitschang and I discussed this briefly offline and are in agreement.
I'm going to go ahead and do this now to resolve this issue as a lot of the creation code evaporates as we can fully leverage pydantic.

Well... actually maybe not, giving it another 2 sec of more thought, there's stuff, i.e., locks etc that might end up adding more work since not everything needs to be a part of the config.


[Feature Request]: rerank onnx

Feature Description
It would be nice to allow onnx models for reranking, as the inference of pytorch with cpu is very slow.
Reason
onnx is faster on cpu and allows to use a bigger llm.
Value of Feature
Onnx makes cpu brrr

I fiddled a little bit around and got it working myself. How can I commit this?


pull_request_target is almost always used insecurely

pull_request_target is more secure than pull_request, which is why it gets access to secrets. There's nothing "fundamentally insecure" about it.
Can you elaborate on this?

Hi @ljharb, thanks for filing an issue.
The docs (https://woodruffw.github.io/zizmor/audits/#dangerous-triggers) have some sources for this, but to summarize:

pull_request_target (and workflow_call) are somewhat unique in that they're often triggered by third-party users, but have the same context as their upstream repo. That means that they have secret and other context access for the upstream repo, rather than the repo sending the PR. There are other triggers that run in "upstream" contexts, but most can only be triggered by that upstream itself (e.g. push, since pushing to a repo already implies a degree of access to it).
On its own, this is not an issue. However, users consistently misuse pull_request_target (and workflow_call) in workflows in ways that allow someone to achieve code execution in the upstream context just by sending a PR.

The "classic" example of this is documented in a 2021 post from GitHub's security lab:
on:
  pull_request_target

jobs:
  build:
    name: Build and test
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      with:
        ref: ${{ github.event.pull_request.head.sha }}

    - uses: actions/setup-node@v1
    - run: |
        npm install
        npm build

The upstream's intent here is (typically) just to install some dependencies. However, because they've checked out attacker-controlled code into the upstream context, npm install is fully controlled by the attacker instead, and can exfiltrate the default GH_TOKEN, any members of secrets.*, etc.
TL;DR: pull_request_target runs in the parent repo context but, unlike most parent repo triggers, can be triggered by external users as well. This enables footguns around attacker-controlled code in the parent context that alternative triggers (like pull_request) don't allow.
(To my understanding, GitHub is aware of this footgun and considers it a security issue, per the GHSL post above and warnings in the docs. However, they aren't interested in removing or neutralizing pull_request_target because they consider it the user's responsibility to use it correctly. Hence zizmor flagging on it.)

How can i override that warning, then, given that the entire point of pull_request_target is to have access to secrets, and that i am using it securely?

How can i override that warning, then, given that the entire point of pull_request_target is to have access to secrets, and that i am using it securely?

You can't yet, at least not directly within zizmor itself -- I need to add support for some kind of ignoring mechanism per #12. If you're using the SARIF integration, then you can use GitHub's UI to dismiss the finding, however.
Out of curiosity, can you share the workflow that you think is using pull_request_target securely? IME with security reviews, anything that uses it beyond its most locked-down case (no code execution, just used to send comments/change issues) tends to get something wrong with it.
Longer term, it's possible that I could refine the dangerous-triggers audit to be more precise about pull_request_target. The reason that hasn't happened yet is because I'm currently erring more towards false positives than false negatives, since this was originally designed primarily as an auditing assistant.

Email, Twitter, whatever you like - I’m ljharb on everything.

The issue should probably still be open, though, because I need a way to not warn on the allow edits workflow, at least.

Sure!

Thanks for the links. require-allow-edits.yml looks fine (and is a good example of how pull_request_target can be used safely),

But can zizmor be changed to detect these sorts of workflows? Ones without substitutions or user-induced input should be safe (IIUC).
I don't want to ignore the check; I want zizmor to tell me concretely if it becomes insecure.

(Sorry for the wall of text that follows.)

But can zizmor be changed to detect these sorts of workflows?

In the long term yes, but not at the moment. There are two big limitations (which I'm working on/thinking about how to accomplish) that prevent this currently:

All audits are currently independent of each other, i.e. dangerous-triggers doesn't know about template-injection or vice versa. Long term audits should be able to declare dependency relationships on each other, and use a previous audit's results to decide whether or not to emit findings of their own.
zizmor currently audits only workflows, not the contents of the actions they call. Once we're able to do the latter, (1) will compose nicely so that we can express things like "only warn about a dangerous trigger if the workflow itself looks fine and every action it calls looks fine."

These are pretty long term goals, however, and I'm not sure if there's an actual stable generality here yet -- many of zizmor's analyses are imprecise because they have to jump between YAML, bash, JS, etc., which means that I've erred on the site of emitting false positives for "very hard to use correctly" things like pull_request_target.

Ones without substitutions or user-induced input should be safe (IIUC).

For the most part, although there are caveats here:

Any pull_request_target or similar that modifies the environment via GITHUB_ENV is susceptible to code execution, if the attacker can manipulate the environment variable names/values to some degree.
It's hard to describe "input" to a given workflow with full generality, because of how dynamic they are at runtime. For example: a pull_request_target might not "take" any input, but it might pull data from an upstream or a fork based on some event state. That data won't show up as an explicit ${{ some.context }} reference, but it could be attacker-controllable in dangerous ways. GHSA-5xr6-xhww-33m4 is kind of an example of this (not exactly since it's a different kind of ambiguity, but it demonstrates how GitHub's internal "provenance" of information diverges from how users think about user/repo/workflow identities).
More broadly, public understanding of GHA's attack surface is still pretty young -- template injections are well understood, but the dangers of GITHUB_ENV weren't super well understood until last year and cache key exfil/poisoning is new to this year. So more generally, I don't feel comfortable asserting that zizmor knows that a given pull_request_target-driven workflow is safe, given that people keep finding new ways to punch holes in them :slightly_smiling_face:

TL;DR: I have plans to make this more precise, but it's not trivial!


refactor broadcastNetwork

Extracting createAndConnectNode, as @mrBliss suggesteed on #773. I'm doing it separately, because having that commit on that branch makes it difficult to rebase onto master.

I took the liberty to drop the commit "add missing other-module" (2256cf657c4e707ab9c83a7090af4aab3b5078d4) from this branch because it was not strictly necessary for this PR.

bors r+

bors r+


How the autoencoder in the example project bind input data as the label data

I've read the code in the autoencoder, but I couldn't figure out how it binds the input data as the label data. The autoencoder needs to share weights between stacks and finetuned net, so the training process is done manually with codes, this example is more complicated than others, hope there could be a readme file to help understanding.
@piiswrong

there is no difference between data/label in executor. The difference is in FeedForward model. If you are writing your own training loop you can use the same input as data and label

I figure it out, it binds the label data when it construct the net, guess I just miss it.


Update azure-sql-python-quickstart.md

remove word install before pydantic

@bobtabor-msft
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

@Oury-MSFT
PRMerger will only accept sign off from the listed 'author'. However, if you are a Microsoft employee authorized to update this content, you can send a request to<EMAIL_ADDRESS>for this PR to be merged.  Include your role, and an explanation of why the PR should be merged without approval from the listed author.

Checking with @David-Engel to see if we can just merge this.

Closed as duplicate of https://github.com/MicrosoftDocs/sql-docs/pull/9273.


Pylance incorrectly complains about inner-class reference even when postponed evaluation of annotations is enabled

 Environment data

Language Server version: 2022.10.20
OS and version: linux x64
Python version (and distribution if applicable, e.g. Anaconda): 3.10.6
python.analysis.indexing: false
python.analysis.typeCheckingMode: basic

 Code Snippet
from __future__ import annotations
from typing import TypedDict

class MyClass:
    class SomeIdFieldDict(TypedDict):
        some_nested_field: str

    class SomeDict(TypedDict):
        id: SomeIdFieldDict | None

def f(x: MyClass.SomeDict) -> None:
    print(x)

if __name__ == '__main__':
    f({"id": {"some_nested_field": "a"}})
    f({"foo": "bar"})

 Repro Steps

Install "Python" extension v2022.16.1 in Visual Studio Code
Open the file with the above code.

 Expected behavior
Pylance correctly resolves a reference to SomeIdFieldDict type thanks to the "postponed evaluation of annotations" functionality enabled by importing from __future__ import annotations (see PEP 563).
The code itself works, by invoking python test.py with the above code put in test.py file - the output is:
{'id': {'some_nested_field': 'a'}}
{'foo': 'bar'}

When the __future__ import is removed, the following error expectedly occurs:
Traceback (most recent call last):
  File "/.../test.py", line 4, in <module>
    class MyClass:
  File "/.../test.py", line 8, in MyClass
    class SomeDict(TypedDict):
  File "/.../test.py", line 9, in SomeDict
    id: SomeIdFieldDict | None
NameError: name 'SomeIdFieldDict' is not defined

 Actual behavior
Pylance complains with reportUndefinedVariable error that "SomeIdFieldDict" is not defined:

This isn't an issue of "postponed evaluation". After all, the SomeIdFieldDict symbol is defined before it is used in this case.
This is an issue of scoping rules and symbol resolution. In Python's scoping rules, a symbol defined in a class scope is not visible (i.e. not captured by) an inner class or function defined within that same outer class.
You can see this if you attempt to do something like this:
class Foo:
    class A: ...

    class B:
        x = A() # Error: A not defined

Since SomeIdFieldDict isn't visible from that scope, you can't access it by simple name. If you were to use this in a runtime type checker (which would probably call typing.get_type_hints()), it would receive an exception trying to evaluate this symbol in a postponed manner.
If you want to use SomeIdFieldDict in a type annotation at this location, you need to refer to it by its qualified name starting with MyClass, which is visible from the inner scope of SomeDict.
    class SomeDict(TypedDict):
        id: MyClass.SomeIdFieldDict | None

So, this isn't a bug in pylance. This is a bug in your code, and pylance is correct to flag it as an error.

Ok that's clear now, thanks for taking time to explain this. I'll close the issue.


Try it out

Can you please add "Try it out" posibility like it is in Swagger UI?

@raderio this is a dup of https://github.com/Rebilly/ReDoc/issues/53

Try redoc-try, use swagger Try it out in redoc project.

There is currently no npm library released, I feel it is unnecessary. :)


Added thread factory so we can set thread-name

This is useful if we want to set the thread-name (for example) on the pub-sub worker threads.

I have no idea how to fix this because of the AutoValue crap.  The error says:

Method without arguments should be a build method returning com.google.api.gax.core.InstantiatingExecutorProvider or a getter method with the same name and type as a getter method of com.google.api.gax.core.InstantiatingExecutorProvider
public abstract ThreadFactory getThreadFactory();

I do have a method with the same name/type in InstantiatingExecutorProvider.  Does it not like the fact that it is abstract?  I wanted to make it concrete so it wouldn't break others.


Web-exported game - Error: Could not load game data at path '.'. Is the .pck file missing?

Godot version
3.3.2 stable
System information
Windows 11
Issue description
When exporting the game for web and running, I get:
Error: Could not load game data at path '.'. Is the .pck file missing?

Steps to reproduce
install latest emsdk, scons, and git godot master, custom.py: no 3d, optomize size, ...
Command build:
scons platform=javascript tools=no threads_enabled=yes target=release
Minimal reproduction project
No response

@nguyenkhoa222
You should upload it to the actual web server or use the button in Editor for HTML5 testing export, because it won't work locally, just by opening the generated HTML file in web browser, because of security policy i think.
You can also try this button in Editor itself, which starts a webserver for you on localhost address


Image support

Hi @Iikeli
Trust you are doing good.
I want to build an application that needs to display a markdown in a cell,
in my quest I found your library...
I see it needs enhancement and would love to contribute to the library...
Any guidelines, I need to know!?

Hi @wptechprodigy as long as you structure the code well, write tests where useful, and add documentation, all is good.
I did notice a while back that Apple also now has a markdown library. I have not yet played around with it myself, so don't know if it will solve your problems. https://github.com/apple/swift-markdown

Thanks @Iikeli...I'll check it out!

Did this enhancement get off the ground (e.g. on a fork)? Looking for a viable SwiftUI markdown view that handles images. MarkdownUI (https://github.com/gonzalezreal/MarkdownUI) looked promising but is hanging/crashing in certain circumstances for us.

Hey @tsfischer
This has not been implemented yet. Do you want to look into the implementation or should I invest some time this week?

If you're able to, that would be awesome. I have not heard back from the owner of MarkdownUI on the issue we're having there, and it seems like this library might be the next best alternative. But we would really like to have image support.

@tsfischer I'll take a look at it during the weekend or early next week. Hopefully can add that functionality for you guys quickly.


A simple alternative to widgets.

One of the complaints we have had from users is the lack of support for Jupyter widgets.
This change addresses some of that limitation by allowing the variables in %%sql modules
that are displayed with %%chart to include simple dynamic controls which will re-execute
the query with the new values when changed (with client-side caching of past results).
This is controlled via the YAML chart options, in a 'variables' section. There are five
types of controls supported, 'text' (textbox), 'set' (checkbox group), 'picker' (dropdown
allowing a single value), 'slider' (combined slider/textbox for numeric value with
min/max/step parameters), and 'boolean' (a checkbox for a Boolean value).
Each can have an optional 'label' property that defaults to the variable name, and a
'value' property to set the initial value, which, if not present, defaults to the default
value in the %%sql cell. The control type is set by a 'type' property which will default to
'text' for strings and 'set' for lists but must otherwise be specified.
For sliders, if s 'step' property is not specified and 'min' and 'max' are ints a 'step' of
1 will be used; else a step of (max - min) / 10.0 will be used.
For pickers, if the variable is not a list, a list of values can be provided with the
'choices' property.
Some examples:
%%sql -m mother_ages

state=['WA', 'TX', 'AZ']

SELECT mother_age, COUNT(*) AS count FROM [publicdata:samples.natality]
WHERE state IN $state
GROUP BY mother_age
ORDER BY mother_age

%chart paged_table -d mother_ages
variables:
  state:
    label: 'State'

The above will create a set of three checkboxes which can be checked/unchecked in different
combinations to affect the resulting table via altering the IN clause of the query.
%%sql -m total_for_age_by_state

age=20

SELECT state, COUNT(*) AS count FROM [publicdata:samples.natality]
WHERE mother_age==$age
GROUP BY state
ORDER BY state

%chart table -d total_for_age_by_state
variables:
  age:
    type: slider
    min: 10
    max: 60

The above will create a slider to allow an age in the range 10 through 60 to be selected.
%%sql -m total_by_state
state = 'WA'
married = True

SELECT mother_age, COUNT(*) AS count FROM [publicdata:samples.natality]
WHERE state==$state AND mother_married==$married
GROUP BY mother_age
ORDER BY mother_age

%chart columns -d natality
title: Births by Age and Marital Status of Mother
hAxis:
  title: Age
variables:
  state:
    type: picker
    choices: [AZ, TX, WA]
  married:
    type: boolean

The above will create both a picker for the three choices of state and a checkbox to toggle
the value of martial status.

It seems this might be even better if it were generalized a bit... example:
%%form
config to create pickers etc. and associated python variables in the notebook

%%sql and %%chart would use those as any other variable.

Regarding generalizing these, I have mixed feelings. It seems to me that there is a degree of coupling between a chart and the type of controls that is tighter than between the query and the control.
OTOH I see some benefit to making our variable definitions in %%sql cells richer, which could allow us to in most cases simplify what we need to specify in %chart. E.g. like we have source/datestring functions for some variables we could have functions for each of these controls in %%sql cell (not specifying the type, necessarily, but - say - specifying min/max/step values for variables that fall in a range). Then %chart would not need to specify these.
We could also infer these if the user doesn't specify them explicitly. E.g. a var that is just assigned an int could be inferred to be a textbox typed to int values - and we could automatically add controls for all non-overridden variables in %chart. So going back to the first example above we wouldn't need to specify anything extra in %chart to get the control. Similarly for the second example if we had declared in the %%sql something like:
age = range_var(10, 60, default=20)
In the third example, without specifying anything we could get a text box for 'state' and a checkbox for 'married'.
That seems to allow us factor out much of the behavior that may be coupled to the %sql (the choice of values, ranges, etc), while leaving the choice of control type (if not a sensible default) to %chart.
Thoughts?

I am not suggesting to put this into a SQL cell. Just making sure. I am
suggesting creating a new %%form cell, and having widgets to be its output
rather than in the chart. With better notebook ux these would be elevated
to a notebook level sidebar or something else rather than lost in the
vertical scroll.
I think there is some overlap with args in a SQL cell. I think it would be
a good time to think how args, form etc generalize to notebook level
arguments... esp. as we build out deployment from a notebook.
On Dec 21, 2015 7:54 PM, "Graham Wheeler"<EMAIL_ADDRESS>wrote:

Regarding generalizing these, I have mixed feelings. It seems to me that
there is a degree of coupling between a chart and the type of controls that
is tighter than between the query and the control.
OTOH I see some benefit to making our variable definitions in %%sql cells
richer, which could allow us to in most cases simplify what we need to
specify in %chart. E.g. like we have source/datestring functions for some
variables we could have functions for each of these controls in %%sql cell
(not specifying the type, necessarily, but - say - specifying min/max/step
values for variables that fall in a range). Then %chart would not need to
specify these.
We could also infer these if the user doesn't specify them explicitly.
E.g. a var that is just assigned an int could be inferred to be a textbox
typed to int values - and we could automatically add controls for all
non-overridden variables in %chart. So going back to the first example
above we wouldn't need to specify anything extra in %chart to get the
control. Similarly for the second example if we had declared in the %%sql
something like:
age = range_var(10, 60, default=20)
In the third example, without specifying anything we could get a text box
for 'state' and a checkbox for 'married'.
That seems to allow us factor out much of the behavior that may be coupled
to the %sql (the choice of values, ranges, etc), while leaving the choice
of control type (if not a sensible default) to %chart.
Thoughts?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/datalab/pull/726#issuecomment-166499374
.

Bradley, regarding YAML help, I think we would want to do this via samples. The YAML can include all the options from Google Charts so trying to add inline help for YAML is problematic.

Nikhil, one of my concerns with moving them out and then treating them as implicit variables is that we are breaking the current model where all variables are declared locally (unless they refer to other 'modules' like queries and UDFs). We're starting to have a mixed model - which begs the question of whether we should just have a %%parameters section with a notebook-scoped set of variables (a discussion we had before and decided against).
The other issue with making the controls separate - either in a cell (which raises all sorts of usability issues if the control cell and the affected cell are not both in scroll view, as you note) or later elevating them to a sidebar -  is that notebook execution gets more complex when a control is changed. Changing a control can affect multiple cells and we need to find and update all of these, but some may not have been executed and/or execution may be in progress. Perhaps that isn't too bad but it seems it could get complicated. Currently with the controls being local to a cell they only appear when the cell gets executed and the event handlers get bound at the same time; if you clear the cell or start re-executing it they are gone. This keeps things pretty simple.
My suggestion is to keep this as is, as it has value, is simple, and does not break existing things and to deal with the broader issue of separate controls (and dashboards) separately, in conjunction with an updated notebook UX (and perhaps then the answer will be a set of global variables that can populate a sidebar and/or a dashboard).

Samples are fine if they cover all the options. Maybe create a notebook to make sure all options are described and some are in demo? Maybe we can create such notebook and check it in, and then provide a github link in the helper message of %chart?

What's keeping back the support for Jupyter widgets? Is it an issue in their code?

Jupyter widgets rely on a communication channel that doesn't exist in
Datalab.
On Fri, Jan 15, 2016 at 11:38 AM, Aaron Toth<EMAIL_ADDRESS>wrote:

What's keeping back the support for Jupyter widgets? Is it an issue in
their code?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/datalab/pull/726#issuecomment-172063439
.

2 years later, is it supported? I see #1689 suggest that it is supported now, but I had the same issue as here when trying to do it, so am curious if it is and I'm doing it wrong.

@yebrahim? Was a new release made after the fix?

Yes, the widgets should be working in the latest version of Datalab.
@selwyth can you open a new issue and post the error you're getting, as well as the version of Datalab you're using?

Posted to #1980


Undefined symbol error when trying to import tensorrt in dustynv/l4t-pytorch:r36.4.0 container

Hi!
I have recently started working in a docker container on Jetson AGX Orin 64GB. I started with TensorRT 8.x but then I decided to upgrade to 10.x to hopefully take an advantage of newer optimizations. I tried to follow the steps here and using one of your newer containers: dustynv/l4t-pytorch:r36.4.0 but both resulted in the following error:
root@tree:/# python3 -c "import tensorrt" Traceback (most recent call last):   File "<string>", line 1, in <module>   File "/usr/local/lib/python3.10/dist-packages/tensorrt/__init__.py", line 75, in <module>     from .tensorrt import * ImportError: /lib/aarch64-linux-gnu/libnvinfer.so.10: undefined symbol: _ZN5nvdla19IConcatenationLayer7setAxisEi
I tried to resolve it myself for quite some time, however, I was unsuccessful. These are the things I have tried/notices:

This happens only for dustynv/l4t-pytorch:r36.4.0, if I use earlier version dustynv/l4t-pytorch:r36.2.0 I can easily import tensorrt.
When following the official steps or installing official jetpack nvcr.io/nvidia/l4t-jetpack:r36.4.0 I first get a different error:
root@tree:/# python3 -c "import tensorrt" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.10/dist-packages/tensorrt/__init__.py", line 76, in <module>  from .tensorrt import * ImportError: libnvcudla.so: cannot open shared object file: No such file or directory
However, this can be resolved by modifying LD_LIBRARY_PATH such that it includes/usr/local/cuda/compat. Once I do it I get the same error as above.
I have found this thread  with a similar error and followed the steps suggested by @dusty-nv, i.e. checking the drivers with root@tree:/# ls -ll /usr/lib/aarch64-linux-gnu/tegra/libnvdla* which resulted in:
-rw-r--r-- 1 root root 7487576 Aug 28 09:03 /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so
-rw-r--r-- 1 root root 6480496 Aug 28 09:03 /usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.soand checked the csv file in/etc/nvidia-container-runtime/host-files-for-container.d/, which did not contain /usr/lib/aarch64-linux-gnu/tegra/libnvcudla.so`.
Given the previous two points I thought that there is something wrong with DLA installation so I tried to reinstall CUDA using steps listed here , but for cuda-12.6. Interestingly I could install cuda-toolkit this way but cuda-compat threw the following error:
root@tree:/# apt-get -y install cuda-toolkit-12-6 cuda-compat-12-6 Reading package lists... Done Building dependency tree... Done Reading state information... Done cuda-toolkit-12-6 is already the newest version (12.6.1-1). Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: cuda-compat-12-6 : PreDepends: nvidia-l4t-core but it is not installable E: Unable to correct problems, you have held broken packages.
One more suspicious thing I noticed is that after starting the dustynv/l4t-pytorch:r36.4.0 docker (I ran all dockers using docker run -it --rm --net=host --runtime nvidia dustynv/l4t-pytorch:r36.4.0) I cannot find tensorrt installation by using dpkg -l | grep nvinfer or dpkg-query -W tensorrt, the only way I can find it is through pip3 show tensorrt. If this is expected behaviour could you elaborate?

Orin runs on, Ubuntu 20.04 and has CUDA 11.4.
I would be very grateful for help with this issue!
Best,
Jakub

Orin runs on, Ubuntu 20.04 and has CUDA 11.4.

Hi @J4Q8, I think your issue is related to trying to run a JetPack 6 container on JetPack 5, and would recommend to reflash the system with JetPack 6.1 / L4T r36.4 if possible.  Actually I am surprised that the r36.2 image worked for you, under some circumstances things like that may work and others not.

I will try that, thanks for quick response!

I reflashed Orin with JetPack 6.1 and the error is no more. Thanks!


How to fix "hxcodec/vlc/VLCBitmap.hx:144: characters 23-38 : Type not found : LibVLC_Instance"

Describe your problem here.
I need help how can i fix this error? : hxcodec/vlc/VLCBitmap.hx:144: characters 23-38 : Type not found : LibVLC_Instance
I'm using Psych Engine 0.6.3
Are you modding a build from source or with Lua?
Source
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
no

0.6.3 is not supported anymore since December 2023


[wip] Mute install logs from @now/node builder

We need to release a new version of @now/build-utils with https://github.com/zeit/now-builders/pull/338, before this change can actually works.

Codecov Report

Merging #2171 into canary will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           canary   #2171      +/-   ##
=========================================
- Coverage    4.42%   4.42%   -0.01%
=========================================
  Files         243     243
  Lines        8670    8673       +3
  Branches      906     906
=========================================
  Hits          384     384
- Misses       8269    8272       +3
  Partials       17      17

Impacted Files
Coverage Δ

src/commands/dev/lib/dev-builder.ts
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23d4434...9d07a1f. Read the comment docs.

@amio 338 has been merged and released as<EMAIL_ADDRESS>


hello,why when i am install this ext,it's warn me requires ext-mongodb > 1.0.0 and i has install the driver

What steps will reproduce the problem?
hello,why when i am install this ext,it's warn me requires ext-mongodb > 1.0.0 and i has install the driver
What's expected?
i has install ext-mongodb and php -m will display mongo
What do you get instead?
Additional info

Q
A

Yii version

Yii MongoDB version

MongoDB server version

PHP version

Operating system

@smartyPHP, please use Official Yii 2 forum to ask a questions - GitHub is a bug tracking system.


Log the engine id when opening Kyuubi connection

Why are the changes needed?

Log the engine id when opening Kyuubi connection, it is more user friendly.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #2448 (f05ffa1) into master (71bc0dc) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master    #2448   +/-   ##
=========================================
  Coverage     63.18%   63.18%
  Complexity       69       69
=========================================
  Files           366      366
  Lines         17441    17442    +1
  Branches       2342     2342
=========================================
+ Hits          11020    11021    +1
+ Misses         5395     5394    -1
- Partials       1026     1027    +1

Impacted Files
Coverage Δ

.../org/apache/kyuubi/session/KyuubiSessionImpl.scala
76.71% <100.00%> (+0.32%)
:arrow_up:

...ache/kyuubi/operation/KyuubiOperationManager.scala
94.44% <0.00%> (-1.86%)
:arrow_down:

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
96.44% <0.00%> (-0.12%)
:arrow_down:

...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala
80.74% <0.00%> (+0.62%)
:arrow_up:

.../org/apache/kyuubi/operation/KyuubiOperation.scala
68.00% <0.00%> (+1.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71bc0dc...f05ffa1. Read the comment docs.

thanks, merging to master


External EHR integration resource identification Issue

Hello,
We are integrating external EHR systems Epic and Cerner and others.  We want to update existing patients with updated data from the respective EHRs. To identify a patient, we are thinking of using root resource "id" which is a unique identifier within respective EHRs. However, it's getting lost on saving the resource in Medplum DB as Medplum creates its own UUID for the resource.
Sample patient data from Epic/Cerner below:
Patient data from Cerner:
{
    "resourceType": "Patient",
    "id": "12724066",
    "meta": {
        "lastUpdated": "2023-06-15T13:37:18.000Z",
        "versionId": "1211"
    },
    ...
}

Patient data from Epic:
{
    "resourceType": "Patient",
    "id": "erXuFYUfucBZaryVksYEcMg3",
    "active": true,
    ...
}

For Epic, On the identifier block, we can use FHIR query

http://localhost:8103/fhir/R4/Patient?identifier=erXuFYUfucBZaryVksYEcMg3

{
"use": "usual",
"type": {
"text": "FHIR STU3"
},
"system": "http://open.epic.com/FHIR/StructureDefinition/patient-fhir-id",
"value": "erXuFYUfucBZaryVksYEcMg3"
}
However, the same patient resource id is not present for other EHRs such as Cerner or NextGen in the identifiers.
What is the best way to preserve the resource id? Should we create a custom identifier element to preserve the original EHR resource id?

Hi @santoshchandak -
Over the next 2-3 months, we plan to release initial support for Client Assigned IDs, which will allow you to use any FHIR compliant ID directly in the id field.
In the short term, you can follow the convention of copying the id into the identifier, similar to your Epic example.  The identifier field is an array, so you can add multiple identifiers:
{
  "resourceType": "Patient",
  "identifier": [
    {
      "system": "https://epic.com",
      "value": "my-epic-id"
    },
    {
      "system": "https://cerner.com",
      "value": "my-cerner-id"
    }
  ]
}

That is an imperfect solution, because it requires some manual work.  We know this is a pain point for the "multiple system synchronization" use case, so we're working hard to release Client Assigned IDs.
Let me know if you have any other questions.


How can I do to open the children list by default？

I use the most easy expandable recyclerview.And I'd like to open the children list when they load first.

Go through this https://github.com/thoughtbot/expandable-recycler-view/issues/54#issuecomment-284171806

Thanks you a lot.And it works for me.

How to collapse all the children at one go?