Document dvcfs API

Report
Dvcfs can be a useful public api that is more flexible than the methods in the current dvc api. Some use cases for it:

It can support walking through a repo or directory and getting the url or opening the files inside.
It can also be used to pass arbitrary credentials or other configuration to connect to remote storage.

Have we considered simplifying API refs and moving the details to auto-generated sites like https://docs.iterative.ai/dvc-task/reference/dvc_task/ ?

@jorgeorpinel, that’s a “Developer docs”, not a user facing one. It does not make sense to me to redirect to a different site for what is part of dvc’s offering.
But I am not against auto-generation if the engine can support.

We had that discussion for dvc.api and decided against it (primarily bc team wanted to keep docstrings really simple - e.g. no full examples like we have, etc, etc). Do we plan in this case bring those into docstring?

We do have docstrings now in dvc.api thanks to @daavoo. Docstrings are useful when you are working in IDEs/Editors. Personally, I wouldn't want to leave my text-editor/REPL to docs as much as possible because it involves context switching from the task I have in hand. So I prefer docstrings.
But I also think the docs have a different responsibility compared to docstrings, such as introducing the dvcfs to new users, suggesting scenarios that it can solve, etc that @dberenbaum mentioned above. That said, we also need an API reference too in the docs (which could be generated from the docstrings).

Yes, I was talking specifically about API reference. If you feel that docstring that we have (and plan to have for dvcfs) cover enough of a doc like this https://dvc.org/doc/api-reference/get_url , we should def auto generate. Since docstrings are 100% needed (I would never want to jump into browser also).

simplifying API refs and moving the details to auto-generated sites like https://docs.iterative.ai/dvc-task/reference/dvc_task/ ?

that’s a “Developer docs”, not a user facing one

Not sure it's a meaningful distinction @skshetry: The user is a "developer" here (API ref).
If the sites in docs.iterative.ai just repeat docstrings, do we even need them? Apparently unrelated, but the idea was: if we do need them, then we could leave all the dvc.api formalities (signature, types, arguments, etc.) that can be easily auto-generated there, and only use dvc.org/docs/api-ref for the Description and Examples.

does not make sense to me to redirect to a different site

It's pretty common to see different apps for technical explanations (example) vs. specifications (example). That said if the engine could incorporate auto-generated content that would be nice indeed.

Remaining work here is to have all supported methods linked from the docs:

It makes sense to start small with examples for this PR, but can we hold off closing #3927 until have a reference of supported methods either in the docs or linked from the docs (to an auto-generated docs site)? Compare to the s3fs docs and gcsfs docs. Both of those document the full API of methods they support. Otherwise, we are leaving users to either look through the codebase (without pointing to where they should look) or go to the fsspec AbstractFileSystem docs and use trial and error to find which methods are implemented.

Originally posted by @dberenbaum in https://github.com/iterative/dvc.org/pull/3932#pullrequestreview-1110727121

Related discussion: https://github.com/iterative/mlem.ai/issues/171
And initial proposal (with more discussion): https://github.com/iterative/mlem.ai/pull/172

Docstrings were added in https://github.com/iterative/dvc/pull/8332. Now how can we publish some docs from them?

Docstrings were added in iterative/dvc#8332. Now how can we publish some docs from them?

That's THE question 😅

Is there a plan with plugins to separate dvcfs from dvc?

Do you mean to generate something like this? https://docs.iterative.ai/dvc-task/ That could work. We just need to link these sites from DVC docs more, I think.

Yup @jorgeorpinel! The problem here is that dvcfs is one of the few things we haven't made into a separate package, so there's no obvious way to generate that.

OK but if the plan is to document it outside of dvc.org maybe we should move this ticket to the core repo.

Added a ticket in https://github.com/iterative/dvc/issues/8428. I'd vote to keep this one also to remember to link to whatever gets generated, but if you prefer to close it to reduce noise, no big deal @jorgeorpinel .

@yathomasi @rogermparent technical question - can we use docstring from an external repo as a gatsby node?

Yep! Anything that can be accessed with Node.js can be turned into a Gatsby node. I was thinking of making prism commands exported like this to get around our multi-repo command woes. We can also consume that node in any way we want, including having doc pages display the docstring content alongside some optional extra elaboration.

@rogermparent that sounds cool! and I see that multiple repos need this. It can simplify the docs flow - for certain pages we'll use Python files as a source, not Markdown. Let's create a ticket and discuss during the next call.

The generated reference docs will have the same content as fsspec's docs. What is our goal with the reference docs?
Do we want to expand upon those docs with examples? If so, then example-driven docs might be better than having a reference. I prefer User Guides that help users get things done (and looking at fastapi, they don't seem to have reference docs).
The only issue that I think this will solve is regarding discoverability, and that won't be fixed if we use a separate site for just dvcfs.

and looking at fastapi, they don't seem to have reference docs).

Not a popular decision, though: https://github.com/tiangolo/fastapi/issues/804 😅

Not a popular decision, though

For a popular project like that, someone will be unhappy, and only few people have commented. And dvcfs is not in the scale of fastapi. I am not against reference docs, but just want to set goals and how can we improve compared to fsspec's existing reference docs (given the same content except for the initializer/constructor).

and only few people have commented.

Well . . .

I am not against reference docs, but just want to set goals and how we can improve compared to fsspec's existing reference docs (given the same content except for the initializer/constructor).

Yes, in this case, I agree that there is too much overlap with the existing fsspec API ref and that we already cover the differences with examples, so I am ok with not duplicating.

My main concern is that it's unclear which methods from fsspec are implemented by dvcfs.

My main concern is that it's unclear which methods from fsspec are implemented by dvcfs.

Maybe I am biased here, but I think "Read only filesystem" may be enough here. Most of the API will raise NotImplementedError or EROFS during runtime.
Also, most of the APIs are implemented in fsspec and the write-operations are not available during runtime. So this may require us to extend some methods to add docs that they are unavailable.
Anyway, here is the current state of APIs that are unsupported:

mkdir/makedirs/rmdir: are no-ops + optional to implement.

All of the following raises or should raise NotImplementedError/OSError(EROFS):

cp_file/sign/created/modified/_rm/rm_file: not extended in dvcfs, default implementation is to raise NotImplementedError.
write_text/pipe_file/put_file/touch -> because open does not support write operations.
copy -> depends on cp_file.
mv -> depends on copy/rm.
rm -> depends on _rm/rm_file.
put -> depends on put_file.
pipe -> depends on pipe_file.

Good points @skshetry. It looks like auto suggestions and docstrings from within the IDE are working well, and between that, the existings fsspec reference, and the "read only filesystem" description, it should be enough. I'm fine to close this, but if someone else feels more is needed, feel free to reopen.


Should there be any changes in the inference code for Korean inputs?

Hi! I noticed that you have a pretrained model trained on KSS dataset and I have trained a mel synthesizer model on the same dataset. When I use Griffin-Lim algorithm to reconstruct the linear output, I get a good enough generated speech. However, when I used the mel output of the synthesizer as the input to your model, the output is 11 seconds of nothing. Should there be any modifications when using Korean inputs?

Hello @jodumagpi ,
When I train the model, I change the sampling rate  with 22050.
(i.e., 44100 -> 22050)
I recommend that test with ground-truth input first, so that you can check whether vocoder is problematic or not.

Hi! Thank you! It works now. But the problem was really with the mel input. When I used a different synthesizer to generate the mel, it worked.

Hi! Thank you! It works now. But the problem was really with the mel input. When I used a different synthesizer to generate the mel, it worked.

That must be the problem of Text2Mel network. However, keep in mind that the pretrained model trained with KSS dataset only converts female voices.
If you want to generalize VocGAN for unkown speakers, you must train the VocGAN from scratch using massive and diverse speech dataset.
Thank you.


Implement refresh token rotated feature for public clients gh-297

As per https://datatracker.ietf.org/doc/html/draft-ietf-oauth-browser-based-apps-07#section-8 (#297) refresh token for public clients must be issuing with reduced time to live based on current refresh token time to live and duration of disuse.
@jgrandja, did I correctly understand and implement best practice?

Thanks for the PR @boggard.
OAuth 2.0 for Browser-Based Apps documents the best practice for refresh tokens, however, it also states:

Authorization servers may choose whether or not to issue refresh tokens to browser-based applications

We're going to apply gh-296 for the 0.2.0 release as this eliminates the risk of a leaked refresh_token by a public client.
I'll keep this PR open for now and we'll revisit this later and possibly merge this feature or provide a customization hook that would allow applications to provide this feature.

@boggard I'm going to close this PR as the current implementation of OAuth2AuthorizationCodeAuthenticationProvider does not support issuing refresh tokens to public clients.
We have limited support for Public Clients at the moment but we do plan on building support later on at some point. Please keep track of gh-297 as when we start work on Public Client support, then would be a good time to resubmit this PR. Thanks.

@boggard I'm going to close this PR as the current implementation of OAuth2AuthorizationCodeAuthenticationProvider does not support issuing refresh tokens to public clients.
We have limited support for Public Clients at the moment but we do plan on building further support later on at some point. Please keep track of gh-297 as when we start that work, then would be a good time to resubmit this PR. Thanks.

In my opinion the leakage of refresh_token by public clients is as dangenerous as the leakage of session ids. Brutely banning the issuing of refresh tokens to public clients really limits the use cases of this project to public clients, which is really annoying at this stage.

A better solution would be to make it configurable, before you can fully implement gh-297.

@zhenchuan9r

Brutely banning the issuing of refresh tokens to public clients really limits the use cases of this project to public clients, which is really annoying at this stage

Can you please provide reason(s) why this limits use cases for public clients? Why is a refresh_token needed? Why can't the public client obtain a new access token after the existing one is expired?

reason(s) why this limits use cases for public clients

@jgrandja as someone who's been following this issue, I hope you don't mind the interjection. Correct me if I'm wrong but obtaining a new access token would require the end-user to re-enter their credentials, which for mobile apps is less convenient than using a refresh token.
We could perhaps use longer-lived access tokens, but (as I understand it) those are more difficult to revoke than refresh tokens if needed.

@zhenchuan9r

Brutely banning the issuing of refresh tokens to public clients really limits the use cases of this project to public clients, which is really annoying at this stage

Can you please provide reason(s) why this limits use cases for public clients? Why is a refresh_token needed? Why can't the public client obtain a new access token after the existing one is expired?

Thanks for reply, and my thoughs are already expressed by chrisrhut.
I agree that requiring end-users to re-authenticate just because access_token expires would indicate bad UE.
Actually I still don't quite understand the extra risk by allowing refresh_token to be issued.
Most of our sites are javascript sites, without refresh_token we may be forced to use session_id.
If we are forced to replace refresh_token by session_id, session_id can also live forever if stealed, which seems not making any sense on improving security.

Thanks for the feedback @chrisrhut @zhenchuan9r.
After we release 0.2.2 at the end of the month, I'll start looking into gh-297 and will log separate issues to start working on building more support for Public Clients.

Following, I'm also working with public clients and find a long lived access token not ideal in this case.

@chrisrhut @zhenchuan9r @nickmelis Just putting this out there in case you are not aware of the Backend-for-Frontend (BFF) design pattern?
You might want to look into this as this is the recommended approach for securing access tokens (and refresh tokens) using a SPA. See this comment for additional context.
As I'm sure you are aware, storing (or passing through) access tokens or refresh tokens in the browser via a SPA is not secure and discouraged.

IMHO even though the BFF pattern is great and can work in some cases, it's too intrusive and inefficient for others: when you have multiple clients, you use a CDN for serving the SPA, etc.
PKCE was meant to be used by public clients that can't store secrets but I think that being able to get a refresh token when using this flow is highly needed. If there isn't a way to refresh the token, we basically have two alternatives:

Ask the user to login again frequently.
Increase the access_token expire time.

I guess that most developers will end up setting really long access_token expiration times and we all know that this is a bad idea and even more dangerous than leaking a refresh_token that is going to be rotated.
I think that allowing refresh tokens shouldn't be that bad if we follow good practices: they must be retrieved using a secure communication (https and PKCE take care of that), they shouldn't be stored in local or session storage (use memory instead), and refresh tokens should be rotated in order to mitigate the consequences of a leak.

Thanks for the feedback @aberasarte. We'll be looking into this shortly.

@boggard @zhenchuan9r @chrisrhut @nickmelis @aberasarte
I reviewed the relevant specifications in detail and provided a high-level summary and final recommendation for best practices when developing browser-based apps. Please see gh-297.

@jgrandja I appreciate the careful consideration you and your team have given this, but I'm still a little bit at a loss. Your comment mentions Token Mediating and Session Information Backend For Frontend - but that document says (emphasis mine):

As a prerequisite for the flow described below, the backend MUST have established a secure session with the user agent, so that all requests from that user agent toward the backend occur over HTTPS and carry a valid session artifact (such as a cookie) that the backend can validate.

My hope had been to use spring-authorization-server to do exactly that - establish the secure session between a mobile client and the server - using OAuth2 (JWT access and refresh tokens).
It sounds like the only alternative is to use an alternate mechanism (a la "plain" spring-security backend OIDC login and session tokens) to do so, but as zhenchuan9r mentioned in the comments above, I'm not seeing the risk distinction between a long-lived opaque session token (which can be leaked) and a long-lived refresh token; in fact as they mentioned the former is somewhat more risky as there's no standard for token revocation.
It seems (and forgive me if I'm misinterpreting) you're suggesting that generally speaking, mobile apps just shouldn't provide long-lived sessions - or if they do, you want to make sure that it's not spring-authorization-server that's allowing it?

@chrisrhut Regarding your comment:

My hope had been to use spring-authorization-server to do exactly that - establish the secure session between a mobile client and the server - using OAuth2 (JWT access and refresh tokens)

Authentication is NOT a concern of an OAuth 2.0 Provider, it's only concern is Authorization. This is the basis of OAuth 2.0, where Authorization Servers mint tokens and Resource Servers authorize tokens. Although, OpenID Connect 1.0 was introduced to address Authentication, you still need to leverage an Authentication sub-system integration with the OAuth 2.0 Provider, e.g. LDAP, SAML, Federated OIDC (social login), custom form login, etc.
As far as an "authenticated session" goes, this is the responsibility of the Authentication sub-system that is integrated with the OAuth 2.0 Provider (Spring Authorization Server). As I'm sure you are aware, a client cannot obtain an access token using the authorization_code grant, without an authenticated session of the Resource Owner.

As a prerequisite for the flow described below, the backend MUST have established a secure session with the user agent, so that all requests from that user agent toward the backend occur over HTTPS and carry a valid session artifact (such as a cookie) that the backend can validate.

The backend would need to be configured with some form of authentication mechanism to implement this pattern. It would store the sensitive data (access tokens, refresh tokens, etc.) and could only be fetched by an authenticated secure session between the frontend and backend. Spring Security provides many different integration options for various authentication mechanisms, so I would recommend looking there. However, I still favour the BFF pattern because tokens NEVER pass through the browser agent, which is what I would ensure when building an SPA. The bottom line is that the browser agent is NOT a secure environment, so I personally would not allow tokens to pass through or even worse store them in storage/memory.
As far as session-related attacks, e.g. session fixation, Spring Security provides out-of-the-box protection against well-known vulnerabilities, see Session Management.
Finally, our ultimate goal is to build the most secure OAuth 2.0 / OIDC framework for the Java platform and limit the potential attack surface. Therefore, we need to ensure we don't introduce features that could potentially introduce a vulnerability down the road. Allowing Public Clients to obtain refresh tokens opens things up so this is not something we want to introduce.

@jgrandja I think your final paragraph drives it home and that's definitely an understandable stance! It sounds like there might be a slight mismatch with our use case, but I wanted to thank you for taking the time to respond with such consideration. :pray:  I didn't intend to derail this thread/issue and make it my personal technical support line, but hopefully all of this is helpful added context for anyone who follows too.

@jgrandja

Allowing Public Clients to obtain refresh tokens opens things up so this is not something we want to introduce.

Does that mean the option of issuing access and refresh tokens to public clients will never find its way into spring-authorization-server? Not even as a disabled-by-default config option?

@chrisrhut

I didn't intend to derail this thread/issue and make it my personal technical support line

Not at all. I'm glad you understand our stance on this and that is all I wanted to communicate to you and the community. We are doing our best to make Spring Authorization Server accessible to ALL users including Javascript developers. But at the same time, we need to keep the balance between accessibility and security and therefore need to ensure we always keep to a very tight security posture.
FYI, we're doing a webinar on Spring Authorization Server and best practices developing a SPA on Mar 10. You might be interested in checking it out. You can register here.

@nickmelis

Does that mean the option of issuing access and refresh tokens to public clients will never find its way into spring-authorization-server?

A public client is able to obtain an access token using authorization_code with PKCE. This capability has been available for quite some time now and we don't plan on removing it.
As far as refresh_token capability for public clients go, there are no plans to add this capability. See "Refresh Tokens for Public Clients" in gh-297 for additional details.

Hi @jgrandja
First, thanks for taking the time to answer all these concerns, they are really insightful especially in understanding why having a refresh token for public client is a bad idea.
Although, I think it would be bit forceful to make users stop using the 'Remember Me' functionality on front end clients. My opinion on the BFF pattern is that its not meant to act as a security layer, rather a consolidation layer for microservices. It does not solve the issue of the 'Remember Me' functionality in terms of security, and its not feasible option for smaller projects.
This may affect or force dev's to use confidential client with authorization code flow, storing the private key on client just so that they do not break existing functionality. This would seem to be a bigger security risk than having refresh token being leaked in my opinion.
I agree to what @nickmelis mentioned at least having refresh-token disabled as a default configuration, or forcing the dev to implement DPoP for refresh token, would seem to be a better approach.

Hi @jgrandja,
As others already said, thanks for your thorough explanation about the refresh_token in public clients and for taking into account our opinion. I still think that not having this, makes the flow pretty much unfeasible to use and will push developers to set really long expiration access tokens which IMHO is even more dangerous. I guess that this is the reason why all of the Authz servers I know, support refreshing tokens when using the PKCE flow: Google, Microsoft, Hydra, Okta...
That being said, I really appreciate the work you are doing in this project. It's a great alternative for those who want a lightweight Authorization Server in the Java/Spring ecosystem.
Keep up the good work 👌 !

Just to add some more context to @aberasarte's comment, we recently moved from Auth0 to our own auth server, and Auth0 too supports sending refresh tokens together with access token with the PKCE flow (it's disabled by default though).

@rayman245

I think it would be bit forceful to make users stop using the 'Remember Me' functionality on front end clients

I'm not seeing the connection between "Remember Me" functionality and Refresh Token? Are you using a Refresh Token for the Remember Me token? Can you provide more insight on your usage as your comments are not that clear to me.

@aberasarte @nickmelis Can you provide me links to the documentation for Refresh Token public client support for any/all of the providers you listed -> Google, Microsoft, Hydra, Okta, Auth. I'd like to dig into it a bit more and see how they have implemented it.
The one major question I have is how do those providers authenticate the public client when it performs the refresh_token grant? The client MUST authenticate at the token endpoint and since public clients are not assigned credentials, how do they authenticate with the provider to assert their identity?

I'm not seeing the connection between "Remember Me" functionality and Refresh Token? Are you using a Refresh Token for the Remember Me token? Can you provide more insight on your usage as your comments are not that clear to me.

Sure @jgrandja ,
In the legacy OAuth server, front end clients would implement the remember me functionality by using refresh tokens to get new access tokens without the need for the user to go through the authorization flow every time.
This article explains how it was done: https://www.baeldung.com/spring-security-oauth2-remember-me
Also we have built apps with google sign-in using this guide -> https://developers.google.com/identity/protocols/oauth2/native-app
The refresh token will be returned after authorization.

@aberasarte @nickmelis Can you provide me links to the documentation for Refresh Token public client support for any/all of the providers you listed -> Google, Microsoft, Hydra, Okta, Auth0. I'd like to dig into it a bit more and see how they have implemented it.

@jgrandja I haven't used Auth0 in production yet, but we're planning to migrate to it eventually so I have looked at its documentation on this subject. I believe the relevant pages are https://auth0.com/docs/secure/tokens/refresh-tokens and https://auth0.com/docs/secure/tokens/refresh-tokens/refresh-token-rotation.
There is also this blog post introducing the topic in 2020 - https://auth0.com/blog/securing-single-page-applications-with-refresh-token-rotation/.
The Auth0 SPA SDK has docs on how to use rotating refresh tokens:
https://auth0.com/docs/libraries/auth0-single-page-app-sdk#use-rotating-refresh-tokens

@jgrandja probably redundant but we followed this document: https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow-with-proof-key-for-code-exchange-pkce

@aberasarte @nickmelis Can you provide me links to the documentation for Refresh Token public client support for any/all of the providers you listed -> Google, Microsoft, Hydra, Okta, Auth0. I'd like to dig into it a bit more and see how they have implemented it.

Sure!

Google https://developers.google.com/identity/protocols/oauth2/native-app#exchange-authorization-code.

Exchange authorization code for refresh and access tokens...

Microsoft https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow#refresh-the-access-token

Refresh tokens for web apps and native apps don't have specified lifetimes. Typically, the lifetimes of refresh tokens are relatively long...

Hydra https://www.ory.sh/oauth2-for-mobile-app-spa-browser/#proof-key-for-code-exchange-pkce . BTW Hydra is an open source project I really like and I think that it could be a great source of inspiration.

The code verifier, only known to the legitimate application, is transmitted when the authorization code is exchanged for access and refresh tokens...

Okta https://developer.okta.com/docs/guides/refresh-tokens/main/

... Refresh token rotation works with SPAs, native apps, and web apps in Okta.

@rayman245 The article you referenced OAuth2 Remember Me with Refresh Token is quite an unusual pattern. To the point, a refresh token should NOT be used as an authentication token.
Remember Me functionality typically stores user authentication context information via a cookie. Because this cookie contains user authentication context, a pre-authentication scenario is possible.
On the other hand, a refresh token is an authorization grant NOT an authentication token containing user context information. It's meant to be used by the client to exchange an expired access token.
Furthermore, the article states the following:

On a first authentication attempt using the password grant type

the purpose of the refresh token is to obtain a new valid access token or just revoke the previous one

To receive a new access token using the refresh_token grant type, the user no longer needs to enter their credentials, but only the client id, secret and of course the refresh token.

This article does not apply to the flow being discussed in this ticket. The flow in the article is password grant type and it's using a confidential client NOT public client, since the client_secret is being passed during the refresh_token grant flow.

Also we have built apps with google sign-in using this guide -> https://developers.google.com/identity/protocols/oauth2/native-app

This article describes Native (Mobile and Desktop) apps NOT JavaScript (SPA) apps. The section Step 5: Exchange authorization code for refresh and access token demonstrates how to refresh an expired access token using a Confidential Client (client_secret required) NOT a Public Client.
The article OAuth 2.0 for Client-side Web Applications is meant for SPA's, however it references usage of the implicit grant which is deprecated.

@jgrandja

@rayman245 The article you referenced OAuth2 Remember Me with Refresh Token is quite an unusual pattern. To the point, a refresh token should NOT be used as a user authentication token.

I mentioned that the refresh token is used to retrieved a new authentication token, just without the need for users to go through authentication flow each time.

This article describes Native (Mobile and Desktop) apps NOT JavaScript (SPA) apps. The section Step 5: Exchange authorization code for refresh and access token demonstrates how to refresh an expired access token using a Confidential Client (client_secret required) NOT a Public Client.

Also according to OAuth mobile apps are considered public clients please refer to -> https://oauth.net/2/client-types/
And yes, the only option for mobile app integration now is treating them as confidential client, which would be unwise, since its easy for anyone to reverse engineer apk/ipa files.

@aberasarte The Google reference you provided:

Google https://developers.google.com/identity/protocols/oauth2/native-app#exchange-authorization-code.

... uses a Confidential Client NOT a Public Client when authenticating on the refresh_token grant flow.
I provided more details in this comment:

This article describes Native (Mobile and Desktop) apps NOT JavaScript (SPA) apps. The section Step 5: Exchange authorization code for refresh and access token demonstrates how to refresh an expired access token using a Confidential Client (client_secret required) NOT a Public Client.

The article OAuth 2.0 for Client-side Web Applications is meant for SPA's, however it references usage of the implicit grant, which is deprecated.

As of RFC8252 (OAuth 2.0 for Native Apps) it recommends to use authorization grand with refresh tokens. But it also states that native apps should still be classified as public (section 8.4). My understanding is that public native clients should be able to obtain a refresh token even if they are considered public (and do not have a client secret).

@bschoenmaeckers

My understanding is that public native clients should be able to obtain a refresh token even if they are considered public (and do not have a client secret)

Can you please provide the exact reference in the relevant spec that states this. Even though native clients are considered public, how do they authenticate during the refresh_token grant flow if they do not have credentials (client_secret) assigned? The token endpoint REQUIRES client authentication.
If a public client receives a refresh token but is not able to authenticate itself on the token endpoint, then the refresh token is useless, since the exchange will simply fail. NOTE: If a native client is assigned credentials then the exchange will succeed, however, the native client in this particular flow is considered a confidential client since it uses it's credentials to authenticate on the token endpoint.

Can you please provide the exact reference in the relevant spec that states this?

RFC6749 - Section 9 (Native Applications)

Native applications that use the authorization code grant type
SHOULD do so without using client credentials, due to the native
application's inability to keep client credentials confidential.

RFC6749 - Section 10.1 (Client Authentication)

The authorization server MUST NOT issue client passwords or other
client credentials to native application or user-agent-based
application clients for the purpose of client authentication.  The
authorization server MAY issue a client password or other credentials
for a specific installation of a native application client on a
specific device.
When client authentication is not possible, the authorization server
SHOULD employ other means to validate the client's identity -- for
example, by requiring the registration of the client redirection URI
or enlisting the resource owner to confirm identity.  A valid
redirection URI is not sufficient to verify the client's identity
when asking for resource owner authorization but can be used to
prevent delivering credentials to a counterfeit client after
obtaining resource owner authorization.

If I understand this these two parts correctly then native apps should not be treated as confidential clients and should not have a client secret issued to them.
Furthermore, as stated in the next paragraph. Clients without a secret should consider extra measures to limit exposure of for example refresh tokens. This is where the extra security considerations of RFC8252 OAuth 2.0 for Native Apps - Section 8 should be used.

The authorization server must consider the security implications of
interacting with unauthenticated clients and take measures to limit
the potential exposure of other credentials (e.g., refresh tokens)
issued to such clients.

As stated in the shown paragraphs below, the token endpoint does NOT REQUIRE a client secret if the client is not confidential.
RFC6749 - Section 4.1.3 (Authorization Code Grant - Access Token Request)

If the client type is confidential or the client was issued client
credentials (or assigned other authentication requirements), the
client MUST authenticate with the authorization server as described
in Section 3.2.1.

This is also true when using a refresh token: RFC6749 - Section 6 (Refreshing an Access Token)

RFC6749 - Section 6 (Refreshing an Access Token

Because refresh tokens are typically long-lasting credentials used to
request additional access tokens, the refresh token is bound to the
client to which it was issued.  If the client type is confidential or
the client was issued client credentials (or assigned other
authentication requirements), the client MUST authenticate with the
authorization server as described in Section 3.2.1.

@jrmcdonald @aberasarte @nickmelis
Auth0 does in fact support refresh tokens for public clients.
Key points in the Refresh Tokens docs:

The refresh token behaviour is applicable to OIDC-conformant applications

For native applications, refresh tokens improve the authentication experience significantly. The user has to authenticate only once, through the web authentication process. Subsequent re-authentication can take place without user interaction, using the refresh token.

If you want to allow users to get refresh tokens while offline, you can select the Allow Offline Access switch in API Settings.

You can increase security by using refresh token rotation which issues a new refresh token and invalidates the predecessor token with each request made to Auth0 for a new access token. Rotating the refresh token reduces the risk of a compromised refresh token.

Key points in the Get Refresh Tokens  docs:

To get a refresh token, you must include the offline_access scope when you initiate an authentication request through the /authorize endpoint. Be sure to initiate Offline Access in your API. For more information, read API Settings.

If you are requesting a Refresh Token for a mobile app using the corresponding Native Client (which is public), then you don't need to send the client_secret in the request since it's only required for confidential applications.

Key points in the Refresh Token Rotation docs:

Refresh tokens are often used in native applications on mobile devices in conjunction with short-lived access tokens to provide seamless UX without having to issue long-lived access tokens

With refresh token rotation enabled in the Auth0 Dashboard, every time an application exchanges a refresh token to get a new access token, a new refresh token is also returned. Therefore, you no longer have a long-lived refresh token that, if compromised, could provide illegitimate access to resources. As refresh tokens are continually exchanged and invalidated, the threat is reduced.

Unfortunately, long-lived refresh tokens are not suitable for SPAs because there is no persistent storage mechanism in a browser that can assure access by the intended application only. As there are vulnerabilities that can be exploited to obtain these high-value artifacts and grant malicious actors access to protected resources, using refresh tokens in SPAs has been strongly discouraged

Based on my analysis, Auth0 supports refresh tokens for public clients based on the following conditions:

The application/client must perform the OpenID Connect 1.0 Authentication using the Authorization Code Flow (with PKCE). NOTE: The standard OAuth 2.0 Authorization Code Grant would not apply.

The offline_access scope must be passed in the Authentication Request.

Refresh Token Rotation SHOULD be enabled to help mitigate refresh token leakage.

The KEY POINT that I want to relay here is that even if all the above conditions are met, an exposure is still possible.
Given the following scenario where a legitimate public client (SPA) receives an access token and refresh token via an OIDC auth code flow and subsequently a malicious client obtains the refresh token by leakage via browser, and the malicious client uses the refresh token before the legitimate client does than the exposure has happened. Assuming access tokens expire every 1 hour, the exposure can last up to one hour. So although Refresh Token rotation helps reduce the threat, it DOES NOT completely eliminate it in this specific scenario.
The other implementation behaviour that does not sit well with me is that a Public Client can perform the refresh_token grant flow WITHOUT needing to authenticate on the token endpoint. This is NOT an implementation behaviour we would provide as a default configuration.
The bottom line is that Refresh Token rotation helps REDUCE the threat of refresh token leakage it DOES NOT ELIMINATE the threat completely. Therefore, enabling the above capabilities effectively "opens up" the potential attack surface, which obviously reduces the overall security posture.
We really are trying to figure out a secure way for SPA's to integrate with Spring Authorization Server. At the same time, we need to remain diligent on maintaining a tight security posture.
We'll be presenting a webinar next week on this topic so I would encourage you to attend as we will answer a lot of your questions there. You can register here.

Thanks for the detailed response @jgrandja, I completely understand your position. This balance between risk and convenience to our users is something we are evaluating in our project. Thank you for the link the webinar, I look forward to it.

Thanks @bschoenmaeckers for the references! I can certainly see how people may interpret parts of the spec differently because it is written generically in some sections and is left up to the implementor to choose the appropriate strategy for implementation.
The spec states:

If the client type is confidential or the client was issued client
credentials (or assigned other authentication requirements), the
client MUST authenticate with the authorization server as described
in Section 3.2.1.

This MAY be interpreted....if the client does not have any credentials assigned then it DOES NOT require authentication on the token endpoint. This most definitely will not be a default configuration that Spring Authorization Server will provide out-of-the-box. I hope the reasons are obvious.

The authorization server must consider the security implications of
interacting with unauthenticated clients and take measures to limit
the potential exposure of other credentials (e.g., refresh tokens)
issued to such clients.

We have considered the implications of adding support for refresh tokens for public clients alongside refresh token rotation but this scenario will increase the potential attack surface as detailed in this comment.

@aberasarte Okta's refresh token support for public clients is very much similar to Auth)'s implementation. However, it also supports the prompt parameter in the Authentication Request.
Please see my comments for Auth0 as the same points would apply for Okta.

@aberasarte Microsoft's refresh token support for public clients has some security gaps - refresh token rotation is not implemented.

For refresh tokens sent to a redirect URI registered as spa, the refresh token expires after 24 hours. Additional refresh tokens acquired using the initial refresh token carries over that expiration time, so apps must be prepared to re-run the authorization code flow using an interactive authentication to get a new refresh token every 24 hours. Users do not have to enter their credentials, and usually don't even see any user experience, just a reload of your application. The browser must visit the login page in a top level frame in order to see the login session. This is due to privacy features in browsers that block third party cookies.

Although refresh token TTL is shorter than typical, re-authentication is still required after 24 hrs, which comes back to the main issue that SPA's face, which is not a great user experience.

Thanks for the feedback @nickmelis and I'm glad you found it informative 👍

it would be good to be able to put together some documentation on all available options for public clients as of today (BFF, silent refresh, mutual TLS, etc)

Everything we presented yesterday is fully documented in the OAuth 2.0 for Browser-Based Apps BCP and I've provided a summary of it in gh-297.
The sender-constrained method will be implemented in gh-101 and documented at that point. There will be some challenged getting it working for a browser-based app but I'll figure something out. Please track that issue for progress.
We're planning on adding the BFF sample to this repository soon as well.
@sjohnr Can you provide some references for "silent authentication"?

I found these resources useful in my research:

angular-auth-oidc-client client library docs on silent renew
Auth0's SDK docs on Get Access Token with no interaction

<IP_ADDRESS>.  Authorization Server Authenticates End-User states:

The Authorization Server MUST NOT interact with the End-User in the following case:

The Authentication Request contains the prompt parameter with the value none. In this case, the Authorization Server MUST return an error if an End-User is not already Authenticated or could not be silently Authenticated.

While there is some additional support for OIDC still missing, this flow does work out of the box even though (to my knowledge) the prompt parameter is not yet supported. I believe this is the case because, as I demonstrated in the webinar, you have a session and a saved consent available for the currently logged in user.

Hello @jgrandja, following up on the silent renew mentioned in the webinar, our authenticator app returns X-Frame-Options=DENY by default, which prevents implementing silent renew in an iframe. What immediately comes to my mind is to set the header to SAMEORIGIN. Have you got any opinion/security consideration around that? Thanks


Add Contribution rules and guidelines.

Refer to any other open source project and add the documentation into a contribution folder as a readme file.

More can be referred on the main readme file on this repo

@AmreetKumarkhuntia I can tackle this issue if that is okay with you.

Rename the pull request to Added contributions and in the description you can keep added code_of_conduct and contribution. md .

Ok will do.


Data Hub Uploads prevent horizontal scaling

As discussed in the scaling-out risks document, we cannot have multiple instances of our API in production without double-submitting our data to the Data Hub. We should fix that.

Ben suggested the best way to implement this to use postgressql's advisory locks.

Ben suggested the best way to implement this to use postgressql's advisory locks.


Add support for Xamarin.Mac

In addition to iOS and Android, please support Xamarin.Mac.

Hi,
unfortunately, we won't add OS X in the near feature.
Christoph


update deployment issue workflow so that the issues have project metadata fields updated for better sorting and filtering

Have to use the new graph API to access beta projects
Project ID's
related article
can use their graph api explorer and run this query to find the project ID's for the org:
{
  organization(login: "CoveredCA") {
    login
    projectsV2(first: 100) {
      edges {
        node {
          id
          title
        }
      }
    }
  }
}

Field ID's
gh api graphql -f query='
            query($org: String!, $number: Int!) {
              organization(login: $org){
                projectV2(number: $number) {
                  id
                  fields(first:20) {
                    nodes {
                      ... on ProjectV2Field {
                        id
                        name
                      }
                      ... on ProjectV2SingleSelectField {
                        id
                        name
                        options {
                          id
                          name
                        }
                      }
                    }
                  }
                }
              }
            }' -f org=$ORGANIZATION -F number=$PROJECT_NUMBER

run update operations on a given item
gh api graphql -f query='
    mutation {
        updateProjectV2ItemFieldValue(
            input: {
                projectId: "PVT_kwDOAfwVjc4AV0R8"
                itemId: "40946050"
                fieldId: "121975948"
                value: {
                    text: "hello world"
                }
            }
        )
        {
            projectV2Item {
                id
            }
        }
    }'

jq query to get the results from issue query to grab ID for final graph QL update
jq '.data.organization.projectV2.items.nodes[] | select(.content.number == 17) | select(.content.title | test("cca-[a-zA-Z0-9-]+-[a-zA-Z]api"))' deployment_issues.json | jq -r .content.id

This bash script could use pagination to get all items in the project to make it easier to manage with jq
# Initial cursor is empty
CURSOR=""

# Loop until all pages are fetched
while : ; do
    # Execute GraphQL query
    RESPONSE=$(gh api graphql -f query='
    query($org: String!, $number: Int!, $cursor: String) {
        organization(login: $org) {
            projectV2(number: $number) {
                items(first: 100, after: $cursor) {
                    pageInfo {
                        hasNextPage
                        endCursor
                    }
                    nodes {
                        id
                        content {
                            ... on Issue {
                                id
                                number
                                title
                            }
                        }
                    }
                }
            }
        }
    }' -f org="$ORGANIZATION" -F number=$PROJECT_NUMBER -f cursor="$CURSOR")

    # Parse items and do something with them, e.g., append to a file
    echo "$RESPONSE" | jq '.data.organization.projectV2.items.nodes[]' >> deployment_issues.json

    # Check if there's a next page; if not, break the loop
    HAS_NEXT_PAGE=$(echo "$RESPONSE" | jq '.data.organization.projectV2.items.pageInfo.hasNextPage')
    if [ "$HAS_NEXT_PAGE" != "true" ]; then
        break
    fi

    # Update cursor to fetch next page
    CURSOR=$(echo "$RESPONSE" | jq -r '.data.organization.projectV2.items.pageInfo.endCursor')
done

echo "All items fetched successfully."

jq '. | select(.content.number == 17) | select(.content.title | test("cca-salesforce-sapi"))' deployment_issues.json

2024-07-19
got things to start working but i had to change the secrets in the org to be available to all repositories in the org including public, but this is just temporary for testing on workflow call directly from common-devops repository.
after im done testing this reusable flow will need to be called from the pipeline which is called by the child processes and thats how the secrets get accessed. so ultimately the secrets need to be returned to their original state of private and internal repositories only.

further work to be done on this issue:

[ ] make new fields for major, minor, and patch versions so the app can actually be sorted correctly as it is still sorting based on text string values incorrectly

[ ] make those fields
[ ] update the reusable action to modify those fields
[ ] parse the version number value to correctly grab the parts

2024-07-23
the new fields are added and populated correctly now for the deployment issue which allows for easier sorting in the deployment dashboard pane and overall filtering is possible too by the version text field now when filtering in the search bar.
for the time being we're calling this work good, theres more that could probably be done here with custom fields but at the moment i think this work is complete.


Unclear error message for unauthorized user

When an unauthorized user tries to install a service in the CLI they are presented with this message:
$ dcos package install spark
Note that the Apache Spark DCOS Service is beta and there may be bugs, incomplete features, incorrect documentation or other discrepancies.
We recommend a minimum of two nodes with at least 2 CPU and 2GB of RAM available for the Spark Service and running a Spark job.
Note: The Spark CLI may take up to 5min to download depending on your connection.
Continue installing? [yes/no] yes
Installing Marathon app for package [spark] version [1.6.0]
<IP_ADDRESS>'s username: alic
<EMAIL_ADDRESS>password:
<IP_ADDRESS>'s username: alice
<EMAIL_ADDRESS>password:
Error decoding response from [http://<IP_ADDRESS>/marathon/v2/info]: HTTP 403: Forbidden

This error should be made clearer - "Error decoding response" is ambiguous.

What auth type are you using? Any wording recommendations?

@tamarrow i'm using the default auth with 1.6. i think RBAC?
[2:52 PM]
here's a working draft of th topic: https://docs.mesosphere.com/installing-enterprise-edition-1-6/enabling-authorization/.

For Kafka, I get a different message: $ dcos kafka broker list <IP_ADDRESS>'s username: joel<EMAIL_ADDRESS>password:  Error: java.io.IOException: 401 - Unauthorized
Are these messages service-specific?

The kafka error is in the kafka subcommand, which is up to whoever owns the kafka cli. I'll look into the other one.

@tamarrow Okay, I have opened a JIRA to track the Kafka issue: https://mesosphere.atlassian.net/browse/INFINITY-68.


Migrate gem to inherit from metanorma-generic

Need to deal with blank Word cover setting in YAML.


hub pull-request assuming the wrong branch

Command attempted:
hub pull-request
What happened:
it is assuming the wrong branches. i have a local branch just pushed to 'origin', i confirmed that it's set to track the remote branch. i do hub pull-request and it shows me the wrong branch... I read in the forums about the same issue happening to other but it was suggested to ensure local branch is tracking by doing a push & track remote branch... i'm still getting the wrong branches despite that...
Any definitive solution? this is a wonderful tool; and this has ruined it for me...
More info:

The set-upstream command is used to track changes between your remote branch and your local branch:
git  --set-upstream
#to view your current remote branch:
$ git remote -v
Displays:

origin  https://github.com/OWNER/REPOSITORY.git (fetch)
origin  https://github.com/OWNER/REPOSITORY.git (push)
destination  https://github.com/FORKER/REPOSITORY.git (fetch)
destination  https://github.com/FORKER/REPOSITORY.git (push)

#to remove your origin branch:
$ git remote rm destination
Set a new remote
$ git remote add origin https://github.com/user/repo.git

@wasims1 Sorry for the bad experience!
You may explicitly specify the head and base branches like so:
hub pull-request --base main -head mybranch


cmd/compile: confusing error: cannot use x (variable of type []string) as []string value

What version of Go are you using (go version)?
Same for go 1.18 to 1.20.1
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
n/a
What did you do?
type Set[T comparable] map[T]struct{}

func (s *Set[string]) String() string {
	keys := make([]string, 0, len(*s))
	for k := range *s {
		keys = append(keys, k)
	}
	sort.Strings(keys)             // cannot use keys (variable of type []string) as []string value in argument to sort.Strings (!!!)
	return strings.Join(keys, ",") // cannot use keys (variable of type []string) as []string value in argument to strings.Join (!!!)
}

https://go.dev/play/p/YhkT3b691OB
What did you expect to see?
cannot use keys (variable of type []comparable) as []string value in argument to sort.Strings

instead
What did you see instead?
cannot use keys (variable of type []string) as []string value in argument to sort.Strings

ps: I'm now aware that using Set[string] in that context doesn't specialize the Set to string, but this was super confusing
An even better error message could be in the receiver to say something
Set[string] should be Set[T comparable] or some such as while string isn't a reserved keyword (sadly in this case) it's very confusing

The string in the method is a type parameter. The whole method declaration body is equivalent to
func (s *Set[T]) String() T {
	keys := make([]T, 0, len(*s))
	for k := range *s {
		keys = append(keys, k)
	}
	sort.Strings(keys)             // cannot use keys (variable of type []T) as []string value in argument to sort.Strings (!!!)
	return strings.Join(keys, ",") // cannot use keys (variable of type []T) as []string value in argument to strings.Join (!!!)
}

Yes I realized as much after scratching my head and asking for help, but that doesn't make the compiler message helpful as it is

Please see https://go.dev/doc/faq#types_in_method_declaration .
But, yes, perhaps the compiler can give a better error message if the error has two different types with the same name.

CC @griesemer

Someone suggested to use [type] instead of type for that error / generic type names, that would have given me a better hint (though [comparable] would have been even better)

Agreed that the compiler should do a better job in cases like these.

@ldemailly FWIW I don't think that using [T] instead of T for type parameters would be a good fix, as you get the same problem if you do type string int. Though arguably that would be less confusing, as it is less obvious that the type parameter name introduces a new type, while type string int is pretty abundantly clear. Either way, I also don't think it's obvious that [T] refers to a type parameter and [][string] most definitely looks confusing.
IMO it would be a better fix if the compiler would say something like can't use []string (with string definde in foo.go:42:23) as []string (with string predeclared) or something, if the stringified type names are equal. No idea how complex that'll be. I'm sure someone can figure something out that's clear.

Thx for super quick fix! one question though, I am not super clear from the mr what the new error message looks like?

For the program in the playground it will now report:
./x.go:15:15: cannot use keys (variable of type []string /* with string declared at ./x.go:10:14 */) as []string value in argument to sort.Strings
./x.go:16:9: cannot use strings.Join(keys, ",") (value of type string) as string /* with string declared at ./x.go:10:14 */ value in return statement
./x.go:16:22: cannot use keys (variable of type []string /* with string declared at ./x.go:10:14 */) as []string value in argument to strings.Join

It's not the most beautiful error message but it hopefully makes it clear which of the string types is user-defined. This is a rare case, after all, and pointing out that something unusual is happening should help a lot.
More generally, whenever a type parameter name matches a name of a predeclared type or object (even true or false) we get the special annotation. Again, usually programs won't use names of predeclared objects in situations like these.
(The new error message should actually be present in the dev playground, but it doesn't appear to be up-to-date at the moment.)


Docs

Yeah... gotta add a few docs

Codecov Report

Merging #264 into master will increase coverage by 0.64%.
The diff coverage is 93.35%.

@@            Coverage Diff             @@
##           master     #264      +/-   ##
==========================================
+ Coverage   74.41%   75.06%   +0.64%
==========================================
  Files          97       97
  Lines        8009     8177     +168
  Branches     1409     1409
==========================================
+ Hits         5960     6138     +178
+ Misses       1578     1570       -8
+ Partials      471      469       -2

Impacted Files
Coverage Δ

AngouriMath/Convenience/AngouriMathExtensions.cs
37.93% <ø> (ø)

...gouriMath/Convenience/StringToEachClass.Classes.cs
62.50% <ø> (ø)

AngouriMath/Convenience/SympySyntax.cs
0.00% <ø> (ø)

AngouriMath/Core/Domains.Classes.cs
12.19% <ø> (ø)

AngouriMath/Core/Domains.cs
89.65% <ø> (ø)

...gouriMath/Core/Entity/Continuous/Number/Complex.cs
77.47% <ø> (+1.37%)
:arrow_up:

...uriMath/Core/Entity/Continuous/Number/Operators.cs
86.27% <ø> (ø)

...Core/Entity/Discrete/Entity.Discrete.Definition.cs
95.23% <ø> (ø)

AngouriMath/Core/Entity/Entity.Definition.cs
90.00% <ø> (+6.66%)
:arrow_up:

AngouriMath/Core/EquationSystem.cs
62.50% <ø> (ø)

... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8afb7a8...c5a5f9c. Read the comment docs.


Remove problematic covariates on the fly

This PR adds a script in step 1, which does the following:

For binary covariates, if the covariate has only one variate (e.g. sex imputed being only 1 for a female-specific phenotype), it will be removed from the list of covariates. Quantitative endpoints are passed through. NAs in covariates will not count towards inclusion. Missing endpoints (e.g. PC{1:10} shorthand) will be passed through. For multiple phenotypes, the rows are counted like they would be if the phenotypes' missing values were imputed, i.e. only if all phenotypes have NA, that row will not be counted.
The covariates are passed on to step2, and are included as step1 outputs.

There is currently no logging of e.g. how many covariate 0/1 there were for each covariate, but that can be added if helpful.

Added printing covar name, one count, zero count, mask status to stderr. Works on local testing.


perf: cache view methods

These are nicely cacheable, and don't have to be part of the class.

codegen         mean runtime:  221.53ms, runs:   255.83,  215.28,  197.83,  201.14,  219.83,  205.83,  206.70,  220.71,  198.30,  293.85
methodcache     mean runtime:  165.73ms, runs:   210.85,  159.71,  155.94,  158.69,  169.23,  158.60,  165.78,  150.29,  175.99,  152.22
profile         mean runtime:  916.79ms, runs:   805.64,  902.98,  891.13,  902.14,  938.34,  939.29,  997.60,  923.44,  923.32,  944.03

to

codegen         mean runtime:  200.05ms, runs:   270.67,  186.91,  177.84,  172.52,  183.54,  196.58,  184.18,  249.55,  179.50,  199.21
methodcache     mean runtime:  160.23ms, runs:   151.52,  160.38,  148.77,  151.17,  149.59,  150.05,  165.42,  156.59,  202.07,  166.77
profile         mean runtime:  731.61ms, runs:   698.33,  721.21,  721.92,  691.91,  721.54,  821.83,  780.41,  736.11,  709.44,  713.36

No idea why the diff is shot, I de-indented and added a cache decorator.

Changes made in tinygrad/:
------------------------------------------------------------
files                             insertions       deletions
------------------------------------------------------------
tinygrad/shape/shapetracker.py            30              27
------------------------------------------------------------
lines added in the tinygrad folder: 3

Is this more or less readable?

This is fine, the github diff makes it seem really bad. If you check in codespaces or in vscode it renders the proper diff.

There's a slightly faster version of this, where I pass only the actual parts of the view that are used,to each function. Better cache hits but that's a bit less readable.

Test failures seem to be the failing model download.

its the openpilot one, we should probably try to cache it in this repos actions cache.

Less readable


Update

Coverage remained the same at 8.113% when pulling 540bf8b1ccf8d1fcd52851aef60e3c90353d2459 on update into 4b793af23bd157925baf74e4aa89354b518152ab on master.


[PF2 Tooltip] option to remove colors from icons, or standartize them with one color

PF2 Tooltip Info panels look too crass with various colors in a non-standardized way, which causes OCD triggers. Is there a chance to get a monochrome or standardized one color for the header icons, rather than the full rainbow of various colors.

Included in 2.3.5 - default is monochrome, with a setting to enable colorized


Fix Windows Close

fix #328
Windows was producing two windows, which meant that the windows-all-closed event wasn't being thrown.

Ty ❤


Out of memory when running spring tests

Hello.
I've tried last version of im-aop-loggers on a Java17, SpringBoot 2.6.0 version project and during the spring boot tests I got out of memory.

stacktrace or didn't happen 😀

Hello.
I've tried last version of im-aop-loggers on a Java17, SpringBoot 2.6.0 version project and during the spring boot tests I got out of memory.

Hi @lucimihai , could you share the code of your tests?

I did some additional investigations. The memory issue is not related to im-aop-loggers lib. I developed custom aspects and the memory still occurred. So I think is more related to AspectJ or Spring AOP.
I managed to solve it by configuring Xmx=1GB for spring boot integration tests.
So I'm closing the issue.


"Stream Quest Microphone"

So I've got this checked, and I use vb-cable frequently so I know how all the audio routing works, but I can't seem to actually get this setting to do anything. Is there any guide explaining how I can use the quest microphone as my mic in game?
Excellent work on this ALVR project. I was playing some pavlov earlier and the quality is just unrivaled. Truly a great experience.

the way i got it to work was.
open your audio settinig on window 10 and selected the input as cable output. make sure the Cable input is not disable. (it does not need to be selected. )

closed due to inactivity


use_treesitter_scope breaks show_current_context

Hello, I tried testing use_treesitter_scope with today's update but it breaks show_current_context. Any clues?

How does it break?

Never mind, found it. I released a fix


Anyway, I figured out you can set up environment variables for your cloud environment, of course these were not being used since the plugin only looks for .env.* and throws when importing something that has not been set.
On the other hand, from dotenv rules:

What happens to environment variables that were already set?
We will never modify any environment variables that have already been set. In particular, if there is a variable in your .env file which collides with one that already exists in your environment, then that variable will be skipped.

So, what do you think about, before checking presence of the variable in the dotEnv config result, check if process.env already has a variable set for that name and replace it with that instead of the .env file, replicating the behavior of dotenv package.
edit: I guess this is related to with #32

I found a temporary solution:
Create and commit a .env.example file that has any env variables required for testing(also you could do .env.test)
In your testing environment run this command first or at least before you npm install
cp .env.example .env
This creates a new .env file with the exact contents of .env.example
You still shouldn't store secret keys your .env.example file but with this you can store things like api urls, endpoints, public keys, and anything else that's okay being public and needed for testing.


Should we use fixed-width types (e.g. stdint) exclusively?

Should we use fixed-width types (e.g. stdint) exclusively or favor primitive types unless a size-guaranteed type is required for the particular exercise?

I personally feel that fixed width types (almost) always make the most sense. So my 2 cents is yes, we should.

I agree. The only real downside is adding some confusion for people new to the language, but it will be mostly hidden from them anyways.


RDM-1547: updated api with pagination metadata endpoint

the page is passed as a query parameter, so it's inside the searchCriteria map

The build is failing for checkstyle issues


OpenMap(URL) remove faulty webmap from sample

Description

Remove the webmap for the world-terrestrial-ecosystems as it's not displaying correctly
Update the sample image and metadata

Checklist:

[ ] Set target branch to main (current release) or v.next (next release)
[ ] Request a reviewer
[ ] Assign a reviewer
[ ] Tag reviewer in comment

Hi @jenmerritt, can you do a first review please?

Ready for a second look @Rachael-E, thanks.


Extra mounted volumes seem to be ignored by deploy_docker.py

Hi, and thanks for this great tool!
I would like to deploy DSA on Microsoft Azure, reading data from Azure blob storage and overlaying annotations.
For now, I'm just trying to deploy locally using deploy.sh. I have found that whatever I pass using --mount can't be found in the container. When I exec into it, I can't find it. I have a folder ~/.histomics_data that is successfully mounted on /data. I was wondering why the volumes mounted by deploy.sh were working, but not the ones I passed myself. I replaced /data with another path (/datum), but I couldn't find a /datum folder in the container. Moreover, /opt/digital_slide_archive/mounts is always empty, which seems inconsistent with the docs for --mount:
https://github.com/DigitalSlideArchive/digital_slide_archive/blob/b9f245401aaf6017338ca8d01c16eec6df306a0f/ansible/deploy_docker.py#L959-L964
What does ro mean in the help message?
Am I doing anything wrong?
I hope this all makes sense. I'm don't have much experience with Docker or web applications. Please let me know if you need more information or if I you would like me to contribute with a PR.

Additionally, and please let me know if I should open a new issue for this, mounted folders and symbolic links in ~/.histomics_data don't seem to work either. Is this expected? I can give you more details about what I tried, if necessary.

The mount command can is used like in standard docker run commands and is of the form: --mount <path on your machine>:<path inside the docker container OR a simple name>:ro, where the path inside the docker container portion and the ro option are both optional.  ro makes the mount read-only (so it can't be used to store new data).  The path inside the docker container can either be a complete path (e.g., /data or /mnt/sample_data) or a single name which will be used internally as /opt/digital_slide_archive/mounts/<name>, or, if not specified, the name is auto-assigned as mount1, mount2, etc.
If you show the full command you are trying, I can advise what needs to be changed.  I imagine you want to do something like --mount /home/user/.histomics_data:/data.

The mount command can is used like in standard docker run commands and is of the form: --mount ::ro, where the path inside the docker container portion and the ro option are both optional. ro makes the mount read-only (so it can't be used to store new data). The path inside the docker container can either be a complete path (e.g., /data or /mnt/sample_data) or a single name which will be used internally as /opt/digital_slide_archive/mounts/, or, if not specified, the name is auto-assigned as mount1, mount2, etc.

Thanks for the explanation, that's very clear.

If you show the full command you are trying, I can advise what needs to be changed. I imagine you want to do something like --mount /home/user/.histomics_data:/data.

I guess some code is worth a thousand words:
$ mkdir myfolder

$ realpath myfolder
/home/fernando/git/digital_slide_archive/myfolder

$ devops/deploy.sh --mount /home/fernando/git/digital_slide_archive/myfolder:/mymounted start
Starting rabbitmq:management - dsa_rabbitmq
Starting mongo:latest - dsa_mongodb
Starting memcached:latest - dsa_memcached
Starting dsarchive/dsa_worker:latest - dsa_worker
Starting dsarchive/dsa_girder:latest - dsa_girder

$ docker exec -it dsa_girder bash -c "ls /"
bin   data  etc   lib    lib64   media  opt   root  sbin  sys  usr
boot  dev   home  lib32  libx32  mnt    proc  run   srv   tmp  var

As you can see, mymounted is missing from / in the container. In WSL, /data is also missing.

Hmm.. Did you stop and remove the containers before running the start command with the extra mount parameter?  Docker doesn't add mounts to running containers.

Yes, I ran devops/deploy.sh stop before running the lines above.
Tested on Docker 19.03.6, Docker 20.10.11+azure-3 and Docker 20.10.7 (WSL).

stop isn't sufficient -- it stops the running containers, but doesn't remove them.  start will then restart the stopped containers without adding new mounts.  rm will stop and remove the containers, after which start will recreate them from the docker images with all of the mount options.

rm before mounting worked. Thanks! I'll open more issues regarding some other questions.

For completeness, I ran devops/deploy.sh rm before trying again.


TypeError: Cannot read property 'source' of undefined

My next version is 2.0.0. I'm trying to run next build but I got this error message. Because of the failure of building. It doesn't generate the .next/ directory and I can't run next start to deploy.
(node:75113) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'source' of undefined
(node:75113) DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Can you provide repo to reproduce issue?

This the error I got. Does this look familiar @Larry850806? https://github.com/zeit/next.js/issues/1545

@sedubois OK, I trying to make a repo to reproduce this issue.

@timmywil @sedubois Here!!
https://github.com/Larry850806/next-error-reproduce
Clone this repo and run yarn && yarn run build to install the dependencies and build. This is my package.json
{
  "name": "Testing",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "next": "^2.0.0",
    "react": "^15.4.2",
    "react-dom": "^15.4.2"
  }
}

@Larry850806 thanks for the repo.
I just did a fix for this: https://github.com/zeit/next.js/pull/1569
Wait for the new release.


Lift requirement that .egg-info filenames must include version

Summary
PR #4533 introduced (almost) spec compliant parsing of .egg-info filenames, but added the overly strict requirement that the distribution version must be present. This causes various uv pip operations to fail in environments where there are .egg-info files without a version component, so loosen this check by making the version component optional and reading the version from the egg metadata when it is not present.
As an example of the issue, running uv pip list on my system currently results in
error: Failed to read metadata from: `/usr/lib/python3.12/site-packages/PySide6.egg-info`
  Caused by: The `.egg-info` filename "PySide6.egg-info" is missing a version

whereas regular pip list succeeds:
$ pip list | rg -S pyside
PySide6                   6.7.2

Test Plan
This has been tested by altering the .egg-info filename tests as needed and ensuring the full test suite passes locally.

Thanks for contributing! This seems reasonable to me, but I'm not the expert on this feature. I've assigned a reviewer and they'll get to this shortly.


Is there auto build & packaging script for all ppa's files?

I want to build & package all ppa's files myself.
Is there auto build & packaging script for all ppa's files (pipeline configuration), or some instructions?

No need to generate packages for old builds head to https://launchpad.net/~pipewire-debian/+archive/ubuntu/pipewire-upstream/+packages?field.name_filter=pipewire&field.status_filter=&field.series_filter=focal  download *.deb packages in a directory install them with dpkg -i *.deb

Such a script would be helpful to build newer versions of pipewire. This PPA is very helpful, thanks a lot.

Closing as Completed


Move code from hpp to cpp

Can you tell for which module and why?
Having header only files is not really a problem.
It can just make compile time longer.. but make module file easier to use for third party projects..

Yes, the goal is to reduce the build time especially when you are developing. Coding many stuff in headers invalidate a lot of code for nothing. For third party usage, we already have compiled modules, so IMHO it will not change much.
Why is it a problem to have more builded modules?

It's not a problem but you know some third party like Eigen are header only and efficient.
But an analysis of which header files make the more compilation time would be great.
Did you know any tool(s) that allow to know which headers are the bottleneck?

Yes, Eigen doesn't require linking and that's useful because it simplifies library usage. But having an header only library is not the ambition of openMVG, and having 3 or 7 compiled modules doesn't really change much for 3rd party usage. On the other hand, having a good split between hpp and cpp can reduce a lot the compile time for all openMVG developpers.
I don't know any tool for global build statistics. But there are small tools to help cleaning the code to reduce includes propagation. I have used include-what-you-use and cppclean a long time ago.
Why IWYU?

Just for the sake of the argument, the modules that are header only (and so a new module definition is required) are:

cameras [numeric, geometry, stl]
color harmonization [linearProgramming, image, matching, features]
geometry [multiview, linearProgramming, robust_estimation, numeric ]
graph
robust estimation [numeric, features, matching, cameras]
stl
tracks [matching]

The [ ] are the internal module dependencies (got it using a simple grep include, maybe not 100% reliable)
@pmoulon Is/was there a criterium on which you decided whether to make it hpp only or not? That would help to better understand the needs.
Also i created the graph of the  module dependencies in which I added the above ones (red ellipses). I lacks of the dependencies of the existing modules towards the header only ones though.

dep.pdf
with an image it's maybe better

My criteria was (reasonable):

if the module use template only method, => header only module,
if the module is so tiny that it provides only some basic functionality that can be seen as independent from openMVG (like stl extensions) => header only module.

My ultimate goal is to make OpenMVG powerful, understandable and easy to use. And it don't consists in making a complex system to use and build.
As you said @simogasp each contributor have it's own idea of what the library should be...
My initial intuition was to make each module as independent as possible to make it's usage the more spreadable as possible.
As said on the main openMVG page:
The openMVG credo is: "Keep it simple, keep it maintainable". OpenMVG targets readable code that is easy to use and modify by the community.
Note: Boost & Eigen are two library that have shown that header only library is not a problem even in complex applications.
My concern is to focus on features and usability and not on compilation time ;-)

I'm really surprised by your reaction, the idea was to clean the code progressively to make it more consistent, to reduce compilation time and to "keep it maintainable". I understand that you prefer to focus on features instead of doing this boring job, but we propose to contribute on that. I disagree that it's over-engineering. Cleaning the code progressively and defining clear rules is important to help all contributors to navigate through the code.
Cleaning include dependencies becomes harder and harder over time because all files get a lot of indirect includes and they succeed to compile due to all those indirect includes instead of declaring them explicitely.
The other point behind this issue was to remove all the "static" methods from headers. "static" keyword means internal linkage (only accessible in one translation unit), so it should not be used in header files because all those functions are duplicated in each translation unit.
I agree on the ambition of making OpenMVG powerful, understandable and easy to use. The ambition of this task was to contribute on that. But if you are convinced that keeping all the code in header files is a good thing, we can keep it as is.

I just tried to put openMVG_tracks and openMVG_geometry to cpp files and the build time remains exactly the same.

Hi,
Just to jump into the conversation, there is a gcc option "-ftime-report" that can give you statistics about the times and resources spent during the different stages of the compilation. I ran a compilation with this option and the results are enclosed in the file compilation_times.txt.zip
compilation_times.txt.zip
.
Even if I am not totally sure how to interpret correctly the numbers, the template instanciation part seems quite high for some files, this can give you some hints on where to freshen up the code if you want to.
I found a good explanation about slow compilation time on stackoverflow:
http://stackoverflow.com/questions/28919285/identify-slow-to-compile-function
Hope that helps,

Thanks @cpichard
It will help to see which files take the more time to be compiled.
It would be interesting to list the files that take the longer time to be compiled.
Perhaps the main compilation bottleneck is when numeric.h is included somewhere.


[docs]: link shows 404

Hi.
In this file, the "Documentation" text, links to a 404 page.

Hey @babakfp you must have caught that page at the perfect time just as the website was being redeployed. It's now loading as expected 👍 thank you for pointing it out.


Fix: Ensure consistent dates for G-Sheets export

This is a fix for https://github.com/brafdlog/caspion/issues/476 ("google spreadsheet dates inconsistent cell format")
The fix is to  use the format of YYYY-MM-DD which has no ambiguity and Google Sheets will automatically detect it as a date field.
I did not see any unit-testing in this project so added no tests. If I am missing the tests, I will add.

I see the test suite now... missed it somehow. I can add tests, let me know if that's a prerequisite as there are no tests for Google Sheets exporter today.


Request: Map files to a project

Can I map single files to a project? I can map (alias) projects to a project but haven't found an option to do that for files. I have several files that belong to an actual project, but count toward "unknown" project. I forgot to setup the .wakatime-project file for these projects when starting out.
Thank you!

Hi @fusion44, unfortunately, this is neither currently supported, nor planned in the future. With a self-hosted instance, you could accomplish this with a quick database query. Or, if you're using Wakapi.dev, I can offer to perform this one-time replacements for you, if you send me an e-mail with the according information.

I'm using a self hosted instance. I'll look into it. Thank you!

It will probably be something like:
update heartbeats set project = 'foo' where entity like ''%/foo%.js and user = 'your-user';

Followed by a Regenerate summaries process triggered on the Settings page.

@fusion44 did you have any success with this approach?

@gingerbeardman Sorry, I've never tried it. Ultimately the amount of time affected didn't matter in the grand scheme of things.


New internal functions

We need to make it so that internal functions can take in any so we don't need to write individual functions for everything.

We could add generics

Yeah, let's do that in this branch too, I guess you can work on that, and ill work and new internal functions.

How did you commit something from __pycache__? https://github.com/nbuilding/N-lang/pull/23/files/5d51a2e5efc011e8977e9a4c217357a1cbb6eed9..a3dddeb7b50ba6f4ae24df9d99bacca37c5063b3#diff-a7608e0f32a65611d198e7406121c7a392052ccd8a5ff73b56d05604d05c8987

How did you commit something from __pycache__? https://github.com/nbuilding/N-lang/pull/23/files/5d51a2e5efc011e8977e9a4c217357a1cbb6eed9..a3dddeb7b50ba6f4ae24df9d99bacca37c5063b3#diff-a7608e0f32a65611d198e7406121c7a392052ccd8a5ff73b56d05604d05c8987

When'll we merge this?

When'll we merge this?


python312Packages.llama-parse: 0.4.0 -> 0.4.1, python312Packages.llama-index-readers-file: 0.1.17 -> 0.1.18

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 304190 run on x86_64-linux 1

20 packages built:

python311Packages.llama-index
python311Packages.llama-index-readers-file
python311Packages.llama-index-readers-file.dist
python311Packages.llama-index-readers-llama-parse
python311Packages.llama-index-readers-llama-parse.dist
python311Packages.llama-index-readers-s3
python311Packages.llama-index-readers-s3.dist
python311Packages.llama-index.dist
python311Packages.llama-parse
python311Packages.llama-parse.dist
python312Packages.llama-index
python312Packages.llama-index-readers-file
python312Packages.llama-index-readers-file.dist
python312Packages.llama-index-readers-llama-parse
python312Packages.llama-index-readers-llama-parse.dist
python312Packages.llama-index-readers-s3
python312Packages.llama-index-readers-s3.dist
python312Packages.llama-index.dist
python312Packages.llama-parse
python312Packages.llama-parse.dist


Add custom delimiter to Import-DbaCsv

Summary of new feature
I will be great to be able to have be able to use custom delimiters in CSV files.

Clear/concise description of what the problem is that the new feature can solve.
I have a process that need to import big TXT file with a custom (2-char) delimiter ^!

Proposed technical details (if applicable)
-Delimeter ^! to Import-DbaCsv

e.g. Blog post reference that shows example code or functionality

Latest version of dbatools as of writing
Latest

Please provide the latest released version of dbatools module.

This is restricted by what the LumenWorks library allows...

Yeah, Lumenworks only allows a char as seen here: https://github.com/phatcher/CsvReader/blob/master/code/LumenWorks.Framework.IO/Csv/CsvReader.cs#L36
I'll have to rewrite the whole command with a new library to accommodate a lot of our requests, it's on my agenda but it's pretty tough to do, so it may be a bit.


Enhancements to the book [Getting Started] section

It might be better to just add the :nrepl    {:port 9000} as a sensible default in the configuration section like so
{:deps     {:aliases []}
 :nrepl    {:port 9000}
 :jvm-opts ["-Xmx2G"]
 :builds   {:book {:target        :browser
                   :output-dir    "docs/js/book"
                   :asset-path    "/js/book"
                   :module-loader true
                   :modules       {:book    {:entries [book.main]
                                             :init-fn book.main/init}
                                   ; For the dynamic code splitting demo
                                   :ui-main {:entries    [book.demos.dynamic-ui-main]
                                             :init-fn    book.demos.dynamic-ui-main/init
                                             :depends-on #{:book}}}
                   :devtools      {:preloads  [com.fulcrologic.fulcro.inspect.preload]
                                   :http-root "resources/public"
                                   :http-port 9005}}}}

Mention and link Cursive for IntelliJ setup.


Amqp over websocket is not working and leeds to a connection error

Amqp over websocket connection error
UAMQP Package Version: 1.6.3
Operating System: Windows / WSL / Docker
Python Version: 3.8, 3.9, 3.10
Describe the bug:
Connection to IoTHub failed with transport type:  amqp.TransportType.AmqpOverWebsocket at versions > v1.2.13
Currently using the version: v1.2.13 (the last version where the amqp over ws works for me) and trying to update to v1.6.3
The websocket connection is needed because some firewalls are blocking the plain amqp connection
This behavior is the same on multiple platforms:

x86_x64
Armv7 (Raspberry 3b+)
Armv8 (Raspberry 4)

The same error ocurred with the async and sync client.

Logg message
Logs
  2023-01-16 10:11:32,134 uamqp.c_uamqp DEBUG    Creating SASL Mechanism
  2023-01-16 10:11:32,138 uamqp.client DEBUG    Opening client connection.
  2023-01-16 10:11:32,138 uamqp        DEBUG    Initializing platform.
  2023-01-16 10:11:32,138 uamqp.connection INFO     Connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0>
  2023-01-16 10:11:32,140 uamqp.c_uamqp DEBUG    Wrapping value type: <AMQPType.CompositeType: 23>
  2023-01-16 10:11:32,141 uamqp.c_uamqp DEBUG    Wrapping value type: <AMQPType.CompositeType: 23>
  2023-01-16 10:11:32,142 uamqp.sender INFO     Message sender b'sender-link-2aaa5d30-aa2d-4a79-a028-ee3168794609' state changed from <MessageSenderState.Idle: 1> to <MessageSenderState.Opening: 2> on connection: b'7475125f-f01a-4a3a-936c-7fdc8c69f607'
  2023-01-16 10:11:32,224 uamqp.connection INFO     Connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0>
  2023-01-16 10:11:32,553 uamqp.c_uamqp INFO     b'wsio_close when not open.' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\deps\\azure-c-shared-utility\\src\\wsio.c':b'internal_close':153)
  2023-01-16 10:11:32,554 uamqp.connection INFO     Connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607' state changed from <ConnectionState.START: 0> to <ConnectionState.END: 13>
  2023-01-16 10:11:32,554 uamqp.connection INFO     Connection with ID b'7475125f-f01a-4a3a-936c-7fdc8c69f607' unexpectedly in an error state. Closing: False, Error: None
  2023-01-16 10:11:32,555 uamqp.sender INFO     Sender link failed to open - expecting to receive DETACH frame.
  2023-01-16 10:11:32,555 uamqp.connection WARNING  ConnectionClose('ErrorCodes.UnknownError: Connection in an unexpected error state.')
  2023-01-16 10:11:32,556 uamqp.c_uamqp DEBUG    Destroying cMessageSender
  2023-01-16 10:11:32,556 uamqp.c_uamqp DEBUG    Destroying cLink
  2023-01-16 10:11:32,557 uamqp.c_uamqp DEBUG    Deallocating 'CompositeValue'
  2023-01-16 10:11:32,557 uamqp.c_uamqp DEBUG    Destroying 'CompositeValue'
  2023-01-16 10:11:32,557 uamqp.c_uamqp DEBUG    Deallocating 'CompositeValue'
  2023-01-16 10:11:32,558 uamqp.c_uamqp DEBUG    Destroying 'CompositeValue'
  2023-01-16 10:11:32,558 uamqp.c_uamqp DEBUG    Deallocating cLink
  2023-01-16 10:11:32,558 uamqp.c_uamqp DEBUG    Deallocating cMessageSender
  2023-01-16 10:11:32,559 uamqp.client DEBUG    Closing non-CBS session.
  2023-01-16 10:11:32,559 uamqp.c_uamqp DEBUG    Destroying cSession
  2023-01-16 10:11:32,560 uamqp.c_uamqp DEBUG    Deallocating cSession
  2023-01-16 10:11:32,560 uamqp.client DEBUG    Closing exclusive connection.
  2023-01-16 10:11:32,560 uamqp.connection DEBUG    Unlocked connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607' to close.
  2023-01-16 10:11:32,561 uamqp.connection INFO     Shutting down connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607'.
  2023-01-16 10:11:32,561 uamqp.c_uamqp DEBUG    Destroying Connection
  2023-01-16 10:11:32,561 uamqp.c_uamqp INFO     b'send called while not open' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\src\\saslclientio.c':b'saslclientio_send_async':1181)
  2023-01-16 10:11:32,562 uamqp.c_uamqp INFO     b'Cannot send encoded bytes' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\src\\connection.c':b'on_bytes_encoded':268)
  2023-01-16 10:11:32,562 uamqp.c_uamqp INFO     b'saslclientio_close called while not open' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\src\\saslclientio.c':b'saslclientio_close_async':1130)
  2023-01-16 10:11:32,562 uamqp.c_uamqp INFO     b'xio_close failed' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\src\\connection.c':b'on_bytes_encoded':272)
  2023-01-16 10:11:32,563 uamqp.connection INFO     Connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607' state changed from <ConnectionState.END: 13> to <ConnectionState.END: 13>
  2023-01-16 10:11:32,563 uamqp.c_uamqp INFO     b'-> [CLOSE]* {}'
  2023-01-16 10:11:32,563 uamqp.connection INFO     Connection b'7475125f-f01a-4a3a-936c-7fdc8c69f607' state changed from <ConnectionState.END: 13> to <ConnectionState.END: 13>
  2023-01-16 10:11:32,563 uamqp.c_uamqp INFO     b'saslclientio_close called while not open' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\src\\saslclientio.c':b'saslclientio_close_async':1130)
  2023-01-16 10:11:32,564 uamqp.c_uamqp INFO     b'xio_close failed' (b'D:\\a\\1\\s\\src\\vendor\\azure-uamqp-c\\src\\connection.c':b'connection_close':1437)
  2023-01-16 10:11:32,564 uamqp.c_uamqp DEBUG    Destroying XIO
  2023-01-16 10:11:32,564 uamqp.c_uamqp DEBUG    Destroying XIO
  2023-01-16 10:11:32,565 uamqp.c_uamqp DEBUG    Destroying SASLMechanism
  2023-01-16 10:11:32,565 uamqp.connection INFO     Connection shutdown complete b'7475125f-f01a-4a3a-936c-7fdc8c69f607'.
  2023-01-16 10:11:32,565 uamqp        DEBUG    Deinitializing platform.

System / dependency's
WSL
Test inside docker container with different build environments:

Dockerfile: python 3.8
Dockerfile: python 3.9

RaspberryPi 4 / RaspberryPi3b+
Test inside docker container with different build environments:

Dockerfile: python 3.8
Dockerfile: python 3.9

The test where also made using the install openssl install script
(https://github.com/Azure/azure-uamqp-python/blob/main/utils/install_openssl.sh)

FROM python:3.8
RUN apt-get update
RUN apt-get install -y build-essential libssl-dev uuid-dev cargo curl cmake libcurl4-openssl-dev pkg-config
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
RUN pip install -U pip \
    && pip install -r requirements.txt \
    --no-binary :all:

Windows:
Python
on Windows:

python 3.8
python 3.9
python 3.10

pip

pip list (venv)

certifi    2022.12.7
pip        22.3.1
setuptools 56.0.0
six        1.16.0
uamqp      1.6.3

To Reproduce

Create a IoTHub instance (currently using Baltimore CyberTrust root but also tested with DigiCert Global G2)
Add a device and get the connection string
Try to connect and send a message with uamqp.SendClient(uri, debug=True, transport_type=uamqp.TransportType.AmqpOverWebsocket) (see example)

Example code
Send to IoTHub
logging.basicConfig(level=logging.DEBUG,
                  format='%(asctime)s %(name)s %(levelname)s >> %(message)s',
                  datefmt='%Y-%m-%d %H:%M:%S')

def generate_connection_uri(config):
  """
  Generates the amqp enpoint url depending on the provided connection string
  :return: amqp endpoint
  """
  username =<EMAIL_ADDRESS>      device_id=config["device_id"],
      iot_hub_name=config["hub_name"])

  # Create token
  ttl = int(time.time()) + 3600
  uri = '{hostname}/devices/{device_id}'.format(
          hostname=config["hub_address"],
          device_id=config["device_id"])
  url_to_sign = urllib.parse.quote(uri, safe='')
  hash_auth = hmac.new(
      base64.b64decode(config["device_key"]),
      msg="{0}\n{1}".format(url_to_sign, ttl).encode('utf-8'),
      digestmod='sha256'
  )
  sas_token = "SharedAccessSignature sr={0}&sig={1}&se={2}".format(
      url_to_sign,
      urllib.parse.quote(base64.b64encode(hash_auth.digest()), safe=''),
      ttl
  )

  operation = '/devices/{device_id}/messages/events'.format(device_id=config["device_id"])
  uri =<EMAIL_ADDRESS>      quote_plus(username),
      quote_plus(sas_token),
      config["hub_address"],
      operation)
  return uri

def send_iot_hub(config):
  data = b"test message"
  message = uamqp.Message(data)
  uri = generate_connection_uri(config)

  client = uamqp.SendClient(uri, debug=True, transport_type=uamqp.TransportType.AmqpOverWebsocket)
  client.queue_message(message)
  states = client.send_all_messages()
  assert not [state for state in states if state == MessageState.SendFailed]

if __name__ == '__main__':
  config = {}
  REG_EXP_CONNECTION_STRING = r"^HostName=(.*)\.azure-devices.net;DeviceId=(.*);SharedAccessKey=(.*)$"
  m_obj = re.match(REG_EXP_CONNECTION_STRING, os.environ['CONNECTION_STRING'])
  config['hub_address'] = m_obj.group(1) + ".azure-devices.net"
  config['hub_name'] = m_obj.group(1)
  config['device_id'] = m_obj.group(2)
  config['device_key'] = m_obj.group(3)
  print(config)

  send_iot_hub(config)

Expected behavior
Expected would be a amqp connection over websocket to the IoTHub like with plain amqp (no connection errors)

It looks like the default port for amqp over websocket is not handled correctly.
It is possible to set the port in the auth (SASLPlain) object directly.
That is the workaround that i now use:
    auth = uamqp.authentication.SASLPlain(
        hostname="<hostname>",
        username="<username>",
        password="<password>",
        port=443,
        transport_type=uamqp.TransportType.AmqpOverWebsocket
    )
    client = uamqp.SendClient(uri, auth, debug=True, transport_type=uamqp.TransportType.AmqpOverWebsocket)

Closing this issue as the PR is merged. Thank you @MkKiefer


gpus not accessible on gpu nodes

I've reported this to schedmd at https://bugs.schedmd.com/show_bug.cgi?id=7901, but I'm posting it here as well to alert to the problem and to also solicit any advice for other approaches.

This issue is a result of recent updates to CentOS from 7.6 to 7.7. Google has not updated the base CentOS image yet. The inconsistency between the CentOS kernel version  (957) and the kernel-devel and kernel-headers (1062) leads to a failed install of Nvidia drivers.
We will push up a patch to this repository as soon as we can.

Thank you very much for your prompt and helpful response. This is very good for the public image of FluidNumerics.

We accept pull requests on this repository if you are able to get to it before we can.

Very much understood. I spent a few days trying to diagnose this issue having assumed the issue lay with my local modifications before I tested the upstream repositories sufficiently. The install scripts from Nvidia seem to silently fail until you try to manually install the driver by itself (before installing cuda). I think that's the way to get some reliability here, but I don't actually know how to work around this at the moment. I'll update if I get time free to look again.
Essentially if this can run with exit code 0, it should be actually install the drivers correctly. This script even has a way to specify the kernel source directories, but I'm really not sure if that fixes the issue or if the incompatibility is worse than just a file path issue.
wget http://fr.download.nvidia.com/tesla/410.79/NVIDIA-Linux-x86_64-410.79.run
sh NVIDIA-Linux-x86_64-410.79.run --silent

The 418.67 driver install script should behave the same way.

I've not had the time to look at this yet. Any progress from your side by chance?

The issue is due to the CentOS 7 image on Google having an incompatible kernel with the kernel-devel and kernel-headers available from rpm. Once the image is updated to 7.7 this will be resolved.

I believe I've been able to work around this issue by preparing a custom centos 7.7 image and using that in the slurm.jinja file.
Steps to reproduce:

Launch centos 7 VM
Update kernel (yum check-update then yum update) and reboot - unsure how to automate in startup-script with reboot, which is the main blocker from upstreaming this process.
Install nvidia driver and cuda  - driver first while manually specifying the kernel source path

  yum -y groupinstall "GNOME Desktop" "Development Tools"
  # Install Extra Packages for Enterprise Linux (EPEL) and other  dependencies
  yum install -y yum-utils epel-release kernel-devel kernel-headers which gcc dkms wget
  yum-config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo
  echo "Installing kernel headers and development packages."
  [ ! -e NVIDIA-Linux-x86_64-410.79.run ] && wget http://fr.download.nvidia.com/tesla/410.79/NVIDIA-Linux-x86_64-410.79.run
  sh NVIDIA-Linux-x86_64-410.79.run --silent --kernel-source-path="/lib/modules/$(uname -r)/source"
  yum install -y cuda
  grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
  mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img
  dracut /boot/initramfs-$(uname -r).img $(uname -r)
  grub2-mkconfig -o /boot/grub2/grub.cfg
  echo blacklist nouveau >> /etc/modprobe.d/blacklist.conf

Create an image in gcloud GUI from the VM's disk
Use that image in slurm.jinja as sourceImage

Steps 4 and 5 could be ignored if we could reboot in the startup-script in the middle of the setup process.

Another issue was that the default GID for slurm was already used on the gcloud VM when using Centos-7.7. Setting it to a much higher number (like 1337) enabled launch of the cluster.


Wrong texture url in case of more than one texture

If more than one texture is defined, the export is wrong:
texture DEF IM_COLOR_MAP ImageTexture {
  url [ ""toilet/map/stargalaxy granite color.jpg" "stargalaxy granite color.jpg" "/home/david/Téléchargements/toilet/map/stargalaxy granite color.jpg"" ]
}

Fix in #4