Add the ability to support scoped RBAC with a scoped namespace

First of all thanks for this amazing project!
I created this PR first and am looking for some helps on this PR before I go too far off so any assistance is greatly appreciated.
I have an issue building using the main branch as it is very different from the v0.4.4 tag.
This PR is built on top of v0.4.4 and modified based on the changes in the main branch.
Summary
This PR is a follow-up on the scopedNamesapce and is addressing the comment for scoped RBAC roles.
The current implementation of scopedNamespace is using Cluster roles to access Kube API.
There are two major security concerns:

The operator can access ExternalSecret outside the scoped namespace
The operator can alter and read Kube resources such as secrets outside the scoped namespace

To address these, we need to scope the RBAC within the scoped namespace by creating Roles instead of ClusterRoles.
This PR adds a new option in the chart called scopedRBAC which must be used alongside scopedNamespace.
When scopedRBAC is true, it will create RBAC roles under the scoped namespace.
This will limit access to the Kube API within the scoped namespace.
However, there is a side-effect of such scoped RBAC that ClusterSecretStore cannot be supported because it is a cluster scoped resource.

@gusfcarvalho @moolen maybe we should remove the PR limit from the bot? Especially with dependable enabled....

BTW, @kinyat thanks a lot for the PR, and so sorry for this bad experience with the bot closing it :sweat_smile: We will reopen as soon as we sort this out

@knelasevero Thanks for the help 🙏 Any possibility we can bring this forward together? This is something that is blocking us to use your awesome tool 🙂

Hey @kinyat ! Thank you for your contribution!
Your main idea here would be to actually deploy ESO without a ClusterSecretStore CRD deployed, is that correct?
I do believe we can go with this approach if we take into account that no ClusterSecretStore CRD would be installed; however, if they are, I believe we would need to investigate a better way to implement this feature. I don't believe it would work as intended by just not adding the CSS Reconciler.

Hey @kinyat ! Thank you for your contribution!
Your main idea here would be to actually deploy ESO without a ClusterSecretStore CRD deployed, is that correct?
I do believe we can go with this approach if we take into account that no ClusterSecretStore CRD would be installed (we would need still see the best way to do it / inform users to do it); however, if they are, I believe we would need to investigate a better way to implement this feature. I don't believe it would work as intended by just not adding the CSS Reconciler.
Edit:
On a side note, my main issue here would be to not have a reconciler for ClusterSecretStores, but still allow users to point to them anyways on the ExternalSecret (the ClusterSecretStore reconciler merely checks if the Store definition is valid).
Is there any specific requirement to not allow a ClusterSecretStore object be created in your use case (aside from the fact that it will fail whenever fetching an object outside of RBAC scope)? Is it the ClusterRole needed to access ClusterSecretStores?

Yes, our use case is that we will deploy one operator per scoped namespace.This is to limit the blast radius in case one of the operators get compromised as each operator should have limited access only within its scoped namespace.  ClusterSecretStore and the rest of the CRDs are pre-created beforehand. However, we do not use ClusterSecretStore.

With my proposed changes, one can still use ClusterSecretStore with the operator when it is deployed separately without scopedRBAC as the CRDs will still be there to use.

I see, thank you for your clarification.
What are you expecting to happen if a user creates a clusterSecretStore, and then creates an ExternalSecret referencing it?

@gusfcarvalho I have pushed a new commit to skip processing cluster secret store and log an info message.
I have trouble running the main branch on Kube at the moment. cert-controller and webhook are not happy and throw a branch of different errors.
Do you mind helping me to have a test, please?

Hi @kinyat ! To test it locally, what I usually do is the following setup:
# Install vault
helm install vault -n vault --create-namespace hashicorp/vault \
    --set "server.dev.enabled=true" \
    --set "injector.enabled=false" \
    --set "csi.enabled=false

# add a secret
kubectl exec -n vault vault-0 -- vault kv put secret/foo my-value=hey

# expose port
kubectl port-forward -n vault vault-0 8200:8200

#Install CRDs with no webhook constraint:
kubectl create -f config/crds/bases
# run ExternalSecret locally
make run --enable-cluster-store-reconciler=false

Then you can add a SecretStore and a ClusterSecretStore binding to localhost:
cat << EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://<IP_ADDRESS>:8200"
      path: "secret"
      version: v1
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"
---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://<IP_ADDRESS>:8200"
      path: "secret"
      version: v1
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"
          namespace: default
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
data:
  token: cm9vdA== # "root"

Lastly, you can create any ExternalSecrets you like, and check if the error is properly output.
After that, to run add a unit test, you'll need to add a new reconciler configuration at the test setup, if you want to test the reconciler loop with the new command. Optionally, you can encapsulate your check in a method, and unit test the method (I would go for t his one, for instance).

@gusfcarvalho Thanks. Got the operator running locally. The logic seems fine :)

@gusfcarvalho updated as per your comment and added a unit test

@gusfcarvalho Sorry to bother you but may I ask if there is any update to move this PR forward, pls?

hi @kinyat ! I just took a look at it this morning, and I still have issues here.
If an user still tries to create a ClusterSecretStore and an  ExternalSecret bound to it, effectively not having the permissions causes the controller to hang.
You can simulate this on your setup by adding the following manifests in this order:
Create a ClusterSecretStore:
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: fake
spec:
  provider:
    fake:
      data:
      - key: "/foo/bar"
        value: "HELLO1"
        version: "v1"
      - key: "/foo/bar"
        value: "HELLO2"
        version: "v2"
      - key: "/foo/baz"
        valueMap:
          foo: example
          other: thing

Create an ExternalSecret bound to the Store:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: fake
    kind: ClusterSecretStore
  target:
    name: secret-to-be-created
  data:
  - secretKey: foo_bar
    remoteRef:
      key: /foo/bar
      version: v1
  dataFrom:
  - key: /foo/baz

Create a SecretStore:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: fake
spec:
  provider:
    fake:
      data:
      - key: "/foo/bar"
        value: "HELLO1"
        version: "v1"
      - key: "/foo/bar"
        value: "HELLO2"
        version: "v2"
      - key: "/foo/baz"
        valueMap:
          foo: example
          other: thing

Create an ExternalSecret bound to the Scoped Store:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example-2
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: fake
    kind: SecretStore
  target:
    name: secret-to-be-created-2
  data:
  - secretKey: foo_bar
    remoteRef:
      key: /foo/bar
      version: v1
  dataFrom:
  - key: /foo/baz

Finally, if you do a kubectl get es:
NAME        STORE   REFRESH INTERVAL   STATUS
example     fake    1h0m0s
example-2   fake    1h0m0s

It is expected that example could be in a no status situation, but also example-2 is not working.
If you check the logs we see controller runtime trying to cache ClusterSecretStore resources:
E0322 10:21:15.799230       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.ClusterSecretStore: failed to list *v1beta1.ClusterSecretStore: clustersecretstores.external-secrets.io is forbidden: User "system:serviceaccount:es:es-external-secrets" cannot list resource "clustersecretstores" in API group "external-secrets.io" at the cluster scope

The problem here is not that it stops working for ClusterSecretStores, but the fact that this also blocks the behavior of any other ExternalSecrets resources.
There are two ways to go around this:

Add some logic to not load into cache any ClusterSecretStore objects if the flag is selected;

add get,list,watch permissions to ClusterSecretStore anyways.

I am keen for the second option here. Would your use case still allow it?

@gusfcarvalho I tested it in my local kube cluster but it is all working just fine.

# kubectl get es -n external-secrets
NAME            STORE           REFRESH INTERVAL   STATUS
hello-world-2   vault-backend   10s
hello-world-3   vault-backend   10s                SecretSynced

# kubectl get ss -n external-secrets
NAME            AGE   STATUS
vault-backend   40m   Valid

# kubectl get css
NAME            AGE
vault-backend   10d

Can you check if you have a ClusterRole installed there? =)

@gusfcarvalho sorry I just realised I was using a very old version of docker mac without the RBAC properly implemented. I will try again after I upgrade it 😢

@gusfcarvalho I can only replicate your error message for the caching when I forcibly enable the CSS Reconciler with Scoped RBAC.
{"level":"info","ts":1648062680.0170197,"logger":"controller.clustersecretstore","msg":"Starting Controller","reconciler group":"external-secrets.io","reconciler kind":"ClusterSecretStore"}
W0323 19:11:20.018713       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta1.ClusterSecretStore: clustersecretstores.external-secrets.io is forbidden: User "system:serviceaccount:external-secrets:swyftx-external-secrets" cannot list resource "clustersecretstores" in API group "external-secrets.io" at the cluster scope
E0323 19:11:20.018804       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.ClusterSecretStore: failed to list *v1beta1.ClusterSecretStore: clustersecretstores.external-secrets.io is forbidden: User "system:serviceaccount:external-secrets:swyftx-external-secrets" cannot list resource "clustersecretstores" in API group "external-secrets.io" at the cluster scope

when CSS Reconciler is disabled
{"level":"info","ts":1648062752.8129458,"logger":"controllers.ExternalSecret","msg":"skipping cluster secret store as it is disabled","ExternalSecret":"external-secrets/hello-world-2"}
{"level":"info","ts":1648062902.1755433,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":"external-secrets/hello-world-3","SecretStore":"external-secrets/vault-backend"}

However, I can still observe that it does not honour the refresh interval of the Secret Store.
I think something has stopped the controller to refresh the store but probably not related to the CSS caching error.

@gusfcarvalho sorry just an update.

I thought it stopped the reconciliation but it was because I didn't set my log level to be "debug" to see a more verbose log.
I can see it is constantly updating as expected.

hi @kinyat ! you are right! It seems my setup had some residue as well.
One more thing we should add before merging: last week we merged a ClusterExternalSecret feature as well into root.go (see here. We need to add this block to the enableClusterStoreReconciler conditional as we will have permission errors if we don't do it.

@gusfcarvalho no worries. I am more than happy to help 😄
I have added processClusterExternalSecret and processClusterStore to have better fine-grant control and make them more explicit. See what you think.
Tested locally and both flags worked together with scopedRBAC

/ok-to-test sha=8dd8ca4

/ok-to-test sha=8dd8ca4d922b39a6a1a7644ee471130f7eac1d42

Github actions seems to be in a lazy day today.. I just want to run once again the e2e test suite for this latest change, but other than that, LGTM.

/approve

/merge

Thanks again for your contribution @kinyat ! :100:


fix(TCOMP-541): replace docker registry url with a property in TCOMP …

…service pom
What is the current behavior? (You can also link to an open issue here)
docker registry url is wrong
What is the new behavior?
fix docker registry url
Please check if the PR fulfills these requirements

[x] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[ ] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[x] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:
Related to https://jira.talendforge.org/browse/TCOMP-541

Pull request test result: success :star:


A lot of fonts aren't working

I want this font: https://www.reddit.com/r/2007scape/comments/ojepcx/comment/h516xk9/?utm_source=share&utm_medium=web2x&context=3
But it instead gives me a default font, same case with a lot of other fonts. Ones which seems to work are Impact, Georgia, Times New Roman, Arial. Probably some others
Thanks.

For a font to be picked up by the plugin the name must match (but are case insensitive) and the font must have been installed system wide. The font in the reddit post works on my w10 system by entering bankgothic in the font setting (and setting font style to BOLD to reach the same effect)

https://github.com/l2-/template-plugin/issues/21 also see this issue.

Thank you for all your help - appreciated
On Thu, 16 Jun 2022 at 4:37 pm, l2- @.***> wrote:

https://github.com/l2-/template-plugin/issues/21 also see this issue.
—
Reply to this email directly, view it on GitHub
https://github.com/l2-/template-plugin/issues/32#issuecomment-1157808415,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJRO7ZUMV7DBV4S7PNQ67HTVPNC5FANCNFSM5Y6J3HRQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


Custom B2C policy enable Javascript

Hi,
I'm trying to enable javascript on a custom b2c policy using my own themed page.
However when following your guide and uploading the policy I get the following error .Please use page contract in content definitions when enabling JavaScript.
Having followed the steps here https://docs.microsoft.com/en-us/azure/active-directory-b2c/javascript-samples

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 818e437f-75d7-30f1-d6fe-e57288abbc54
Version Independent ID: 876c370e-e6f1-0e30-4ab7-e502643196bd
Content: JavaScript samples - Azure Active Directory B2C
Content Source: articles/active-directory-b2c/javascript-samples.md
Service: active-directory
Sub-service: b2c
GitHub Login: @mmacy
Microsoft Alias: marsma

I came across this https://stackoverflow.com/questions/57260178/enable-javascript-on-b2c-custom-policy which would be great if it was covered off/made more clear on that page

@jordanfinners
Thanks for your feedback! We will investigate and update as appropriate.

@jordanfinners, have you tried the steps mentioned in this article:
https://docs.microsoft.com/en-us/azure/active-directory-b2c/user-flow-javascript-overview
Check if this helps you.

I did look at those steps @souravmishra-msft however that didn't work for custom policies

@jordanfinners, Even after following the steps, are you still getting the same error as mentioned above?

Yes as it doesn't mention And then add the page contracts to ALL the contentDefinitions in your files where a dataUri is mentioned.
Clearly like the stackoverflow post does

@jodanfinners, please refer to this https://docs.microsoft.com/en-us/azure/active-directory-b2c/page-layout#replace-datauri-values
Change all  Old DataUri value	 in your policy with New DataUri value

Hi, yes that works like the Stackoverflow recommends, I feel it could just be made clearer on that page about the steps that you need to do, as it brings quite a few different elements of documentation together

If you are still facing the error, then replace the DataUri tag from urn:com:microsoft:aad:b2c:elements:idpselection:1.2.0 to urn:com:microsoft:aad:b2c:elements:contract:providerselection:1.2.0 for the ContentDefinition api.idpselections and api.idpselections.signup.

i can't get this to work. wouldn't even want to enable javascript if it wasn't literally impossible to redirect the "forgot password" link when using SSO with SAML in an external site like salesforce. i added every single content definition as listed in https://docs.microsoft.com/en-us/azure/active-directory-b2c/contentdefinitions#migrating-to-page-layout and still get

Please use page contract in content definitions when enabling JavaScript
when uploading the custom policy for SAML.

TrunstFrameworkBase.xml:
<ContentDefinitions>
	  <ContentDefinition Id="api.error">
		<DataUri>urn:com:microsoft:aad:b2c:elements:contract:globalexception:1.2.0</DataUri>
	  </ContentDefinition>
	  <ContentDefinition Id="api.idpselections">
		<DataUri>urn:com:microsoft:aad:b2c:elements:contract:providerselection:1.2.0</DataUri>
	  </ContentDefinition>
	  <ContentDefinition Id="api.idpselections.signup">
		<DataUri>urn:com:microsoft:aad:b2c:elements:contract:providerselection:1.2.0</DataUri>
	  </ContentDefinition>

      <!-- CUSTOM HTML -->
      <ContentDefinition Id="api.signuporsignin">
        <LoadUri>https://redacted</LoadUri>
        <RecoveryUri>~/common/default_page_error.html</RecoveryUri>
        <DataUri>urn:com:microsoft:aad:b2c:elements:contract:unifiedssp:1.2.0</DataUri>
        <Metadata>
          <Item Key="DisplayName">Signin and Signup</Item>
        </Metadata>
      </ContentDefinition>
      <ContentDefinition Id="api.selfasserted">
        <LoadUri>https://redacted</LoadUri>
        <RecoveryUri>~/common/default_page_error.html</RecoveryUri>
        <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:1.2.0</DataUri>
        <Metadata>
          <Item Key="DisplayName">Collect information from user page</Item>
        </Metadata>
      </ContentDefinition>
      <ContentDefinition Id="api.selfasserted.profileupdate">
        <LoadUri>https://redacted</LoadUri>
        <RecoveryUri>~/common/default_page_error.html</RecoveryUri>
        <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:1.2.0</DataUri>
        <Metadata>
          <Item Key="DisplayName">Collect information from user page</Item>
        </Metadata>
      </ContentDefinition>
      <ContentDefinition Id="api.localaccountsignup">
        <LoadUri>https://redacted</LoadUri>
        <RecoveryUri>~/common/default_page_error.html</RecoveryUri>
        <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:1.2.0</DataUri>
        <Metadata>
          <Item Key="DisplayName">Local account sign up page</Item>
        </Metadata>
      </ContentDefinition>
      <ContentDefinition Id="api.localaccountpasswordreset">
        <LoadUri>https://redacted</LoadUri>
        <RecoveryUri>~/common/default_page_error.html</RecoveryUri>
        <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:1.2.0</DataUri>
        <Metadata>
          <Item Key="DisplayName">Local account change password page</Item>
        </Metadata>
      </ContentDefinition>
    </ContentDefinitions>

@markusdresch I have the same problem, have you been able to fix it?

@emaborsa i implemented the new subjourney: https://docs.microsoft.com/en-us/azure/active-directory-b2c/add-password-reset-policy?pivots=b2c-custom-policy
unfortunately the "cancel" button does not behave the way it should, it does not go back to the main user journey but returns to the calling website (salesforce in my case - leading to another saml error). still trying to solve this.


Punktinformasjon tagget BS som MI, fikset

A picture tells a thousand words
Before this PR
After this PR

Coverage remained the same at 40.487% when pulling d0ab172dab601f91e6c07d7d2d236290a9af7b38 on BS_ikke_MI into ad44f0732f44535e8bb968b7fa85ead8c553fe13 on master.


Use Get-Content instead of ReadAllText to read file contents on Win

Get-Content can read files that are in use by another process, such as log files, where ReadAllText can not.
Example output from powershell for the differences:
PS C:\Users\vagrant> [System.IO.File]::ReadAllText("C:\Windows\Temp\test.log")
Exception calling "ReadAllText" with "1" argument(s): "The process cannot access the file 'C:\Window
cause it is being used by another process."
At line:1 char:30
+ [System.IO.File]::ReadAllText <<<< ("C:\Windows\Temp\test.log")
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : DotNetMethodException

PS C:\Users\vagrant> (Get-Content("C:\windows\Temp\test.log") | Out-String)
2015-04-14 03:32:07 nil INFO -> -> MARK <- <-
2015-04-14 03:33:16 nil INFO -> -> MARK <- <-
2015-04-14 03:34:16 nil INFO -> -> MARK <- <-
2015-04-14 03:35:16 nil INFO -> -> MARK <- <-
2015-04-14 03:36:16 nil INFO -> -> MARK <- <-

Thanks.

Released as v2.28.2.


Map ISV2PurchLineVersionedFieldSet

I think the new map should be named ISV2PurchLineMap in the diagram, instead of ISV2PurchLineVersionedFieldSet?
Because there is already a class called ISV2PurchLineVersioningFieldSet, which makes it a bit confusing. And it makes the difference between these two more clear.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c49540ec-9be0-49a5-e795-377984c406b4
Version Independent ID: a7ae6f12-fe91-31d8-704c-489f98662eac
Content: Extend table maps that are used for versioning - Finance & Operations | Dynamics 365 | #MSDyn365FO
Content Source: articles/dev-itpro/extensibility/maps-with-versioning.md
Service: dynamics-ax-platform
GitHub Login: @LarsBlaaberg
Microsoft Alias: lolsen

You are right. Thank you for the feedback.


Storage types

Is there a reason why we can't use GRS storage?  Is the recommendation LRS only?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 706b2e1a-e90e-d70e-431f-5476b0ca52c1
Version Independent ID: 70721d4c-642c-61f8-6c61-84b1dd1ae96c
Content: Deploy a two-node Storage Spaces Direct SOFS for UPD storage in Azure
Content Source: WindowsServerDocs/remote/remote-desktop-services/rds-storage-spaces-direct-deployment.md
Service: unspecified
GitHub Login: @Haley-Rowland
Microsoft Alias: harowl

Premium Storage is LRS only.


Create an increment_x method, not next_x

This creates better naming at the attribute level:
Old (attribute itself makes little sense):
counter :build_number

x.build_number
# => 0

x.next_build_number
# => 1

New (incrementing method and attribute make sense on their own):
counter :builds_count

x.builds_count
# => 0

x.increment_builds_count
# => 1

/cc @wfleming

:+1:


refactor(examples/d-o/musica): clean up no-param-reassign violations

#990
Clean up no-param-reassign from "examples/data-objects/musica/**"

@atiqueahmedziad our testing CI has some instability right now.  I have retrigger your run.

@atiqueahmedziad our testing CI has some instability right now. I have retriggered your run.

Thanks @curtisman :-) I was a bit worried :p


[QwenBlogBridge] Add bridge

Qwen is a Chinese AI team.

@dvikan Their index.xml includes only the first paragraph.


Move @types/lodash to devDependency

I think the @types/lodash dependency in package.json can be moved to devDependencies instead since it doesn't look like the typemoq.d.ts file references it. Right now there's a bug in the lodash definition that causes a compilation error in the newer versions of typescript.

Yep, you're right. I've moved @types/lodash to dev dependencies.


Planner 2328: enable swagger

JIRA

https://issues.redhat.com/browse/PLANNER-2328
Referenced pull requests

How to retest this PR or trigger a specific build:
For quarkus and all use-case apps, I followed the readme file instructions and run $ ./runQuickstartsFromSource.sh
For spring-boot app, maven version, I also followed instructions and run mvn spring-boot:run
Gradle version of spring-boot app did not compile.  Since I am not used to gradle, I was unable to fix it. Below is the error thrown.

$ gradle bootRun
FAILURE: Build failed with an exception.

Where:
$ gradle bootRun --scan

FAILURE: Build failed with an exception.

Where:
Auto-applied by using --scan

What went wrong:
Plugin [id: 'com.gradle.enterprise', version: '3.6.3', artifact: 'com.gradle:gradle-enterprise-gradle-plugin:3.6.3'] was not found in any of the following sources:

Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
Plugin Repositories (could not resolve plugin artifact 'com.gradle:gradle-enterprise-gradle-plugin:3.6.3')
Searched in the following repositories:
Gradle Central Plugin Repository

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 596ms

please send this PR against https://github.com/apache/incubator-kie-optaplanner-quickstarts repository


x/talks: 2 sets of deploy instructions

I've recently realized that, due to a series of unfortunate events spanning 2014-2020, we have accidentally ended up with 2 sets of deploy instructions for talks.golang.org:

golang.org/x/build/cmd/tip README
golang.org/x/talks README

We should pick one as canonical (likely the one in x/build/cmd/tip, because it has an auto-deploy feature), and update READMEs in both places to avoid any future confusion.
Making this issue quickly so I don't forget; will add more history/background later.
/cc @cagedmantis @toothrot @andybons

Definitely agree about not having two sets of instructions.
It's unfortunate that the x/build version requires Docker,
but if it happens automatically on push so that mere mortals
never have to run it, that's fine.

I investigated whether the deploy instructions in x/build/cmd/tip (which have the auto-deploy feature) can still be used today, but they have atrophied and cannot be used without making changes to get them functional again. I tried making small and safe changes to get them working, but it turned out larger changes are required to get them functional without also breaking tip.golang.org in the process. (I've described how they're broken in a separate issue #38120.)
The deploy instructions in x/talks are functional and have been used most recently. They lack the auto-deploy feature, unfortunately.
We could resolve this issue by continuing to use deploy instructions in golang.org/x/talks README for now, delete the non-functional instructions in x/build/cmd/tip (or at least mark them as "broken" and in need of fixing), and file a tracking issue about restoring the auto-deploy of x/talks content feature that used to exist. It can be resolved by making x/build/cmd/tip functional for serving talks.golang.org again, or another way.
Alternatively, we can fix x/cmd/tip deploy instructions (i.e., resolve #38120) and use them. I'll investigate that issue some more first.

Now that we have //go:embed I believe the right fix here is to use //go:embed so that no files have to be copied across repos as part of a deploy.

Now that we have //go:embed I believe the right fix here is to use //go:embed so that no files have to be copied across repos as part of a deploy.

The deploy instructions in x/talks are functional and have been used most recently.

As we found out from a deploy attempt for #43455, they stopped being functional. I opened issue #44164 and sent CL 290192 to fix it. (As part of that work, I also revisited this issue; see below for updates.)

Now that we have //go:embed I believe the right fix here is to use //go:embed so that no files have to be copied across repos as part of a deploy.

The cmd/present code lives in x/tools, while the talk content is in x/talks, so it seems some copying is unavoidable. I considered moving the App Engine-specific code from x/tools/cmd/present to x/talks (similarly to what we've done in the x/blog repo) to get around having to copy code, but that resulted in much code duplication.

Alternatively, we can fix x/cmd/tip deploy instructions (i.e., resolve #38120) and use them. I'll investigate that issue some more first.

I investigated that option and pretty much got it working in a staging instance. However, I decided not to continue with that approach for reasons described in https://github.com/golang/go/issues/38120#issuecomment-775365482.
So, I'll update this issue to be about removing the non-functional x/build/cmd/tip deploy for talks, and we can continue to use the simpler deployment in x/talks README instead (which will be functional as soon as #44164 is resolved). More can be done later as part of future work.


Doubt: How to destroy CfCT resources?

We have added this to backlog to update the documentation.

You can delete the solution's cloudformation stack to destroy most of the resources created by the solution.
Resources that require manual deletion:

The S3 buckets and related artifacts created by the solution will be retained and needs to be manually deleted (CustomControlTowerPipelineS3Bucket, CustomControlTowerPipelineArtifactS3Bucket, CustomControlTowerS3AccessLogsBucket)
If you created a Code Commit repo using the solution, that will be retained and needs to be manually deleted

Hey @evairmarinho, let us know if you've got any further context here, closing this issue for now. Thanks!


Setup automatic dependency updates

I have good experiences with Depfu, however other tools such as e.g. Snyk would also be fine.

Since v1.2.1 this should be covered by Snyk now.


Add methods to get the size of VTCM on device as well as the allocated size of the HexagonVtcmPool

Expose methods so that callers can get both the size of VTCM on device as well as the allocated size of the pool for the session.
cc: @masahi

Can we expose the API to python too?


Discussion: v3 proposal to simplify API

A lot of options have literal git log counterparts:

--starting-date maps to git log --since
--commit-limit maps to git log --max-count
--ignore-commit-pattern maps to git log --exclude
--tag-pattern maps to git tag -l '<pattern>'
--tag-prefix is included in git tag -l '<pattern>'
--unreleased-only maps to git log $(git describe --abbrev=0)...HEAD
--release-summary maps to git log --simplify-by-decoration
What #223 asks maps to git log <target-branch>
What #216 asks maps to git log --no-merges
What #207 asks maps to git log --simplify-by-decoration -n <n> --tags

There's a lot of requests to add very specific use-case options that git would handle much faster and in a more reliable way than any abstraction on top of it could. So my thought is that maybe it is a better approach to execute different git commands as starting point instead of doing 1 big cmd and mapping everything in JS.
Benefits to this approach are:

probably much faster for many use cases without extended sorting/ filtering requirements
probably less unit tests and specific options required

I think the reason why a lot of these option feature requests exist, is because the --append-git-log and --append-git-tag options are not self-evident (I've been using this lib for over a year and only realized now). If it were called --git-log-args or if most options mapped directly to a git-native option, another burden of documentation could be shifted onto the git-scm docs.

The repo is now over 7 years old, so there is no doubt that this could be much more efficient if it was written from scratch. There is nothing stopping anyone creating a similar library that uses git log more heavily.
Although, keep in mind that not everyone is running the latest git, and not everyone knows or wants to learn the nuances of git compared to more straightforward CLI arguments.
The original intention of the library was to be a very simple script to generate a changelog for any repo, with no need for loads of configuration or parameters. I think it still is if you ignore the amount of options it now has, but I was eager to please anyone who had a specific use case or something they wanted to tweak. Maybe that was a mistake on my part.
I appreciate the comment and agree with your sentiment, but I simply don’t have time for any big rewrites these days.

@cookpete totally understand & am grateful for the work done so far.
Note I named the issue proposal for v3, meaning in an unspecified amount of time (could be 1-x years).
So should you ever feel like doing a rewrite for a new major version, I hope this analysis helps


Antag status doesn't clear from deconverted revs

Description

I think there's a similar issue with zombies, it's not a major issue but it does clog the admin panel when I simply what to see who the actual main antags of the round are.

I see Smug's PR addresses the mindshield, but I also wanted to mention it doesn't clear it when the heads all die either. Just in case those work differently.


Enable more specs with the Haskell backend

The #Ceil(#buf) rule is removed because it is obsolete. In the past, it was necessary to define #Ceil(#buf) so that we could use the SMT solver to refute the #Not(#Ceil(#buf)) remainder after applying rules. This is no longer necessary because the backend correctly infers #Ceil(#buf) from the #buf appearing in the configuration; then the remainder is refuted by the obvious contradiction.

There are some issues with the SMT solver on Jenkins, so we probably need to wait for the latest version of K before all these tests will work.

I thought this would use the latest K now, but I'm still getting weird responses from the SMT solver, so I need to investigate locally.


adding notebook and cell metadata

The first part of #38, this adds round-trip support for cell metadata.
I probably started in the wrong place, and ended up touching too much stuff. But maybe not. Since I think it has to have a new rule in the lexer, it would have been a bit odd to overload that elsewhere.
Obviously still lots to be done, but wanted to get this out there!

looking good so far! thanks!

Thanks! The test setup you built is really nice... Though i confess i had
to do some manual round trips to fix little issues.
Quick question? Why the mistune fork (har)? Nothing is too deeply buried in
the upstream (grammar, lexer) and since it's already an nbconvert dep...
On 03:10, Tue, Jun 23, 2015 Cyrille Rossant<EMAIL_ADDRESS>wrote:

looking good so far! thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/rossant/ipymd/pull/62#issuecomment-114386618.

IIRC I had to make minor changes in some classes, but I can't remember the specifics. I'd be happy to move to a regular dependency if it's technically feasible without too much work.

@bollwyvl just wondering if it's still a WIP, or if it can be merged as is, as a first step?

@rossant if there is any reason that you can't make mistune a dependency, I'd like to help.

@rossant gah, totally have a draft response from 11 days ago! anyhow, long and short is i just got back to this now.
I added notebook metadata, maybe not very elegantly. kernel info has to be first, and has to follow the ---<stuff>--- while cells use ---<stuff>... to be more YAML-like.
it works for the content manager (yay!), but of course it is injecting the language_info and kernelspec, and I can't yet decide how to break it down in tests. How do you want to handle this? obviously this is how we would support multiple kernels, but it would be surprising to have all that get injected the first time you round trip. once we do !kernel python2 or whatever, we can obviously hide all that stuff, which would be less surprising, but i think it would be good and useful to get this resolved at least somewhat now before adding the whole alias tag scheme.

kernel info has to be first, and has to follow the ------ while cells use ---... to be more YAML-like.

hmm.. is there a reason for this?
btw can you remind me (or send me a link) about the rationale for --- and ... instead of double ---? Is this standard?
so: about notebook metadata and language info. I haven't thought much about that. A few random remarks

as far as I'm concerned, if I'm working on a book (as I am now), I'll always use the same language for all my .md notebooks, and I wouldn't want to have a new header in all of them. So it would be good to have an option for the default language. In other words, the notebook metadata should be optional. And the lib default should probably be Python 3 (ipymd is a Python library after all :)

maybe the notebook metadata should be opt-out by default? Ideally, the round trip should have no effect by default, I think.

good idea to have aliases like !kernel python2 or whatever -- do you know if there's anything standard for this sort of things?

kernel info has to be first, and has to follow the ------ while cells use ---... to be more YAML-like.

hmm.. is there a reason for this?

The first-ness and ---/----ness is the front-matter convention borrowed from Jekyll. Github will render it there, and nowhere else.
It also makes sense, because when it comes back over the wire as an ipynb struct, there will be no way to tease it back out into some other structure that couldn't have been modified by the user, so first is probably as good as any... aside from last.
also there can be only one :zap:.

btw can you remind me (or send me a link) about the rationale for --- and ... instead of double ---? Is this standard?

The YAML spec actually demands ---/..., I don't know why jekyll decided to go with /---. We could allow for /---... but then there would be no distinguisher as to whether something is notebook or cell level. but since we're going to be allowing rather a lot of YAML features, it makes sense to me to go with the spec.

as far as I'm concerned, if I'm working on a book (as I am now), I'll always use the same language for all my .md notebooks,

So if a Configurable setting (i.e. default_kernel) in the profile or cmdline matches what the frontend says in the meta, and there is nothing else at all in there...

maybe the notebook metadata should be opt-out by default? Ideally, the round trip should have no effect by default, I think.

...it doesn't get written out. works for me!

good idea to have aliases like !kernel python2 or whatever -- do you know if there's anything standard for this sort of things?

I presume we can interrogate the kernelmanager or something, and get the list of names. otherwise, it will use the same tag mechanism we talked about for cell meta, like !slide.

Still failing, due to how i've temporarily stored notebook meta, but it now strips out either the native kernel or a configurable one (needs doc). Probably pretty close to reviewable... but keeping [wip] because it's late :zzz:

thanks for the explanations again, looks all quite reasonable!

Ok, tests and doc up! Would love some feedback on the choices: be brutal!

Looks good, thanks! I've left some minor comments. I'll play with it a bit later!

btw, the alias ! thingy doesn't seem to be documented?

Thanks for the outstanding code review! :facepunch: The insight about the crufty is_notebook and is_meta cleaned a lot of things up.
Now that I think about it, I don't think I hit the doc on default_kernel: that's there to let you decide which kernel (gah, should be kernels, probably) shouldn't be stored in notebook metadata.

btw, the alias ! thingy doesn't seem to be documented? and it's not used in the test examples?

That's not in yet! It's different and weird enough that I wanted to get all this stuff in first before we explore how that should look. It should be entry_pointy, will need a fair amount of doc, might introduce new dependencies (if we need to do the patch thing). So I guess let me know what you think about what's in, and we'll go from there!

Ok, removed the spurious metadata, documented the kernelspec switches... and made sure they were turned back on.
I have to say, when editing the README, it does seem a bit odd that every paragraph gets its own cell. Also the treatment of things-in-lists is a bit weird. It's outside the scope of this issue, but it is surprising to have a single cell broken into many on roundtrip. With meta now in there, cell boundaries become more important...

awesome, thanks @bollwyvl !!
I'm gonna open a new issue for the split issue


PostCSS plugins not working as expected

When I add the following package.json configuration on the readme.md for adding PostCSS plugins, I get a build error.
{
  "devDependencies": {
    "postcss-simple-vars": "1.1.0"
  },
  "cssModules": {
    "postcssPlugins": {
      "postcss-simple-vars": {},
      "postcss-modules-local-by-default": {},
      "postcss-modules-extract-imports": {},
      "postcss-modules-scope": {},
    }
  }
}

This is the build error I keep getting:
/~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Processing Step: CSS Modules / PostCSS compilation
Unable to compile /home/admin/Projects/stencil/node_modules/react-toolbox/src/components/layout/theme.css
CssSyntaxError: postcss-simple-vars: /home/admin/Projects/stencil/node_modules/react-toolbox/src/components/layout/theme.css:161:1: Undefined variable $platform

@define-mixin sidebarIncrements $platform {
^
  & .sidebarDrawer {

/~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is an issue with React Toolbox's use of CSS Modules with CSS next, which oddly compiles just fine without any PostCSS plugins added to package.json. But when I revert to the default settings, it still doesn't allow me to use any CSS next in my own CSS modules in the /imports folder.
I'm sure I'm missing something with the set up here, but I can't find anything in the README.md. Any idea what I'm doing incorrectly?
This project is running Meteor 1.6.1, React v16 (with some old Blaze templates), and React Router v4.

I haven't had time to look into this yet - it's on my radar but I've been very busy of late


the software cannot be opened.

Microsoft PowerToys version
0.87.1
Installation method
Other (please specify in "Steps to Reproduce")
Running as admin
Yes
Area(s) with issue?
General
Steps to reproduce
I am using Scoop to install, but the software cannot be opened. It can still be displayed in the status.Even using an administrator to open it doesn't work
✔ Expected Behavior
No response
❌ Actual Behavior
No response
Other Software
No response

/bugreport

Hi there!We need a bit more information to really debug this issue. Can you add a "Report Bug" zip file here?  Then drag the zipfile from your desktop onto the GitHub comment box in this issue. Thanks!

I can see the icon in the bottom right corner, but no menu appears when I right-click
I ran the program and encountered a failure prompt

C:\Users\huch40\scoop\apps\powertoys\current\Tools\PowerToys.BugReportTool.exe

@htcfreek


ContainerRuntime mirror registries v1beta2 API

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1551
Fixes #1547
Fixes #1470
Special notes for your reviewer:

machine-controller already can handle dockerd registry-mirrors
machine-controller needs additional flags handling for containerd

Does this PR introduce a user-facing change?:
ContainerRuntime mirror registries v1beta2 API

/hold
Requires machine-controller support

/hold cancel
will remove draft status once machine-controller counter-part is ready

/lgtm


chore: onchain usd send comments

some potential improvment for onchain-usd-send
@vindard feel free to cherry pick the commit that you want

If it's ok, I'd like to cherry-pick https://github.com/GaloyMoney/galoy/pull/2094/commits/ff5cca9a708e4bb5c9a8c142b57ccc6e279ef52a as a separate PR to keep the #2072 diff clean


Pull request description code block syntax highlighting

A big feature of this project is that it allows you to review a pull request with the syntax highlighting of your favourite VS-Code theme.
However when you view code blocks in the Pull Request description they are completely void of any syntax highlighting:

It shouldn't be too hard to apply the users preferred syntax highlighting to pull request description code blocks if the language is known.

Duplicate of https://github.com/microsoft/vscode-pull-request-github/issues/54


Seq2Seq: why length of input sequences should be equal

As I see more LSTM implementation on keras, it seems that the length of input sequences should be equal. for example in addition_rnn, we use padding to make length of sequences equal.
But as you now LSTM is able to learn sequences with different lengths. So why we use padding?
I asked this question generally (not about keras) which people says there should not be such equalization.
https://stats.stackexchange.com/questions/282319/how-to-feed-time-series-with-different-length-into-lstm
I appreciate if anyone explain this issue.

I asked this question on slak channel of keras. so I am closing the issue here.

Any result or thoughts about this? Or now APIs in Keras not need to using padding?

@jinfagang
I didn't get exact answer but people advised me to use this features:

Masking layer
sample weight initialization alongside with padding

I did some exploration but results were not good when I added masking layer. finally I decided not to use masking and just normalized my data to have sequence lengths and so I lost part of training data.

@jinfagang
I recommend to read this too:
http://machinelearningmastery.com/data-preparation-variable-length-input-sequences-sequence-prediction/

@mehrdadscomputer Thanks pal, nice help.

@mehrdadscomputer I have the same problem, but my situation is a bit different. I want to train with a little part of the whole data and than use the whole sequence/data in prediction. So I can not cut my data for prediction. Padding masking did not work good, what can I do ?


Fully qualify types in quasiquotes

While browsing the sources I noticed some unqualified types in quasiquotes. They haven't caused any hygiene issues for me but I think those are potential bugs.

Good catch! Many thanks ... merging :-)


net: Another duplicate function to get pkt address

Following https://github.com/zephyrproject-rtos/zephyr/issues/4421 , this is 3rd function with almost the same functionality (previous duplication case fixed). Well, this time, it has signature:
static void pkt_get_sockaddr(sa_family_t family, struct net_pkt *pkt,
                             struct sockaddr_ptr *addr)

i.e., it takes struct sockaddr_ptr*, whereas net_pkt_get_dst_addr() takes struct sockaddr*. Still, I guess it's worth looking into to avoid duplication.

@askawu : Let me know if you're interested to look into this, based on the previous experience. If not, this is in my queue.

@pfalcon , please assign this to me. :)

@askawu : Done, thanks!

it's actually a feature, not a bug ;)

it's actually a feature, not a bug ;)

Well, it's definitely not a "bug", but:

Objectively, a missed optimization opportunity. I didn't hear about code size comparison of Zephyr's IP stack with other IP stack, but given such cases, it's hard to shake off a hunch that we don't compare favorably.
It shows that we have some inconsistency/technical debt. Actually, somewhere around 1.10 I got an idea that we have a pretty vast and featureful stack, and no single party (except for hopefully @jukkar) knows it well enough, so there's a chance that a particular developer when adding new feature might reinvent/duplicate some functionality (even if partially).

I don't think that we're ready for full-fledged optimization (that's tracked in https://github.com/zephyrproject-rtos/zephyr/issues/3547), but IMHO, p.2 is a real concern, especially is we're talking about things like MISRA C (what's the point of MISRA C, which tells us how to write expressions and statements, if we have problems with how code is being written?).
Anyway, in the current situation, this is definitely prio:low.

@nashif

you should try to assign bugs to one person only, otherwise everyone will be relying on the others

Per above, I try to assign @jukkar to any networking issue, to "keep him in loop". Not doing that would mean giving up an idea that "@jukkar for sure knows what happens in the IP stack, down to small ideas" ;-). He didn't raise any concerns, if there're suggestions for better workflow (e.g. just @mention), let's go over them.
To make the above more fair, I now assign ticket I might work on to myself too.
And @askawu volunteered to work on this, hence 3 assignees %). My guess he's on Chinese New Year vacation now, and I myself have enough a queue I think, so this isn't marked for 1.11.

This looks like a stale issue atm as there has been lot of changes in net_pkt API -> closing.


No arrow for folding

https://github.com/microsoft/vscode/blob/d86a6137cb58c57eb586605422001fdfb5d725c6/src/vs/workbench/browser/layout.ts#L52

In a PR with comments, seems to affect ranges that changed.

Can you give some more specific details about what's wrong/missing?
Are you missing a folding range on #region ? You need to add a //#endregion somewhere

Yes, quite sure it has an end region, it works fine now that it was pushed to main:

i missed the endregion


Mistake in French translation

The mistake is quite misleading because it is in the main table headers:
The English headers are "User configured/App configured" which meaning is very clear. The current French translation is "L'utilisateur est configuré/L'application est configurée"  which means "the user is configured/the app is configured" which is meaningless in this context (this mistake occurs very often alas). A proper translation would be "Configuré par l'utilisateur/Configuré par l'application" (which means "Configured by the user/Configured by the app", which is what the English title means).

@delahondes Thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


Streaming llama output

@jmorganca @mxyng  Testing llama2 output after building and serving the model and with the example why the sky is blue I noticed that the streaming does not work correctly. Using the curl or python requests either on ubuntu 22.04
I got something like:
{"model":"llama2","created_at":"2023-08-01T22:02:11.894420812Z","response":" sc","done":false}
{"model":"llama2","created_at":"2023-08-01T22:02:12.151293915Z","response":"at","done":false}
{"model":"llama2","created_at":"2023-08-01T22:02:12.409555353Z","response":"ters","done":false}
Is there anyway to disable streaming?

@osamanatouf2 this is working as expected. Each stream response is a complete message to be consumed independently. If you're only interested in the full response, you can buffer the stream responses until done. For an example of this using Python requests, check the linked Discord message.
The short version:
def unstream(model, prompt):
    r = requests.post('http://localhost:11434/api/generate',
                      json={'model': model, 'prompt': prompt},
                      stream=True)
    r.raise_for_status()
    response = ''
    for line in r.iter_lines():
        body = json.loads(line)
        response += body.get('response', '')
        if body.get('done', False):
            body['response'] = response
            return body

https://discord.com/channels/1128867683291627614/1128867684130508875/1134313869221838868

@mxyng My point is if I want to use stream buffer, I cannot since the words are split. The issue shows one example but there is more. I did try a workaround to get the full response and that seems to be working. But I could not get streaming to work properly. I am trying to use Python. I always end up with case as follow:
Greate question, this is due to phenomeno n that caused by  Ray le igh sc at tering

Can you provide an example of your Python code? If the output is not being reconstructed properly, it'll be caused by either the model producing mangled outputs or the reconstruction. The stream is only a means of passing the decoded tokens from the model to the client.
Keep in mind spacing is handled by the model. The full response must be joined without any additional values, e.g. ''.join(responses)

@mxyng you are correct. ''.join(responses) does actually solve the issue.


Add word-wrap support based on terminal size

Highly experimental and still WIP.
Refers to #287

I opened #1119 with this change + a speculative fix for windows.

I opened #1119 with this change + a speculative fix for windows.

Ah thank you for taking this over, I'll close this one here.


Protocol guest leaks child nodes when entire subtree removed

When a subtree is removed, we only remove the root item from the map. We need to walk the tree and remove all children.

The host-side also has this bug (#1900), so we can use this guest-side walk to send the entire tree's IDs for removal. However, once the host side bug is fixed we don't want to do that. In order to do it conditionally we need something like (#1901).


IE issue

The  picker does not show proper UCT time values. Ideally it should show GMT+5.30. However, it shows only GMT.

The times shown (and their format) and determined by jquery-timepicker, and not this directive.


feat-US34012: Add proposal Status Stepper to Proposal Details

@roshan-bisht-lab49 The code looks good but the stepper UI needs to match the wireframes in terms of size, padding, etc.


Add set_container to elements

related to #507

This could use a test to demonstrate the usage (which will also fix the code coverage CI failure).


Support initial value for metadata

It would be interesting to be able to set a default value for metadata in the yaml file

In GitLab by @ksiero on Feb 6, 2020, 09:43
Closes with MR !221

removed milestone

In GitLab by @ksiero on Jun 22, 2020, 08:31
closed via merge request !240

In GitLab by @ksiero on Jun 20, 2020, 12:43
mentioned in commit 178e7010045a76b143de3973319439df8a0712bd

In GitLab by @ksiero on Jun 22, 2020, 08:31
closed via commit 178e7010045a76b143de3973319439df8a0712bd


fix mailIssueCommentBatch for pull request (#14252)

Backport #14252

make L-G-T-M work

make L-G-T-M work


Project scope and organisation

There have been several discussions about the organisation and scope of this project, for example https://github.com/delta-incubator/delta-sharing-rs/pull/19 and https://github.com/delta-incubator/delta-sharing-rs/pull/22. Let's discuss some options here.

What I would love to see in this project:

A Delta Sharing server that could be configured and launched from the command line that is at least at feature parity with the reference server implementation here
A library with implementation of the Delta Sharing Protocol (i.e. the REST API) that can be completely customised.

The Delta Sharing server implementation is already making good progress and is on its way to be at parity with the reference implementation. In doing so the server did make some assumptions, for example it uses a postgres database to store the shared tables, it includes some admin routes, etc. I think this is great, because it allows people to get up and running quick and easy.
The second point, a library only focused on implementing the protocol, would be aimed at more advanced users who want to customize their Delta Sharing server (e.g. auth middleware, rate limiting, storing shared tables in a different database, etc.). These users could depend on this crate for the core implementation (and maybe also some middleware) and customise it by implementing traits. I have some idea's/findings from experimentation in https://github.com/tdikland/delta-sharing-server-rs.
Regarding organisation, I think it makes sense to have a crate that implements the Delta Sharing protocol, a crate with useful extensions/middleware for the sharing server and finally a reference implementation that depends on the aforementioned crates and that is plug&play.


85 Support persitent harvesting delegation transaction

I think we should make this function an independent command.
I‘ve already added this function to transaction transfer, but there are too many useless parameters which will confuse users.

Pull Request Test Coverage Report for Build 386

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 93.385%

Totals

Change from base Build 384:
0.0%

Covered Lines:
163

Relevant Lines:
177

💛  - Coveralls

Closes #133


Link are Broken in README.md

Record conversion API and Hadoop API links are broken in README.md
Reporter: nihed mbarek / @nihed
Assignee: nihed mbarek / @nihed
Note: This issue was originally created as PARQUET-663. Please see the migration documentation for further details.

nihed mbarek / @nihed:
PR https://github.com/apache/parquet-mr/pull/350

Ryan Blue / @rdblue:
Merged. Thanks for contributing, @nihed!


Week 1: Weekly report

Till Friday 1 pm small summary here: https://docs.google.com/spreadsheets/d/17-2DJjB6mGZHfVXO47SnjktxSKkP6gE9nC8RD2ZEeQg/edit#gid=0

Documented:
"- Business understanding done

EDA done
Base Model Friday"


Update sbt-softwaremill-common, ... to 2.0.2

Updates

com.softwaremill.sbt-softwaremill:sbt-softwaremill-common
com.softwaremill.sbt-softwaremill:sbt-softwaremill-publish

from 1.9.15 to 2.0.2.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.softwaremill.sbt-softwaremill" } ]

labels: sbt-plugin-update, semver-major

Superseded by #456.


Add MVTec data format support

Summary

How to test

Checklist

[x] I submit my changes into the develop branch
[ ] I have added description of my changes into CHANGELOG
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] I have linked related issues

License

[ ] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[ ] I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

Codecov Report
Base: 77.60% // Head: 77.24% // Decreases project coverage by -0.37% :warning:

Coverage data is based on head (a65f3db) compared to base (3cc2e1c).
Patch coverage: 43.36% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #808      +/-   ##
===========================================
- Coverage    77.60%   77.24%   -0.37%
===========================================
  Files          179      183       +4
  Lines        23238    23423     +185
  Branches      5023     4856     -167
===========================================
+ Hits         18034    18093      +59
- Misses        4141     4270     +129
+ Partials      1063     1060       -3

Flag
Coverage Δ

macos-11_Python-3.10
?

macos-11_Python-3.7
?

macos-11_Python-3.8
?

macos-11_Python-3.9
?

ubuntu-20.04_Python-3.10
?

ubuntu-20.04_Python-3.7
76.81% <42.18%> (-0.30%)
:arrow_down:

ubuntu-20.04_Python-3.8
77.19% <43.36%> (-0.29%)
:arrow_down:

ubuntu-20.04_Python-3.9
77.20% <43.36%> (-0.29%)
:arrow_down:

windows-2019_Python-3.10
?

windows-2019_Python-3.7
?

windows-2019_Python-3.8
?

windows-2019_Python-3.9
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

datumaro/util/mask_tools.py
71.03% <6.66%> (-5.75%)
:arrow_down:

datumaro/plugins/data_formats/mvtec/base.py
28.98% <28.98%> (ø)

datumaro/plugins/data_formats/mvtec/exporter.py
37.66% <37.66%> (ø)

datumaro/plugins/data_formats/mvtec/format.py
100.00% <100.00%> (ø)

datumaro/plugins/data_formats/mvtec/importer.py
100.00% <100.00%> (ø)

datumaro/plugins/synthetic_data/utils.py
83.33% <0.00%> (-1.67%)
:arrow_down:

datumaro/util/test_utils.py
83.06% <0.00%> (-1.59%)
:arrow_down:

datumaro/util/os_util.py
70.45% <0.00%> (-1.52%)
:arrow_down:

datumaro/plugins/data_formats/kitti_raw/base.py
85.00% <0.00%> (-1.12%)
:arrow_down:

datumaro/plugins/data_formats/coco/exporter.py
85.65% <0.00%> (-0.43%)
:arrow_down:

... and 11 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Use Chudnovsky rational approximation of pi^2 for zetasEven

To evaluate Riemann zeta function on even arguments (zetasEven) we use formula:

The main issue here is that for reasonably big n we divide one huge number by another one, meaning a catastrophic loss of precision. For instance,
> approximateValue (zetasEven !! 25) :: Double
0.9999999999999996
> approximateValue (zetasEven !! 999) :: Double
NaN

which does not make any sense, because the zeta function is always finite and > 1. Currently we fix this behaviour by flooring values of zetasEven by 1.
This discrepancy hurts evaluation of Riemann zeta function on odd arguments and propagates further to numeric errors in Dirichlet beta function.

This is a rough idea, I have not given it much thought.
Chudnovsky algorithm gives us a rapidly converging series of rational approximations to pi^2. We can use it to generate a rational approximation of zetasEven !! n and consequently calculate not only zetaEven !! n itself (which is any way close to 1), but also zetasEven !! n - 1 using usual Double with a fairly good relative precision.

Surely the issue here is in approximateValue from ExactPi, rather than in zetasEven? Specifically, zetasEven returns an exact value, and the loss of precision occurs when converting the exact value into a Double.

:+1: for loss of precision in Dirichlet beta. The code introduced by #120 works, but not very well. Error propagation from zetasOdd and zetasEven is quite strong.

Cool!
@b-mehta do you know any rapidly converging rational series for pi? If yes, then we could possibly combine it with Chudnovsky algorithm for pi^2 and squeeze them together into exact-pi instead of Data.ExactPi. rationalApproximations.

@Bodigrim I don't know of any off the top of my head, but I'll research. In the meantime, I've just pushed to b-mehta/arithmoi@23b385d for a solution which should work if the answer turns out to be no: I've made a datatype ExactPiSq as a simple analogue to ExactPi along with a new approximateValue which uses Chudnovsky's algorithm to get the best approximation for a given Fractional instance.
The test value in the docs for zeta(50) used to differ from the true value after only around 10 non-zero places, but now agrees to a Fixed Prec50 at least. In addition, we can now get values for zetasEven !! n-1 as you mentioned (cf WolframAlpha):
> import Data.Number.Fixed
> let Exact z q = zetasEven !! 50
> approximateValue (Exact z q) :: Fixed Prec50
1.00000000000000000000000000000078886090522101180736
> getRationalLimit [q * p^z - 1 | p <- chudnovsky]
7.888609052210118e-31

One possible solution would be to use Chudnovsky's algorithm directly, and use a rational approximation to the square root using continued fraction convergents which is easy to calculate and should be fairly easy to accelerate also.

Looks awesome!
The approach with rational approximation to the square root sounds reasonable.
I suggest passing comparison function a -> a -> Bool to getRationalLimit as an argument and do not rely on Eq instance. This is not only more flexible in terms of precision, but is also a cure against NaN /= NaN thing, which is a common reason for fix-point evaluations cycling forever.

Sounds good, I'd like to do some numerical testing myself before making changes in exact-pi, but the modifications to getRationalLimit sound good. Would it be better to move the Chudnovsky work here to that repo instead?

Yes, IMHO Chudnovsky algorithm itself fits into exact-pi better.

Yes, if nothing happens I will proceed with my fork.

I managed to upload exact-pi-<IP_ADDRESS> with @b-mehta's improvements to rationalApproximations.

The issue has been merged, I'm guessing we can proceed with a PR to improve zetas.

I patched arithmoi to benefit from rationalApproximations. It has an immediate effect even for Double. For example, betas 1e-15 !! 31 gives a correct value of 0.9999999999999983 instead of 0.9999999999999976 previously.
@b-mehta thanks for valuable contribution!


Documentation Error

New Issue Checklist

I have read the guidelines for contributing and I understand:

[X] My issue is happening in the latest version of SideMenu.
[X] My issue was not solved in the README.
[X] My issue can not be answered on stackoverflow.com.
[X] My issue is not a request for new functionality that I am unwilling to build and contribute with a pull request.
[X] My issue is reproducible in the demo project.

Issue Description

Just wanted to point out the documentation under usage is out of date or was just mis-typed.
In Step 1: "Navigation Controller to be SideMenuNavigationController in the Identity Inspector"
The class should be "UISideMenuNavigationController". The screen shot under it also doesn't show the correct class. The correct class is being used in the demo app.

Hi @Ludicrous2k, UISideMenuNavigationController was actually deprecated in favor of SideMenuNavigationController in the latest release. Please update your source accordingly. Thank you.

Oh thanks for the heads up. Sorry I didn't realize.

No worries. I appreciate the effort.


Could you release the checkpoints pretrained on Kinetics 400

Hi Yuan, in the paper you mentioned that "Specifically, we train the model on Kinetics-400 (K400) dataset and report the top-1 accuracy on Kinetics-Sounds", I'm wondering if it is possible that you could release the checkpoints pretrained on K400 dataset for action recognition tasks. It appears that the repo currently provides checkpoints only for the event classification tasks. Thanks a lot!

hi there,
Thanks for the inquiry. Unfortunately I do not have time to release this checkpoint. It is not just a checkpoint, but also an inference script (add probably also data preprocessing). Before release, I do a cleanup on my research code and run a brief test. There are many experiments in the appendix and I only have time to release codes for experiments in the main manuscript.
However, I am willing to answer any questions related to that.
-Yuan


Need icon 'fa-university' in clr-icons

http://fontawesome.io/icon/university/
We're using this icon in the context of a budget value. Let me know if it can make its way to the next release.
Select one ...  (check one with "x")
[ ] bug
[x] feature request
[ ] enhancement

Environment details

Clarity version:
0.8.4

bank_icons_svg.zip
@Shijir I've attached the icon and its variants. I named it "bank" rather than "university" because of CCureau's use case.

@lil-kim Awesome. Thank you!


navtoogle or navtoggle in scss?

I was trying to modify the color of the menu button, and found its class is .navtoogle (which is also what it's called in the javascripts)
But page.scss also has .navtoggle
.no-js .nav .navtoggle {
	display: none;
}

I'm assuming this wants to be .navtoogle to match?

Line 171 in page.scss is where you can change the color and/or styling of the menu button that appears on smaller devices.

Got it. Thanks!
But I should have been more explicit about my question: is the typo in navtoogle (two o) or in navtoggle (two g) or there isn't a typo or it doesn't matter?
(I'm a photographer, not a coder; I only know enough to get myself in trouble)

Looks like it's a typo. Should be navtoggle in the CSS and in the HTML.


Eloquent model terrible bug (Delete all table rows!)

Hi there,
Last day I found a terrible bug (I guess) in Laravel Eloquent model which cause loosing all of my data!
Imagine you have a model called Invoice, and you want to forceDelete one of its records.
If you get an instance of Invoice and save it in a variable called $invoice, then you forceDelete that instance, and somewhere later, you make an update to that instance, Eloquent execute that update on all of your model rows!!!
$invoice = Invoice::find(541);
$invoice->forceDelete();

//... somewhere later

// These lines of code cause all rows in Invoice table updated with 'status'=>'failed'
$invoice->update([
    'status' => 'failed',
]);

So, Is this really a bug?

Looks more like a usage bug to me. Why would you still want to update a deleted record?

When you run delete() or forceDelete() it would change the Invoice::$exists value to false, and then when you attempt to run update() it would run the update on all the records because of the Invoice::$exists value.
https://github.com/laravel/framework/blob/5.2/src/Illuminate/Database/Eloquent/Model.php#L1478-L1513

This is indeed just a bug in your code.


Add VScode settings and recommendations

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#37
#36
#35
#34  👈
#33
#32
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @JackPlowman and the rest of your teammates on  Graphite

Merge activity

Sep 25, 5:02 PM EDT: @JackPlowman merged this pull request with Graphite.


Navigator fetches all Data columns when Name field is missing

The problem:
When using the Table.ToNavigationTable function from the sample here, it seems to retrieve all Data columns when there is no Name column.
Reproduction scenario from Power BI:

Get Data
Choose Invantive for Exact Online:

The navigator opens. The spinner keeps running until all data is fetched from all APIs in the backing source (in this case Exact Online, but the same holds for other sources with a lot of data).

Expected result is:
The navigator will only fetch the Data column values when the Data is actually requested.
Faulting code
[DataSource.Kind="Invantive", Publish="Invantive.UI"]
shared Invantive.Schema = () as table =>
    let
        Source = Invantive.Database(),
        smallSet = Table.SelectColumns(Source,{"table_catalog", "table_schema", "table_name"}),

        addItemKind = Table.AddColumn(smallSet, "ItemKind", each "Table"),
        adddItemName = Table.AddColumn(addItemKind, "ItemName", each "Table"),
        addLeaf = Table.AddColumn(adddItemName, "IsLeaf", each true),
        addDataTables = Table.AddColumn(addLeaf, "Data", each GetDataFor([table_catalog], [table_schema], [table_name]), type table),

        NavigationSchema = Table.ToNavigationTable(addDataTables, { "table_catalog", "table_schema", "table_name" }, "table_name", "Data", "ItemKind", "ItemName", "IsLeaf")
    in
        NavigationSchema
    ;

Working code:
[DataSource.Kind="Invantive", Publish="Invantive.UI"]
shared Invantive.Schema = () as table =>
    let
        Source = Invantive.Database(),
        smallSet = Table.SelectColumns(Source,{"table_catalog", "table_schema", "table_name"}),

        // renaming makes a difference here!
        rename1 = Table.RenameColumns(smallSet,{"table_catalog", "Catalog"}),
        rename2 = Table.RenameColumns(rename1,{"table_schema", "Schema"}),
        rename3 = Table.RenameColumns(rename2,{"table_name", "Name"}),
        // end difference

        addItemKind = Table.AddColumn(rename3, "ItemKind", each "Table"),
        adddItemName = Table.AddColumn(addItemKind, "ItemName", each "Table"),
        addLeaf = Table.AddColumn(adddItemName, "IsLeaf", each true),
        addDataTables = Table.AddColumn(addLeaf, "Data", each GetDataFor([Catalog], [Schema], [Name]), type table),

        NavigationSchema = Table.ToNavigationTable(addDataTables, { "Catalog", "Schema", "Name" }, "Name", "Data", "ItemKind", "ItemName", "IsLeaf")
    in
        NavigationSchema
    ;

I can't seem to reproduce in PBI Desktop. The [Data] column is pulled lazily if the AddColumn line sets type table on the column.


Dmitri/3733 switch storage driver

Add ability to switch Docker storage driver during update. The driver is a command line option to gravity upgrade atm (i.e. no UI changes were introduced):
# given a cluster with devicemapper as storage driver, change it to overlay2:
./gravity upgrade --docker-storage-driver=overlay2

Changing the storage driver for Docker is straightforward - the Docker daemon's command line is updated with the new driver. This also means that all containers will be lost and need to be repopulated when kubernetes starts. This is usually not an issue - even if it is, the containers with ephemeral storage can be committed and exported prior to update and loaded back after as additional preparation steps (out of scope in this PR).
Requires https://github.com/gravitational/gravity.e/pull/3749.
Fixes https://github.com/gravitational/gravity.e/issues/3733

Just occurred to me that I don't support -> devicemapper path which would require the specification of device.
Even though the main use-case is to go from devicemapper to overlay (or any other supported driver), I think the other way should be supported too.

@a-palchikov Have a question just reading the PR description: if you change storage driver in manifest, will it be detected too? Or is only upgrade command flag supported?

@r0mant question is important one - it should be supported. Also, is this flag a new flag or it always existed?

I removed the command line option to simplify the implementation (which also led to discovering new corner cases), so the update will only use the value from the manifest.

There's actually - there're tests that install with devicemapper and then upgrade which switches to overlay2. What's missing is the validation that the storage driver has been actually changed.


Improve Text Alignment and Style for Enhanced Readability

To create a more polished and visually appealing presentation of content on the webpage, it is necessary to enhance the alignment and style of the text. Currently, the text lacks a cohesive structure and visual hierarchy, which can hinder readability and overall user experience.
Text Alignment:
Left Align: For paragraphs, keep the text left-aligned for better readability.
Font Style
Font Choice: Use a clean, modern font (like Arial, Roboto, or Open Sans) that enhances readability.
Font Weight: Use bold for headings to create a hierarchy, while keeping body text regular or light.
Contrast: Ensure the text color contrasts well with the background for readability. Dark text on a light background or vice versa works well.
Accent Colors: Use a distinct color for keywords like "Privacy" or "Secured" to highlight important concepts.
Spacing
Line Height: Increase line height (1.5 to 1.6) for better readability.
Padding and Margins: Add padding around text blocks and margins between sections to avoid clutter.
Background
Subtle Backgrounds: Consider a subtle gradient or a soft color that complements the text rather than a stark white or black.

To create a more polished and visually appealing presentation of content on the webpage, it is necessary to enhance the alignment and style of the text. Currently, the text lacks a cohesive structure and visual hierarchy, which can hinder readability and overall user experience. Text Alignment: Left Align: For paragraphs, keep the text left-aligned for better readability.
Font Style Font Choice: Use a clean, modern font (like Arial, Roboto, or Open Sans) that enhances readability. Font Weight: Use bold for headings to create a hierarchy, while keeping body text regular or light.
Contrast: Ensure the text color contrasts well with the background for readability. Dark text on a light background or vice versa works well. Accent Colors: Use a distinct color for keywords like "Privacy" or "Secured" to highlight important concepts.
Spacing Line Height: Increase line height (1.5 to 1.6) for better readability. Padding and Margins: Add padding around text blocks and margins between sections to avoid clutter.
Background Subtle Backgrounds: Consider a subtle gradient or a soft color that complements the text rather than a stark white or black.

please show a preview image

@PragyaTripathi990 HEY!! Can you update me on how you will work on it?


all: test failures on netbsd blocked on waiting for subprocesses

#!watchflakes
post <- builder ~ `netbsd-.*` && (log ~ `^\s*os\.wait6` || log ~ `^\s*syscall.wait4` || `DETECTED A HANGING GO COMMAND` || `test timed out while running command` || `script_test\.go:\d+: .*: signal: killed`)

(attn @golang/netbsd)
See previously:

#56180
#50138
#56215
#48789

@golang/netbsd, is anyone looking into this? These failures are quite frequent.

@golang/netbsd, this failure rate is still quite high compared to the other ports. Is anyone looking into it?

@golang/netbsd, the Go porting policy requires that each port must have at least two developers “making required updates in a timely manner”.
Since no one has even responded to this issue since it was reported in January, and given the frequency of these failures, I believe that this port is no longer meeting that requirement. (If I am mistaken and someone is actually looking into this problem, please comment here with some details about when we can expect progress on a fix.)

I didn't check every log but each log that I checked is consistent with a hang in the wait6 system call.  I would vote for rolling back CL 431855 (again).  CC @tklauser

Would it help if we updated all the NetBSD 9.x test machines to the latest NetBSD 9_STABLE? That should have the rtld fix.
(Relatedly, I have a bug assigned to me to provide NetBSD 10 build VMs.)

The Go bug behind #50138 was fixed in https://go.dev/cl/442478. It is possible that related bugs remain, but the cause of the hands must be different from that cause.
The libc bug behind #56810 was supposed to have been fixed by upgrading the builders in #54773.

I don't think that history is quite right.  We don't have to implement blockUntilWaitable at all.  For NetBSD our first attempt at implementing blockUntilWaitable led to #48789.  We fixed that issue and tried again.  I am suggesting that that exposed this issue.  I think we should now try not implementing blockUntilWaitable at all on NetBSD.

To be clear, I am suggesting that this is a NetBSD kernel bug in wait6.

Just to be sure, can you share the output of ident /usr/lib/libpthread.so (or whatever is the pthread library that golang links against) on the affected builders?

Note that Go by default does not link against libpthread.so at all.  It calls the system call lwp_create directly.  That said, there are cases where it calls pthread_create, and I haven't checked which case applies for these failures.  The output of the ident is:

/usr/lib/libpthread.so:
     $NetBSD: crti.S,v 1.1 2010/08/07 18:01:35 joerg Exp $
     $NetBSD: crtbegin.S,v 1.2 2010/11/30 18:37:59 joerg Exp $
     $NetBSD: tss.c,v 1.1 2019/04/24 11:43:19 kamil Exp $
     $NetBSD: thrd.c,v <IP_ADDRESS> 2019/09/13 06:54:09 martin Exp $
     $NetBSD: mtx.c,v 1.1 2019/04/24 11:43:19 kamil Exp $
     $NetBSD: cnd.c,v 1.1 2019/04/24 11:43:19 kamil Exp $
     $NetBSD: call_once.c,v 1.2 2019/04/24 21:41:15 kamil Exp $
     $NetBSD: sem.c,v 1.9 2019/02/21 21:33:34 christos Exp $
     $NetBSD: pthread_spin.c,v 1.6 2012/08/16 04:49:47 matt Exp $
     $NetBSD: pthread_once.c,v 1.3 2013/03/21 16:49:12 christos Exp $
     $NetBSD: pthread_cancelstub.c,v <IP_ADDRESS> 2021/10/05 11:02:18 martin Exp $
     $NetBSD: pthread_tsd.c,v 1.17 2019/03/05 01:35:52 christos Exp $
     $NetBSD: pthread_specific.c,v 1.26 2013/03/21 16:49:12 christos Exp $
     $NetBSD: pthread_rwlock.c,v <IP_ADDRESS> 2020/01/26 10:55:16 martin Exp $
     $NetBSD: pthread_mutex.c,v <IP_ADDRESS> 2020/01/26 10:55:16 martin Exp $
     $NetBSD: pthread_misc.c,v <IP_ADDRESS> 2020/01/26 10:55:16 martin Exp $
     $NetBSD: pthread_lock.c,v 1.34 2008/04/28 20:23:01 martin Exp $
     $NetBSD: pthread_cond.c,v <IP_ADDRESS> 2020/01/26 10:55:16 martin Exp $
     $NetBSD: pthread_barrier.c,v 1.20 2016/07/03 14:24:58 christos Exp $
     $NetBSD: pthread_attr.c,v 1.18 2017/08/01 12:31:45 martin Exp $
     $NetBSD: pthread.c,v <IP_ADDRESS> 2022/08/08 17:16:53 martin Exp $
     $NetBSD: pthread_makelwp_netbsd.c,v 1.2 2014/12/17 01:49:08 pooka Exp $
     $NetBSD: crtend.S,v 1.1 2010/08/07 18:01:34 joerg Exp $
     $NetBSD: crtn.S,v 1.1 2010/08/07 18:01:35 joerg Exp $

Is there a summary of how golang uses the wait family of system calls and handles SIGCHLD?  I see there's a lot of past discussion and changes, so it would be helpful if I had a clear idea of the current picture.

Go will normally install a signal handler for SIGCHLD, but for most programs, including the ones that are failing, that signal handler won't do anything.
When Go is waiting for a process, it first calls wait6(P_PID, pidOfProcess, WEXITED|WNOWAIT).  When that call returns, Go will call wait4(pidOfProcess, &status, 0, &rusage).  (This two step process is to avoid a race in sending a signal to a process and accidentally sending it to a PID of a process that has been waited for and then recycled.)

golang doesn't by any chance use SA_NOCLDWAIT, or set SIGCHLD to SIG_IGN, does it?

It does not.

Does golang have any mechanism to guarantee that it only calls waitid/wait6 without WNOWAIT at most once -- or, at most once with a successful return if WNOHANG -- for each child?  (I suppose the answer is probably yes but I want to rule out easy problems before going into a wild goose chase of harder ones!)

  If a program calls os.Process.Wait from two different goroutines simultaneously, then Go will call wait4 from two different threads.  That said, that is not happening in these test cases.

Is there any ptrace involved?

No.

Is there any SIGTSTP/SIGSTOP/SIGCONT/WSTOPPED/WUNTRACED involved?

No.

Can you get any kind of core dump or stack trace of the child process when it hangs like this?

As far as I can tell, the child process has exited.

Are there any reports that consistently happen running the same test (after the rtld/pthread fix and the idtype numbering fix), so I could try running that one in a loop to more reliably reproduce it?

I haven't been able to recreate the problem.  It's fairly rare.  But I also haven't tried very hard.

Is there any chance you could start ktrace in the test harness, and save the ktrace file when it fails in this manner?  Something like ktrace("/tmp/ktrace.out", KTROP_SET|KTRFLAG_DESCEND|KTRFACv2, KTRFAC_SYSCALL|KTRFAC_SYSRET|KTRFAC_NAMEI|KTRFAC_PSIG|KTRFAC_EMUL|KTRFAC_MIB, getpid()) when each test starts, and ktrace(NULL, KTROP_CLEAR|KTRFLAG_DESCEND|KTRFACv2, KTRFAC_MASK, 1) when it ends, and, if the test fails, either save /tmp/ktrace.out or include the output of kdump -T -f /tmp/ktrace.out in the report?  (@bsiegert, maybe you could rig this up?)

I'll leave that for @bsiegert.

I took a look at a few other logs and they all appear to be blocking in wait6 under blockUntilWaitable, which is consistent with the hypothesis that the subprocess just hasn't exited yet. If that is the case, we need to know why the subprocess hasn't exited, not why wait6 isn't returning.

That is a fair point.  My only real reason for thinking that the problem is with wait6 is that we aren't seeing the builder hang.  We're only seeing the builder time out while waiting for child processes.

I took a look at a few other logs and they all appear to be blocking in wait6 under blockUntilWaitable, which is consistent with the hypothesis that the subprocess just hasn't exited yet. If that is the case, we need to know why the subprocess hasn't exited, not why wait6 isn't returning.

Several of the failures in this log are for subtests of os/exec.TestCancelErrors — namely, subtests beginning with killed_.
In normal operation, those subtests terminate the child process by sending it SIGKILL.
For example, in https://build.golang.org/log/0889975bf55f90c65f04d4535e58591d7c30fa4f we see:
goroutine 172 [chan send, 29 minutes]:
runtime.gopark(0x40001d9df8?, 0xaf9f8?, 0x0?, 0x0?, 0x4000024000?)
	/var/gobuilder/buildlet/go/src/runtime/proc.go:398 +0xc8 fp=0x40001d9dd0 sp=0x40001d9db0 pc=0x4adf8
runtime.chansend(0x4000185560, 0x40001d9f90, 0x1, 0x40001d9e50?)
	/var/gobuilder/buildlet/go/src/runtime/chan.go:259 +0x3c4 fp=0x40001d9e40 sp=0x40001d9dd0 pc=0x166e4
runtime.chansend1(0x40001d9f10?, 0x3d4608?)
	/var/gobuilder/buildlet/go/src/runtime/chan.go:145 +0x18 fp=0x40001d9e70 sp=0x40001d9e40 pc=0x16308
os/exec.(*Cmd).watchCtx(0x4000235080, 0x4000185560)
	/var/gobuilder/buildlet/go/src/os/exec/exec.go:813 +0x328 fp=0x40001d9fb0 sp=0x40001d9e70 pc=0x10c368
os/exec.(*Cmd).Start.func4()
	/var/gobuilder/buildlet/go/src/os/exec/exec.go:715 +0x2c fp=0x40001d9fd0 sp=0x40001d9fb0 pc=0x10bdbc
runtime.goexit()
	/var/gobuilder/buildlet/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x40001d9fd0 sp=0x40001d9fd0 pc=0x7da24
created by os/exec.(*Cmd).Start in goroutine 120
	/var/gobuilder/buildlet/go/src/os/exec/exec.go:715 +0x788

That goroutine is blocked on this line:
https://cs.opensource.google/go/go/+/master:src/os/exec/exec.go;l=813;drc=29253f4d513f1d3fe549db39a8daa311492138b3
which is dominated by an unconditional call to Kill on the command's *os.Process:
https://cs.opensource.google/go/go/+/master:src/os/exec/exec.go;l=768;drc=29253f4d513f1d3fe549db39a8daa311492138b3
The implementation of *os.Process.Kill is pretty straightforward, so I would be pretty shocked if that doesn't end up making the correct syscalls on NetBSD. 😅
os/exec.TestCancelErrors and os.TestStdPipe might be good cases to focus on, because they involve only single subprocesses. (Many of the other failures here end up stuck on go build, but in those cases more often than not go build is itself blocked on a stuck invocation of cmd/compile, which makes the goroutine dump for the test itself look — correctly! — like it is blocked on a go subprocess that is still running.)

Is there a document anywhere that lays out how this works on each OS, or roughly how things like the syscalls and cgo dynamic symbol calls are put together?

I'm not aware of any particular document. (I would expect to find it in https://cs.opensource.google/go/go/+/master:src/runtime/HACKING.md, but it's not there.)
A short summary of my understanding:

On darwin (including ios), aix, solaris (including illumos), openbsd (#61546 notwithstanding), and windows, the runtime always makes its system calls through a dynamically-linked libc or kernel32.dll.

On linux (including android), freebsd, netbsd, dragonfly, plan9, and wasm, the runtime makes direct system calls using assembly routines.

However, if cgo is enabled and the program ends up linking against a package that uses it (often the net package), the runtime calls through libc for functions specific to thread management (declared in src/runtime/cgo.go).
That behavior is toggled at link time using the runtime.iscgo variable.

It looks like the os test doesn't have any dependencies that would involve dynamic linking. (Notably, it doesn't end up importing net.) I'm not entirely sure whether it is purely statically linked in practice, but you could check by building the test binary with go test -c os.
One of these hangs was reported in os.TestStdPipe in https://github.com/golang/go/issues/57999#issuecomment-1684522180.

I examined a go1.20 executable on netbsd-9 with readelf --dynamic and it certainly looks like it's linked against libpthread:

As @bcmills says, this is going to vary depending on whether the Go program imports any packages that call into C code.  I've verified that on NetBSD the os package tests are running a program that is not dynamically linked.  This is the test used for TestStdPipe that @bcmills mentioned.

If you're calling the _lwp_create syscall directly instead of using the pthread_create symbol, but you're calling any other functions via dynamic symbols, that will get you in trouble and having a newer libpthread.so with the rtld/pthread fix won't help. There is probably lots of other trouble you can get into if you bypass the pthread symbols -- I would have to think more about it.

When dynamically linked, Go programs will call pthread_create to create new threads.
That said, there are cases where dynamically linked programs will call the LWP functions.  The functions I see that may be called are lwp_park, lwp_unpark, lwp_self, and lwp_kill.

Can you be more specific about what the difference is?

What we are seeing is that a Go program starts a subprocess, waits for it to complete, and the wait6(P_PID, pid, WEXITED|WNOWAIT) call never returns.  If there were a problem such that programs fail to exit, I would expect to also see cases where the build and test run does not complete without reporting any error about waiting for a subprocess.  As far as I know we aren't seeing such a problem.

Hmm. I guess it would be possible to use runtime.Stack to detect when the test has been stuck in wait6 for too long — but then wouldn't we have to start some other new process to prevent the test from deadlocking when it sends itself SIGSTOP?
(You're welcome to send a CL to implement that, and I'll review it, but it seems like it would add a lot of extra complexity to an already-complex situation. 😅)

The flake above is on 10_BETA. So much for my hope that this was fixed in a newer NetBSD version :(

@golang/netbsd, is it possible that this is (somehow) a kqueue-related race condition?
#63937 on darwin is plausibly kqueue-related, and I wonder if they might share a similar root cause.

@golang/netbsd, is it possible that this is (somehow) a kqueue-related race condition?
#63937 on darwin is plausibly kqueue-related, and I wonder if they might share a similar root cause.

Are you thinking of a race in the way Go uses kqueue, or a race in kqueue itself?

I suspect a race in the way Go uses kqueue.


Connection not terminated for invalid/ill-formed requests

If a client sends an incorrect request the connection should be terminated. What happens now is that the connection is not terminated and new data are appended to the buffer but the data from the buffer are never removed so the buffer grows. Kestrel does throw if the data is incorrect (https://github.com/aspnet/KestrelHttpServer/blob/dev/src/Microsoft.AspNet.Server.Kestrel/Http/Frame.cs#L560)  but the exception is then caught logged and swallowed (https://github.com/aspnet/KestrelHttpServer/blob/dev/src/Microsoft.AspNet.Server.Kestrel/Http/Connection.cs#L87)
Repro steps:

With telnet send \r\n to the server

Expected behavior:

the connection is closed

Actual behavior:

you can continue typing. with each new character an exception will be thrown on the server side. buffer grows. the connection is never closed

Should we do some fuzzing tests in general?

Generally, any malformed request currently throws off Kestrel in various ways. I had created a couple of failing tests where generally a connection would leak, but I guess no one looked at it.
BTW, is there any way to test for leaking connections? @DamianEdwards mentioned these in various community standups and I guess they have been fixed, but there doesn't seem to be infrastructure to actually verify it.

This commit which just made it into RC1 allows Kestrel to handle requests with empty headers.
We still should add fuzz testing and limit start line length, header length, etc...

This is a potential stress issue so I'd like us to take a look at this for RC2.

/cc @yuwaMSFT @taoliseki FYI.

The last commit of https://github.com/aspnet/KestrelHttpServer/pull/619 should resolve this

Though not all @Tragetaschen's scenarios

Fixed in https://github.com/aspnet/KestrelHttpServer/issues/683.

Actually this is a dup of https://github.com/aspnet/KestrelHttpServer/issues/683.


Add env vars to configuration the tusd service

Is your feature request related to a problem? Please describe.
I using golang-plugins for external behaviours and using internal abstractions of tusd (composer, store) then I would like configuring main tusd and my plugins via common var envs, but tusd use only flags.
Describe the solution you'd like
I propose to use env tags. For example:
S3Bucket   string `envconfig:"S3_BUCKET"  required:"false"`
S3Endpoint string `envconfig:"S3_ENDPOINT"  required:"false"`
Basepath   string `envconfig:"BASE_PATH"  required:"false"`
HttpHost   string `envconfig:"HTTP_HOST"  required:"false"`
HttpPort   string `envconfig:"HTTP_PORT"  required:"false"`

Can you provide help with implementing this feature?
Yes. I can add envconfig tags (or an alternative lib) and rename the config abstraction from "flags" to config for use flags and envs together

I am open to adding this as it seems useful in certain situations. We should probably just discuss which library to use for this. A quick search also revealed https://github.com/jnovack/flag, which is advertised as a drop-in replacement for Go's flag package. Since we currently use flag, this would mean minimal changes to our code. Do you have other recommendations?

@Acconut thanks, https://github.com/jnovack/flag seems good compromise!

Would you be interested in trying it out? :)


Upcoming pickups box is merging with the navigation bar when scrolled

 Chechout the above link you can find a screenshot in which upcoming box was coinciding with navigation bar. Sorry if I am wrong.

I can't reproduce this bug. As your browser says 'localhost', I guess you did not actually encounter this behaviour on karrot.world or dev.karrot.world?

Yeah you are right, thankyou. Closing the issue.

It seems like you are using Safari, right? Currently, karrot is not well-tested on Safari, so I would be happy about feedback (best with screenshots).
I would start to mark these issues with a Safari and a Starter Task label, so new contributors that also use Safari can fix these kinds of bugs.


C# sample for using Text-to-Speech

Would be helpful to have a C# sample (and in other languages) of speech synthesis to help people get started quickly with using the API.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6ba5bbd9-c69d-fbfe-6392-f5cea4ebfb08
Version Independent ID: 617eeeb5-0ad6-f54c-7d36-edeb2e6d9581
Content: Use Text to Speech using Speech services - Microsoft Cognitive Services
Content Source: articles/cognitive-services/Speech-Service/how-to-text-to-speech.md
Service: cognitive-services
GitHub Login: @v-jerkin
Microsoft Alias: v-jerkin

@jjgriff93 Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

the samples are here:  https://github.com/Azure-Samples/Cognitive-Speech-TTS

@szhaomsft Thanks!
@jjgriff93 We will now proceed to close this thread. If there are further questions regarding this matter, please respond here and @YutongTie-MSFT and we will gladly continue the discussion.


Add the Qwt library

Also includes some other cleanup

A few optional comments.  Biggest I suppose is qwt not having a dynamic/shared option at all.  If that is ok and intentional, this should be good to go.

#approved  looks good @bnhhughes


findjvm.py: print() is a function in Python 3

- print "MessageRate =", float(NUMMSGS) / (time.time() - t0)
+ print("MessageRate =", float(NUMMSGS) / (time.time() - t0))

And similar changes to avoid SyntaxErrors on Python 3.

Thanks for fixing that. Did you actually try to run these examples? Do they provide any meaningful output?
I think they are not used/updated for many years.


onyo mv: confusing output

I did
❱ onyo mv repair/dp/Jan\ Leonhardt/laptop_lenovo_ThinkpadP14s.fauxC4D6tW dp/Jan\ Leonhardt
The following will be moved:
/home/jleo/onyo-test/repair/dp/Jan Leonhardt/laptop_lenovo_ThinkpadP14s.fauxC4D6tW -> /home/jleo/onyo-test/dp/Jan Leonhardt

the output reads like the new name will be "Jan Leonhardt". So suggestions might be:
laptop_lenovo_ThinkpadP14s.fauxC4D6tW:  /home/jleo/onyo-test/repair/dp/Jan Leonhardt/-> /home/jleo/onyo-test/dp/Jan Leonhardt/

or
/home/jleo/onyo-test/repair/dp/Jan Leonhardt/laptop_lenovo_ThinkpadP14s.fauxC4D6tW -> /home/jleo/onyo-test/dp/Jan Leonhardt/laptop_lenovo_ThinkpadP14s.fauxC4D6tW

I think this is a duplicate of #442. @j-leo can you confirm?

Yes, it seems to be a duplicate, sorry for that!
closing :)