Implements unmanaged-cluster profiles

Users can specify --profile via flag or config to automatically
install a profile from an installed package repo.
Users can set the --profile-version to install a specific version.
This is optional and will default the the latest semver if not
specified
Users can set the --profile-config-file for a values yaml to be
consumed by the installed profile
Also includes a change to logging of packages being reconciled
so the name is visible to users when reconciling them
Updates our usage function for cobra so a nicely formatted line break
for long flag messages don't wrap around

Which issue(s) this PR fixes
Fixes: #3537
Describe testing done for PR
Install a profile by name:
$ tanzu unmanaged-cluster create test --profile fluent-bit.community.tanzu.vmware.com

📁 Created cluster directory

🧲 Resolving and checking Tanzu Kubernetes release (TKr) compatibility file
   projects.registry.vmware.com/tce/compatibility
   Compatibility file exists at /home/jmcb/.config/tanzu/tkg/unmanaged/compatibility/projects.registry.vmware.com_tce_compatibility_v3

🔧 Resolving TKr
   projects.registry.vmware.com/tce/tkr:v0.17.0-dev-1
   TKr exists at /home/jmcb/.config/tanzu/tkg/unmanaged/bom/projects.registry.vmware.com_tce_tkr_v0.17.0-dev-1
   Rendered Config: /home/jmcb/.config/tanzu/tkg/unmanaged/test/config.yaml
   Bootstrap Logs: /home/jmcb/.config/tanzu/tkg/unmanaged/test/bootstrap.log

🔧 Processing Tanzu Kubernetes Release

🎨 Selected base image
   projects.registry.vmware.com/tce/kind:v1.22.4

📦 Selected core package repository
   projects.registry.vmware.com/tce/repo-10:0.10.0

📦 Selected additional package repositories
   projects.registry.vmware.com/tce/main:0.11.0

📦 Selected kapp-controller image bundle
   projects.registry.vmware.com/tce/kapp-controller-multi-pkg:v0.30.1

🚀 Creating cluster test
   Cluster creation using kind!
   ❤  Checkout this awesome project at https://kind.sigs.k8s.io
   Base image downloaded
   Cluster created
   To troubleshoot, use:

   kubectl ${COMMAND} --kubeconfig /home/jmcb/.config/tanzu/tkg/unmanaged/test/kube.conf

📧 Installing kapp-controller
   kapp-controller status: Running

📧 Installing package repositories
   tkg-core-repository package repo status: Reconcile succeeded
   projects.registry.vmware.com-tce-main-0.11.0 package repo status: Reconcile succeeded

🌐 Installing CNI
   calico.community.tanzu.vmware.com:3.22.1

🌐 Installing Profile fluent-bit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 1.7.5
   Installing profile with no configuration file

✅ Cluster created

🎮 kubectl context set to test

View available packages:
   tanzu package available list
View running pods:
   kubectl get po -A
Delete this cluster:
   tanzu unmanaged delete test

$ k get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
fluent-bit           fluent-bit-k8xq6                             0/1     Running   0          51s
kube-system          calico-kube-controllers-556b886dc8-4zc4q     1/1     Running   0          48s
kube-system          calico-node-94s4j                            1/1     Running   0          48s
kube-system          coredns-78fcd69978-cxzbd                     1/1     Running   0          95s
kube-system          coredns-78fcd69978-hkknt                     1/1     Running   0          95s
kube-system          etcd-test-control-plane                      1/1     Running   0          108s
kube-system          kube-apiserver-test-control-plane            1/1     Running   0          108s
kube-system          kube-controller-manager-test-control-plane   1/1     Running   0          108s
kube-system          kube-proxy-hj9ns                             1/1     Running   0          95s
kube-system          kube-scheduler-test-control-plane            1/1     Running   0          108s
local-path-storage   local-path-provisioner-85494db59d-bf9ws      1/1     Running   0          95s
tkg-system           kapp-controller-779d9777dc-k8xm7             1/1     Running   0          95s

$ k get packageinstalls -n tkg-system
NAME                                    PACKAGE NAME                            PACKAGE VERSION   DESCRIPTION           AGE
cni                                     calico.community.tanzu.vmware.com       3.22.1            Reconcile succeeded   64s
fluent-bit.community.tanzu.vmware.com   fluent-bit.community.tanzu.vmware.com   1.7.5             Reconcile succeeded   64s

Users may also provide just the prefix of the package name and the install will do a best effort to match the prefix of a package:
$ tanzu unmanaged-cluster create --profile fluent-bit
...

🌐 Installing Profile fluent-bit
   Installing profile without version specified. Using version 1.7.5
   Installing profile with no configuration file

...

❯ k get packageinstalls -A
NAMESPACE    NAME         PACKAGE NAME                            PACKAGE VERSION   DESCRIPTION   AGE
tkg-system   cni          calico.community.tanzu.vmware.com       3.22.1            Reconciling   15s
tkg-system   fluent-bit   fluent-bit.community.tanzu.vmware.com   1.7.5             Reconciling   15s

Install a profile and specify the version in the profile mapping:
$ tanzu unmanaged-cluster create test --profile external-dns.community.tanzu.vmware.com:0.8.0
...

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Using profile version 0.8.0
   Installing profile with no configuration file

...

Selects the latest version if no version tag specified. (note, this includes some code from golang/x/mod/semver to parse semantic versions)
$ tanzu unmanaged-cluster create test --profile external-dns.community.tanzu.vmware.com
...

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Installing profile without version specified. Using version 0.10.0
   Installing profile with no configuration file

...

The latest keyword is also reserved and will select the latest semver tag
$ tanzu unmanaged-cluster create test --profile external-dns.community.tanzu.vmware.com:latest
...

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Installing profile without version specified. Using version 0.10.0
   Installing profile with no configuration file

...

If also using a config, users can use :: to skip defining a version and select the latest one (or the latest keyword)
$ tanzu unmanaged-cluster create test --profile external-dns.community.tanzu.vmware.com::my-config.yaml
...

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Installing profile without version specified. Using version 0.10.0
   Using config my-config.yaml

...

Include a values yaml file to be consumed by the package profile. Using external DNS as an example here
---
#! The namespace in which to deploy ExternalDNS.
namespace: my-cool-namespace

#! Deployment related configuration
deployment:
  args:
  - --source=service
  - --source=contour-httpproxy
  - --txt-owner-id=k8s
  - --domain-filter=k8s.example.org
  - --namespace=my-services-ns
  - --provider=rfc2136
  - --rfc2136-host=<IP_ADDRESS>
  - --rfc2136-port=53
  - --rfc2136-zone=k8s.example.org
  - --rfc2136-tsig-secret=MTlQs3NNU=
  - --rfc2136-tsig-secret-alg=hmac-sha256
  - --rfc2136-tsig-keyname=externaldns-key
  - --rfc2136-tsig-axfr
  env: []
  securityContext: []
  volumeMounts: []
  volumes: []

#! Service account related configuration
serviceaccount:
  annotations:
    key: value

Using the values yaml file:
$ tanzu unmanaged-cluster create test \
  --profile external-dns.community.tanzu.vmware.com:0.10.0:values.yaml

...

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Using profile version 0.10.0
   Using config values.yaml

...

$ k get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS      AGE
kube-system          calico-kube-controllers-6849b5c7d-c5w9z      1/1     Running   0             6m5s
kube-system          calico-node-cht4q                            1/1     Running   0             6m5s
kube-system          coredns-78fcd69978-4csnt                     1/1     Running   0             6m55s
kube-system          coredns-78fcd69978-tjsng                     1/1     Running   0             6m55s
kube-system          etcd-test-control-plane                      1/1     Running   0             7m8s
kube-system          kube-apiserver-test-control-plane            1/1     Running   0             7m8s
kube-system          kube-controller-manager-test-control-plane   1/1     Running   0             7m8s
kube-system          kube-proxy-wrrfk                             1/1     Running   0             6m55s
kube-system          kube-scheduler-test-control-plane            1/1     Running   0             7m8s
local-path-storage   local-path-provisioner-85494db59d-c9nj5      1/1     Running   0             6m55s
my-cool-namespace    external-dns-557c9c6487-7d9q5                1/1     Running   4 (50s ago)   6m8s
tkg-system           kapp-controller-779d9777dc-sq6pb             1/1     Running   0             6m55s

$ k get packageinstalls -n tkg-system
NAME                                      PACKAGE NAME                              PACKAGE VERSION   DESCRIPTION           AGE
cni                                       calico.community.tanzu.vmware.com         3.22.1            Reconcile succeeded   6m33s
external-dns.community.tanzu.vmware.com   external-dns.community.tanzu.vmware.com   0.10.0            Reconcile succeeded   6m33s

Users can specify multiple profiles via multiple flags
$ tanzu unmanaged-cluster create test --profile external-dns.community.tanzu.vmware.com --profile fluent-bit.community.tanzu.vmware.com

...

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Using profile version 0.10.0
   Using config values.yaml

🌐 Installing Profile fluent-bit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 1.7.5
   Installing profile with no configuration file

Users can use a multiple profile mappings delimitted by a ,
# this will generate 3 profiles:
# - external-dns version 0.10.0 using the external-values.yaml file
# - a fluent-bit profile at the latest version with no config file
# - the app toolkit, version 0.1.0 with the values.yaml file
# Each profile is delimited by a `,` and each value is delimited by `:`
$ tanzu unmanaged-cluster create test \
  --profile external-dns.community.tanzu.vmware.com:0.10.0:external-values.yaml,fluent-bit.community.tanzu.vmware.com,app-toolkit.community.tanzu.vmware.com:0.1.0:values.yaml

results in:
🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Using profile version 0.10.0
   Using config external-values.yaml

🌐 Installing Profile fluent-bit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 1.7.5
   Installing profile with no configuration file

🌐 Installing Profile app-toolkit.community.tanzu.vmware.com
   Using profile version 0.1.0
   Using config values.yaml

Users can specify profiles in their config file:
ClusterName: test
KubeconfigPath: ""
ExistingClusterKubeconfig: ""
NodeImage: ""
Provider: kind
ProviderConfiguration: {}
Cni: calico
CniConfiguration: {}
PodCidr: <IP_ADDRESS>/16
ServiceCidr: <IP_ADDRESS>/16
TkrLocation: ""
AdditionalPackageRepos: []
PortsToForward: []
SkipPreflight: false
ControlPlaneNodeCount: "1"
WorkerNodeCount: "0"
Profiles:
- name: fluent-bit.community.tanzu.vmware.com
- name: external-dns.community.tanzu.vmware.com
  config: external-values.yaml
  version: 0.10.0
- name: app-toolkit.community.tanzu.vmware.com
  config: values.yaml

and during deploy:
$ tanzu unmanaged-cluster create -f test.yaml
...

🌐 Installing Profile fluent-bit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 1.7.5
   Installing profile with no configuration file

🌐 Installing Profile external-dns.community.tanzu.vmware.com
   Using profile version 0.10.0
   Using config external-values.yaml

🌐 Installing Profile app-toolkit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 0.1.0
   Using config values.yaml

Bigger example installing app toolkit:
$ tanzu unmanaged-cluster create test --profile app-toolkit.projects.registry.vmware.com::values.yaml
...

🌐 Installing Profile app-toolkit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 0.1.0
   Using config values.yaml

...

$ k get packageinstalls -A
NAMESPACE                   NAME                                     PACKAGE NAME                                          PACKAGE VERSION   DESCRIPTION           AGE
tanzu-package-repo-global   cartographer                             cartographer.community.tanzu.vmware.com               0.2.2             Reconcile succeeded   65s
tanzu-package-repo-global   cert-manager                             cert-manager.community.tanzu.vmware.com               1.6.1             Reconcile succeeded   99s
tanzu-package-repo-global   contour                                  contour.community.tanzu.vmware.com                    1.19.1            Reconcile succeeded   99s
tanzu-package-repo-global   fluxcd-source-controller                 fluxcd-source-controller.community.tanzu.vmware.com   0.21.2            Reconcile succeeded   99s
tanzu-package-repo-global   knative-serving                          knative-serving.community.tanzu.vmware.com            1.0.0             Reconcile succeeded   99s
tanzu-package-repo-global   kpack                                    kpack.community.tanzu.vmware.com                      0.5.1             Reconcile succeeded   99s
tkg-system                  app-toolkit.community.tanzu.vmware.com   app-toolkit.community.tanzu.vmware.com                0.1.0             Reconcile succeeded   109s
tkg-system                  cni                                      calico.community.tanzu.vmware.com                     3.22.1            Reconcile succeeded   109s

$ k get pods -A
NAMESPACE             NAME                                          READY   STATUS      RESTARTS   AGE
cartographer-system   cartographer-controller-86996bbb9-72gnz       1/1     Running     0          46s
cert-manager          cert-manager-56d644484c-8hz72                 1/1     Running     0          103s
cert-manager          cert-manager-cainjector-c77cb865-pwzm4        1/1     Running     0          103s
cert-manager          cert-manager-webhook-d7c4c8dc9-dsllw          1/1     Running     0          103s
flux-system           source-controller-5cfc8d5f94-jdn2g            1/1     Running     0          54s
knative-serving       activator-5d89dc486f-tmwgx                    1/1     Running     0          70s
knative-serving       autoscaler-7c7d7bb749-sz2k6                   1/1     Running     0          69s
knative-serving       controller-8c878f645-xd87v                    1/1     Running     0          70s
knative-serving       domain-mapping-544594f79b-5cs6v               1/1     Running     0          68s
knative-serving       domainmapping-webhook-5c56fcf66-5dk2s         1/1     Running     0          68s
knative-serving       net-certmanager-controller-8657b7f5dc-pxrn6   1/1     Running     0          70s
knative-serving       net-certmanager-webhook-765c755c57-l6lrx      1/1     Running     0          70s
knative-serving       net-contour-controller-59b89f9594-qhhzg       1/1     Running     0          70s
knative-serving       webhook-84679d6476-jxjst                      1/1     Running     0          68s
kpack                 kpack-controller-8b84485dc-pf7wh              1/1     Running     0          70s
kpack                 kpack-webhook-7674b5447-dttg5                 1/1     Running     0          70s
kube-system           calico-kube-controllers-6487584b54-tcsgt      1/1     Running     0          113s
kube-system           calico-node-tjvs2                             1/1     Running     0          113s
kube-system           coredns-78fcd69978-4x2qt                      1/1     Running     0          3m5s
kube-system           coredns-78fcd69978-qcljl                      1/1     Running     0          3m5s
kube-system           etcd-test-control-plane                       1/1     Running     0          3m21s
kube-system           kube-apiserver-test-control-plane             1/1     Running     0          3m18s
kube-system           kube-controller-manager-test-control-plane    1/1     Running     0          3m19s
kube-system           kube-proxy-jkkqx                              1/1     Running     0          3m5s
kube-system           kube-scheduler-test-control-plane             1/1     Running     0          3m18s
local-path-storage    local-path-provisioner-85494db59d-q6qdw       1/1     Running     0          3m5s
projectcontour        contour-6bc54547c4-5sclw                      1/1     Running     0          105s
projectcontour        contour-6bc54547c4-q7lv4                      1/1     Running     0          105s
projectcontour        contour-certgen-v1.19.1--1-gx458              0/1     Completed   0          105s
projectcontour        envoy-q9d97                                   2/2     Running     0          98s
tkg-system            kapp-controller-779d9777dc-qvnbk              1/1     Running     0          2m43s

Users can drop in the profile flag and mappings into the config command:
$ tanzu unmanaged-cluster config --profile fluent-bit:1.7.5:values.yaml test
Wrote configuration file to: test.yaml

$ cat test.yaml
ClusterName: test
KubeconfigPath: ""
ExistingClusterKubeconfig: ""
NodeImage: ""
Provider: kind
ProviderConfiguration: {}
Cni: calico
CniConfiguration: {}
PodCidr: <IP_ADDRESS>/16
ServiceCidr: <IP_ADDRESS>/16
TkrLocation: ""
AdditionalPackageRepos: []
PortsToForward: []
SkipPreflight: false
ControlPlaneNodeCount: "1"
WorkerNodeCount: "0"
Profiles:
  - name: fluent-bit
    config: values.yaml
    version: 1.7.5

Pushed another commit that makes it possible to set multiple profiles - would be good to get a few eyes on this implementation!
This PR still needs a bit of work:

[ ] Documentation updates
[ ] Update the help messages in the CLI to reflect setting multiple profiles
[ ] Write a few more tests for profile parsing
[ ] Re-reviews of code implementations

@jpmcb @joshrosso I can't comment about code, but I have to say that I am not in favour of this implementation as it will lead to a lot of problems/issues coming from the community abusing the loose checks of this feature.
An example, from the testing John did.
tanzu unmanaged-cluster create test --profile external-dns.community.tanzu.vmware.com

This package will never be reconciled because it requires a value that is not provided https://github.com/vmware-tanzu/community-edition/tree/main/addons/packages/external-dns/0.10.0#external-dns-configuration

Heard and thanks for voicing this. We're still going to move ahead with this implementation and mark it as an experimental feature.
It will be on the user to know whether they need to include a values file.
In the future we'll aim to make this smarter, especially once our packages have YTT schemas where they declare their required values. This plumbing is still valid for that future state.

I have to say that I am not in favour of this implementation as it will lead to a lot of problems/issues coming from the community abusing the loose checks of this feature.

Totally heard. Unfortunately, today, we don't have a great way to "pre-check" that a values yaml file is needed or that a provided values yaml file matches the expected schema.
This first pass will give users the ability to install and based on future user feedback, we can iterate and make this a better user experience.

@jorgemoralespou could be good to get you sync'd up with @vkorbes and I if the primary contention is naming.
Let's get some time to talk it through, we have room to change the name prior to April 19th.

@jpmcb question (i'm not sure)
should we fully fail when a profile doesn't install?
$ tanzu unmanaged-cluster create my-cluster3 --tkr=projects.registry.vmware.com/tce/tkr:v0.17.0 --profile fluent-bit.community.tanzu.vmware.com

📁 Created cluster directory

🧲 Resolving and checking Tanzu Kubernetes release (TKr) compatibility file
   projects.registry.vmware.com/tce/compatibility
   Compatibility file exists at /home/josh/.config/tanzu/tkg/unmanaged/compatibility/projects.registry.vmware.com_tce_compatibility_v3
   Custom TKr projects.registry.vmware.com/tce/tkr:v0.17.0 NOT found in compatibility file. Proceed with caution, the provided TKr may not work with this version of unmanaged-cluster

🔧 Resolving TKr
   projects.registry.vmware.com/tce/tkr:v0.17.0
   TKr exists at /home/josh/.config/tanzu/tkg/unmanaged/bom/projects.registry.vmware.com_tce_tkr_v0.17.0
   Rendered Config: /home/josh/.config/tanzu/tkg/unmanaged/my-cluster3/config.yaml
   Bootstrap Logs: /home/josh/.config/tanzu/tkg/unmanaged/my-cluster3/bootstrap.log

🔧 Processing Tanzu Kubernetes Release

🎨 Selected base image
   projects.registry.vmware.com/tce/kind:v1.22.4

📦 Selected core package repository
   projects.registry.vmware.com/tce/repo-10:0.10.0

📦 Selected kapp-controller image bundle
   projects.registry.vmware.com/tce/kapp-controller-multi-pkg:v0.30.1

🚀 Creating cluster my-cluster3
   Cluster creation using kind!
   ❤  Checkout this awesome project at https://kind.sigs.k8s.io
   Base image downloaded
   Cluster created
   To troubleshoot, use:

   kubectl ${COMMAND} --kubeconfig /home/josh/.config/tanzu/tkg/unmanaged/my-cluster3/kube.conf

📧 Installing kapp-controller
   kapp-controller status: Running

📧 Installing package repositories
   tkg-core-repository package repo status: Reconcile succeeded

🌐 Installing CNI
   calico.community.tanzu.vmware.com:3.22.1

🌐 Installing Profile fluent-bit.community.tanzu.vmware.com
could not resolve profile. Error: no package was resolved for named fluent-bit.community.tanzu.vmware.com with version
Error: exit status 14

I wonder if, similar to CNI, we'd be better off printing a warning and still letting the cluster go through with the rest of its bootstrap.

should we fully fail when a profile doesn't install? ... I wonder if, similar to CNI, we'd be better off printing a warning and still letting the cluster go through with the rest of its bootstrap

@joshrosso - I agree, I think we should probably hand the cluster back over to the user (like we do with CNI) and not fail fully. Good catch.

How will the user, if he gota cluster back that some packages were not installed successfully? Will that be part of the status or output?
Is this the behavior you would expect in the tools you use?

How will the user, if he gota cluster back that some packages were not installed successfully? Will that be part of the status or output?

It will not. Similar to CNI, we will not block the install on the reconciliation of PackageInstall->App.
We are open to considering this out-of-band from this work in a separate issue.

Is this the behavior you would expect in the tools you use?

Depends on the tool. Every time I do a kubectl apply I expect this exact behavior

Looks like we may have a regression in cluster deletion. If the cluster fails to be created, that normally creates the cluster metadata folder under ~/.config/tanzu/tkg/unmanaged/[cluster-name]. That does still happen.
The difference is normally we can then call tanzu uc rm and it will delete the metadata directory. When doing that after failing to create a cluster with this change, I now get:
$ ./artifacts/linux/amd64/cli/unmanaged-cluster/v0.12.0-dev.1/tanzu-unmanaged-cluster-linux_amd64 create foo --profile=kpack.community.tanzu.vmware.com:0.5.1
directory /home/smcginnis/.config/tanzu/tkg/unmanaged/foo already exists, this cluster must be deleted before proceeding
~/src/vmware-tanzu/community-edition (unmanaged/profiles)
$ ./artifacts/linux/amd64/cli/unmanaged-cluster/v0.12.0-dev.1/tanzu-unmanaged-cluster-linux_amd64 rm foo

🧪 Deleting cluster: foo
Failed to delete cluster. Error: failed to locate a config file at /home/smcginnis/.config/tanzu/tkg/unmanaged/foo/config.yaml

It should try to get information, if it can't get enough to delete the cluster assume it is gone, then remove the metadata folder.
Not sure if that is directly caused by any of these changes, or if the failure condition this is hitting is different than most I've encountered before.

should we fully fail when a profile doesn't install?

I think we actually should. This functionality makes it less likely that a user is interactively running this and helps enable the use case of scripting the deployment of a tanzu cluster in a pipeline or something like that.
In this case, the only way they have to validate that the deployment was successful is to inspect the return code from calling create. If they are saying they want a cluster created with a certain set of packages, and we return 0, they continue on thinking everything worked and they have the environment they expect to have for their given scenario. Then if they try to use this cluster to deploy a workload, it won't be immediately clear why it is failing if the cluster environment they are deploying isn't actually configured the way they expect it to be.

Trying to test locally and I can't get past this error:
🧲 Resolving and checking Tanzu Kubernetes release (TKr) compatibility file
   projects.registry.vmware.com/tce/compatibility
   Downloaded to: /home/smcginnis/.config/tanzu/tkg/unmanaged/compatibility/projects.registry.vmware.com_tce_compatibility_v3
failed parsing the TKr compatibility file. Error: could not find compatible CLI version in compatibility file

Looks like we may have a regression in cluster deletion. If the cluster fails to be created, that normally creates the cluster metadata folder under ~/.config/tanzu/tkg/unmanaged/[cluster-name]. That does still happen.
The difference is normally we can then call tanzu uc rm and it will delete the metadata directory. When doing that after failing to create a cluster with this change, I now get:
$ ./artifacts/linux/amd64/cli/unmanaged-cluster/v0.12.0-dev.1/tanzu-unmanaged-cluster-linux_amd64 create foo --profile=kpack.community.tanzu.vmware.com:0.5.1
directory /home/smcginnis/.config/tanzu/tkg/unmanaged/foo already exists, this cluster must be deleted before proceeding
~/src/vmware-tanzu/community-edition (unmanaged/profiles)
$ ./artifacts/linux/amd64/cli/unmanaged-cluster/v0.12.0-dev.1/tanzu-unmanaged-cluster-linux_amd64 rm foo

🧪 Deleting cluster: foo
Failed to delete cluster. Error: failed to locate a config file at /home/smcginnis/.config/tanzu/tkg/unmanaged/foo/config.yaml

It should try to get information, if it can't get enough to delete the cluster assume it is gone, then remove the metadata folder.
Not sure if that is directly caused by any of these changes, or if the failure condition this is hitting is different than most I've encountered before.

I don't believe this is a regression produced by this PR. I have requested it's fixed as part of the minikube provider work.
https://github.com/vmware-tanzu/community-edition/pull/3674#issuecomment-1086229724

should we fully fail when a profile doesn't install?

I think we actually should. This functionality makes it less likely that a user is interactively running this and helps enable the use case of scripting the deployment of a tanzu cluster in a pipeline or something like that.
In this case, the only way they have to validate that the deployment was successful is to inspect the return code from calling create. If they are saying they want a cluster created with a certain set of packages, and we return 0, they continue on thinking everything worked and they have the environment they expect to have for their given scenario. Then if they try to use this cluster to deploy a workload, it won't be immediately clear why it is failing if the cluster environment they are deploying isn't actually configured the way they expect it to be.

Thanks for the input. At this point, I'm not aligned, because I feel as though we're taking a UX hit for the sake of having exhaustive error codes. Open to reconsidering blocking and validating package installs in the context of a separate issue. If we do that, we need to start doing the same for CNI, which we don't do today.

OK, things are looking good now:
$ tanzu uc create test --tkr projects.registry.vmware.com/tce/tkr:v0.17.0-dev-1 --profile fluent-bit.community.tanzu.vmware.com

📁 Created cluster directory

🧲 Resolving and checking Tanzu Kubernetes release (TKr) compatibility file
   projects.registry.vmware.com/tce/compatibility
   Compatibility file exists at /home/smcginnis/.config/tanzu/tkg/unmanaged/compatibility/projects.registry.vmware.com_tce_compatibility_v3
   Custom TKr projects.registry.vmware.com/tce/tkr:v0.17.0-dev-1 NOT found in compatibility file. Proceed with caution, the provided TKr may not work with this version of unmanaged-cluster

🔧 Resolving TKr
   projects.registry.vmware.com/tce/tkr:v0.17.0-dev-1
   TKr exists at /home/smcginnis/.config/tanzu/tkg/unmanaged/bom/projects.registry.vmware.com_tce_tkr_v0.17.0-dev-1
   Rendered Config: /home/smcginnis/.config/tanzu/tkg/unmanaged/test/config.yaml
   Bootstrap Logs: /home/smcginnis/.config/tanzu/tkg/unmanaged/test/bootstrap.log

🔧 Processing Tanzu Kubernetes Release

🎨 Selected base image
   projects.registry.vmware.com/tce/kind:v1.22.4

📦 Selected core package repository
   projects.registry.vmware.com/tce/repo-10:0.10.0

📦 Selected additional package repositories
   projects.registry.vmware.com/tce/main:0.11.0

📦 Selected kapp-controller image bundle
   projects.registry.vmware.com/tce/kapp-controller-multi-pkg:v0.30.1

🚀 Creating cluster test
   Cluster creation using kind!
   ❤  Checkout this awesome project at https://kind.sigs.k8s.io
   Base image downloaded
   Cluster created
   To troubleshoot, use:

   kubectl ${COMMAND} --kubeconfig /home/smcginnis/.config/tanzu/tkg/unmanaged/test/kube.conf

📧 Installing kapp-controller
   kapp-controller status: Running

📧 Installing package repositories
   tkg-core-repository package repo status: Reconcile succeeded
   projects.registry.vmware.com-tce-main-0.11.0 package repo status: Reconcile succeeded

🌐 Installing CNI
   calico.community.tanzu.vmware.com:3.22.1

🌐 Installing Profile fluent-bit.community.tanzu.vmware.com
   Selected package fluent-bit.community.tanzu.vmware.com
   Installing profile without version specified. Using version 1.7.5
   Installing profile with no configuration file

✅ Cluster created

🎮 kubectl context set to test

View available packages:
   tanzu package available list
View running pods:
   kubectl get po -A
Delete this cluster:
   tanzu unmanaged delete test

$ k get po -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
fluent-bit           fluent-bit-g5bvb                             1/1     Running   0          58s
kube-system          calico-kube-controllers-5757f95646-hgs4t     1/1     Running   0          57s
kube-system          calico-node-5ch9g                            1/1     Running   0          57s
kube-system          coredns-78fcd69978-klx89                     1/1     Running   0          102s
kube-system          coredns-78fcd69978-xhcpn                     1/1     Running   0          102s
kube-system          etcd-test-control-plane                      1/1     Running   0          116s
kube-system          kube-apiserver-test-control-plane            1/1     Running   0          115s
kube-system          kube-controller-manager-test-control-plane   1/1     Running   0          116s
kube-system          kube-proxy-frkbj                             1/1     Running   0          102s
kube-system          kube-scheduler-test-control-plane            1/1     Running   0          116s
local-path-storage   local-path-provisioner-85494db59d-4fjnn      1/1     Running   0          102s
tkg-system           kapp-controller-779d9777dc-vfnvd             1/1     Running   0          102s

$ k get app -A
NAMESPACE    NAME                                    DESCRIPTION           SINCE-DEPLOY   AGE
tkg-system   cni                                     Reconcile succeeded   70s            78s
tkg-system   fluent-bit.community.tanzu.vmware.com   Reconcile succeeded   68s            77s

🧲 Resolving and checking Tanzu Kubernetes release (TKr) compatibility file
projects.registry.vmware.com/tce/compatibility
Downloaded to: /home/smcginnis/.config/tanzu/tkg/unmanaged/compatibility/projects.registry.vmware.com_tce_compatibility_v3
failed parsing the TKr compatibility file. Error: could not find compatible CLI version in compatibility file

Hmmm what version do you get when you run tanzu um version? This is the top of the v3 compatibility file:
version: v1
unmanagedClusterPluginVersions:
- version: dev
  supportedTkrVersions:
  - image: projects.registry.vmware.com/tce/tkr:v0.17.0-dev-1

Might be because I've been running the binary directly which gets dev as the version:
$ ./unmanaged-cluster version
dev

but yes, 💯 to what josh said on specifying the --tkr flag!

@joshrosso @stmcginnis - addressed a nit and ready for another review/merge 👀

Hmmm what version do you get when you run tanzu um version?

I initially was running that too:
$ ./artifacts/linux/amd64/cli/unmanaged-cluster/v0.12.0-dev.1/tanzu-unmanaged-cluster-linux_amd64 version
v0.12.0-dev.1

The comments on #3895 sound good to me.

Necromancing this to pick up the naming discussion.
We want something that means "packages that get installed during cluster setup." On a scale from descriptive to facetious, suggestions:

--install-packages
--setup-packages
--pre-install
--setup
--preload
--bootstrap
--kickoff

We're picking one by EOD tomorrow (Apr 14), please do chime in with your preferences or other suggestions. Thanks!

My vote would be --preload or --preinstall, in that order.

My vote would be --install-packages or --pre-install in that order

I don't mind for any of these 3: --preload, --pre-install, --install-packages

I would go for --pre-install-packages

For a potential interim solution, my preference is --install-packages.
My primary tiny criticism for options like --preload are that they feel just as ambiguous as --profile.

From those in the list, --install-packages is my vote

I think --preload is not clear in what it means and doesnt indicate something would be installed rather it indicates something would be imported into a cluster like kind has the idea of preloaded images but they are just added to the cri level and not run in the cluster

+1 to --install-packages

+1 for --install-packages

+1 --install-packages with the hopes that also includes meta-packages

@thesteve0 isn't a meta-package just another package?

A meta package is simply a package that installs other packaves and in yesterdays demo app toolkit was installed this way so shouldnt be an issue at all

As per the majority of votes, we're going with --install-packages. Thanks so much everyone who chimed in!


AppCenter Test + Device Leases

Are there any plans to support "leasing" a device for use in a DevOps pipeline for a period of time instead of only supporting test workloads that run directly on the device unassisted?  I have scenarios where using adb directly on a device is necessary to orchestrate testing - ideally a DevOps task could be used to get a device that is accessible for all of the remaining tasks in the pipeline.

@cofitz1 The comments feature of our documentation is intended to enable customers to suggest changes to the docs or ask clarifying questions about the docs.
App Center support is provided directly from within the App Center portal. Any time you need help, just click the blue chat button in the lower-right corner of any page and our dedicated support team will respond. For additional information, see the App Center Help Center.

Redirecting this to the correct github repo :)


ocrmypdf: 11.3.3 -> 11.6.0

Motivation for this change

Update to latest upstream release 11.6.0
Update license (was relicensed from GPL3 to MPL 2.0 and MIT)

Change log: https://github.com/jbarlow83/OCRmyPDF/blob/master/docs/release_notes.rst#v1160
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 112411 run on x86_64-linux 1

1 package built:

ocrmypdf

@ofborg eval

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 112411 run on x86_64-linux 1

1 package built:

ocrmypdf


Add support for Laravel 9

hello,
do you plan support for Laravel 9 ?
thanks

Hi,
Yes, I'm going to run some tests tomorrow and create a new release within the next few days.

Version 1.3.0 with Laravel 9 supported has been released.


Need to compile templates into one file for production

Recompiling is very slow so we should make it once.

Landed in 0.1.2


Adding a word about reform_2_0! in README

Just addressed the same issue in #208

Your version is better :)
Closing this ;)


Android 10 looping in boot screen

Hi,
System setup: $uname -a Linux useruser 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux. CPU: Ryzen 5 3600, 32GBRAM
I'm currently trying to build & run Android 10 using the provided Tutorial.
Building seems to work fine, executing m -j results in #### build completed successfully (53:59 (mm:ss)) ####.
However, when I try to run the emulator the GUI gets stuck in the boot screen (Image attached). In my shell I get the following output in a loop.
[  674.288299] init: Sending signal 9 to service 'zygote' (pid 1295) process group...
[  674.289602] libprocessgroup: Successfully killed process cgroup uid 0 pid 1295 in 0ms
[  674.290992] init: updatable process 'zygote' exited 4 times before boot completed
[  674.291499] init: Init cannot set 'ro.init.updatable_crashing' to '1': Read-only property was already set
[  674.296649] init: Command 'write /sys/power/state on' action=onrestart (<Service 'zygote' onrestart>:2) took 0ms and failed: Unable to write to file '/sys/power/state': Unable to write file contents: Invalid argument
[  674.309014] init: Sending signal 9 to service 'audioserver' (pid 1296) process group...
[  674.338266] libprocessgroup: Successfully killed process cgroup uid 1041 pid 1296 in 28ms
[  674.362204] init: Command 'restart audioserver' action=onrestart (<Service 'zygote' onrestart>:3) took 53ms and succeeded
[  674.362828] init: Command 'restart cameraserver' action=onrestart (<Service 'zygote' onrestart>:4) took 0ms and failed: service cameraserver not found
[  678.703257] healthd: battery l=100 v=5000 t=25.0 h=2 st=4 c=900 fc=3000000 cc=10 chg=

The full emulator log is attached (At the end i just ctrl+c the process in the shell)
I also waited 3:30h until i canceled it in another run, it kept looping the same messages like in the attached log (I also redownloaded and rebuilt everything from the start in this second run).
Anyone got ideas what I did wrong or how to fix this? Is my CPU too weak? If you need more information or other logs I'm happy to provide them.
log.txt

please use the android 12 branch instead or check the System boot fail section in https://github.com/riscv-android-src/riscv-android/blob/main/doc/common_faults_and_treatment_methods.md


nn_pruning doesn't seem to work for T5 Models, Roberta-based Models

Hi @madlag @julien-c @co42 @srush @Narsil
I am trying to use nn_pruning for Pruning different transformer models.
Code:
model_checkpoint = "t5-small"
t5small_model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint).to(device)
mpc.patch_model(t5small_model)

t5small_model.save_pretrained("models/patched")

Error:
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
[<ipython-input-47-602943fc51a1>](https://localhost:8080/#) in <module>()
     1
     2 t5small_model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint).to(device)
----> 3 mpc.patch_model(t5small_model)
     4
     5 t5small_model.save_pretrained("models/patched")

[/usr/local/lib/python3.7/dist-packages/nn_pruning/patch_coordinator.py](https://localhost:8080/#) in patch_model(self, model, trial)
   640             patched_count += 2 * layers_count
   641
--> 642         assert (patcher.stats["patched"] == patched_count)
   643
   644         if layer_norm_patch:

AssertionError:

[Colab] (https://colab.research.google.com/drive/1Gz7rozG8NbeBtsiWXjGNQ5wnVU7SE_Wl?usp=sharing)

Hi @madlag @julien-c @co42 @srush @Narsil
I am trying to use nn_pruning for Pruning different transformer models.
Code:
model_checkpoint = "t5-small"
t5small_model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint).to(device)
mpc.patch_model(t5small_model)

t5small_model.save_pretrained("models/patched")

Error:
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
[<ipython-input-47-602943fc51a1>](https://localhost:8080/#) in <module>()
     1
     2 t5small_model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint).to(device)
----> 3 mpc.patch_model(t5small_model)
     4
     5 t5small_model.save_pretrained("models/patched")

[/usr/local/lib/python3.7/dist-packages/nn_pruning/patch_coordinator.py](https://localhost:8080/#) in patch_model(self, model, trial)
   640             patched_count += 2 * layers_count
   641
--> 642         assert (patcher.stats["patched"] == patched_count)
   643
   644         if layer_norm_patch:

AssertionError:

[Colab] (https://colab.research.google.com/drive/1Gz7rozG8NbeBtsiWXjGNQ5wnVU7SE_Wl?usp=sharing)

I have the same problem, did you fix it? @shubham-krishna

I have the same problem, did you fix it? @robotsp @ghost

sorry I don't know, I can't really help.


bug: Image always builds (context always out of date)

What happened?
pulumi up always says the contextHash has changed, and always rebuilds, only to discover that the output has not changed. Immediately re-running will again say context hash is different, event after refresh etc.
Example
pulumi up diff:
    ~ docker-build:index:Image: (update)
        [id=sha256:0a49b414b7272b3b4458cadce0cc62d5c658f5a1c44e5997cf2c12c4cf56f62c]
        [urn=urn:pulumi:dev::myproject::docker-build:index:Image::image-indexer]
        [provider=urn:pulumi:dev::myproject::pulumi:providers:docker-build::default_0_0_6::286a454e-0463-46e7-a196-75e5bd1d6601]
      - contextHash: "0e6f2bec5f86c06b6e95001047445b621ace8b81ac0508bd8770f4f8654a882e"

Output of pulumi about
CLI
Version      3.129.0
Go Version   go1.22.6
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host
OS       darwin
Version  14.6.1
Arch     arm64

...

Backend
Name          <my-machine>
URL            file://<my-home-dir>
User           <my-username>
Organizations
Token type     personal

Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@brettwillis sorry you're running into this. We need some additional information in order to debug the problem:

Can you please include the full output of pulumi about, including the version of docker-build you're using?
Can you please include the code you're running as part of your program?

In general it's very difficult to diagnose issues like this that are sensitive to the build environment, so the more information you can provide the better!

Sorry yes here's the provider versions from the stack output, docker-build v0.0.6.
pulumi:providers:cloudflare                       urn:pulumi:dev::hhc::pulumi:providers:cloudflare::default_5_36_0
pulumi:providers:gcp                              urn:pulumi:dev::hhc::pulumi:providers:gcp::default_7_37_0
pulumi:providers:docker-build                     urn:pulumi:dev::hhc::pulumi:providers:docker-build::default_0_0_6
pulumi:providers:pulumi-nodejs                    urn:pulumi:dev::hhc::pulumi:providers:pulumi-nodejs::default

And the relevant part of the program (sorry I should have known to include this in the first place!):
const indexerImage = new docker.Image('image-indexer', {
  tags: [
    pulumi.interpolate`${imageRegistryUrl}/indexer:v${indexerPkg.version}`,
    pulumi.interpolate`${imageRegistryUrl}/indexer:latest`,
  ],
  context: {
    location: '../service-indexer',
  },
  platforms: [
    'linux/amd64',
  ],
  push: true,
  pull: true,
  secrets: {
    'bun': pulumi.secret(pulumi.interpolate`[install.scopes]\nmyorg = { url = "https://npm.pkg.github.com/", token = "${config.requireSecret('github_packages_token')}" }\n`),
  },
}, {
  dependsOn: imageRegistry,
  deletedWith: imageRegistry,
});

@brettwillis ahh I see now, thank you!
It's always re-building due to pull: true -- this should definitely be clearer in the docs.
The original thinking behind this behavior was that pull-ing images is typically intended to build against the most current tags, so we should always perform the build to be safe. If we didn't perform the build, and if you didn't touch your Dockerfile for a while, your image could get out of sync with the latest available base images which seems undesirable.
I'm curious about your use case for pulling, though? And does the reasoning above make sense to you or is it unreasonable?

@blampe ah yes that seems obvious now when you point it out.

I'm curious about your use case for pulling, though? And does the reasoning above make sense to you or is it unreasonable?

Yeah I imagined something a little different in my mind: when I need to do a rebuild due to actual context changes, then you might as well pull the latest images while you're at it... or when it's been a while since the last build... However I can certainly see the reasoning behind the current behaviour.
So maybe what I should be doing instead is specifying the fully-qualified tags of my images in my Dockerfile, then manually updating those when I re-deploy whenever I deem it appropriate.
But yes at least two points of feedback:

making this clearer in the docs
perhaps (if possible) making it clearer in the update diff somehow (e.g. highlighting pull: true or something instead of missing context)

Yep, that's all super reasonable.
One of the aims of the provider has been to stay as close as possible to the native CLI, and this is one area where we're deviating in a surprising way. For example running docker buildx build --pull won't do a full re-build unless the base images have changed. We took a shortcut here because it's a little involved to figure out which images you depend on, but strictly speaking it's possible -- especially considering that we're already parsing your Dockerfile.
In other words, we could:

Parse your Dockerfile and pull out all your base images.
Store an output like "dependencies" which tracks all your dependent base images and tags. This might be mildly useful just to see when your dependencies change, and it would make the diff a little more obvious.

Then, when pull: true is specified:

Issue a HEAD request on your dependent images to find their current digests -- this is fast and can be parallelized.
Compare remote digests with existing dependencies and perform a build if they differ.

That way you would get something much closer to Docker's own behavior -- if your app and remotes are both unchanged nothing happens.

That sounds like it would be amazing!


Local config file

I recently found fw and am interested in this project as a good way to manage all of my projects. I do have some issues using fw for my needs and would like to help contribute some ideas and features to help improve fw.
I use multiple different machines with varying different configurations. To help keep my sanity I manage a set of dotfiles that I use on all my machines.  I would like to be able to commit the config file of fw into my dotfiles. My problem is that on different machines my workspace root is in different locations.
Arch: workspace = ~/path/to/workspace
Windows: workspace = D:\path\to\workspace
I would like some way to override my global config file settings depending on the machine I am on. My suggestion is to be able to set a local config file that will be loaded after the global and override any settings defined in there. This way I can have the default path on windows in config file defined by some env variable like $FW_LOCAL_PATH
Also if I am at work and only want to add projects on this local machine this would be a great place to add them.
Would like to know your thoughts on this.

We already had a lot of discussions about this. I think @icepuma suggested we should support a tag folder. Ever .tag file would define one tag. This would allow us to not only share tags between configurations but also with each other. The same thing could be done with .repo files. Although I think tags would be sufficient.
If you want to only share some tags you could achieve this using 2 folders and symlinks.
This would be a really powerful feature and the only reason we haven't implemented it yet is because we're lazy.
What do you think?
The annoying thing to change are commands that modify the configuration. Some of them would require some serious thought about how they should behave.
On March 4, 2019 9:20:01 PM GMT+01:00, James Simpson<EMAIL_ADDRESS>wrote:

I recently found fw and am interested in this project as a good way to
manage all of my projects. I do have some issues using fw for my needs
and would like to help contribute some ideas and features to help
improve fw.
I use multiple different machines with varying different
configurations. To help keep my sanity I manage a set of
dotfiles that I use on all my
machines.  I would like to be able to commit the config file of fw into
my dotfiles. My problem is that on different machines my workspace root
is in different locations.
Arch: workspace = ~/path/to/workspace
Windows: workspace = D:\path\to\workspace
I would like some way to override my global config file settings
depending on the machine I am on. My suggestion is to be able to set a
local config file that will be loaded after the global and override
any settings defined in there. This way I can have the default path on
windows in config file defined by some env variable like
$FW_LOCAL_PATH
Also if I am at work and only want to add projects on this local
machine this would be a great place to add them.
Would like to know your thoughts on this.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/brocode/fw/issues/89

How about splitting up all the structures in the config file into their own files. fw would now by default look for a config file in the directory $XDG_CONFIG_HOME/fw. This file can be overridden by the env variable $FW_CONFIG_PATH. In this directory we could have a tag folder to store all of the tags.
This folder can be then added to someone's versioned dotfiles and be used on any machine they use.
To have local configuration for a machine we could also define $FW_LOCAL_PATH. fw would also read this directory after the global one and merge the two settings files together where the local config would override the global one. any projects and tags would only be defined for this machine.
An example would be
.
├── .config
│  └── fw
│     ├── config
│     ├── projects
│     └── tags
│        ├── one.tag
│        ├── three.tag
│        └── two.tag
├── .local
│  └── fw
│     ├── config
│     ├── projects
│     └── tags
│        └── another.tag
└── tree

Also while refactoring how configuration is stored could we add some way to define that override-path in a project or workspace in a tag is relative to the workspace root. This would make it easier to organize my workspace. Have to create a tag and then add it to all projects that I want to group together.

I guess that would be a really good way to remove the drawbacks of the current implementation. It would require a major release. I don't like it when tools stop working with an upgrade so fw should be able to migrate automatically to the new structure.
@mriehl thoughts?
If we do this I will grant you access to a branch here so we can work together on this.
I have to think about this.

I created a 2.0 branch. You should have an invite.

I also like this idea a lot, would simplify many things without being a hassle to maintain. +1 for backwards compatiblity, maybe an interactive upgrade that ensures user consent before we start moving/deleting files around.

For upgrading we could have a switch in the setup command like fw setup --upgrade '~/path/to/.fw.json' or create an actual command called upgrade that would handle the upgrade for you. If we are wanting to change the version again I think the upgrade command might be a better option as it would be more flexible.

I was also looking at the other issues listed for fw and in regards to the issue #74, with this restructure is there any ideas on how to handle this request? Or is it necessary with the ability to set a local paths well and relative override-paths?

I think fw could automatically detect that an upgrade is necessary. No need for an extra command. Then interactively prompt the user asking for permission to perform the upgrade and backup the old configuration file.
Regarding #74:I think everything asked for can be done with the new layout. You can have 2 fw config folders and share items using symlinks.
I'm not sure how we should sync what we do. Maybe create issues for different parts of the refactoring? I don't have time until the weekend anyway.

For help with merging the global and local folders together we could lever https://crates.io/crates/config

Looks like they only support reading. I'm nor sure if such an abstraction is good for us. The commands to manipulate the config are probably harder to migrate to the new system.
We should probably start with the migration to the new layout so we can test and discuss the structure before we modify anything else.

To add another use-case, I'd love to commit my public-facing projects into a fw config in my public dotfiles repo, but work/private projects be manually added to the machine I'm using at the time. I don't want references to those projects floating around on the public internet. I think the solution you're talking about supports that!
In terms of commands that edit the config, perhaps default to modifying the global config, and allow modifying the local config with a flag? That should (hopefully) remove the need to use the config library to write.
For example:
$ fw add git@github.com:/brocode/fw.git
$ fw add --local<EMAIL_ADDRESS>

@CruizeMissile
@mriehl and I started with the migration.
Idea:
fw
+ projects
  + default
     - fw
     - fblog
  + tags
     - t1
- settings.toml

@CruizeMissile i pushed some code on the 2.x branch which is how I picture that reading the 2.0 configuration should work. Can you have a look if that's compatible with what you had in mind? The idea is that we walk the config directories, and you can just symlink your custom config in there to get what you want.

Hey @mriehl! This is a good start but, I am unsure of using symlinks to add any local to that machine files. With most
people that have their config files managed in a vcs they would have $FW_CONFIG_PATH (default $XDG_CONFIG_HOME/fw)
tracked. They would have to create an ignore for every local link that you made to somewhere else, or ignore everything
and manualy state file files they don't want to ignore. This is one of the reasons that I would like to see a $FW_LOCAL_PATH that
users can declare for that machine. There would be some rules on how the program will read file two config directories.

$FW_CONFIG_PATH will be read for parsed for the settings information.
If $FW_LOCAL_PATH is defined then this folder will be parsed for the settings information. The settings information
will be merged into the settings that have been loaded by the first step. If there are any settings that overlap then
the information in the $FW_LOCAL_PATH will override $FW_CONFIG_PATH
Default projects and tags can be read from $FW_CONFIG_PATH
Default projects and tags can be read from $FW_LOCAL_PATH
If there are any projects or tags that are defined in both locations, $FW_LOCAL_CONFIG will be used

These parsing rules make it easy to override any default behavior for a local machine. With this approach we do not
touch the tracked $FW_CONFIG_PATH directory with machine specific information and this is stored in one location. Also
this helps people like me that would like to use this to manage all of their personal projects and projects on their
work computers that should not be anywhere near my personal projects. A global option can be defined to say that I am
operating in the local path (-l, --local). So say that I only want to sync local projects I can say that like:
fw sync --local
@mriehl  and @bomgar  Let me know your thoughts on these ideas.

It should be easy to add FW_LOCAL_PATH to the current implementation. I don't think the drawbacks without it are that bad. You don't have to link multiple files. One local folder symlink for projects is enough to have a local subtree for projects. This would mean one entry in the gitignore file. Or if it just links to .local/fw/projects/ you could even commit that.
The one thing that is missing with this approach is the settings file. This is probably important because of the credentials. But if I would separate settings for work and home I'm not even sure I would need a global one.
My only gripe with this is the added maintenance overhead.

Now all commands in the 2.0 branch use the new config layout. A lot of this is untested but the basic stuff seems to work. This is not really finished yet because the cli interface does not support the new folder layout. At the moment everything new is added to the default folder.
Anyway this is something that can be tested and improved.
@CruizeMissile @liamdawson thoughts?

fw migrate fails to execute because I was using the $XDC_CONFIG_HOME directory as my old config path and it has a file already exists error. Not sure how to bet way to ignore just a file already exists error. Other than that looks good so far

The migration with a custom config location has more problems than that. The code uses the same variable for the old and new location. So it is impossible to migrate a custom setup.
But this should be easy to fix.

@CruizeMissile i reverted the commits you made. I told you I'm not happy with the config library and with the local path. I want to discuss this first. I also don't like commits directly into the 2.0 branch with code that we have an active discussion about. Please use pull requests from now on.

I wonder if we should allow a settings file per directory in the project dir. So the user could use different github tokens or other things based on the "group".
@mriehl thoughts?

I did a lot of test. The biggest challenge no is to update the documentation. Everything else seems to be working properly. I probably start using this version for a while to see if anything weird happens.

@bomgar I apologize for not discussing more about the changes I made. As for the changes I made they are addressing the core of the issue I opened. Right now there is no way to override configuration locally. I understand that you can add local projects and tags with symlinks and ignore the directories in your dotfiles. As for the settings file there is no way to override the settings. There needed to be a way to define the override for the machine (env variable) and then a way to merge the settings from the local and global. There is no crate that handles toml/config merging, that is why I was using the config crate. @bomgar @mriehl do you have any ideas on how to tackle these?

I agree that we should have a special handling for the settings file. I'm not sure what to do about it. I need to think about this. As for the config libray: fw has enough dependencies already and it is annoying to update them. I would like to reduce them. The settings struct has 6 fields. It shouldn't be that hard to write something to merge that.

That is understandable, I could work on something to merge toml files.

The toml file makes less sense the more I think about it. Maybe we should just have one file per setting.
On May 10, 2019 4:35:45 PM GMT+02:00, James Simpson<EMAIL_ADDRESS>wrote:

That is understandable, I could work on something to merge toml files.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/brocode/fw/issues/89#issuecomment-491311369

Ok the options are:

As @CruizeMissile suggested add FW_LOCAL_SETTINGS and point to a settings file.
a. merge with settings file
b. use just the new file
Move every setting to a separate file.
Support $HOSTNAME_settings.toml in config dir
a. merge with settings file
b. use just the new file
Support ${FW_ENV}_settings.toml in config dir
a. merge with settings file
b. use just the new file
Allow everything to be set by environment variables (we probably should do this anyway for the auth tokens)
a. only do this
b. Do one of the others and this

@CruizeMissile @mriehl @icepuma I would really appreciate your input on this. I don't want to just decide.

IMHO the current tree structure solves all use cases. But we need to support config file overlaying I think.
Something like
.config/fw
-------settings.toml (?)
-------personal
-----------settings.toml
------------------projects
----------------------my-cool-project
----------------------my-other-cool-project
------------------tags
-----------------------my-go-workspace-tag
-------work
-----------settings.toml
------------------projects
----------------------cash-script
----------------------ibims-1-repo
------------------tags
-----------------------such-professional-very-taglike

the settings.toml in personal would dictate the config for the projects in personal.
At that point I'm not even sure if the root settings.toml even makes sense. Merging is a delicate thing prone to all sorts of clusterfucks, and maybe each "module" folder should simply have its own, explicit config. And maybe at that point it also makes sense to simply dictate that everything is in a flat subfolder of .config/fw and that all commands support a module (or something) argument that dictates what is used.
Things like fw sync --module personal, fw add --module afterwork<EMAIL_ADDRESS>etc (with the argument being mandatory on things like add).
That's my 2c anyway

@mriehl that looks good but would mean we need to refactor everything again. If someone would pay me to do that I would, but I think it is to late now.

Seeing @mriehl suggestion would be the best approach although I feel that the global config would need to be present and 'module' settings would override it.

Yeah as @bomgar wrote it requires quite a lot of work (currently there is a globally loaded configuration for the projects, but we'd need to have a per-project configuration in memory or at least some kind of tree structure)

I favor 4b because it is really easy to implement and I don't see any problems it would not solve. For security reasons we should support tokens as environment variables no matter what else we do.
@CruizeMissile anything with your use case that would not work? Is one of the others better.
If someone is willing to reimplement everything like @mriehl suggested I would also be ok with that but I'm not really sure the benefits are worth that investment (I personally won't invest more time to do that).
The question is what problem is not solved by one of the easy solution?

@CruizeMissile @mriehl I would like to continue working on this and finishing the 2.0 release. It would be great if we reach some kind of conclusion.

4b would work for me as i can just override the envs on each machine

All done. I'm going to use this from now on. if it works for a while without issues we can release it.
@CruizeMissile it should be easy to add 1b and/or 6 to the current implementation. I implemented 4b now and it is compatible with both other options. So if you want you can add one or both.
I updated the readme. Would be great if someone could have a look at it. (@icepuma, @mriehl )


[Sidenav] - Disabling backdrop / grey overlay

Bug, feature request, or proposal:
Was trying to remove/disable the grey backdrop when the sidenav is displayed.
I don't see any properties to set them.
I tried the two solutions provided here - https://stackoverflow.com/questions/33333446/removing-the-grey-overlay-backdrop-from-mdsidenav
But they did not work.
Below is my plunker link
https://plnkr.co/edit/Zxup6owofUqA18tQ01Z5?p=preview
What is the expected behavior?
I am trying to add a sidenav in the right side of the page.
When the sidenave is opened, i need to push the content, so that user can still continue working in the main content. For this I do not want the grey backdrop. I tried but it is not working.
What is the current behavior?
The backdrop covers the main content thus disabling it for interaction and edits.
What are the steps to reproduce?
Provided in plunker link: https://plnkr.co/edit/Zxup6owofUqA18tQ01Z5?p=preview
What is the use-case or motivation for changing an existing behavior?
Trying few work around, but did not work. Is there a way to do so ?
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
All Latest
Is there anything else we should know?
NO

Try this
https://plnkr.co/edit/uALEO3f6IrHm1gJosNnr?p=preview

@willshowell  It took some tome to realize the change you made in index.html

Hacky it is than, for angular material v5 its
.mat-drawer-backdrop.mat-drawer-shown {
      display:none;
}

This css did a trick for me:
::ng-deep .mat-drawer-backdrop.mat-drawer-shown {
  display: none;
}

Is there a better way so far?


Type for options for POST request not exported

The type used for POST requests is IWithData<phin.IOptions> but the IWithData is not exported in types.d.ts so there's no way to construct a typed options object without making a custom type.
Also, there's a note in the types about using the generics to make JSON/Form posting mutually exclusive, but wouldn't this work just as well?
export interface IOptions extends IOptionsBase {
 parse?: 'none'
}

interface WithData {
  data: object;
}

interface WithForm {
  form: {
    [index: string]: string;
  }
}

export type IOptionsWithData = IOptions & WithData;
export type IOptionsWithForm = IOptions & WithForm;
// etc

Thanks for pointing that out. A PR would be appreciated!

@ethanent PR submitted https://github.com/ethanent/phin/pull/65
I kept the generics but explicitly made the T extend IOptionsBase and updated the data property of IWithData to be compatible with what Centra accepts.


get next_run from within job

Consider the example:
#!/usr/bin/env python3

import time
import logging

import schedule

LOGGER = logging.getLogger()

def perform_job(notify_user_id):
    time.sleep(10.0)
    LOGGER.info('Job done. Will sleep now until %s.', schedule.next_run())

def main():
    schedule.every().day.at("10:00").do(perform_job)
    schedule.every().day.at("18:00").do(perform_job)
    LOGGER.info('Scheduler started. Will now sleep until %s.',
                schedule.next_run())
    while True:
        schedule.run_pending()
        time.sleep(1)

if __name__ == '__main__':
    main()

The problem is, Job done. Will sleep now until %s will tell me about the "next job" that's actually the current one because perform_job hadn't finished yet. If run_pending returned a boolean saying whether anything was executed I could use it to solve the problem, but that's not the case. Are there any other solutions?

Hi @d33tah, thanks for raising this question. Unfortunately it is not possible to retrieve the next_run of a job from within the job itself. This is because the timestamp at which the job finishes is used to calculate the next run of the job. That's why the next_run is calculated after the job finishes. Since we don't know when the job finishes while the job is in progress there is no way to calculate when the next run takes place.


Redesign settings menu to avoid undecorated Godot UI

[ ]  Tab container
[ ] OK button, quit button
[ ] Sound + Graphics Tab; sliders, checkboxes, dropdowns
[ ] Gameplay tab; checkboxes, dropdowns, buttons
[ ] Controls tab; buttons
[ ] Miscellaneous tab; buttons, dropdowns

I think we should go with a simple UI like Puyo Puyo or Fall Guys -- monochrome, big bold fonts, easy to read. I think the fun colorful font is OK for the game menus, but not for complex places like this.

I've sketched out some mockups for the kind of UI we might use. I'll solicit feedback from the community and come up with a decision.

The settings menu has been redesigned to use a custom UI.

Notes:
border width: 3, 6
corner radius: 6
content margin: 4, 8
expand margin: 0, 3


Plugins in Webassembly

Description
I know crashing is fun and stuff, but would you as the developers of Hyprland be opposed to injecting plugins via WASM ?
It could offer many features, security, allow C++ and other languages to compile plugins, and most probably: less bug reports by users who used plugins but clain the plugin is not responsible for a crash ;)
I dont want you to include the whole WASM runtime here, just asking for a general direction if you have a strong opinion about it because someone starts working and investing time into this

No, defeats the purpose of plugins.


Improvements made to the O365 starter intranet solution

Q
A

Bug fix?
yes?

New feature?
yes

What's in this Pull Request?

Added a carousel component and links on the home page + sample data.
Added some additional details in the README file about prerequisites and troubleshooting.
Added a custom theme for the site (.spcolor)
Updated the PnP provisioning schema to 201605
Fixed a bug with moment.js where dates were not translated in french pages.
Updated Nuget package versions in the extensibility provider solution. Now it uses the latest version of the PnP core assembly.

thx Franck.


[process-agent] Set HintMask in CollectorProc during process checks.

What does this PR do?
Process check can now populate the hints field added into the payload. The process discovery hint will be added every certain amount of checks done. This is a configurable setting that is currently set at every 60 checks.
Motivation
https://datadoghq.atlassian.net/jira/software/projects/PROC/boards/132?selectedIssue=PROC-2069
Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

I ran some benchmarks to compare the bit method against the enum method and these are the results:
Bits
BenchmarkProcessCheck-10          8227      143255 ns/op    120182 B/op      996 allocs/op
Enum
BenchmarkProcessCheck-10          7095      169513 ns/op    120267 B/op      997 allocs/op
Seems like the bits method does perform better.


Docstring Improvements - linting

Description
I finally used the ruff linter in ARMI (and hand tweaked the results)!  This PR just cleans up our docstrings and unifies them with our standards.  While this is a very large PR in terms of number of files/lines, I believe it is exceptionally safe to merge as it is just docstrings.
(I have spoken to John B and even he agrees that docs-only PRs are safe for the release cycle.)
Tony/Arrielle, this PR is so big I thought maybe two reviewers could split the files they look at?  Maybe splitting all the boring scrolling?
Low Priority: I know everyone is busy.

Checklist

[X] This PR has only one purpose or idea.
[X] Tests have been added/updated to verify that the new/changed code works.

[X] The code style follows good practices.
[X] The commit message(s) follow good practices.

[X] The release notes (location doc/release/0.X.rst) are up-to-date with any bug fixes or new features.
[X] The documentation is still up-to-date in the doc folder.
[X] The dependencies are still up-to-date in setup.py.

Okay, the code coverage difference is that this line was uncovered in the code coverage report:
https://github.com/terrapower/armi/blob/02d3d2b9b58bbb27b6d2f5c4b270fa98ea6c30f0/armi/utils/__init__.py#L420
I would argue that I did not touch that line, and this is some wiggle in our code coverage.  (Our unit tests have some random nature to them, so they don't touch the same exact lines of code every time.)
This is a totally new one to me; I have never seen this one before. I thought I had squashed all of these testing inconsistencies.  I will have to add a test to ensure these line is covered every time. OH well.


Remove the old JBoss image on the PATRIC web page

See the image at the browser tab:

this if favicon. Not sure whether we have an alternative.


Made assembly color more noticeable

It's bothered me for a while; the Java and Assembly color look almost exactly alike. Here's a picture for a comparison.
Anyway, this pull request changes the Assembly color from #a67219 to #6E4C13, which makes it look darker and more distinctive compared to Java.

:metal: thanks!


Update: Postman.Postman version 7.24.0

Removing old versions to reduce clutter
Microsoft Reviewers: Open in CodeFlow

/azp run


EVER marketing strategy

Is your feature request related to a problem? Please describe.
It looks like we have a problem with a product development. Not a code, but product at all. It is not a decentralisation problem, but just a task we can fulfill
Describe the solution you'd like
Revise marketing.
First of all, an analysis of all available data is needed, this will require the participation of the entire community. Even if these data are not too impressive. The results would be nice to discuss with Initials.
Then, we must create closed working groups. At least 1. Groups should include not only developers - also marketers, sociologists, economists. Group proposals can be processed according to all bureaucratic procedures in MainGov.
...
PROFIT!!!111
Describe alternatives you've considered
Can wait, maybe it will happen randomly, before we reach the bottom
Additional context
Budget: 0 EVER

@EVER2d the DeFi Alliance already has groups for marketing and business development
Please contact me in telegram (https://t.me/isheldon) if you want to join


Show only current persp in modestring

This is more of a suggestion than anything,
What about being able to only show the current persp in the mode/headerline instead of the entire list with the dividers and all?
I have it in my modeline, and since I have quite a few different perspectives, this list completely clutters my modeline.
Having just the current selected persp in the modestring, I could still be able to quickly see the entire list by calling "persp-switch".
Cheers!

Good idea. I keep persp-show-modestring off in my own use, which is why I haven’t run into this problem. I’ll try to get to implementing this in a few days, but in the meanwhile, try putting the following in your configuration after Perspective loads:
(defun persp-update-modestring ()
  (when persp-show-modestring
    (set-frame-parameter nil 'persp--modestring (persp-current-name))))

This is completely untested. 😁 If it doesn’t work, you’ll have to reload Perspective, since it overrides an existing function.

This is mor a suggestion and workaround than a direct answer but I think it can be useful information.
To avoid this problem I use mini-modeling (https://github.com/kiennq/emacs-mini-modeline) which displays the modeline in the modeline buffer. Effectively it means that there is only one modeline, instead of one per window. The result is a much longer modeline, with enough space for displaying perspective names.

@gcv
Thanks, this works for now. It, for some reason is stuck to the rest of the modeline content. By that I mean it could use a space or something...but it works!
@dams That's a brilliant suggestion! I think I'll even use it in tandem with my suggestion, since the less clutter the better in my book.

@titibandit: You can fix the spacing by adjusting the format string in the definition below:
(defun persp-update-modestring ()
  (when persp-show-modestring
    (set-frame-parameter nil 'persp--modestring (format " %s" (persp-current-name)))))

I added a new customization setting, persp-modestring-short. Set it to t to enable this feature (don't forget to get rid of the hacked up persp-update-modestring).

Great, this works!
I can then close the issue.
However, I would like to still have the perspective to be displayed using the persp-selected-face face, which I could do using:
(defun persp-update-modestring ()
  "Update the string to reflect the current perspectives.
Has no effect when `persp-show-modestring' is nil."
  (when persp-show-modestring
    (let ((open (list (nth 0 persp-modestring-dividers)))
          (close (list (nth 1 persp-modestring-dividers)))
          (sep (nth 2 persp-modestring-dividers)))
      (set-frame-parameter nil 'persp--modestring
           (append open
                   (if persp-modestring-short
                       (list (if (equal (persp-current-name) nil) nil (propertize (persp-current-name) 'face 'persp-selected-face)))
                     (persp-intersperse (mapcar 'persp-format-name
                                                (persp-names)) sep))
                   close)))))

And of course you can still "reverse" it by having a persp-selected-face that does nothing.
I don't know if that should be included, or if that only belongs to my personal config... you tell me!
Anyway, thanks for the commit :)

How about: (1) make a dedicated face for this purpose, like persp-modestring-short-face, and (2) make it a bit less loud than persp-selected-face, maybe just (:weight bold).
You don't need to do (if (equal something nil) ...), just (if something ...) suffices. :)
Feel free to open a PR.


S3 Storage (Digitalocean Spaces) compatibility issues

Describe the Bug
Digitalocean Spaces is an S3-compatible storage. Compared to others, it is a budget solution based on download data usage and API usage.
However, the BookStack arises following an error message when I try to upload an image as my book cover:
League\Flysystem\UnableToCheckDirectoryExistence
Unable to check existence for: uploads/images/cover_book/2023-09/test.jpg

Here is my .env configuration related to the storage part:
# s3 Storage Type
STORAGE_TYPE=s3
STORAGE_S3_KEY=*********************
STORAGE_S3_SECRET=*********************
STORAGE_S3_BUCKET=&&&&&&&&
STORAGE_S3_ENDPOINT=https://&&&&&&&&.sfo3.digitaloceanspaces.com

Steps to Reproduce
Create Digital Ocean Spaces

Log into the Digital Ocean
Under MANAGE, click Spaces
Click Create Spaces Bucket
Set datacenter region in SFO3
Disable Content Delivery Network
Set a unique Spaces Bucket name
Click Create a Spaces Bucket

Obtain API Key

Log into the Digital Ocean
Click API and choose Spaces Key
Click Generate New Key
Fill the KEY and SECRET into the .env configuration file

Upload an image
Debug mode should be activated before uploading:
# Debug Mode
APP_DEBUG=true

Create a new book
Select a random cover image (< 1Mb)
Save book
Error Message

Expected Behaviour
Files should be successfully uploaded to the Digitalocean Spaces.
Screenshots or Additional Context
PHP Version: 8.1.7
Browser Details
No response
Exact BookStack Version
v23.08.3

Hi @NianwenDan,
Unfortunately I cannot assure that every service that advertises "S3 compatibility" will be supported. Can ultimately come down to their implementation and level of support.
Looking at the error, this seems to be specifically failing on a ListObjectsV2 call, which DigitalOcean advertises to support, but I have not tested this live myself.
I get the feeling this may be related to permission control within spaces.
Does DigitalOcean offer permissions/controls on the "space", or key used for access, at all?
Worth looking for anything permission related that may affect "listing", or just accessing, files & folders.

Hi @NianwenDan, Unfortunately I cannot assure that every service that advertises "S3 compatibility" will be supported. Can ultimately come down to their implementation and level of support.
Looking at the error, this seems to be specifically failing on a ListObjectsV2 call, which DigitalOcean advertises to support, but I have not tested this live myself. I get the feeling this may be related to permission control within spaces.
Does DigitalOcean offer permissions/controls on the "space", or key used for access, at all? Worth looking for anything permission related that may affect "listing", or just accessing, files & folders.

Hi @ssddanbrown,
DigitalOcean Spaces does offer permissions to list objects, and I tried to set it as Enable File Listing. Which should give permission to everyone who wants to list objects in the bucket. It didn't work.

I also noticed that "The Spaces API does not support list-objects-v2 pagination." is a known issue. According to https://docs.digitalocean.com/products/spaces/details/limits/
I am more concerned if DigitalOcean Spaces running a different API or return type resulting in this issue.
So far, I do not think it is a bug for BookStack upload module. But could DigitalOcean Spaces be supported in the future?

@NianwenDan I've just been testing things against Digital Ocean spaces myself. All seems to work find from my testing, but a few specific things to note:

Your STORAGE_S3_ENDPOINT should not include the bucket name. That will result in the error you experienced.
"File Listing" in Digital Ocean does not need to be enabled.
You will need to make the uploads/images folder within your space publicly accessible, while ideally keeping other paths inaccessible. Could't find a nice way to do this in-platform, but you could set S3 policies via the API. I used s3cmd to set the policy in my bucket for this:

s3cmd setpolicy policy.json s3://bookstack-testing

With policy.json being as per the below. You'd have to change bookstack-testing in this to match your bucket/space name.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::bookstack-testing/uploads/images/*"
            ]
        }
    ]
}

For reference, here are the relevant BookStack settings I used:
STORAGE_TYPE=s3
STORAGE_S3_KEY=DO00DQNxxxxxxxxxxxx
STORAGE_S3_SECRET=PB9y5sR8Q4wP0lmxxxxxxxxxxxxxxxxxxxxxxx
STORAGE_S3_BUCKET=bookstack-testing
STORAGE_S3_ENDPOINT=https://ams3.digitaloceanspaces.com
STORAGE_URL=https://bookstack-testing.ams3.digitaloceanspaces.com

Hi @ssddanbrown,
I am sorry I didn't read Digital Ocean Spaces API documentation before.
That method did work for me.
Also, this issue tends to be more focused on how to set up the configuration file correctly, specifically for DigitalOcean Spaces. It would be better to change the tag from Bug to Support, and mark it closed.
Thanks for your time to help me out again!


Updating empty documents fails

Tries to get an _id, and fails.

Moving this to 0.4.0 since it's unlikely user will delete all contents from their document


Is it only possible to train one field at a time?

Is there a possibility to train the model on all fields in my dataset?
Something like:
python train.py --field { FIELDS } --batch_size
Now, I can only train one field at a time?

Ah, yes that is a limitation. You can only train individual models for one field at a time. Each trained InvoiceNet model can only handle a single field currently.


helmet-csp package vulnerability

Remove helmet and react-helmet to resolve package vulnerability. These packages are no longer used.
https://issues.redhat.com/browse/COST-513

Codecov Report

Merging #1650 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1650   +/-   ##
=======================================
  Coverage   77.36%   77.36%
=======================================
  Files         242      242
  Lines        4105     4105
  Branches      742      742
=======================================
  Hits         3176     3176
  Misses        830      830
  Partials       99       99

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2b471b...01f1336. Read the comment docs.


add expand and collapse icons

name: New icon
about: Add a new icon to the library
labels: "🎨 <icon"

Name of the icon : expand-[right | left | up | down] and collapse[right | left | up | down]

Tags (alternative names for this icon) (add them in as a separate json file using the same icon name) :

What is the purpose of this icon? : Shows that one can click it to expand or collapse an area (example: sidenav)

100% scale preview :

Have you considered alternative possibilities for its naming or design? :

Not sure, feels a lot like got to last/first, I would assume a collapse being just a simple chevron.

Closing this, as this has been superseded by #1158 which has more generic icon names.


Support spymemcached 2.x

hi @wu-sheng:
I provide a plugin about memcache client spymemcached lib.

Thanks for the contribution. Still, same as before, you need to provide screenshot about topology graph, trace stack and span info.

@IluckySi  you should resolve the conflicts first.

@wu-sheng How can I provide the screenshot about topology graph, trace stack and span info. I have images in local. Email? or other way?

Coverage increased (+0.09%) to 17.737% when pulling f3ce4037c911a0067345fea2a0ac5554775aadb7 on IluckySi:ilucky-skywalking-spymemcached into 4a874676e595808b3cc31dc8110a898af331d184 on wu-sheng:master.

ilucky-spymemcached-2.11.1.zip
readme.txt

Besides codes review, which I have no time to do, the concern is about git commit logs. These is nothing important there, but this pr includes 8 commits. @IluckySi Since this is a relative simple pr, the commit log number should stay in 1. So please use git rebase in your branch.

I am sorry for  you @wu-sheng @ascrutae @coveralls , Maybe I have som wrong operation, I will give a new pr for this! Thank you very much!

@IluckySi You can do rebasing in this branch. Github will update after that. So, don't worry.

Coverage increased (+0.04%) to 17.685% when pulling 8ba1640d443c2ef2c623cee789b484c312c64017 on IluckySi:ilucky-skywalking-spymemcached into 4a874676e595808b3cc31dc8110a898af331d184 on wu-sheng:master.

When I execute git rebase remotes/origin/ilucky-skywalking-spymemcached ilucky-skywalking-spymemcached, I encouter  lots of conflic files! @wu-sheng

That may be caused by your sync from the origin. I don't know whether do you have other choices than send a new one. :) Maybe you can ask someone know Git better than me.

@IluckySi  you can rebase your commits first. and then check new branch from master, and last you can cherry pick the commit to you new branch.


Allow user to override opam command

The user may override the opam command by setting the OPAM environment
variable. The default is equivalent to OPAM=opam.
opam is not required (or even available) in some environment, such as under Nix.

What problem does this solve? You don't need to run make ocaml-deps if you don't want ocaml, and make build doesn't build the OCaml backend by default.

@ehildenb It lets me run make build-ocaml and make test-conformance with the OCaml backend.


[Bug]: 'NoneType' object has no attribute 'client' with Prowler 4.2.1

Steps to Reproduce
Executing prowler with a mutelist results on the error:
[File: mutelist.py:109]         [Module: mutelist]       CRITICAL: AttributeError -- 'NoneType' object has no attribute 'client'[51]
Expected behavior
Works as earlier version
Actual Result with Screenshots or Logs
[File: mutelist.py:109]         [Module: mutelist]       CRITICAL: AttributeError -- 'NoneType' object has no attribute 'client'[51]
How did you install Prowler?
Cloning the repository from github.com (git clone)
Environment Resource
3
OS used

Docker official image

Prowler version
4.2.1
Pip version
One on docker image
Context
It works correctly on version 4.1.0

Hi @christiandavilakoobin I've created the above PR in the branch PRWLR-3837-bug-none-type-object-has-no-attribute-client-with-prowler-4-2-1-4141. Please test it and tell me if this works as expected.
Thanks for using Prowler 🚀

Hi @jfagoagas,
Yes, it's working now.
Thanks!

Hi @jfagoagas,
Yes, it's working now.
Thanks!

That's great! I'll fix some failing tests tomorrow and include some others to prevent this to happen again. Probably we will release a 4.2.2 tomorrow or Monday.
Thanks again 🙌

Hi @christiandavilakoobin could you please test it again with the same branch? I've refactor some parts of the code make it more simpler to work with and also to separate concerns about AWS and other providers.
More tests has been included.

Hi @jfagoagas
It fails 😢 :
Traceback (most recent call last):
  File "/home/prowler/.local/bin/prowler", line 8, in <module>
    sys.exit(prowler())
             ^^^^^^^^^
  File "/home/prowler/.local/lib/python3.12/site-packages/prowler/__main__.py", line 204, in prowler
    global_provider.mutelist = args.mutelist_file
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prowler/.local/lib/python3.12/site-packages/prowler/providers/aws/aws_provider.py", line 322, in mutelist
    mutelist = get_mutelist_file_from_lambda(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: get_mutelist_file_from_lambda() takes from 1 to 2 positional arguments but 3 were given

You were faster than me! I'm just including tests for that part to prevent this from happening again.

@christiandavilakoobin pushed changes and tests, thanks!

@jfagoagas same error. It seems you fixed the s3, but lambda still have 3 parameters instead 2.

It should be fixed with the last commit maybe you didn't push, could you check it?

You're right. It works now! Thanks!