Arctic Ocean diagnostics

Short description of the diagnostic
The main focus of this diagnostics is evaluation of ocean components of climate models in the Arctic Ocean, however most of the diagnostics are implemented in a way that can be easily expanded to other parts of the World Ocean. Most of the diagnostics aim at model comparison to climatological data (PHC3), so we target historical CMIP simulations. However scenario runs also can be analysed to have an impression of how Arcti Ocean hydrography will chnage in the future.
Branch and pull request
Branch: https://github.com/ESMValGroup/ESMValTool/tree/applicate_trr
PR: https://github.com/ESMValGroup/ESMValTool/pull/721

Working on the codacy issues.

@bouweandela Should I address Too many local variables and Too many arguments or just try to reduce the function to 50 lines?

@zklaus Can we, please,  agree on the review decision?

@bouweandela In plotting functions, I use names of the variables like fig, ax, cb, cs. This is basically a standard in python plotting code, and make things clear for those who use matplotlib a lot. The 'codacy' naturally complains about the length of the variables. Do you think I still have to change them?

Should I address Too many local variables and Too many arguments or just try to reduce the function to 50 lines?

Preferably you should address them, but if you have something like 10 or so left it's fine. Reducing the functions to about 50 lines is a great rule of thumb, because that means you can see the entire function without scrolling, which really helps to avoid unseen bugs.

In plotting functions, I use names of the variables like fig, ax, cb, cs. This is basically a standard in python plotting code, and make things clear for those who use matplotlib a lot. The 'codacy' naturally complains about the length of the variables. Do you think I still have to change them?

I think it would be good to stick with the Codacy advice. I think names like figure, axis, colorbar would be nicer for those who are not intimately familiar with matplotlib and just as easy to understand for those who do use matplotlib a lot.

Hi @bouweandela, @zklaus, @mattiarighi
Here is the status. I think I ticked all the boxes:

provenance is added to all generated files.
There are only 11 codacy issues left, among them, two yellow ones seem to be codacy mistakes:

interpolate_esmf has only 18 variables (at least VSCode thinks so), not 31, anyway the function is quite short now.
"Unused argument 'kwargs'" for genfilename seem to be a pylint issue https://github.com/PyCQA/pylint/issues/2214

Most of the rest of the codacy issues related to too many local arguments in plotting functions. I don't see how to make them shorter without doing unreadable (to me at least). Moreover looking at the plotting functions in other diagnostics I can see that long plotting functions seem to be tolerated.
CI tests are all green.

@zklaus Thank you for the comments. I have implemented changes you requested and remove cmorization part, I will do a separate PR with it.
@bouweandela @mattiarighi Can you, please, comment if something else should be done?

I'll test the PR once @bouweandela has approved it, as usual.

I have updated the environment and fx files do not work anymore. Looking at the issue.  Maybe @valeriupredoi can give a quick hint of where to look. I try to get areacello variables like this:
areacello:
        project: CMIP5
        mip: fx
        exp: historical
        ensemble: r0i0p0
        start_year: 2000
        end_year: 2002

and when ESMValTool tries to get areacello from climatology, that has only one time step and defined like this:
additional_datasets:
      - {dataset: PHC,  project: OBS6,  tier: 2, type: clim, version: 3, start_year: 1950,  end_year: 1950}

It seems to try to concatenate two areacello files and naturally fails with:
ValueError: Can not concatenate cubes.

If it helps the names of climatological files are:
OBS6_PHC_clim_3_Omon_so_195001-195012.nc
OBS6_PHC_clim_3_Omon_thetao_195001-195012.nc
OBS6_PHC_clim_3_fx_areacello.nc

And the complete recipe can be found here:
https://github.com/ESMValGroup/ESMValTool/blob/applicate_trr/esmvaltool/recipes/recipe_arctic_ocean.yml

@koldunovn I am running your recipe without the PHC data (I dont have it, nor is it on Jasmin/esmeval) and am not noticing any concatenation errors: INFO    Successfully completed task arctic_ocean/areacello (priority 0) in 0:01:25.616730 - can you pls tell me which files it is trying to concatenate and it's failing? :beer:

Ok, the problem is solved, I had two areacello files in the directory.
But now I have another one :) Related to https://github.com/ESMValGroup/ESMValTool/pull/1459 As suggested by @mattiarighi I changed mip for observations to Oyr. Now I am in the situation when my data (thetao, so) have mip: Omon , areacello data have mip: fx, and my additional dataset (PHC) has mip: Oyr.

If I don't specify mip in the additional dataset, the code searches for OBS6_PHC_clim_3_Omon_thetao[_.]*nc (observations with mip: Omon, while they have mip: Oyr)
If I add mip: Oyr to additional dataset, then the code searches for OBS6_PHC_clim_3_Oyr_areacello[_.]*nc, while areacello file should have mip: fx.

The easiest option would be to revert changes in https://github.com/ESMValGroup/ESMValTool/pull/1459 and keep mip: Omon for climatology data. But maybe there is a better option (ping @valeriupredoi ).
Complete recipe:
https://github.com/ESMValGroup/ESMValTool/blob/applicate_trr/esmvaltool/recipes/recipe_arctic_ocean.yml

@koldunovn I am running your recipe without the PHC data (I dont have it, nor is it on Jasmin/esmeval) and am not noticing any concatenation errors: INFO Successfully completed task arctic_ocean/areacello (priority 0) in 0:01:25.616730 - can you pls tell me which files it is trying to concatenate and it's failing? 🍺

@valeriupredoi thanks for the answer! Sorry, it was a stupid mistake of having two areacello files in one directory (in subdirectory actually).

this is why I tried to convince @mattiarighi and @bouweandela that my proposed priority assignment of mip should be: if mip specified in variable then that should be prioritized over the mip specified in the dataset dictionary! This case here is not solved without such a priority assignment; the only way around it in a nasty way is to make the PHC data Omon

this is why I tried to convince @mattiarighi and @bouweandela that my proposed priority assignment of mip should be: if mip specified in variable then that should be prioritized over the mip specified in the dataset dictionary! This case here is not solved without such a priority assignment; the only way around it in a nasty way is to make the PHC data Omon

For me, it's the easiest solution, so I am happy :) Thanks for looking into this!

okeydokey :grin:

@bouweandela is there a chance that you will have time to look at the PR in the near future? 😳

@koldunovn My apologies for the slow reply. I just reviewed the code.

The easiest option would be to revert changes in #1459 and keep mip: Omon for climatology data. But maybe there is a better option

Yes, there is a better option: you can specify the additional datasets per variable:
    variables:
      areacello:
        project: CMIP5
        mip: fx
        exp: historical
        ensemble: r0i0p0
        additional_datasets:
          - {dataset: PHC, project: OBS6, tier: 2, type: clim, version: 3}
      thetao: &variable
        mip: Omon
        project: CMIP5
        exp: historical
        ensemble: r1i1p1
        start_year: 2002
        end_year: 2005
        additional_datasets:
          - {dataset: PHC, project: OBS6, mip: Oyr, tier: 2, type: clim, version: 3, start_year: 1950, end_year: 1950}
      so: *variable

The thing to keep in mind is that files are defined using a combination of the keys in the variable section and dataset section, where keys from the dataset section take precedence. This is also explained in the documentation: https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/esmvalcore/recipe.html#variable-and-dataset-definitions
I also used a yaml anchor to avoid some duplication and removed the start_year and end_year keys from the fx variable, because it probably doesn't have a start and end year.

@bouweandela Thanks a lot for the review! I will test the changes in the next couple of days.

Hi @bouweandela
I tested the diagnostic with all suggested changes and the latest master and it works for me at DKRZ. Do you think you can give it a go, so @mattiarighi test it?

The change in the YAML file does not work, it gives:
Missing keys {'start_year', 'end_year'} from variable areacello in diagnostic arctic_ocean

I would keep it as is. Other diagnostics also use "fake" time range for  areacello.

The change in the YAML file does not work

Did you update your ESMValCore installation to the latest version?

The change in the YAML file does not work

Did you update your ESMValCore installation to the latest version?

I merged with master and did:
conda env update --name esmvaltool --file environment.yml
pip install .

That should give me the latest released version (available from pip), I guess :)

This is what I get now:
esmvalcore.__version__
'2.0.0b3'

not from pip remote but from local - did you git pull the latest master? Note that conda env update is not needed unless dependencies have changed

not from pip remote but from local - did you git pull the latest master? Note that conda env update is not needed unless dependencies have changed

By master I mean ESMValTool master, sorry. I use ESMValCore from the pip.

That is probably the problem.
Try to install the core in development mode, see instructions.
That would give you the very last version.

That is probably the problem.
Try to install the core in development mode, see instructions.
That would give you the very last version.

But isn't it a bit inconsistent? I thought the idea of having Core and Tools separated is to make it possible to develop diagnostics against something stable, not a moving target? If this diagnostic is changed to work with the latest version, merged and the user will try to use it the next day, it will not work (in this particular case at least), because he/she will get 2.0.0b3 by following instructions in the documentation.
On the other hand, one would like to have all diagnostics working with the latest version of the tool, that's understandable. I guess the way the YAML file is now will work in both cases (2.0.0b3 and the latest).
Let me know how to proceed - I can, of course, install the bleeding edge version of ESMValCore and test against it. Or maybe it is time to bump the version of ESMValCore?

yes, but when we'll have a stable major release then we'll use that one. Also @bouweandela could update the tagged minor releases for Core more often :grin: :imp:

But isn't it a bit inconsistent? I thought the idea of having Core and Tools separated is to make it possible to develop diagnostics against something stable, not a moving target?

But in this case you are (probably) trying to use a feature which has not been released yet.

Let me know how to proceed - I can, of course, install the bleeding edge version of ESMValCore and test against it.

Yes, let's try to see if it solves the problem.

But in this case you are (probably) trying to use a feature which has not been released yet.

It's not like I want to :) I am fine with the current version.

Yes, let's try to see if it solves the problem.

Sure, I will try it now.

It's not like I want to :) I am fine with the current version.

What I mean is that the issue with the missing keys in areacello is related to something that was recently changed in ESMValCore and I can't remember whether it is already in the latest release. ;-)

Well, after installing the latest version of ESMValCore the suggestion from @bouweandela works, and of course it more readable and understandable, when setup is formulated this way.
But again, this version not going to work with current stable :) Let me know if I should commit this version of YAML file, or stick to the previous one :stuck_out_tongue_winking_eye:

Well, the current stable is ESMValTool-1.1.0 since after that there were only pre-releases. Though I admit that the current beta release is impressively stable already.

Let me know if I should commit this version of YAML file, or stick to the previous one

Just go ahead if it works.
There will be anyway a new release pretty soon.

we don't talk about 1.1.0 :hushed:

Well, the current stable is ESMValTool-1.1.0 since after that there were only pre-releases. Though I admit that the current beta release is impressively stable already.

Of course, by "stable", I mean tagged release, or more generally the latest ESMValCore you get from pypi. You right there is no stable version 2 yet :wink:

I added the new version of the yaml file. The code is working and all tests are green (except couple of acceptable codacy issues). @bouweandela @mattiarighi please let me know what actions are further necessary from my side.


Add access restrictions to vending machines

Various vending machines now require an appropriate ID to function.

This Pull Request, its content and its author are all property of PP Station 13, Copyright © woross 2020


Installation check fails on notebook

Reported by @alaity47
python check_env.py fails to find module "notebook".
(navo-env) garibaldi:navo-workshop laity$ python check_env.py
Found jupyter
Error: Failed import: No module named 'notebook'
Found numpy 1.24.3
Found matplotlib 3.7.1
Found jupyterlab 4.0.0
Found astropy 5.2.2
Found pyvo 1.4.1
Found astroquery 0.4.6

There are errors that you must resolve before running the tutorials.

Also, we don't check for the existence of the "scipy" module even though it is installed in the notebook.
Notes:

Newest Intel miniconda threw an "incompatible architecture" error. Second-newest miniconda worked.
Step 5 of the installation instructions took a long time to run some of its pieces, might want to to note it's expected to spin for a while.

Maybe time to switch to mamba? That may resolve step5 being too slow.

Indeed, I switched the astropy-workshop installation instructions to use mamba. Could copy those instructions.


Add TOML support with [BaseTomlService]

Add base toml service to enable fetch of toml files
Add spec file for the new toml service for automated testing
This was added to allow a new way to retrive python version from pyproject.toml as described in issue #9410
Note: this adds a new dependency smol-toml which requires nodejs 18 or newer. i was testing this in nodejs 20 on my machine.
Note: when parsed, toml dates are a new extended date object (TomlDate) due to limitations in native js date, read more here.

Seems to fail due to node16 in github workflow.
I will keep this draft until #9385 is merged

Thanks. I've now merged the node 18 PR so you can update this branch to finish it off.
I've removed [BaseTomlService] from the PR title. The base classes are just covered in the core tests - there are no service tests to run.
I'll have a proper look over this at the weekend


Undefined method when host_validation mode is "syntax"

I recently added this to my code:
EmailAddress::Config.configure(
  host_validation: :syntax
)

When I try to use EmailAddress to validate an address, an error occurs:
 3) Msb::CartsController checkout action failure email address is invalid returns 422
     Failure/Error: return if EmailAddress.valid?(email_address)

     NoMethodError:
       undefined method `mx_ips' for nil:NilClass
     # /Users/strike/.rvm/gems/ruby-2.4.2@memberhub-commerce-api/gems/email_address-0.1.9/lib/email_address/host.rb:417:in `valid_mx?'
     # /Users/strike/.rvm/gems/ruby-2.4.2@memberhub-commerce-api/gems/email_address-0.1.9/lib/email_address/host.rb:393:in `valid?'
     # /Users/strike/.rvm/gems/ruby-2.4.2@memberhub-commerce-api/gems/email_address-0.1.9/lib/email_address/address.rb:233:in `valid?'
     # /Users/strike/.rvm/gems/ruby-2.4.2@memberhub-commerce-api/gems/email_address-0.1.9/lib/email_address.rb:43:in `public_send'
     # /Users/strike/.rvm/gems/ruby-2.4.2@memberhub-commerce-api/gems/email_address-0.1.9/lib/email_address.rb:43:in `block (2 levels) in singleton class'

This is the line that blows up:
if self.exchangers.mx_ips.size > 0

Because this...
def exchangers
  return nil if @config[:host_type] != :email || !self.dns_enabled?
  @_exchangers ||= EmailAddress::Exchanger.cached(self.dns_name)
end

... returns nil.

And it looks as if host_validation is something other than :mx or :a, then DNS is disabled.

Also running into this while trying to disable host_validation in a unit test.

Looks like #28 fixes this. My quick test of this case worked. Please re-open if it didn't


Fix api_docs for to_ordinal

This PR will resolve two issues and add an explanation for to_ordinal utility, recently merged in #17419.
It will,

Resolve a grammatical error in Return
Resolve abnormality in api_docs due to a new line in the docstring. api_docs link
Add a little explanation for to_ordinal

cc: @haifeng-jin

Thanks for the PR!

Glad to contribute :)


Initial commit for speculative execution in MR mode

This is the first PR of https://docs.google.com/document/d/1Vf5A7pHbFKwr7-Bbw2X3zBBVZSvT7HXEPE6bsi9qGco/
Main changes:

It has the new GobblinMultiTaskAttempt, which does not contain any real execution but provides the layout.
Gobblin Task has new commit method, which is empty in this PR.
It has the new interface SpeculativeAttemptAwareConstruct. Some most commonly used gobblin writers, and writer wrappers, have implemented this interface.

@chavdar @ibuenros Can you review?

@chavdar Comments addressed.

Changes Unknown when pulling 75a1a072c3f52429c4857e1572cc685b514681b0 on ydai1124:master into ** on linkedin:master**.

@ibuenros comments addressed.


Implementing some functions for dict

Coverage increased (+2.6%) to 92.297% when pulling 175f2810a5d3947074c6c1a7fee07d845cec2156 on trnsnt:trnsnt/dict into 4190aca5b593ad0ccc4cf0baa7f048a8032b8e82 on Tygs:master.

Coverage increased (+2.6%) to 92.297% when pulling 175f2810a5d3947074c6c1a7fee07d845cec2156 on trnsnt:trnsnt/dict into 4190aca5b593ad0ccc4cf0baa7f048a8032b8e82 on Tygs:master.

Coverage increased (+2.6%) to 92.297% when pulling 175f2810a5d3947074c6c1a7fee07d845cec2156 on trnsnt:trnsnt/dict into 4190aca5b593ad0ccc4cf0baa7f048a8032b8e82 on Tygs:master.

Coverage increased (+2.6%) to 92.297% when pulling 175f2810a5d3947074c6c1a7fee07d845cec2156 on trnsnt:trnsnt/dict into 4190aca5b593ad0ccc4cf0baa7f048a8032b8e82 on Tygs:master.

Coverage increased (+2.8%) to 92.407% when pulling 2d3a9fa35f657cb94904eed6f42ed1eb66c1e589 on trnsnt:trnsnt/dict into 522ee84c770b0669c5487a4051a52407442fa9d1 on Tygs:master.


qline with VIM 9.1.378

Vim version: 9.1.378
I get the following errors:
Error detected while compiling function qline#Statusline[1]..<SNR>74_Statusline:
line    9:
E1048: Item not found in script: Get

Thank you for your plugin.

can be closed ... it might be an issue between qline and vim9-jetpack. I changed back for Plug and everything is working now.

Most likely not qline related

Sorry for late reply.
I've also tried with latest vim and I got no problem.
Thank you for using my plugin!


Virtualservices/Destination rules not working after 1.2.5 upgrade

Bug description
I’ve recently upgraded my istio setup from 1.0.5 to 1.2.5, basically running a fresh setup.
After that my set of VirtualServices / Destination rules seems to be totally ignored.
I’ve a Kong Api gw with istio sidecar acting as a ingress and a couple of nginx labelled “version: green” and “version: blue” I want to access via a given header “nginx” (green is the live one, blue the “dark” alternative)
here is my destinationRule
kind: DestinationRule
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: nginx-service-destination
  namespace: staging
  labels:
    app: nginx-service
    profile: staging
spec:
  host: nginx
  subsets:
    - labels:
        version: blue
      name: blue-nginx
    - labels:
        version: green
      name: green-nginx

and my virtual service
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: nginx-service-vservice
  namespace: staging

  labels:
    app: nginx-service
    profile: staging
spec:
  hosts:
    - nginx
  http:
    - match:
        - headers:
            nginx:
              exact: green
      name: green
      route:
        - destination:
            host: nginx
            port:
              number: 80
            subset: green-nginx
    - match:
        - headers:
            nginx:
              exact: blue
      name: blue
      route:
        - destination:
            host: nginx
            subset: blue-nginx
    - name: default
      route:
        - destination:
            host: nginx
            subset: green-nginx

Such configuration was working perfectly on old 1.0.5 and I could drive my browser from green to blue simply adding http header “nginx: blue”.
I’ve checked 1.2.5 documentation several time but could’n find any significant change and validation webhook and Kiali says everything is fine.
Nevertheless my requests are simply doing a roundRobin between green and blue as they where totally ignored by istio.

Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ X ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Request should be routed to proper pod according to header presence/value
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version)
istio 1.2.5
kiali 1.4.2
kubernetes 1.12.6 (local installation)
How was Istio installed?
helm template + kubectl apply
values.yaml
# Top level istio values file has the following sections.
#
# global: This file is the authoritative and exhaustive source for the global section.
#
# chart sections: Every subdirectory inside the charts/ directory has a top level
#       configuration key in this file. This file overrides the values specified
#       by the charts/${chartname}/values.yaml.
#       Check the chart level values file for exhaustive list of configuration options.

#
# Gateways Configuration, refer to the charts/gateways/values.yaml
# for detailed configuration
#
gateways:
  enabled: false

#
# sidecar-injector webhook configuration, refer to the
# charts/sidecarInjectorWebhook/values.yaml for detailed configuration
#
sidecarInjectorWebhook:
  enabled: true
  enableNamespacesByDefault: false

#
# galley configuration, refer to charts/galley/values.yaml
# for detailed configuration
#
galley:
  enabled: true

#
# mixer configuration
#
# @see charts/mixer/values.yaml, it takes precedence
mixer:
  policy:
    # if policy is enabled the global.disablePolicyChecks has affect.
    enabled: false

  telemetry:
    enabled: true
#
# pilot configuration
#
# @see charts/pilot/values.yaml
pilot:
  enabled: true
  traceSampling: 100.0

#
# security configuration
#
# DO NOT TURN OFF OR YOU WILL GET MISSING SECRETS FOR OTHER COMPONENTS
security:
  enabled: true

#
# nodeagent configuration
#
nodeagent:
  enabled: false

#
# addon grafana configuration
#
grafana:
  enabled: false

#
# addon prometheus configuration
#
prometheus:
  enabled: false

#
# addon jaeger tracing configuration
#
tracing:
  enabled: false
  jaeger:
    tag: 1.13.1

#
# addon kiali tracing configuration
#
kiali:
  enabled: true
  #tag: v0.21
  tag: v1.4.2
  dashboard:
    # You must create a secret with this name - one is not provided out-of-box.
    secretName: kiali
    # Bind the service account to a role with only read access
    viewOnlyMode: false
    # If you have Grafana installed and it is accessible to client browsers, then set this to its external URL.
    # Kiali will redirect users to this URL when Grafana metrics are to be shown.
    grafanaURL:  https://monitor.test.k8.mydomain.int/
    # If you have Jaeger installed and it is accessible to client browsers, then set this property to its external URL.
    # Kiali will redirect users to this URL when Jaeger tracing is to be shown.
    jaegerURL:  https://jaeger.test.k8.mydomain.int/
  prometheusAddr: http://prometheus-server.kube-system:80

#
# addon certmanager configuration
#
certmanager:
  enabled: false

#
# Istio CNI plugin enabled
#   This must be enabled to use the CNI plugin in Istio.  The CNI plugin is installed separately.
#   If true, the privileged initContainer istio-init is not needed to perform the traffic redirect
#   settings for the istio-proxy.
#
istio_cni:
  enabled: false

# addon Istio CoreDNS configuration
#
istiocoredns:
  enabled: false

# Common settings used among istio subcharts.
global:
  # Default hub for Istio images.
  # Releases are published to docker hub under 'istio' project.
  # Daily builds from prow are on gcr.io, and nightly builds from circle on docker.io/istionightly
  hub: docker.io/istio

  # Default tag for Istio images.
  tag: 1.2.5

  # Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>
  # The control plane has different scopes depending on component, but can configure default log level across all components
  # If empty, default scope and level will be used as configured in code
  logging:
    level: "default:warn"

  # monitoring port used by mixer, pilot, galley
  monitoringPort: 15014

  k8sIngress:
    enabled: false
    # Gateway used for k8s Ingress resources. By default it is
    # using 'istio:ingressgateway' that will be installed by setting
    # 'gateways.enabled' and 'gateways.istio-ingressgateway.enabled'
    # flags to true.
    gatewayName: ingressgateway
    # enableHttps will add port 443 on the ingress.
    # It REQUIRES that the certificates are installed  in the
    # expected secrets - enabling this option without certificates
    # will result in LDS rejection and the ingress will not work.
    enableHttps: false

  proxy:
    image: proxyv2

    # cluster domain. Default value is "cluster.local".
    clusterDomain: "cluster.local"

    # Resources for the sidecar.
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 2000m
        memory: 1024Mi

    # Controls number of Proxy worker threads.
    # If set to 0 (default), then start worker thread for each CPU thread/core.
    concurrency: 2

    # Configures the access log for each sidecar.
    # Options:
    #   "" - disables access log
    #   "/dev/stdout" - enables access log
    accessLogFile: ""

    # Configure how and what fields are displayed in sidecar access log. Setting to
    # empty string will result in default log format
    accessLogFormat: ""

    # Configure the access log for sidecar to JSON or TEXT.
    accessLogEncoding: JSON

    # Log level for proxy, applies to gateways and sidecars.  If left empty, "warning" is used.
    # Expected values are: trace|debug|info|warning|error|critical|off
    logLevel: "warning"

    # Per Component log level for proxy, applies to gateways and sidecars. If a component level is
    # not set, then the global "logLevel" will be used. If left empty, "misc:error" is used.
    componentLogLevel: ""

    # Configure the DNS refresh rate for Envoy cluster of type STRICT_DNS
    # This must be given it terms of seconds. For example, 300s is valid but 5m is invalid.
    dnsRefreshRate: 300s

    #If set to true, istio-proxy container will have privileged securityContext
    privileged: false

    # If set, newly injected sidecars will have core dumps enabled.
    enableCoreDump: false

    # Default port for Pilot agent health checks. A value of 0 will disable health checking.
    statusPort: 15020

    # The initial delay for readiness probes in seconds.
    readinessInitialDelaySeconds: 1

    # The period between readiness probes.
    readinessPeriodSeconds: 2

    # The number of successive failed probes before indicating readiness failure.
    readinessFailureThreshold: 30

    # istio egress capture whitelist
    # https://istio.io/docs/tasks/traffic-management/egress.html#calling-external-services-directly
    # example: includeIPRanges: "<IP_ADDRESS>/16,<IP_ADDRESS>/16"
    # would only capture egress traffic on those two IP Ranges, all other outbound traffic would
    # be allowed by the sidecar
    includeIPRanges: "*"
    excludeIPRanges: "{{ansible_default_ipv4.network}}/24"
    excludeOutboundPorts: ""

    # pod internal interfaces
    kubevirtInterfaces: ""

    # istio ingress capture whitelist
    # examples:
    #     Redirect no inbound traffic to Envoy:    --includeInboundPorts=""
    #     Redirect all inbound traffic to Envoy:   --includeInboundPorts="*"
    #     Redirect only selected ports:            --includeInboundPorts="80,8080"
    includeInboundPorts: "*"
    excludeInboundPorts: ""

    # This controls the 'policy' in the sidecar injector.
    autoInject: disabled

    # Sets the destination Statsd in envoy (the value of the "--statsdUdpAddress" proxy argument
    # would be <host>:<port>).
    # Disabled by default.
    # The istio-statsd-prom-bridge is deprecated and should not be used moving forward.
    envoyStatsd:
      # If enabled is set to true, host and port must also be provided. Istio no longer provides a statsd collector.
      enabled: false
      host: # example: statsd-svc.istio-system
      port: # example: 9125

    # Sets the Envoy Metrics Service address, used to push Envoy metrics to an external collector
    # via the Metrics Service gRPC API. This contains detailed stats information emitted directly
    # by Envoy and should not be confused with the the Istio telemetry. The Envoy stats are also
    # available to scrape via the Envoy admin port at either /stats or /stats/prometheus.
    #
    # See https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/metrics/v2/metrics_service.proto
    # for details about Envoy's Metrics Service API.
    #
    # Disabled by default.
    envoyMetricsService:
      enabled: false
      host: # example: metrics-service.istio-system
      port: # example: 15000

    # Specify which tracer to use. One of: lightstep, zipkin, datadog
    tracer: "zipkin"

  proxy_init:
    # Base name for the proxy_init container, used to configure iptables.
    image: proxy_init

  # imagePullPolicy is applied to istio control plane components.
  # local tests require IfNotPresent, to avoid uploading to dockerhub.
  # TODO: Switch to Always as default, and override in the local tests.
  imagePullPolicy: IfNotPresent

  # controlPlaneSecurityEnabled enabled. Will result in delays starting the pods while secrets are
  # propagated, not recommended for tests.
  controlPlaneSecurityEnabled: false

  # disablePolicyChecks disables mixer policy checks.
  # if mixer.policy.enabled==true then disablePolicyChecks has affect.
  # Will set the value with same name in istio config map - pilot needs to be restarted to take effect.
  disablePolicyChecks: true

  # policyCheckFailOpen allows traffic in cases when the mixer policy service cannot be reached.
  # Default is false which means the traffic is denied when the client is unable to connect to Mixer.
  policyCheckFailOpen: false

  # EnableTracing sets the value with same name in istio config map, requires pilot restart to take effect.
  enableTracing: true

  # Configuration for each of the supported tracers
  tracer:
    # Configuration for envoy to send trace data to LightStep.
    # Disabled by default.
    # address: the <host>:<port> of the satellite pool
    # accessToken: required for sending data to the pool
    # secure: specifies whether data should be sent with TLS
    # cacertPath: the path to the file containing the cacert to use when verifying TLS. If secure is true, this is
    #   required. If a value is specified then a secret called "lightstep.cacert" must be created in the destination
    #   namespace with the key matching the base of the provided cacertPath and the value being the cacert itself.
    #
    lightstep:
      address: ""                # example: lightstep-satellite:443
      accessToken: ""            # example: abcdefg1234567
      secure: true               # example: true|false
      cacertPath: ""             # example: /etc/lightstep/cacert.pem
    zipkin:
      # Host:Port for reporting trace data in zipkin format. If not specified, will default to
      # zipkin service (port 9411) in the same namespace as the other istio components.
      address: "{{jaeger_deploy_name}}-collector.{{istio_namespace}}:9411"
    datadog:
      # Host:Port for submitting traces to the Datadog agent.
      address: "$(HOST_IP):8126"

  # Default mtls policy. If true, mtls between services will be enabled by default.
  mtls:
    # Default setting for service-to-service mtls. Can be set explicitly using
    # destination rules or service annotations.
    enabled: false

  # ImagePullSecrets for all ServiceAccount, list of secrets in the same namespace
  # to use for pulling any images in pods that reference this ServiceAccount.
  # For components that don't use ServiceAccounts (i.e. grafana, servicegraph, tracing)
  # ImagePullSecrets will be added to the corresponding Deployment(StatefulSet) objects.
  # Must be set for any cluster configured with private docker registry.
  imagePullSecrets:
    # - private-registry-key

  # Specify pod scheduling arch(amd64, ppc64le, s390x) and weight as follows:
  #   0 - Never scheduled
  #   1 - Least preferred
  #   2 - No preference
  #   3 - Most preferred
  arch:
    amd64: 2
    s390x: 2
    ppc64le: 2

  # Whether to restrict the applications namespace the controller manages;
  # If not set, controller watches all namespaces
  oneNamespace: false

  # Default node selector to be applied to all deployments so that all pods can be
  # constrained to run a particular nodes. Each component can overwrite these default
  # values by adding its node selector block in the relevant section below and setting
  # the desired values.
  defaultNodeSelector:
    devops: test

  # Default node tolerations to be applied to all deployments so that all pods can be
  # scheduled to a particular nodes with matching taints. Each component can overwrite
  # these default values by adding its tolerations block in the relevant section below
  # and setting the desired values.
  # Configure this field in case that all pods of Istio control plane are expected to
  # be scheduled to particular nodes with specified taints.
  defaultTolerations: []

  # Whether to perform server-side validation of configuration.
  configValidation: true

  # Custom DNS config for the pod to resolve names of services in other
  # clusters. Use this to add additional search domains, and other settings.
  # see
  # https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#dns-config
  # This does not apply to gateway pods as they typically need a different
  # set of DNS settings than the normal application pods (e.g., in
  # multicluster scenarios).
  # NOTE: If using templates, follow the pattern in the commented example below.
  #podDNSSearchNamespaces:
  #- global
  #- "[[ valueOrDefault .DeploymentMeta.Namespace \"default\" ]].global"

  # If set to true, the pilot and citadel mtls will be exposed on the
  # ingress gateway
  meshExpansion:
    enabled: false
    # If set to true, the pilot and citadel mtls and the plain text pilot ports
    # will be exposed on an internal gateway
    useILB: false

  multiCluster:
    # Set to true to connect two kubernetes clusters via their respective
    # ingressgateway services when pods in each cluster cannot directly
    # talk to one another. All clusters should be using Istio mTLS and must
    # have a shared root CA for this model to work.
    enabled: false

  # A minimal set of requested resources to applied to all deployments so that
  # Horizontal Pod Autoscaler will be able to function (if set).
  # Each component can overwrite these default values by adding its own resources
  # block in the relevant section below and setting the desired resources values.
  defaultResources:
    requests:
      cpu: 10m
    #   memory: 128Mi
    # limits:
    #   cpu: 100m
    #   memory: 128Mi

  # enable pod distruption budget for the control plane, which is used to
  # ensure Istio control plane components are gradually upgraded or recovered.
  defaultPodDisruptionBudget:
    enabled: true
    # The values aren't mutable due to a current PodDisruptionBudget limitation
    # minAvailable: 1

  # Kubernetes >=v1.11.0 will create two PriorityClass, including system-cluster-critical and
  # system-node-critical, it is better to configure this in order to make sure your Istio pods
  # will not be killed because of low priority class.
  # Refer to https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
  # for more detail.
  priorityClassName: ""

  # Use the Mesh Control Protocol (MCP) for configuring Mixer and
  # Pilot. Requires galley (`--set galley.enabled=true`).
  useMCP: true

  # The trust domain corresponds to the trust root of a system
  # Refer to https://github.com/spiffe/spiffe/blob/master/standards/SPIFFE-ID.md#21-trust-domain
  # Indicate the domain used in SPIFFE identity URL
  # The default depends on the environment.
  #   kubernetes: cluster.local
  #   else:  default dns domain
  trustDomain: ""

  # Set the default behavior of the sidecar for handling outbound traffic from the application:
  # ALLOW_ANY - outbound traffic to unknown destinations will be allowed, in case there are no
  #   services or ServiceEntries for the destination port
  # REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well
  #   as those defined through ServiceEntries
  # ALLOW_ANY is the default in 1.1.  This means each pod will be able to make outbound requests
  # to services outside of the mesh without any ServiceEntry.
  # REGISTRY_ONLY was the default in 1.0.  If this behavior is desired, set the value below to REGISTRY_ONLY.
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY

  # The namespace where globally shared configurations should be present.
  # DestinationRules that apply to the entire mesh (e.g., enabling mTLS),
  # default Sidecar configs, etc. should be added to this namespace.
  # configRootNamespace: istio-config

  # set the default set of namespaces to which services, service entries, virtual services, destination
  # rules should be exported to. Currently only one value can be provided in this list. This value
  # should be one of the following two options:
  # * implies these objects are visible to all namespaces, enabling any sidecar to talk to any other sidecar.
  # . implies these objects are visible to only to sidecars in the same namespace, or if imported as a Sidecar.egress.host
  #defaultConfigVisibilitySettings:
  #- '*'

  sds:
    # SDS enabled. IF set to true, mTLS certificates for the sidecars will be
    # distributed through the SecretDiscoveryService instead of using K8S secrets to mount the certificates.
    enabled: false
    udsPath: ""
    useTrustworthyJwt: false
    useNormalJwt: false

  # Configure the mesh networks to be used by the Split Horizon EDS.
  #
  # The following example defines two networks with different endpoints association methods.
  # For `network1` all endpoints that their IP belongs to the provided CIDR range will be
  # mapped to network1. The gateway for this network example is specified by its public IP
  # address and port.
  # The second network, `network2`, in this example is defined differently with all endpoints
  # retrieved through the specified Multi-Cluster registry being mapped to network2. The
  # gateway is also defined differently with the name of the gateway service on the remote
  # cluster. The public IP for the gateway will be determined from that remote service (only
  # LoadBalancer gateway service type is currently supported, for a NodePort type gateway service,
  # it still need to be configured manually).
  #
  # meshNetworks:
  #   network1:
  #     endpoints:
  #     - fromCidr: "<IP_ADDRESS>/24"
  #     gateways:
  #     - address: <IP_ADDRESS>
  #       port: 80
  #   network2:
  #     endpoints:
  #     - fromRegistry: reg1
  #     gateways:
  #     - registryServiceName: istio-ingressgateway.istio-system.svc.cluster.local
  #       port: 443
  #
  meshNetworks: {}

  # Specifies the global locality load balancing settings.
  # Locality-weighted load balancing allows administrators to control the distribution of traffic to
  # endpoints based on the localities of where the traffic originates and where it will terminate.
  # Please set either failover or distribute configuration but not both.
  #
  # localityLbSetting:
  #   distribute:
  #   - from: "us-central1/*"
  #     to:
  #       "us-central1/*": 80
  #       "us-central2/*": 20
  #
  # localityLbSetting:
  #   failover:
  #   - from: us-east
  #     to: eu-west
  #   - from: us-west
  #     to: us-east
  localityLbSetting: {}

  # Specifies whether helm test is enabled or not.
  # This field is set to false by default, so 'helm template ...'
  # will ignore the helm test yaml files when generating the template
  enableHelmTest: false

Environment where bug was observed (cloud vendor, OS, etc)
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

can anybody provide me any hint?
This is blocking me from upgrading to latest version.

have you tried to use istioctl proxy-config to check if the VS and DR configuration in your gateway or pod?

@aroundthecode  Are you using kong ingress controller?  and is it working as expected in the current istio version ?

in the end this resulted in some configuration/conflict errors which where note reported by Istio validation webook / pilot nor Kiali causing the whole destination rule to be dropped and ignored.
The only point where the error was reported was the policy pod, but if you miss such log all the rest of the mesh is report to be working properly with given configuration when in fact it's not.
I think such errors should be propagated with much more evidence mainly because the "entry check" performed by the validation webook doesn't seem to perform the same checks.
is there any prometheus metrics which can be used to detect such "dropped" configurations?

I suspect your issue is similar to this one https://github.com/istio/istio/issues/19835
Try changing your service port to 8080 and check if that works


testClient does not open websockets for testing

What version of Hono are you using?
4.5.1
What runtime/platform is your app running on?
Bun
What steps can reproduce the bug?
Step 1: Create a route with an upgrade to web socket handler.
const app = factory
  .createApp()
  .get(
    '/ws',
    upgradeWebSocket(async c => {
      return {
        onOpen(evt, ws) {
          console.log('WebSocket opened');
          ws.send('Hello from the server');
        },
      };
    })
  );

Bun.serve({
  fetch: app.fetch,
  port: Bun.env.PORT || 3000,
  websocket
})

Step 2a: Now have two variations of test files. Let's call the first one withHc.test.ts:
const client = hc<typeof app>("http://localhost:3000");
const socket = client.ws.$ws(0);

test("testing websocket", async () => {
  const res = await new Promise<Event>((resolve, reject) => {
    socket.addEventListener(
      'open',
      ev => {
        console.log('WebSocket connection opened');
        resolve(ev);
      },
      { once: true }
    );
  });

  console.log(res);
  expect(res).toBeTruthy();
})

Start the server on port 3000. Run the tests with bun test and you will see the console logs and the test passes 🥳

Step 2b: Let's call the second test file withClient.test.ts:
const client = testClient(app); // <- This is the only thing that has changed!
const socket = client.ws.$ws(0);

test("testing websocket", async () => {
  const res = await new Promise<Event>((resolve, reject) => {
    socket.addEventListener(
      'open',
      ev => {
        console.log('WebSocket connection opened');
        resolve(ev);
      },
      { once: true }
    );
  });

  console.log(res);
  expect(res).toBeTruthy();
})

Don't start the server. Simply run bun test and the code will timeout 😢 . This is because the socket object from testClient never opens a connection.
What is the expected behavior?
Any sockets created using testClient should also open connections on request. In a testing environment, it is important to have this helper work to maintain consistency around the project.
What do you see instead?
testClient does not open a connection so I cannot test websocket server on the route.
Additional information
No response

Hi @cybercoder-naj
In short, you cannot use testClient to test WebSockets in Bun.
To test WebSockets in Bun, you have to run the actual server on like localhost:3000.
With Step 2a, you run the server at 3000 port, but with Step 2b, you don't run a server. The testClient can't execute a server, and it's not created for that purpose.

Is it possible to have a websocket test helper planned for Hono?

This may be related to #3185
@nakasyou Isn't it?

@yusukebe Yes.
But I don't know about better API for that. It needs to discuss.


[GB>SCT] Invalid date format

During some automated tests, we've discovered an issue with new certificates from Scotland.
@andrewkay18981 you've added new certificates, but these seem to have an invalid date format
GB/1.3.0/TEST_SCT.png has a date '2022-01-31T05:00:00+0'
According to the ISO specification (https://en.wikipedia.org/wiki/ISO_8601), you need to have 2 numbers behind the +
<time>Z
--
<time>±hh:mm
<time>±hhmm
<time>±hh

This results in an error thrown in our validation app, causing these certificates to be marked as invalid, even though they may be valid. Can you confirm the production certificates of Scotland don't all contain this invalid format? If so, we'll have to add a workaround on our side and other countries may block these codes accidentally.

Thanks @ikbendewilliam I will follow up on this

I have confirmed that this is the case in production. An update will be released to fix this. In the meantime, do we think it's best that the QR code in question be changed to be a special case? I have an updated version of the QR code with this issue fixed that can be uploaded alongside.

If you expect us to be able to process this type of date, then you should leave it, but if this will be updated and isn't something we should expect to encounter, then I think it would be better to update the current QR-code 🙂

Example QR codes are updated and issuing in production should be patched. Issue can be closed now.
Thanks

Thank you for fixing this 👍


Identifiers validation failed for dataset HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão

Identifier validation failed for the dataset HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão:

Crawler attempt: 438
Publishing organization: Universidade Tecnológica Federal do Paraná - Campus Campo Mourão
Publishing country: BRAZIL
Installation: speciesLink Network / INCT-HVFF IPT
Number of occurrences indexed for this dataset at the time of the issue: 38678
Cause: GBIF ID problems exceed 50% threshold: 100% duplicates; 38786 total records; 38786 absent records
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/4171748/step

New IDs sample:

Old IDs sample:

BRA:UTFPR:HCF:38060
BRA:UTFPR:HCF:38061
BRA:UTFPR:HCF:38062
BRA:UTFPR:HCF:38063
BRA:UTFPR:HCF:38064
BRA:UTFPR:HCF:38065
BRA:UTFPR:HCF:38066
BRA:UTFPR:HCF:38067
BRA:UTFPR:HCF:38068
BRA:UTFPR:HCF:38069

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão : https://doi.org/10.15468/sfdea2.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.

Identifier validation still  failing for the dataset HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão:

Crawler attempt: 439
Publishing organization: Universidade Tecnológica Federal do Paraná - Campus Campo Mourão
Publishing country: BRAZIL
Installation: speciesLink Network / INCT-HVFF IPT
Number of occurrences indexed for this dataset at the time of the issue: 38678
Cause: GBIF ID problems exceed 50% threshold: 100% duplicates; 38786 total records; 38786 absent records
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/4193668/step

New IDs sample:

Old IDs sample:

BRA:UTFPR:HCF:38060
BRA:UTFPR:HCF:38061
BRA:UTFPR:HCF:38062
BRA:UTFPR:HCF:38063
BRA:UTFPR:HCF:38064
BRA:UTFPR:HCF:38065
BRA:UTFPR:HCF:38066
BRA:UTFPR:HCF:38067
BRA:UTFPR:HCF:38068
BRA:UTFPR:HCF:38069

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão : https://doi.org/10.15468/sfdea2.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.

Identifier validation still  failing for the dataset HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão:

Crawler attempt: 440
Publishing organization: Universidade Tecnológica Federal do Paraná - Campus Campo Mourão
Publishing country: BRAZIL
Installation: speciesLink Network / INCT-HVFF IPT
Number of occurrences indexed for this dataset at the time of the issue: 38678
Cause: GBIF ID problems exceed 50% threshold: 100% duplicates; 38786 total records; 38786 absent records
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/4212025/step

New IDs sample:

Old IDs sample:

BRA:UTFPR:HCF:38060
BRA:UTFPR:HCF:38061
BRA:UTFPR:HCF:38062
BRA:UTFPR:HCF:38063
BRA:UTFPR:HCF:38064
BRA:UTFPR:HCF:38065
BRA:UTFPR:HCF:38066
BRA:UTFPR:HCF:38067
BRA:UTFPR:HCF:38068
BRA:UTFPR:HCF:38069

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the HCF - Herbário da Universidade Tecnológica Federal do Paraná Campus Campo Mourão : https://doi.org/10.15468/sfdea2.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.


Proguard will obfuscate class names.

Previous proguard example left class names unchouched.

I merged #161.


[webui enhancement] about the tooltip content for default metric chart

What would you like to be added:
show more infomation on webui default metric chart, like chart, add trial id message. And on overview page, the default metric chart show less message when hover to the point, It's not friendly.

Why is this needed: 信息一致，更加友好
Without this feature, how does current nni work：
Components that may involve changes:
Brief description of your proposal if any:

closed as it has been merged.


feat: add issue templates

Adds structured templates to submit either a feedback request/bug fix, new issues are now automatically tagged on creation.
TODO: Make sure people do not use issues for security incident reports.

TODO: Make sure people do not use issues for security incident reports.

Yeah we should point people to the right place for security issues, good call. I'll figure out the link (might have to make one) & ping you after, can be included in the template.


installdeps: Add libzstd-devel

Signed-off-by: John Eckersberg<EMAIL_ADDRESS>

Actually it looks like this is already included in the fcos container, so not needed.


Direct link between atlases and tools/services that use them?

This is just an idea, but it might be worthwhile thinking about it. Atlases are obviously research outputs themselves, but (one of) their main purpose is to be used somehow, e.g. as a visual reference or within tools/services.
So, wouldn't it make sense to capture this somehow?
Example: QuickNII
It's a tool used to register data to a reference atlas. Right now it supports the Allen mouse brain atlas and the Waxholm rat brain atlas (and many more to come).
QuickNII is/will be registered as a Software(Version) and both atlases will be/are registered as BrainAtlas(Versions), but there is no connection between the two...

@bweyers and @jagru20 any idea how this linkage could be established in a Software? (it is a rather specific case, but maybe you know about similar cases for e.g., software that only supports certain models?)

Isn't it the case that atlases are (from a software point of view) just a specific type of data to be read and written on? Thus, a linkage similar to datasets could be thought...

@bweyers generally speaking atlases can match a single dataset, but could also be composed of multiple datasets. Datasets and software are currently linked through contentTypes. I think contentTypes are not sufficient to support a particular atlas or model within a software, because their original contentType is often irrelevant in these cases. They are neither input nor output format, but integrated (typically hard-coded) within the software directly, no? For example, QuickNII supports two atlases and allows datasets with certain contentTypes to be registered to these software-integrated atlases. The contentTypes of the atlases do not play a role in this context, and other atlases with the same contentTypes are not supported by the software. I think the atlases are more to be seen as implementation aspect of the software itself. What do you think?

@bweyers and @jagru20 did you have the time to think about this a bit?

@bweyers and @jagru20 Could you comment on this issue? I could imagine the following:
Adding an additional optional property to SoftwareVersion schema "supported construct" (feel free suggest a different name) which can get as input (type array (0-N)) a BrainAtlasVersion, a ModelVersion, a MetaDataModelVersion, a CommonCoordinateSpace (for now).

Hey Lyuba, please apologize the delay from our side. Introducing a property like the one you proposed sounds like a good idea for this purpose. If it is an Array I would rather suggest naming it "supportedConstructs" but that is nitpicking.


This and next few methods should not be public?

https://github.com/xhsun/gw2wrapper/blob/11b95c973cbdebd9cff8be11ee6ee1ca5066656e/src/main/java/me/xhsun/guildwars2wrapper/AsynchronousRequest.java#L2610

Oh my... They should be public instead of private. I'll fix it now!


refactor: use matches!

Thank you @toku-sa-n , but this is obsolet because of some code refactorings and improvements in #2


Fix button adaptive width for long CTAs and translations

^ UI issue observed in languages with lengthy words.
@ironng lmk if there's anything i might need to provide in terms of design/rules. is this even possible to adjust for?

We can adjust!  I'm guessing that we want to keep both buttons the same width? It's a little harder to keep them the same but also flexible to text length. The best bet here might be to put in a larger minimum button width for ru maybe, but will try to figure out something more generic.

Is there a minimum/maximum width we should use here? I think this is really a “how far can we afford to go” thing.
We can also use padding in conjunction with overflow: hidden/text-overflow: ellipsis/white-space: nowrap, so if the text still overflows we get proper elision and a tooltip with the full content.

See #232, #250 for specific examples here.


Fix error with morphing toggled relation manager tabs

This PR fixes the bug described in #8788
Defining a key to the element will help livewire/alpine to morph the dom as expected.
After hiding a currently active manager, the activeRelationManager is set to the first item.

[x] Changes have been thoroughly tested to not break existing functionality.

Tested with all possible relationmanager configuration what i could think of (with groups,  with hasCombinedRelationManagerTabsWithContent, with associative array keys,with more than 2 relationmanagers .. )

[x] New functionality has been documented or existing documentation has been updated to reflect changes.

some inline comment describing the activemanager reset case

Thanks. I feel like it's kinda weird to be setting public properties in Livewire from the view though. Can we use a hook or something, maybe rendering(), to make sure that $activeRelationManager is valid?
We have a HasRelationManagers trait, so you could add a renderingHasRelationManagers() method to it that should be called at the correct time.
Since this will require a rewrite, I will close this PR and you can reopen it when the changes have been made. Thanks!

Actually gonna reopen this so I can merge the wire:key, but a separate PR can be used for the  renderingHasRelationManagers() thing


[cleanup][broker] Delete duplicate method definitions

Motivation
NonPersistentDispatcher's superclass Dispatcher already declares exactly the same methods, so definitions of those methods are completely unnecessary.
https://github.com/apache/pulsar/blob/b6351f73481fc78d8ba13f7355194c9c51dc59c4/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcher.java#L34-L50
Modifications
Delete these methods
Need to update docs?

[ ] doc-required
[x] no-need-doc
[ ] doc
[ ] doc-added

/pulsarbot run-failure-checks


feat(pingcap/tidb): tidb split release-6.5 pipelines

split release-6.5 pipelines from master

/hold

/unhold


WASM Local-blob overwrite

This is the first PR to add overrides for on-chain WASM with WASM stored locally on a users machine.
rel #7035
This PR introduces two new CLI arguments,

wasm-overwrite a flag for enabling WASM overwrites
wasm-overwrite-path path where the WASM blobs are stored, defaults to $BASE_PATH/$chain/wasm_runtime_overwrites

Most of the logic is contained within wasm_overwrite.rs in client/service/src/client/. The LocalCallExecutor uses this module to try and replace on-chain WASM with local WASM if wasm overwrites are enabled.

There are quite some spaces, trailing whitespaces and lack of newline file termination. Is there any chance you could configure your editor to fix those on save?

I've since installed an editor-config plugin, which was missing in my setup :sweat_smile: . Sorry for all the whitespace, I didn't even realize I had that many

Would be really nice to have some tests.
There exists the test-runtime which you can use to get a wasm file.

Added two tests:
One for runtime_version in WasmOverwrite and one for the CheckOverwrite method in LocalCallExecutor. I am not entirely sure on the strategy for testing methods that scrape for files locally, so I skipped over the scrape_overwrites method. Is there an established way to do that in substrate?

Added two tests, should_check_for_duplicates and should_scrape_wasm using tempdir and the wasm_binary_unwrap()

@insipx please merge master to fix the companion check.

@insipx is C1-low the appropriate label here?

@insipx is C1-low the appropriate label here?

Yes, this does not elevate any release.

@insipx Could you add some usage notes to the PR description?

@insipx Could you add some usage notes to the PR description?

done


[gha] docker build in CI

Create some CI that does the docker build of the backend service. After this lands, you can trigger a build via workflow_dispatch. This is to avoid building on PRs automatically
Images available here: https://console.cloud.google.com/artifacts/docker/aptos-registry/us/docker/graffio-backend?project=aptos-registry
Test plan:

canary here, by triggering on push to this PR branch: https://github.com/aptos-labs/graffio/actions/runs/6398012521

NOTE:

We're using GH hosted runners for now. We can change to self hosted to speed up the build time
This PR copies over some GHA composite actions from other repos for boilerplate. In the future though, we will likely have a central repo for all shared composite actions to reduce copy pasta. now using https://github.com/aptos-labs/actions/tree/main

I want this stuff now so I'll just land, feel free to push any changes directly to main in another commit.


groupId for camel-servlet-starter was incorrect

The camel-servlet-starter artifact is present in groupId org.apache.camel and in org.apache.camel.springboot. However in org.apache.camel.springboot only release 3.0.0 is present. In org.apache.camel all versions including 3.0.0 are present. For the documentation to provide a consistent dependency for all releases, groupId org.apache.camel seems a better fit. Another solution would be to mention two distinct dependencies in the documentation, depending on the version; maybe for the future versions you want to make org.apache.camel.springboot the groupId and keep org.apache.camel for the old ones.

The documentation is up to date with the current master so it must be as is. This PR is invalid. Thanks anyway.

This is mentioned in migration guide too

In top right corner on website you can switch to 2.x version. By default is loaded 3.0.0 (latest) for which is groupId org.apache.camel.springboot correct.

You're correct, I did not check the migration guide and also did not check the documentation for 2.x and 3.x and look at the difference there. My apologies for this pull request and thank you for explaining!


Refactor UDF management process to support table model user-defined function

Description

Refactor UDF management process. Most important change is to introduce an enum type Model. UDFTable use Pair<Model, String> as hash key to support table model user-defined function now.
Introduce pre create and pre drop process to avoid some corner cases  due to partial create/drop. Function with UNAVAILABLE state in SHOW FUNCTIONS will perform some unexpected behaviors.

See https://timechor.feishu.cn/docx/TCyYdm8pYowe0HxEGb9c2UfenCf for more details.

What a spectacular work 🦾! Fabulous design and implementation⚡⚡🔥


breaking change in a minor<EMAIL_ADDRESS>put browser-sync to optional deps

Command
other
Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
17.0.0
Description
(I'm creating this new ticket, because the discussion in relevant ticket and PR is already locked and limited contributors only)
After simple npm-bumping the version of @angular-devkit/build-angular from v17 to v17.1, and running the command npm run dev:ssr I got an error "browser-sync" is not installed".
Note: I already migrated my app from ng16 to ng17 some time ago - so I'm still using the old builder @angular-devkit/build-angular:ssr-dev-server.
I've discovered only now there exists a 17.1 migration schematics for installing browser-sync. But when I'm bumping an Angular minor I'm surprised I'm expected by the framework authors to run migrations scripts - because minors shouldn't contain breaking changes. 🤔
I've seen the comment: https://github.com/angular/angular-cli/issues/26349#issuecomment-1809580491

Latest localtunnel (2.0.2) is still using a vulnerable axios (0.21.4). There's already an issue filed to localtunnel to update the dependency: https://github.com/localtunnel/localtunnel/issues/632. > Until that's fixed, there's not much we can do about this.
browser-sync is only used for live reloading in ng serve and is used in local development only, so the risk here is fairly > minimal. Rennovate should automatically pick up a version bump when it is available.

And now I know I need to install browser-sync manually.
Instead of moving browser-sync to optional deps in v17.1, did you consider the following?:

in v17.1 only bump version of the transitive dependency of browser-sync (for security reasons), but not move it to optional deps
in the nearest major (v18) move browser-sync to optional deps + provide migrations for it

Minimal Reproduction

migrate an Angular SSR app from ng16 to ng17.0.0 (not 17.1.0)
npm-install @angular-devkit/build-angular in version 17.1.0
run npm run dev:ssr

and get an error that "browser-sync" is not installed".

No response

Hello @Platonn, in this particular scenario, the decision was made considering various factors. One significant factor was the lack of progress in resolving the security vulnerability at the time. This isn't the first instance of such an issue with browser-sync, given its reliance on a substantial number of older dependencies. Additionally, browser-sync is utilized in one of the less popular builders, @angular-devkit/build-angular:ssr-dev-server. Despite the inconvenience of resolving errors when browser-sync isn't installed, the process is relatively straightforward. Thus, for now, we prefer to maintain the current setup.
Furthermore, it's worth noting that while migrations are typically reserved for major versions, there have been instances where migrations were included in minor versions. Migrations serve broader purposes beyond solely addressing breaking changes.


Make sure the capstan wheels don't destroy the fiber

The wheels either crush and destroy the fiber or it slips. We will have the find a fix if we ever wnat to automate the process

We can wait until the spool is automated to work on this issue

Closing this issue and combining with #31.


CLI quickstart: quickstart -h should show default addresses and ports

Minor issue. Defaults are shown for some options but not all

closed with #1387


MS OAuth - making "/me" API call configurable

Reason for change
When working with Microsoft oauth, you can either pass in MSGraph scopes like: "User.Read" or you can pass in scopes that are needed for your own resources like: "api://resources/Api.Write". However, you cannot pass in both scopes as requesting a singular token for multiple audiences is not possible (noted here and many other places). If you only pass in the scope for your custom resource, you then you get a 401 trying to call the "/me" endpoint (because the user.read scope is missing).
Changes

"useUser" flag added to config
"useUser" default to false (I figured a limited user object is better than error)
if/else block that either fetches from "/me" or decodes the JWT using jwt-decode

Other considerations

I used 3.1.2 of jwt-decode because I was getting some errors with the 4.0 version that I couldn't figure out
It's not possible to type the jwtDecode object as it could be two different JWT formats. I didn't what to hardcode this to the "1.0" version as to be considerate of future changes to this library and the access token used.
I used "displayName" and "mail" as the keys in the user object to match what those keys are in the "/me" response for consistency and to not break any downstream parsing in users app code.

Testing
Within my personal project, I used the module locally and tested with both true/false flags on the useUser configuration key. Everything functioned as I expected it to.

Sorry for the late review, happy to fix the conflicts?
I would also like another review from @amandesai01

Reason for change
When working with Microsoft oauth, you can either pass in MSGraph scopes like: "User.Read" or you can pass in scopes that are needed for your own resources like: "api://resources/Api.Write". However, you cannot pass in both scopes as requesting a singular token for multiple audiences is not possible (noted here and many other places). If you only pass in the scope for your custom resource, you then you get a 401 trying to call the "/me" endpoint (because the user.read scope is missing).
Changes

"useUser" flag added to config
"useUser" default to false (I figured a limited user object is better than error)
if/else block that either fetches from "/me" or decodes the JWT using jwt-decode

Other considerations

I used 3.1.2 of jwt-decode because I was getting some errors with the 4.0 version that I couldn't figure out
It's not possible to type the jwtDecode object as it could be two different JWT formats. I didn't what to hardcode this to the "1.0" version as to be considerate of future changes to this library and the access token used.
I used "displayName" and "mail" as the keys in the user object to match what those keys are in the "/me" response for consistency and to not break any downstream parsing in users app code.

Testing
Within my personal project, I used the module locally and tested with both true/false flags on the useUser configuration key. Everything functioned as I expected it to.

If you request a access_token with a custom scope from Entra ID, it is not possible to query the user info endpoint independent of the scope you provide, as the classic /me endpoint is part of the graph api and you don't have a graph api token anymore.
If you request a access_token for the graph api, you will get an access token for graph api, which is not even a JWT token and thereby not parsable. This is not a token you should ever use in your apis, just in your application for the sake of requesting user information. If you need information about the user, you should adjust the responseType request param to code id_token or id_token. If you have id tokens enabled in Entra ID, you can then add additional user claims to the id_token in the token configuration.
If you need a token to authenticate against your api, you would request a token for your api, which is represented by a separate application registration in Entra ID. The id of that api app should be announced by adding the resource field to the code request and is represented by the audience field in the response access token. This token must be validated by your api and first checked if the apis id is in the audience claim and then the signature has to be verified.
If one of these fails, the token needs to be rejected.


[error] uv self update does not work from v0.5.12 to v0.5.13

$ uv self update
error: error decoding response body
  Caused by: there are extra bytes after body has been decompressed

I am on version: uv 0.5.12 (351d602d8 2024-12-26)
Windows 11 23H2
Update from v0.5.11 to v0.5.12 went fine.

Having this same issue on macOS.  I am currently using uv 0.5.12 (351d602d8 2024-12-26).
uv self update
info: Checking for updates...
error: error decoding response body
  Caused by: there are extra bytes after body has been decompressed

Same, but with uv pip install:
Using Python 3.11.0rc1 environment at: /databricks/python3
error: Failed to fetch: `https://<private_domain>/pypi/<repo_name>/simple/<package_name>/`
  Caused by: error decoding response body
  Caused by: there are extra bytes after body has been decompressed

Downgrading to 0.5.11 has resolved the issue

👍 This was a known issue in v0.5.12 due to a reqwests update. Sorry about that. I suggest re-installing with cURL (e.g., curl -LsSf https://astral.sh/uv/install.sh | sh). Subsequent self-updates should work without issue.

@charliermarsh Given that version 0.5.12 is unable to network, would you like to proceed with yanking it from PyPI?

👍 This was a known issue in v0.5.12 due to a reqwests update. Sorry about that. I suggest re-installing with cURL (e.g., curl -LsSf https://astral.sh/uv/install.sh | sh). Subsequent self-updates should work without issue.

That solved the issue quickly! Thank you for being swift and engaged.


Branch-switcher

Added branch-switcher in rasqberry-config to switch between available branches

Thank you for your contribution to the RasQberry project, Bengt !


WIP: Remove server input

Closes #252.
This PR removes all sending of player input and window sizes to the server, and uses messages to send it from client WASM instead. This has some up- and downsides, but I think it works out in the long run (especially once we have dependencies for implementing common operations).

[x] Port tictactoe
[x] Port minigolf
[x] Port pong

CI not responding, closing and reopening

Need to provide

function to get user id from source if available
faster user id to entityid lookup


Various fixes & improvements

Summary
This PR includes fixes and improvements for the following:

Follow user preferences to show or hide join/leave messages
Fix the android issue when the app first loads and the post text input was off screen
Use the device locale as the default locale instead of the en
When tapping on a channel or a team in the drawer now we have visual feedback
When switching channels the center channel title updates before the loading animation ends

Ticket Link
https://mattermost.atlassian.net/browse/RN-166
https://mattermost.atlassian.net/browse/RN-158
https://mattermost.atlassian.net/browse/RN-181
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] All new/modified APIs include changes to mattermost-redux
[x] Has UI changes

Device Information
This PR was tested on:

IOS 10.3.1 iPhone 6s
Android 6.0.1  Samsung J5

Also, you've got a conflict

right, I'll avoid them later on.
Fixing the conflicts and the test failures


__resolveType should be optional in interface resolvers

Describe the bug

Currently, Resolvers interface for GraphQL Interface types requires the __resolveType resolver to be defined. This is an issue in any project that split resolvers in multiple files: __resolveType doesn't need to be implemented each time a resolvers object is declared for the GraphQL Interface.
To Reproduce
Steps to reproduce the behavior:

My GraphQL schema:

interface Animal {
  id: ID!
  computedField: String
}

type Cat implements Animal {
  id: ID!
  computedField: String
}

type Dog implements Animal {
  id: ID!
  computedField: String
}

Resolvers:

File A:
export const resolvers: Resolvers = {
  Animal: {
    __resolveType: (item) => item.type,
  },
}

File B:
export const resolvers: Resolvers = {
  // Error thrown here: Property '__resolveType' is missing
  Animal: {
    computedField: (item) => item.foo + 'bar',
  },
}

Indeed, in the generated types, __resolveType is always required in the resolvers object:
export type AnimalResolvers = {
  __resolveType: TypeResolveFn
}

Expected behavior

No error in File B.
The generated type should look like:
export type AnimalResolvers = {
  __resolveType?: TypeResolveFn
}

Environment:

OS: Linux
"@graphql-codegen/cli": "^1.9.1",
"@graphql-codegen/typescript": "^1.9.1",
"@graphql-codegen/typescript-operations": "^1.9.1",
"@graphql-codegen/typescript-resolvers": "^1.9.1",
"@graphql-codegen/fragment-matcher": "^1.9.1",
"@graphql-codegen/introspection": "^1.9.1",
NodeJS: 12.13.0

Additional context

@Akryum I added this in this PR: https://github.com/dotansimha/graphql-code-generator/pull/3123
Can you please try the alpha version? 1.9.2-alpha-976a8ac6.41
You should be able to set now optionalResolveType: true in your config, and it will generate __resolveType as optional field for both union and interface resolvers.

Fixed in v1.10.0.


Do culling per-scene, not per-eye.

Currently, frustum culling is done in renderCamera() in renderer.cpp.  renderCamera() is called per-eye.  It would be more ideal to do that culling once per frame, rather than twice.  That culling should be perhaps done against a combined left/right eye frustum.

I'm working on this now.
The first step here will be to create a center camera for the camera rig.  That is done as part of https://github.com/Samsung/GearVRf/pull/202
Second step will be to move the culling to a method that happens before calling of renderCamera for either left or right eyes.
Third step will be to make the bounding volumes hierarchical so that whole subgraphs can be eliminated if their parent's bounding volume is not in the frustum.
Forth actually traverse the tree as a tree (rather than getWholeSceneObjects which returns the whole tree as a linked list) and do testing for frustum, visibility, and render mask.

this was done back in 2.3 or so.


[PLAT-749] Allow embedding Quick Files in a wiki

Purpose
Right now quickfiles links are not embedding properly in the wiki markdown, creating an "OK" message. This fix makes it so the whole thing is rendered.
Changes

makes file embed link more explicit.

QA Notes
Grab a guid for a quickfiles node and put it in a @[osf](<guid>) tag in the wiki. If it renders, it's all good.
Documentation
🐞 fix, no docs.
Side Effects
None that I know of.
Ticket
<https://openscience.atlassian.net/browse/PLAT-749

Headed to staging 3 🚢

@Johnetordoff, doing some cleanup of our board, do you mind getting this up-to-date so we can stick it in the next plat release?

thank you!


[ENH] Use Sphinx-Gallery API Usage Map in Documentation

Related to https://github.com/mne-tools/mne-python/pull/10999.
It would be nice to link the output in doc/sg_api_usage.html somewhere in the documentation, maybe near the roadmap. This might help new contributors find where the software is not heavily documented as far as example usage and fix that. Or even reduce build time and find redundant examples or how to refactor examples to show things more logically.

To me this is a "developer option" and shouldn't be exposed / linked to in the docs. The right place for these in general I think is a contributor's page or something. I'd rather make a recommendation like this than add any option (or by default with no option) add these to the docs automatically


Change Select Deployment Measure to Select Measure

Background
Due to change in tool naming we need to change references from Deployment measures to measures.
Task

On cover page change variety of deployment approaches to multiple workforce measures
On Main tool tab change Select Deployment Measure to Select Measure
On school to school tab change Select Deployment Measure to Select Measure

Closed via pull request


Faucet button within the Tokel coin wallet

I suggest we add an easy to find faucet option into the wallet (noting this will have to be done over nSPV).
It could be presented as a coloured button that the user clicks and it actions the faucet. Once the faucet has been activated, we could add a timer onto the button to let the user know when they can use the faucet again (potentially changing the colour of the button during this time).
We can fund the faucet and limit it to a couple of transactions a day per address. Otherwise people can go through the normal process of buying some Tokel and holding it in their wallet.

Just to add to the above, the faucet will allow users to make txs within the TOKEL network without ending up in the situation when they want to make a transaction but have to purchase TOKEL to do it and then they just get annoyed.
So basically we will allow users to make 1-2 txs per day for free. And then if they wanna make more they will have to buy TOKEL but if its just a one off case they don't need to worry.

We are no longer using faucetCC as it wasn't mechanised the way we wanted and would take CC developer time that we currently do not have.


NTR: cuproptosis

Please provide as much information as you can:

Suggested term label:
cuproptosis

Definition (free text)
Programmed cell death resulting from mitochondrial proteotoxic stress mediated by copper and FDX1.

Reference
PMID:35354936

Gene product name and ID to be annotated to this term
FDX1 (P10109)

Parent term(s)
GO:0012501

Children terms (if applicable) -

Synonyms (please specify, EXACT, BROAD, NARROW or RELATED) -

Cross-references -

Any other information -

+[Term]
+id: GO:0160119
+name: cuproptosis
+namespace: biological_process
+def: "A programmed cell death process that is induced by intracellular copper accumulation." [PMID:35298263, PMID:35354936]
+is_a: GO:0012501 ! programmed cell death
+property_value: term_tracker_item "https://github.com/geneontology/go-ontology/issues/25942" xsd:anyURI
+created_by: rynl
+creation_date: 2023-10-27T21:19:19Z


Migrate to using the security context v3 for the signature suite and key definitions

Once https://github.com/w3c-ccg/security-vocab/pull/53 is merged this can be addressed

With this merged we can now proceed with updating this library to use the unstable version of the v3 vocab

WIP branch to work off: https://github.com/mattrglobal/jsonld-signatures-bbs/compare/tl/update-to-security-v3

Outstanding work on this is to update the samples in the repo including what is in the root readme to show the new context element.

Outstanding work on this is to update the samples in the repo including what is in the root readme to show the new context element.


Antigen are regenerating the .zcompdump

My antigen are regenerating the .zcompdump on every tab I open in my terminal and it is too slow.

I just update to v1.2.1 and removed my .zcompdump, and it works.

:+1: Cool.


Hide EnhancedStreamChat?

Hey there. I'm the same person who talked to you on Discord recently about having a visibleObjects setting for hiding (visible) cameras from other cameras (that would still be great, though not super important since you can just hide the cameras entirely).
What you can't do anything about, though, is EnhancedStreamChat, when that mod is present. You have an avatar setting in visibleObjects. Would it likewise be possible to integrate EnhancedStreamChat, or just to make it be covered by the UI setting? Thanks!

If ESC is not hidden by turning off UI that is something that needs to be handled by ESC as it would then essentially incorrectly describing "what it is" / not saying that its something to do with UI, so I would suggest you ask them to make sure the Chat is on the UI layer ;)
This is beyond the things that Cam2 should do

Reopening to have a reminder for the other cam visibility option


400 less fps when drawing circles/arcs in 3.3

Godot version:
Godot_v3.3-rc8_win64
i also tested 3.2.4 rc1 (both win32 and win64) and same issue
(no issues in 3.2.3, 3.2.2 or 3.2.1)
OS/device including version:
windows 10
gtx 1080
gles2 and gles3
Issue description:
drawing circles/arcs lowers fps a lot more than it does in 3.2.3
using the repro i get 104 fps in 3.3 and 519 in 3.2.3
i tried drawing rects and lines and it worked fine not sure about anything else
also my projects that dont draw circles are about the same fps and in one case i even got a 1500 fps increase using 3.3
Steps to reproduce:
func _draw():
	for i in 200:
		draw_circle(Vector2(0, 0), 4.0, Color(1,0,0))
		draw_arc(Vector2(0,0), 4.0, 0, 7, 32, Color(0,0,0), 1.0, false)

Minimal reproduction project:
repro.zip
a solution:
out of curiosity i tried the (outdated) circle arc methods from the docs which gives me 1140 fps in 3.3 and 186 in 3.2.3
the only problem is that the arc doesnt have aa (although irrelevant to this issue)
heres that if you want to compare or something
func draw_circle_arc(center, radius, angle_from, angle_to, color):
	var nb_points = 8
	var points_arc = PoolVector2Array()

	for i in range(nb_points + 1):
		var angle_point = deg2rad(angle_from + i * (angle_to-angle_from) / nb_points - 90)
		points_arc.push_back(center + Vector2(cos(angle_point), sin(angle_point)) * radius)

	for index_point in range(nb_points):
		draw_line(points_arc[index_point], points_arc[index_point + 1], color)

func draw_circle_arc_poly(center, radius, angle_from, angle_to, color):
	var nb_points = 32
	var points_arc = PoolVector2Array()
	points_arc.push_back(center)
	var colors = PoolColorArray([color])

	for i in range(nb_points + 1):
		var angle_point = deg2rad(angle_from + i * (angle_to - angle_from) / nb_points - 90)
		points_arc.push_back(center + Vector2(cos(angle_point), sin(angle_point)) * radius)
	draw_polygon(points_arc, colors)

func _draw():
	for i in 200:
		draw_circle_arc_poly(Vector2(0,0), 4.0, 0, 360, Color(1,0,0))
		draw_circle_arc(Vector2(0,0), 4.0, 0, 360, Color(0,0,0))

This seems to be device/driver-specific as I can't reproduce the issue on Linux with Intel HD Graphics 630 and AMD Radeon RX Vega M. Details below.
The main change between 3.2.3 and 3.3 which might affect this is the extension of 2D batching to GLES3 (in 3.2.3 batching was only for GLES2).
Can you see if you get different results if you turn off batching in the project settings? (Rendering > Batching). There are also various options you can play with to see if they make a difference.

Test results
Note that to avoid having a performance hit from running the editor itself, I'm running the project directly from the command line, not from the editor.
$ inxi -CSG
System:    Host: cauldron Kernel: 5.11.13-desktop-2.mga9 x86_64 bits: 64 Desktop: KDE Plasma 5.21.3 Distro: Mageia 9 mga9
CPU:       Info: Quad Core model: Intel Core i7-8705G bits: 64 type: MT MCP cache: L2: 8 MiB
           Speed: 3590 MHz min/max: 800/4100 MHz Core speeds (MHz): 1: 3590 2: 3863 3: 3775 4: 3840 5: 3837 6: 3894 7: 3880
           8: 3860
Graphics:  Device-1: Intel HD Graphics 630 driver: i915 v: kernel
           Device-2: Advanced Micro Devices [AMD/ATI] Polaris 22 XL [Radeon RX Vega M GL] driver: amdgpu v: kernel
           Device-3: Cheng Uei Precision Industry (Foxlink) HP Wide Vision FHD Camera type: USB driver: uvcvideo
           Display: x11 server: Mageia X.org 1.20.10 driver: loaded: intel,v4l resolution: 1920x1080~60Hz
           OpenGL: renderer: Mesa Intel HD Graphics 630 (KBL GT2) v: 4.6 Mesa 21.0.2

I seem to get more or less the same metrics with both GLES2 and GLES3 (maybe +20 FPS for AMD with GLES2 on both 3.2.3 and 3.3 RC 8 compared to GLES3).

GPU
3.2.3 stable
3.3 RC 8

Intel
210 FPS
380 FPS

AMD
400 FPS
530 FPS

Testing on Windows 10 with my laptop with the AMD Radeon RX Vega 56, I seem to be able to reproduce a performance drop in 3.3 RC 8 compared to 3.2.3 stable, though both give me surprisingly bad performance metrics on this repo, compared to Linux drivers on the same hardware.

GPU
3.2.3 stable
3.3 RC 8

AMD
188 FPS
80 FPS

I'm not so familiar with debugging GPU drivers on Windows, but I can see in the Task Manager's Performance tab that the GPU is 100% busy on 3.2.3 stable, and only 66% busy on 3.3 RC 8. In both releases the CPU is only 5% busy.
Disabling batching doesn't seem to make a difference.
Here's some performance metrics from Radeon Software:

I can confirm similar results on another AMD GPU, 315 fps vs 170 fps on average. Interestingly enough, the number of draw calls is the same for both versions, and there is no output in the console from batching doing anything. So my assumption is just general degradation for some reason.
Frame times tell a different story. In 3.2.3 lows are lower, but highs tend to be higher. It gets from 2.9-3 ms to 7 ms on average and up to 10+ ms sometimes. In 3.3 the averages are more stable, it flactuates from 5.9 ms to 7+ ms, with no outliers in either direction.

Radeon 5700 XT, driver version 21.3.1.

Note that neither circles or arcs (polyline) are batched, so there shouldn't be a huge amount of differences codewise. It's possible that an extra state being set (possibly to fix a bug) may have introduced slower performance on some hardware. Or if one version is using line smoothing and one not.
I'll try and investigate, I just have a couple more crucial bugs to fix first (or clayjohn or anyone interested, it's not a massive amount of code to compare).

If anyone getting this drop can try out 3.2.4 beta 2, and try these settings and see if there is any effect, that will be useful:
https://github.com/godotengine/godot/issues/43038#issuecomment-722336615

If anyone getting this drop can try out 3.2.4 beta 2, and try these settings and see if there is any effect, that will be useful:
#43038 (comment)

Tested. Only turning rendering->options->api_usage_legacy->flag_stream on has any effect, but it's a positive one. I get an fps boost even compared to 3.2.3, over 410 fps on average in the same sample project. Frame times are also better, giving constant stream of 2.3ms

See https://github.com/godotengine/godot/issues/47833#issuecomment-818506326 in the other issue, as although the observed results are slightly different, the underlying problems / discussion are similar.

I can confirm @pycbouh's results with the new options in #47864, turning ON rendering/2d/legacy_stream makes performance go from 80 FPS to 315 FPS (compared with 3.2.3-stable which had 188 FPS).

I can confirm @pycbouh's results with the new options in #47864, turning ON rendering/2d/legacy_stream makes performance go from 80 FPS to 315 FPS (compared with 3.2.3-stable which had 188 FPS).

Out of interest what are the results of this one with legacy_orphan set to off, and legacy_stream set to off?

I think I've tried all combinations between each pair of settings from 3.2.4b2 and nothing else provided any meaningful difference. Including both set to off.

I can confirm @pycbouh's results with the new options in #47864, turning ON rendering/2d/legacy_stream makes performance go from 80 FPS to 315 FPS (compared with 3.2.3-stable which had 188 FPS).

Testing with the same device (Radeon RX Vega 56) on Linux, I get:

Default:

Project FPS: 876 (1.1 mspf)
Project FPS: 874 (1.1 mspf)
Project FPS: 857 (1.1 mspf)
Project FPS: 831 (1.2 mspf)
Project FPS: 801 (1.2 mspf)
Project FPS: 869 (1.1 mspf)
Project FPS: 926 (1.0 mspf)
Project FPS: 831 (1.2 mspf)
Project FPS: 787 (1.2 mspf)
Project FPS: 789 (1.2 mspf)
Project FPS: 840 (1.1 mspf)
Project FPS: 842 (1.1 mspf)

Legacy Stream ON:

Project FPS: 954 (1.0 mspf)
Project FPS: 922 (1.0 mspf)
Project FPS: 924 (1.0 mspf)
Project FPS: 946 (1.0 mspf)
Project FPS: 924 (1.0 mspf)
Project FPS: 928 (1.0 mspf)
Project FPS: 908 (1.1 mspf)
Project FPS: 930 (1.0 mspf)
Project FPS: 910 (1.0 mspf)
Project FPS: 847 (1.1 mspf)
Project FPS: 923 (1.0 mspf)
Project FPS: 911 (1.0 mspf)

Fluctuates a lot but I guess that can be summarized as a +100 FPS gain.

I was just wondering because I hadn't yet worked out what had caused the drop in FPS from 3.2.3 to 3.3, as aside from the orphaning the code was mostly the same if I remember correctly. Maybe there is some other change since 3.2.3.

Something at least improves frame times in 3.3 overall. Not the average, maybe, but the spread. I definitely see it going off both ends in 3.2.4, but not so in 3.3 even with default/auto settings.
If there is anything else we can track, do tell.

Some tests all done on the same machine with AMD FirePro D300 GPU:

OS / Driver
Godot 3.2
Godot 3.3-rc8
Godot 3.x + 47864 and legacy options enabled

macOS 11.2.3 (20D91)
105 FPS
470 FPS
460 FPS

Windows 10 (Radeon Software 21.3.1)
230 FPS
120 FPS
210 FPS

Linux (Mesa 20.2.6/AMDGPU)
270 FPS
350 FPS
350 FPS

As an aside, something I had noted a while ago is that the legacy draw_polygon and draw_generic etc functions are very inefficient API usage wise. (I think these are used to draw the circles).
They call glBufferSubData multiple times per call which could well be not good. I did make a bit of an attempt to improve this to a better method a while ago but made a mistake and reverted, but I've just not had it as a priority to fix. If we pre-prepared the whole buffer before uploading as a one off these calls would probably be a lot more efficient.
It's just not been a massive priority as I'm not sure how much these legacy primitives are used (they may be important in some editor modes though).

Out of curiousity I've also tried this project on my GTX 960m laptop. In 3.2.3 I get anywhere between 200 and 300 fps, sometimes dipping below 200. In 3.2.4 beta 2/3.3 RC 8 I get values between 35 and 50 fps, varying run to run. Changing flags in 3.2.4 beta 2 doesn't have any effect.
That's on Windows 10 and docked (not on battery). I didn't update either drivers or Windows on that laptop for some time though.

RC9 tests on the same machine (average over 1 minute of measurement, with first and last 5 seconds excluded), using official binaries for all tests, on Linux FPS have huge fluctuation for some reason:

Godot version and config
macOS 11.2.3 (20D91)
Windows 10 (Radeon Software 21.3.1)
Ubuntu 20.10 (Mesa 20.2.6)

3.3 RC9, orphan_buffers=2, stream=1
450 ± 20
120 ± 10
465 ± 200

3.3 RC9, orphan_buffers=1, stream=2
470 ± 20
205 ± 10
430 ± 200

3.3 RC9, orphan_buffers=2, stream=2
445 ± 20
205 ± 10
460 ± 200

3.3 RC9, orphan_buffers=1, stream=1
440 ± 20
120 ± 2
430 ± 100

3.2.3 stable
110 ± 2
210 ± 2
265 ± 2

Those results look encouraging, it should be good with the defaults we set in RC9.
The fluctuation on linux I'm not sure we can do anything obvious about, probably something you'd need internal knowledge of the driver to diagnose. Interesting that on windows the performance is halved with the same workload.

Fixed by #47864. Please comment if you still reproduce this specific issue in 3.3 RC 9 or stable.


Unable to install the package

In RemoteFilesystem.php line 515
The "http://packagist.org/p/sebastian/object-enumerator%2490946e571931b2f7d5f583632b48b05dbc3a27717cb469462d36b09268a84d41.json" file could not be downloaded: failed to open stream: Network is unreachable.
Internet is working fine

It got resolved when tried installing it with sudo


Generator Function with Yield in condition causes maybe-uninitialized warning with GCC

Here is a sample function in question:
def first_or_last(int[:] values, bint first):
    cdef Py_ssize_t i

    if first:
        i = -1
        yield values[i]
    else:
        i = 0
        yield values[i]

In both cases, i should always be initialized when used, but gcc emits something along the lines of
hello.c: In function ‘__pyx_gb_5hello_2generator’:
hello.c:2326:32: warning: ‘__pyx_t_2’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     __pyx_cur_scope->__pyx_t_0 = __pyx_t_2;

Moving the yields outside of the branches works fine, but this is a problem showing up in a larger function where I don't know that that is possible
I've attached the generated code. I think this is an issue where the generator saving the state of the function but not entirely sure
hello.c.zip

The warning seems correct: that temp variable is only used in the first branch and not in the second. There is no reason why it would need to get stored away in the closure there.
It could be that some state from the first branch leaks into the second during the code generation phase, e.g. an unreleased temp variable. Needs some more debugging there.


Create new project board for Caltrans

The ask is to create a project board specifically for Caltrans to work with as we get further into handoff.

New project board created, it turns out items from the original project board like size, priority, sprint, and status do not automatically carry over. For next steps I will need to manually set those for active issues. Then we will need to work with Caltrans to pick a day (possibly this coming Monday 10/14) to cut over and use the new project board.
I will also need to recreate the automations from the old board but that's an easy lift.

All active issues in the new board have been assigned the correct size/priority/sprint/status. Will likely need to do it again for any issues created between now and cutover but if we do it next week there shouldn't be too many new ones.
Will talk to Caltrans in Sprint Planning this week to plan the cutover.