net:sockets:tls MBEDTLS Heap optimisation and API support for chained X509 certificates in DER format

1. Context
@mrfuchs and I need to optimize our MBEDTLS heap usage in an application which gathers :

2 TLS/TCP clients which share the same X509 client certificates chain (PEM) : 1 leaf + 3 intermediate + 2 CA (PEM)
1 HTTPS server : public key certificate (PEM)

As all our certificates are persistent for the duration of the connections, we would like to take advantage of the MBEDTLS function mbedtls_x509_crt_parse_der_nocopy which (compared to its neighbor mbedtls_x509_crt_parse_der) doesn't create a duplicate of the provided DER certificate in heap. See Section #Parsing X.509 certificates without copying the raw certificate data.
Currently there are two limitations to this in Zephyr :

Zephyr doesn't take advantage of the function mbedtls_x509_crt_parse_der_nocopy.
There is an API limitation where we cannot provide a chain of DER certificates, the reason is that the credential provided using the API tls_credential_add() is itself provided to the mbedtls_x509_crt_parse()  which is able to parse concatenated PEM certificates but cannot parse more than 1 DER certificate. The two locations I identified : CA certificate here and server certificates here

2. Describe the solution you'd like
We would like :

An API which allows to provide :

A chain of X509 "client" certificates in DER format (credential type TLS_CREDENTIAL_SERVER_CERTIFICATE)
A list of X509 CA certificates in DER format (credential type TLS_CREDENTIAL_CA_CERTIFICATE)

A specific or general configuration option which enables the mbedtls_x509_crt_parse_der_nocopy optimisation for these types of credentials.

In this draft PR #40255, I'm proposing a very first implementation of what we would need as an API and behavior.
A. API : Providing several DER certificates as a credential
My first idea was to add new credential types TLS_CREDENTIAL_CA_CERTIFICATE_DER_LIST and TLS_CREDENTIAL_SERVER_CERTIFICATE_DER_CHAIN which handle a list of CA certificates in DER format and a list of server certificates in DER format respectively.
These credential types are almost handled the same as the original types TLS_CREDENTIAL_CA_CERTIFICATE and TLS_CREDENTIAL_SERVER_CERTIFICATE except that the DER certificates are parsed independently and assembled in the x509 using the mbedtls_x509_crt_parse_der_nocopy function instead of being passed to the function mbedtls_x509_crt_parse which do the complete work itself (but allow less possibilities).
Here is two examples to how to handle this new credential types :

For server certificates :

static unsigned char *der_list[] = {
  ders[0], /* pointer to leaf certificate in flash */
  ders[1], /* pointer tointermediate ca certificate 1 in flash */
  ders[2], /* pointer tointermediate ca certificate 2 in flash */
  ders[3], /* pointer tointermediate ca certificate 3 in flash */
  NULL     /* null pointer to indentify the end of the list */
};
tls_credential_add(HTTPS_SECURE_TAG, TLS_CREDENTIAL_SERVER_CERTIFICATE_DER_CHAIN,
                   der_list, sizeof(*ders));

For CA certificates :

static const unsigned char *ca_der_list[] = {
  ca_baltimore_root_der,
  ca_global_root_g2_der,
  NULL
};
tls_credential_add(HTTPS_SECURE_TAG, TLS_CREDENTIAL_CA_CERTIFICATE_DER_LIST,
                   ca_der_list, MAX(sizeof(ca_baltimore_root_der), sizeof(ca_global_root_g2_der)));

Notes :

I made the choice to add new credential types in order to not change the behaviour of the original types.
One problem to me here is the way to handle each DER certificates size, as a first implementation I decided to use a fixed size for the buffers. This size is provided as argument of the tls_credential_add() function

B. Optimisation : Prevent certificate duplication in HEAP by using mbedtls_x509_crt_parse_der_nocopy
To make it simple, I firstly introduced a general Kconfig option which, if enabled, use function mbedtls_x509_crt_parse_der_nocopy instead of mbedtls_x509_crt_parse_der for all certificates provided through introduced types TLS_CREDENTIAL_CA_CERTIFICATE_DER_LIST and TLS_CREDENTIAL_SERVER_CERTIFICATE_DER_CHAIN.
Note :

Doesn't apply to original credential types
Problem : This is a general configuration option and it cannot be enabled or disabled for specific credentials.

3. Problems of this design
In addition to API choices that I need your opinion on, the two mains problems with this design for me are :

(Major) Not being able to enable or disable the _der_nocopy for some specific credentials.
(Minor) Not being able to provide the accurate certificate size for each DER certificate.

One (good/bad) Idea I have which could solve all at once is to use an array of descriptors (see bellow), which gather the certificates buffer pointers, the certificates sizes and if they are volatile (should be copied in heap : mbedtls_x509_crt_parse_der) or persistent (shouldn't be copied : mbedtls_x509_crt_parse_der)
struct cred_der_certificate {
	/* A pointer to the DER certificate buffer */
	const void *buf;

	/* Size of the DER certificate buffer */
	size_t size;

	/* Indicates whether the der certificate should be copied to the heap or not */
	enum { CRED_VOLATILE = 0, CRED_PERSISTENT = 1 } persistent;
};

struct cred_der_certificate der_list[] = {
   {ders[0], 1024, CRED_PERSISTENT},
   {ders[1], 1060, CRED_PERSISTENT},
   {ders[2], 980, CRED_PERSISTENT},
   {ders[3], 1002, CRED_VOLATILE},
};

tls_credential_add(HTTPS_SECURE_TAG, TLS_CREDENTIAL_SERVER_CERTIFICATE_DER_CHAIN,
                   der_list, ARRAY_SIZE(der_list));

4. Describe alternatives you've considered
There is no actual alternative I think, both problematics come together as we need to provide certificates in DER format in order use function mbedtls_x509_crt_parse_der_nocopy. And as we always need to provide several certificates, we need a new API to do this.
5. Additional context
By implementing the solution described above for CA and client certificates we managed to save up to 5420B of heap during both connect and "TCP established state"  :

MAXIMUM heap usage (connect) before: 36048 B after optimisation : 30628 B
TCP established state : heap usage before : 24076 B after optimisation : 18632 B

Conclusion
All this is technically very feasible but needs a good API, to me this enhancement enables optimisations for most of the secure applications of this kind which keep their certificates (in FLASH for example) for the entire connection duration. Moreover it helps to precisely size heap needs for secure applications.
I am attentive to any comment, idea or advice. Thank you.

@rlubos, what is your take on this? It looks like this feature has to be added in the tls_credentials module you authored. It would be great if you had some time to review @Adecy's proposal.

Hi, thank you for describing the issue clearly.
Let's focus on the certificate chains first.
The general idea behind this was to register multiple certificates with different secure tags, and then feed the TLS socket with an array of tags. Unless I miss something, this approach should already work out of the box with CA certificates, I see however there might be a problem configuring a certificate chain for own certificate since currently it's expected the certificate is coupled with private key. I think we could relax this requirement a bit, i. e. by default just parse the certificate (and thus add it to the tls->own_cert chain), and register own certificate in a similar way we register a CA chain , after going through all of the tags.
I'm not really in favor of increasing the number of credential types with more specialized types - IMO it might be confusing to the users on which credential type to use in which situation (for instance whether to use TLS_CREDENTIAL_CA_CERTIFICATE or TLS_CREDENTIAL_CA_CERTIFICATE_DER_LIST with a single entry), I think we should strive to avoid such ambiguity.
Now as for the no-copy approach.
I agree this might be a nice addition to reduce heap usage. If I understand correctly though, this can only be used with DER certificates, so we need some extra work to determine if we deal with DER or PEM certificate. mbed TLS handles this with a very simple check - perhaps we could provide a helper function in TLS socket implementation that'd allow to identify certificate type. In case DER certificate is detected, and no-copy option is enabled, tls_add_ca_certificate() could simply use mbedtls_x509_crt_parse_der_nocopy() instead of mbedtls_x509_crt_parse().
Another question is how to enable the no-copy functionality. I see 3 options here:

Enable it globally with Kconfig option (I'd consider an option in TLS socket Kconfig space however, we're not really altering mbed TLS functionality here).
Enable it at a socket level with some new socket option - this would allow limiting the functionality only to sockets that explicitly enable it.
Introduce a generic wrapper structure for TLS credential, and add flags field there (something like you've already suggested, but I think the structure shouldn't be limited to a certain credential type). That way we could enable the no-copy for each credential individually.

Personally, I think 2 is a good tradeoff, with a minimal set of changes needed, but if you think the third option is more viable I think it'd be acceptable as well, it just requires more work to adapt the existing codebase to the API changes.

Hi, thank you for describing the issue clearly.
Let's focus on the certificate chains first.
The general idea behind CA chains was to register multiple certificates with different secure tags, and then feed the TLS socket with an array of tags. Unless I miss something, this approach should already work out of the box with CA certificates, I see however there might be a problem configuring a certificate chain for own certificate since currently it's expected the certificate is coupled with private key. I think we could relax this requirement a bit, i. e. by default just parse the certificate (and thus add it to the tls->own_cert chain), and register own certificate in a similar way we register a CA chain , after going through all of the tags.
I'm not really in favor of increasing the number of credential types with more specialized types - IMO it might be confusing to the users on which credential type to use in which situation (for instance whether to use TLS_CREDENTIAL_CA_CERTIFICATE or TLS_CREDENTIAL_CA_CERTIFICATE_DER_LIST with a single entry), I think we should strive to avoid such ambiguity.
Now as for the no-copy approach.
I agree this might be a nice addition to reduce heap usage. If I understand correctly though, this can only be used with DER certificates, so we need some extra work to determine if we deal with DER or PEM certificate. mbed TLS handles this with a very simple check - perhaps we could provide a helper function in TLS socket implementation that'd allow to identify certificate type. In case DER certificate is detected, and no-copy option is enabled, tls_add_ca_certificate() could simply use mbedtls_x509_crt_parse_der_nocopy() instead of mbedtls_x509_crt_parse().
Another question is how to enable the no-copy functionality. I see 3 options here:

Enable it globally with Kconfig option (I'd consider an option in TLS socket Kconfig space however, we're not really altering mbed TLS functionality here).
Enable it at a socket level with some new socket option - this would allow limiting the functionality only to sockets that explicitly enable it.
Introduce a generic wrapper structure for TLS credential, and add flags field there (something like you've already suggested, but I think the structure shouldn't be limited to a certain credential type). That way we could enable the no-copy for each credential individually.

Personally, I think 2 is a good tradeoff, with a minimal set of changes needed, but if you think the third option is more viable I think it'd be acceptable as well, it just requires more work to adapt the existing codebase to the API changes.

Thanks for your comments and suggestions @rlubos.

The general idea behind CA chains was to register multiple certificates with different secure tags, and then feed the TLS socket with an array of tags. Unless I miss something, this approach should already work out of the box with CA certificates

You're right, I missed that.
We discussed @mrfuchs and I agree that your option 2 is a good solution for us. It fits our needs and as you pointed out it requires only "few" changes. Option 3 could be another future enhancement.
To illustrate, I added a commit https://github.com/zephyrproject-rtos/zephyr/pull/40255/commits/1e269f98dbfc577c38318e733e2dfc1651a079f6 to the draft PR #40255 with a implementation of how I understood the option 2 (that I tested with our application) :

Removed the 2 credentials types I firstly added
Added the TLS socket option "TLS_DER_PERSISTANCE" which can have values VOLATILE=0 (default) or PERSISTANT=1.

If set, the option is effective only when parsing DER certificates, if parsing a PEM certificate the option is ignored. Could this be ambiguous to the developper ? We wondered if it would make sense to reject PEM certificates if the option is set ?
This new option also only apply for credentials of type TLS_CREDENTIAL_CA_CERTIFICATE and TLS_CREDENTIAL_SERVER_CERTIFICATE.
A consequence is that this requires a new member in the "tls_context" structure

As we are using this feature in the context of the MQTT client, I added support for this new option in it.
As private key and server certificate are handled at the same time in the current implementation, I changed it to parse the private key independently when it appears when iterating the tags. And server certificates are appended in a chain the same way as CA certificates.

Another consequence is that we need as many tags as we have certificates in the chain. (CA or own certificate)

What is your opinion on this design ?

Hi, thanks for the update, the current proposal corresponds pretty well with what I had in mind, just a few comments.

Added the TLS socket option "TLS_DER_PERSISTANCE" which can have values VOLATILE=0 (default) or PERSISTANT=1.
If set, the option is effective only when parsing DER certificates, if parsing a PEM certificate the option is ignored. Could this be ambiguous to the developper ? We wondered if it would make sense to reject PEM certificates if the option is set ?

Up for discussion (this might only be my personal preference) but I'm wondering about the naming here. For me TLS_DER_PERSISTANCE is a bit cryptic in its meaning, I've been rather tinking on following the mbed TLS nomenclature here and providing something like TLS_CERT_NO_COPY. Theoretically, all certificates provided in tls_credentials storage should be considered non-volatile (at least until they're removed from the storage).
As for the PEM certificates, personally, I'd just ignore the option if set. We could document it explicitly in the option description, that the socket will try to avoid extra copying if possible, not giving guarantees this will actually happen. In the end we're constrained with what mbed TLS allows us to do here. Rejecting PEM in such case would also put an extra limitation on the impelmentation - preventing from mixing DER and PEM certificates on a socket (not likely, but I guess it still should be possible).

As private key and server certificate are handled at the same time in the current implementation, I changed it to parse the private key independently when it appears when iterating the tags.

Just to clarify, in the comment in the code you wrote:
		/* private key is not set anymore with a server certificate,
		 *  there could have several DER server certificates for a
		 *  single private key
		 */

I don't think this is correct. The private key is still related only to a single server certificate (the one that holds the corresponding public key), other intermediate certificates in the chain are only provided to allow the other side to authenticate the "leaf" certificate. So it should still be expected that a single tag corresponds to both, cert and private key. But I guess it should still work with the current implementation, as we have this credential iterator that will fetch all of the credentials using the same tag.

Another consequence is that we need as many tags as we have certificates in the chain. (CA or own certificate)

I don't think this is an issue? The idea being tags was that single tag corresponds to a single credential (or to a credential parir in case of private key/public cert for instance). You can then create certificate chains by providin a list of tags to use, this way you can re-use certificate in different chains w/o a need to register each chain explicitly in tls_credentials.
So to summarize, I think the current proposal looks pretty decent, and I think we could proceed with the actual PR (non-draft) after a small cleanup (TODOs etc.)

Hi, thanks for your feedback @rlubos
I updated the PR (as "non-draft" now) #40255, feel free to request/make changes at your convenience (I'm not yet very confident with PR but I assume this is possible ?).

So it should still be expected that a single tag corresponds to both, cert and private key. But I guess it should still work with the current implementation, as we have this credential iterator that will fetch all of the credentials using the same tag.

With the current implementation, the private key would be expected with the certificate for each tags. So the function will fail to add an intermediate certificate (which isn't provided with a private key).
https://github.com/zephyrproject-rtos/zephyr/blob/525fa76f4dd13d42549261c9fc846a150ee291bc/subsys/net/lib/sockets/sockets_tls.c#L738-L747
In my updated PR the function tls_mbedtls_set_credentials() only checks at the end that the private key is set if the own certificate is provided. However, I'm wondering if we could not just trust mbedTLS returned error in this case, which is : https://github.com/zephyrproject-rtos/mbedtls/blob/3e3e58a92de76069730c63e0d524f40fea948a61/include/mbedtls/pk.h#L54

Another consequence is that we need as many tags as we have certificates in the chain. (CA or own certificate)

I don't think this is an issue? The idea being tags was that single tag corresponds to a single credential (or to a credential parir in case of private key/public cert for instance). You can then create certificate chains by providin a list of tags to use, this way you can re-use certificate in different chains w/o a need to register each chain explicitly in tls_credentials.

It was just to point it out, your argument is good, i'll take it


Full RoadRunner Implimentation

Once finished, this PR will close #1. Currently, it's an untested work-in-progress. Some notes on the latest commit:
This commit implements RoadRunner, but in a way that I don't actually think works yet. We'll have to check, but some things come to mind:

We need to check how RoadRunner sets the motor power, possibly using DcMotorEx instead of DcMotor
I'm not great with Java so some places here are using things like List.get and List instead of ArrayList - we may want to change those
I will refactor this code to make it look good - this is a draft PR for now

Note
Currently, the IDE lists several errors, but I have confirmed this builds without an issue. We still need to check the new methods on the robot though, potentially altering StraightTest (which I used) to use MecanumDriver in place of SampleMecanumDrive.

Do we have an IMU config?

Pinging @jander15 and @Thomas011608 as well on this PR. Does this look OK in the implimentation for now? Do you know if we have an IMU config right now at all? If not, I should be able to refactor that out of the method entirely, but I wasn't sure if we had it. Also, this code still needs to be tested on the robot - we should test it tomorrow most likely, but there's a large chance it won't work at all.


API: Add CREATE/CALL support to API compose-raw-transaction

Sorry for the trouble. Could you rebase your changes on API v2.3.0?

Not a problem! Will update

Ok, think it looks clean to review.


[stable/datadog] Conf.d does not template postgresql configuration

Describe the bug
When trying to configure postgresql integration with the agent, it creates a file within conf.d instead of the right integration within conf.d/postgres.d
Version of Helm and Kubernetes:
Helm v2.13.1 and Kubernetes v1.15.11-gke.5
Which chart:
datadog/stable v1.39.8
What happened:
I've defined the following values for postgres.d integration but the configuration does not get created or gets created in the wrong folder
https://docs.datadoghq.com/integrations/postgres/
datadog:
  confd:
    postgres.d: |-
      init_config:
      instances:
        - host: mydb.database.svc
          port: 5432
          username: datadog
          password: password1
         ssl: prefer

Also tried
datadog:
  confd:
    postgresql.yaml: |-
      init_config:
      instances:
        - host: mydb.database.svc
          port: 5432
          username: datadog
          password: password1
         ssl: prefer

What you expected to happen:
This should've created a configuration within /etc/datadog-agent/conf.d/postgres.d/ in the agent for the the data to be sent to datadog postgres integration
How to reproduce it (as minimally and precisely as possible):
Define the sample configuration above as a value to the chart
Anything else we need to know: no

I'm running into the same and I can't get it to work, I can't see anything in pod logs about it either.


Fix "Loaded resource as image file:" warning spam

"Loaded resource as image file, this will not work on export: "res://icon.png". Instead, import the image file as an Image resource and load it normally as a resource."
"res://" prefix cause this warning arrive when loading it. Its happened for project icon and boot splash screen image, it could be loaded without it, which remove these irritating messages from developer view
Bugsquad edit: Fixes #21072.

Closed as its not a proper way to fix it as discussed in IRC


Regent: CUDA generator fails with particular triple nested loops

The following code fails to run through the CUDA code generator:
import "regent"

__demand(__cuda, __leaf)
task f(r : region(ispace(int3d), int), s : rect3d)
  for i in r do
    for j = 0, 10 do
      for k in s do
      end
    end
  end
end

task main()
end
regentlib.start(main)

Result:
$ ./regent.py test_regent_cuda.rg -fcuda 1
...ght/legion_compute/language/terra.build/src/terralib.lua:1172: Errors reported during typechecking.
/scratch2/eslaught/legion_compute/language/src/regent/std_base.t:308: definition of variable with symbol $offset is not in scope in this context
    return `([lhs] + [rhs])
                        ^

stack traceback:
        [C]: in function 'error'
        ...ght/legion_compute/language/terra.build/src/terralib.lua:396: in function 'finishandabortiferrors'
        ...ght/legion_compute/language/terra.build/src/terralib.lua:3437: in function 'typecheck'
        ...ght/legion_compute/language/terra.build/src/terralib.lua:1172: in function 'defineobjects'
        ...h2/eslaught/legion_compute/language/src/regent/codegen.t:8931: in function 'fn'
        ...ht/legion_compute/language/terra.build/src/terralist.lua:150: in function 'block'
        ...h2/eslaught/legion_compute/language/src/regent/codegen.t:11203: in function 'compile_thunk'
        ...2/eslaught/legion_compute/language/src/regent/std_base.t:1660: in function 'complete'
        /scratch2/eslaught/legion_compute/language/src/regent/std.t:4519: in function </scratch2/eslaught/legion_compute/language/src/regent/std.t:4517>
        /scratch2/eslaught/legion_compute/language/src/regent/std.t:4521: in function 'compile_tasks_in_parallel'
        /scratch2/eslaught/legion_compute/language/src/regent/std.t:4655: in function 'start'
        test_regent_cuda.rg:15: in main chunk

As best I can tell this is just a slightly odd triple-nested loop, so I have no idea why it's not working.

Some of the sensitivities I'm observing:

Any two of these loops seem to pass. You need three loops to make it fail.
Looping over r and r in the innermost and outermost loops respectively pass. Same with s/r. Only r/s and s/s fail: i.e., with s in the innermost loop.
Replacing s with an ispace(int3d) makes this pass. Somehow this is specific to rects.

I found that the following code passes:
import "regent"

__demand(__cuda, __leaf)
task f(r : region(ispace(int3d), int), s : rect3d)
where reads writes(r) do
  for i in r do
    r[i] = 0
    for j = 0, 10 do
      for k in s do
      end
    end
  end
end

task main()
end
regentlib.start(main)

So adding a region access to the outermost loop somehow makes this work.

Ok, this seems to be sufficient to develop a workaround for the real code I'm working on. It is still a really strange behavior that we should fix.

Actually, I'm still having issues. When I run the version in https://github.com/StanfordLegion/legion/issues/1351#issuecomment-1307880003, and actually enable the GPU, I see:
$ ./regent.py test_regent_cuda.rg -fcuda 1 -ll:gpu 1
[0 - 7fc86bf8c780]    0.000000 {5}{gpu}: Failed to load CUDA module! Error log: ptxas application ptx input, line 347; error   : Call has wrong number of parameters
ptxas fatal   : Ptx assembly aborted due to errors
CU: cuModuleLoadDataEx = 218 (CUDA_ERROR_INVALID_PTX): a PTX JIT compilation failed

The difference vs before is that I added the -ll:gpu 1 flag. So the code compiles to PTX but the PTX does not compile.

this should be fixed by https://gitlab.com/StanfordLegion/legion/-/commit/1818051ddd53d37c91d7bfe28a2961c31013c71f. @elliottslaughter can you pull and try again?

The issue seems to be fixed, thanks!


Fixing query with Get-AzureAdLicense

DEBUG: Function: Get-AzureAdLicenseServicePlan: Splitting at '': PAM_ENTERPRISE b1188c4c-1b36-4018-b48b-ee07604f6feb)
WARNING: Cannot read Entry 'PAM_ENTERPRISE b1188c4c-1b36-4018-b48b-ee07604f6feb)' - malformed string. Reading this requires open and close parenthesis around ServicePlanId - please open issue against
Documentation: https://docs.microsoft.com/en-us/azure/active-directory/users-groups-roles/licensing-service-plan-reference
fixed by adding a "("

DEBUG: Function: Get-AzureAdLicenseServicePlan: This ServicePlan: AAD_SMB (de377cbc-0019-4ec2-b77c-3f223947e102)EXCHANGE_S_FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)INTUNE_SMBIZ
(8e9ff0ff-aa7a-4b20-83c1-2f636b600ac2)INTUNE_A (c1ec4a95-1f05-45b3-a911-aa3fa01094f5)
DEBUG: Function: Get-AzureAdLicenseServicePlan: Splitting at '': AAD_SMB (de377cbc-0019-4ec2-b77c-3f223947e102)
DEBUG: Function: Get-AzureAdLicenseServicePlan: Splitting at '': EXCHANGE_S_FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)
DEBUG: Function: Get-AzureAdLicenseServicePlan: Splitting at '': INTUNE_SMBIZ (8e9ff0ff-aa7a-4b20-83c1-2f636b600ac2)
DEBUG: Function: Get-AzureAdLicenseServicePlan: Splitting at '': INTUNE_A (c1ec4a95-1f05-45b3-a911-aa3fa01094f5)
DEBUG: Function: Get-AzureAdLicenseServicePlan: Splitting at '':
WARNING: Cannot read Entry '' - malformed string. Reading this requires open and close parenthesis around ServicePlanId - please open issue against Documentation:
https://docs.microsoft.com/en-us/azure/active-directory/users-groups-roles/licensing-service-plan-reference
WARNING: Cannot read Entry '' - malformed string. Reading this requires open and close parenthesis around ServicePlanId - please open issue against Documentation:
https://docs.microsoft.com/en-us/azure/active-directory/users-groups-roles/licensing-service-plan-reference
fixed by removing a superflous ""

@DEberhardt : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@Nicholak-MS: I'm sorry - only the author of this article, @daveba, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

Thanks for approving @Nicholak-MS .


What's the correct matcher of 'tf.nn.conv2d_transpose' in quantization aware training ?

To some reason, I have to use TF1.X (actually TF1.14.0) to quantize a custom model.
As general,
g = tf.get_default_graph()
tf.contrib.quantize.create_training_graph(input_graph=g,quant_delay=0)

was used to create fakequant min/max variable and I could find the min/max value of 'tf.nn.conv2d_transpose' in checkpoint. But when
tf.contrib.quantize.create_eval_graph()

created the Fakequant Evaluation Graph, 'tf.nn.conv2d_transpose' was not added fake quantization nodes. I tried transpose only, transpose --> relu6, transpose --> bias --> relu6, transpose --> bacthnorm --> relu6, but all of them did not work.
Thus the direct question: What's the correct matcher of 'tf.nn.conv2d_transpose' in quantization aware training ? Is there any resources documented the correct matchers of every op ?

@ethkim could you triage this issue?


Error in application configuration

Describe the bug
after update package I got the error
To Reproduce
Steps to reproduce the behavior:

Configure a django application with django import export and django import export celery
call python manage.py check in terminal

Versions:

Django Import Export: [2.9.0]
Python [3.10.9]
Django [4.2.5]

Screenshots

This error is coming from a different library 'django-import-export-celery'.  Please raise this issue there.


Create Items

Can this actually create extra items in pokestops?

Be aware: you can see it, you can touch it - you won't own it


feat(core): add scss mixin support

PR Checklist
Please check if your PR fulfills the following requirements:

[X] The commit message follows Conventional Commits
[X] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[X] Feature
[ ] Refactoring (no functional changes, no api changes)
[ ] Other... Please describe:

What is the current behavior?
SASS users cant use
@import '~@taiga-ui/core/styles/taiga-ui-local';

Closes #164
What is the new behavior?
SASS users can safely use below
@import '~@taiga-ui/core/styles/taiga-ui-local';

Does this PR introduce a breaking change?

[ ] Yes
[X] No

Other information

I added new files next to less files as a reminder to change scss files when working on less files:

I added new files next to less files as a reminder to change scss files when working on less files:

@waterplea  do not accept yet. Found an issue

@waterplea  do not accept yet. Found an issue

I realized my SASS was outdated from previous branch.

I realized my SASS was outdated from previous branch.

@waterplea  do not accept yet. Found an issue

Been fixed

@waterplea  do not accept yet. Found an issue

Been fixed

I don't think this change allows us to use the grid mixins from a scss file, or does it?
The grid mixins are in markup/tui-row.less. I don't see a scss version of the files in the markup directory.
Side question 1: why there is not a scss version of taiga-ui-global.less (which import tui-row.less)? Probably because the files in markup don't have a scss version yet either?
Side question 2: when would one use taiga-ui-global.less instead of taiga-ui-local.less?
Thanks for your response!

Hi @JohanHeyvaert
Grid is in global styles. These are global classes and not mixins. There is no SCSS global file because it doesn't matter what it uses, you just add it to angular.json and Angular does everything you need. You add global there once. You add local whenever you need to use a mixin in a file.
I would recommend to use your own layout classes or some dedicated library like Tailwind. Global classes are somewhat limited in Taiga UI and I believe we should drop most of them someday in a major release. We cannot compete with dedicated solutions 🙂


Example usage

Some more examples on how to use the s3-uploader would be nice. Maybe in an example folder, that way we can more easily integrate them with the test suite making sure no example gets broken between releases.

While figuring out the broken cache property in the readme was not working I was thinking about an examples folder as well yesterday!

Do you mean a couple of functional tests that could serve as documentation as well?
Like: is the image actually uploaded to path n on aws, are the cache headers actually set to n?

Yes. The thought was to move the Integration Tests as separate examples in the example directory and have a way to integrate them back into the test suite.


Fix dev stacks containing less than 3 prometheis

Why I am making this change
By default dev stacks only have 1 instance, however this causes a problem when getting the fqdns as it was expecting at least 2. This is a quick fix whilst we are in the middle of migrating to EC2 prometheis which will mean that the EC2 prometheus will only be shown when the number of prometheis is 3 not less.
What approach I took
As we are migrating the EC2 prometheis there is not much point in spending too long on a solution. This solution should get the dev stack working for EC2 prometheis.

needs terraform fmt tho


send email with collectives' donations/expenses to their host

We should let hosts know on a regular basis when their collectives are receiving donations to make their management easier and more efficient. I suggest hosts can opt-in between every activity/daily digest/weekly digest. I wouldn't offer monthly since they get the monthly email anyway.

It's already happening for a host. But not for the members of the collectives. Is that what you meant?

My mistake. No I don't think we should send that to members unless they specifically opt-in. Spam risk is too big. Closing this.


Terraform provider

I'm looking for a Terraform provider for Neuvector, is there one out there or is someone working on it?

I have started working on a provider at https://registry.terraform.io/providers/naitmare01/neuvector. It is still in early stages and I'm still trying to wrap my heaad around the API.

Thanks for your contributions. There are several references you can check out. They are a bit dated so might need updating.
https://github.com/devseclabs/nv-deployment
https://blog.neuvector.com/article/an-introduction-to-secure-infrastructure-as-code-iac-using-terraform
https://blog.neuvector.com/article/how-to-use-terraform-to-deploy-secure-infrastructure-as-code

@gkosaka thanks for the documents. Will have a look through them. The provider I was after and now is writing will focus on configure the setup after e.g. Helm has deployed the pods.
Has started some initial work on managing the oidc-setting through terraform for example.

Hi,
I've worked with NeuVector and had to install it on several clusters. I was also looking for a terraform provider for NeuVector but there was none really useful apart from @naitmare01 one.
So I decided to make a wrapper for the client and abstract some NeuVector resources (https://github.com/theobori/go-neuvector) that I use through the Terraform plugin (https://registry.terraform.io/providers/theobori/neuvector/0.4.2 and https://github.com/theobori/terraform-provider-neuvector) to configure NeuVector (eula, user, roles, etc..)  and manage the life cycle of some NeuVector resources (registries, policies, admission rules, etc..).
Hashicorp recommends to do Dedicated Upstream Librarie, that's why I have made a wrapper =]
It's not finished yet, any help is welcome :)


Make Website More Accessible

In the following pull request:

Edit some alt attributes and make it more descriptive.
No linter errors are recorded.
-Suggested some general changes in regard to the structure of the page.

In the following pull request:

[ ] Edit some alt attributes and make it more descriptive.

[ ] No linter errors are recorded.

[ ] Suggested some general changes in regard to the structure of the page.


The location in generate_site_positions.py script

Dear,
I want to use the generate_site_positions.py script to creat a restriction sites file for my study genome.
But I don't know what is the  [location] parameter in this python script?
generate_site_positions.py <restriction enzyme> <genome> [location]
Thanks.

Please post to our forum at aidenlab.org/forum.html and someone will answer
you.
The github is for bug reports.
Thanks!
On Sat, Mar 24, 2018 at 10:19 AM, hope<EMAIL_ADDRESS>wrote:

Dear,
I want to use the generate_site_positions.py script to creat a
restriction sites file for my study genome.
But I don't know what is the [location] parameter in this python script?
generate_site_positions.py   [location]
Thanks.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/theaidenlab/juicer/issues/55, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABWiW2S9qRIZp8xKORd7E_6sDDUr-ViOks5thlYBgaJpZM4S5wHk
.

--
Neva Cherniavsky Durand, Ph.D.
Staff Scientist, Aiden Lab
www.aidenlab.org


Was called negation is missing in assertion signature in documentation

It is possible to assets expect(mySpy, 'was not called'). But the documentation doesn't show this assertion in the left hand menu, nor does it appear in the DSL for was called, which I would expect to say was [not] called

The assertions are here: https://github.com/unexpectedjs/unexpected-sinon/blob/master/lib/unexpected-sinon.js#L404-L411

PR welcome ;)

I guess it's just an artifact of the assertions being defined separately. If you turn them into one that checks expect.flags.not I think it'll come out right.


Extension doesn't show any hints

Suggestions don't appear to be working in my vscode for some reason.  I've put in some code that I know should trigger a refactor suggestion, but no suggestions appear on the code or the suggestions sidebar.  I tried uninstalling and reinstalled and all of the suggestions settings are set to hint.
Does anyone have any suggestions on how I can fix this?
P42
v1.86.0
VScode
Version: 1.64.2 (user setup)
Commit: f80445acd5a3dadef24aa209168452a3d97cc326
Date: 2022-02-09T22:02:28.252Z
Electron: 13.5.2
Chromium: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows_NT x64 10.0.22000

Thanks for the bug report!
Can you try the following example?
let a = "123";
f(a);

It should show up as follows:

The 3 dots under let are the suggestion indicator. Does it show for you?
Btw, in v1.86.0 the default suggestion level was changed from info (blue squiggly underline) to hint (3 dots) to make the suggestions look less distractive and indicate that they are not problems (many users perceived the squiggly line as such).

No joy :(

The red squiggle on f is just eslint complaining that f is not defined.

Do you have a p42.toml in the root folder of your workspace? If so, please check if the refactoring is deactivated there (see https://p42.ai/documentation/p42-for-vscode/configuration#configuration-file for details, specific paths and/or refactorings could be disabled).
If not, can you set "p42.logging.error.enabled": true in your VS Code settings, restart VS Code, open the file that should show suggestions, and then check if any errors are logged in the 'Output' panel (select P42)?

I don't have a p42.toml file so no worries there.
I added the setting (although vscode greys it out saying "Unknown Configuration Setting") and restarted vscode.  There was nothing logged to the output panel.
The extension appears to be working because vscode shows that it had a 153ms activation time.

Thanks for checking! The "unknown configuration setting" is expected (the options are not exposed in the UI).
Just to make sure I'm not missing anything, what is the extension of the JavaScript file that you are working on?
P42 only works with certain extensions such as .js, .ts or .mjs (see https://p42.ai/documentation/p42-for-vscode/faq for all extensions).

It is a .js file, but I think I've narrowed down the problem.  It appears that the issue is related to multiple folders in the workspace.  I created a test.js file with your code and saved it to a folder called p42-tests then ran the following tests:
Test 1

Open folder with vscode
P42 suggestions visible
Save workspace
Close and re-open workspace
Suggestions still visible

Test 2

Open folder with vscode
P42 suggestions visible
Add another folder to workspace
Suggestions disappear
Save workspace
Close and reopen workspace
Suggestions still not visible

Thanks for the investigation! I'll take a look at the multiple folders issue today and see if I can confirm & fix it.

Update: I have reproduced the issue, now working on a fix.

Works!  You are awesome, thanks for the super quick response and a great addon!

Thanks! I hope you find the extension helpful!


How to add pthread extension to php container

Hi there,
I'm bogged down with php extensions. I need to add pthread extension to my drupal application. I was wondering if I'll have to write my custom Dockerfile from scratch or I'll can extend the wodby/drupal-php or add some configurations to docker-compose.yml file.
Thanks.

We don't add pthreads extension to wodby/php image because it requires php to be compiled with the flag --enable-maintainer-zts which can cause can cause significant and asymmetrical slowdowns. So, you'll have to fork wodby/base-php (which is already a fork of _/php) to compile with this flag, then fork wodby/php where you add the extension via pecl.


RPC connection problem

For iOS applications, after the application is placed for a period of time, the data is not available through GRPC. After several tens of seconds, the data can be obtained. Is the PRC disconnected? What parameters should be set to stay connected?
Setting the following code didn't work.
[GRPCCall setKeepaliveWithInterval:30000 timeout:150000 forHost:[UCUserDataManager shareInstance].grpc];

It sounds like the connection was broken and was in backoff mode, and resumed after the backoff period. You can turn on the logging to confirm if that is what happened

Closing for no activity.


Podfile issue

I had to fix the Podfile as follows in order to run pod update without errors:
target 'Enesco' do

    pod 'DOUAudioStreamer', '0.2.11'
    pod 'SDWebImage'
    pod 'GVUserDefaults'
    pod 'MBProgressHUD'
    pod 'Mantle', '1.5.5'
    pod 'AFNetworking', '~> 2.0'
end

Fixes here.


Update Klement to exploit the diagonal structure

For the General Klement updates (which in practice don't work that well) one can use the one in NonlinearSolve
We were anyways constructing a diagonal jacobian throughout but were not exploiting the structure.

This should also be good to go


Use of this library for huge xml processor

Instead of chunking the file, I wonder if this can act as a processor for a huge XML file. For example, given a file of 8GB XML, and with chunks of 100MB, perhaps we can calculate the number of chunks and loop through each chunk, or go directly to a specific chunk to process it reliable? Do you think it's possible?

Hi there! If I understand your message correctly, you mean something like a callback, which is called every time a chunk is created with (let's say) the chunked data as parameter to that callback? So that a custom action can be bound on every chunk's completition. If yes, absolutely! If not, feel free to elaborate :)


await-interactions: false positive and bad fix if await result is passed to expect in one line

Describe the bug
Rule reports problem on following code:
const promise = new Promise<{foo: string}>((resolve) => {
  setTimeout(() => {
    resolve({foo: 'bar'});
  }, 500);
});

expect((await promise).foo).toBe('bar');

if it's written this way:
const promise = new Promise<{foo: string}>((resolve) => {
  setTimeout(() => {
    resolve({foo: 'bar'});
  }, 500);
});

const {foo} = await promise;
expect(foo).toBe('bar');

rule doesn't complain.
Fix is proposed to prepend expect((await promise).foo).toBe('bar') which is incorrect as it doesn't need that in the first place.
That's true Jest expect can be awaited, but then
expect(promise).toBe(resolveValue)

should become
await expect(promise).resolves.toBe(resolveValue)

and fixer just prepends the await keyword
Additional context

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

<EMAIL_ADDRESS>

After a brief look into the rule code I can narrow the cause of what I see as a problem down to this section:
https://github.com/storybookjs/eslint-plugin-storybook/blob/611b70eae394a491d3284f060c0566d74e6bc545/lib/rules/await-interactions.ts#L101-L109
There's a design decision to enforce all expect calls to be awaited, but it's not even covered fully (e.g. it won't trigger the rule if it's expect(…).not.toBe(…) where the pattern of CallExpression>MemberExpression>CallExpression, which this chunk above detects is not preserved)
I went through the history of changes, of the rule, and I see that expect initially was one of FUNCTIONS_TO_BE_AWAITED but then some refinements came, but I'm not sure why was it considered as "it needs to be awaited" at all - it's not at all obvious it should


Mismatched annotation tables caused by recordings without annotations

Describe the bug
I am training a custom model with two classes: HAAM (Hawaiian 'Amakihi) and noise. To generate my training, validation, and testing data, I generated three random subsets of 2-minute files, one for each dataset. After training the model, I ran it on my validation dataset, which was composed of 42 files.
In BirdNET, I chose to "Combine selection tables". The resulting BirdNET output selection table found 46 detections from 13 of the 42 files.
When I loaded in the input files (Select input directory (recursive)), I noticed that the file order matched the order of the files that I manually annotated in Raven.
However, when I ran the model and saw the list of files at the bottom of the program, I noticed that the files were not analyzed in the same order, and this lead to the selection tables not aligning. The Start Times and End Times did not match up because the file order was different. I think this is due to the fact that I am running the model on 7 CPU threads. I switched to run it 1 by 1 and it looks like the order was corrected.
So, is there a way to keep the correct file order even when running in parallel? I think this would be very helpful for larger batch analyses.
Thank you
To Reproduce
Steps to reproduce the behavior:
For the GUI:

Go to '...'
Click on '....'
Scroll down to '....'
See error
For the CLI provide us with the arguments used.

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. MacOS, Win10]
Version [e.g. 1.0.2]

Additional context
Add any other context about the problem here.

I created a pull request for a fix, that ensures that the combined selection tables has the same order the input files.
If you have the option of running from the command line you might be able to workaround until this is released, as the analyze script does not mess up the ordering of the result files.

Fixed in #473


Infinite scroll with list view and grid view

I implemented infinite scroll for list view and grid view, when scroll up on grid view it does not send request mean it's working fine, but when scroll up on list view it sends request, while data.length!==0. I don't know why this happening.

It's fixed


shortcut command error

When i use shortcut command for login (/l) say me that: Login failed! Invalid password. When i use normal command (/login) work: Successfully logged in.
I used /l and /login with same password.

This should be fixed in the latest version.


stats.scipy.multinomial is not vectorized?

It appears that stats.scipy.multinomial does not support vectorized operations in a way that is consistent with other scipy.stats distributions that I have used:
from scipy.stats import binom, multinomial

# 3 binomial RVs, each with n=1, but with different probabilities of success
pbinom = np.array([0, 0, 1])
xbinom = binom(n=1,p=pbinom)
xbinom.rvs()
>>> array([0, 0, 1]) # returns the result of the 3 "coin flips"

# 3 multinomial RVs, each with n=1, but with different probabilities of the (3) classes
pmulti = np.array([[0,1,0],[1,0,0],[0,0,1]])
xmulti = multinomial(n=1,p=pmulti)
xmulti.rvs()
>>> array([[0, 1, 0]]) # returns the result of only the first "die roll"

# expected:
# array([[0,1,0],
#            [1,0,0],
#            [0,0,1]])

It also seems that n can't be vectorized:
xmulti = multinomial(n=np.array([1,2]), p=np.array([[0,0,1],[1,0,0]]))
xmulti.rvs()

returns an error, whereas
pbinom = np.array([0, 0, 1])
xbinom = binom(n=np.array([1,1,5]),p=pbinom)
xbinom.rvs()
>>> array([0, 0, 5])

does exactly what one would expect.

multinomial.rvs() calls random_state.multinomial(n, p, size)
https://github.com/scipy/scipy/blob/7e3cda70f5448bc2afa019d7b49533eaba33090e/scipy/stats/_multivariate.py#L3218-L3220
random_state.multinomial doesn't appear to deal with the multi-dimensional pmulti. I.e. It returns only the values corresponding to the first row of pmulti
pmulti = np.array([[0,1,0],[1,0,0],[0,0,1]])
random_state = np.random.mtrand._rand
random_state.multinomial(1, pmulti, None)
#  array([0, 1, 0])
random_state.multinomial(1, pmulti[0], None)
#  array([0, 1, 0])
random_state.multinomial(1, pmulti[2], None)
#  array([0, 0, 1])

# What you would like is the equivalent of
np.array([rs2.multinomial(1, _p, None)  for _p in pmulti])
# array([[0, 1, 0],
#        [1, 0, 0],
#        [0, 0, 1]])

The multinomial docstring is
https://github.com/scipy/scipy/blob/7e3cda70f5448bc2afa019d7b49533eaba33090e/scipy/stats/_multivariate.py#L2949-L2952
It gives a list of some methods that support broadcasting and rvs is not on that list. There is a comment in the original PR https://github.com/scipy/scipy/pull/6375#issuecomment-233080752 that suggests this issue was known at the time.

@pvanmulbregt thanks for looking into it! I didn't assume this was a bug per se, but I wanted to point out that regardless of the reason, the behavior of .rvs is inconsistent in this case in a way that makes building generic packages on top of scipy.stats difficult. It's especially weird now that I know that .pmf actually does support broadcasting! :)

@alejandroschuler would you be interested in submitting a PR with a fix?

FYI: in one of the examples presented np.random.Generator appears to broadcast. Is this the correct behaviour?
>>> rg = np.random.default_rng()
>>> rg.multinomial(n=np.array([1, 2]), pvals=np.array([[0, 0, 1], [1, 0, 0]]))
array([[0, 1],
       [0, 2]])

Upon further investigation, multinomial is not taking the first row.
When passed an array of shape (m,p), it decides the number of possible values is m (not p), and takes the first (m-1) elements of the array and uses those. This selection may cross rows. E.g.
In[86]: pmulti = np.array([[0,1,0],[0.5,0,0.5],[0,0,1], [0.1, 0.1, 0.8], [0.2, 0.2, 0.6]])
In [87]: pmulti
Out[87]:
array([[0. , 1. , 0. ],
       [0.5, 0. , 0.5],
       [0. , 0. , 1. ],
       [0.1, 0.1, 0.8],
       [0.2, 0.2, 0.6]])

In [88]: np.shape(pmulti)
Out[88]: (5, 3)

In [89]: rng.multinomial(20, pmulti)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-85-541ce7a1994a> in <module>
----> 1 rng.multinomial(20, pmulti)

generator.pyx in numpy.random.generator.Generator.multinomial()

ValueError: sum(pvals[:-1]) > 1.0

The error occurs because it treated the (5,3) array as an array of length 5, and used 3 values from row 0, and 2 values from row 1.  It summed the first 4 elements of  [0, 1, 0, 0.5, 0, 0.5, ..., 0.2, 0.2, 0.6] to get 1.5 which is bigger than 1, so raised an Exception.

@ev-br I'm not at all familiar with the scipy.stats codebase so I'm probably not the right person. It looks like another issue has been raised (https://github.com/numpy/numpy/issues/15492) that's more specific so hopefully folks will narrow in on the issue and resolve it there.
The reason I bring this up is because I'm one of the authors of the newly-popular NGBoost package. Right now, the process for developing a new distribution to add to NGBoost is not as user-friendly as I'd like it to be. In my ideal world, it would be as easy as pointing to an implementation in scipy.stats, but I can't quite get there because of some inconsistencies and my unfamiliarity with some necessary internals.
By any chance do you know if there someone who is an involved contributor to scipy.stats that might be willing to email or chat with me about a tighter integration between NGBoost and scipy.stats?


Whether there is a directory access function

I was looking for a good editor to integrate into my system, and I found jodit very good. I need a directory function, and clicking the right directory can scroll the text area, I did not find this function in the document, I wonder if it can be supported?

What editor has this feature? Can you provide such a function?

What editor has this feature? Can you provide such a function?

https://www.wangeditor.com/demo/catalog.html


[TASK] Minimum resource requirement investigation

This issue is for tracking the minimum CPU requirement for different deployment.
The reason for this is we are seeing sometime due to CPU starvation, longhorn-engine controller and longhorn-engine replica don't have enough CPU to perform I/O operation, which caused I/O read/write timeout. This may impact user experience and data storage.
We need to find some formula that can give us a guidance about the guaranteed CPU resource.
For example, for 3 node, 1 volume (20GB size) with 3 replicas, how much CPU is needed. And for 3 node, 2 volumes (20GB size) with 3 replicas each, how much CPU is needed. And how about 6 node, 10 node. Also for large volume size, like 50GB, 100GB, 200GB.

Also, we need to know about the network and disk IO requirements as well.

Minimal CPU usage test:

Use image 1shuowu/longhorn-manager:8af120dd-dirty1 to deploy Longhorn, which can set CPU limits for engine manager pods or replica manager pods.
Make sure the request CPU resource is much smaller than the limit. (set to 0.001)
Attach and mount volumes. Then run

while true; do dd if=/dev/urandom of=/mnt/data/t1 bs=512 count=10000000 && rm /mnt/data/t1; done &
while true; do dd if=/dev/urandom of=/mnt/data/t2 bs=1M count=10000 && rm /mnt/data/t2; done &

Check manager log. And monitor CPU usage via kubectl -n longhorn-system top pods

For each replica process, 25~30m CPU resource is needed as the minimum requirement. Here are some test results:
               20m        25m        30m         40m         60m         75m
1 replica       x          ✓
2 replicas                             x           ✓
3 replicas                                         x           x           ✓

Good work @shuo-wu !
Can you check if we can safely claim 30m CPU per engine/replica even there are more engines/replicas on the node? I want to see if the minimal requirement holds. Pay attention to the potential IO bandwidth limitation on the disk or network when you're testing with more engines and replicas.
Also, pay attention to the volume attached node's dmesg. There are might be recoverable errors that show up but that should be considered as a failure too.
We also need to measure the performance in this condition. We can use https://github.com/longhorn/dbench .
BTW, there is a missing checkmark for the 3 engines case.

It won’t help if we control the bandwidth for each engine since the user won’t do that. Try to find a total workable bandwidth instead.

@shuo-wu Can you add a checklist regarding what measurement we were planning to do? So other engineers can pick it up when you're working on other items.

OK. Here is the TODO list:

Minimum disk performance requirement: Bandwidth requirement when writing sequential data to multiple volumes/replicas simultaneously
Minimum disk performance requirement: IOPS/Latency requirement when writing random data to multiple volumes/replicas simultaneously
Minimum disk performance requirement: IO depth requirement when writing sequential/random data to multiple volumes/replicas simultaneously

Additionally, we may need to re-test the network requirement if we can allocate network resources to all engine-replica sessions uniformly (based on time slot) for each node: https://github.com/longhorn/longhorn/issues/2021
@yasker Can you check if there is anything else missing?

IO Depth is a parameter we can tweak on the client iSCSI side. It's more like a setting than resource.
Otherwise LGTM.

Reassign to @PhanLe1010 since he is working on #2206 , so he needs to measure the minimal IOPS requirement.

Reassign to @PhanLe1010 since he is working on #2206 , so he needs to measure the minimal IOPS requirement.

ref: https://forums.rancher.com/t/general-io-performance-of-longhorn-storage-backend/14970/5 (about IO speed)

Need to schedule this after #2206 , since the fix of #2206 will affect the stability.

I think we should reprioritize this since #2206 's fix has been merged. It should greatly increase the stability of the data plane in low resource environment.

Discussed with @innobead , this should be done as a part of https://github.com/longhorn/longhorn/issues/2598

@PhanLe1010 will communicate with @khushboo-rancher for the testing plan. @khushboo-rancher will coordinate the following efforts.

Analysis - In progress
Result with Longhorn master-head -08/09/2021 is as below
Minimum network resource requirement
Bandwidth
1 volume with 3 replicas

Bandwidth/bs
512
4k
64k
256k
1m
4m

10mbit
317KiB/s
396KiB/s
448KiB/s
x
x
x

50mbit

2951KiB/s
x
x

80mbit

3155KiB/s
3153KiB/s
3143KiB/s
x

100mbit

5777KiB/s
5253KiB/s
3909KiB/s

200mbit

7876KiB/s
7845KiB/s

The minimum bandwidth for 1 volume to work fine is little improved from the last result and is 100~120 Mbit
Multiple volumes

Volume/bandwidth
200 mbit/volume
250mbit/volume
300mbit/volume
350mbit/volume

2 volumes
x
x
✓ (Observed 1 rebuilding)
✓

I didn't test limiting network bandwidth when I tested #2206.    Upon analysis, the volume are failing with limited bandwidth because of ping timeouts.    With limited bandwidth, it is possible that system calls to write messages over the TCP socket would block.   The time computed for when the ping response should be received is computed before the system call to write the message over the TCP socket is completed.  https://github.com/longhorn/longhorn-engine/blob/870f06d50ece703aed65eb4db617c013bbdf3cd0/pkg/backend/remote/remote.go#L305
I want to see if the tests that fail with bandwidth limiting fail with a build with ping timeouts removed.  Maybe we don't need ping timeouts at all, or only ping when there are no read or write operations pending.

Sounds good. We need a ping when there is no traffic to keep the TCP connection alive at least. But that should be about it. We definitely don't need to ping if there are read/write operations.

It seems TCP keepalive is enabled by default. Well, we might not need ping at all then.
cc @joshimoo

Or can Longhorn guarantee a min network resource/split the resource for each connection? Otherwise, heavy r/w operations for an engine would affect/block the requests to other engines anyway.

I tested longhorn-engine without ping timeouts and the fio single volume write test at 10mbit networking passed for all the 256k-4m blocksizes.    The networking code doesn't explicitly use setsockopt to set the TCP keepalive to something other than the default.  If I'm reading this right, the default TCP keepalive timeout is 7200 + 75 * 9 seconds or 2 hours 11 minutes and 15 seconds:
`       tcp_keepalive_intvl (integer; default: 75; since Linux 2.4)
The number of seconds between TCP keep-alive probes.
   tcp_keepalive_probes (integer; default: 9; since Linux 2.2)
          The maximum number of TCP keep-alive probes to send before  giv‐
          ing  up  and  killing  the connection if no response is obtained
          from the other end.

   tcp_keepalive_time (integer; default: 7200; since Linux 2.2)
          The number of seconds a connection needs to be idle  before  TCP
          begins sending out keep-alive probes.  Keep-alives are sent only
          when the SO_KEEPALIVE socket option  is  enabled.   The  default
          value  is  7200 seconds (2 hours).  An idle connection is termi‐
          nated after approximately an additional 11 minutes (9 probes  an
          interval of 75 seconds apart) when keep-alive is enabled.

          Note that underlying connection tracking mechanisms and applica‐
          tion timeouts may be much shorter.`

@shuo-wu I think it might be hard. On the CPU and disk, it's decided by the scheduler which can be relatively easily influenced. I am not sure how network traffic is divide across the processes.

@keithalucas 2 hours... I guess we need something faster than that. I am in fact curious about why it will take so long? It cannot be only we want to detect the disconnection faster than 2 hrs.

I was considering the option to make the ping occur only when there are no read or writes in progress.  I think the problem with this could be that on a single node, an individual longhorn-engine controller process isn't aware of the other longhorn-engine controller processes running on the system.
Given that a single longhorn-engine controller process that is handling a lot of write requests (fio with blocksize of 256k and iodepth of 32) over a 10mbit network with fail to receive to ping responses from a replica within 8 seconds,  if pings were only performed when a longhorn-engine controller isn't handling read or writes, the longhorn-engine controller will still time out if there is another longhorn-engine controller on that node handling a rate of write requests that caused it fail with pings occurring all the time.  The other longhorn-engine process will be using so much bandwidth that the idle longhorn-engine process can't send a ping and receive a response within 8 seconds.
The idea of only performing pings when a longhorn-engine controller process is idle/non-performing IO is under the assumption when it is idle the node isn't so saturated with network traffic that it can send pings and receive ping responses.  However, there could be another volume and longhorn-engine controller process on the same node which saturates the network link between the controller and replica such that the idle controller receive a ping response from a replica within 8 seconds.  So unless an engine controller could determine that all other processes on the node aren't saturating the network link and only use ping then, it seems that pings could always fail.
So what are we trying to detect with ping?

A replica node's network is down.  In this case, longhorn-manager already will detect this through another mechanism and remove the replicas on this node from volumes that have a replica on this node.  I am curious if we could make kubernetes or longhorn-manager think that a node is down at a kubernetes level because there is so much controller to replica traffic that kubernetes services are unresponsive.
To determine that there is a bug in the replica process and disconnect it from the volume before a read or write occurs which will cause it to disconnect it via a read or write timeout.

I feel like the ping mechanism is not very useful.  The worst network bandwidth test was fio with a blocksize of 4m and an iodepth of 32 over a 10mbit network .  32 4MiB blocks is 128 MiB.  It should take 102.4 seconds to transmit over a 10mbit network<PHONE_NUMBER>4)/(101024*1024/8), without taking into account the ethernet/IP/TCP headers and headers in the longhorn protocol.   So maybe the ping timeout should be 2 minutes if we retain it to account allow worst network bandwidth test.

It looks like go sets its own default keep alive value of 15 seconds:  https://github.com/golang/go/issues/31510

@keithalucas
I agree with your analyze. Ping was mainly for:

Detect replica disconnection earlier. Though if it happens, read/write will also do it. So it's not really needed here.
Keep the connection alive, which should be covered by TCP keepalive anyway (even though the timeout is extremely long).

I am leaning towards removing the ping totally from Longhorn now.
Btw: data path shouldn't rely on Longhorn manager to handle failures. The data path failure should be handled by the engine alone.

It looks like go sets its own default TCP keep alive value of 15 seconds: golang/go#31510

Great! No reason to keep ping now. 😄

It looks like go sets its own default TCP keep alive value of 15 seconds: golang/go#31510

Great! No reason to keep ping now. 😄

I'm going to test removing the ping and using this default of 15 seconds.    I'm concerned that idle volume could timeout with another controller running on the same node that is using all the bandwidth.

Create two best effort volumes with three replicas and attach them to the same node.
Restrict the bandwidth of all the nodes to 10mbit.
Run fio with blocksize of 4m and iodepth of 32 on one of the volumes block device.
Verify that the other idle volume does not timeout because of TCP keepalive.

We can wrap up this after 1.2.0 out.

Need to wrap up after 1.2.1.

The CPU and RAM usage can be easily figured out using the scalability test framework with additional information about CPU and RAM usage of the instance manager. So I think we can do it together with the ticket https://github.com/longhorn/longhorn/issues/8627 in the milestone v1.8.0 WDYT? @derekbit @innobead


[FEATURE] Sorting and filtering item list

Sort and filter by name, category, value, room, etc...

This issue is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in a week.


Fixed bug with Azure DevOps unable to pull Yara rules

ADO does not end its repos with ".git" which causes a failure when trying to pull Yara rules from ADO.

Initial solution was to use git ls-remote to determine if it's a git repo or otherwise, however, less changes are required with this solution. If it goes against community coding standard I can rework the change.


Upgrade hotprotein framework from ESM1 to ESM2, and train it with deepspeed

Thank you for your great work, I am very interested in it.
I have managed to upgrade the hotprotein framework from ESM1 to ESM2, this is the repo
I finetune the esm2 models with d1_1_classification.pkl and HP-S2C2 data, then test the mean accuracy for 10 folders of HP-S2C2 dataset.

esm2_t6_8M_UR50D : 88.45%
esm2_t33_650M_UR50D: 93.74%
esm2_t36_3B_UR50D : 96.13%
ESM-1B + FST + Aug. + SAP( hotprotein paper  table1) : 92.36±0.58%

Also, I finetune the ESM2 models and test it with HP-S dataset.

esm2_t48_15B_UR50D:  77.31%
esm2_t36_3B_UR50D:  76.11%
esm2_t33_650M_UR50D:  74.11%
ESM-1B + FST + Aug. + SAP( hotprotein paper  table1) : 73.21±0.13 %

Then I have several questions:

Why do you train the esm1b and linear model with different loss, optimizer and scheduler.
I checked the ESM1b module attributes of each layers' weights, requires_grad is False. that means when we finetune classification function, we just train the linear layer only.

I try to train the hotprotein with deepspeed, now it supports distribute training with multiple GPU in FP16 dtype, but the accuracy is worse.
I only test esm2_t6_8M on HP-S2C2 dataset, the best Accuracy for d1_1_classification is only 84.26%, but I finetune esm2_t6_8M classification without deepspeed, it gets 90.36%.

If you have any suggestion, let me know.
Thank you very much.

Hi, Thank you for your integrating ESM2 work. I have some questions, as follows:
1.Does the key point SAP in the paper only provide the model parameters, such that it's difficult to integrate into esm2?
2.Why the accuracy of the bigger dataset HP-S lower than that of small dataset HP-S2C2? Does it means there is more overfitting problem in the small dataset, even the sequences have less than 50% identity according to the author?
I'll appreciate to you reply.

SAP
According to the Hotprotein paper, you can see how to train the SAP model:
We use AlphaFoldDB (Jumper et al., 2021) for SAP protein pre-training. We filter the data with sequence length and data quality and finally get 270K data. ESM-1B (Rives et al., 2019) and ESM-IF (Hsu et al., 2022) backbone are used to process the sequence and 3D coordinate inputs, and an average pooling layer is applied to the final-layer token representations of ESM models and get protein embeddings. A momentum encoder with τ = 1.0, momentum encoder coefficient α = 0.9999 and memory bank of size 65, 536 is used and the model is trained for 4 epochs, with AdamW optimizer, weight decay 10 −12 , batch size 512 and an initial learning rate 10 −6 decayed with OneCycle (Smith & Topin, 2019) decay scheduler.

So SPA is only compatible with ESM-1B, so when I train the classification, I just only use esm-2b pretrain module without SAP.
2.
HP-S
Total：182305
train：145844

label 0: 5120
label 1：28002
label 2：24286
label 3：63126
label 4：25310

test：36461

0:  1270
1： 6944
2：6047
3：15961
4：6239

HP-S2C5
Total: 1040
Train: 936

0: 64
1: 352
2: 304
3: 180
4: 36

Test: 104

0: 9
1: 35
2: 39
3: 16
4: 5

you see HP-S is much bigger than HP-S2C5 dataset. According to my training results, using a larger model of ESM2 can unearth more sequence information, thereby improving accuracy.
You are right, when I train the esm2_t33_650M  on HP-S2C5, it gets 93.75%, when I test it on HP-S, the accuracy is only 38.92%.

老哥好呀，我也在深圳。昨天简单看了你的repo，发现你只是把esm的代码照搬过来了，没有get到这篇论文的精髓。他ESM的模型参数是冻住了，但是在每个注意力层的q和v都添加了用于微调的LoRA层（这些层可学习的，Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[J]. arXiv preprint arXiv:2106.09685, 2021.），还有一些sparse层也是可以学习的。建议你看看这个仓库esm/module/Transformers以及esm/sparse_multihead_attention/SparseMultiheadAttention 呢。注意rank参数和use_sparse参数，这些都是微调相关的。祝好运！


Add Workflow file for Github Actions

Upload esplora-tapyrus binary

本PRでビルドしたバイナリがAmazon Linux 2で動作しない現象が発生。以下は実行時の出力
electrs: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by electrs)
electrs: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by electrs)
electrs: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by electrs)
electrs: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by electrs)
electrs: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by electrs)
electrs: /lib64/libc.so.6: version `GLIBC_2.30' not found (required by electrs)
electrs: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by electrs)
electrs: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by electrs)
electrs: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by electrs)

実行環境でのglibcのバージョンが古いのが原因
ec2-user@ip-10-0-1-149 bin]$ /lib64/libc.so.6
GNU C Library (GNU libc) stable release version 2.26, by Roland McGrath et al.


Add CSV Data Table rendering

see #19 and #12
use d3-dsv to parse csv data: https://github.com/d3/d3-dsv

needs restbook cell output mime type fix: https://github.com/tanhakabir/rest-book/issues/114

this should do for now:


help with %search%

I just started using this program and I'm having an issue with automatic directory naming.
For save path I have tried c:\images\%search% (also search_1, etc) to set my download location to the tag I searched for.  What's happening is it's saving to c:\images\%search% rather than, say, c:\images\witch
When naming files, %character%_(%artist%)_%md5%.%ext% works great.
I could be missing something obvious.  Any help?  Thank you.

Tokens are only allowed in the "filename" part.
So in your case, you would want to use:

Directory: c:\images
Filename: %search%\%character%_(%artist%)_%md5%.%ext%

If there's a / or \ in your filename, it will create sub-directories automatically.


Scale Anchor Misalignment in Bounce & Zoom Animation Parts

Some items that are meant to scale together are slightly misaligned when animating in. May be a precision error or transform-origin not being applied properly.
Primarily seen with link dots. Visible in Fashion & Fitness template on link pages.

https://github.com/google/web-stories-wp/pull/3112#issuecomment-656321512

not for stable as per @samitron7


Support Truffleruby in CI

@eregon I need your help here, not sure why but truffle ruby head version is not working for github actions

@eregon works now

@Jbern16 @davishmcclurg please review/merge

Thanks!


ViewController doesn't compile on Xcode 6.1

on Xcode 6.1 the code bellow on class ViewController doesn't compile
override func supportedInterfaceOrientations() -> Int {
    return Int(UIInterfaceOrientationMask.Portrait.toRaw())
}

Change to :
    return Int(UIInterfaceOrientationMask.Portrait.rawValue)

Sorry for taking so long – I appreciate you taking the time to file this. Thanks!


Add new redis commands

Overview
Add instrumentation for new redis commands.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.91%. Comparing base (60fe0b6) to head (ee15710).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1190      +/-   ##
==========================================
- Coverage   81.28%   78.91%   -2.37%
==========================================
  Files         196      196
  Lines       21736    21736
  Branches     3795     3795
==========================================
- Hits        17668    17154     -514
- Misses       2969     3490     +521
+ Partials     1099     1092       -7

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Open RTSP port from Addon

PLEASE READ THIS FIRST
Describe the issue
Could you please add an option to the addons configuration to open the rtsp port on the docker addon container? I saw how you would do it via a standard docker install, but not how it would be done via the addon. I would like to potentially send the stream to something like frigate for object detection. I know that it would need to have the user/pass configured on it, and the stream would only be local, not open to the internet.

I'm not sure I understand the request, the ability to expose the RTSP port via the addon is already there, it's just disabled by default since it's not needed for the local streaming case.  Just scroll down to the network section on the config page and put in whatever TCP port you want to forward.

I did not see that option there, I was looking in the configuration. Thanks, this is exactly what I needed.


add block v3 API

PR Description
Add ProduceBlock API /eth/v3/validator/blocks/{slot}
Link to the spec

Add new /eth/v3/validator/blocks/{slot} endpoint
Deprecate GetNewBlock /eth/v2/validator/blocks/{slot}
Deprecate GetNewBlindedBlock /eth/v1/validator/blinded_blocks/{slot}

Fixed Issue(s)

fixes #7390
Documentation

[x] I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

[x] I thought about adding a changelog entry, and added one if I deemed necessary.

Missing parts to implement in a separate PR:

Implement ProduceBlockRequest (Like the current CreateBlockRequest) and add the new endpoint response deserialisation
Make the BlockProductionDuty use the new createUnsignedBlock(UInt64 slot, BLSSignature randaoReveal, Optional<Bytes32> graffiti) (without passing the blind parameter)


Add Button to retrieve and render current data

In the past, Streets GL used current data from the Overpass API. This ensured rapid verifiability of one's edits.
This was an essential part of my editing workflow, because verification is essential for complex buildings. Currently, I feel unable to edit complex buildings because I cannot check the results in Streets GL. (Admittedly, I do this also for pleasure of viewing it there.)
As I understood it (from here https://github.com/strandedkitty/streets-gl#-data-sources), a vector tile cache is implemented and used instead, that is updated only once a week. This seems appropriate for many users and uses, but not for verification of edits.
Therefore the enhancement request:

Add a preference button to retrieve data (as before) from Overpass API.

If an on-the-fly tile update could be achieved that would be marvellous in replacement of the Ctrl+P key now archived. 1 tile at the time, or maybe the tile at centre of screen plus the 8 around it... don't know the area of a tile.
(I've reverted to F4 to check buildings, albeit the gloss is missing. New adds work near immediate, changes not so much.)


Compiles fine, but wont run?

Compiles without errors, Server is installed and running, compiled binary is a little over 10mb
When I run UpdateService.exe "http://payload.server.URL/ENCRYPTION KEY" nothing happens, when I try browsing to "http://payload.server.URL/ENCRYPTION KEY"  I get the 1st stage as encoded test in the browser (reloading rick rolls)
Seems to compile fine, although debugging throws this error
UpdateService.exe' has exited with code 0 (0x0)
tried opening FW ports, nothing.  Nothing being blocked by AV?
Victim,
Newest/updated Win10 pro
Compiled with VS community  15.8.1
Attacking machine, Kali-rolling

Compiled again using VS15


AppWorkload has a resources field like TaskWorkload

Is there a related GitHub Issue?
 #1408
What is this change about?
 See issue
Does this PR introduce a breaking change?
 Yes. Existing deployments will break due to the change to AppWorkload
Acceptance Steps
See issue
Tag your pair, your PM, and/or team
 @gcapizzi
Things to remember

Neat!


FIX: Remove the extra scrollbar on the left navbar

Resolves #624
🔧 What changed
The left navbar had a double scrollbars. I have removed the outer scrollbar, and kept the inner scrollbar so the Filters are always visible while scrolling through the doctors.

Removed unnecessary overflows
Removed unnecessary hover effect

🧪 Testing instructions
📸 Screenshots

Before

After

@Anissa3005 I don't see the double scrollbars that you're showing in your screenshot. 🤔  When I visit the site in production, this is what I see:

@theyokohamalife I use chrome, on my laptop screen. In production there are two scroll bars. There were three overflows applied so I think that was anyway unnecessary.

I'd like to confirm if other people are experiencing this issue in production or not before making these changes.

I've also seen in prod

I'd like to confirm if other people are experiencing this issue in production or not before making these changes.
I am experiencing these issues. It could be an OS thing with how scrollbars are applied. Like the discussion I had with Natalja one time about how windows and macs apply scrollbars different

@theyokohamalife  can you re-review?


unformat with the decimal specified to deal with decimal for thousands

Signed-off-by: Brad Midgley<EMAIL_ADDRESS>formatMoney will fail on currencies that have a dot for the thousands separator.
fixes: https://github.com/openexchangerates/accounting.js/issues/129

After looking into this issue it is apparent that the intended usage of formatMoney does not securely allow for number to be passed as a string. Numbers which use "." as a thousand separator must be wrapped in quotes, since JavaScript will not correctly interpret a number such as 1.234,56. The proposed fix may not be reliable since options to formatMoney can be passed either individually:
formatMoney( number, "R$", 2, ".", ",", "%s %v" ); // 6 arguments
... or, as an object:
formatMoney( number, {
  symbol: "R$",
  precision: 2,
  thousand: ".",
  decimal: ",",
  format: "%s %v"
} ); // 2 arguments

I believe the best work around for currencies which use ".' as a thousand separator is this two step process:
// Manually unformat the number passed as a string:
var number = accounting.unformat( "1.234,56", "," ); // number => 1234.56

// Pass the unformatted number to formatMoney with appropriate options:
accounting.formatMoney(number, {
  symbol: "R$",
  thousand: ".",
  decimal: ",",
  format: "%s %v"
}); // "R$ 1.234,56"


Java 8 not supported?

While using smartid client in java 8 application JRE throws error:
Caused by: java.lang.UnsupportedClassVersionError: com/sun/istack/FinalArrayList has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 (unable to load class [com.sun.istack.FinalArrayList])
This dependency is from jaxb.runtime 4.0 and if I understood correctly its java 11+ only.
I got application running with following gradle dependency setup:
implementation('ee.sk.smartid:smart-id-java-client:2.3') {
exclude group: 'org.glassfish.jaxb', module: 'jaxb-runtime'
}
implementation('org.glassfish.jaxb:jaxb-runtime:3.0.2')
implementation('jakarta.activation:jakarta.activation-api:2.1.0')
implementation('org.eclipse.angus:angus-activation:1.0.0')
however now runtime error is thrown:
java.lang.ClassNotFoundException: Provider for jakarta.ws.rs.client.ClientBuilder cannot be found
at jakarta.ws.rs.client.FactoryFinder.find(FactoryFinder.java:168) ~[jakarta.ws.rs-api-3.0.0.jar:3.0.0]
at jakarta.ws.rs.client.ClientBuilder.newBuilder(ClientBuilder.java:62) ~[jakarta.ws.rs-api-3.0.0.jar:3.0.0]
at ee.sk.smartid.rest.SmartIdRestConnector.prepareClient(SmartIdRestConnector.java:180) ~[smart-id-java-client-2.3.jar:2.3]
at ee.sk.smartid.rest.SmartIdRestConnector.postRequest(SmartIdRestConnector.java:258) ~[smart-id-java-client-2.3.jar:2.3]
at ee.sk.smartid.rest.SmartIdRestConnector.postAuthenticationRequest(SmartIdRestConnector.java:233) ~[smart-id-java-client-2.3.jar:2.3]
at ee.sk.smartid.rest.SmartIdRestConnector.authenticate(SmartIdRestConnector.java:168) ~[smart-id-java-client-2.3.jar:2.3]
at ee.sk.smartid.AuthenticationRequestBuilder.getAuthenticationResponse(AuthenticationRequestBuilder.java:360) ~[smart-id-java-client-2.3.jar:2.3]
at ee.sk.smartid.AuthenticationRequestBuilder.initiateAuthentication(AuthenticationRequestBuilder.java:298) ~[smart-id-java-client-2.3.jar:2.3]
at ee.sk.smartid.AuthenticationRequestBuilder.authenticate(AuthenticationRequestBuilder.java:282) ~[smart-id-java-client-2.3.jar:2.3]
I could not figure out what provider jar is still missing.
But is smartid java client still java 8 compatible?

I managed to resolve error "java.lang.ClassNotFoundException: Provider for jakarta.ws.rs.client.ClientBuilder cannot be found" - missing dependency
org.glassfish.jersey.connectors
jersey-apache-connector
This includes it as test dependency but its also required on runtime. This also needs to be fixed.


Add support for custom differ

Add support for custom differ
I have the need for my CI workflow to output custom diffs. I was quite surpised to see that this was not possible even though a DifferInterface exists.
Usage
The idea is that anyone willing to use a custom differ can create one in his project and then specify it in the diff-format argument.
vendor/bin/php-cs-fixer fix --diff-format="\MyNamespace\MyLib\MyCustomDiffer"

What's next ?
Tell me if any change is needed to get this (hopefully) merged :)

Regarding the phpstan error. I've never used this tool so have no clue what the error is about 🤷♂
If you could point me in the right direction that would be cool.

Hi and thanks for the suggestion!
I like the proposed change, however can you add a check that $option is a string? Like;

if (is_string($option) && is_subclass_of($option, DifferInterface::class)) {

it will prevent people passing an instance of a differ already and causing a crash on the next line.
I think the SCA might be an issue with PHPStan itself TBH, lets see what it does after the added check!

So I've tried to add the is_string check inside the if but I had an error on the elseif because it was tryng to use the object as an index.
I therefore went for a type check right before the first if. I thought about putting it before that would have led to this :
if (isset($options['diff-format'] && !is_string($options['diff-format'])))

Which I found a bit too verbose.

Thanks for the updates, the PR looks good to me :)

Oh by the way, I also have an implementation that supports passing the diffed file to the differ. If that's of any interest to you I can make a PR for that one too :)

You mean on update for DifferInterface.php to pass along the filename as well? That is in the planning for 3.0 so would be welcome there :+1:

This looks like a new feature to me, I would merge this into master.

Do you want me to change the target branch to master then ?
@SpacePossum I'll have to polish a bit that implementation as it is really rough and open a pull request then ;)

nice, please retarget to master :+1:
on the other one, feel free to open a rough first version so we might understand a bit better what it will be like etc., but no rush on it at all :)

It seems that the travis build is failing again now that I have rebased my branch on master. An idea of what the issue might be?

You have to wait for https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/4966

baseline has been revert, sorry for the noise!

@SpacePossum No problem, I've rebased my work on master to make the check pass 😉
Oh by the way the PR to add support for passing the file to the differ is on its way

Well it looks like I've broken someting, I'll find what it is

@paulhenri-l you did not, the same issue is on other PRs. phive makes a mess again.

Ok so it appears that master is still broken, should I rebase on another branch?

please don't, it looks good :)

@paulhenri-l can you fix the project itself (php php-cs-fixer fix), rebase and squash the commits? After that it looks good to me merge to me

@SpacePossum Sure I'll try to do that today

It should be ok now

@paulhenri-l almost there, only 1 failing test.
If you move FakeDiffer to PhpCsFixer\Tests\Fixtures namespace it would be fixed.

@kubawerlos I'll try to do that today

Thank you @paulhenri-l.


chore: reduce gas price multiplier

Description
This change changes the oracle client gas price multiplier to ~2~ 1.5.
Other changes

N/A

Tested

Updated config locally and verified setting overrides the default value.

Related issues

Closes https://github.com/mento-protocol/mento-general/issues/165

Backwards compatibility

N/A

Documentation

N/A

@nvtaveras @bayological I'm also changing this in the PR for USDCEUR


datasheet for the Broadcom B50612D

Any idea where i can find the datasheet for the ethernet phy Broadcom B50612D used on this board?

I know there is one for  B50610 but I was not able to download it anywhere. http://www.51hei.com/bbs/dpj-122718-1.html

B50612D at least exists on chinese resources like https://download.csdn.net/download/mydream_sky/11088168
But I can't to download it.

The B50612D data sheet is available at https://github.com/wuxx/Colorlight-FPGA-Projects/doc

@derekmulcahy indeed!
Thanks for sharing and it seems google not so good indexing github


All fonts track Linux Libertine serifs

Description
In the reference image below, the left edge of each text block using default font (Linux Libertine) lines up exactly with the left edge of its container.
The text blocks using other fonts don't, instead leaving room for LL's large serifs.
This is most obvious in the two sans fonts tested (Arial and Segoe UI), but it appears to be true of other serif fonts as well.
The misalignment of the left edge is clear, but it feels like it could be affecting the tracking throughout the text.
Apologies if this is a known issue. I couldn't find it in the repo, but I don't have the font design terminology to be confident it hasn't already been flagged.
Reference Image

Typst code
Default serif

#box(fill: rgb("#cff7ff"), width: 100%)[
  #block(text(size: 50pt)[III (50pt)])
  #block(text(size: 28pt)[III (28pt)])
  #block(text(size: 16pt)[III (16pt)])
  #block(text(size: 11pt)[III (11pt)])
]

#v(20pt)

#text(font: "Garamond")[Garamond]

#box(fill: rgb("#cff7ff"), width: 100%)[
  #block(text(font: "Garamond", size: 50pt)[III (50pt)])
  #block(text(font: "Garamond", size: 28pt)[III (28pt)])
  #block(text(font: "Garamond", size: 16pt)[III (16pt)])
  #block(text(font: "Garamond", size: 11pt)[III (11pt)])
]

#v(20pt)

#text(font: "Arial")[Arial]

#box(fill: rgb("#cff7ff"), width: 100%)[
  #block(text(font: "Arial", size: 50pt)[III (50pt)])
  #block(text(font: "Arial", size: 28pt)[III (28pt)])
  #block(text(font: "Arial", size: 16pt)[III (16pt)])
  #block(text(font: "Arial", size: 11pt)[III (11pt)])
]

#v(20pt)

#text(font: "Segoe UI")[Segoe UI]

#box(fill: rgb("#cff7ff"), width: 100%)[
  #block(text(font: "Segoe UI", size: 50pt)[III (50pt)])
  #block(text(font: "Segoe UI", size: 28pt)[III (28pt)])
  #block(text(font: "Segoe UI", size: 16pt)[III (16pt)])
  #block(text(font: "Segoe UI", size: 11pt)[III (11pt)])
]

Reproduction URL
No response
Operating system
Windows
Typst version

[X] I am using the latest version of Typst

I think it's rather a quirk of the sans-serif fonts themselves, as it looks the same in plain HTML (with Arial):

Good to know!
In case its useful to anyone, I found a workaround that adds a negative left inset to the heading block.

#let sans-heading(x, font: "Arial", size: 11pt) = {
  let i = (size / -11)
  block(
    inset: (left: i),
    text(
      font: font,
      size: size,
      x
    )
  )
}

#box(fill: rgb("#cff7ff"), width: 100%)[
  #for size in (11pt, 16pt, 28pt, 50pt) {
    sans-heading(size: size)[III (#size)]
  }
]


⚠ Secret Website 1 has degraded performance

In ac47691, Secret Website 1 ($SECRET_WEBSITE_1) experienced degraded performance:

HTTP code: 200
Response time: 6057 ms

Resolved: Secret Website 1 performance has improved in fb9f68f.


Duplicate error message for exceptions

I don't like the repetition in the representation of an "unexpected exception" very much.

Example: submission 7528170
Originally posted by @pdawyndt in https://github.com/dodona-edu/universal-judge/issues/202#issuecomment-823237503

Do you suggest to only display the message one, for each exception with stacktrace?

The stack trace is indeed more informative this case. There's no "expected exception" to compare with whatsoever. I don't know if  placng the stack trace in the "generated exception" would look good, but we could give it a try.

We added the message to allow linking the file lines in the stacktrace.
Placing it in the "generated exception" will lose this functionality.

While theoretically it's nice to have different channels and actual/expected values, I think it's not that intuitive, e.g. (an example from the js judge):

I would propose omitting the exception feedback channel (except when there is an expected exception of course) and just showing the error with the stack trace, like the Python judge:

(I'm unsure why the Python judge doesn't have the gray-ish box around the stracktrace, because that looks a lot nicer.)

Ideal would be to have the “exception” label of the test (in red) followed by the stack trace (without grey box), but no comparison with expected. Alas, think Dodona does not support tests with only a message for the moment.

Duplicate issue: #121